sort ingesters when generating token lookup table fix #6513 #6535
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
January 23, 2025 14:36
Signed-off-by: Daniel Sabsay <[email protected]>
Signed-off-by: Daniel Sabsay <[email protected]>
dsabsay
force-pushed
the
flaky-ring-consistency-test-fix
branch
from
2b87236 to
28860c7
Compare
|
Converting to draft because after rebasing |
|
Okay, repro is behaving as expected again (failed twice in a row, passed twice in a row with this fix). This is ready for review. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Sort ingesters when generating token lookup table in ring to fix consistency issues.
Since map iteration order is undefined, successive calls to
getTokensInfo()can return different results when tokens are owned by multiple instances in the ring. I don't know enough about the ring to say whether that is expected to occur, but in the tests it does and this fixes that test.See the linked issue for more details and here for a reproducible test: https://github.com/dsabsay/cortex/tree/flaky-ring-consistency-test-repro
I validated the test locally by running my reproducer (linked above) twice (failed both times), applying this patch and running the same test (passed both times). I don't think it's worth merging the reproducer as it takes 5mins to run utilizing 16 cores fully.
Which issue(s) this PR fixes:
Fixes #6513
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]