KAFKA-18618: Improve leader change handling of acknowledgements

[AndrewJSchofield]

There is a problem in the share consumer where records fetched from node A might be acknowledged with node B, if node B used to be the leader of the topic-partition and A becomes unavailable.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[clolov]

I know this is still marked as a Draft, but how difficult will it be to add a test which tests the edge case which uncovered this? I believe I can follow the reasoning otherwise i.e. you only want to send the acknowledgements to the nodes from which the in-flight records originated.

[AndrewJSchofield]

I know this is still marked as a Draft, but how difficult will it be to add a test which tests the edge case which uncovered this? I believe I can follow the reasoning otherwise i.e. you only want to send the acknowledgements to the nodes from which the in-flight records originated.

Not that difficult. Might be a follow-on PR to complete the testing, but we have two pieces of work in progress. First, we are developing tests using MockClient which can pretend to be a cluster of multiple brokers in which we can very deliberately introduce specific changes of leadership to exercise all of the code paths. Then, we are starting to run system tests which roll the cluster while performing a long-running workload. It was a rolling restart system test that showed the problem initially.

[clolov]

Awesome! Looking forward to seeing those 😊