Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[release/9.0-staging] Remove thread contention from Activity Start/Stop #109359

Merged

Conversation

github-actions[bot]
Copy link
Contributor

@github-actions github-actions bot commented Oct 29, 2024

Fixes #109060

Backport of #107333 to release/9.0-staging

/cc @tarekgh @AlgorithmsAreCool

Customer Impact

  • Customer reported
  • Found internally

Users running tracing in high-throughput scenarios may experience lock contention, which can lead to performance degradation in their applications or services.

  • The issue was reported by the ASP.NET/Aspire team while enabling CRUD operations over a PostgreSQL database with OpenTelemetry, resulting in a 15% drop in requests per second. Benchmark numbers are available in dotnet/runtime issue #107158.
  • A customer also reported that in their high-throughput services, lock contention was contributing significantly (around 10–20%) to service latency. The issue Consider releasing #107333 into dotnet 9 #109060 has more details and benchmark numbers.

Regression

  • Yes
  • No

While this isn't a regression, it appears that users have started enabling scenarios that lead to this issue. We may expect this get reported more in the near future if we don't service this fix.

Testing

Manual testing, #107333 (comment) has some numbers before and after fix. Also, we have provided a private built package to the customer who reported the issue, and they have confirmed the fix resolve their issue. #109060 (comment)

Risk

Medium to low risk. The whole change is touching the type we use for caching which was causing the lock contention. The fix didn't change any actual functional behavior more than avoiding the lock when notifying the listeners.

IMPORTANT: If this backport is for a servicing release, please verify that:

  • The PR target branch is release/X.0-staging, not release/X.0.

  • If the change touches code that ships in a NuGet package, you have added the necessary package authoring and gotten it explicitly reviewed.

Copy link
Contributor

Tagging subscribers to this area: @dotnet/area-system-diagnostics-activity
See info in area-owners.md if you want to be subscribed.

@tarekgh tarekgh added the Servicing-consider Issue for next servicing release review label Oct 29, 2024
@tarekgh
Copy link
Member

tarekgh commented Oct 29, 2024

CC @ericstj @artl93

@rbhanda rbhanda modified the milestones: 9.0.x, 9.0.1 Nov 5, 2024
@rbhanda rbhanda added Servicing-approved Approved for servicing release and removed Servicing-consider Issue for next servicing release review labels Nov 5, 2024
@tarekgh tarekgh merged commit 409f6a4 into release/9.0-staging Nov 5, 2024
85 of 86 checks passed
@tarekgh tarekgh deleted the backport/pr-107333-to-release/9.0-staging branch November 5, 2024 21:09
carlossanlop pushed a commit to carlossanlop/runtime that referenced this pull request Nov 7, 2024
…op (dotnet#109359)

* Remove thread contention from Activity Start/Stop

Author:    algorithmsarecool <[email protected]>

* Fix ref parameters and whitespace

Author:    algorithmsarecool<[email protected]>

* PR feedback. - Reduce duplication - add comments and make code more obvious - Use IndexOf

Author:    algorithmsarecool <[email protected]>

* PR feedback to simplify locking strategy

* PR feedback, final nits

* package authoring

* Fix package authoring

* revert package authoring as it is not needed anymore in net9.0

---------

Co-authored-by: algorithmsarecool <[email protected]>
Co-authored-by: Tarek Mahmoud Sayed <[email protected]>
@carlossanlop carlossanlop modified the milestones: 9.0.1, 9.0.2 Nov 21, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Dec 22, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants