Add exemplar support to Prometheus exporter

[saul]

Fixes #3449
Design discussion issue - not found

Changes

Adds support for exporting histogram exemplars to OpenTelemetry.Exporter.Prometheus.AspNetCore and OpenTelemetry.Exporter.Prometheus.HttpListener. There are a couple of aspects that I'd appreciate some guidance on:

• The spec states "If no exemplars exist on a bucket, the highest exemplar from a lower bucket MUST be used, even though it is a duplicate of another bucket’s exemplar" but my implementation doesn't follow this. I'm wondering what the rationale is behind this as for large histograms I can imagine this would produce a lot of extraneous data.
• The spec states "For Prometheus pull endpoints, only a single exemplar is able to be added to each bucket, so the largest exemplar from each bucket MUST be used, if attaching exemplars" but exemplars as implemented in OTel.Net do NOT preserve the largest exemplar in each bucket on histograms. Simply the most recent value for that bucket is stored as the exemplar, regardless of whether it is larger than the current exemplar or not
• Do the exemplars need to be 'wiped' between scrapes of the /metrics endpoint? Or should the exemplars be persistent in that once a bucket has an exemplar, it will always be returned by the /metrics endpoint until it is replaced by another exemplar.
• Is this considered an API-breaking change? I wonder if it would be best to add a bool option to PrometheusAspNetCoreOptions to enable exemplars, with the default being disabled (i.e. current behaviour)
  • "Prometheus Exporters are not stable yet, so no concern about breaking changes here." @cijothomas

Merge requirement checklist

• CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
• Unit tests added/updated
• Appropriate CHANGELOG.md files updated for non-trivial changes
• Changes in public API reviewed (if applicable) - see question above re adding a PrometheusAspNetCoreOptions option

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: cijothomas / name: Cijo Thomas (4261b1d)
• ✅ login: saul / name: Saul Rennison (606b69e, 9cac3e7, 8550c31, 24b2c6a)
• ✅ login: Kielek / name: Piotr Kiełkowicz (0b0e001, 725a730, bf24a25, bb24629, 86cb1c7)

[cijothomas]

Is this considered an API-breaking change? I wonder if it would be best to add a bool option to PrometheusAspNetCoreOptions to enable exemplars, with the default being disabled (i.e. current behaviour)

I think you can enable it by default, given Exemplars itself is off-by-default in the SDK! So if prometheus exporter is exporting exemplars, users has already made an explicit decision to enable it at the sdk.
Another thing - Prometheus Exporters are not stable yet, so no concern about breaking changes here.

[cijothomas]

• ❌ - login: @saul / name: Saul Rennison . The commit (fe78b00, a609c15) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

@saul Can you sign the CLA part, as it is a mandatory pre-req for accepting any contributions to OpenTelemetry. If you are working on behalf on an employer, you may want to consult your employer before CLA. (Note: This is a one time requirement only)

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[saul]

Hi @cijothomas the CLA has been signed and this is ready for review when you have time. Thanks

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[cijothomas]

@robertcoltheart Tagging you here, to see if you have some bandwidth to help review this PR, given you have made several contributions to the Prometheus Exporter!

[cijothomas]

The spec states "If no exemplars exist on a bucket, the highest exemplar from a lower bucket MUST be used, even though it is a duplicate of another bucket’s exemplar" but my implementation doesn't follow this. I'm wondering what the rationale is behind this as for large histograms I can imagine this would produce a lot of extraneous data.

I am not sure of the reasoning for this either...You can put a TODO in the code and state this is not implemented, and we can follow up with the spec.

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[github-actions]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

[saul]

Hi team - this PR isn't stale, it's ready to merge - please can I get a review? If you let me know what the hold up I can work through any issues

[cijothomas]

@saul It looks like some comments are already left and are unresolved? Could you take a look at them and comment/resolve.

(Unfortunately, I won't have bandwidth to do full review, Tagging @CodeBlanch @Kielek @rajkumar-rangaraj to help review.)

[Kielek]

I will start with reopening this PR. It allows us to occur on typical board and notifications.

[Kielek]

@saul, I fixed 2 obvious issues, but your changes are still failing CI pipeline. Could you please investigate what is the issue and fix it?

[saul]

I believe I've fixed the failing test now.

Unfortunately when I push, my commits don't seem to trigger CI - do I need to manually trigger CI?

[Kielek]

Unfortunately when I push, my commits don't seem to trigger CI - do I need to manually trigger CI?

First contribution requires approval to execute pipeliene (until PR is merged). Any furter changes will trigger pipeline automatically.

CI triggered.

[saul]

Thanks for your help @Kielek. From what I can see all of the checks have passed now.

[Kielek]

@robertcoltheart, do you have some time to make full review of this PR?