Traces and Metrics gems for instrumenting libraries.

[ioquatix]

I want to move the expectations of tracing and instrumentation out of a gem like OTEL (Ruby) and into the actual libraries. My goal is for it to retain zero overhead as the current implementations achieve, but also adopt a flexible model for adding instrumentations.

As such, I've created and released two new gems which I'm hoping to develop in this direction:

• https://github.com/socketry/metrics - a light weight interface for reporting metrics.
• https://github.com/socketry/traces - a light weight interface for capturing traces.

These two gems have zero overhead if not in use, but allow library authors to opt into explicitly providing metrics and traces on a class-by-class basis.

To use this gem, a backend must be used. An example of a backend would be OTEL. We will provide some basic testing backends but ideally the backends will be provided by specific gems, e.g. traces-backend-open_telemetry.

[ahayworth]

@ioquatix What you're describing is what the OpenTelemetry API itself is supposed to accomplish. Ideally, library authors (like yourself!) would instrument their application with the OpenTelemetry API. When no SDK is configured, everything is essentially a no-op, and nothing really happens up until the point that a user of your library wires up the SDK (and then boom, telemetry!).

(And of course, users can configure their OpenTelemetry SDK instance to point to whatever backends they'd like.)

I was curious if you'd tried that approach or not? If the ruby implementation of the API is lacking, I think we would want to try and fix its shortcomings rather than create another API that is doing something so similar?

(Regardless, I am excited to have people working in this space! 😄 )

[ahayworth]

I had a few specific thoughts about what you mentioned:

OpenTelemetry does not have a zero-overhead-if-not-enabled interface AFAIK.

This is certainly one of the goals of the project. I don't know that the ruby opentelemetry libraries are necessarily zero overhead (there's always a tiny penalty for method dispatch, etc - even in other tracing systems as well), but I think we're certainly interested in getting closer to zero. As it stands today, using the ruby API but not the ruby SDK results in a low-overhead no-op right now.

OpenTelemetry pulls in a lot of instrumentation by default which isn't always appropriate.

This is true if you use the opentelemetry-instrumentation-all gem, which is certainly what we tend to push in our documentation. But it's not necessary at all; you can select individual instrumentation by installing/requiring just those specific instrumentation libraries (or none at all!).

We have test backends which validate traces and metrics behaviour and data models. This allows you to run tests with traces and metrics enabled and validate they would behave correctly with any supported backend

I think a testing backend is something that would be welcomed and supported here. We do have some ability to do testing around what traces are generated, but it could be made easier.

---

I won't respond to every single point in detail other than to say "I agree with some of what you're saying" and "it seems like you have different design goals than the opentelemetry project." And that's totally okay, and we welcome cross-collaboration regardless!

Thank you for sharing the libraries you're working on!

[ioquatix]

OpenTelemetry does not have a zero-overhead-if-not-enabled interface AFAIK. - This is certainly one of the goals of the project. I don't know that the ruby opentelemetry libraries are necessarily zero overhead.

Yes, there are really two options here:

• Only load the monkey patch if you want instrumentation (by not loading it, you don't get it, which is by definition zero overhead), OR
• Use meta-programming to only apply the monkey patch if instrumentation is enabled.

The model followed by OpenTelemetry appears to be essentially load the monkey patch, but the patch itself is as close to a no-op as possible if tracing is disabled. The problem with this is you still pay some overhead cost - whether it matters in practice I'm not sure.

The traces gem family uses both approaches - only load what you want to trace and only applied if tracing is enabled - if tracing is disabled, no instrumentation or monkey patches are applied. It could be relatively straight forward for OpenTelemetry use the same model.

[arielvalentin]

Ideally I think what we would hope to eventually see is first party integration in gems or some way to have native instrumentation as part of the core VM spec & implementations.

Given your experience working with the core team is this something that you think would be feasible?

[fbogsany]

To be clear, there's a distinction between the OpenTelemetry API and the OpenTelemetry auto-instrumentation gems. The auto-instrumentation gems generally use monkey-patching, and only load and apply the monkey-patches if the instrumentation is enabled. Requiring but disabling some auto-instrumentation will be zero overhead.

The same is not true of explicit tracing in first-party instrumentation. There will be some overhead to this, for example, even in the absence of the SDK:

tracer.in_span('foo', attributes: { 'bar' => bar }) do
  ...
end

This involves several method calls, a few allocations, exception handling, and fiber-local context lookup. This is a convenience method, though, to make common use-cases easy. It is possible to avoid most of this at the cost of explicit context management, which might be preferable in storage client libraries, for example.

Alternatively, the model used by the auto-instrumentation gems could be used for first-party instrumentation. The downside of this is that instrumentation is separate from the code it instruments, which is a little more verbose and harder to maintain. The tradeoff may be worth it some cases.

[ioquatix]

Ideally I think what we would hope to eventually see is first party integration in gems

I thought a lot about this and the conclusion I came to is it's almost always better to have a separate gem with tracing/instrumentation.

or some way to have native instrumentation as part of the core VM spec & implementations.

At one point I was also thinking about doing this. However, it turns out a library like traces can get 90% of the way there with good backwards compatibility. What we are missing is tracing around core IO operations, but the fiber scheduler can help with this.