manually propagating trace & span

[tobert]

I'm adding tracing to a legacy Ruby project and am kinda stuck. The app has a custom worker queue based on Postgres. Fortunately the worker data structure had a place I could stash the trace id & span id in hex. I've retrieved them on the worker and am having trouble creating a new span around doing the work.

I'm still learning all the otel terminology and inconsistencies, so apologies if I'm missing something obvious here...

Here's a small test script the reproduces the error, doing what seems like the obvious thing, but maybe wrong.

#!/usr/bin/env ruby

require 'opentelemetry/sdk'

tracer = OpenTelemetry.tracer_provider.tracer("test-application", "0.1")

hex_trace_id = "636231f67e0be6c1097b083fd5029ea7"
hex_span_id = "abff042c9a17b06b"

tid = [hex_trace_id].pack('H*')
sid = [hex_span_id].pack('H*')

parent = OpenTelemetry::Trace::SpanContext.new(trace_id: tid, span_id: sid)

tracer.start_span("test", with_parent: parent) do |span|
    sleep 0.1
end

gets

tobert@hostname ~/src/project $ ruby test2.rb
Traceback (most recent call last):
	2: from test2.rb:15:in `<main>'
	1: from /home/tobert/.rbenv/versions/2.5.8/lib/ruby/gems/2.5.0/gems/opentelemetry-api-0.15.0/lib/opentelemetry/trace/tracer.rb:52:in `start_span'
/home/tobert/.rbenv/versions/2.5.8/lib/ruby/gems/2.5.0/gems/opentelemetry-api-0.15.0/lib/opentelemetry/trace.rb:62:in `current_span': undefined method `value' for #<OpenTelemetry::Trace::SpanContext:0x000055cd61f01f98> (NoMethodError)

[tobert]

OK I see that an OpenTelemetry::Context is expected, but cannot find how to actually create one?

Is the answer "use a propagator or get out?"

[fbogsany]

You likely want to write your own Getter and Setter and then use the global OpenTelemetry.propagation to extract the context, then use it to start a child span. That's the golden path, but you can hack it...

Your example is breaking because with_parent: expects an OpenTelemetry::Context rather than an OpenTelemetry::Trace::SpanContext. You could "fix" it by wrapping the SpanContext, like:

sc = OpenTelemetry::Trace::SpanContext.new(trace_id: tid, span_id: sid)
parent_span = OpenTelemetry::Trace:Span.new(span_context: sc)
parent = OpenTelemetry::Trace.context_with_span(parent_span)
tracer.start_span("test", with_parent: parent) do |span|
    sleep 0.1
end

[tobert]

Awesome, this is exactly what I was looking for, thank you :)

Also TIL the difference between start_span and in_span. I think in my example it should have been tracer.in_span for that to work, and indeed it does in my test code.

[fbogsany]

The golden path is:

parent = OpenTelemetry.propagation.extract(carrier, getter: my_getter)
tracer.start_span("test", with_parent: parent) do |span|
    sleep 0.1
end

[fbogsany]

If your carrier is using a unique propagation format, then you can instead use your own propagator rather than what you're using for, e.g., HTTP requests:

parent = my_propagator.extract(carrier)
tracer.start_span("test", with_parent: parent) do |span|
    sleep 0.1
end

[ahayworth]

👋 I'm doing something a little similar to this with a background job, and I figured I'd leave it here as a note. The general approach I'm taking is:

1. Stuff the current context into a text map before we enqueue the job.
2. Pass that map as an argument to the job as its enqueued.
3. On the other side, inflate a context object from the job's arguments.
4. Do the work of the job wrapped inside a span (setting the span's parent_context to the context object we just retrieved).

I'm doing it in ActiveJob, and so I leverage the callbacks to do this automatically for every job, like so:

class ApplicationJob < ActiveJob::Base
  around_enqueue do |job, block|
    # Fetches the current context (if any) and turns it into an easy-to-work-with map of string keys/values:
    context_object = OpenTelemetry.propagation.inject({})  # The hash is the "carrier" for propagation.
    # You could also be more explicit if you aren't using the default text_map_setter and text_map_getter in your app:
    # context_object = OpenTelemetry.propagation.inject({}, setter: OpenTelemetry::Context::Propagation.text_map_setter)
    
    job.arguments.push(context_object)
    block.call
  end
  
  around_perform do |job, block|
    # The last argument is our text map we passed along with the job.
    context_carrier = job.arguments.pop
    
    # We rehydrate that into a parent context object:
    parent_context = OpenTelemetry.propagation.extract(context_carrier)
    # Again, being explicit with getters and setters if our environment has changed the defaults:
    # parent_context = OpenTelemetry.propagation.extract(context_carrier, getter: OpenTelemetry::Context::Propagation.text_map_getter)
    
    # Do the work of the job wrapped inside a span:
    OpenTelemetry.tracer_provider.tracer("my-app", "0.1").in_span("span-name", with_parent: parent_context) do |span|
      block.call
    end
  end
end

Since this is ActiveJob, I have handy callbacks available around enqueuing and performing the job. And I know that Rails is able to serialize and deserialize a string map as a job argument for me - so it's rather tidy.

However, you should be able to adapt the example to your queuing system, as long as you're able to somehow serialize/deserialize additional data as a plain text map, like I did.

[ahayworth]

Modifying your test script directly, presuming we'd serialized things as described above beforehand:

#!/usr/bin/env ruby

require 'opentelemetry/sdk'

tracer = OpenTelemetry.tracer_provider.tracer("test-application", "0.1")

# No need to pack the string back into binary, because
# we serialized the parent context as an alphanumeric string in this job system.
# There's a way to do this using constants defined in the API, but for just
# the parent trace/span ID, the format is simple enough:
traceparent_id = "00-e7766b70cfa5d0fec059a16237d6d613-ecc6e3ee8bd08ab0-01"
context_carrier = { "traceparent" => traceparent_id }

parent = OpenTelemetry.propagation.extract(context_carrier)

tracer.in_span("test", with_parent: parent) do |span|
  sleep 0.1
  puts "look, a span! #{span.inspect}"
end

Yields:

~/github/github-telemetry-ruby main*
❯ ruby test2.rb
look, a span! #<OpenTelemetry::Trace::Span:0x00007fd6d5845310 @context=#<OpenTelemetry::Trace::SpanContext:0x00007fd6d58451d0 @trace_id="\x04%\xEB\xEF\x12\x0E\xF4\x8E]\xB2\xDF.;\x16\x88\xFD", @span_id="\xB3\x82\x1C\xFA\xC4\x96`\xF3", @trace_flags=#<OpenTelemetry::Trace::TraceFlags:0x00007fd6d3159c38 @flags=0>, @tracestate=#<OpenTelemetry::Trace::Tracestate:0x00007fd6d31509d0 @hash={}>, @remote=false>>

~/github/github-telemetry-ruby main*