[Exporter.Geneva] Adding user_events support for logs on Linux - user_events data transport utility

[xiang17]

Towards #2474

Changes

Add a UnixUserEventsDataTransport utility class which will be used for sending user_events data.

Testing

Prerequisites

• Perf Tool: Install the perf tool to capture the user_events.
• Decode-Perf Tool: Install the decode-perf tool to decode the user_events.

Steps

To capture the user_events, the perf tool has to be run while dotnet test is running. The most simple way is to do in two terminals.

Terminal 1

Run the tests:

$ sudo dotnet test --configuration Debug --framework net8.0 --filter CategoryName=Geneva:user_events --no-build
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.8.2+699d445a1a (64-bit .NET 8.0.12)
[xUnit.net 00:00:00.12]   Discovering: OpenTelemetry.Exporter.Geneva.Tests
[xUnit.net 00:00:00.45]   Discovered:  OpenTelemetry.Exporter.Geneva.Tests
[xUnit.net 00:00:00.46]   Starting:    OpenTelemetry.Exporter.Geneva.Tests
[xUnit.net 00:00:00.51]     OpenTelemetry.Exporter.Geneva.Tests.UnixUserEventsDataTransportTests.UserEvents_Enabled_Success_Linux [SKIP]
[xUnit.net 00:00:00.51]       This would fail on Ubuntu. Skipping for now.
cat /sys/kernel/tracing/user_events_status
cat /sys/kernel/tracing/user_events_status [OUT] Active: 0
cat /sys/kernel/tracing/user_events_status [OUT] Busy: 0
cat /sys/kernel/debug/tracing/trace_pipe
cat /sys/kernel/tracing/events/user_events/MicrosoftOpenTelemetryLogs_L4K1/enable
cat /sys/kernel/tracing/events/user_events/MicrosoftOpenTelemetryLogs_L4K1/enable [OUT] 0
sh -c "echo '1' > /sys/kernel/tracing/events/user_events/MicrosoftOpenTelemetryLogs_L4K1/enable"
------------- ready to write events -------------
About to write tracepoint:
Written tracepoint.
cat /sys/kernel/debug/tracing/trace_pipe [OUT]   .NET TP Worker-19820   [004] ..... 620527.192577: MicrosoftOpenTelemetryLogs_L4K1: eventheader_flags=(7) version=(0) id=0x0 (0) tag=0x0 (0) opcode=(0) level=(4)
Writing events from listener:
eventheader_flags=(7), version=(0), id=0x0 (0), tag=0x0 (0), opcode=(0), level=(4)
Total events: 1
sh -c "echo '0' > /sys/kernel/tracing/events/user_events/MicrosoftOpenTelemetryLogs_L4K1/enable"
[xUnit.net 00:00:10.63]     OpenTelemetry.Exporter.Geneva.Tests.UnixUserEventsDataTransportTests.UserEvents_Disabled_Success_Linux [SKIP]
[xUnit.net 00:00:10.63]       This would fail on Ubuntu. Skipping for now.
[xUnit.net 00:00:10.63]   Finished:    OpenTelemetry.Exporter.Geneva.Tests
  OpenTelemetry.Exporter.Geneva.Tests test net8.0 succeeded (11.6s)

Test summary: total: 3, failed: 0, succeeded: 1, skipped: 2, duration: 11.6s
Build succeeded in 12.1s

Terminal 2

1. Check if events are ready with this command: sudo ls /sys/kernel/tracing/events/user_events/.

2. When the dotnet test outputs reached ------------- ready to write events -------------, the test would wait for 5 seconds before actually sending user_events. You can see events are ready by running the first command:

$ sudo ls /sys/kernel/tracing/events/user_events/
MicrosoftOpenTelemetryLogs_L4K1  enable  filter  otlp_metrics

3. Before the tests send out user_events in 5 seconds, run the perf tool: sudo ./perf record -v -e user_events:MicrosoftOpenTelemetryLogs_L4K1.

4. Once the dotnet test command finishes, ctrl-c to terminate the capture.

5. Run sudo /mnt/c/repos/LinuxTracepoints/bin/perf-decode ./perf.data to decode the user_events data:

$ sudo ./perf-decode ./perf.data
{
"./perf.data": [
  { "n": "MicrosoftOpenTelemetryLogs:_", "__csver__": "0x400", "partA": { "time": "2025-01-24T02:43:03.938446Z" }, "PartB": { "_typeName": "Log", "severityNumber": 21, "severityText": "Critical", "name": "CheckoutFailed" }, "PartC": { "book_id": "12345", "book_name": "The Hitchhiker's Guide to the Galaxy" }, "meta": { "time": 620719.617558900, "cpu": 4, "pid": 89238, "tid": 89259, "level": 4, "keyword": "0x1" } } ]
}

Formatted:

{
    "./perf.data": [{
        "n": "MicrosoftOpenTelemetryLogs:_",
        "__csver__": "0x400",
        "partA": {
            "time": "2025-01-24T02:43:03.938446Z"
        },
        "PartB": {
            "_typeName": "Log",
            "severityNumber": 21,
            "severityText": "Critical",
            "name": "CheckoutFailed"
        },
        "PartC": {
            "book_id": "12345",
            "book_name": "The Hitchhiker's Guide to the Galaxy"
        },
        "meta": {
            "time": 620719.617558900,
            "cpu": 4,
            "pid": 89238,
            "tid": 89259,
            "level": 4,
            "keyword": "0x1"
        }
    }]
}

Merge requirement checklist

• CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
• Unit tests added/updated
• [ ] Appropriate CHANGELOG.md files updated for non-trivial changes: will add when the feature is ready
• [ ] Changes in public API reviewed (if applicable)

[xiang17]

Disable the test case for now. The kernel in test runner doesn't support user_events yet. Error message:

  Failed OpenTelemetry.Exporter.Geneva.Tests.UnixUserEventsDataTransportTests.UserEvents_Logs_Success_Linux [2 s]
  Error Message:
   System.NotSupportedException : Kernel does not support user_events. Verify your distribution/kernel supports user_events: https://docs.kernel.org/trace/user_events.html.
  Stack Trace:
     at OpenTelemetry.Exporter.Geneva.Tests.UnixUserEventsDataTransportTests.EnsureUserEventsEnabled() in D:\a\opentelemetry-dotnet-contrib\opentelemetry-dotnet-contrib\test\OpenTelemetry.Exporter.Geneva.Tests\UnixUserEventsDataTransportTests.cs:line 254
   at OpenTelemetry.Exporter.Geneva.Tests.UnixUserEventsDataTransportTests.UserEvents_Logs_Success_Linux() in D:\a\opentelemetry-dotnet-contrib\opentelemetry-dotnet-contrib\test\OpenTelemetry.Exporter.Geneva.Tests\UnixUserEventsDataTransportTests.cs:line 151
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

There is an issue tracking this: #2326.

[cijothomas]

eventid and event name too?

[cijothomas]

is "L4" hardcoded? or its based on severity?
Also, is K1 fixed with no ability to change?