Tracking: Stabilize declarative configuration

[jack-berg]

We've made a lot of progress in declarative configuration, and I'd like to discuss what scope should be included in an initial cut at stabilization, and blockers.

The declarative config spec is broken into a variety of pieces:

• Data model: SDK implementation requirements around data model, and definition of schema. Subcomponents:
  • opentelemetry-configuration: repository for data model JSON schema
  • file-based YAML representation, including env var substitution syntax
  • SDK in-memory representation
• Instrumentation configuration API: defines an API for instrumentation to participate in configuration. Subcomponents:
  • ConfigProperties for programmatic access of structured config
  • ConfigProvider for accessing instrumentation config
• SDK: SDK user facing capabilities. Subcomponents:
  • ConfigProvider SDK implementation of ConfigProvider API component
  • ComponentProvider defines mechanism for referencing / loading SDK extension plugin components (i.e. exporters, samplers, etc) in declarative config
  • Parse and Create operations for producing configured SDK components from YAML file representation
• OTEL_EXPERIMENTAL_CONFIG_FILE, for automatic initialization based on YAML file representation

I propose the following scope for initial stability:

• opentelemetry-configuration
• file-based YAML representation
• SDK in-memory representation
• ConfigProperties
• ComponentProvider
• Parse and Create operations
• OTEL_EXPERIMENTAL_CONFIG_FILE, adjusted to OTEL_CONFIG_FILE

That means the following would be out of scope:

• ConfigProvider (i.e. instrumentation config API)

Assuming this scope, the follow lists the known blockers to stability

• Cut stable 1.0.0 release opentelemetry-configuration#161 (see list of sub-issues)
• Use ABNF syntax instead of PCRE2 in file configuration env var substitution syntax #4027
• Alignment with OpenTelemetry Collector configuration #3963
  • [file configuration] Escaping environment variable substitution #3914

The following list are not specifically blocking, but they should be reviewed to ensure the stability of declarative configuration doesn't prevent us from addressing them in the future:

• Environment variables specification should require _FILE variations to support reading from files #2739

Additionally, we need evaluation / feedback of the state of the spec from the current prototype language implementations.

If you have feedback on the scope of initial stability, or on additional blockers, please discuss in this issue.

[codeboten]

Not strictly blocking, but we should ensure we're not preventing addressing issues like this one #2739

[pellared]

Please consider: #4384

I was also thinking if the declarative configurations should allow adding custom components e.g. custom span processor created by the user. Right now it looks like the declarative configuration can be used only in a sandbox.