add qualification phase for agent 6

[sabrina-datadog]

What does this PR do?

add qualification phase for agent 6 by modifying the release tasks and the create rc pr workflow

CHANGES IN RELEASE TASKS

1. start qualification phase

• added new --start-qual argument to build_rc task
• this will be manually triggered by running the build rc task with the --start-qual argument
• inv release.build-rc 6.53.x --start-qual

2. automatically move qualification tag when RC is bumped

• added is_qualification() function to check if the current state is in qualification phase
• use git tag --force to move qualification tag to new RC commit
• inv release.build-rc 6.53.x (automatically checks if in qualification phase without any extra arguments)

3. end qualification phase

• according to the release coordinator guide we use the tag_version task to tag the final version
• used the same task to end the qualification phase by checking if it is agent 6 and a final version
• inv release.tag-version --version 6.53.2

CHANGES IN CREATE RC PR WORKFLOW

if current state of agent 6 is in qualification phase and there are no changes (no new commits in 6.53.x branch)

1. do not bump rc pr
2. rebuild previous rc pipeline

Motivation

ACIX-525

Describe how you validated your changes

ran the release tasks locally
but plan to test it fully through the next couple of agent 6 RCs

Possible Drawbacks / Trade-offs

Additional Notes

[agent-platform-auto-pr]

Uncompressed package size comparison

Comparison with ancestor bc6097051d789973d4572d1c1710779a0682b454

Diff per package

package	diff	status	size	ancestor	threshold
datadog-agent-arm64-deb	0.00MB	✅	916.09MB	916.09MB	0.50MB
datadog-dogstatsd-amd64-deb	0.00MB	✅	59.01MB	59.01MB	0.50MB
datadog-dogstatsd-x86_64-rpm	0.00MB	✅	59.09MB	59.09MB	0.50MB
datadog-dogstatsd-x86_64-suse	0.00MB	✅	59.09MB	59.09MB	0.50MB
datadog-dogstatsd-arm64-deb	0.00MB	✅	56.50MB	56.50MB	0.50MB
datadog-heroku-agent-amd64-deb	0.00MB	✅	477.89MB	477.89MB	0.50MB
datadog-iot-agent-amd64-deb	0.00MB	✅	93.88MB	93.88MB	0.50MB
datadog-iot-agent-x86_64-rpm	0.00MB	✅	93.95MB	93.95MB	0.50MB
datadog-iot-agent-x86_64-suse	0.00MB	✅	93.95MB	93.95MB	0.50MB
datadog-iot-agent-arm64-deb	0.00MB	✅	89.93MB	89.93MB	0.50MB
datadog-iot-agent-aarch64-rpm	0.00MB	✅	90.00MB	90.00MB	0.50MB
datadog-agent-aarch64-rpm	-0.00MB	✅	925.81MB	925.81MB	0.50MB
datadog-agent-x86_64-rpm	-0.00MB	✅	938.66MB	938.66MB	0.50MB
datadog-agent-x86_64-suse	-0.00MB	✅	938.66MB	938.66MB	0.50MB
datadog-agent-amd64-deb	-0.00MB	✅	928.92MB	928.92MB	0.50MB

Decision

✅ Passed

[agent-platform-auto-pr]

[Fast Unit Tests Report]

On pipeline 53975971 (CI Visibility). The following jobs did not run any unit tests:

Jobs:

• tests_deb-arm64-py3
• tests_deb-x64-py3
• tests_flavor_dogstatsd_deb-x64
• tests_flavor_heroku_deb-x64
• tests_flavor_iot_deb-x64
• tests_rpm-arm64-py3
• tests_rpm-x64-py3
• tests_windows-x64

If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-devx-help

[cit-pr-commenter]

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: e0bf71e0-0143-4faf-af66-9c0c93532083

Baseline: bc60970
Comparison: df7233b
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	quality_gate_logs	% cpu utilization	+0.89	[-2.13, +3.91]	1	Logs
➖	quality_gate_idle	memory utilization	+0.88	[+0.84, +0.91]	1	Logs bounds checks dashboard
➖	file_to_blackhole_500ms_latency	egress throughput	+0.20	[-0.56, +0.97]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	+0.11	[+0.07, +0.15]	1	Logs bounds checks dashboard
➖	file_to_blackhole_1000ms_latency_linear_load	egress throughput	+0.11	[-0.37, +0.58]	1	Logs
➖	file_to_blackhole_0ms_latency_http1	egress throughput	+0.10	[-0.78, +0.99]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	+0.09	[-0.74, +0.93]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.01	[-0.03, +0.01]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.01	[-0.29, +0.27]	1	Logs
➖	file_to_blackhole_300ms_latency	egress throughput	-0.01	[-0.66, +0.63]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	-0.03	[-0.84, +0.78]	1	Logs
➖	file_to_blackhole_0ms_latency_http2	egress throughput	-0.06	[-0.89, +0.78]	1	Logs
➖	file_tree	memory utilization	-0.09	[-0.14, -0.04]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-0.10	[-0.17, -0.04]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	-0.46	[-1.25, +0.32]	1	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-2.58	[-3.46, -1.70]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	links
✅	file_to_blackhole_0ms_latency	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http1	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http1	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http2	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http2	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency_linear_load	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	lost_bytes	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_300ms_latency	lost_bytes	10/10
✅	file_to_blackhole_300ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	lost_bytes	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	quality_gate_idle	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10
✅	quality_gate_logs	lost_bytes	10/10
✅	quality_gate_logs	memory_usage	10/10

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

• quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
• quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.

[CelianR]

I think you need to add an agent_context + a branch argument here to switch to the target branch

[chouetz]

nit: I think it's preferable to put the explicit argument

Suggested change

	inv -e release.run-rc-pipeline 6.53.x
	inv -e release.run-rc-pipeline -r 6.53.x

[chouetz]

Suggested change

	MINOR_RC_VERSION_RE = re.compile(r'\d+[.]\d+[.]0-rc\.\d+')
	MINOR_RC_VERSION_RE = re.compile(r'^\d+[.]\d+[.]0-rc\.\d+$')

As you change it for the RC_VERSION_RE we could align the regex (knowing it's only used with re.match we can add the boundaries ^$)

[chouetz]

Nit: This block could stand in a get_qualification_rc_tag method, to better illustrate its role (and to ease its testing)