v1.19.0-rc.1

What's Changed

• [CHANGE] Deprecate -blocks-storage.tsdb.wal-compression-enabled flag (use blocks-storage.tsdb.wal-compression-type instead). #6529
• [CHANGE] OTLP: Change OTLP handler to be consistent with the Prometheus OTLP handler. #6272

• target_info metric is enabled by default and can be disabled via -distributor.otlp.disable-target-info=true flag
• Convert all attributes to labels is disabled by default and can be enabled via -distributor.otlp.convert-all-attributes=true flag
• You can specify the attributes converted to labels via -distributor.promote-resource-attributes flag. Supported only if -distributor.otlp.convert-all-attributes=false

• [CHANGE] Change all max async concurrency default values 50 to 3 #6268
• [CHANGE] Change default value of -blocks-storage.bucket-store.index-cache.multilevel.max-async-concurrency from 50 to 3 #6265
• [CHANGE] Enable Compactor and Alertmanager in target all. #6204
• [CHANGE] Update the cortex_ingester_inflight_push_requests metric to represent the maximum number of inflight requests recorded in the last minute. #6437
• [CHANGE] gRPC Client: Expose connection timeout and set default to value to 5s. #6523
• [CHANGE] Enable analysis on Thanos PromQL engine #6472
• [CHANGE] Log grafana headers along with query request before query execution #6391
• [CHANGE] Log query requests in QFE before query execution #6390
• [CHANGE] Log when a request starts running in querier #6525
• [CHANGE] Remove openstack swfit experimental in blocks storage #6322
• [CHANGE] Remove openstack swift experimental #6316
• [CHANGE] Replace cespare/xxhash with cespare/xxhash/v2 #6467
• [CHANGE] Stop using global instant query codec #6328
• [CHANGE] Unify query frontend instant and range protos #6180
• [FEATURE] Ruler: Add an experimental flag -ruler.query-response-format to retrieve query response as a proto format. #6345
• [FEATURE] Ruler: Pagination support for List Rules API. #6299
• [FEATURE] Query Frontend/Querier: Add protobuf codec -api.querier-default-codec and the option to choose response compression type -querier.response-compression. #5527
• [FEATURE] Ruler: Experimental: Add ruler.frontend-address to allow query to query frontends instead of ingesters. #6151
• [FEATURE] Ruler: Minimize chances of missed rule group evaluations that can occur due to OOM kills, bad underlying nodes, or due to an unhealthy ruler that appears in the ring as healthy. This feature is enabled via -ruler.enable-ha-evaluation flag. #6129
• [FEATURE] Store Gateway: Add an in-memory chunk cache. #6245
• [FEATURE] Chunk Cache: Support multi level cache and add metrics. #6249
• [FEATURE] Distributor: Accept multiple HA Tracker pairs in the same request. #6256
• [FEATURE] Ruler: Add support for per-user external labels #6340
• [FEATURE] Query Frontend: Support a metadata federated query when -tenant-federation.enabled=true. #6461
• [FEATURE] Query Frontend: Support an exemplar federated query when -tenant-federation.enabled=true. #6455
• [FEATURE] Ingester/StoreGateway: Add support for cache regex query matchers via -ingester.matchers-cache-max-items and -blocks-storage.bucket-store.matchers-cache-max-items. #6477 #6491
• [FEATURE] Add reason why the key was evicted in the cortex_ingester_expanded_postings_cache_evicts metric #6318
• [FEATURE] Create feature flag to switch between current shuffle sharding group planner and partition compaction group planner #6141
• [FEATURE] Hook up partition compaction end to end implementation #6510
• [FEATURE] Implement partition compaction grouper #6172
• [FEATURE] Implement partition compaction planner #6469
• [FEATURE] Make LivenessCheck Timeout Configurable #6227
• [FEATURE] Querier: Add day range limit for LabelNames and LabelValues #6233
• [ENHANCEMENT] Query Frontend: Add more operation label values to the cortex_query_frontend_queries_total metric. #6519
• [ENHANCEMENT] Query Frontend: Add a source label to query stat metrics. #6470
• [ENHANCEMENT] Query Frontend: Add a flag -tenant-federation.max-tenant to limit the number of tenants for federated query. #6493
• [ENHANCEMENT] Querier: Add a -tenant-federation.max-concurrent flags to configure the number of worker processing federated query and add a cortex_querier_federated_tenants_per_query histogram to track the number of tenants per query. #6449
• [ENHANCEMENT] Query Frontend: Add a number of series in the query response to the query stat log. #6423
• [ENHANCEMENT] Store Gateway: Add a hedged request to reduce the tail latency. #6388
• [ENHANCEMENT] Ingester: Add metrics to track succeed/failed native histograms. #6370
• [ENHANCEMENT] Query Frontend/Querier: Add an experimental flag -querier.enable-promql-experimental-functions to enable experimental promQL functions. #6355
• [ENHANCEMENT] OTLP: Add -distributor.otlp-max-recv-msg-size flag to limit OTLP request size in bytes. #6333
• [ENHANCEMENT] S3 Bucket Client: Add a list objects version configs to configure list api object version. #6280
• [ENHANCEMENT] OpenStack Swift: Add application credential configs for Openstack swift object storage backend. #6255
• [ENHANCEMENT] Query Frontend: Add new query stats metrics cortex_query_samples_scanned_total and cortex_query_peak_samples to track scannedSamples and peakSample per user. #6228
• [ENHANCEMENT] Ingester: Add option ingester.disable-chunk-trimming to disable chunk trimming. #6300
• [ENHANCEMENT] Ingester: Add blocks-storage.tsdb.wal-compression-type to support zstd wal compression type. #6232
• [ENHANCEMENT] Query Frontend: Add info field to query response. #6207
• [ENHANCEMENT] Query Frontend: Add peakSample in query stats response. #6188
• [ENHANCEMENT] Ruler: Add new ruler metric cortex_ruler_rule_groups_in_store that is the total rule groups per tenant in store, which can be used to compare with cortex_prometheus_rule_group_rules to count the number of rule groups that are not loaded by a ruler. #5869
• [ENHANCEMENT] Ingester/Ring: New READONLY status on ring to be used by Ingester. New ingester API to change mode of ingester #6163
• [ENHANCEMENT] Ruler: Add query statistics metrics when --ruler.query-stats-enabled=true. #6173
• [ENHANCEMENT] Ingester: Add new API /ingester/all_user_stats which shows loaded blocks, active timeseries and ingestion rate for a specific ingester. #6178
• [ENHANCEMENT] Distributor: Add new cortex_reduced_resolution_histogram_samples_total metric to track the number of histogram samples which resolution was reduced. #6182
• [ENHANCEMENT] StoreGateway: Implement metadata API limit in queryable. #6195
• [ENHANCEMENT] Ingester: Add matchers to ingester LabelNames() and LabelNamesStream() RPC. #6209
• [ENHANCEMENT] KV: Add TLS configs to consul. #6374
• [ENHANCEMENT] Ingester/Store Gateway Clients: Introduce an experimental HealthCheck handler to quickly fail requests directed to unhealthy targets. #6225 #6257
• [ENHANCEMENT] Upgrade build image and Go version to 1.23.2. #6261 #6262
• [ENHANCEMENT] Ingester: Introduce a new experimental feature for caching expanded postings on the ingester. #6296
• [ENHANCEMENT] Querier/Ruler: Expose store_gateway_consistency_check_max_attempts for max retries when querying store gateway in consistency check. #6276
• [ENHANCEMENT] StoreGateway: Add new cortex_bucket_store_chunk_pool_inuse_bytes metric to track the usage in chunk pool. #6310
• [ENHANCEMENT] Distributor: Add new cortex_distributor_inflight_client_requests metric to track number of ingester client inflight requests. #6358
• [ENHANCEMENT] Distributor: Expose cortex_label_size_bytes native histogram metric. #6372
• [ENHANCEMENT] Add new option -server.grpc_server-num-stream-workers to configure the number of worker goroutines that should be used to process incoming streams. #6386
• [ENHANCEMENT] Distributor: Return HTTP 5XX instead of HTTP 4XX when instance limits are hit. #6358
• [ENHANCEMENT] Ingester: Make sure unregistered ingester joining the ring after WAL replay. #6277
• [ENHANCEMENT] Distributor: Add a new -distributor.num-push-workers flag to use a goroutine worker pool when sending data from distributor to ingesters. #6406
• [ENHANCEMENT] Ingester: If a limit per label set entry doesn't have any label, use it as the default partition to catch all series that doesn't match any other label sets entries. #6435
• [ENHANCEMENT] Querier: Add new cortex_querier_codec_response_size metric to track the size of the encoded query responses from queriers. #6444
• [ENHANCEMENT] Distributor: Added cortex_distributor_received_samples_per_labelset_total metric to calculate ingestion rate per label set. #6443
• [ENHANCEMENT] Added metric name in limiter per-metric exceeded errors. #6416
• [ENHANCEMENT] StoreGateway: Added cortex_bucket_store_indexheader_load_duration_seconds and cortex_bucket_store_indexheader_download_duration_seconds metrics for time of downloading and loading index header files. #6445
• [ENHANCEMENT] Add cleaner logic to clean partition compaction blocks and related files #6507
• [ENHANCEMENT] Add support for native histograms in querier protobuf codec #6368
• [ENHANCEMENT] Add timeout on lifecycler heartbeat #6212
• [ENHANCEMENT] Added UserReplicaGroupMetrics #6463
• [ENHANCEMENT] Blocks Storage: Allow use of non-dualstack endpoints for S3 blocks storage via -blocks-storage.s3.disable-dualstack. #6522
• [ENHANCEMENT] Discarded samples per labelset metrics for throttle by labelset #6492
• [ENHANCEMENT] Expanded Postings Cache can cache results without the nearly created series under high load. #6417
• [ENHANCEMENT] Ingester: Disable chunk trimming. #6270
• [ENHANCEMENT] Improve consistency check warn log #6366
• [ENHANCEMENT] Improve streaming on MetricsForLabelMatchersStream method #6436
• [ENHANCEMENT] Improve validation metrics for discarded samples and exemplars #6218
• [E...

v1.19.0-rc.0

What's Changed

• [CHANGE] Deprecate -blocks-storage.tsdb.wal-compression-enabled flag (use blocks-storage.tsdb.wal-compression-type instead). #6529
• [CHANGE] OTLP: Change OTLP handler to be consistent with the Prometheus OTLP handler. #6272

• target_info metric is enabled by default and can be disabled via -distributor.otlp.disable-target-info=true flag
• Convert all attributes to labels is disabled by default and can be enabled via -distributor.otlp.convert-all-attributes=true flag
• You can specify the attributes converted to labels via -distributor.promote-resource-attributes flag. Supported only if -distributor.otlp.convert-all-attributes=false

• [CHANGE] Change all max async concurrency default values 50 to 3 #6268
• [CHANGE] Change default value of -blocks-storage.bucket-store.index-cache.multilevel.max-async-concurrency from 50 to 3 #6265
• [CHANGE] Enable Compactor and Alertmanager in target all. #6204
• [CHANGE] Update the cortex_ingester_inflight_push_requests metric to represent the maximum number of inflight requests recorded in the last minute. #6437
• [CHANGE] gRPC Client: Expose connection timeout and set default to value to 5s. #6523
• [CHANGE] Enable analysis on Thanos PromQL engine #6472
• [CHANGE] Log grafana headers along with query request before query execution #6391
• [CHANGE] Log query requests in QFE before query execution #6390
• [CHANGE] Log when a request starts running in querier #6525
• [CHANGE] Remove openstack swfit experimental in blocks storage #6322
• [CHANGE] Remove openstack swift experimental #6316
• [CHANGE] Replace cespare/xxhash with cespare/xxhash/v2 #6467
• [CHANGE] Stop using global instant query codec #6328
• [CHANGE] Unify query frontend instant and range protos #6180
• [FEATURE] Ruler: Add an experimental flag -ruler.query-response-format to retrieve query response as a proto format. #6345
• [FEATURE] Ruler: Pagination support for List Rules API. #6299
• [FEATURE] Query Frontend/Querier: Add protobuf codec -api.querier-default-codec and the option to choose response compression type -querier.response-compression. #5527
• [FEATURE] Ruler: Experimental: Add ruler.frontend-address to allow query to query frontends instead of ingesters. #6151
• [FEATURE] Ruler: Minimize chances of missed rule group evaluations that can occur due to OOM kills, bad underlying nodes, or due to an unhealthy ruler that appears in the ring as healthy. This feature is enabled via -ruler.enable-ha-evaluation flag. #6129
• [FEATURE] Store Gateway: Add an in-memory chunk cache. #6245
• [FEATURE] Chunk Cache: Support multi level cache and add metrics. #6249
• [FEATURE] Distributor: Accept multiple HA Tracker pairs in the same request. #6256
• [FEATURE] Ruler: Add support for per-user external labels #6340
• [FEATURE] Query Frontend: Support a metadata federated query when -tenant-federation.enabled=true. #6461
• [FEATURE] Query Frontend: Support an exemplar federated query when -tenant-federation.enabled=true. #6455
• [FEATURE] Ingester/StoreGateway: Add support for cache regex query matchers via -ingester.matchers-cache-max-items and -blocks-storage.bucket-store.matchers-cache-max-items. #6477 #6491
• [FEATURE] Add reason why the key was evicted in the cortex_ingester_expanded_postings_cache_evicts metric #6318
• [FEATURE] Create feature flag to switch between current shuffle sharding group planner and partition compaction group planner #6141
• [FEATURE] Hook up partition compaction end to end implementation #6510
• [FEATURE] Implement partition compaction grouper #6172
• [FEATURE] Implement partition compaction planner #6469
• [FEATURE] Make LivenessCheck Timeout Configurable #6227
• [FEATURE] Querier: Add day range limit for LabelNames and LabelValues #6233
• [ENHANCEMENT] Query Frontend: Add more operation label values to the cortex_query_frontend_queries_total metric. #6519
• [ENHANCEMENT] Query Frontend: Add a source label to query stat metrics. #6470
• [ENHANCEMENT] Query Frontend: Add a flag -tenant-federation.max-tenant to limit the number of tenants for federated query. #6493
• [ENHANCEMENT] Querier: Add a -tenant-federation.max-concurrent flags to configure the number of worker processing federated query and add a cortex_querier_federated_tenants_per_query histogram to track the number of tenants per query. #6449
• [ENHANCEMENT] Query Frontend: Add a number of series in the query response to the query stat log. #6423
• [ENHANCEMENT] Store Gateway: Add a hedged request to reduce the tail latency. #6388
• [ENHANCEMENT] Ingester: Add metrics to track succeed/failed native histograms. #6370
• [ENHANCEMENT] Query Frontend/Querier: Add an experimental flag -querier.enable-promql-experimental-functions to enable experimental promQL functions. #6355
• [ENHANCEMENT] OTLP: Add -distributor.otlp-max-recv-msg-size flag to limit OTLP request size in bytes. #6333
• [ENHANCEMENT] S3 Bucket Client: Add a list objects version configs to configure list api object version. #6280
• [ENHANCEMENT] OpenStack Swift: Add application credential configs for Openstack swift object storage backend. #6255
• [ENHANCEMENT] Query Frontend: Add new query stats metrics cortex_query_samples_scanned_total and cortex_query_peak_samples to track scannedSamples and peakSample per user. #6228
• [ENHANCEMENT] Ingester: Add option ingester.disable-chunk-trimming to disable chunk trimming. #6300
• [ENHANCEMENT] Ingester: Add blocks-storage.tsdb.wal-compression-type to support zstd wal compression type. #6232
• [ENHANCEMENT] Query Frontend: Add info field to query response. #6207
• [ENHANCEMENT] Query Frontend: Add peakSample in query stats response. #6188
• [ENHANCEMENT] Ruler: Add new ruler metric cortex_ruler_rule_groups_in_store that is the total rule groups per tenant in store, which can be used to compare with cortex_prometheus_rule_group_rules to count the number of rule groups that are not loaded by a ruler. #5869
• [ENHANCEMENT] Ingester/Ring: New READONLY status on ring to be used by Ingester. New ingester API to change mode of ingester #6163
• [ENHANCEMENT] Ruler: Add query statistics metrics when --ruler.query-stats-enabled=true. #6173
• [ENHANCEMENT] Ingester: Add new API /ingester/all_user_stats which shows loaded blocks, active timeseries and ingestion rate for a specific ingester. #6178
• [ENHANCEMENT] Distributor: Add new cortex_reduced_resolution_histogram_samples_total metric to track the number of histogram samples which resolution was reduced. #6182
• [ENHANCEMENT] StoreGateway: Implement metadata API limit in queryable. #6195
• [ENHANCEMENT] Ingester: Add matchers to ingester LabelNames() and LabelNamesStream() RPC. #6209
• [ENHANCEMENT] KV: Add TLS configs to consul. #6374
• [ENHANCEMENT] Ingester/Store Gateway Clients: Introduce an experimental HealthCheck handler to quickly fail requests directed to unhealthy targets. #6225 #6257
• [ENHANCEMENT] Upgrade build image and Go version to 1.23.2. #6261 #6262
• [ENHANCEMENT] Ingester: Introduce a new experimental feature for caching expanded postings on the ingester. #6296
• [ENHANCEMENT] Querier/Ruler: Expose store_gateway_consistency_check_max_attempts for max retries when querying store gateway in consistency check. #6276
• [ENHANCEMENT] StoreGateway: Add new cortex_bucket_store_chunk_pool_inuse_bytes metric to track the usage in chunk pool. #6310
• [ENHANCEMENT] Distributor: Add new cortex_distributor_inflight_client_requests metric to track number of ingester client inflight requests. #6358
• [ENHANCEMENT] Distributor: Expose cortex_label_size_bytes native histogram metric. #6372
• [ENHANCEMENT] Add new option -server.grpc_server-num-stream-workers to configure the number of worker goroutines that should be used to process incoming streams. #6386
• [ENHANCEMENT] Distributor: Return HTTP 5XX instead of HTTP 4XX when instance limits are hit. #6358
• [ENHANCEMENT] Ingester: Make sure unregistered ingester joining the ring after WAL replay. #6277
• [ENHANCEMENT] Distributor: Add a new -distributor.num-push-workers flag to use a goroutine worker pool when sending data from distributor to ingesters. #6406
• [ENHANCEMENT] Ingester: If a limit per label set entry doesn't have any label, use it as the default partition to catch all series that doesn't match any other label sets entries. #6435
• [ENHANCEMENT] Querier: Add new cortex_querier_codec_response_size metric to track the size of the encoded query responses from queriers. #6444
• [ENHANCEMENT] Distributor: Added cortex_distributor_received_samples_per_labelset_total metric to calculate ingestion rate per label set. #6443
• [ENHANCEMENT] Added metric name in limiter per-metric exceeded errors. #6416
• [ENHANCEMENT] StoreGateway: Added cortex_bucket_store_indexheader_load_duration_seconds and cortex_bucket_store_indexheader_download_duration_seconds metrics for time of downloading and loading index header files. #6445
• [ENHANCEMENT] Add cleaner logic to clean partition compaction blocks and related files #6507
• [ENHANCEMENT] Add support for native histograms in querier protobuf codec #6368
• [ENHANCEMENT] Add timeout on lifecycler heartbeat #6212
• [ENHANCEMENT] Added UserReplicaGroupMetrics #6463
• [ENHANCEMENT] Blocks Storage: Allow use of non-dualstack endpoints for S3 blocks storage via -blocks-storage.s3.disable-dualstack. #6522
• [ENHANCEMENT] Discarded samples per labelset metrics for throttle by labelset #6492
• [ENHANCEMENT] Expanded Postings Cache can cache results without the nearly created series under high load. #6417
• [ENHANCEMENT] Ingester: Disable chunk trimming. #6270
• [ENHANCEMENT] Improve consistency check warn log #6366
• [ENHANCEMENT] Improve streaming on MetricsForLabelMatchersStream method #6436
• [ENHANCEMENT] Improve validation metrics for discarded samples and exemplars #6218
• [E...

v1.18.1

This release includes security fixes. Thanks to @99-not-out and @jeanlouisboudart for reporting it

What's Changed

• [BUGFIX] Backporting upgrade to go 1.22.7 to patch CVE-2024-34155, CVE-2024-34156, CVE-2024-34158 #6217 #6264

Full Changelog: v1.18.0...v1.18.1

Cortex v1.18.0

This release contains 230 contributions from 29 contributors. We also have 9 new contributors. Thank you all for the contributions!

Some notable changes release are:

• Experimental native histogram ingestion via -blocks-storage.tsdb.enable-native-histograms flag
• Support for filtering alerts on ListRules API
• Add query rejection mechanism to protect queries
• Introduce token bucket limiter to protect store gateway
• Implement ingester metadata API limits
• Remove -querier.query-store-for-labels-enabled flag
• Remove -querier.at-modifier-enabled flag
• Remove oltp_endpoint config
• ruler.evaluation-delay-duration mark as deprecated. Use ruler.query-offset
• -querier.max-outstanding-requests-per-tenant and -query-scheduler.max-outstanding-requests-per-tenant mark as deprecated. Use frontend.max-outstanding-requests-per-tenant

What's Changed

• [CHANGE] Ingester: Remove -querier.query-store-for-labels-enabled flag. Querying long-term store for labels is always enabled. #5984
• [CHANGE] Server: Instrument cortex_request_duration_seconds metric with native histogram. If native-histograms feature is enabled in monitoring Prometheus then the metric name needs to be updated in your dashboards. #6056
• [CHANGE] Distributor/Ingester: Change cortex_distributor_ingester_appends_total, cortex_distributor_ingester_append_failures_total, cortex_distributor_ingester_queries_total, and cortex_distributor_ingester_query_failures_total metrics to use the ingester ID instead of its IP as the label value. #6078
• [CHANGE] OTLP: Set AddMetricSuffixes to true to always enable metric name normalization. #6136
• [CHANGE] Querier: Deprecate and enable by default querier.ingester-metadata-streaming flag. #6147
• [CHANGE] QueryFrontend/QueryScheduler: Deprecate -querier.max-outstanding-requests-per-tenant and -query-scheduler.max-outstanding-requests-per-tenant flags. Use frontend.max-outstanding-requests-per-tenant instead. #6146
• [CHANGE] Ingesters: Enable 'snappy-block' compression on ingester clients by default. #6148
• [CHANGE] Ruler: Scheduling ruler.evaluation-delay-duration to be deprecated. Ruler will use the highest value between ruler.evaluation-delay-duration and ruler.query-offset #6149
• [CHANGE] Querier: Remove -querier.at-modifier-enabled flag. #6157
• [CHANGE] Tracing: Remove deprecated oltp_endpoint config entirely. #6158
• [CHANGE] Store Gateway: Enable store gateway zone stable shuffle sharding by default. #6161
• [FEATURE] Ingester/Distributor: Experimental: Enable native histogram ingestion via -blocks-storage.tsdb.enable-native-histograms flag. #5986 #6010 #6020
• [FEATURE] Querier: Enable querying native histogram chunks. #5944 #6031
• [FEATURE] Query Frontend: Support native histogram in query frontend response. #5996 #6043
• [FEATURE] Ruler: Support sending native histogram samples to Ingester. #6029
• [FEATURE] Ruler: Add support for filtering out alerts in ListRules API. #6011
• [FEATURE] Query Frontend: Added a query rejection mechanism to block resource-intensive queries. #6005
• [FEATURE] OTLP: Support ingesting OTLP exponential metrics as native histograms. #6071 #6135
• [FEATURE] Ingester: Add ingester.instance-limits.max-inflight-query-requests to allow limiting ingester concurrent queries. #6081
• [FEATURE] Distributor: Add validation.max-native-histogram-buckets to limit max number of bucket count. Distributor will try to automatically reduce histogram resolution until it is within the bucket limit or resolution cannot be reduced anymore. #6104
• [FEATURE] Store Gateway: Introduce token bucket limiter to enhance store gateway throttling. #6016
• [FEATURE] Ruler: Add support for query_offset field on RuleGroup and new ruler_query_offset per-tenant limit. #6085
• [ENHANCEMENT] Ruler: Add support to persist tokens in rulers. #5987
• [ENHANCEMENT] Query Frontend/Querier: Added store gateway postings touched count and touched size in Querier stats and log in Query Frontend. #5892
• [ENHANCEMENT] Query Frontend/Querier: Returns warnings on prometheus query responses. #5916
• [ENHANCEMENT] Ingester: Allowing to configure -blocks-storage.tsdb.head-compaction-interval flag up to 30 min and add a jitter on the first head compaction. #5919 #5928
• [ENHANCEMENT] Distributor: Added max_inflight_push_requests config to ingester client to protect distributor from OOMKilled. #5917
• [ENHANCEMENT] Distributor/Querier: Clean stale per-ingester metrics after ingester restarts. #5930
• [ENHANCEMENT] Distributor/Ring: Allow disabling detailed ring metrics by ring member. #5931
• [ENHANCEMENT] KV: Etcd Added etcd.ping-without-stream-allowed parameter to disable/enable PermitWithoutStream #5933
• [ENHANCEMENT] Ingester: Add a new limits_per_label_set limit. This limit functions similarly to max_series_per_metric, but allowing users to define the maximum number of series per LabelSet. #5950 #5993
• [ENHANCEMENT] Store Gateway: Log gRPC requests together with headers configured in http_request_headers_to_log. #5958
• [ENHANCEMENT] Ingester: Add a new experimental -ingester.labels-string-interning-enabled flag to enable string interning for metrics labels. #6057
• [ENHANCEMENT] Ingester: Add link to renew 10% of the ingesters tokens in the admin page. #6063
• [ENHANCEMENT] Ruler: Add support for filtering by state and health field on Rules API. #6040
• [ENHANCEMENT] Ruler: Add support for filtering by match field on Rules API. #6083
• [ENHANCEMENT] Distributor: Reduce memory usage when error volume is high. #6095
• [ENHANCEMENT] Compactor: Centralize metrics used by compactor and add user label to compactor metrics. #6096
• [ENHANCEMENT] Compactor: Add unique execution ID for each compaction cycle in log for easy debugging. #6097
• [ENHANCEMENT] Compactor: Differentiate retry and halt error and retry failed compaction only on retriable error. #6111
• [ENHANCEMENT] Ruler: Add support for filtering by state and health field on Rules API. #6040
• [ENHANCEMENT] Compactor: Split cleaner cycle for active and deleted tenants. #6112
• [ENHANCEMENT] Compactor: Introduce cleaner visit marker. #6113
• [ENHANCEMENT] Query Frontend: Add cortex_query_samples_total metric. #6142
• [ENHANCEMENT] Ingester: Implement metadata API limit. #6128
• [BUGFIX] Configsdb: Fix endline issue in db password. #5920
• [BUGFIX] Ingester: Fix user and type labels for the cortex_ingester_tsdb_head_samples_appended_total TSDB metric. #5952
• [BUGFIX] Querier: Enforce max query length check for /api/v1/series API even though ignoreMaxQueryLength is set to true. #6018
• [BUGFIX] Ingester: Fix issue with the minimize token generator where it was not taking in consideration the current ownership of an instance when generating extra tokens. #6062
• [BUGFIX] Scheduler: Fix user queue in scheduler that was not thread-safe. #6077 #6160
• [BUGFIX] Ingester: Include out-of-order head compaction when compacting TSDB head. #6108
• [BUGFIX] Ingester: Fix cortex_ingester_tsdb_mmap_chunks_total metric. #6134
• [BUGFIX] Query Frontend: Fix query rejection bug for metadata queries. #6143

New Contributors

• @Yaxhveer made their first contribution in #5920
• @KrisBuytaert made their first contribution in #5933
• @wilguo made their first contribution in #5935
• @rapphil made their first contribution in #5987
• @harshitasao made their first contribution in #6061
• @deradiri made their first contribution in #6099
• @shekeriev made their first contribution in #6125
• @klingerf made their first contribution in #6131
• @SungJin1212 made their first contribution in #6142

Full Changelog: v1.17.1...v1.18.0

Cortex v1.18.0-rc.0

This release contains 230 contributions from 29 contributors. We also have 9 new contributors. Thank you all for the contributions!

Some notable changes release are:

• Experimental native histogram ingestion via -blocks-storage.tsdb.enable-native-histograms flag
• Support for filtering alerts on ListRules API
• Add query rejection mechanism to protect queries
• Introduce token bucket limiter to protect store gateway
• Implement ingester metadata API limits
• Remove -querier.query-store-for-labels-enabled flag
• Remove -querier.at-modifier-enabled flag
• Remove oltp_endpoint config
• ruler.evaluation-delay-duration mark as deprecated. Use ruler.query-offset
• -querier.max-outstanding-requests-per-tenant and -query-scheduler.max-outstanding-requests-per-tenant mark as deprecated. Use frontend.max-outstanding-requests-per-tenant

What's Changed

• [CHANGE] Ingester: Remove -querier.query-store-for-labels-enabled flag. Querying long-term store for labels is always enabled. #5984
• [CHANGE] Server: Instrument cortex_request_duration_seconds metric with native histogram. If native-histograms feature is enabled in monitoring Prometheus then the metric name needs to be updated in your dashboards. #6056
• [CHANGE] Distributor/Ingester: Change cortex_distributor_ingester_appends_total, cortex_distributor_ingester_append_failures_total, cortex_distributor_ingester_queries_total, and cortex_distributor_ingester_query_failures_total metrics to use the ingester ID instead of its IP as the label value. #6078
• [CHANGE] OTLP: Set AddMetricSuffixes to true to always enable metric name normalization. #6136
• [CHANGE] Querier: Deprecate and enable by default querier.ingester-metadata-streaming flag. #6147
• [CHANGE] QueryFrontend/QueryScheduler: Deprecate -querier.max-outstanding-requests-per-tenant and -query-scheduler.max-outstanding-requests-per-tenant flags. Use frontend.max-outstanding-requests-per-tenant instead. #6146
• [CHANGE] Ingesters: Enable 'snappy-block' compression on ingester clients by default. #6148
• [CHANGE] Ruler: Scheduling ruler.evaluation-delay-duration to be deprecated. Ruler will use the highest value between ruler.evaluation-delay-duration and ruler.query-offset #6149
• [CHANGE] Querier: Remove -querier.at-modifier-enabled flag. #6157
• [CHANGE] Tracing: Remove deprecated oltp_endpoint config entirely. #6158
• [CHANGE] Store Gateway: Enable store gateway zone stable shuffle sharding by default. #6161
• [FEATURE] Ingester/Distributor: Experimental: Enable native histogram ingestion via -blocks-storage.tsdb.enable-native-histograms flag. #5986 #6010 #6020
• [FEATURE] Querier: Enable querying native histogram chunks. #5944 #6031
• [FEATURE] Query Frontend: Support native histogram in query frontend response. #5996 #6043
• [FEATURE] Ruler: Support sending native histogram samples to Ingester. #6029
• [FEATURE] Ruler: Add support for filtering out alerts in ListRules API. #6011
• [FEATURE] Query Frontend: Added a query rejection mechanism to block resource-intensive queries. #6005
• [FEATURE] OTLP: Support ingesting OTLP exponential metrics as native histograms. #6071 #6135
• [FEATURE] Ingester: Add ingester.instance-limits.max-inflight-query-requests to allow limiting ingester concurrent queries. #6081
• [FEATURE] Distributor: Add validation.max-native-histogram-buckets to limit max number of bucket count. Distributor will try to automatically reduce histogram resolution until it is within the bucket limit or resolution cannot be reduced anymore. #6104
• [FEATURE] Store Gateway: Introduce token bucket limiter to enhance store gateway throttling. #6016
• [FEATURE] Ruler: Add support for query_offset field on RuleGroup and new ruler_query_offset per-tenant limit. #6085
• [ENHANCEMENT] Ruler: Add support to persist tokens in rulers. #5987
• [ENHANCEMENT] Query Frontend/Querier: Added store gateway postings touched count and touched size in Querier stats and log in Query Frontend. #5892
• [ENHANCEMENT] Query Frontend/Querier: Returns warnings on prometheus query responses. #5916
• [ENHANCEMENT] Ingester: Allowing to configure -blocks-storage.tsdb.head-compaction-interval flag up to 30 min and add a jitter on the first head compaction. #5919 #5928
• [ENHANCEMENT] Distributor: Added max_inflight_push_requests config to ingester client to protect distributor from OOMKilled. #5917
• [ENHANCEMENT] Distributor/Querier: Clean stale per-ingester metrics after ingester restarts. #5930
• [ENHANCEMENT] Distributor/Ring: Allow disabling detailed ring metrics by ring member. #5931
• [ENHANCEMENT] KV: Etcd Added etcd.ping-without-stream-allowed parameter to disable/enable PermitWithoutStream #5933
• [ENHANCEMENT] Ingester: Add a new limits_per_label_set limit. This limit functions similarly to max_series_per_metric, but allowing users to define the maximum number of series per LabelSet. #5950 #5993
• [ENHANCEMENT] Store Gateway: Log gRPC requests together with headers configured in http_request_headers_to_log. #5958
• [ENHANCEMENT] Ingester: Add a new experimental -ingester.labels-string-interning-enabled flag to enable string interning for metrics labels. #6057
• [ENHANCEMENT] Ingester: Add link to renew 10% of the ingesters tokens in the admin page. #6063
• [ENHANCEMENT] Ruler: Add support for filtering by state and health field on Rules API. #6040
• [ENHANCEMENT] Ruler: Add support for filtering by match field on Rules API. #6083
• [ENHANCEMENT] Distributor: Reduce memory usage when error volume is high. #6095
• [ENHANCEMENT] Compactor: Centralize metrics used by compactor and add user label to compactor metrics. #6096
• [ENHANCEMENT] Compactor: Add unique execution ID for each compaction cycle in log for easy debugging. #6097
• [ENHANCEMENT] Compactor: Differentiate retry and halt error and retry failed compaction only on retriable error. #6111
• [ENHANCEMENT] Ruler: Add support for filtering by state and health field on Rules API. #6040
• [ENHANCEMENT] Compactor: Split cleaner cycle for active and deleted tenants. #6112
• [ENHANCEMENT] Compactor: Introduce cleaner visit marker. #6113
• [ENHANCEMENT] Query Frontend: Add cortex_query_samples_total metric. #6142
• [ENHANCEMENT] Ingester: Implement metadata API limit. #6128
• [BUGFIX] Configsdb: Fix endline issue in db password. #5920
• [BUGFIX] Ingester: Fix user and type labels for the cortex_ingester_tsdb_head_samples_appended_total TSDB metric. #5952
• [BUGFIX] Querier: Enforce max query length check for /api/v1/series API even though ignoreMaxQueryLength is set to true. #6018
• [BUGFIX] Ingester: Fix issue with the minimize token generator where it was not taking in consideration the current ownership of an instance when generating extra tokens. #6062
• [BUGFIX] Scheduler: Fix user queue in scheduler that was not thread-safe. #6077 #6160
• [BUGFIX] Ingester: Include out-of-order head compaction when compacting TSDB head. #6108
• [BUGFIX] Ingester: Fix cortex_ingester_tsdb_mmap_chunks_total metric. #6134
• [BUGFIX] Query Frontend: Fix query rejection bug for metadata queries. #6143

New Contributors

• @Yaxhveer made their first contribution in #5920
• @KrisBuytaert made their first contribution in #5933
• @wilguo made their first contribution in #5935
• @rapphil made their first contribution in #5987
• @harshitasao made their first contribution in #6061
• @deradiri made their first contribution in #6099
• @shekeriev made their first contribution in #6125
• @klingerf made their first contribution in #6131
• @SungJin1212 made their first contribution in #6142

Full Changelog: v1.17.1...v1.18.0-rc.0

Cortex v1.17.1

This release includes one bug fix and two changes related to compatibility:

• [CHANGE] Query Frontend/Ruler: Omit empty data, errorType and error fields in API response. #5953 #5954
• [ENHANCEMENT] Ingester: Added upload_compacted_blocks_enabled config to ingester to parameterize uploading compacted blocks. #5959
• [BUGFIX] Querier: Select correct tenant during query federation. #5943

Cortex 1.17.0

This release contains 168 contributions from 29 contributors. We also have 16 new contributors. Thank you all for the contributions!

Some notable changes release are:

• Experimental OTLP ingestion
• Experimental minimize spread token generator strategy on Ingester
• Advanced query scheduling with Query Priority
• ListRules API high availability by rule group replication and backup
• Various improvements on Store Gateway Index Cache
• mem-ballast-size-bytes flag has been marked as deprecated and not functional anymore
• -querier.ingester-streaming flag has been marked as deprecated and ingester streaming is always enabled now
• querier.iterators and querier.batch-iterators flags have been marked as deprecated and batch iterator is always enabled in Querier now

Cortex

• [CHANGE] Azure Storage: Upgraded objstore dependency and support Azure Workload Identity Authentication. Added connection_string to support authenticating via SAS token. Marked msi_resource config as deprecating. #5645
• [CHANGE] Store Gateway: Add a new fastcache based inmemory index cache. #5619
• [CHANGE] Index Cache: Multi level cache backfilling operation becomes async. Added -blocks-storage.bucket-store.index-cache.multilevel.max-async-concurrency and -blocks-storage.bucket-store.index-cache.multilevel.max-async-buffer-size configs and metric cortex_store_multilevel_index_cache_backfill_dropped_items_total for number of dropped items. #5661
• [CHANGE] Ingester: Disable uploading compacted blocks and overlapping compaction in ingester. #5735
• [CHANGE] Distributor: Count the number of rate-limited samples in distributor_samples_in_total. #5714
• [CHANGE] Ruler: Remove cortex_ruler_write_requests_total, cortex_ruler_write_requests_failed_total, cortex_ruler_queries_total, cortex_ruler_queries_failed_total, and cortex_ruler_query_seconds_total metrics for the tenant when the ruler deletes the manager for the tenant. #5772
• [CHANGE] Main: Mark mem-ballast-size-bytes flag as deprecated. #5816
• [CHANGE] Querier: Mark -querier.ingester-streaming flag as deprecated. Now query ingester streaming is always enabled. #5817
• [CHANGE] Compactor/Bucket Store: Added -blocks-storage.bucket-store.block-discovery-strategy to configure different block listing strategy. Reverted the current recursive block listing mechanism and use the strategy Concurrent as in 1.15. #5828
• [CHANGE] Compactor: Don't halt compactor when overlapped source blocks detected. #5854
• [CHANGE] S3 Bucket Client: Expose -blocks-storage.s3.send-content-md5 flag and set default checksum algorithm to MD5. #5870
• [CHANGE] Querier: Mark querier.iterators and querier.batch-iterators flags as deprecated. Now querier always use batch iterators. #5868
• [FEATURE] OTLP ingestion experimental. #5813
• [FEATURE] Ingester: Add per-tenant new metric cortex_ingester_tsdb_data_replay_duration_seconds. #5477
• [FEATURE] Query Frontend/Scheduler: Add query priority support. #5605
• [FEATURE] Tracing: Add kuberesolver to resolve endpoints address with kubernetes:// prefix as Kubernetes service. #5731
• [FEATURE] Tracing: Add tracing.otel.round-robin flag to use round_robin gRPC client side LB policy for sending OTLP traces. #5731
• [FEATURE] Ruler: Add ruler.concurrent-evals-enabled flag to enable concurrent evaluation within a single rule group for independent rules. Maximum concurrency can be configured via ruler.max-concurrent-evals. #5766
• [FEATURE] Distributor Queryable: Experimental: Add config zone_results_quorum_metadata. When querying ingesters using metadata APIs such as label names and values, only results from quorum number of zones will be included and merged. #5779
• [FEATURE] Storage Cache Clients: Add config set_async_circuit_breaker_config to utilize the circuit breaker pattern for dynamically thresholding asynchronous set operations. Implemented in both memcached and redis cache clients. #5789
• [FEATURE] Ruler: Add experimental experimental.ruler.api-deduplicate-rules flag to remove duplicate rule groups from the Prometheus compatible rules API endpoint. Add experimental ruler.ring.replication-factor and ruler.ring.zone-awareness-enabled flags to configure rule group replication, but only the first ruler in the replicaset evaluates the rule group, the rest will just hold a copy as backup. Add experimental experimental.ruler.api-enable-rules-backup flag to configure rulers to send the rule group backups stored in the replicaset to handle events when a ruler is down during an API request to list rules. #5782
• [ENHANCEMENT] Store Gateway: Added -store-gateway.enabled-tenants and -store-gateway.disabled-tenants to explicitly enable or disable store-gateway for specific tenants. #5638
• [ENHANCEMENT] Compactor: Add new compactor metric cortex_compactor_start_duration_seconds. #5683
• [ENHANCEMENT] Index Cache: Multi level cache adds config max_backfill_items to cap max items to backfill per async operation. #5686
• [ENHANCEMENT] Query Frontend: Log number of split queries in query stats log. #5703
• [ENHANCEMENT] Logging: Added new options for logging HTTP request headers: -server.log-request-headers enables logging HTTP request headers, -server.log-request-headers-exclude-list allows users to specify headers which should not be logged. #5744
• [ENHANCEMENT] Query Frontend/Scheduler: Time check in query priority now considers overall data select time window (including range selectors, modifiers and lookback delta). #5758
• [ENHANCEMENT] Querier: Added querier.store-gateway-query-stats-enabled to enable or disable store gateway query stats log. #5749
• [ENHANCEMENT] AlertManager: Retrying AlertManager Delete Silence on error #5794
• [ENHANCEMENT] Ingester: Add new ingester metric cortex_ingester_max_inflight_query_requests. #5798
• [ENHANCEMENT] Query: Added query_storage_wall_time to Query Frontend and Ruler query stats log for wall time spent on fetching data from storage. Query evaluation is not included. #5799
• [ENHANCEMENT] Query: Added additional max query length check at Query Frontend and Ruler. Added -querier.ignore-max-query-length flag to disable max query length check at Querier. #5808
• [ENHANCEMENT] Querier: Add context error check when converting Metrics to SeriesSet for GetSeries on distributorQuerier. #5827
• [ENHANCEMENT] Ruler: Improve GetRules response time by refactoring mutexes and introducing a temporary rules cache in ruler/manager.go. #5805
• [ENHANCEMENT] Querier: Add context error check when merging slices from ingesters for GetLabel operations. #5837
• [ENHANCEMENT] Ring: Add experimental -ingester.tokens-generator-strategy=minimize-spread flag to enable the new minimize spread token generator strategy. #5855
• [ENHANCEMENT] Query Frontend: Ensure error response returned by Query Frontend follows Prometheus API error response format. #5811
• [ENHANCEMENT] Ring Status Page: Add Ownership Diff From Expected column in the ring table to indicate the extent to which the ownership of a specific ingester differs from the expected ownership. #5889
• [BUGFIX] Distributor: Do not use label with empty values for sharding #5717
• [BUGFIX] Query Frontend: queries with negative offset should check whether it is cacheable or not. #5719
• [BUGFIX] Redis Cache: pass cache_size config correctly. #5734
• [BUGFIX] Distributor: Shuffle-Sharding with IngestionTenantShardSize == 0, default sharding strategy should be used #5189
• [BUGFIX] Cortex: Fix GRPC stream clients not honoring overrides for call options. #5797
• [BUGFIX] Ring DDB: Fix lifecycle for ring counting unhealthy pods as healthy. #5838
• [BUGFIX] Ring DDB: Fix region assignment. #5842

New Contributors

• @testwill made their first contribution in #5644
• @dsabsay made their first contribution in #5684
• @pawarpranav83 made their first contribution in #5719
• @Kramer0x0 made their first contribution in #5743
• @tesla59 made their first contribution in #5746
• @blorby made their first contribution in #5767
• @CharlieTLe made their first contribution in #5784
• @lekaf974 made their first contribution in #5793
• @euniceek made their first contribution in #5794
• @mustafain117 made their first contribution in #5823
• @availhang made their first contribution in #5826
• @erlan-z made their first contribution in #5827
• @yj-yoo made their first contribution in #5775
• @kindknow made their first contribution in #5856
• @momantech made their first contribution in #5863
• @till made their first contribution in #5874

Full Changelog: v1.16.1...v1.17.0

Cortex 1.17.0-rc.1

Over v1.17.0-rc.0 to include one bug fix and one change.

• [CHANGE] Ruler: Remove experimental.ruler.api-enable-rules-backup flag and use ruler.ring.replication-factor to check if rules backup is enabled. #5901
• [BUGFIX] Fix random string used in test allocating 50MB in Cortex binary. #5903

Cortex 1.17.0-rc.0

This release contains 166 contributions from 29 contributors. We also have 16 new contributors. Thank you all for the contributions!

Some notable changes release are:

• Experimental OTLP ingestion
• Experimental minimize spread token generator strategy on Ingester
• Advanced query scheduling with Query Priority
• ListRules API high availability by rule group replication and backup
• Various improvements on Store Gateway Index Cache
• mem-ballast-size-bytes flag has been marked as deprecated and not functional anymore
• -querier.ingester-streaming flag has been marked as deprecated and ingester streaming is always enabled now
• querier.iterators and querier.batch-iterators flags have been marked as deprecated and batch iterator is always enabled in Querier now

Cortex

• [CHANGE] Azure Storage: Upgraded objstore dependency and support Azure Workload Identity Authentication. Added connection_string to support authenticating via SAS token. Marked msi_resource config as deprecating. #5645
• [CHANGE] Store Gateway: Add a new fastcache based inmemory index cache. #5619
• [CHANGE] Index Cache: Multi level cache backfilling operation becomes async. Added -blocks-storage.bucket-store.index-cache.multilevel.max-async-concurrency and -blocks-storage.bucket-store.index-cache.multilevel.max-async-buffer-size configs and metric cortex_store_multilevel_index_cache_backfill_dropped_items_total for number of dropped items. #5661
• [CHANGE] Ingester: Disable uploading compacted blocks and overlapping compaction in ingester. #5735
• [CHANGE] Distributor: Count the number of rate-limited samples in distributor_samples_in_total. #5714
• [CHANGE] Ruler: Remove cortex_ruler_write_requests_total, cortex_ruler_write_requests_failed_total, cortex_ruler_queries_total, cortex_ruler_queries_failed_total, and cortex_ruler_query_seconds_total metrics for the tenant when the ruler deletes the manager for the tenant. #5772
• [CHANGE] Main: Mark mem-ballast-size-bytes flag as deprecated. #5816
• [CHANGE] Querier: Mark -querier.ingester-streaming flag as deprecated. Now query ingester streaming is always enabled. #5817
• [CHANGE] Compactor/Bucket Store: Added -blocks-storage.bucket-store.block-discovery-strategy to configure different block listing strategy. Reverted the current recursive block listing mechanism and use the strategy Concurrent as in 1.15. #5828
• [CHANGE] Compactor: Don't halt compactor when overlapped source blocks detected. #5854
• [CHANGE] S3 Bucket Client: Expose -blocks-storage.s3.send-content-md5 flag and set default checksum algorithm to MD5. #5870
• [CHANGE] Querier: Mark querier.iterators and querier.batch-iterators flags as deprecated. Now querier always use batch iterators. #5868
• [CHANGE] Query Frontend: Error response returned by Query Frontend now follows Prometheus API error response format. #5811
• [FEATURE] Experimental: OTLP ingestion. #5813
• [FEATURE] Query Frontend/Scheduler: Add query priority support. #5605
• [FEATURE] Tracing: Use kuberesolver to resolve OTLP endpoints address with kubernetes:// prefix as Kubernetes service. #5731
• [FEATURE] Tracing: Add tracing.otel.round-robin flag to use round_robin gRPC client side LB policy for sending OTLP traces. #5731
• [FEATURE] Ruler: Add ruler.concurrent-evals-enabled flag to enable concurrent evaluation within a single rule group for independent rules. Maximum concurrency can be configured via ruler.max-concurrent-evals. #5766
• [FEATURE] Distributor Queryable: Experimental: Add config zone_results_quorum_metadata. When querying ingesters using metadata APIs such as label names and values, only results from quorum number of zones will be included and merged. #5779
• [FEATURE] Storage Cache Clients: Add config set_async_circuit_breaker_config to utilize the circuit breaker pattern for dynamically thresholding asynchronous set operations. Implemented in both memcached and redis cache clients. #5789
• [FEATURE] Ruler: Add experimental experimental.ruler.api-deduplicate-rules flag to remove duplicate rule groups from the Prometheus compatible rules API endpoint. Add experimental ruler.ring.replication-factor and ruler.ring.zone-awareness-enabled flags to configure rule group replication, but only the first ruler in the replicaset evaluates the rule group, the rest will just hold a copy as backup. Add experimental experimental.ruler.api-enable-rules-backup flag to configure rulers to send the rule group backups stored in the replicaset to handle events when a ruler is down during an API request to list rules. #5782
• [FEATURE] Ring: Add experimental -ingester.tokens-generator-strategy=minimize-spread flag to enable the new minimize spread token generator strategy. #5855
• [FEATURE] Ring Status Page: Add Ownership Diff From Expected column in the ring table to indicate the extent to which the ownership of a specific ingester differs from the expected ownership. #5889
• [ENHANCEMENT] Ingester: Add per-tenant new metric cortex_ingester_tsdb_data_replay_duration_seconds. #5477
• [ENHANCEMENT] Store Gateway: Added -store-gateway.enabled-tenants and -store-gateway.disabled-tenants to explicitly enable or disable store-gateway for specific tenants. #5638
• [ENHANCEMENT] Query Frontend: Write service timing header in response even though there is an error. #5653
• [ENHANCEMENT] Compactor: Add new compactor metric cortex_compactor_start_duration_seconds. #5683
• [ENHANCEMENT] Index Cache: Multi level cache adds config max_backfill_items to cap max items to backfill per async operation. #5686
• [ENHANCEMENT] Query Frontend: Log number of split queries in query stats log. #5703
• [ENHANCEMENT] Compactor: Skip compaction retry when encountering a permission denied error. #5727
• [ENHANCEMENT] Logging: Added new options for logging HTTP request headers: -server.log-request-headers enables logging HTTP request headers, -server.log-request-headers-exclude-list allows users to specify headers which should not be logged. #5744
• [ENHANCEMENT] Query Frontend/Scheduler: Time check in query priority now considers overall data select time window (including range selectors, modifiers and lookback delta). #5758
• [ENHANCEMENT] Querier: Added querier.store-gateway-query-stats-enabled to enable or disable store gateway query stats log. #5749
• [ENHANCEMENT] Querier: Improve labels APIs latency by merging slices using K-way merge and more than 1 core. #5785
• [ENHANCEMENT] AlertManager: Retrying AlertManager Delete Silence on error. #5794
• [ENHANCEMENT] Ingester: Add new ingester metric cortex_ingester_max_inflight_query_requests. #5798
• [ENHANCEMENT] Query: Added query_storage_wall_time to Query Frontend and Ruler query stats log for wall time spent on fetching data from storage. Query evaluation is not included. #5799
• [ENHANCEMENT] Query: Added additional max query length check at Query Frontend and Ruler. Added -querier.ignore-max-query-length flag to disable max query length check at Querier. #5808
• [ENHANCEMENT] Querier: Add context error check when converting Metrics to SeriesSet for GetSeries on distributorQuerier. #5827
• [ENHANCEMENT] Ruler: Improve GetRules response time by reducing lock contention and introducing a temporary rules cache in ruler/manager.go. #5805
• [ENHANCEMENT] Querier: Add context error check when merging slices from ingesters for GetLabel operations. #5837
• [BUGFIX] Distributor: Do not use label with empty values for sharding #5717
• [BUGFIX] Query Frontend: queries with negative offset should check whether it is cacheable or not. #5719
• [BUGFIX] Redis Cache: pass cache_size config correctly. #5734
• [BUGFIX] Distributor: Shuffle-Sharding with ingestion_tenant_shard_size set to 0, default sharding strategy should be used. #5189
• [BUGFIX] Cortex: Fix GRPC stream clients not honoring overrides for call options. #5797
• [BUGFIX] Ruler: Fix support for keep_firing_for field in alert rules. #5823
• [BUGFIX] Ring DDB: Fix lifecycle for ring counting unhealthy pods as healthy. #5838
• [BUGFIX] Ring DDB: Fix region assignment. #5842

New Contributors

• @testwill made their first contribution in #5644
• @dsabsay made their first contribution in #5684
• @pawarpranav83 made their first contribution in #5719
• @Kramer0x0 made their first contribution in #5743
• @tesla59 made their first contribution in #5746
• @blorby made their first contribution in #5767
• @CharlieTLe made their first contribution in #5784
• @lekaf974 made their first contribution in #5793
• @euniceek made their first contribution in #5794
• @mustafain117 made their first contribution in #5823
• @availhang made their first contribution in #5826
• @erlan-z made their first contribution in #5827
• @yj-yoo made their first contribution in #5775
• @kindknow made their first contribution in #5856
• @momantech made their first contribution in #5863
• @till made their first contribution in #5874

Full Changelog: v1.16.1...v1.17.0-rc.0

Cortex 1.16.1

This release includes two security fixes:

• [ENHANCEMENT] Upgraded Docker base images to alpine:3.18. #5684
• [ENHANCEMENT] Upgrade to go 1.21.9 #5879 #5882