Before Creating the Enhancement Request
Summary
This PR introduces an opt-in broker config suppressMinValueMetrics (default false, backward compatible) that suppresses exporting data points whose value is not greater than a
minimal threshold.
Motivation
The consumer lag gauges are asynchronous ObservableGauges that emit one data point per consumer_group × topic × is_retry combination. In deployments with many consumer groups and topics, the vast majority of these combinations are idle at any given sampling instant, so their values are 0 — yet they are still recorded and exported on every collection cycle. This inflates both metric cardinality and the OTLP export payload.
In production, we observed zero-valued points accounting for ~95% of gauge data points and the bulk of the payload. Suppressing them is safe as long as the monitoring pipeline tolerates discontinuous series.
Describe the Solution You'd Like
BrokerConfig: add suppressMinValueMetrics (default false).BrokerMetricsManager: add a general guard shouldRecordValue(currentValue, minValue).
When suppression is enabled, a data point is recorded only if currentValue > minValue.- Apply the guard at the record site of the 5 consumer lag gauges, all passing
minValue = 0 for now, i.e. zero-valued points are dropped.
Describe Alternatives You've Considered
.
Additional Context
When enabled, threshold-based dashboards/alerts (e.g. lag > N) are unaffected; the only difference is
that idle series appear as gaps instead of a flat zero line. Therefore, before turning this on, make sure
there are no alert rules based on absent() / absent_over_time() (or any equivalent "no data" detection) targeting these metrics.
Before Creating the Enhancement Request
Summary
This PR introduces an opt-in broker config
suppressMinValueMetrics(defaultfalse, backward compatible) that suppresses exporting data points whose value is not greater than aminimal threshold.
Motivation
The consumer lag gauges are asynchronous ObservableGauges that emit one data point per consumer_group × topic × is_retry combination. In deployments with many consumer groups and topics, the vast majority of these combinations are idle at any given sampling instant, so their values are 0 — yet they are still recorded and exported on every collection cycle. This inflates both metric cardinality and the OTLP export payload.
In production, we observed zero-valued points accounting for ~95% of gauge data points and the bulk of the payload. Suppressing them is safe as long as the monitoring pipeline tolerates discontinuous series.
Describe the Solution You'd Like
BrokerConfig: addsuppressMinValueMetrics(defaultfalse).BrokerMetricsManager: add a general guardshouldRecordValue(currentValue, minValue).When suppression is enabled, a data point is recorded only if
currentValue > minValue.minValue = 0for now, i.e. zero-valued points are dropped.Describe Alternatives You've Considered
.
Additional Context
When enabled, threshold-based dashboards/alerts (e.g.
lag > N) are unaffected; the only difference isthat idle series appear as gaps instead of a flat zero line. Therefore, before turning this on, make sure
there are no alert rules based on
absent()/absent_over_time()(or any equivalent "no data" detection) targeting these metrics.