Repository navigation
Add Prometheus metrics for jobs and queues - #84
Merged
Merged
Conversation
Exposes job queue/execution metrics (queued, running, success, failed) per task via /metrics endpoint using VictoriaMetrics client, following the pattern from zerodha/kaf-relay. Closes zerodha#46
- Count jobs enqueued in groups and rename the counter to jobs_enqueued_total. - Escape label values so unusual task names no longer crash startup. - Replace success/failed counters with job_attempts_total by outcome (success, error, canceled). - Add execution duration, queue wait, pending jobs and worker concurrency metrics. - Document the metrics and the worker-only limitation in the README.
# Conflicts: # go.mod # go.sum
shreyaabaranwal
left a comment
shreyaabaranwal
left a comment
Contributor
There was a problem hiding this comment.
Went through the final diff after this landed and compared it with my original commit from #70. Thanks for picking it up and keeping the original commit, @kalbhor.
A few things I learned from the changes, mostly noting them for myself:
- Escaping label values. I hadn't thought about task names with quotes. Since the metrics are created in
initQueue(), a bad name would have stopped the server from starting at all. job_attempts_totalwith anoutcomelabel instead of separate success/failed counters. My callback ran once per attempt, so with retries my counters were really counting attempts anyway. Splitting outcanceledalso stops deleted jobs from showing up as failures.- Checking
ctx.Err()instead of comparing withcontext.Canceled, since pq returns its own error on cancel. - Measuring queue wait only on the first attempt, and keeping
ReadyAtsafe for jobs that older versions enqueued without it.
I also checked the GOMAXPROCS fallback in registerQueueMetrics against tasqueue. It matches the default NewServer sets, so the concurrency gauge is right for tasks without a concurrency value.
One thing I noticed: the pending jobs gauge calls GetPending() on every scrape, which is a full LRANGE on Redis. tasqueue v2.3.0 has GetPendingCount() on the Broker interface, so I've opened #86 to switch to it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Prometheus metrics for jobs and queues at
/metrics. Builds on #70 (original commit kept) and addresses the review feedback there. Closes #46.