Skip to content

feat(metrics): expose orphaned vGPU mdev gauge - #471

Draft
yummybomb wants to merge 1 commit into
mainfrom
hypeship/gpu-mdev-orphan-metric
Draft

feat(metrics): expose orphaned vGPU mdev gauge#471
yummybomb wants to merge 1 commit into
mainfrom
hypeship/gpu-mdev-orphan-metric

Conversation

@yummybomb

Copy link
Copy Markdown
Contributor

Summary

Adds a hypeman_gpu_mdevs_total gauge with state="claimed" and state="orphaned", emitted from the existing instance metrics callback on hosts that have mdevs present.

  • claimed: mdevs under /sys/bus/mdev/devices whose UUID matches the GPUMdevUUID stored on some instance.
  • orphaned: mdevs no instance references. Each one holds a vGPU slot that nothing will release until ReconcileMdevs runs at the next API startup.

Why

Leaked mdevs are currently only visible by subtracting used GPU slots from a count of running instances on a dashboard. That derivation has to guess which instances are GPU-backed (there is no gpu label on hypeman_instances_total), hardcodes host capacity, and mixes two gauges that are scraped at different instants, so it jitters and can't be alerted on. This gauge answers the question directly from one snapshot, using the same definition the startup reconciliation applies.

Notes

  • The series is only emitted when the host has at least one mdev, matching how hypeman_resources_gpu_slots is gated on GPU presence. Dashboards should use or vector(0) if they want a flat line on idle GPU hosts; an alert on > 0 works as-is.
  • Instances in any state count as owners. A stopped instance that retained its assignment after a failed release is a tracked retry, not an orphan.
  • ListMdevDevices is a directory read plus a couple of readlinks per mdev, bounded by the VF count, so per-collect cost is negligible.

Testing

  • go vet ./lib/instances/ on linux and GOOS=darwin cross-vet.
  • go test ./lib/instances/ -run 'TestCountGPUMdevs|Metric' passes, including the new unit test for the claimed/orphaned split.
  • Not exercised on a live vGPU host from this branch.

🤖 Generated with Claude Code

Add hypeman_gpu_mdevs_total{state="claimed"|"orphaned"}, computed in the
instance metrics callback by comparing host mdevs against the mdev UUIDs
stored on instances. An mdev no instance references holds a vGPU slot that
only startup reconciliation would release, and until now the only way to
see one was to diff two independently scraped gauges on a dashboard.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant