Detect wedged vGPU VFs from the guest and report to the health store

[yummybomb]

Summary

Top half of the wedged-VF work, stacked on #462 (the VF health store, placement exclusion, admission, and /resources fields). This PR adds the detection path that feeds that store.

• The guest agent watches /dev/kmsg for kernel-facility NVRM: ... RmInitAdapter failed! records, probes driver init at boot with nvidia-smi -L when the image has it, and exposes the current GPU initialization state over GetGPUInitStatus via vsock.
• A host sentinel controller polls vendor-VFIO instances with up to 64 concurrent checks and reports failures and successes into the VF health store from Quarantine unhealthy vGPU VFs via a persisted health store #462.
• Only instances with a live VMM (control socket present) are polled; a stopped or standby instance whose failed release left its claim in metadata is skipped, since QEMU vsock dials by guest CID alone and a stale CID could be reused by an unrelated instance.
• Reports are keyed on GPUClaimedAt from Quarantine unhealthy vGPU VFs via a persisted health store #462, the identity a claim receives when it is persisted.

Rebuilt on the claim-first allocator

This branch was rebuilt from the new #462 head after #321 merged as the claim-first rewrite. The guest agent, guest RPC, sentinel controller, wiring, metrics, and docs are the previously reviewed code, squashed into one commit. What changed in the port:

• GPUAssignedAt is gone from main; the sentinel keys reports on GPUClaimedAt, which Quarantine unhealthy vGPU VFs via a persisted health store #462 sets in the same metadata save as the claim.
• The GPURetainedForCleanup skip is gone with the retention-record design. The existing control-socket check already excludes instances without a VMM.
• The "allocate the vGPU immediately before metadata persistence" change to create.go is dropped. Claim-first already persists the claim before the VF is touched.
• The reconcile liveness change is re-hooked into main's two hypervisorMayBeAlive call sites; reconcile and the create/start cleanup guard share one wrapper that logs the resolution error and increments the liveness counter when it fails closed.
• The retention-record tests that the old branch adjusted no longer exist and were dropped with them.

Rebased onto the current #462 head (cc6b625, itself rebased onto main after #428): the store now takes the threshold in InitVFHealth, retries a failed persist on read, and passes the quarantine set into the selector struct. The four review-round commits were squashed into the detection commit for the rebase; the review follow-ups (sentinel keyed on devices.FormatVFAssignedAt, throttled repair warning, lock comment on RepairVFHealthStore) are a separate commit on top.

Safety and failure handling

• GPU initialization success is terminal in the guest agent, so a delayed failure watcher cannot replace a confirmed success.
• The host reads GPU init state only from the guest-agent RPC; workload output on the shared serial console cannot influence the tally, and userspace writes to /dev/kmsg are rejected by the kernel-facility check. The guest is still the reporter, so a root workload that replaces the agent can report a failure; the per-assignment threshold and randomized VF selection bound how fast that drains capacity.
• A slow nvidia-smi attempt is killed after 30 seconds. The guest agent waits for that process to be reaped before retrying, so an attempt stuck in uninterruptible I/O cannot accumulate concurrent probes; the independent kmsg watcher still reports the underlying init failure.
• Unreachable guests do not serialize a host-wide scan; checks run with a fixed concurrency limit.
• Repeated polls and controller restarts do not double-count an assignment. The sentinel revalidates the VF assignment before reporting, and the health store deduplicates reports by assignment.
• Reconciliation preserves claims when hypervisor liveness is uncertain and records that condition in logs and metrics.

Observability

• hypeman_instances_vgpu_sentinel_init_failures_total
• hypeman_instances_vgpu_sentinel_quarantines_total
• hypeman_instances_vgpu_sentinel_checks_total by result (ok, failed, unknown, rpc_error, unsupported_agent, or list_error)
• hypeman_instances_vgpu_quarantined_vfs
• hypeman_instances_vgpu_vf_health_store_unavailable
• hypeman_instances_vgpu_liveness_uncertain_total

lib/devices/GPU.md gains the detection and sentinel documentation.

Known gap

The nvidia-smi -L probe is the only source of an OK state. An image without nvidia-smi, or a driver that takes longer than the 10 minute probe window to initialize, stays UNKNOWN: its failures are still detected and tallied, but its assignments can never clear a tally or rescind a quarantine. Fleets running such images recover VFs only through the manual runbook in lib/devices/GPU.md.

Out of scope

An operator force-cycle endpoint. Recovery remains the documented manual DCGM quiesce, SR-IOV cycle, state edit, restart, and verification flow.

Testing

Passed locally:

go vet ./lib/instances ./lib/devices ./lib/guest ./lib/system/guest_agent ./lib/providers ./cmd/api
go test -race ./lib/devices ./lib/guest ./lib/system/guest_agent ./lib/resources ./lib/providers
go test -race ./lib/instances -run 'VGPU|Sentinel|Reconcile|Metric|HypervisorMayBeAlive|ResolveLiveHypervisor|Select|Quarantin|StoredVGPU|Cleanup|ReleaseStored|Snapshot.*GPU|Fork.*GPU'

cmd/api/wire_gen.go was regenerated with wire and matches. lib/guest/guest.pb.go and guest_grpc.pb.go are carried over from the previous head; guest.proto has not changed on main since, and the header records the same protoc and protoc-gen-go versions as the committed files.

The full lib/instances suite was not run here; tests that boot real VM images need a host this environment does not provide.

The underlying wedge signal and manual recovery sequence were previously validated on L40S hardware. A live end-to-end run of the guest watcher and host controller is still required before merge, including two failed assignments and the success path.

---

Note

High Risk
Changes vGPU capacity management (quarantine), runs continuous vsock polling against live VMs, and tightens when stale VF claims are released—mistakes could mis-quarantine hosts or leave wedged VFs in rotation.

Overview
Adds an end-to-end path to detect guest NVIDIA RmInitAdapter failures on vendor VFIO vGPUs and feed the existing VF health store (quarantine / placement exclusion).

The guest agent watches kernel /dev/kmsg for NVRM init-failure lines, optionally probes with nvidia-smi -L, and exposes GetGPUInitStatus over vsock. Shared scanKmsg replaces ad-hoc kmsg reading in the clock keeper.

The vGPU sentinel controller runs in the API process (wire-injected), polls only running vendor-VFIO instances (control socket present), and records failures/successes keyed on GPUClaimedAt. It repairs an unavailable health store once per poll and emits sentinel-specific metrics; instance metrics add quarantine and store-unavailable gauges plus hypeman_instances_vgpu_liveness_uncertain_total when reconcile/cleanup cannot prove the hypervisor is dead.

VF health store changes: RepairVFHealthStore, stricter persist behavior (no-op reports succeed while persist is failed; mutations fail closed), and a fast path so routine OK reports skip vendorVFIOMu when the VF has no tallies.

Reconcile preserves claims on ambiguous hypervisor liveness via vgpuHypervisorMayBeAlive (replacing the removed package-level helper). GPU.md documents detection, limits (OK only from nvidia-smi), and observability.

^{Reviewed by Cursor Bugbot for commit b56faa6. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit b23a654. Configure here.}