Repository navigation
Description
1. Quick Debug Information
- OS/Version(e.g. RHEL8.6, Ubuntu22.04): Ubuntu 22.04.5
- Kernel Version: 5.15.0-179-generic
- Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): containerd 2.1.5
- K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): RKE2 v1.34.3
- Device plugin: v0.18.1 (deployed by GPU Operator), default config — full GPUs, no MIG, no time-slicing
- Driver: 580.159.03 (open kernel module), 8x GeForce RTX 5090, no NVLink
2. Issue or feature description
Expected behavior. When one GPU becomes unusable, pods that fit in the remaining healthy GPUs should still be admitted. The unhealthy device should not be able to break allocation for the rest of the node.
Current behavior. When a GPU falls off the PCIe bus (Xid 79), the health checker correctly marks that device Unhealthy and the node's nvidia.com/gpu allocatable drops from 8 to 7 — but from that moment every GPU pod scheduled to the node is rejected at kubelet admission, including pods that request a single GPU:
Warning UnexpectedAdmissionError pod/... Allocate failed due to device plugin GetPreferredAllocation rpc failed
with err: rpc error: code = Unknown desc = error getting list of preferred allocation devices:
unable to get device link information: ...
The scheduler only looks at allocatable = 7, so it keeps binding pods to the node while kubelet keeps rejecting them. Any controller-owned pod (a Deployment in our case) therefore enters a tight create → Failed loop: we measured ~1,000 Failed pods per 5 minutes per Deployment, sustained for 3 days until the workloads were stopped by hand, with the pod objects churning against --terminated-pod-gc-threshold. The device plugin itself kept running the whole time (it had initialised before the fault), so nothing restarted it, and gpu-feature-discovery on the same node crash-looped with error getting device handle for index '4': Unknown Error.
Where it fails (v0.18.1; the same code is on main today). On a node that exposes whole GPUs — i.e. not MIG devices and not replicated/shared ones, which is what Devices.AlignedAllocationSupported() checks — getPreferredAllocation always takes the aligned path, and that path builds the topology graph for all devices on the node before the available list is applied — internal/rm/nvml_manager.go:
func (r *nvmlResourceManager) alignedAlloc(available, required []string, size int) ([]string, error) {
linkedDevices, err := gpuallocator.NewDevices(
gpuallocator.WithNvmlLib(r.nvml),
)
if err != nil {
return nil, fmt.Errorf("unable to get device link information: %w", err)
}
availableDevices, err := linkedDevices.Filter(available) // filtering happens only after the full walkIn go-gpuallocator, NewDevices → build() enumerates every device by index (devicelib.GetDevices() → DeviceGetHandleByIndex(i)) and then queries P2P/NVLink topology for every pair of devices. A lost GPU fails either at the handle lookup or at one of the pairwise topology calls, so the whole call returns an error. The plugin passes that error to kubelet, and kubelet's callGetPreferredAllocationIfAvailable has no fallback — the error becomes an admission failure.
Two consequences worth calling out:
- The plugin already knows which device is unhealthy (it reported it through
ListAndWatch, and kubelet excludes it fromAvailableDeviceIDs), but the allocation path never uses that information. - There is no way to opt out:
GetPreferredAllocationAvailableis unconditionallytrueand no flag disables it.
Steps to reproduce
- A node with >= 2 full GPUs, device plugin v0.18.1, default configuration.
- Make one GPU unreachable over PCIe. We hit a genuine Xid 79; the host then shows
nvidia-smi -L→Unable to determine the device handle for gpu <bdf>: Unknown Error, andlspci -s <bdf> -xxxreads all0xff(the device no longer answers config reads). - Confirm
allocatable= N-1 and the plugin log lineXidCriticalError: Xid=79 ... marking device as unhealthy. - Create a pod requesting
nvidia.com/gpu: 1with a nodeSelector for that node →UnexpectedAdmissionErrorwith the message above. - Wrap the same pod in a Deployment → unbounded Failed-pod churn.
Proposed fix (happy to send the PRs)
- A. Degrade instead of reject (plugin only, small): in
alignedAlloc, ifgpuallocator.NewDevicesfails, log a warning and fall back to a topology-unaware allocation (requiredfirst, thenavailablein order, up tosize) instead of returning an error. This removes both the admission failure and the pod churn; the only thing lost is topology optimisation while the node is degraded. It does not change health reporting, so no device is marked unhealthy that isn't known to be. - B. Build the graph only from the candidate devices (
go-gpuallocator+ plugin): add an option such asWithDeviceUUIDs(uuids []string)so thatNewDevicesenumerates by UUID and computes links only within that subset, and have the plugin passavailable ∪ required. A lost GPU is then never touched. This is the more complete fix but spans two repositories and a release ordering.
I would suggest landing A first (a few lines plus unit tests that inject a lost device and assert the healthy ones are still allocated), then B — but the direction is of course yours to decide, and I am happy to go straight to B, to a variant of either, or to drop A entirely if you would rather fix this only in go-gpuallocator.
Worth noting that the two do not overlap: once B is in place this class of failure never reaches the fallback, because the lost device is no longer enumerated. A would remain as a safety net for any other reason the link discovery can fail (NVML initialisation problems, transient driver errors), where returning a topology-unaware allocation is still preferable to failing admission for the whole node.
One design question I expect: should the fallback trigger on any NewDevices failure, or only on a lost device? I went with any failure, because the plugin cannot reliably classify the underlying NVML error, and the fallback result is still a correct allocation — it is only not topology-optimal, which is the same trade-off the plugin already accepts on nodes where links are reported as P2PLinkUnknown. Narrowing it to specific NVML return codes is easy to add if you prefer.
I already have the patch and unit tests for A ready locally (one file changed, no new dependencies, make test/make lint/make check-modules clean) and will open a PR shortly. Happy to be assigned this if that helps.
Relation to existing issues. #231 (2021) reports exactly this failure — "a single GPU crashing causes the other GPUs to become unallocatable", with the same error string and the same Failed-pod accumulation — and was closed by the stale bot in 2025 with a request to reopen in 2026. #1640 describes the same cascading pattern from a different trigger (NVML init failure) and was closed asking for a retest. #1641 proposed marking all devices unhealthy when the health check fails to start and was closed with the guidance that an NVML init failure should make the plugin exit instead. This report is deliberately narrower and different from that case: NVML is initialised and healthy for the other seven devices, and only per-device queries for the lost GPU fail, so neither "exit the plugin" nor "mark everything unhealthy" addresses it — the allocation path simply needs to tolerate (or avoid) the lost device.
3. Information to attach (optional if deemed irrelevant)
Common error checking:
- The output of
nvidia-smi -aon your host — the lost GPU is not enumerated at all:Unable to determine the device handle for GPU 0000:99:00.0: Unknown Error # the other 7 GPUs report normally (idle, ~30 C, ~13 W) - Your docker configuration file — N/A (containerd)
- The k8s-device-plugin container logs — health check reacted correctly:
No further plugin errors and no restart; the plugin kept serving with 7 healthy devices.
health.go:143] Processing event {Device:{...} EventType:8 EventData:79 ...} health.go:169] XidCriticalError: Xid=79 on Device=GPU-3d04...; marking device as unhealthy. server.go:278] 'nvidia.com/gpu' device marked unhealthy: GPU-3d04... - The kubelet logs / pod events on the node — one line per rejected pod:
Allocate failed due to device plugin GetPreferredAllocation rpc failed with err: rpc error: code = Unknown desc = error getting list of preferred allocation devices: unable to get device link information: ...
Additional information that might help better understand your environment and reproduce the bug:
- Docker version — N/A (containerd 2.1.5)
- Image and tag used:
nvcr.io/nvidia/k8s-device-plugin:v0.18.1(GPU Operator deployment) - Kernel version:
Linux 5.15.0-179-generic #189-Ubuntu SMP x86_64 - Relevant kernel output from
dmesg(host name removed):For completeness on how long the bad window can last in practice: a softNVRM: GPU at PCI:0000:99:00: GPU-3d04... NVRM: Xid (PCI:0000:99:00): 79, GPU has fallen off the bus. NVRM: GPU 0000:99:00.0: GPU has fallen off the bus. NVRM: kgspRcAndNotifyAllChannels_IMPL: RC all channels for critical error 79. NVRM: Xid (PCI:0000:17:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (Node Reboot Required) ... (Xid 154 repeated for all 8 GPUs)reboothung in the shutdown path waiting for the lost device to quiesce, so the node needed a full power cycle. After the power cycle all 8 GPUs enumerated again and passed a load test, but the plugin had been rejecting every GPU pod on that node for three days. - NVIDIA driver version: 580.159.03 (open kernel module,
nvidia-driver-580-server-open) - NVIDIA Container Toolkit: 1.18.x (GPU Operator managed)
Activity
sirzzang commentedon Sep 17, 2026
Opened #2039 with the option A fallback (one file changed, no new dependencies, unit tests for the lost-device path). Happy to iterate there or to change direction if you would rather fix this in
go-gpuallocatoronly.ZacxDev commentedon Sep 30, 2026
Independent reproduction, on a hardware class that I think strengthens the case for the unaligned fallback in #2039: consumer RTX 4090 nodes, which have no NVLink at all.
Environment:
k8s-device-plugin:v0.18.1, driver580.65.06, 8× RTX 4090 per node, Kubernetes 1.35, Ubuntu 22.04.Two nodes independently hit
Xid 79on one card. The plugin's health check did the right thing and logged it:The node then sits at
capacity 8 / allocatable 7, and NVML no longer enumerates the lost card —nvidia-smilists seven.A pod requesting 6 GPUs — fewer than both capacity and allocatable — schedules, and is then rejected by the kubelet:
The owning DaemonSet then recreates the pod roughly every 26 s indefinitely.
The part I want to add to the thread:
nvidia-smi nvlink --statuson a healthy node of this model returns empty output with rc 0 — there is no NVLink on these GPUs. So the aligned path is building link information for a topology that does not exist, and it can never yield a better placement than the unaligned one, yet it is what takes the node out entirely. On this hardware the fallback in #2039 is not a degradation, it is the only sensible path.Net effect, which is worth stating plainly because it surprised us: on a node with one off-bus GPU there is no request value that works. A request equal to capacity never schedules (allocatable is one lower); anything smaller schedules and then fails to allocate. One dead card idles all eight.
We have no workaround. We tried lowering the request — which is how we found this — and reverted it, because it converts a quiet
Pendinginto an admission-retry loop with no gain. The only thing that restores capacity is a cold power-cycle to re-enumerate the card.sirzzang commentedon Sep 30, 2026
Thanks for the independent reproduction. Different GPU model, driver and Kubernetes version, same outcome, and your traceback points at exactly the branch #2039 changes: in v0.18.1
alignedAllocwraps thegpuallocator.NewDevicesfailure asunable to get device link information, and the innererror getting NVLink for devices (1, 0)comes from link discovery walking every pair of physical devices, including the one the health check has already marked unhealthy and removed fromavailable.Your trace also narrows down where it fails.
build()iterates i-major, so every pair with i=0, including (0, 1), completed; the failing call isGetNVLink(dev1, ...), which queries the NVLink state of its first argument, so the lost card is index 1. The plugin's NVML session still enumerated that card: its handle, UUID, PCI info and the pairwiseGetP2PLinkquery all succeeded, even though a freshnvidia-smilists only seven, and only the NVLink state query returnedGPU_IS_LOST. The tests in #2039 injected the failure at handle lookup, so I have added a case for this path too, with an assertion that discovery fails on the NVLink state query rather than earlier: it reproduceserror getting NVLink for devices (1, 0): failed to get nvlink remote pci info: failed to get nvlink state: ERROR_GPU_IS_LOST(go-nvml's default string, since the test does not load the NVML library), and the allocation cases fail without the fallback.One narrowing on the NVLink point, because the argument is stronger without it. The best-effort policy scores PCIe-level relations as well as NVLink:
P2PLinkCrossCPUthroughP2PLinkSameBoardcontribute 10 to 60 per pair, NVLink 100 and up, and link discovery queries both P2P and NVLink for every pair. On our own 8-GPU consumer node, which also has no NVLink, the matrix is not flat.nvidia-smi topo -mon one of our nodesNODEmaps toP2PLinkSameCPU(20) andSYStoP2PLinkCrossCPU(10), so a 4-GPU set taken from one socket scores 6 pairs × 20 = 120, against 2 × 20 + 4 × 10 = 80 for a 2+2 split. An aligned allocation does have something to choose on this class of hardware. Peer-to-peer DMA is disabled on these cards (nvidia-smi topo -p2p rreportsCNSfor every pair), which shrinks that gap because copies stage through host memory, but a cross-socket pair still traverses the SMP interconnect.So the claim I would defend is the narrower one, and it holds on every hardware class: the fallback is entered only when link discovery fails, and at that moment no topology information is available to the allocator. The comparison is therefore not an aligned placement against an unaligned one, it is an unaligned placement against no placement at all. On a healthy node the code path is unchanged.
Your summary that no request value works matches what we see: a request equal to capacity stays Pending because allocatable is one lower, anything below it is admitted and then rejected. We have since hit it a second time on the same host, 11 days after the first, with the same card and the same PCI address. One difference worth recording for severity: the owning controller sets the pace. Your DaemonSet keeps one pod per node and retries about every 26 s, while the Deployment in our case created roughly a thousand failed pods every five minutes, for three days before anyone noticed.
On your last point it helps to separate the two cards. The power cycle is what the lost card itself needs: after Xid 79 it stops answering config reads, and on our host a soft reboot hung waiting for it. The other seven are a different matter. They are healthy and still advertised as allocatable, and they sit idle only because allocation walks the lost card. That is the part #2039 addresses. Your run on Kubernetes 1.35 is useful here too: it shows the kubelet side is unchanged there as well, since a failed
GetPreferredAllocationstill ends the admission with no fallback of its own.sirzzang commentedon Nov 13, 2026
I have been following how this issue was tackled in this repository.
First of all, thanks to @ZacxDev for independently reproducing this on 09-30. Since the fallback approach was put on hold by a maintainer in #2039 (comment), I'd like to follow the direction suggested there instead.
#2149 limits the GPU enumeration in
alignedAllocto the union of available and required UUIDs. It does this without modifyinggo-gpuallocator: it passes a wrapper (requestedDeviceLib) through the existingWithDeviceLiboption. I think this has the same effect as "building topology only fromAvailableDeviceIDs" as suggested in that comment.The unit test
a broken device that was not offered is not queriedin #2149 matches the situation in this issue. A failed GPU is already marked Unhealthy, so kubelet excludes it fromAvailableDeviceIDs, and the new enumeration never queries it.Given that, I'll close #2039 in favor of #2149 and leave this issue open until #2149 is merged.
aryangorwade commentedon Nov 16, 2026
Hi @sirzzang thanks for bringing this up! We wanted to avoid updating
go-gpuallocatorif possible, hence the wrapper workaround. We are looking into merging #2149 soon.