Repository navigation
[DO NOT REVIEW] openvmm_helpers: refuse a FIFO artifact without waiting for a writer - #5004
Merged
Pedro Henrique Penna (ppenna) merged 1 commit intoNov 10, 2026
Conversation
Restore hung, before it validated anything, when a FIFO without a writer stood in place of a snapshot's manifest.bin, state.bin, memory.bin, or paired scratch.img. Restore opens each artifact without following a final link and then checks the type of the file that it opened, but a read-only open of a FIFO waits until a writer opens the FIFO, so the open never returned to the check: --restore-snapshot, and a management-RPC CreateVm with a restore_path, waited in the open forever (microsoft/nvx#1154). The path-based opener, which the claimed scratch link and write_snapshot's memory backing file use, waited the same way. On Unix, both openers now add O_NONBLOCK, so that the open of a FIFO returns at once and the existing check refuses it like any other file that is not a regular file: "<artifact> is not a regular file". The check then clears O_NONBLOCK on the descriptor of a regular file, so the descriptors that restore keeps and hands to the VM worker, such as the one that the worker maps as RAM, have the flags of a blocking open, as before. Windows needs no change: NTFS holds no FIFOs, and the AF_UNIX sockets and WSL FIFOs that it can hold are reparse points, which the relative open opens as themselves, without waiting, and refuses. Tests replace each artifact of a paired-scratch resume snapshot with a FIFO without a writer and open the snapshot, and its paired scratch, and open a FIFO with each artifact opener, on another thread: each open must fail with the type error within 10 s, and the handle of a regular file must not be nonblocking. Without the change, the opens wait until the tests give the FIFOs writers, and the tests fail. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
added a commit
to microsoft/nvx
that referenced
this pull request
Nov 9, 2026
Pins cba21d109eeb99fbc573c79b2ed9f4754f734b25, the head of microsoft/openvmm#5004, one commit on ac3dc27a7, the head of microsoft/openvmm#5003, on which it stacks, which follows b4028c1aa, the head of feature/microvm/dev: - cba21d109 openvmm_helpers: refuse a FIFO artifact without waiting for a writer A FIFO without a writer in place of a snapshot's manifest.bin, state.bin, memory.bin, or paired scratch.img made restore wait forever before it validated anything (#1154). OpenVMM opened each artifact without following a final link and only then checked the type of the file that it opened, and a read-only open of a FIFO waits until a writer opens it. The pinned revision opens artifacts with O_NONBLOCK on Unix, so it refuses such a FIFO at once, with "<artifact> is not a regular file", before a restore can claim a resume snapshot, and then makes a regular file's descriptor blocking again. Windows is unchanged: its relative opens never wait for a file's type. The protocol vectors, the excerpts and contract versions that nvx-rs copies from OpenVMM, and its CPU profiles match the revision's. With 904a33c32, the same patch on d501e5066, on nvx-kvm-test and nvx-mshv-test (KVM and MSHV virtual machines), a FIFO in place of each artifact of a workload-start and an instance-checkpoint snapshot made the OpenVMM of release v0.1.0-dev.a0173e7b7d97, at 6cb6ba87c, wait in the kernel's wait_for_partner until it was killed 30 s later, while the fix's glibc and musl builds exited with status 1 and that error within 0.01 s; none of the 32 restores claimed the instance checkpoint. On Windows, an AF_UNIX socket, which is a reparse point, in place of each artifact made that release's openvmm.exe refuse it at once, so WHP had no such wait. The snapshot-tiers and scratch-snapshot scenarios passed with it on KVM, MSHV, and WHP. Tests: OpenVMM's openvmm_helpers tests pass at the pinned revision, 135 of 135 on Linux (glibc) and 127 of 127 on Windows, as they did at 904a33c32, 131 of 131 on Linux, glibc and musl, and 123 of 123 on Windows, where the two new ones failed without the change. The openvmm_vectors and openvmm_cpu_profiles comparisons pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
added a commit
to microsoft/nvx
that referenced
this pull request
Nov 9, 2026
Maps NVX's new pin, cba21d109, the head of microsoft/openvmm#5004, at position 237, after position 236, ac3dc27a7, the head of microsoft/openvmm#5003, on which it stacks, which follows position 235, b4028c1aa, the head of feature/microvm/dev. The pin is 106 commits past the mapped tip, and positions 132-237 stay unassigned. Position 237 changes the artifact openers of the snapshot file system module that PR11's position 032 adds, with the path-based opener and the type check, and its position 033 extends with the directory-relative opener, through which PR12's position 035 restores: on Unix both now open with O_NONBLOCK, so that the type check refuses a FIFO without waiting for a writer, and then make a regular file's descriptor blocking again. It adds tests beside positions 032's and 035's, and extends the restore paragraph that PR12's position 036 adds to the snapshot format guide. Replaying it on upstream main conflicts in all three of its files, since upstream main lacks the file system module of position 032, the directory-relative restore of position 035, and that paragraph, so it follows PR11, PR12, and PR19 upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
added a commit
to microsoft/nvx
that referenced
this pull request
Nov 9, 2026
A guest's snapshot request could crash a multi-vCPU microVM, or lose the device accesses of its other vCPUs (#1138). The snapshot boundary gates host input before it stops the vCPUs, so they run while each device's state unit changes the device's gate, and the unit held the device's mutex closed for the change, until the end of its wait. A port or MMIO access that another vCPU made then was dropped, a read returning all ones; an interrupt from another device was lost; and an EOI or a PCI configuration access panicked the VM worker with "lock should not be closed". Pin 46bfe02ed65697e858f66ce55eb60831b3ac3461, the head of microsoft/openvmm#5005, one commit on cba21d109, the head of microsoft/openvmm#5004, on which it stacks, which follows ac3dc27a7, the head of microsoft/openvmm#5003, and b4028c1aa, the head of feature/microvm/dev: - 46bfe02ed vmotherboard: gate a device's input without closing it to the vCPUs The revision splits a device's gate change into a synchronous part, which the unit makes under the lock that the vCPUs' accesses take, and a wait that does not borrow the device, which the unit awaits without it. The virtio transports and the remote device proxy send their gate's request in the synchronous part, in the order of the guest's accesses. Evidence: a microVM guest with 2 and 4 vCPUs requested 2000 snapshots, each of which gated host input and then failed its preflight, since its destination existed, while its other vCPUs read the magic value of the virtio-fs device's MMIO window, and with 4 vCPUs the console's too, in a loop. With OpenVMM at d6addf8c1, 145 to 2358 of a run's reads returned all ones, and OpenVMM logged that the device did not respond, in each of 3 runs per count on nvx-kvm-test (KVM), nvx-mshv-test (MSHV), and nvx-whp-test (WHP), all virtual machines; with a91d4c7a4, the same patch on d6addf8c1, no read did. With a91d4c7a4, the default microVM suite passed on all three, with 1, 2, and 4 vCPUs. At the pinned revision, the OpenVMM tests of the changed packages and the virtio devices pass, 545 on Windows and 550 on Linux, as they did at a91d4c7a4. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
added a commit
to microsoft/nvx
that referenced
this pull request
Nov 9, 2026
Maps NVX's new pin, 46bfe02ed, the head of microsoft/openvmm#5005, at position 238, after position 237, cba21d109, the head of microsoft/openvmm#5004, on which it stacks, which follows positions 236, ac3dc27a7, the head of microsoft/openvmm#5003, and 235, b4028c1aa, the head of feature/microvm/dev. The pin is 107 commits past the mapped tip, and positions 132-238 stay unassigned. Position 238 changes the input gate that PR09's position 018 adds: the chipset device unit, which changed a device's gate with the device's mutex closed, now makes the gate's synchronous part under the device's ordinary lock and awaits the rest without the device, since the vCPUs run through a gate change. It changes the gate's methods of ChangeDeviceState, the erased chipset devices, and the remote device proxy, which position 018 adds, and the virtio transports' forwarding of the gate, which PR13's position 040 adds, and adds tests of the unit, both virtio transports, and the proxy. Replaying it on upstream main conflicts in all eight of the files that it changes, since upstream main lacks the input gate of position 018 and its forwarding by position 040, so it follows PR09 and PR13 upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Nov 9, 2026
Pedro Henrique Penna (ppenna)
changed the base branch from
copilot/nvx-issue-1364-expansion-overflow
to
feature/microvm/dev
November 10, 2026 00:54
Pedro Henrique Penna (ppenna)
marked this pull request as ready for review
November 10, 2026 00:55
Pedro Henrique Penna (ppenna)
requested a review
from a team
as a code owner
November 10, 2026 00:55
Pedro Henrique Penna (ppenna)
merged commit Nov 10, 2026
cba21d1
into
feature/microvm/dev
73 of 74 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes a snapshot restore refuse a FIFO in place of an artifact at once, instead of waiting forever for a writer, for microsoft/nvx#1154. On Linux, a FIFO without a writer in place of a snapshot's
manifest.bin,state.bin,memory.bin, or pairedscratch.imgmade--restore-snapshot, and a management-RPCCreateVmwith arestore_path, wait in the open of the artifact, before restore validated anything or claimed a resume snapshot.One commit on
ac3dc27a7, the head of #5003, which it stacks on. microsoft/nvx#1424 pins this head. It is a draft, in a stack of three; see Stack.Refuse a FIFO artifact without waiting for a writer (
cba21d109)OpenedSnapshotDirectory::open_regular_fileopened it withopenat(O_RDONLY | O_CLOEXEC | O_NOFOLLOW), and a read-only open of a FIFO waits, in the kernel'swait_for_partner, until a writer opens the FIFO, so it never returned to the check. The path-basedopen_regular_file_impl, which the claimed scratch link andwrite_snapshot's memory backing file use, waited the same way.O_NONBLOCK, so that the open of a FIFO returns at once and the existing check refuses it like any other file that is not a regular file: "<artifact>is not a regular file". A newaccept_opened_regular_file, which both openers call, makes that check and then clearsO_NONBLOCKon a regular file's descriptor, since the handles outlive validation, as restore guards, as the descriptor that the VM worker maps as RAM, and as the scratch source, andO_NONBLOCKis not inert for every consumer of a file. It also lets openvmm: nonregular paired scratch is accepted before a resume generation can be consumed nvx#1157, which this issue blocks, check the type ofscratch.imgbefore the claim without a blocking open. Directory opens are unchanged, sinceO_DIRECTORYfails withENOTDIRbefore any wait. Windows is unchanged: NTFS holds no FIFOs, and the AF_UNIX sockets and WSL FIFOs that it can hold are reparse points, which the relative open opens as themselves, without waiting, and refuses.artifact_opens_refuse_a_writerless_fifo_without_waitingopens a FIFO without a writer with each artifact opener, andrestore_refuses_a_writerless_fifo_artifact_without_waitingreplaces each artifact of a paired-scratch resume snapshot with one and opens the snapshot and its paired scratch. Each open runs on another thread, through the shared helperrefusal_of_writerless_fifo, and must fail with the type error within 10 s; a regular file's handle must not be nonblocking. A waiting open gets a writer from anO_RDWRopen of the FIFO, so the tests fail rather than hang without the change. Both are Linux tests.O_NONBLOCKand refuses a FIFO for its type.Validation
At this head,
cba21d109, with Rust 1.97.1. It is the commit that NVX's phase-1 validation tested,904a33c32ond501e5066, rebased ontoac3dc27a7;git range-diffshows the patch identical.cargo nextest run --profile agent -p openvmm_helpers, 127 passed (the new tests are Linux tests);cargo clippy --all-targets -p openvmm_helpersandcargo doc --no-deps -p openvmm_helpers, without warnings.cargo check --all-targets,cargo clippy --all-targets, andcargo doc --no-depsforopenvmm_helpers, without warnings, andcargo nextest run --profile agent -p openvmm_helpers, 135 passed, both new tests among them.cargo xtask fmtat the head of the stack, which holds this commit's files as they are here, clean.At
904a33c32:openvmm_helperspassed, 123 tests on Windows, and 131 on Linux for glibc on NVX's KVM test VM and forx86_64-unknown-linux-muslon its MSHV test VM. Without theO_NONBLOCKflags, both new tests failed after their 10 s bound, on both. On the KVM VM, removing only the directory-relative flag failed both; only the path-based flag, the opener test; and only the clearing ofO_NONBLOCK, the opener test's flag assertion.6cb6ba87c), 8 of 8 waited inwait_for_partneruntil killed after 30 s, on NVX's KVM and MSHV test VMs; with the fix, built for glibc on KVM and for musl on MSHV, 8 of 8 exited with status 1 within 0.01 s with "<artifact>is not a regular file", and none claimed the instance checkpoint.manifest.bin, so WHP had no such wait.snapshot-tiersandscratch-snapshotscenarios passed with OpenVMM built from it on KVM, MSHV, and WHP.Stack
Three drafts, one for each of three microsoft/nvx issues, each based on the branch of the one before it:
feature/microvm/dev.NVX pins each one's head in its own draft, stacked the same way: microsoft/nvx#1423, microsoft/nvx#1424, and microsoft/nvx#1425, in this order, with microsoft/nvx#1427, part 2 of microsoft/nvx#544, which moves no pin, on top.
This one lands after #5003: once #5003 has landed, retarget this one to
feature/microvm/dev(gh pr edit 5004 --repo microsoft/openvmm --base feature/microvm/dev) and fast-forwardfeature/microvm/devto this head,cba21d109eeb99fbc573c79b2ed9f4754f734b25, since this repository's merge button squashes and would leave NVX's pin off the branch.