[Pedro Henrique Penna (ppenna)]

Makes the snapshot boundary's input gating leave each device open to the vCPUs that still run, for microsoft/nvx#1138. A guest's snapshot request could crash a multi-vCPU microVM, or lose the device accesses of its other vCPUs: the chipset device unit held a device's mutex closed while it changed the device's input gate, so a port or MMIO access that another vCPU made then went unanswered, a read returning all ones and a write lost, an interrupt from another device was dropped, and an EOI or a PCI configuration access panicked the VM worker with "lock should not be closed".

One commit on cba21d109, the head of #5004, which it stacks on. microsoft/nvx#1425 pins this head. It is a draft, at the top of a stack of three; see Stack.

Gate a device's input without closing it to the vCPUs (46bfe02ed)

• Cause. A snapshot boundary gates host input before it stops the vCPUs, and resumes a gate that failed before any of them stops, so the vCPUs run through both. The chipset device unit nevertheless changed each gate with the device's mutex closed, from the call to the end of the change's wait, as it changes the device's other states, which run only while the vCPUs are stopped. The window lasted the whole wait of a virtio transport's gate for its device task, and of a remote device's gate for its worker, and even a gate that completes at once, as most devices' gates do, closed the device for an instant, which a vCPU that signals EOIs in a loop met within one to five gates.
• Change. A gate change now has two parts. ChangeDeviceState::quiesce_input and resume_input make the synchronous part and return what remains as an InputGateWait, an owned, boxed future that does not borrow the device. The chipset device unit makes the call under the device's ordinary lock, as a vCPU's access does, releases the lock, and then awaits the wait, so a vCPU's access comes before the synchronous part or after it and is served either way. The virtio transports (MMIO and PCI) and the remote device proxy send their gate's request in the synchronous part, on the channel that carries the commands of the guest's accesses, so that their device takes the gate in the order of those accesses, and wait for its answer without the device. The erased chipset devices of chipset_device_resources forward both parts. The remote device worker is unchanged.
• Scope. Only the input gate changed. The other state changes still close the device, since they run only while the vCPUs are stopped.
• Alternatives. Holding the device's lock across the wait would stall the vCPUs that access the device for the whole wait and risk a deadlock with a device task that needs a vCPU's access to finish. Tolerating a closed device in the vCPUs' access paths would hide the dropped accesses rather than serve them.
• Tests. vp_accesses_during_an_input_gate_change_reach_the_device makes a vCPU's port write, EOI, and interrupt while a device's gate change waits, for a quiesce and for a resume, and requires them to reach the device after the change, without a panic. immediate_input_gates_serve_concurrent_vp_accesses takes and resumes the gate 1000 times while a vCPU writes a port and signals EOIs in a loop, and requires every access to be served. gate_changes_keep_the_order_of_guest_accesses_mmio and _pci, and remote_input_gate_changes_keep_the_order_of_requests, require a gate change to reach the device in the order of the guest's accesses around it, on both virtio transports and through the remote device proxy, and a remote gate's error to be reported.

A pre-existing, related defect that review found, a guest's virtio reset clearing a device's input gate while a boundary holds host input, is microsoft/nvx#1416, outside this change.

Validation

At this head, 46bfe02ed, with Rust 1.97.1. It is the commit that NVX's phase-1 validation tested, a91d4c7a4 on d6addf8c1, rebased onto cba21d109; git range-diff shows the patch identical, and the commits from d6addf8c1 to cba21d109 change none of its files.

• Windows: cargo clippy --all-targets for vmcore, chipset_device_resources, virtio, vmotherboard, and chipset_device_worker, and for the virtio devices virtiofs, virtio_net, and virtio_console, and cargo doc --no-deps for the eight, without warnings; cargo nextest run --profile agent for the eight and state_unit, 545 passed and 2 skipped.
• Linux (NVX's KVM test VM, Azure Linux 3.0, glibc): the same clippy, doc, and nextest commands, without warnings, 550 passed and 2 skipped.
• cargo xtask fmt, clean.

At a91d4c7a4:

• Without the fix, the first test failed, with the port write dropped ("device didn't respond"), the EOI panicking ("lock should not be closed"), and the interrupt lost, and the stress test failed in 10 of 10 runs; with it, 20 of 20 runs passed. Phase 1 also ran mutation checks of the virtio and proxy gates. The same package checks passed, 545 tests on Windows and 550 on Linux.

• End to end, a microVM guest requested 2000 snapshots, each of which gated host input and then failed its preflight, while its other vCPUs read a virtio-mmio register in a loop, 3 runs per cell. Reads that returned all ones:

  Backend (NVX test VM)	2 vCPUs, before	4 vCPUs, before	After
  KVM	586-786	2068-2358	0
  MSHV	271-360	898-1184	0
  WHP	145-166	246-358	0

  OpenVMM logged "device didn't respond" before, and nothing after. NVX's snapshot and SMP scenarios and its full default microVM suite, with 1, 2, and 4 vCPUs, passed with OpenVMM built from it on KVM, MSHV, and WHP.

Stack

Three drafts, one for each of three microsoft/nvx issues, each based on the branch of the one before it:

1. [DO NOT REVIEW] openvmm_helpers: compute memory-expansion ranges without overflow #5003, for openvmm: memory-expansion range math overflows near the top of the GPA space nvx#1364, on feature/microvm/dev.
2. [DO NOT REVIEW] openvmm_helpers: refuse a FIFO artifact without waiting for a writer #5004, for openvmm: a writerless FIFO snapshot artifact hangs restore before validation nvx#1154, on [DO NOT REVIEW] openvmm_helpers: compute memory-expansion ranges without overflow #5003's branch.
3. This pull request, for openvmm: snapshot input gating closes device mutexes while other vCPUs still access them nvx#1138, on [DO NOT REVIEW] openvmm_helpers: refuse a FIFO artifact without waiting for a writer #5004's branch.

NVX pins each one's head in its own draft, stacked the same way: microsoft/nvx#1423, microsoft/nvx#1424, and microsoft/nvx#1425, in this order, with microsoft/nvx#1427, part 2 of microsoft/nvx#544, which moves no pin, on top.

This one lands last, after #5004: once #5004 has landed, retarget this one to feature/microvm/dev (gh pr edit 5005 --repo microsoft/openvmm --base feature/microvm/dev) and fast-forward feature/microvm/dev to this head, 46bfe02ed65697e858f66ce55eb60831b3ac3461, since this repository's merge button squashes and would leave NVX's pin off the branch.

[github-actions]

⚠️ Unsafe Code Detected

This PR modifies files containing unsafe Rust code. Extra scrutiny is required during review.

For more on why we check whole files, instead of just diffs, check out the Rustonomicon

[github-actions]

3 Petri tests failed (0 unstable)