[Pedro Henrique Penna (ppenna)]

Fixes #1416

What and why

A guest that reset a virtio device while OpenVMM held host input gated let
host input past the gate. The VM worker gates host input at a snapshot
boundary before it stops the vCPUs, and during a gated restore until the
guest acknowledges it, and the vCPUs run in both windows. When the guest
wrote 0 to a device's status there, the transport reset the device, and each
device's reset cleared its own gate: virtio-net cleared the latch under which
its endpoint queues start, the consoles their input_gated flag, and
virtio-fs reopened its request admission. Nothing gated the device again, so
once the guest set DRIVER_OK again, the device delivered host packets,
console input, the control console's host records among it, and virtio-fs
replies that the gate held back, while the worker still recorded input as
gated. A capture whose virtio-fs device the guest reset also failed its save
(virtio-fs save was not preceded by input quiesce) and rolled back.

Root cause: the VM worker takes and releases the input gate (quiesce_input,
resume_input), but each device kept it in state that its
VirtioDevice::reset returned to its initial value, so a reset, which the
guest controls, opened it.

microsoft/openvmm#5022, virtio: keep a held input gate across a device reset, makes each device's reset keep its gate, so that only resume_input
releases it:

• virtio-net keeps the latch, from which the coordinator that the next start
  of the queues creates gates its new endpoint queues, as it already did for a
  gate taken before the queues first start;
• the console keeps input_gated, so its direct worker reads no host input
  and its broker neither reads the host's records nor times its
  authentication;
• virtio-fs leaves admission closed, so the guest's requests stay on their
  virtqueues.

VirtioDevice::reset and quiesce_input, ChangeDeviceState::quiesce_input
and reset, and OpenVMM's snapshot guide state the rule.

Commits:

• openvmm: pin input gates that a guest's device reset keeps moves the
  gitlink, and nothing else, from cd5cd3814, the head of
  feature/microvm/dev, to 61cdd63978ea58ce045d159208ed4ca22f30cbb8, the
  head of [DO NOT REVIEW] virtio: keep a held input gate across a device reset openvmm#5022, one commit on it; its message carries the
  host evidence below. NVX needs no coordinated change: the control-session
  vectors, source excerpts, contract versions, and CPU profiles that nvx-rs
  copies from OpenVMM match the revision's.
• doc: map the input gates that resets keep in the OpenVMM roadmap maps
  position 247, 116 commits past the mapped tip, as a PR13 fixup: it folds
  into PR13's positions 043 and 048, its virtio-fs change into PR18's
  position 088, and its other parts into the positions that add what they
  change, as the fold sentences and the commit mapping say. Replaying it on
  upstream main conflicts in 13 of its 15 files.

Decisions:

• Where the gate survives a reset. In each device, by keeping its own gate
  state across its reset, rather than in a latch in the transport's device
  task that quiesces the device again after the guest's reset, which openvmm: a guest's virtio device reset lets host input past a held input gate #1416
  also proposed. The gate state already lives in the devices, and their reset
  is what cleared it; a latch in the task would give the gate two owners, and
  a quiesce after a reset could fail and need handling of its own, while the
  task and the transports stay what [DO NOT REVIEW] vmotherboard: gate a device's input without closing it to the vCPUs openvmm#5005 made them,
  forwarders of the gate in the order of the guest's accesses.
• A reset of the VM keeps the gate too. The VM worker records the gate
  (snapshot_restore.input_gated, the snapshot boundary), and its reset
  changes neither record: the worker refuses a VM reset while a boundary is
  active, and a gated restore's resume gates input again before the units
  start. Releasing the gate on a VM reset would recreate the mismatch between
  the device and the worker's record that openvmm: a guest's virtio device reset lets host input past a held input gate #1416 describes, so only
  resume_input releases it. The alternative, telling a VM reset from the
  guest's in the device task and opening the gate on the former, has no
  caller that needs it.
• One OpenVMM commit, recorded as a PR13 fixup. The fix, its
  documentation, and its tests are one commit, as position 238 was for openvmm: snapshot input gating closes device mutexes while other vCPUs still access them #1138.
  Since it fixes what PR13's positions 043 and 048 and PR18's position 088
  add, the roadmap records it as a fixup, as it records positions 241-243 as
  PR09 fixups, with the parts that fold elsewhere named.
• The end-to-end reproduction uses virtio-net. OpenVMM's tests show the
  consoles' and virtio-fs's cases, through the devices and the virtio-mmio
  transport.

Landing order

1. Land [DO NOT REVIEW] virtio: keep a held input gate across a device reset openvmm#5022 by fast-forwarding feature/microvm/dev to
   61cdd63978ea58ce045d159208ed4ca22f30cbb8, since OpenVMM's merge button
   squashes and would leave the pin off the branch.
2. Merge this pull request through the merge queue.

Validation

OpenVMM, Rust 1.97.1, at 61cdd6397, which is 7870c09e2, the commit that
the first validation below tested on 46bfe02ed, rebased onto cd5cd3814:
git range-diff shows the patch identical.

• The seven new tests fail without the fix and pass with it: the console's
  direct worker and broker, after a reset made as the transport makes the
  guest's, receive no host input until input resumes (without the fix, the
  input and the host's attachment are read); a guest that writes 0 to the
  status of the running console through the virtio-mmio transport and starts
  it again receives none (without the fix, it is delivered); virtio-net
  quiesces the endpoint queues of the next start (without the fix, it never
  does); virtio-fs leaves a FUSE_INIT on its virtqueue (without the fix, it
  is answered); and after a status write of 0 to an inactive virtio-fs
  device, or a reset of the VM, the save succeeds (without the fix, it fails
  with virtio-fs save was not preceded by input quiesce).
• Windows: cargo clippy --all-targets and cargo doc --no-deps for
  vmcore, virtio, virtio_net, virtio_console, and virtiofs, without
  warnings; cargo nextest run --profile agent for the five, 515 passed and
  2 skipped; cargo xtask fmt, clean.
• Linux (nvx-kvm-test, Azure Linux 3.0, glibc): the same clippy, doc, and
  nextest commands, without warnings, 520 passed and 2 skipped.

End to end, the platform tier of the snapshot-tiers scenario, whose restore
gates host input until the guest acknowledges it, with a runtime post-restore
hook, which runs before the acknowledgement, that resets the guest's
virtio-net device by unbinding and binding its driver, configures it, and
pings the gateway, whose reply is host input. "Before" is the OpenVMM of the
release that CI built from the named revision; "after" is OpenVMM built on
each host from the named revision with build-openvmm (GNU for KVM, musl for
MSHV, MSVC for WHP). Runs whose reply arrived before the acknowledgement:

Profile	Backend	Host type	Before, 46bfe02ed	After, 7870c09e2	Before, a8a0e3517	After, 61cdd6397
nvx-kvm-test	kvm	virtual machine	4 of 4	0 of 3	2 of 2	0 of 3
nvx-mshv-test	mshv	virtual machine	4 of 4	0 of 3	2 of 2	0 of 3
nvx-whp-test	whp	virtual machine	4 of 4	0 of 3	2 of 2	0 of 3

46bfe02ed was the base of the first validation, and a8a0e3517, the
OpenVMM of release v0.1.0-dev.705412fddc32, differs from this pin's base,
cd5cd3814, only in tests. Every run passed the scenario's own checks, among
them that the input that the host sends while input is gated reaches the
guest only after the acknowledgement. The binaries that the "after" runs on
61cdd6397 used had the SHA-256 digests 9498d617… (KVM), b32de940…
(MSHV), and FB907106… (WHP), and those of the "before" runs on
a8a0e3517, b4aaea17…, 1a46efb6…, and 088820BC…. With OpenVMM at
61cdd6397, as with 7870c09e2, test-microvm --processors 1 2 4 passed
guest-boot, lifecycle, virtio-net, snapshot-core, snapshot-rollback,
snapshot-tiers, network-snapshot, console-snapshot,
filesystem-snapshot, platform-filesystem, platform-network-policy,
untiered-network-policy, run-console-restore, and smp-snapshot on all
three hosts.

NVX, at this head:

• python scripts/nvx.py verify, passed.
• python scripts/local_gates.py --base origin/dev --only test-changes markdown-links test-layout openvmm-vectors openvmm-cpu-profiles --run,
  passed, with no change to an existing test and the 4 vector and 1
  CPU-profile comparisons passing against 61cdd6397.
• python scripts/openvmm_pin.py check-roadmap --worktree and
  check-branch --topic, passed.
• nvx-cli, run before the rebase on scripts identical to dev's, which
  this pull request leaves unchanged: of its 2188 tests, 88 were skipped and
  7 subtests of two tests, the host-loopback and private-ingress script tests
  of test_microvm_tests, errored when their scripts timed out after 10
  seconds under Git's sh.exe on a loaded Windows workstation, again when
  rerun alone; the others passed, and so did test_adversarial and the help
  commands. CI runs the gate on Linux and Windows.
• openvmm-ci is left to CI, which waits for OpenVMM's CI of the pin.

[Copilot]

🔵 Needs a closer look

The pin changes snapshot input isolation, but OpenVMM #5022 remains unmerged and exact-head build/test checks are skipped, requiring maintainer verification of upstream review, required CI, and landing before NVX merges.

No open issues 🎉

What changed in this PR

Updates the OpenVMM upstream roadmap for the pin that preserves host-input gates across guest device resets, addressing #1416’s snapshot and gated-restore issue.

Changes:

• Records revision 61cdd6397 at position 247 and updates the chain counts.
• Classifies position 247 as a PR13 fixup and identifies where its device, documentation, and test changes should fold upstream.
• Documents the fix’s dependencies and upstream replay conflicts.

File	Description
doc/​openvmm-upstream-roadmap.md	Updates the pin, commit mapping, and upstream folding plan for reset-safe input gates.

_{Review effort: Balanced}

[Pedro Henrique Penna (ppenna)]

No change needed here. As Copilot's overview says, microsoft/openvmm#5022 must land first, and this pull request already orders it so: its "Landing order" section has the maintainer fast-forward feature/microvm/dev to the exact pin, 61cdd63978ea58ce045d159208ed4ca22f30cbb8, before this pull request merges. The OpenVMM checks that show as skipped belong to OpenVMM's optional release workflow, which skips pull requests. OpenVMM's own CI of the pin, run 42864231169, is in progress, and this pull request's openvmm-ci job waits for it and requires its unit-test and VMM-test jobs before Required status check can pass.