Skip to content

[DO NOT REVIEW] virtio: keep a held input gate across a device reset - #5022

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
feature/microvm/devfrom
copilot/nvx-issue-1416-virtio-reset-input-gate
Nov 10, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
feature/microvm/devfrom
copilot/nvx-issue-1416-virtio-reset-input-gate

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Keeps the host-input gate that the VM worker holds across a guest's reset of a virtio device, for microsoft/nvx#1416. A guest that reset a virtio device while OpenVMM held host input gated, between a snapshot boundary's input gate and the vCPU stop, or during a gated restore before the guest acknowledged it, cleared the device's gate: once the guest set DRIVER_OK again, the device delivered host packets, console input, and virtio-fs replies that the gate held back, while the VM worker still recorded input as gated, and a capture whose virtio-fs device the guest reset failed its save and rolled back.

One commit on cd5cd3814, the head of feature/microvm/dev. microsoft/nvx#1452 pins this head.

Keep a held input gate across a device reset (61cdd6397)

  • Cause. Each device kept the gate in state that its VirtioDevice::reset returned to its initial value: virtio-net's latch under which its endpoint queues start, the console's input_gated, and virtio-fs's request admission, which its reset reopened. The transport resets the device when the guest writes 0 to its status (Disable for a DRIVER_OK device, Reset for another), when DRIVER_OK fails, and on ChangeDeviceState::reset, so a reset, which the guest controls, opened the gate.
  • Change. A reset now keeps the gate, and only resume_input releases it. virtio-net keeps the latch, from which the coordinator that the next start of the queues creates gates its new endpoint queues, as it already does for a gate taken before the queues first start; the console keeps input_gated, so its direct worker reads no host input and its broker neither reads the host's records nor times its authentication; virtio-fs leaves admission closed, so the guest's requests stay on their virtqueues. VirtioDevice::reset and quiesce_input, ChangeDeviceState::quiesce_input and reset, and the snapshot guide state the rule.
  • A reset of the VM keeps the gate too. The VM worker records the gate, and its reset changes neither that record nor, when it resumes a gated restore, the gate, which it takes again before the units start; the worker refuses a VM reset while a snapshot boundary is active. Releasing the gate on a VM reset would recreate the mismatch between the device and the worker's record.
  • Alternatives. A latch in the transport's device task that quiesces the device again after the guest's reset would give the gate a second owner, and its quiesce after a reset could fail; keeping the gate where the devices already keep it needs neither.
  • Tests. On each device, while input is gated, a reset made as the transport makes the guest's and a new start of the queues: the console's direct worker and its broker receive no host input, the host's attachment included, until input resumes; virtio-net quiesces the endpoint queues that the new start creates; virtio-fs leaves a FUSE_INIT on its virtqueue until input resumes. Through the virtio-mmio transport: a guest that writes 0 to the status of the running console and starts it again receives no host input until input resumes; after a status write of 0 to an inactive virtio-fs device, or a reset of the VM, the save succeeds. Without the fix, all seven fail: host input reaches the guest, the new endpoint queue is never quiesced, the FUSE_INIT is answered, and the save fails with "virtio-fs save was not preceded by input quiesce".

Validation

At this head, 61cdd6397, with Rust 1.97.1. It is the commit that NVX's phase-1 validation tested, 7870c09e2 on 46bfe02ed, rebased onto cd5cd3814: git range-diff shows the patch identical, and the commits from 46bfe02ed to cd5cd3814 change none of its files but the snapshot guide, in another section.

  • Windows: cargo clippy --all-targets and cargo doc --no-deps for vmcore, virtio, virtio_net, virtio_console, and virtiofs, without warnings; cargo nextest run --profile agent for the five, 515 passed and 2 skipped; cargo xtask fmt, clean.
  • Linux (NVX's KVM test VM, Azure Linux 3.0, glibc): the same clippy, doc, and nextest commands, without warnings, 520 passed and 2 skipped.
  • End to end, on NVX's KVM, MSHV, and WHP test VMs: a platform-tier restore whose guest, before it acknowledges the restore, resets its virtio-net device by unbinding and binding its driver and pings its gateway. With the OpenVMM that CI built from 46bfe02ed and from a8a0e3517, which differs from this base only in tests, the reply, which is host input, arrived before the acknowledgement in every run, 4 and 2 runs on each backend; with OpenVMM built from 7870c09e2 and from this head, in none of 3 runs each, and NVX's affected microVM scenarios, the snapshot, restore, network, console, and filesystem ones among them, passed on all three. See openvmm: keep a held input gate across a guest's device reset nvx#1452 for the table.

A snapshot boundary gates host input before it stops the vCPUs, and a
gated microVM restore keeps host input gated while the vCPUs run the
guest's repair, until the guest acknowledges the restore. While the
gate was held, the guest could reset a virtio device by writing 0 to
its device status, and the device's reset cleared its own gate:
virtio-net cleared the latch under which its endpoint queues start,
the console its input_gated flag, and virtio-fs reopened its request
admission. Nothing gated the device again, so once the guest set
DRIVER_OK again, the device delivered host packets, console input, or
virtio-fs replies that the gate held back, while the VM worker still
recorded host input as gated (microsoft/nvx#1416). In a capture, the
reopened virtio-fs admission also failed the save, which rolled the
capture back.

A reset now keeps the gate, and only resume_input releases it. The
gate is the VM worker's: the worker takes it, and releases it when it
releases the snapshot boundary or the guest acknowledges the restore,
and no reset changes the worker's record of it. So a reset of the VM
keeps the gate too, as does the reset that follows a failed DRIVER_OK.
Each device's reset keeps its gate. virtio-net keeps the latch, from
which the coordinator that the next start of the queues creates gates
its new endpoint queues, as it already does for a gate taken before
the queues first start. The console keeps input_gated, so its worker
reads no host input and its broker does not time the host's
authentication. virtio-fs leaves admission closed, so the guest's
requests stay on their virtqueues. VirtioDevice::reset and
quiesce_input, ChangeDeviceState::quiesce_input and reset, and the
snapshot guide state it.

Tests: on each device, while input is gated, a reset made as the
transport makes the guest's, and a new start of the queues: the
console's direct worker and its broker read no host input, the host's
attachment included, until input resumes; virtio-net quiesces the
endpoint queues that the new start creates; and virtio-fs leaves a
FUSE_INIT on its virtqueue until input resumes. Through the
virtio-mmio transport: a guest that writes 0 to the status of the
running console and starts it again gets no host input until input
resumes, and after a status write of 0 to an inactive virtio-fs
device, or after a reset of the VM, while input is gated, the save
that follows succeeds. Without the fix, all seven fail: host input
reaches the guest, the new endpoint queue is never quiesced, the
FUSE_INIT is answered, and the save fails with "virtio-fs save was not
preceded by input quiesce".

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added Guide release-nvx Targets a branch under feature/microvm/. unsafe Related to unsafe code labels Nov 10, 2026
@github-actions

Copy link
Copy Markdown

⚠️ Unsafe Code Detected

This PR modifies files containing unsafe Rust code. Extra scrutiny is required during review.

For more on why we check whole files, instead of just diffs, check out the Rustonomicon

@ppenna

Copy link
Copy Markdown
Contributor Author

This pull request adds no unsafe code. The check matched the phrase "cancellation-unsafe issued I/O" in a doc comment in vm/vmcore/src/device_state.rs, which #4863 added and which this pull request leaves as is; the pull request only edits other doc comments in that file. None of the Rust files that it changes contains an unsafe block, function, or attribute.

@ppenna
Pedro Henrique Penna (ppenna) merged commit 61cdd63 into feature/microvm/dev Nov 10, 2026
73 of 74 checks passed
syeedali pushed a commit to syeedali/nvx that referenced this pull request Nov 11, 2026
A guest that reset a virtio device while OpenVMM held host input gated
let host input past the gate (microsoft#1416): between a snapshot boundary's
input gate and the vCPU stop, or during a gated restore before the
guest acknowledged it, the device's reset cleared its gate, and once
the guest set DRIVER_OK again, the device delivered the host input
that the gate held back, virtio-net's packets, a console's input, and
virtio-fs's replies, while the VM worker still recorded input as
gated. A capture whose virtio-fs device the guest reset failed its
save and rolled back.

Pin 61cdd63978ea58ce045d159208ed4ca22f30cbb8, the head of
microsoft/openvmm#5022, one commit on cd5cd3814, the head of
feature/microvm/dev:

- 61cdd6397 virtio: keep a held input gate across a device reset

The revision makes each device's reset keep its input gate, so that
only resuming input releases it, also for a reset of the VM: the VM
worker holds the gate and releases it when it releases the boundary
or the guest acknowledges the restore, and no reset changes its
record.

Evidence: the platform tier of the snapshot-tiers scenario, whose
restore gates host input until the guest acknowledges it, with a
runtime post-restore hook, which runs before the acknowledgement, that
resets the guest's virtio-net device by unbinding and binding its
driver, configures it, and pings the gateway, whose reply is host
input. The reply arrived before the acknowledgement in every run on
nvx-kvm-test (KVM), nvx-mshv-test (MSHV), and nvx-whp-test (WHP), all
virtual machines, with the OpenVMM of the releases that CI built from
46bfe02ed, 4 of 4 runs each, and from a8a0e3517, which differs from
cd5cd3814 only in tests, 2 of 2 runs each. With OpenVMM built on each
host from 61cdd6397, it arrived in none of 3 runs each, every run
passed the scenario, whose host input reached the guest only after
the acknowledgement, and the affected microVM scenarios passed. At the
pinned revision, the OpenVMM tests of the changed packages pass, 515
on Windows and 520 on Linux; without the fix, the 7 new ones fail.

NVX needs no coordinated change: the control-session vectors, the
source excerpts, the contract versions, and the CPU profiles that
nvx-rs copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syeedali pushed a commit to syeedali/nvx that referenced this pull request Nov 11, 2026
Maps NVX's new pin, 61cdd6397, the head of microsoft/openvmm#5022, at
position 247, after position 246, cd5cd3814, the head of
feature/microvm/dev, which microsoft/openvmm#5016 merged. The pin is
116 commits past the mapped tip.

Position 247 is a PR13 fixup. It changes the input gates that PR13's
positions 043 and 048 and PR18's position 088 give the console,
virtio-net, and virtio-fs with their saved state: each device's reset
cleared its gate, and now keeps it. It states that rule on the gate's
methods of VirtioDevice and ChangeDeviceState, which PR13's position
040 and PR09's position 018 add, extends the snapshot guide's
paragraph on the post-restore gate that PR20's position 095 adds, and
adds tests beside those of the devices, which extend the broker's test
harness of PR23's position 113 and the gated-queue tests of position
172. It folds into positions 043 and 048, its virtio-fs change into
PR18's position 088, and its other parts into the positions that add
what they change, as the fold sentences after the DAG and the commit
mapping say; the Scope, the DAG, the dependency graph, the PR summary,
and the commit mapping record it with PR13. Replaying it on upstream
main conflicts in 13 of its 15 files, all but a new module of console
tests and the development dependency that virtio-fs takes for its
transport tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syeedali pushed a commit to syeedali/nvx that referenced this pull request Nov 11, 2026
Maps NVX's new pin, 58d3c3b1d, the head of microsoft/openvmm#5026, at
position 248, after position 247, 61cdd6397, the head of
feature/microvm/dev, which microsoft/openvmm#5022 merged. The pin is
117 commits past the mapped tip.

Position 248 is a PR18 fixup. It checks the prefix of the network
identity that PR18's position 083 adds, in validate_machine_config,
which PR15's position 062 adds, before the command-line check of
position 083 derives the identity's netmask, and gives that position's
parser the same range. It folds into position 083, and its test's
virtio-fs device into position 202, which reserves the virtio-fs slot
on every command line, as the fold sentence after the DAG and the
commit mapping say; the Scope, the DAG, the dependency graph, the PR
summary, and the commit mapping record it with PR18. Replaying it on
upstream main conflicts in its only file: upstream main lacks the
microVM module of PR15's position 055.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Guide release-nvx Targets a branch under feature/microvm/. unsafe Related to unsafe code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant