Repository navigation
nvx-rs: activate a clone from its restore packet's record - #1427
Conversation
A platform clone took its activation record in the adapter's Activate, which the host sent once it attached. The clone's launch now passes the record to OpenVMM with --restore-activation-record, which seals it in the restore packet before the first restored vCPU runs, and the guest activates from the packet as soon as it runs, without waiting for the host (#544). - nvx::guest::clone: the repair hands the packet's record over, Repaired::activation_record, and Activation takes it from there: accept() decodes and checks it, and the activation is ready once the record applied and the host asked for Ready, the first request of each of its sessions. OpenVMM's broker refuses the guest's records while no host is attached, which the guest cannot tell, so the guest cannot send Ready unasked. A record that applies before the host asks leaves the activation in the new Applied state, its fence closed and its deadline holding, so that a clone whose host never asks still ends, and any other request before Ready still fails it. A reset changes nothing of the record, so it only abandons the requests for Ready of the earlier session: the Transaction, Changed, and Reset causes go, and Missing comes for a packet without a record. - nvx::host::template: ClonePlan::launch passes the record that it writes to the sandbox with --restore-activation-record, and the restore sequence asks for Ready, CloneControl::ready in place of activate, so that a session that replaces a lost one resends nothing. A reattach asks for Ready too, and checks it against that record. - nvx::activation::CONTRACT_VERSION, 2, which template keys hold as their activation contract, versions how a clone receives its record, so that no build restores a template whose guest awaits an Activate. The record's schema, VERSION, stays 1. - The runtime adapter's Await, kind 28, replaces Activate, kind 27, which no message takes, and its worker applies the record at the restore's reset, once it wrote its acknowledgement, so that the host's attachment overlaps the application. Its activation probe also reports when the application completed. Tests: the guest activation's state machine, applied before or after the host's request, its deadline, resets, and every failure class; the launch's arguments and the restore sequence with a fake VMM that keeps the record that the launch supplied, retries that resend nothing, and the reattach; the template key's activation contract; the adapter's framing, client, and worker against a fake root. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The guide's restore packet (5.5.5) says that NVX supplies an activation record at every launch of a platform clone, whose guest activates from it as soon as it runs, without waiting for the host, which only asks for the clone's Ready, and that a template key's activation contract refuses a template whose guest takes its record otherwise (#544). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This pull request changes existing testsIt modifies or removes 128 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests. Test code... and 77 more, which the job's summary lists. Test data
Relocated unchanged: 0; added: 13. Compared the merge base |
The clone order tests that ask for Ready took new names when Await replaced Activate, but the suite's scenarios and the README's stage scenarios still named the old ones, so test-runtime-adapter refused to run on every backend: host_session had no tests by those names, and two tests that no scenario named. Name the new tests, in the order of their names, and say that the order sends Await first and that the next session's Await gets the same Ready. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The application timestamp is recorded after buffered messages are drained and state is reset, misordering buffered Await events and inflating the reported application duration.
1 open issue
What changed in this PR
Implements part 2 of #544 by moving clone activation-record delivery from the control session into the restore packet. Guests can apply their configuration without waiting for host attachment; the host then requests and validates Ready.
Changes:
- Passes activation records at launch and replaces Activate with Await for initial attachment and reattachment.
- Updates guest activation, readiness fencing, and template compatibility to the new delivery contract.
- Revises protocol tests, activation timing probes, documentation, and public API snapshots.
| File | Description |
|---|---|
tests/runtime-adapter/tests/vm_clone.rs |
Updates clone scenarios and activation timeline assertions. |
tests/runtime-adapter/tests/host_session.rs |
Tests Await ordering, replies, and replacement sessions. |
tests/runtime-adapter/src/worker/capture.rs |
Documents packet-driven activation after restore. |
tests/runtime-adapter/src/worker/activation/tests.rs |
Updates failure classification and probe tests. |
tests/runtime-adapter/src/worker/activation.rs |
Applies records at reset and handles Await readiness. |
tests/runtime-adapter/src/worker.rs |
Integrates packet-driven activation into the worker loop. |
tests/runtime-adapter/src/root.rs |
Documents Ready as the response to Await. |
tests/runtime-adapter/src/lib.rs |
Updates the adapter’s activation overview. |
tests/runtime-adapter/src/host.rs |
Replaces record transmission with Await requests. |
tests/runtime-adapter/src/hook.rs |
Updates reattachment hook documentation. |
tests/runtime-adapter/src/frame/tests.rs |
Tests Await framing and rejection of retired Activate. |
tests/runtime-adapter/src/frame.rs |
Introduces Await kind 28 and retires Activate kind 27. |
tests/runtime-adapter/src/bin/runtime-adapter-host.rs |
Updates host reattachment documentation. |
tests/runtime-adapter/scenarios.txt |
Registers the renamed host-session scenarios. |
nvx-rs/src/protocol/status.rs |
Updates activation failure descriptions. |
nvx-rs/src/host/template/mod.rs |
Documents launch-time delivery and Ready requests. |
nvx-rs/src/host/template/key/tests.rs |
Tests rejection of the earlier activation contract. |
nvx-rs/src/host/template/key.rs |
Uses the delivery contract in template compatibility keys. |
nvx-rs/src/host/template/clone/tests/launch.rs |
Tests packet delivery and Ready-based reattachment. |
nvx-rs/src/host/template/clone/restore.rs |
Requests Ready without reading or resending the record. |
nvx-rs/src/host/template/clone/records.rs |
Updates progress-log documentation. |
nvx-rs/src/host/template/clone/reattach.rs |
Validates Ready against the persisted activation record. |
nvx-rs/src/host/template/clone/launch.rs |
Passes --restore-activation-record to OpenVMM. |
nvx-rs/src/host/template/clone/failure.rs |
Clarifies failure and progress terminology. |
nvx-rs/src/guest/time/api.rs |
Documents the record’s handoff through repair. |
nvx-rs/src/guest/mod.rs |
Updates the guest activation overview. |
nvx-rs/src/guest/clone/repair.rs |
Preserves packet record bytes in Repaired. |
nvx-rs/src/guest/clone/activation/apply/tests.rs |
Adapts application tests to packet-supplied records. |
nvx-rs/src/guest/clone/activation/apply.rs |
Documents one-time application and repeated Ready requests. |
nvx-rs/src/guest/clone.rs |
Documents activation states, deadlines, and session behavior. |
nvx-rs/src/contract.rs |
Documents activation delivery as a compatibility contract. |
nvx-rs/src/activation/mod.rs |
Adds contract version 2 while retaining record schema version 1. |
nvx-rs/README.md |
Explains the revised host and guest activation flow. |
nvx-rs/public-api/windows.txt |
Records the contract constant and host API changes. |
nvx-rs/public-api/linux.txt |
Records the updated host and guest APIs. |
doc/design/5.5-restore-protocol.md |
Specifies packet-based activation and host-requested Ready. |
Review effort: Balanced
The activation probe's `applied` took the instant at which the worker completed the activation, once it took what the session received while the record applied, not when the application completed. An `Await` that came during the application therefore came before `applied` in the probe's report, and the application's time held the worker's input. Take the instant when the application returns, before the worker takes more input, and keep `ready` at the activation's completion. A new test holds the application until an `Await` came, and checks that the clone reports its reset, its application, the host's first message, and its Ready in that order; it fails without the change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Part of #544
What and why
Part 2 of #544. Part 1, #1419 with microsoft/openvmm#4996, made the restore
packet carry a platform clone's activation record, from OpenVMM's
--restore-activation-record, sealed before the first restored vCPU runs.The libraries still delivered the record in the runtime adapter's Activate,
which the host sent once it attached to the clone. This pull request makes
every clone launch pass its record in the packet, and the guest activate from
it as soon as it runs, without waiting for the host:
nvx::guest::clone: the repair hands the packet's record over,Repaired::activation_record, andActivationtakes it from there:accept()decodes and checks it, and the activation is ready once therecord applied and the host asked for Ready, the first request of each of
its sessions. A record that applies before the host asks leaves the
activation in a new
Appliedstate, its fence closed and its deadlineholding, so that a clone whose host never asks still ends, and any other
request before Ready still fails it. A reset changes nothing of the record,
so it only abandons the earlier session's requests for Ready: the
Transaction,Changed, andResetfailure causes go, andMissingcomesfor a packet without a record.
nvx::host::template:ClonePlan::launchpasses the record that it writesto the sandbox with
--restore-activation-record, and the restore sequenceasks for Ready,
CloneControl::readyin place ofactivate, so that asession that replaces a lost one resends nothing. A reattach asks for Ready
too, and checks it against that record.
nvx::activation::CONTRACT_VERSION, 2, which template keys now hold astheir activation contract, versions how a clone receives its record, so
that no build restores a template whose guest awaits an Activate. The
record's schema,
VERSION, stays 1.message takes any more, and its worker applies the record at the restore's
reset, once it wrote its acknowledgement, so that the host's attachment
overlaps the application. Its activation probe also reports when the
application completed.
says that NVX supplies a record at every launch of a platform clone, whose
guest activates from it without waiting for the host, which only asks for
the clone's Ready, and that a template key's activation contract refuses a
template whose guest takes its record otherwise. The nvx-rs and runtime
adapter READMEs and the public API snapshots follow the code.
Departure from the plan in #544: it had the guest send Ready unprompted. It
cannot, since OpenVMM's broker refuses the guest's records while no host is
attached, and the guest cannot tell whether one is, so the host asks for Ready
instead. The guest still activates from the packet as soon as it runs; only
its Ready waits for the host's request.
Of part 2's tasks in #544, this does the first four: the host passes the
record at launch; the guest activates from the packet, answering Ready when
the host asks; Activate leaves
nvx::guest::clone,nvx::host::template::clone,and the runtime adapter; and template keys refuse templates whose guests
predate the flag. Left open: measuring restore-to-Ready against the Activate
path, source-matched and interleaved, on KVM, MSHV, and WHP.
Commits:
nvx-rs: activate a clone from its restore packet's record, the change andits tests;
doc: say that clones activate from the restore packet's record, the designguide;
runtime-adapter: name the renamed clone order scenarios in the suite, whichnames the two
host_sessiontests that the first commit renamed in thesuite's
scenarios.txtand in the README's stage scenarios, which stillnamed the old ones, so that
test-runtime-adapterrefused to run;runtime-adapter: report a clone's application when it completes, afterreview: the activation probe's
appliedtook the instant at which theworker completed the activation, once it took the input that came while the
record applied, so an
Awaitthat came meanwhile came before it; it nowtakes the instant at which the application returns, and a new test checks
the order.
This layer changes no OpenVMM code and moves no pin: it runs on layer 3's pin,
46bfe02ed, and part 1's--restore-activation-recordis onfeature/microvm/devsinced501e5066.Stack
This is the top, layer 4, of a stack of four pull requests, one for each of
four issues, each based on the branch of the one below it; the OpenVMM pull
requests of layers 1-3 stacked the same way. Layers 1-3 have landed, so this
pull request now targets
dev.dev)feature/microvm/dev), landedac3dc27a7, position 236dev)feature/microvm/dev), landedcba21d109, position 237dev)feature/microvm/dev), landed46bfe02ed, position 238dev)46bfe02ed, layer 3'sLanding order, bottom first, one layer at a time; each OpenVMM pull request
lands before its NVX pull request merges:
feature/microvm/devtoac3dc27a738808d38e1148d3caefc5e89111ee32, since OpenVMM's merge buttonsquashes and would leave the pin off the branch. Then merge openvmm: pin overflow-free memory-expansion ranges #1423
through the merge queue; GitHub deletes its branch and retargets openvmm: pin refusing a FIFO snapshot artifact without waiting #1424
to
dev.feature/microvm/dev(
gh pr edit 5004 --repo microsoft/openvmm --base feature/microvm/dev),fast-forward
feature/microvm/devtocba21d109eeb99fbc573c79b2ed9f4754f734b25, and merge openvmm: pin refusing a FIFO snapshot artifact without waiting #1424.feature/microvm/devto46bfe02ed65697e858f66ce55eb60831b3ac3461, andmerge openvmm: pin input gating that keeps devices open to the vCPUs #1425; GitHub retargets this pull request to
dev.Validation
The branch's developer stopped before validating it: no gate had run, and no
host check. This restack folded its last fixup, which made the guest binary's
malformed-message test send kind 29, now that kind 28 is Await, and ran the
gates that the change selects, with no further change:
test_host_loopback_allow_script_checks_publication_and_trafficandtest_private_network_ingress_script_checks_policy_and_traffic, whose scripts time out in Git for Windows'shon this workstation and pass on Linux and in CI; this change touches no script.test_adversarial(84 tests, 11 skipped) and thenvx.pyhelp commands passThe guest's activation compiles only on Linux, so only the Linux gates build
and test it. The gates ran with
python scripts/local_gates.py --base <base> --run, on the change rebased onto #1422's head,d48786283, whose tree isthat of its merge into
dev,dca5af173; this head differs from what theytested only in the lower layers' gitlink and documents, which none of these
gates reads, and
git range-diffshows its two commits unchanged. On Linux,the gates ran in a scratch clone on nvx-kvm-test;
nvx-rs-macosprinted 15warnings in test files that this change does not touch, since that step
denies none.
python scripts/rebase_check.py verifypassed after bothrebases. At this head,
python scripts/nvx.py verify, test-changes,markdown-links, test-layout, and the nvx-rs vector and CPU-profile comparisons
against layer 3's pin,
46bfe02ed, pass.Then, on hosts, the runtime adapter's suite,
python3 scripts/nvx.py test-runtime-adapter --backend BACKEND --guest-agent ..., as CI's runtimeadapter steps run it, in worktrees of this layer's second commit with OpenVMM
built at
46bfe02edon each host withbuild-openvmm(GNU for kvm, musl formshv, MSVC for whp), the guest agent built at that commit on Linux, and the
kernel, Alpine initramfs, and Ubuntu distro layer of #1424's CI run, whose
initramfs runs the adapter's guest agent in place of NVX's agents. The suite
first refused to run: its
scenarios.txtstill named the twohost_sessiontests that the first commit renamed. With the third commit:
test-runtime-adapter, whosevm_cloneandvm_templatetargets restore clones that activate from their packets' recordstest-changes,markdown-links, andtest_nvx_tools.RuntimeAdapterSuiteTestsat the third commitAt the fourth commit,
a4e7f20d1, this head:check-nvx-rsfor its guest and host configurations:cargo fmt --check,clippy -D warnings,test,doc, and the MSRV'scheckapply, the newa_clone_reports_its_application_before_an_await_that_came_during_itfailed in 5 of 5 runstest-runtime-adapter, with the guest agent built at this headtest-changes,test-layout, andmarkdown-linksagainstdevCI run 42753203986
of this head,
a4e7f20d1, passed: it rebuilt the guest artifacts with thischange's guest binaries, and the microVM shards, guests and sandboxes, and the
runtime adapter's suite passed on KVM, MSHV, and WHP, the debug kernel on
MSHV, as did the nvx-rs jobs, NVX CLI on Linux and Windows, the packages and
platforms of each backend, the performance regression gate, and
Required status check. The merge group tests this pull request again,merged into the latest
dev. Left open in #544, as above: therestore-to-Ready measurement against the Activate path.
Changes to existing tests
python scripts/check_test_changes.pylists 128 changes to existing tests and13 added tests, hence
tests-modified. Each follows from the record's newpath, the packet instead of Activate, and Ready's, the host's request instead
of the guest's message:
nvx-rs/src/guest/clone/activation/tests.rs: the state machine takes itsrecord from the repair, so its helpers build a restored activation with a
packet's record (
ACTIVATEisAWAIT,bootstrapisrestored,restrictedisapplied), and its tests specify the packet's record inplace of an Activate: activation with the packet's record, refusal of any
request but Ready before Ready, the host's requests for Ready joining the
activation or getting its result, decoding and validation of the packet's
record before any side effect, and a deadline that nothing extends. Three
tests went with what they tested: retries of an Activate joining the
running activation, a changed record or a new transaction failing it, and
an Activate too long to read; a record arrives once, in the packet, whose
decoder bounds its length.
nvx-rs/src/guest/clone/activation/apply/tests.rs: the fake adapter handsthe application the packet's record rather than an Activate's, and the
tests of what the application mounts, generates, and tears down take their
record that way; their expectations are unchanged.
nvx-rs/src/host/template/clone/tests/launch.rsandrestore.rs: the fakeVMM keeps the record that the launch passes with
--restore-activation-record, the controls'activateisready, and thetests expect the launch to pass the record, the restore sequence and a
reattach to ask for Ready, and a retry or a replacement session to resend
nothing (
FakeControl::recordswent).tests/runtime-adapter/src/frame/tests.rs: Await, kind 28, replacesActivate, kind 27, in the framing's tables and tests, and kind 27 no longer
decodes.
tests/runtime-adapter/src/worker/activation/tests.rs: the worker appliesthe packet's record at the restore's reset, and its probe also reports when
the application completed.
tests/runtime-adapter/tests/guest_session.rs: a clone activates with itspacket's record and answers the host's Await, takes nothing but resets and
awaits before it is ready, and its failure tests take their record from the
packet; the malformed-message test sends kind 29. Three tests went with
Activate: one longer than a record; a host that does not activate in time,
now a host that never asks for Ready, which the state machine's deadline
test covers in its applied state; and an Activate that is not a retry.
tests/runtime-adapter/tests/host_session.rsandvm_clone.rs: theclient's order for a clone awaits its Ready first, a session that replaces
a lost one gets the same Ready, and the fake controls'
activateisready.tests/runtime-adapter/scenarios.txt: the suite namesa_clone_s_order_awaits_its_ready_first_then_starts_its_sandbox_as_after_a_cold_bootand
a_clone_s_session_that_ends_after_its_bootstrap_is_lost_and_the_next_gets_the_same_ready,the new names of the two
host_sessiontests above, in the order of thenames, as the README's stage scenarios do.