Repository navigation
Description
Goal
Measure which code NVX's tests run, on Linux and Windows, and publish it as dev changes, so that untested code shows up in a report rather than in a regression: the nvx crate under its own tests, the host code of nvx-rs under the microVM scenarios, and the Python host tooling of scripts/. Coverage tells which code runs, and the on-demand mutation testing of nvx-rs-mutants.yml tells whether the tests check what it does, so the two complement each other.
Background
Nothing measures code coverage today. #726 added nvx-rs-coverage.yml and scripts/nvx_rs_coverage.py, which measured only the lines and branches that each family of golden tests alone reached, for the retirements of #722, and 10035af deleted both once the last family was gone. What they needed stays: nvx-rs and nvx-fake-openvmm return their exit statuses from main, so that instrumented builds write their profiles on Windows (84edb1f), and tests/guest_tools.rs ignores the __LLVM_PROFILE_RT_INIT_ONCE variable that an instrumented guest binary passes on (0bc5b9b).
At 82c15cd, on a Windows development host (Linux isn't measured yet):
| Code | Tests | Tool | Lines | Branches |
|---|---|---|---|---|
nvx-rs, every feature |
Its unit and integration tests | cargo-llvm-cov 0.9.1, nightly-2026-07-09 |
91.2% of 29,650 | 86.1% of 5,120 |
scripts/, without its test_*.py modules |
The modules that validate-nvx runs |
coverage.py 7.17.0 | 84.3% of 30,449 statements | 75.7% |
- Every
nvx-rstest passed instrumented, in 165 s, 59 s of which built the crate. - The least-covered code of
nvx-rsmostly needs a VM or the network:src/host/instance/(75.8% of lines, 62.5% of branches),src/host/template/clone/launch.rs(66.2% of lines),src/host/cli/download.rs(74.0% of lines),src/host/template/store/capture.rs(65.3% of branches), andsrc/host/platform/(71.5% of branches). - By default, cargo-llvm-cov counts a test vector that a unit test includes, and the fake OpenVMM of
src/testing/, 1,611 lines, as production code. - The guest code runs only on Linux:
tests/guest_tools.rsran no test on Windows. - The Python modules took 16 minutes under coverage.py, with the Python subprocesses that they start measured, and reported 7 errors among 1,909 tests. Without coverage, the same host reports 4 errors in subtests of
test_host_loopback_allow_script_checks_publication_and_traffic, whose shell script exceeds its 10-second timeout there, and none intest_adversarial.py, which reported one under coverage.
Plan
The sub-issues are the steps, and their blocked-by relationships record what each step needs.
- ci: report the coverage of the nvx crate's tests on Linux and Windows #1323 reports the coverage of
nvx-rsunder its own tests, on Linux and Windows. - ci: show pull requests the nvx-rs lines that they add and no test reaches #1324 shows each pull request the lines of
nvx-rsthat it adds and no test reaches. It needs ci: report the coverage of the nvx crate's tests on Linux and Windows #1323. - ci: measure the nvx-rs host code that the KVM microVM scenarios reach #1325 measures the host code of
nvx-rsthat the microVM scenarios reach on KVM. It needs ci: report the coverage of the nvx crate's tests on Linux and Windows #1323. - ci: report the coverage of the Python host tooling in scripts/ #1326 reports the coverage of
scripts/under its unit tests, independently of the others.
Constraints
- Coverage informs and gates nothing: no required check depends on it, and no threshold fails a run. Some code runs in only some runs, such as the loop of
wait_interruptiblyinsrc/host/cli/sandbox/lifecycle/start.rs, which waits for a process that may already have exited. - Reports count production code: test files, test data, and test doubles are left out or reported apart.
Open questions
- Compiler: the release that
rust-toolchain.tomlpins measures lines, regions, and functions with itsllvm-toolscomponent, but branch coverage still needs a nightly compiler, which nvx-rs: measure line, branch, and mutation coverage on Linux and Windows #726 pinned apart fromrust-toolchain.toml. - Schedule: every push to
dev, or a weekly run, for ci: report the coverage of the nvx crate's tests on Linux and Windows #1323, ci: measure the nvx-rs host code that the KVM microVM scenarios reach #1325, and ci: report the coverage of the Python host tooling in scripts/ #1326. - Publication: workflow artifacts and run summaries only, or also a third-party service such as Codecov, which would need approval.
Out of scope
- OpenVMM, whose own CI owns its coverage (
doc/contribute.md, "Test ownership"). - Kernel coverage, which needs a kernel configuration of its own.
- The binaries that run inside guests, whose profiles would stay in the guest.
- The coverage manifest of nvx-rs: final acceptance for the image-backed runtime #624, which maps requirements to tests rather than measuring code.
Done when
- Every sub-issue is closed.
-
doc/ci.mddescribes each measurement, what it counts and leaves out, and how to run it on a development host. - The first runs on
devrecord the baseline of Linux and Windows here.
Activity
ppenna commentedon Nov 11, 2026
Every sub-issue is closed,
doc/ci.mddescribes the three measurements, and this comment records the baseline of their first runs ondev, so this closes the issue.Done when
dev, ci: show pull requests the nvx-rs lines that they add and no test reaches #1324 by ci: show pull requests the nvx-rs lines that they add and no test reaches #1353, ci: measure the nvx-rs host code that the KVM microVM scenarios reach #1325 after ci: measure the nvx-rs host code that the KVM microVM scenarios reach #1393 and the first run of its workflow on a KVM runner, and ci: report the coverage of the Python host tooling in scripts/ #1326 by ci: report the coverage of the Python host tooling in scripts/ #1344.doc/ci.mddescribes each measurement, what it counts and leaves out, and how to run it on a development host: nvx-rs coverage, with Changed lines that no test reaches, nvx-rs scenario coverage, and scripts coverage each say what they count and leave out, and give the commands that measure on a development host.devrecord the baseline of Linux and Windows here: below. Only nvx-rs coverage measures Windows: as ci: report the coverage of the Python host tooling in scripts/ #1326 and ci: measure the nvx-rs host code that the KVM microVM scenarios reach #1325 planned, scripts coverage runs onubuntu-latest, and scenario coverage on Linux/KVM.Baseline
nvx-rs coverage
42387413841, on 2026-11-07 at 06e8dd5, the merge of #1333, with Rust 1.97.1 and cargo-llvm-cov 0.9.1. No test ran again, and no profile strayed.
The guest code of
src/guest/, which only Linux compiles, accounts for the difference between the OSes: its tests reach 73.7% of its 17,688 lines,src/guest/time52.4% of 4,613 andsrc/guest/sys64.4% of 2,683, and 91.6% of the 27,454 other production lines of Linux. Branches go unmeasured, as the compiler below explains.Pull requests that change
nvx-rs/list the lines that they add or change and that no test reaches, as #1324 requires. #1422's 42690462465 listed line 520 ofsrc/host/template/store/capture.rson both OSes, #1427's 42753203728 listed 10 lines ofsrc/guest/clone/on Linux, which alone compiles them, and #1393's 42446261760, which changed no production code, listed none.scripts coverage
42398882635, on 2026-11-07 at 8394746, the merge of #1344, on Linux with Python 3.12.16 and coverage.py 7.17.0, in 214 s.
All 1,980 tests passed: the 1,892 of
python3 -m unittest, 30 of them skipped, the 84 oftest_adversarial.py, and the 4 oftest_hosts.py. Without coverage, theNVX CLI / Linuxjob of the same commit had the same results, but for one more skip,test_a_measurement_of_a_sample_checkout, which needs coverage.py. The reports hold notest_*.pymodule. Of the modules of more than 400 statements, the least covered areadversarial_executor.py(58.8% of 423),ubuntu.py(65.4% of 648),collect_ubuntu_sources.py(67.7% of 418),adversarial_oracles.py(70.5% of 714),benchmark.py(71.1% of 3,185), andbuild.py(71.5% of 625), and, of the scenario drivers,microvm_tests.py(74.7% of 4,020).nvx-rs scenario coverage
Its first run, the scheduled 42460108426 at bf81488, stopped before it measured, since no KVM runner had the coverage tools yet. The first to measure, 42768407172, dispatched on 2026-11-10 at b6ad700, ran on
azure-kvm-5:The scenarios reach 82 lines that the crate's tests don't, and 57.8% of the 26,336 lines that their builds compile. Each of the 144
nvx-rsprocesses that they started wrote its profile.Open questions
rust-toolchain.tomlpins, with itsllvm-toolscomponent, as ci: report the coverage of the nvx crate's tests on Linux and Windows #1333 chose, so the Rust measurements leave branches out until they no longer need a nightly compiler. coverage.py measures the branches ofscripts/.devand the ready pull requests that changenvx-rs/or how it is measured, scripts coverage for the pushes todevthat changescripts/or how it is measured, and scenario coverage every Sunday at 07:41 UTC. Each also runs on dispatch.Constraints
dev's ruleset requires onlyRequired status check, a job ofci.yml, which refers to none of the three workflows, and no threshold fails a run. A run fails only when its measurement is incomplete, as when tests fail.nvx-rsand the scenario drivers ofscripts/.The summaries of the runs follow, since their artifacts expire on 2026-12-07 and 2026-12-10.
nvx-rs coverage on Linux, run 42387413841
What the tests of the
nvxcrate reached at06e8dd511dfe, with every feature, built by rustc 1.97.1 (8bab26f4f 2026-07-14) and measured by cargo-llvm-cov 0.9.1. Production code leaves out test code, test data, and the test doubles, the fake OpenVMM ofsrc/testing/and itsnvx-fake-openvmmbinary, which count apart.The profiles include processes of
nvx-console-pending,nvx-device-io,nvx-exit,nvx-fake-openvmm,nvx-ingress,nvx-init,nvx-managed-agent,nvx-mmio-write,nvx-reseed,nvx-rs,nvx-time, andnvx-time-probe.Production code by module
A row counts the files of a top-level module, or of one of its submodules that has submodules of its own, that no other row counts.
src/activationsrc/binsrc/contractsrc/errorsrc/execsrc/exec/creditsrc/gptsrc/guestsrc/guest/clonesrc/guest/execsrc/guest/ingresssrc/guest/managed_agentsrc/guest/mappingssrc/guest/rootsrc/guest/syssrc/guest/timesrc/guest/time_probesrc/hostsrc/host/artifactssrc/host/clisrc/host/filesystemsrc/host/instancesrc/host/networksrc/host/platformsrc/host/sandboxsrc/host/storesrc/host/supervisorsrc/host/syssrc/host/templatesrc/hypervisorsrc/netsrc/protocolsrc/protocol/nvxssrc/uuidnvx-rs coverage on Windows, run 42387413841
What the tests of the
nvxcrate reached at06e8dd511dfe, with every feature, built by rustc 1.97.1 (8bab26f4f 2026-07-14) and measured by cargo-llvm-cov 0.9.1. Production code leaves out test code, test data, and the test doubles, the fake OpenVMM ofsrc/testing/and itsnvx-fake-openvmmbinary, which count apart.The profiles include processes of
nvx-fake-openvmmandnvx-rs. No process ofnvx-console-pending,nvx-device-io,nvx-exit,nvx-ingress,nvx-init,nvx-managed-agent,nvx-mmio-write,nvx-reseed,nvx-time, andnvx-time-probewrote one.Production code by module
A row counts the files of a top-level module, or of one of its submodules that has submodules of its own, that no other row counts.
src/activationsrc/binsrc/contractsrc/errorsrc/execsrc/exec/creditsrc/gptsrc/hostsrc/host/artifactssrc/host/clisrc/host/filesystemsrc/host/instancesrc/host/networksrc/host/platformsrc/host/sandboxsrc/host/sealsrc/host/storesrc/host/supervisorsrc/host/templatesrc/hypervisorsrc/netsrc/protocolsrc/protocol/nvxssrc/uuidscripts coverage on Linux, run 42398882635
What the unit tests that validate-nvx runs reached of the Python modules of
scripts/at8394746fd864, with the Python processes that they started, on Python 3.12.16, measured by coverage.py 7.17.0. The unit tests, thetest_*.pymodules, and their test support go unmeasured, and the end-to-end scenario drivers, the*_tests.pymodules ofscripts/nvx_tools/, which only a hypervisor runs end to end, count apart.python3 -m unittestwith 24 modulespython3 scripts/test_adversarial.py -vpython3 .github/skills/nvx-host-connect/scripts/test_hosts.py -vHost tooling by module
scripts/check_markdown_links.pyscripts/check_test_changes.pyscripts/check_test_layout.pyscripts/module_diagrams.pyscripts/nvx.pyscripts/nvx_adversarial_executor.pyscripts/nvx_rs_coverage.pyscripts/nvx_tools/__init__.pyscripts/nvx_tools/adversarial.pyscripts/nvx_tools/adversarial_broker.pyscripts/nvx_tools/adversarial_executor.pyscripts/nvx_tools/adversarial_oracles.pyscripts/nvx_tools/archive.pyscripts/nvx_tools/azurelinux.pyscripts/nvx_tools/benchmark.pyscripts/nvx_tools/benchmark_host.pyscripts/nvx_tools/build.pyscripts/nvx_tools/build_config.pyscripts/nvx_tools/build_constants.pyscripts/nvx_tools/ci.pyscripts/nvx_tools/collect_alpine_sources.pyscripts/nvx_tools/collect_ubuntu_sources.pyscripts/nvx_tools/common.pyscripts/nvx_tools/crate_release.pyscripts/nvx_tools/create_linux_source_archive.pyscripts/nvx_tools/development_release.pyscripts/nvx_tools/doctor.pyscripts/nvx_tools/elf.pyscripts/nvx_tools/guest_artifact_reuse.pyscripts/nvx_tools/guest_binaries.pyscripts/nvx_tools/guests.pyscripts/nvx_tools/host_telemetry.pyscripts/nvx_tools/kconfig.pyscripts/nvx_tools/kernel_config.pyscripts/nvx_tools/kernel_features.pyscripts/nvx_tools/merge_queue.pyscripts/nvx_tools/microvm_durations.pyscripts/nvx_tools/microvm_shards.pyscripts/nvx_tools/nightly_ci.pyscripts/nvx_tools/nvx_rs.pyscripts/nvx_tools/openvmm_ci.pyscripts/nvx_tools/openvmm_process.pyscripts/nvx_tools/performance.pyscripts/nvx_tools/performance_history.pyscripts/nvx_tools/process_supervisor.pyscripts/nvx_tools/release.pyscripts/nvx_tools/shard_plans.pyscripts/nvx_tools/storage_events.pyscripts/nvx_tools/time_abi.pyscripts/nvx_tools/ubuntu.pyscripts/public_api.pyscripts/publish_development_release.pyscripts/publish_nvx_crate.pyscripts/scripts_coverage.pyscripts/setup/azure/build_package.pyscripts/setup/azure/nvx_azure/__init__.pyscripts/setup/azure/nvx_azure/cli.pyscripts/setup/azure/nvx_azure/common.pyscripts/setup/azure/nvx_azure/linux_bootstrap.pyscripts/setup/azure/nvx_azure/linux_disk.pyscripts/setup/azure/nvx_azure/linux_firewall.pyscripts/setup/azure/nvx_azure/linux_mshv.pyscripts/setup/azure/nvx_azure/windows_bootstrap.pyscripts/setup/azure/nvx_azure/windows_disk.pyscripts/setup/azure/nvx_azure/windows_firewall.pyscripts/setup/azure/nvx_azure/windows_stage.pyscripts/setup/azure/nvx_azure/windows_system.pyScenario drivers by module
scripts/nvx_tools/managed_exec_tests.pyscripts/nvx_tools/microvm_tests.pyscripts/nvx_tools/runtime_adapter_tests.pyscripts/nvx_tools/sandbox_lifecycle_tests.pyscripts/nvx_tools/sandbox_proxy_tests.pyscripts/nvx_tools/sandbox_ram_upper_tests.pynvx-rs scenario coverage on Linux/KVM, run 42768407172
What the KVM microVM scenarios and the runtime adapter's suite reached of the production code of the
nvxcrate atb6ad700744bc, alone and together with the crate's own tests, built by rustc 1.97.1 (8bab26f4f 2026-07-14) and measured by cargo-llvm-cov 0.9.1. Production code leaves out test code, test data, and the test doubles, as nvx-rs coverage does. A line or a function counts once, as reached once any build of the crate ran it.The scenarios reach 82 lines that the crate's tests don't. Their profiles include processes of
nvx-rs, the crate's tests that the scenarios run, and the runtime adapter's tests and host program. Of the lines of production code that the builds of the scenarios compile, they reach 57.8% of 26,336. These builds leave out the other lines, among them the guest code, which runs only in guests.nvx-rsprocessesThe scenarios started 144
nvx-rsprocesses, and each wrote its profile.Production lines by module
A row counts the files of a top-level module, or of one of its submodules that has submodules of its own, that no other row counts. The last column counts the lines that the scenarios reach and the crate's tests don't.
src/activationsrc/binsrc/contractsrc/errorsrc/execsrc/exec/creditsrc/gptsrc/guestsrc/guest/clonesrc/guest/execsrc/guest/ingresssrc/guest/managed_agentsrc/guest/mappingssrc/guest/rootsrc/guest/syssrc/guest/timesrc/guest/time_probesrc/hostsrc/host/artifactssrc/host/clisrc/host/filesystemsrc/host/instancesrc/host/networksrc/host/platformsrc/host/sandboxsrc/host/storesrc/host/supervisorsrc/host/syssrc/host/templatesrc/hypervisorsrc/netsrc/protocolsrc/protocol/nvxssrc/uuid