Repository navigation
ci: Benchmark the execution spec tests on CodSpeed - #1730
Merged
Merged
Conversation
Contributor
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1730 +/- ##
==========================================
+ Coverage 97.89% 97.98% +0.08%
==========================================
Files 183 183
Lines 16850 16851 +1
Branches 3856 3856
==========================================
+ Hits 16496 16511 +15
+ Misses 257 250 -7
+ Partials 97 90 -7
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
chfast
force-pushed
the
ci/eest-benchmarks
branch
2 times, most recently
from
September 23, 2026 21:05
c41ba92 to
eaf6642
Compare
The benchmarks tracked so far execute a single contract against a mocked host, which leaves the state, the host calls and the transaction handling unmeasured. Report the execution of each blockchain test `evmone test` runs as a benchmark, named by its fixture: that name is already the "<file>::<test>[<params>]" a benchmark is identified by. The reporting is done by codspeed-cpp, the library of the C++ benchmarks, whose integration CodSpeed knows and generates the flame graphs for. It is linked in only for a CodSpeed build and reports nothing unless the runner is attached, so the one binary stays the ordinary test runner.
chfast
force-pushed
the
ci/eest-benchmarks
branch
from
September 23, 2026 21:46
eaf6642 to
8033f7d
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved benchmark-boundary, naming, and fixture-integrity issues must be addressed.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Adds CodSpeed benchmarking for blockchain execution-spec tests.
Changes:
- Instruments blockchain fixture execution.
- Links CodSpeed into test utilities.
- Adds partitioned fixture benchmark jobs.
| File | Description |
|---|---|
test/utils/test_driver.cpp |
Adds measurement hooks; measurement cleanup on exceptions and file-qualified benchmark names need correction. |
test/CMakeLists.txt |
Links CodSpeed instrumentation into test utilities. |
.github/workflows/codspeed.yml |
Adds benchmark matrix jobs; the downloaded fixture archive requires SHA-256 verification. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Run the worst-case blocks of the execution spec tests as the tests they are: each checks its post state and is measured on the way. Every one is filled for seven block sizes and the smallest is taken, which covers all the benchmarks in a few minutes under the simulator where the largest would need most of an hour. CodSpeed rejects too many benchmarks in a single upload, so the fixtures are split into jobs, and two sweeps measured elsewhere are left out.
chfast
force-pushed
the
ci/eest-benchmarks
branch
from
September 23, 2026 22:56
8033f7d to
43869a5
Compare
chfast
changed the title
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved findings affect CodSpeed linkage, measurement cleanup, benchmark identity, and CI build efficiency.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (2)
Resolved since last review (2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The benchmarks tracked so far execute a single contract against a mocked host, which leaves the state, the host calls and the transaction handling unmeasured. This reports the execution of every blockchain test
evmone testruns to CodSpeed as a benchmark, and adds the worst-case blocks of the execution spec tests to the workflow, run as the tests they are: each checks its post state and is measured on the way.The reporting is codspeed-cpp's own, compiled in only for a CodSpeed build, and it reports nothing unless the runner is attached, so the one binary stays the ordinary test runner. The test driver brackets the test execution with codspeed-cpp's measurement calls.
What is measured
benchmark@v0.0.7fills every benchmark for seven block sizes; the smallest is taken, which covers all of them in a few minutes under the simulator where the largest would need most of an hour. Only the execution of a test is measured, not the loading of its fixture: the state transition, the state root and the post-state check.CodSpeed rejects too many benchmarks in a single upload (750 were), so the fixtures are split into jobs. The two suites sweeping what something else already covers give up their sweep (
modexp, which the precompiles benchmark sweeps already, andlog_benchmark, whose opcodeslog.jsonsweeps).Notes for review
<file>::<test>[<params>]a benchmark is identified by. The file belongs to the repository the fixture was filled from, so CodSpeed cannot link it.evmone-benchandevmone-precompiles-benchrun together in theevmonejob, which fails if either does.🤖 Generated with Claude Code