Repository navigation
dry-run ledger scripts: keep ledger bytes with mina-json-bytes - #19987
Conversation
|
!ci-build-me |
SanabriaRusso
left a comment
There was a problem hiding this comment.
LGTM. I checked that every step that writes a ledger now goes through mina-json-bytes: prepare-test-ledger.sh (filter and transform), update_extra_balances, and the injection and runtime_config_full.json steps in generate-ledger-hf-dryrun.sh. The remaining plain jq calls only read, or build JSON with no account data (hashes.json, the injected key entries). Every flag used (--argjson, --slurpfile, -n, input from stdin) is allowed by the wrapper's rules in v0.0.10, and the toolchain image already ships it. Two non-blocking comments inline. I didn't re-run the scripts end to end.
| # step that writes the ledger uses mina-json-bytes. | ||
| # jq filter to exclude PKs from the ledger | ||
| if ! <"$ledger_file" jq "[.[] | select(.pk | IN($keys_) | not)]" >"$tmpfile"; then | ||
| if ! <"$ledger_file" mina-json-bytes jq "[.[] | select(.pk | IN($keys_) | not)]" >"$tmpfile"; then |
There was a problem hiding this comment.
Non-blocking. A single mainnet zkApp URI can make this step, and so every dry-run ledger build, fail.
Repro: not run locally. The tool's own test (mina-json-bytes/src/transcode.rs, v0.0.10) shows it:
fn encoder_rejects_the_prefix_in_the_input() {
let error = encoded(b"a __MINA_BYTE_ff__ b").unwrap_err();
assert!(matches!(error, Error::PrefixInInput));So a staking ledger dump with an account whose zkapp_uri contains __MINA_BYTE_ makes mina-json-bytes jq exit 1 here.
Why: encode rejects that prefix anywhere in its input, so that decode cannot mistake the text for a token. zkApp URIs are free text that any account owner can set. Token symbols are too short (6 bytes) to hold it. Before this PR, plain jq let such a ledger through; only the invalid bytes were changed.
Suggestion: the failure is loud, not silent corruption, so this doesn't block the PR. It's worth noting in the PR description so that a failed dry-run prep before a hard fork can be traced quickly. A longer-term fix belongs in the tool: escape the prefix instead of rejecting it.
There was a problem hiding this comment.
Agreed. The PR description has a new "Known limit" section: it quotes the tool's error (the input already contains the text __MINA_BYTE_), says that one zkApp URI with that text stops every dry-run ledger build, and that no ledger is written with wrong bytes. The fix belongs in mina-json-bytes, which should escape the text instead of refusing it. v0.0.11, released today, still refuses it.
| fi | ||
|
|
||
| if ! command -v mina-json-bytes >/dev/null 2>&1; then | ||
| missing_deps+=("mina-json-bytes") |
There was a problem hiding this comment.
Nit: when mina-json-bytes is missing, the error says "Missing required dependencies: mina-json-bytes" but not where to get it. Only jq gets a hint line (line 34).
Why: only the toolchain image has the tool (dockerfiles/toolchain/3-toolchain:58). It isn't in flake.nix or any apt package. Someone running this on their own machine or in a Nix shell has to work out where it comes from.
Suggestion: add a hint line next to the jq one at line 34, and in the other two scripts' checks:
echo " - mina-json-bytes: keeps non-UTF-8 ledger bytes exact through jq; install from https://github.com/MinaProtocol/mina-release-toolkit/releases" >&2There was a problem hiding this comment.
Fixed in 61fc0c3. All three scripts print the hint when mina-json-bytes is missing: it keeps non-UTF-8 ledger bytes exact through jq, the toolchain image has it, and elsewhere it comes from the mina-release-toolkit package (releases link). prepare-test-ledger.sh has one check for all its tools, so it prints the hint only when that tool is the missing one. Each script, run without the tool on PATH, prints the hint and exits 1. Shellcheck warnings are the same as before.
Only the toolchain image has it, so a missing tool now points to the mina-release-toolkit package.
|
!ci-build-me |
Why
Token symbols and zkApp URIs are arbitrary bytes, so a ledger is not always valid UTF-8, and these bytes must stay exact. The dry-run ledger scripts rewrite ledgers from the staking ledger dumps with plain
jq, so a dry run can get a ledger with different account data, and so a different hash, from the source ledger.What
prepare-test-ledger.sh,prepare-test-ledger-hf-dryrun.shandgenerate-ledger-hf-dryrun.sh: each step that writes a ledger now usesmina-json-bytes jq.mina-json-byteswith their other dependencies. The toolchain image has it. On any other host, install it from themina-release-toolkitrelease. When it is missing, each script says so and where to get it.Steps that only read numbers or keys from a ledger keep plain
jq.Known limit
mina-json-bytesstops on input that already contains the text__MINA_BYTE_, because its decoder could not tell that text from its own tokens. The error is:A zkApp URI is free text that any account owner sets, so one URI with that text in a staking ledger dump stops every dry-run ledger build. Token symbols are too short (6 bytes) to hold it. Before this change, plain
jqlet such a ledger through and changed only its invalid bytes. The failure is loud: no ledger is written with wrong bytes. The fix belongs in the tool, which should escape the text instead of refusing it. v0.0.11 still refuses it.Update (2026-11-16): fixed.
mina-release-toolkitv0.0.12 (MinaProtocol/mina-release-toolkit#53) escapes the text instead of refusing it, and every toolchain image oncompatiblecarries v0.0.12 since #19998.Tests
The input was a test ledger with a raw
0xfftoken symbol and an invalid UTF-8 zkApp URI (ed b3 bf).prepare-test-ledger.shoncompatibleprepare-test-ledger.shprepare-test-ledger-hf-dryrun.sh -e 1 -b 7genesis.json,staking.jsonandnext.json; extra key balance setprepare-test-ledger.shwithout the tool onPATHPATHgenerate-ledger-hf-dryrun.shwas not run end to end, because it needs the real ledger and binaries. Its--argjsonand three--slurpfileforms were tested throughmina-json-bytes jqon their own, with the same result.Only scripts change, so no changelog entry.