Skip to content

dry-run ledger scripts: keep ledger bytes with mina-json-bytes - #19987

Merged
dkijania merged 2 commits into
compatiblefrom
dkijania/dryrun-ledger-json-bytes
Nov 12, 2026
Merged

dkijania merged 2 commits into
compatiblefrom
dkijania/dryrun-ledger-json-bytes

Conversation

@dkijania

@dkijania dkijania commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Why

Token symbols and zkApp URIs are arbitrary bytes, so a ledger is not always valid UTF-8, and these bytes must stay exact. The dry-run ledger scripts rewrite ledgers from the staking ledger dumps with plain jq, so a dry run can get a ledger with different account data, and so a different hash, from the source ledger.

What

  • prepare-test-ledger.sh, prepare-test-ledger-hf-dryrun.sh and generate-ledger-hf-dryrun.sh: each step that writes a ledger now uses mina-json-bytes jq.
  • The scripts check for mina-json-bytes with their other dependencies. The toolchain image has it. On any other host, install it from the mina-release-toolkit release. When it is missing, each script says so and where to get it.

Steps that only read numbers or keys from a ledger keep plain jq.

Known limit

mina-json-bytes stops on input that already contains the text __MINA_BYTE_, because its decoder could not tell that text from its own tokens. The error is:

the input already contains the text __MINA_BYTE_

A zkApp URI is free text that any account owner sets, so one URI with that text in a staking ledger dump stops every dry-run ledger build. Token symbols are too short (6 bytes) to hold it. Before this change, plain jq let such a ledger through and changed only its invalid bytes. The failure is loud: no ledger is written with wrong bytes. The fix belongs in the tool, which should escape the text instead of refusing it. v0.0.11 still refuses it.

Update (2026-11-16): fixed. mina-release-toolkit v0.0.12 (MinaProtocol/mina-release-toolkit#53) escapes the text instead of refusing it, and every toolchain image on compatible carries v0.0.12 since #19998.

Tests

The input was a test ledger with a raw 0xff token symbol and an invalid UTF-8 zkApp URI (ed b3 bf).

Script Result
prepare-test-ledger.sh on compatible Both sequences lost, 2 U+FFFD in the output
prepare-test-ledger.sh Both sequences exact, no U+FFFD
prepare-test-ledger-hf-dryrun.sh -e 1 -b 7 Exact in genesis.json, staking.json and next.json; extra key balance set
prepare-test-ledger.sh without the tool on PATH Exit 1, missing dependency message with the install hint
The other two scripts without the tool on PATH Same message and hint

generate-ledger-hf-dryrun.sh was not run end to end, because it needs the real ledger and binaries. Its --argjson and three --slurpfile forms were tested through mina-json-bytes jq on their own, with the same result.

Only scripts change, so no changelog entry.

@dkijania

dkijania commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

!ci-build-me

@SanabriaRusso SanabriaRusso left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I checked that every step that writes a ledger now goes through mina-json-bytes: prepare-test-ledger.sh (filter and transform), update_extra_balances, and the injection and runtime_config_full.json steps in generate-ledger-hf-dryrun.sh. The remaining plain jq calls only read, or build JSON with no account data (hashes.json, the injected key entries). Every flag used (--argjson, --slurpfile, -n, input from stdin) is allowed by the wrapper's rules in v0.0.10, and the toolchain image already ships it. Two non-blocking comments inline. I didn't re-run the scripts end to end.

# step that writes the ledger uses mina-json-bytes.
# jq filter to exclude PKs from the ledger
if ! <"$ledger_file" jq "[.[] | select(.pk | IN($keys_) | not)]" >"$tmpfile"; then
if ! <"$ledger_file" mina-json-bytes jq "[.[] | select(.pk | IN($keys_) | not)]" >"$tmpfile"; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking. A single mainnet zkApp URI can make this step, and so every dry-run ledger build, fail.

Repro: not run locally. The tool's own test (mina-json-bytes/src/transcode.rs, v0.0.10) shows it:

fn encoder_rejects_the_prefix_in_the_input() {
    let error = encoded(b"a __MINA_BYTE_ff__ b").unwrap_err();
    assert!(matches!(error, Error::PrefixInInput));

So a staking ledger dump with an account whose zkapp_uri contains __MINA_BYTE_ makes mina-json-bytes jq exit 1 here.

Why: encode rejects that prefix anywhere in its input, so that decode cannot mistake the text for a token. zkApp URIs are free text that any account owner can set. Token symbols are too short (6 bytes) to hold it. Before this PR, plain jq let such a ledger through; only the invalid bytes were changed.

Suggestion: the failure is loud, not silent corruption, so this doesn't block the PR. It's worth noting in the PR description so that a failed dry-run prep before a hard fork can be traced quickly. A longer-term fix belongs in the tool: escape the prefix instead of rejecting it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The PR description has a new "Known limit" section: it quotes the tool's error (the input already contains the text __MINA_BYTE_), says that one zkApp URI with that text stops every dry-run ledger build, and that no ledger is written with wrong bytes. The fix belongs in mina-json-bytes, which should escape the text instead of refusing it. v0.0.11, released today, still refuses it.

fi

if ! command -v mina-json-bytes >/dev/null 2>&1; then
missing_deps+=("mina-json-bytes")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: when mina-json-bytes is missing, the error says "Missing required dependencies: mina-json-bytes" but not where to get it. Only jq gets a hint line (line 34).

Why: only the toolchain image has the tool (dockerfiles/toolchain/3-toolchain:58). It isn't in flake.nix or any apt package. Someone running this on their own machine or in a Nix shell has to work out where it comes from.

Suggestion: add a hint line next to the jq one at line 34, and in the other two scripts' checks:

echo "  - mina-json-bytes: keeps non-UTF-8 ledger bytes exact through jq; install from https://github.com/MinaProtocol/mina-release-toolkit/releases" >&2

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 61fc0c3. All three scripts print the hint when mina-json-bytes is missing: it keeps non-UTF-8 ledger bytes exact through jq, the toolchain image has it, and elsewhere it comes from the mina-release-toolkit package (releases link). prepare-test-ledger.sh has one check for all its tools, so it prints the hint only when that tool is the missing one. Each script, run without the tool on PATH, prints the hint and exits 1. Shellcheck warnings are the same as before.

Only the toolchain image has it, so a missing tool now points to the mina-release-toolkit package.
@dkijania

Copy link
Copy Markdown
Member Author

!ci-build-me

@dkijania
dkijania merged commit a42d9c8 into compatible Nov 12, 2026
78 checks passed
@dkijania
dkijania deleted the dkijania/dryrun-ledger-json-bytes branch November 12, 2026 20:26
@github-project-automation github-project-automation Bot moved this from To triage to Done in Mesa Triage Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants