Skip to content

mina-json-bytes: escape the token prefix instead of refusing the input - #53

Merged
dkijania merged 1 commit into
mainfrom
dkijania/json-bytes-escape-prefix
Nov 12, 2026
Merged

dkijania merged 1 commit into
mainfrom
dkijania/json-bytes-escape-prefix

Conversation

@dkijania

Copy link
Copy Markdown
Member

Problem

mina-json-bytes encode stops with "the input already contains the text __MINA_BYTE_". Without that rule, decode could not tell such text from a token. But a zkApp URI is free text that any account owner sets. One account with that text in its URI stops every ledger rewrite that uses the tool, for example each dry-run ledger build in the mina repository (MinaProtocol/mina#19987, "Known limit").

Change

Where the input holds the text __MINA_BYTE_, encode writes the first _ of it as the token __MINA_BYTE_5f__, and continues after it:

input      __MINA_BYTE_ff__
encoded    __MINA_BYTE_5f___MINA_BYTE_ff__
decoded    __MINA_BYTE_ff__

decode reads from left to right. It gives back the _, and the text after the token is no longer a complete token, so it stays text. decode is unchanged: it already accepted a token for any byte. So decode(encode(x)) == x holds for every input now, and nothing is refused.

  • transcode.rs: the encoder writes the text before each prefix (valid UTF-8 as it is, other bytes as tokens), then the escaped _. A character cut just before a prefix is invalid, because _ follows it. A prefix split across two reads is still found, because the last bytes stay pending as before.
  • error.rs: PrefixInInput is removed.
  • token.rs, README.md: describe the escape. The README no longer lists the refusal as a limit.

Tests

  • New unit tests: an escaped prefix and its round trip; overlapping prefixes (__MINA_BYTE__MINA_BYTE_, ___MINA_BYTE_); a prefix split across reads at every read size; a cut character before a prefix.
  • The random round-trip test now also uses __MINA_BYTE_ and __MINA_BYTE_ff__ as parts, and requires every input to encode. It was also run once with 200,000 inputs instead of 3,000: every input came back exactly at every read size, and every output was valid UTF-8.
  • tests/cli.rs: the two refusal tests now check that a URI holding __MINA_BYTE_ff__ next to a raw 0xff byte comes back unchanged through encode and decode, and through jq -c ..
  • cargo test: 20 unit tests and 12 CLI tests pass. cargo clippy --all-targets -- -D warnings is clean.

After a release with this change, the mina toolchain image needs the new version.

Where the input already holds __MINA_BYTE_, encode writes its first byte as the token __MINA_BYTE_5f__, so decode gives the text back and cannot take it for a token. A zkApp URI is free text, so one account could stop every ledger rewrite before.
@dkijania
dkijania merged commit 099cf45 into main Nov 12, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant