Design: keep original bytes per component, as the ledger does, instead of one format tree per transaction

[solidsnakedev]

Summary

Today the SDK keeps one CBOR format tree per decoded Transaction, in a WeakMap keyed on that object (Transaction.ts L130), and replays it on encode. Most of the recent format bugs come from that design:

• the tree is replayed only where the encoder produces the same shape the decoder read (Transaction: sets without tag 258 gain the tag on re-encode, changing the transaction id #574, CBOR: Plutus data encodings change on re-encode, breaking the script data hash and the transaction id #576, TxOut: map-form outputs without a script ref are re-encoded as arrays, changing the transaction id #577)
• it is lost as soon as any part is rebuilt, or when a component is encoded on its own (Signing: signTx hashes a re-encoded body when given a Transaction object #531, and the datum hash in Data: reject duplicate map keys on decode and make datum hashing byte faithful #397)
• on the explicit edit path it can be applied to a value it no longer fits (CBOR: preservation-aware encode reuses stale chunk sizes, writing different content #575)

The ledger does this differently. It keeps no format for the whole transaction. It keeps the original bytes of each part it hashes on its own, through MemoBytes, and serializes an untouched part by returning those bytes (cardano-ledger MemoBytes/Internal.hs L195, originalBytes = fromShort . mbBytes):

Ledger type (Conway)	Hash it feeds	Source
TxBody	transaction id	Conway/TxBody.hs L320
AlonzoTxWits	witness set	Alonzo/TxWits.hs L265
Redeemers, TxDats	script data hash	Alonzo/TxWits.hs L177, L358
Data	datum hash, inline datums	Plutus/Data.hs L95
AlonzoTxAuxData	auxiliary data hash	Alonzo/TxAuxData.hs L313
Timelock	native script hash	Allegra/Scripts.hs L244

This issue proposes the same model for the SDK.

Proposal

Keep one WeakMap from a decoded component object to the exact bytes it was decoded from, for the SDK types that match the table: TransactionBody, TransactionWitnessSet, Redeemers, the witness datum list, each Data value, AuxiliaryData and NativeScripts.

const originalBytes = new WeakMap<object, Uint8Array>()

// decode: each component records the slice it came from
originalBytes.set(body, bytes.subarray(start, end))

// encode: an untouched object returns its bytes; a new or edited one is encoded
const encodeBody = (body: TransactionBody) => originalBytes.get(body) ?? encodeBodyDefault(body)

The decoder already reports where each item starts and ends (decodeItemAt returns newOffset), so the slices come from the existing walk.

Encoding rule, in order:

1. explicit codec options from the caller: encode with them and ignore saved bytes (as today)
2. a component with saved bytes: return them
3. otherwise: the ledger's own encoding, which is the SDK default today (tag 258 on sets from protocol version 9, Encoder.hs L476-483; array-form outputs unless an inline datum or script ref needs the map, Babbage/TxOut.hs L502-506)

An edit that only appends keeps the edited container's layout. addVKeyWitnesses carries the witness set's captured layout forward (key order, definite or indefinite length, and the tag or its absence on the vkey list), appends the new witnesses, and takes every untouched entry (redeemers, datums, scripts) from its saved bytes. A witness set without a vkey entry gets one after its existing keys.

received witness set:   bf 05 [redeemers] 00 81 [vk1] ff
after adding vk2:       bf 05 [redeemers] 00 82 [vk1] [vk2] ff

What this settles

• A plain round trip returns the exact input, for every form the ledger accepts, because nothing is re-encoded. This covers Transaction: sets without tag 258 gain the tag on re-encode, changing the transaction id #574, CBOR: Plutus data encodings change on re-encode, breaking the script data hash and the transaction id #576 and TxOut: map-form outputs without a script ref are re-encoded as arrays, changing the transaction id #577 on the untouched path.
• An edit keeps every untouched part byte for byte. Adding a witness keeps the body, redeemers and datums, and the witness set's own layout, so the transaction id and script data hash stay valid. Changing the fee rebuilds the body but keeps each output's datums.
• Component hashes become right on decoded objects: TransactionBody.toHash (Signing: signTx hashes a re-encoded body when given a Transaction object #531), the datum hash (Data: reject duplicate map keys on decode and make datum hashing byte faithful #397), AuxiliaryData.toHash.
• Saved bytes cannot go stale: decoded objects are immutable, so a changed value is a new object with no saved bytes.
• Copies work: new Transaction({ ...tx, witnessSet }) keeps the body object and so its bytes.
• The public types, Equal, Hash and toJSON do not change, because the bytes live outside the objects.

Limits and open points

• A component rebuilt by a general edit, such as the body after a fee change, is encoded in the default form. The ledger does not care, since an edited body has a new transaction id anyway. Appending witnesses is not such an edit: the witness set keeps its layout, as described above. The explicit *WithFormat API can stay for callers who want other edited containers kept, with the CBOR: preservation-aware encode reuses stale chunk sizes, writing different content #575 guard.
• A WeakMap key must be an object, so a top-level datum that is a bare bigint falls back to a normal encode. Integers inside a list, map or constructor are covered by their parent's bytes.
• Readonly types do not stop runtime mutation of an array or Map inside a decoded object; that would leave stale bytes. Open: freeze decoded components, or document that decoded objects must not be mutated.
• Every decode path must record bytes, including the Schema path (Transaction.FromCBORHex()), which today never fills the cache.
• Open: whether the whole-transaction format tree stays for anything once components keep their bytes, or is removed.

Not covered here: #578 (repeated keys), #579 (codec options in toScriptDataHash) and #580 (4- and 8-byte tag headers) are decoder and API fixes needed either way.

Suggested stages

1. TransactionBody and Data: transaction id and datum hash
2. Redeemers, witness datums and TransactionWitnessSet: script data hash
3. AuxiliaryData and NativeScripts
4. retire or narrow the transaction-level format cache

Tests

The CML parity suite and the devnet suite built for #574 to #580 serve as acceptance tests. Each stage flips its cases from failing to passing: the original bytes are accepted by the node, and the SDK-processed transaction is accepted too instead of being rejected with 3100, 3107 or 3113.