solana-indexer: revert rolled-back transactions when their slots finalize by squadgazzz · Pull Request #4875 · cowprotocol/services

squadgazzz · 2026-09-07T09:49:25Z

Description

Reorg handling for the indexer, the second half of BE-203, stacked on #4854. Solana cannot reorg past finality, so the moment slots become final is the last place a reorg can be caught.

Confirmed transactions vanish only on an optimistic-confirmation safety failure, so the audit is a no-op in practice: one batched getSignatureStatuses call per finalized tick, and none when the range holds no rows. An RPC failure delays finalization to the next tick, it never skips a range and never reverts on its own.

Changes

The decoder audits each newly finalized range and reverts transactions that the chain no longer knows
The revert cascade removes the transaction's trades (subtracting the fill sums they added, aggregated per order), settlements, and dead letters, and marks the orders it created is_reorged, atomically with the watermark advance. Marked orders are skipped by the solvable-orders query and the API reads, and a re-landed creation clears the flag, mirroring EVM's is_reorged on onchain_placed_orders
V3 migration: solana.order_pda carries created_by_tx, created_in_slot, and is_reorged, all creation provenance and reorg state on the on-chain sidecar next to created_by, solana.orders stays untouched, plus slot indexes for the audit scans
cow-solana-rpc gains known_signatures, the batched recent-status probe

How to test

New ignored postgres tests for the cascade, and the pipeline test drives the audit through a mock RPC.

Related issues

BE-203

linear-code · 2026-09-07T09:49:28Z

BE-203

claude · 2026-09-07T17:45:58Z

Claude finished @squadgazzz's task in 3m 33s —— View job

Review

Reviewed the revert cascade, the finalization audit, the RPC probe, and the V3 migration. The cascade SQL itself (per-order delta aggregation, delete order, is_reorged marking, atomic watermark advance) looks correct and well-tested. Two concerns, both centered on the audit's assumption that it always runs within the RPC's recent-status cache window — this breaks on the catch-up/boot path where the range grows large:

known_signatures false negatives on aged transactions — cow-solana-rpc/src/lib.rs: get_signature_statuses doesn't set searchTransactionHistory, so a healthy but aged-out transaction returns None. Combined with the unbounded audit range in decoder.rs, this can turn a catch-up after downtime into mass false-positive reverts of data that finalized fine. Also flags the 256-signature-per-call cap (no chunking).
Unbounded audit range — decoder.rs finalize: after can be far behind the current finalized slot after an outage or accumulated RPC failures; auditing that whole range against the ~150-slot cache is where the false positives materialize. Suggest bounding to the cache window or using searchTransactionHistory for the aged tail.

Nothing else stood out — the migration, reader filters (NOT COALESCE(p.is_reorged, false)), and re-landing flag-clear all look consistent with the described EVM parity.
· solana-indexer/be-203-rollback-cascade

claude · 2026-09-07T17:49:12Z

+    pub async fn known_signatures(&self, signatures: &[Signature]) -> Result<Vec<bool>, Error> {
+        self.inner
+            .get_signature_statuses(signatures)
+            .await
+            .map(|response| {
+                response
+                    .value
+                    .into_iter()
+                    .map(|status| status.is_some())
+                    .collect()
+            })
+    }


get_signature_statuses has two properties that bite the audit:

256-signature cap. The RPC rejects more than 256 signatures per call. In steady state a single finalized slot carries few of our txs, but the catch-up range (unfinalized_signatures(after, slot) where after is the persisted watermark after any downtime, or after accumulated audit-RPC failures) can easily exceed 256 and the whole call fails. Chunk signatures into batches of ≤256.

searchTransactionHistory defaults to false, so this only consults the recent-status cache (the "~150 slots" in the doc comment). A transaction that landed and finalized fine but has aged out of that cache returns None here — i.e. false does not reliably mean "rolled back"; it also means "too old to still be cached". See the companion note in decoder.rs on why that turns into false-positive reverts.

Fix this →

256-signature cap. [...]

Suggestion here makes sense.

2. searchTransactionHistory defaults to false [...]

I think #4877 is a more adequate mechanism, as mentioned in the other comment.

Fixed in 3d7b809

claude · 2026-09-07T17:49:26Z

+        let signatures = self.persistence.unfinalized_signatures(after, slot).await?;
+        let vanished = if signatures.is_empty() {
+            Vec::new()
+        } else {
+            match self.rpc.known_signatures(&signatures).await {
+                Ok(known) => signatures
+                    .into_iter()
+                    .zip(known)
+                    .filter_map(|(signature, known)| (!known).then_some(signature))
+                    .collect(),
+                Err(err) => {
+                    tracing::warn!(?err, "signature audit failed, finalization delayed");
+                    *finalized_through = Some(after);
+                    return Ok(());
+                }
+            }
+        };
+        if vanished.is_empty() {
+            self.persistence.write_finalized_slot(slot).await?;
+        } else {
+            tracing::warn!(?vanished, "rolled-back transactions reverted");
+            self.persistence.finalize_through(slot, &vanished).await?;


Risk of mass false-positive reverts when the audit range is large.

after is unbounded below: on the first finalized tick after boot it's the persisted watermark (finalized_slot().unwrap_or(slot)), and after an outage the live geyser stream jumps straight to the current finalized slot, so the first (after, slot] range can span thousands of slots. The same happens whenever audit-RPC failures accumulate and delay finalization.

known_signatures only checks the recent-status cache (~150 slots). Any transaction in that range that finalized normally but has already aged out of the cache comes back as not known → vanished → reverted. So an indexer that catches up over an old range would delete healthy settlements/trades and mark healthy orders is_reorged, exactly the data it just indexed.

The PR description's claim that "an RPC failure … never reverts on its own" holds for a single delayed tick, but not for a range that has grown past the cache window. Consider bounding the audit to the cache window (e.g. don't revert on slot - created_in_slot older than the cache; advance the watermark without auditing those), or pass searchTransactionHistory: true for the aged tail.

Fix this →

I believe #4877 is the answer for such long missing slot ranges, so the audit introduced in this PR should not be authoritative over them.

With the history search from 3d7b809, a cache miss no longer reads as vanished, so a large catch-up range cannot mass-revert healthy rows. I see the split as: #4877 recovers data we never saw, the audit removes data the chain dropped, and the backfill cannot do the latter.

tilacog · 2026-09-09T17:55:22Z

+                response
+                    .value
+                    .into_iter()
+                    .map(|status| status.is_some())


Might be a corner case, but worth noting that the TransactionStatus object contained in status might not be Finalized, and the code here assumes so.

IMO the risk is near zero but we might be marking Confirmed slots as being finalized.

The audit only asks whether the transaction exists, finality comes from the stream's finalized status. A confirmed-level answer just means the audit node lags our stream a bit, the transaction itself is fine.

tilacog · 2026-09-09T18:01:42Z

+        if slot <= after {
+            *finalized_through = Some(after);
+            return Ok(());
+        }


Nit: Should this branch also call persistence.write_finalized_slot before returning?

The write would be a no-op there. When that branch hits, after came from the persisted watermark (or the in-memory mirror, which only moves after a successful write), so the DB already stores slot or something newer. The one exception is a fresh DB, where after is just the current slot used as a starting point. Persisting that baseline would work too, but the next boot derives the same value again, so it buys nothing.

tilacog · 2026-09-09T18:14:10Z

+UNION
+SELECT created_by_tx FROM solana.order_pda
+    WHERE created_in_slot > $1 AND created_in_slot <= $2 AND created_by_tx IS NOT NULL
+ORDER BY 1


Is there a reason to order by the transaction signature here?

It is now only required for tests, and I forgot to drop it. Done now.

tilacog · 2026-09-09T18:26:32Z

+UPDATE solana.order_pda AS pda
+SET amount_withdrawn = pda.amount_withdrawn - deltas.sell,
+    amount_received  = pda.amount_received - deltas.buy
+FROM (
+    SELECT order_uid, SUM(sell_amount) AS sell, SUM(buy_amount) AS buy
+    FROM solana.trades WHERE tx_signature = $1 GROUP BY order_uid
+) AS deltas
+WHERE pda.order_uid = deltas.order_uid


Might be a long shot and require further discussion beyond this PR, but what if pda.amount_withdrawn and pda.amount_received were a view over solana.trades instead of maintained columns?
Something like this:

CREATE VIEW solana.order_pda_sums AS SELECT pda.order_uid, COALESCE(SUM(t.sell_amount), 0) AS amount_withdrawn, COALESCE(SUM(t.buy_amount), 0) AS amount_received FROM solana.order_pda pda LEFT JOIN solana.trades t USING (order_uid) GROUP BY pda.order_uid;

I think this could simplify some of the bookkeeping the indexer does, both on regular ops and during this reorg step... it would just come down to which rows exist (or don't) in solana.trades. Revert could skip the subtraction UPDATE entirely and only do the DELETEs.

Not sure about the read cost if something needs these sums for many orders at once, though.

The delete-only revert is a nice property, but I would keep the columns for now. They mirror what the PDA stores on chain, and not every future change to those amounts has to come with a trade row. A view computed from trades would silently diverge the moment that happens. There is also the read side: solvable orders needs these sums for every open order, and I would rather not run the join plus aggregate on that path. If trades end up being the only writer we can switch to a view in a follow-up.

tilacog · 2026-09-09T18:39:20Z

+                "DELETE FROM solana.trades WHERE tx_signature = $1",
+                "DELETE FROM solana.settlements WHERE tx_signature = $1",
+                "DELETE FROM solana.dead_letter WHERE tx_signature = $1",
+                // Orders are marked, not deleted: the rows are the audit
+                // trail, and the stream re-delivering a re-landed creation
+                // clears the flag.
+                "UPDATE solana.order_pda SET is_reorged = true WHERE created_by_tx = $1",


Should we keep reorg'ed settlements and trades around too? For the same reasons we keep orders?

An order row is user intent, a trade row is a record of something that happened on chain. After a reorg, the intent still stands, only the creation transaction is gone, and it usually lands again on the canonical fork. Then the order must return to solvable, so we keep the row and just flag it, the re-delivered creation clears the flag. Deleting also keeps the readers honest, otherwise every consumer of those tables would need an is_reorged filter. Same split as on EVM: events are hard-deleted on reorg, ethflow orders get the is_reorged mark.

…-rollback-cascade # Conflicts: # crates/cow-solana-rpc/src/lib.rs # crates/solana-indexer/src/indexer/decoder.rs # crates/solana-indexer/src/indexer/decoder/tests.rs # crates/solana-indexer/src/persistence.rs

squadgazzz added 4 commits September 3, 2026 11:44

Advance the finalized watermark and chain tip from slot statuses

56bbde4

Fix stale component docs

61b1fa3

Drop the chain tip table, the watermark is the freshness signal

1bbfdab

Shrink the slot routing comment

be99aa8

squadgazzz marked this pull request as ready for review September 7, 2026 17:45

squadgazzz requested a review from a team as a code owner September 7, 2026 17:45

squadgazzz requested a review from tilacog September 7, 2026 17:45

claude Bot reviewed Sep 7, 2026

View reviewed changes

squadgazzz mentioned this pull request Sep 7, 2026

solana-indexer: backfill from RPC history when the replay window is exceeded #4877

Open

tilacog approved these changes Sep 9, 2026

View reviewed changes

tilacog reviewed Sep 9, 2026

View reviewed changes

squadgazzz added 7 commits September 11, 2026 07:23

Flush on confirmed slot statuses and drop the hold-back

7bad48c

Audit newly finalized slots and revert rolled-back transactions

5be0190

Mark reorged orders instead of deleting them, like EVM

ac363cf

Move the reorged mark to the order_pda sidecar

8e7c905

Keep creation provenance on the order_pda sidecar

e2ec78a

Deduplicate the migration comment

64cf65c

Simplify the cascade test doc

70fd3cd

squadgazzz force-pushed the solana-indexer/be-203-rollback-cascade branch from ebe21a7 to 70fd3cd Compare September 11, 2026 07:35

squadgazzz added 2 commits September 11, 2026 07:55

Audit signatures through ledger history in capped batches

3d7b809

Drop the test-only ordering from the audit query

9352df7

Base automatically changed from solana-indexer/be-203-finalized-watermark to main September 11, 2026 16:00

Merge remote-tracking branch 'origin/main' into solana-indexer/be-203…

a101938

…-rollback-cascade # Conflicts: # crates/cow-solana-rpc/src/lib.rs # crates/solana-indexer/src/indexer/decoder.rs # crates/solana-indexer/src/indexer/decoder/tests.rs # crates/solana-indexer/src/persistence.rs

Uh oh!

Conversation

squadgazzz commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

How to test

Related issues

Uh oh!

linear-code Bot commented Sep 7, 2026

Uh oh!

claude Bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review

Uh oh!

claude Bot Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

tilacog Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

claude Bot Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

squadgazzz commented Sep 7, 2026 •

edited

Loading

claude Bot commented Sep 7, 2026 •

edited

Loading

tilacog Sep 9, 2026 •

edited

Loading