solana-indexer: backfill from RPC history when the replay window is exceeded - #4877
solana-indexer: backfill from RPC history when the replay window is exceeded#4877squadgazzz wants to merge 29 commits into
Conversation
b61edc0 to
8a1a37c
Compare
ebe21a7 to
70fd3cd
Compare
8a1a37c to
87b9097
Compare
87b9097 to
301a216
Compare
|
Claude finished @squadgazzz's task in 3m 31s —— View job ReviewSolid, well-documented change — the decode/flush reuse via a stream-shaped
Non-blocking: the |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| pub(crate) async fn backfill(&self) -> Result<(), PersistenceError> { | ||
| let result = self.backfill_inner().await; | ||
| if let Err(err) = &result { | ||
| tracing::error!(?err, "backfill failed, recording the gap as lost"); | ||
| match ( | ||
| self.persistence.last_indexed_slot().await, | ||
| self.rpc.slot().await, | ||
| ) { | ||
| (Ok(Some(from)), Ok(tip)) => { | ||
| if let Err(err) = self | ||
| .persistence | ||
| .record_lost_range(from, Slot(tip), "backfill failed") | ||
| .await | ||
| { | ||
| tracing::error!(?err, "failed to record the lost range"); | ||
| } | ||
| } | ||
| (watermark, tip) => { | ||
| tracing::error!(?watermark, ?tip, "failed to bound the lost range"); | ||
| } | ||
| } | ||
| } | ||
| result | ||
| } |
There was a problem hiding this comment.
Any error out of backfill_inner records the gap as lost and (via run.rs) drops to Resume::LiveTip, permanently skipping the range. But backfill_inner fails on the first transient RPC error — a single signatures_for_address/transaction timeout or 429 aborts the whole scan (? on lines 97, 137, 68). So a momentary RPC blip converts a fully recoverable gap into an "unrecoverable" lost_slot_ranges row and jumps past it on the live tip.
The PR intends lost_slot_ranges for ranges "RPC cannot serve" (pruned history), but the code doesn't distinguish that from a retryable failure. Consider retrying the backfill a bounded number of times (the loop is already paced by STREAM_RETRY) before declaring the gap lost, or only recording as lost when the RPC error is genuinely non-retryable. Otherwise transient failures silently drop protocol transactions.
| let page_len = page.len(); | ||
| before = page.last().map(|(signature, _)| *signature); | ||
| let fresh = entries.len(); | ||
| entries.extend( | ||
| page.into_iter() | ||
| .take_while(|(_, slot)| Slot(*slot) > watermark) | ||
| .map(|(signature, slot)| (Slot(slot), signature)), | ||
| ); | ||
| let reached_watermark = entries.len() - fresh < page_len; | ||
| if reached_watermark || page_len < SolanaRPC::SIGNATURES_PAGE { | ||
| return Ok(entries); |
There was a problem hiding this comment.
page_len is the length after signatures_for_address has dropped any unparsable signatures. The termination check page_len < SolanaRPC::SIGNATURES_PAGE treats a short page as "history exhausted" — but a full 1000-signature page with even one malformed entry filters down to 999, ending the walk early and silently missing older in-window history below it.
To key the "last page" decision on the raw page size rather than the filtered count, have signatures_for_address report the unfiltered length (or return the raw count alongside the parsed entries). Rare in practice, but it produces a silent gap that no lost_slot_ranges row records.
| (watermark, tip) => { | ||
| tracing::error!(?watermark, ?tip, "failed to bound the lost range"); | ||
| } | ||
| } |
There was a problem hiding this comment.
When the bounding calls (last_indexed_slot/rpc.slot) themselves fail, the gap is logged but not recorded to lost_slot_ranges, yet backfill() still returns Err, so run.rs proceeds to Resume::LiveTip and advances the watermark past the gap on the next confirmed slot. Net result: an unrecorded, silently-skipped range — the exact "silently accepted" outcome the table exists to prevent. At minimum the watermark (from) is already known here from backfill_inner's own read; consider persisting a lost range with whatever bound is available rather than dropping the record entirely.
| /// One page of an address's transaction signatures, newest first, | ||
| /// starting below `before` when given. The node serves deep history, so | ||
| /// repeated calls walk arbitrarily far back. | ||
| pub async fn signatures_for_address( | ||
| &self, | ||
| address: &Pubkey, | ||
| before: Option<Signature>, | ||
| ) -> Result<Vec<(Signature, u64)>, Error> { | ||
| let config = solana_rpc_client::rpc_client::GetConfirmedSignaturesForAddress2Config { | ||
| before, | ||
| limit: Some(Self::SIGNATURES_PAGE), | ||
| ..Default::default() | ||
| }; | ||
| let page = self | ||
| .inner | ||
| .get_signatures_for_address_with_config(address, config) | ||
| .await?; | ||
| Ok(page | ||
| .into_iter() | ||
| .filter_map(|status| { | ||
| let signature = status.signature.parse().ok()?; | ||
| Some((signature, status.slot)) | ||
| }) | ||
| .collect()) | ||
| } |
There was a problem hiding this comment.
Just an idea: should this method handle/abstract pagination internally so callers don't have to?
I see signatures_since does it already, so I don't have a strong preference.
(maybe worth revisiting once more consumers need it.)
There was a problem hiding this comment.
Kept it page-level on purpose, since the stop condition (the watermark) lives in the caller, and internalizing the loop would either fetch history we do not need or grow a predicate callback. It would make sense to revisit when a second consumer shows up. Introduced a new type, tho: SignaturesPage.
| let backfiller = { | ||
| let rpc = SolanaRPC::new_with_timeout_and_commitment( | ||
| &config.rpc.endpoint, | ||
| config.rpc.request_timeout, | ||
| CommitmentConfig::confirmed(), | ||
| ); | ||
| let (_closed, rx) = mpsc::channel(1); | ||
| Decoder::new( | ||
| persistence.clone(), | ||
| rpc, | ||
| rx, | ||
| settlement_program, | ||
| solflow_program, | ||
| ) | ||
| }; |
There was a problem hiding this comment.
nit: should this be encapsulated as Decoder::backfiller(...)?
smth like
fn backfiller(config: &Config, persistence: Postgres) -> Self There was a problem hiding this comment.
Moved to Decoder::rpc_driven.
| if result.is_err() | ||
| && let (Ok(Some(from)), Ok(tip)) = ( | ||
| self.persistence.last_indexed_slot().await, | ||
| self.rpc.slot().await, | ||
| ) | ||
| && let Err(err) = self | ||
| .persistence | ||
| .record_lost_range(from, Slot(tip), "backfill failed") | ||
| .await | ||
| { | ||
| tracing::error!(?err, "failed to record the lost range"); | ||
| } |
There was a problem hiding this comment.
Should this be considered a terminal condition / fatal error?
There was a problem hiding this comment.
Crashlooping will likely get to this point faster, but we will be warned via alerting right away. Added a panic.
…-rollback-cascade # Conflicts: # crates/cow-solana-rpc/src/lib.rs # crates/solana-indexer/src/indexer/decoder.rs # crates/solana-indexer/src/indexer/decoder/tests.rs # crates/solana-indexer/src/persistence.rs
…exer/be-204-gap-scan # Conflicts: # crates/cow-solana-rpc/src/lib.rs # crates/solana-indexer/Cargo.toml # crates/solana-indexer/src/indexer/decoder.rs # crates/solana-indexer/src/indexer/decoder/tests.rs
|
@claude review again please, i dropped the lost ranges logic. |
|
Claude finished @squadgazzz's task in 3m 8s —— View job ReviewThe lost-ranges logic is cleanly removed, and the previous round's three findings are all resolved:
The panic-over-skip invariant is sound, and reusing the stream's I left one inline finding:
Non-blocking: the |
| let config = solana_rpc_client::rpc_client::GetConfirmedSignaturesForAddress2Config { | ||
| before, | ||
| limit: Some(Self::SIGNATURES_PAGE), | ||
| ..Default::default() | ||
| }; |
There was a problem hiding this comment.
Commitment mismatch can still skip a small window silently. This config leaves commitment unset, so getSignaturesForAddress runs at the node's default (finalized). But the backfill's tip comes from rpc.slot(), which uses the client's confirmed commitment, and transaction() above also fetches at confirmed.
So backfill_inner scans signatures only up to the finalized tip, then advances the watermark to the confirmed tip (write_last_indexed_slot(tip)). Any tracked settlement/order transaction in the (finalized_tip, confirmed_tip] window — up to ~32 slots — is confirmed but not yet listed by getSignaturesForAddress, so it is neither backfilled nor re-streamed (the resume starts at watermark + 1, past it). That's a silently-skipped gap, the exact outcome this PR sets out to prevent.
Pinning the scan to confirmed keeps the signature listing, the tip, and the transaction fetch on the same commitment:
| let config = solana_rpc_client::rpc_client::GetConfirmedSignaturesForAddress2Config { | |
| before, | |
| limit: Some(Self::SIGNATURES_PAGE), | |
| ..Default::default() | |
| }; | |
| let config = solana_rpc_client::rpc_client::GetConfirmedSignaturesForAddress2Config { | |
| before, | |
| limit: Some(Self::SIGNATURES_PAGE), | |
| commitment: Some(CommitmentConfig::confirmed()), | |
| ..Default::default() | |
| }; |
2 participants
Description
The stream provider replays only ~150 slots, so a restart after real downtime resubscribes from the live tip and silently skips everything in between. This PR fills the gap from RPC instead: walk the program's signature history back to the stored in the DB watermark, re-fetch each transaction, and push it through the same decode and flush path streamed transactions take.
The cost scales with the program's transactions, not with chain length, since empty slots cost nothing. If RPC itself cannot serve the gap, the backfill retries and then panics rather than silently accepting a hole: the watermark never passes an unscanned slot, so the restarted process reruns the recovery until RPC serves it, and the outage shows up as a crashloop in alerting.
Same invariants as the EVM event handler's
getLogscatch-up: the database watermark is the only resume truth, and the backfill commits before the live stream attaches.First half of BE-204, stacked on #4875. Dead-letter replay follows separately.
Changes
getTransactionpayloads, pinned by an equivalence test against the streamed formcow-solana-rpcgainssignatures_for_addressandtransactionHow to test
New unit tests and ignored postgres tests, including an end-to-end backfill through a mock RPC.
Related issues
BE-204