Skip to content

journal: name why a cached timeout vote was dropped - #267

Merged
On1x merged 2 commits into
masterfrom
fix/journal-replay-reason-counts
Nov 9, 2026
Merged

On1x merged 2 commits into
masterfrom
fix/journal-replay-reason-counts

Conversation

@On1x

@On1x On1x commented Nov 9, 2026

Copy link
Copy Markdown
Member

Problem

The start-up line Timeout vote journal: restored N of M cached votes (<k> stale or invalid) merged two cases with opposite meanings:

  • aged — the vote is older than the round window (MAX_VOTE_ROUND_BEHIND = 3600 / block_interval() = 1 h on a 1-second block). Normal: the journal is compacted only on replay, so the file keeps hours of history.
  • rejected — the caller refused the vote (key/window mismatch). A defect.

Measured 09.11.2026 on the four production nodes after the 08.11 redeploy: 215 of 1862 votes restored (11.5 %), spread 180× between nodes (0.2 %…36.5 %). That spread is a function of how long the file had been accumulating, and the single total could not say whether a node was ageing votes or rejecting them.

Change

timeout_vote_journal::replay_result gains aged (subset of dropped, cut by min_round before the signature check), and the start-up line splits the total:

Timeout vote journal: restored 83 of 588 cached votes from … (505 stale or invalid: 500 older than the round window, 5 rejected)

Tests:

  • test_timeout_vote_journal.py asserts aged == 0 for a rejected vote and aged == 1 for a window drop, so the split is measured in both directions.
  • tools/qc-bench/tc-stale-replay.py (viz-tools 423d830) now tolerates the longer tail; without it the acceptance harness would have gone blind on every new log.

Verification

  • tests/chain/test_timeout_vote_journal.py — green, incl. --selftest.
  • Single-TU -fsyntax-only of libraries/chain/database.cpp on host (nice/ionice), 0 errors.
  • Image build and the full suite run via this PR's CI.

Live proof of the split appears in the first restart line after this change is deployed; until then "this is the window, not a defect" rests on the code plus the consistency of the numbers.

🤖 Generated with Claude Code

On1x added 2 commits November 9, 2026 03:31
'stale or invalid' merged two cases with opposite meanings: a vote older
than the round window (normal -- the journal keeps hours of history and is
compacted only on replay) and a vote the caller rejected (a defect). A
restart on a healthy chain therefore printed a number nobody could read;
09.11.2026 the four production nodes restored 215 of 1862 votes and the
single total could not say which case that was.

replay() now counts the window drops separately (replay_result::aged) and
the start-up line splits the total. Tests cover both halves: a rejected
vote leaves aged at 0, a window drop sets it to 1. Syntax-checked as a
single TU on host; image build follows on the PR.
A line cut short by a crash mid-write is also unparsable, and counting it
as 'rejected' would point an incident reader at a key mismatch that does
not exist. Say 'rejected or unparsable' and spell the crash case out in
the comment.
@On1x
On1x merged commit 54b42dc into master Nov 9, 2026
8 checks passed
@On1x
On1x deleted the fix/journal-replay-reason-counts branch November 9, 2026 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant