Skip to content

Commit 373b602

Browse files
authored
Merge pull request #93 from VIZ-Blockchain/fork-resolution
fix(witness): resolve infinite fork collision deferral and stuck head…
2 parents e21b8a1 + 35c4256 commit 373b602

23 files changed

Lines changed: 2012 additions & 483 deletions

File tree

‎.qoder/docs/shared-memory.md‎

Lines changed: 114 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,8 @@ The VIZ node stores all blockchain state in a memory-mapped file (`shared_memory
4242
| `libraries/chain/database.cpp` | Chain-level `open()`, `_resize()`, `check_free_memory()`, `push_block()`, `_generate_block()` |
4343
| `libraries/chain/include/graphene/chain/database.hpp` | Chain database class declaration, resize/memory parameters |
4444
| `plugins/chain/plugin.cpp` | Config option definitions, initialization, snapshot loading |
45+
| `plugins/witness/witness.cpp` | Lockless reads in `maybe_produce_block()` and `is_witness_scheduled_soon()`, guarded by `operation_guard` |
46+
| `plugins/p2p/p2p_plugin.cpp` | Lockless reads in block post-validation (`get_witness_key()`), guarded by `operation_guard` |
4547

4648
---
4749

@@ -204,21 +206,63 @@ The resize is implemented in `chainbase::database::resize()` (`thirdparty/chainb
204206
index_type->add_index(*this) ← Rebuild index pointers from new mapping
205207
```
206208

207-
### Critical Race Condition
209+
### Resize Barrier
208210

209-
**The resize operation is inherently dangerous in a multi-threaded environment.** When `_segment.reset()` destroys the old mapping:
211+
Because `_segment.reset()` invalidates **all** pointers/references into shared memory, the resize must ensure that **no thread** — whether holding a lock or reading locklessly — has any live reference into the mapped segment. A simple write lock is insufficient because several code paths read chainbase indices without holding any lock ("lockless reads").
210212

211-
- All pointers/references into the old segment become **dangling**
212-
- `boost::shared_mutex` does **not** protect against this — a read lock only prevents concurrent writes, but the resize **is** the write
213-
- If any thread holds a read lock and has cached a reference/pointer to an object in the old mapping, that reference becomes invalid after `_segment.reset()`
213+
The **resize barrier** (`chainbase::database`) solves this with an atomic operation counter, a flag, and a condition variable:
214214

215-
The `resize()` function checks `_undo_session_count` but does **not** check for active read locks. The write lock held during `push_block()` prevents other threads from acquiring **new** read locks during the resize, but threads that already hold read locks can still be accessing the old mapping.
215+
```
216+
┌──────────────────────────────────────────────────────────────────┐
217+
│ Resize Barrier Protocol │
218+
│ │
219+
│ Normal operation: │
220+
│ enter_operation() ── wait while _resize_in_progress │
221+
│ ── increment _active_operations │
222+
│ ... access shared memory ... │
223+
│ exit_operation() ── decrement _active_operations │
224+
│ ── notify resize thread if last op │
225+
│ │
226+
│ Resize: │
227+
│ begin_resize_barrier() ── set _resize_in_progress = true │
228+
│ ── wait until _active_operations == 0 │
229+
│ ... _segment.reset() + open() + rebuild indices ... │
230+
│ end_resize_barrier() ── set _resize_in_progress = false │
231+
│ ── notify all waiting threads │
232+
└──────────────────────────────────────────────────────────────────┘
233+
```
234+
235+
**Participation points:**
236+
237+
| Code Path | How It Participates |
238+
|-----------|--------------------|
239+
| `with_read_lock()` / `with_write_lock()` | `operation_guard` acquired automatically inside the lock wrapper before the `boost::shared_mutex` lock |
240+
| `_generate_block()` lockless reads (pre-write-lock) | Explicit scoped `operation_guard` around `get_slot_at_time()`, `get_scheduled_witness()`, `get_witness()`, `find_account()` |
241+
| `_generate_block()` lockless reads (post-write-lock) | Second `operation_guard` (`op_guard2`) around `get_dynamic_global_properties()`, `head_block_id()`, `get_witness()`, `get_hardfork_property_object()`; released via `release()` before `push_block()` |
242+
| Witness plugin `maybe_produce_block()` | Explicit `operation_guard` around `get_slot_at_time()`, `get_scheduled_witness()`, index lookups; released via `release()` before `generate_block()` |
243+
| Witness plugin `is_witness_scheduled_soon()` | Explicit `operation_guard` around `get_slot_at_time()`, `get_scheduled_witness()`, index lookups; released via `release()` before return |
244+
| P2P plugin block post-validation | Explicit `operation_guard` around `get_witness_key()` calls; released via `release()` before `apply_block_post_validation()` |
245+
246+
**Key classes:**
247+
248+
- `chainbase::database::operation_guard` — RAII guard that calls `enter_operation()` on construction and `exit_operation()` on destruction. Supports early release via `release()`. Move-only (non-copyable).
249+
- `chainbase::database::make_operation_guard()` — Factory method returning an `operation_guard`.
250+
- `chainbase::database::begin_resize_barrier()` / `end_resize_barrier()` — Called by `apply_pending_resize()` to establish exclusive access for the resize.
251+
252+
**Immediate resize (reindex) does NOT use the barrier.** During reindex, the caller already holds an exclusive write lock and no API threads are running. Using the barrier would deadlock because the write lock itself holds an operation guard.
216253

217-
**This is the root cause of shared memory corruption** when resize occurs while API threads are reading.
254+
### Historical Context: Pre-Barrier Race Condition
255+
256+
Before the resize barrier was added, the resize used `with_strong_write_lock()` which only blocked threads holding or waiting for a `boost::shared_mutex` lock. This left lockless reads unprotected:
257+
258+
- `boost::shared_mutex` does **not** protect against segment destruction — a read lock only prevents concurrent writes, but the resize **is** the write
259+
- Threads performing lockless reads (witness plugin, `_generate_block()`) could hold stale pointers into the old mapping after `_segment.reset()`
260+
261+
This was the root cause of shared memory corruption symptoms like `CRITICAL: Witness X account object MISSING from database!`.
218262

219263
### Corruption Symptoms
220264

221-
Typical corruption indicators:
265+
Typical corruption indicators (should not occur with the resize barrier in place, but listed for historical reference and diagnostics):
222266

223267
- `CRITICAL: Witness X account object MISSING from database!` — account index entry not found despite `account_index_size` showing entries exist
224268
- `Could not modify object, most likely a uniqueness constraint was violated` — internal index pointers corrupted, uniqueness check fails
@@ -233,7 +277,7 @@ inc-shared-file-size = 500M
233277
min-free-shared-file-size = 1000M ← THRESHOLD > INCREMENT!
234278
```
235279

236-
After one resize (500M → 1000M), free space is still below 1000M, triggering **another resize on the next check**, causing cascading resizes that dramatically increase the chance of corruption.
280+
After one resize (500M → 1000M), free space is still below 1000M, triggering **another resize on the next check**, causing cascading resizes. While the resize barrier prevents corruption, frequent resizes still cause latency spikes as all operations are paused during each resize.
237281

238282
---
239283

@@ -255,25 +299,78 @@ Most `database_api` methods use `with_weak_read_lock()`:
255299
```cpp
256300
// Example: get_accounts
257301
auto result = with_weak_read_lock([&]() {
258-
// Read from chainbase indices
302+
// Read from chainbase indices — operation_guard acquired automatically
259303
return accounts;
260304
});
261305
```
262306

263-
If the read lock cannot be acquired within `read-wait-micro × max-read-wait-retries`, the API returns error: `"Unable to acquire READ lock"`.
307+
If the read lock cannot be acquired within `read-wait-micro × max-read-wait-retries`, the API returns error: `"Unable to acquire READ lock"`. If a resize barrier is active, the `operation_guard` inside `with_weak_read_lock()` will block until the resize completes before attempting the lock.
264308

265309
### Write Path (Block Application)
266310

267311
All state modifications go through `with_strong_write_lock()`:
268312

269313
```cpp
270314
// push_block
271-
with_strong_write_lock([&]() {
315+
apply_pending_resize(); // uses resize barrier (begin/end)
316+
with_strong_write_lock([&]() { // operation_guard acquired inside
272317
_push_block(new_block, skip);
273318
check_free_memory(false, new_block.block_num());
274319
});
275320
```
276321
322+
### Lockless Reads (Witness Plugin, Block Generation, P2P)
323+
324+
Some code paths read from chainbase indices **without** holding a `boost::shared_mutex` lock. These must explicitly use an `operation_guard` to participate in the resize barrier:
325+
326+
```cpp
327+
// witness.cpp — maybe_produce_block()
328+
auto op_guard = db.make_operation_guard();
329+
uint32_t slot = db.get_slot_at_time(now); // lockless read
330+
string scheduled = db.get_scheduled_witness(slot); // lockless read
331+
// ... more lockless reads ...
332+
op_guard.release(); // release before generate_block() which has its own guard
333+
```
334+
335+
```cpp
336+
// witness.cpp — is_witness_scheduled_soon()
337+
auto op_guard = db.make_operation_guard();
338+
uint32_t slot = db.get_slot_at_time(now);
339+
string scheduled = db.get_scheduled_witness(s);
340+
// ...
341+
op_guard.release(); // release before returning true
342+
```
343+
344+
```cpp
345+
// database.cpp — _generate_block() (pre-write-lock reads)
346+
{
347+
auto op_guard = make_operation_guard();
348+
uint32_t slot_num = get_slot_at_time(when);
349+
string scheduled_witness = get_scheduled_witness(slot_num);
350+
const auto& witness_obj = get_witness(witness_owner);
351+
const auto* witness_acct = find_account(witness_owner);
352+
} // op_guard released before with_strong_write_lock
353+
```
354+
355+
```cpp
356+
// database.cpp — _generate_block() (post-write-lock reads)
357+
auto op_guard2 = make_operation_guard();
358+
auto maximum_block_size = get_dynamic_global_properties().maximum_block_size;
359+
// ... with_strong_write_lock, then post-lock reads ...
360+
pending_block.previous = head_block_id();
361+
const auto& witness = get_witness(witness_owner);
362+
const auto& hfp = get_hardfork_property_object();
363+
const auto& dgp_block = get_dynamic_global_properties();
364+
op_guard2.release(); // release before push_block()
365+
```
366+
367+
```cpp
368+
// p2p_plugin.cpp — block post-validation
369+
auto op_guard = chain.db().make_operation_guard();
370+
fc::ecc::public_key w_signing_key = chain.db().get_witness_key(account);
371+
op_guard.release(); // release before apply_block_post_validation()
372+
```
373+
277374
### single-write-thread Mode
278375

279376
When `single-write-thread = true`, all `push_block()` and `push_transaction()` calls are serialized through a single `fc::async()` queue. This:
@@ -393,9 +490,10 @@ FATAL write lock timeout!!!
393490

394491
## Safety Rules
395492

396-
1. **`min-free-shared-file-size` must be less than `inc-shared-file-size`** — otherwise cascading resizes occur, dramatically increasing corruption risk
397-
2. **Pre-allocate generously** — set `shared-file-size` large enough that resize is rare. Resizing is the most dangerous operation.
398-
3. **Use `single-write-thread = true`** in production — prevents write lock contention and reduces the window for resize race conditions
399-
4. **Avoid resize during witness production** — a witness node should have enough pre-allocated memory that resize never triggers during block generation
493+
1. **`min-free-shared-file-size` must be less than `inc-shared-file-size`** — otherwise cascading resizes occur, causing frequent operation pauses
494+
2. **Pre-allocate generously** — set `shared-file-size` large enough that resize is rare. Each resize pauses all operations while the segment is remapped.
495+
3. **Use `single-write-thread = true`** in production — prevents write lock contention
496+
4. **Avoid resize during witness production** — a witness node should have enough pre-allocated memory that resize never triggers during block generation. The resize barrier guarantees safety but introduces latency.
400497
5. **After corruption, always replay** — there is no safe way to repair a corrupted `shared_memory.bin`. Use `--replay-blockchain` or `--snapshot` to rebuild state from block_log/dlt_block_log.
401498
6. **Backup before config changes** — changing `shared-file-size` to a larger value triggers `grow()` on next startup, which is safe. Reducing it has no effect (file doesn't shrink).
499+
7. **Any new lockless read path must use `operation_guard`** — if you add code that reads from chainbase indices without `with_read_lock()`/`with_write_lock()`, wrap it with `make_operation_guard()` to participate in the resize barrier. Failing to do so can cause stale pointer access during resize.

0 commit comments

Comments
 (0)