You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 373b602
Browse filesBrowse the repository at this point in the historyBrowse files
|`plugins/witness/witness.cpp`| Lockless reads in `maybe_produce_block()` and `is_witness_scheduled_soon()`, guarded by `operation_guard`|
46
+
|`plugins/p2p/p2p_plugin.cpp`| Lockless reads in block post-validation (`get_witness_key()`), guarded by `operation_guard`|
45
47
46
48
---
47
49
@@ -204,21 +206,63 @@ The resize is implemented in `chainbase::database::resize()` (`thirdparty/chainb
204
206
index_type->add_index(*this) ← Rebuild index pointers from new mapping
205
207
```
206
208
207
-
### Critical Race Condition
209
+
### Resize Barrier
208
210
209
-
**The resize operation is inherently dangerous in a multi-threaded environment.** When `_segment.reset()` destroys the old mapping:
211
+
Because `_segment.reset()` invalidates **all** pointers/references into shared memory, the resize must ensure that **no thread** — whether holding a lock or reading locklessly — has any live reference into the mapped segment. A simple write lock is insufficient because several code paths read chainbase indices without holding any lock ("lockless reads").
210
212
211
-
- All pointers/references into the old segment become **dangling**
212
-
-`boost::shared_mutex` does **not** protect against this — a read lock only prevents concurrent writes, but the resize **is** the write
213
-
- If any thread holds a read lock and has cached a reference/pointer to an object in the old mapping, that reference becomes invalid after `_segment.reset()`
213
+
The **resize barrier** (`chainbase::database`) solves this with an atomic operation counter, a flag, and a condition variable:
214
214
215
-
The `resize()` function checks `_undo_session_count` but does **not** check for active read locks. The write lock held during `push_block()` prevents other threads from acquiring **new** read locks during the resize, but threads that already hold read locks can still be accessing the old mapping.
|`_generate_block()` lockless reads (post-write-lock) | Second `operation_guard` (`op_guard2`) around `get_dynamic_global_properties()`, `head_block_id()`, `get_witness()`, `get_hardfork_property_object()`; released via `release()` before `push_block()`|
242
+
| Witness plugin `maybe_produce_block()`| Explicit `operation_guard` around `get_slot_at_time()`, `get_scheduled_witness()`, index lookups; released via `release()` before `generate_block()`|
243
+
| Witness plugin `is_witness_scheduled_soon()`| Explicit `operation_guard` around `get_slot_at_time()`, `get_scheduled_witness()`, index lookups; released via `release()` before return |
244
+
| P2P plugin block post-validation | Explicit `operation_guard` around `get_witness_key()` calls; released via `release()` before `apply_block_post_validation()`|
245
+
246
+
**Key classes:**
247
+
248
+
-`chainbase::database::operation_guard` — RAII guard that calls `enter_operation()` on construction and `exit_operation()` on destruction. Supports early release via `release()`. Move-only (non-copyable).
249
+
-`chainbase::database::make_operation_guard()` — Factory method returning an `operation_guard`.
250
+
-`chainbase::database::begin_resize_barrier()` / `end_resize_barrier()` — Called by `apply_pending_resize()` to establish exclusive access for the resize.
251
+
252
+
**Immediate resize (reindex) does NOT use the barrier.** During reindex, the caller already holds an exclusive write lock and no API threads are running. Using the barrier would deadlock because the write lock itself holds an operation guard.
216
253
217
-
**This is the root cause of shared memory corruption** when resize occurs while API threads are reading.
Before the resize barrier was added, the resize used `with_strong_write_lock()` which only blocked threads holding or waiting for a `boost::shared_mutex` lock. This left lockless reads unprotected:
257
+
258
+
-`boost::shared_mutex` does **not** protect against segment destruction — a read lock only prevents concurrent writes, but the resize **is** the write
259
+
- Threads performing lockless reads (witness plugin, `_generate_block()`) could hold stale pointers into the old mapping after `_segment.reset()`
260
+
261
+
This was the root cause of shared memory corruption symptoms like `CRITICAL: Witness X account object MISSING from database!`.
218
262
219
263
### Corruption Symptoms
220
264
221
-
Typical corruption indicators:
265
+
Typical corruption indicators (should not occur with the resize barrier in place, but listed for historical reference and diagnostics):
222
266
223
267
-`CRITICAL: Witness X account object MISSING from database!` — account index entry not found despite `account_index_size` showing entries exist
224
268
-`Could not modify object, most likely a uniqueness constraint was violated` — internal index pointers corrupted, uniqueness check fails
After one resize (500M → 1000M), free space is still below 1000M, triggering **another resize on the next check**, causing cascading resizes that dramatically increase the chance of corruption.
280
+
After one resize (500M → 1000M), free space is still below 1000M, triggering **another resize on the next check**, causing cascading resizes. While the resize barrier prevents corruption, frequent resizes still cause latency spikes as all operations are paused during each resize.
237
281
238
282
---
239
283
@@ -255,25 +299,78 @@ Most `database_api` methods use `with_weak_read_lock()`:
255
299
```cpp
256
300
// Example: get_accounts
257
301
auto result = with_weak_read_lock([&]() {
258
-
// Read from chainbase indices
302
+
// Read from chainbase indices — operation_guard acquired automatically
259
303
return accounts;
260
304
});
261
305
```
262
306
263
-
If the read lock cannot be acquired within `read-wait-micro × max-read-wait-retries`, the API returns error: `"Unable to acquire READ lock"`.
307
+
If the read lock cannot be acquired within `read-wait-micro × max-read-wait-retries`, the API returns error: `"Unable to acquire READ lock"`. If a resize barrier is active, the `operation_guard` inside `with_weak_read_lock()` will block until the resize completes before attempting the lock.
264
308
265
309
### Write Path (Block Application)
266
310
267
311
All state modifications go through `with_strong_write_lock()`:
Some code paths read from chainbase indices **without** holding a `boost::shared_mutex` lock. These must explicitly use an `operation_guard` to participate in the resize barrier:
op_guard.release(); // release before apply_block_post_validation()
372
+
```
373
+
277
374
### single-write-thread Mode
278
375
279
376
When `single-write-thread = true`, all `push_block()` and `push_transaction()` calls are serialized through a single `fc::async()` queue. This:
@@ -393,9 +490,10 @@ FATAL write lock timeout!!!
393
490
394
491
## Safety Rules
395
492
396
-
1.**`min-free-shared-file-size` must be less than `inc-shared-file-size`** — otherwise cascading resizes occur, dramatically increasing corruption risk
397
-
2.**Pre-allocate generously** — set `shared-file-size` large enough that resize is rare. Resizing is the most dangerous operation.
398
-
3.**Use `single-write-thread = true`** in production — prevents write lock contention and reduces the window for resize race conditions
399
-
4.**Avoid resize during witness production** — a witness node should have enough pre-allocated memory that resize never triggers during block generation
493
+
1.**`min-free-shared-file-size` must be less than `inc-shared-file-size`** — otherwise cascading resizes occur, causing frequent operation pauses
494
+
2.**Pre-allocate generously** — set `shared-file-size` large enough that resize is rare. Each resize pauses all operations while the segment is remapped.
495
+
3.**Use `single-write-thread = true`** in production — prevents write lock contention
496
+
4.**Avoid resize during witness production** — a witness node should have enough pre-allocated memory that resize never triggers during block generation. The resize barrier guarantees safety but introduces latency.
400
497
5.**After corruption, always replay** — there is no safe way to repair a corrupted `shared_memory.bin`. Use `--replay-blockchain` or `--snapshot` to rebuild state from block_log/dlt_block_log.
401
498
6.**Backup before config changes** — changing `shared-file-size` to a larger value triggers `grow()` on next startup, which is safe. Reducing it has no effect (file doesn't shrink).
499
+
7.**Any new lockless read path must use `operation_guard`** — if you add code that reads from chainbase indices without `with_read_lock()`/`with_write_lock()`, wrap it with `make_operation_guard()` to participate in the resize barrier. Failing to do so can cause stale pointer access during resize.
0 commit comments