perf(parse): run Tier-1 extraction on a worker-thread pool

[qoole]

Problem

The cold-build parse loop is a single synchronous files.forEach — every tree-sitter extraction serializes on the main thread. On a 4,000-file TypeScript repo that is ~7.8s of the 7.9s cold build (graft build, no --deep), one core busy while the rest of the machine idles.

Change

Cache-miss files now go to a small pool of worker threads:

• graph/parse-worker.ts — the worker half. Receives one strided partition of the misses, warms the WASM grammars its partition needs once, parses strictly sequentially, streams one result message per file.
• graph/parse-pool.ts — the parent half. Partitions strided, spawns one pool per build, terminates after. A worker that dies mid-partition rejects loudly — no silent partial graph.
• graph/build.ts — the loop splits into three passes:
  1. classify + read + hash + replay cache hits (main thread — the read+hash is ~0.05ms/file, so hits never spawn worker work)
  2. extract the misses on the pool
  3. merge every per-file slot strictly in file order

Pass 3 is the determinism contract: node/edge/entry output is merged by the caller's file index, never by completion order, so a cold build is byte-identical to the single-threaded loop regardless of which worker finished first — the invariant test/graph-incremental.test.ts pins down.

Knob

GRAFT_PARSE_CONCURRENCY (default 4, capped by CPUs and by the miss count). 1 skips the pool entirely and runs the misses inline — the old single-threaded behaviour, kept as an escape hatch.

Measurements

Synthetic 4,000-file TS repo, cold build (3 runs each, medians):

mode	time	speedup
sequential (baseline main)	7.85s	—
inline fallback (this patch, GRAFT_PARSE_CONCURRENCY=1)	7.92s	no regression
pool, 4 workers (default)	3.95s	1.99×
pool, 8 workers	2.2s	3.6×

Default kept at 4 (not 8) deliberately — graft often runs alongside other work (MCP refresh, the Stop hook), and the env knob is there for dedicated boxes.

Output equality

wiring.json is sha256-identical across sequential / inline / pool on the same input. (The extract-cache filename/stamp differs between build environments — it hashes build artifacts, not extractor behaviour; cache contents are JSON-parse-identical.)

Tests

1351 pass / 6 fail — the same 6 fail on clean unmodified main in this environment (the git worktree/seed/refresh set: graph-refresh.test.ts:102 etc.). Baseline recorded before any change; the set is unchanged after this patch.

🤖 Generated with Claude Code

[trailhq-graft]

🌱 graft blast radius

1 area changed → 6 areas can be affected. 16 dependent symbols, depth 2.
Tests: Graph Parsing has tests the diff did not touch.
Tag: @anirudhkumar-nanonets — 5 of 7 areas · @shhdwi — 6 of 7 areas · @bhavesh-gupta-investis — Sync Execution

flowchart TB
  A0(("Pull Request Review<br/>8 symbols"))
  A1(("Graph Freshness<br/>3 symbols"))
  A2(("CLI Engine<br/>2 symbols"))
  A3(("Viewer Build<br/>1 symbol"))
  A4(("MCP Tools<br/>1 symbol"))
  AX(("1 smaller area<br/>1 symbol"))
  classDef reached fill:#D9EDF3,stroke:#3AA7C9,stroke-width:1.5px,color:#0E313C;
  class A0,A1,A2,A3,A4 reached;
  classDef tail fill:#EEF2F3,stroke:#9AA4A9,stroke-width:1px,color:#3A4247;
  class AX tail;

Loading

Can be affected	Symbols	Nearest hop	Reached from
Pull Request Review	8	src/app/brain-build.ts:L251-L358 readRepository — calls, depth 1	Graph Parsing
Graph Freshness	3	src/graph/refresh.ts:L150-L227 ensureFreshGraph — calls, depth 1	Graph Parsing
CLI Engine	2	src/engine.ts:L91-L101 graph — calls, depth 1	Graph Parsing
Viewer Build	1	scripts/build-viewer.mjs:L1-L45 build-viewer.mjs — calls, depth 2	Graph Parsing
MCP Tools	1	src/mcp/tools.ts:L216-L244 callTool — calls, depth 2	Graph Parsing
Sync Execution	1	src/claude/sync-run.ts:L19-L33 runSync — calls, depth 2	Graph Parsing

Who knows this code — 3 people across 7 areas

Area	Who knows it
Graph Parsing · changed	@anirudhkumar-nanonets — 17 commits, last 2mo ago · @shhdwi — 5 commits, last 2mo ago
Pull Request Review · affected	@anirudhkumar-nanonets — 12 commits, last 25d ago
Graph Freshness · affected	@anirudhkumar-nanonets — 8 commits, last 2mo ago · @shhdwi — 3 commits, last 2mo ago
CLI Engine · affected	@anirudhkumar-nanonets — 54 commits, last 6d ago · @shhdwi — 24 commits, last 2mo ago
Viewer Build · affected	@shhdwi — 2 commits, last 2mo ago
MCP Tools · affected	@shhdwi — 14 commits, last 2mo ago · @anirudhkumar-nanonets — 7 commits, last 1mo ago
Sync Execution · affected	@shhdwi — 3 commits, last 3mo ago · @bhavesh-gupta-investis — 1 commit, last 1mo ago

Ownership is git history over each area's own files, weighted towards recent work (120-day half-life). Merge commits and bots are dropped, and you are dropped from your own PR. A name with no @ has no GitHub handle in its commit email — tag them by hand, or add a .mailmap entry. A suggestion from history, not a CODEOWNERS rule.

All 16 dependent symbols, grouped by area

Pull Request Review — 8 symbols in 6 files

• src/app/brain-build.ts:L251-L358 — readRepository (calls, depth 1)
  282: await buildGraph(checkout.dir, { graphOnly: true });
• src/app/review.ts:L45-L99 — reviewPullRequest (calls, depth 1)
  55: await buildGraph(checkout.dir);
• src/app/brain-build-worker.ts:L1-L83 — brain-build-worker.ts (calls, depth 2)
  12: import { readRepository, type BrainBuildJob, type RepoReadAuth } from "./brain-build.js";
• src/app/brain-build-worker.ts:L29-L32 — DoneMessage (references, depth 2)
• src/app/brain-build.ts:L237-L239 — buildRepoIntoBrain (calls, depth 2)
• src/app/review-process.ts:L179-L183 — childReviewer (references, depth 2)
• src/app/review-worker.ts:L67-L87 — run (calls, depth 2)
• src/app/server.ts:L34-L46 — AppSeams (references, depth 2)

Graph Freshness — 3 symbols in 2 files

• src/graph/refresh.ts:L150-L227 — ensureFreshGraph (calls, depth 1)
  216: await buildGraph(dir, { contextDir: opts.contextDir, graphOnly: true, onlyDirs });
• src/graph/refresh.ts:L235-L261 — ensureFreshChildren (calls, depth 2)
• src/graph/workspace-cli.ts:L49-L70 — buildChild (calls, depth 2)

CLI Engine — 2 symbols in 2 files

• src/engine.ts:L91-L101 — graph (calls, depth 1)
  92: return buildGraph(dir, {
• src/cli.ts:L205-L215 — refreshBefore (calls, depth 2)

Viewer Build — 1 symbol in 1 file

• scripts/build-viewer.mjs:L1-L45 — build-viewer.mjs (calls, depth 2)
  3: * assets). Runs as part of `npm run build`; the bundle ships in the package

MCP Tools — 1 symbol in 1 file

• src/mcp/tools.ts:L216-L244 — callTool (calls, depth 2)

Sync Execution — 1 symbol in 1 file

• src/claude/sync-run.ts:L19-L33 — runSync (calls, depth 2)

Test signal per changed area — 1 ⚠

Reached = a node under a test path has a resolved edge into the changed symbol. It undercounts anything called indirectly — through a CLI, a spawned process or a dynamic import — so read a low ratio as “look here”, never as a coverage gate.

• ⚠ Graph Parsing — 1 of 7 reached · 33 test files reach it, none changed here
  • not reached: freshSlot, fail, parseWorkerCount, workerEntryUrl, ensureWarmed, parseOne

33 test suites also reference this code

41 symbols, kept out of the diagram and the table so they cannot crowd out the areas a reviewer has to look at.

• test/ask-index.test.ts
• test/ask.test.ts
• test/container-extract.test.ts
• test/context-only-dir.test.ts
• test/context.test.ts
• test/covers.test.ts
• test/generic-extract.test.ts
• test/graph-go.test.ts
• test/graph-incremental.test.ts
• test/graph-invariants.test.ts
• test/graph-java.test.ts
• test/graph-languages.test.ts
• test/graph-php.test.ts
• test/graph-posix-paths.test.ts
• test/graph-python.test.ts
• test/graph-r-classes.test.ts
• test/graph-r-phase3.test.ts
• test/graph-r-phase4.test.ts
• test/graph-r-phase5.test.ts
• test/graph-r.test.ts
• …13 more

_{graft blast · origin/main...HEAD · depth 2 · 3 changed files}

Open the interactive graph → — click an area to see its dependent symbols at file:line.