fix(usage-ingestion): survive GOAWAY on idle grpc connections by benjackwhite · Pull Request #95018 · PostHog/posthog

benjackwhite · 2026-09-04T10:28:18Z

Problem

Every ingestion consumer that reports usage restarts in a loop since fix(usage-ingestion): make scale-ups take traffic #92663 deployed.
fix(usage-ingestion): make scale-ups take traffic #92663 made the usage-ingestion service close each connection after 60 seconds with an HTTP/2 GOAWAY.
The usage client is idle between flushes, so the GOAWAY arrives with no open stream.
connect-node then destroys the session with an error that Node emits only once the socket closes.
A flush in that window reconnects, connect-node removes its error listeners, and the deferred error surfaces as an uncaught exception.
The server lifecycle handler stops the process on any uncaught exception.
Upstream bug: Uncaught exception after GOAWAY on idle connection crashes the process connectrpc/connect-es#1768. Upstream fix: Do not raise deferred connection error as uncaught exception after GOAWAY connectrpc/connect-es#1769, not merged.

Changes

Ingestion consumers stay up when usage-ingestion recycles an idle connection. The next flush reconnects and sends as before.
A pnpm patch on @connectrpc/connect-node attaches an error listener before the GOAWAY destroy() call. The package already uses the same guard in abort().
The patch mirrors Set heroku python runtime to python 3.8.6 #1769 so it can be dropped on the connect-node bump that carries the upstream fix.
The patch covers every connect-node caller in the repo, including the personhog clients, which receive the same GOAWAY every 300 seconds.
Nothing user-visible changes. The lockfile change is mechanical.

How did you test this code?

New test nodejs/src/common/usage-ingestion/goaway.test.ts replays the sequence against a local HTTP/2 server: idle session, server GOAWAY, request before the socket closes.
Without the patch it fails with the same error and stack as production. With the patch it passes. No existing test covers GOAWAY handling.
Not checked: a real usage-ingestion server. The test uses a plain Node HTTP/2 server.

Automatic notifications

Publish to changelog?

Docs update

None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Claude Code (Claude Fable 5.1) in Conductor. Skills invoked: /writing-tests, /writing-pr-descriptions.

The session validated an earlier investigation against the Node http2 source and production logs, and got a deterministic local repro that the earlier pass had not. The test was written first and run red against the unpatched package before the patch was re-applied. pnpm had rewritten pnpm-workspace.yaml wholesale during the first install; the file was restored so the diff is one line. No open PR fixes this (searched "GOAWAY connect-node"). The diff carries no material from the session beyond public upstream links.

connect-node destroys an idle session on GOAWAY with an error that Node emits only once the socket closes. A request in that window removes the manager's error listeners, so the deferred error crashes the process. Patch connect-node with the guard from connectrpc/connect-es#1769 and add a test that replays the sequence against a local HTTP/2 server. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

trunk-io · 2026-09-04T10:28:21Z

😎 Merged successfully - details.

github-actions · 2026-09-04T10:30:17Z

🤖 CI report

✅ Trunk lane — non-backend lane

This PR is assigned to the non-backend lane. It does not run backend Python tests and may merge in parallel with PRs in other lanes.

✅ Bundle size — no change

Uncompressed size of every built .js bundle, compared against the base branch.

Total: 68.29 MiB · no change

No file changed by more than 1000 B.

_{Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report}

✅ Eager graph — within budget

How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.

Root	Eager (shipped)	Δ vs base	Budget
entry (logged-out pages, app bootstrap) `src/index.tsx`	1.45 MiB · 22 files	no change	`███░░░░░░░` 32.3% of 4.51 MiB
authenticated shell (every logged-in page) `src/scenes/AuthenticatedShell.tsx`	8.89 MiB · 3,268 files	no change	`█████████░` 91.6% of 9.71 MiB

🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx

Largest files eagerly shipped from src/index.tsx

Size	File
126.8 KiB	`../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js`
24.6 KiB	`../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js`
6.3 KiB	`../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js`
4.5 KiB	`../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js`
3.9 KiB	`../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js`
1.4 KiB	`../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js`
1.3 KiB	`src/RootErrorBoundary.tsx`
912 B	`../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js`
854 B	`src/scenes/ChunkLoadErrorBoundary.tsx`
762 B	`src/index.tsx`

Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx

Size	File
307.3 KiB	`../node_modules/.pnpm/posthog-js@1.425.1_@types+react@18.3.27_react@18.3.1/node_modules/posthog-js/dist/rrweb.js`
272.4 KiB	`../node_modules/.pnpm/posthog-js@1.425.1_@types+react@18.3.27_react@18.3.1/node_modules/posthog-js/dist/module.js`
267.7 KiB	`../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js`
255.2 KiB	`src/taxonomy/core-filter-definitions-by-group.json`
154.2 KiB	`../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js`
126.8 KiB	`../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js`
104.4 KiB	`src/lib/api.ts`
95.8 KiB	`../packages/quill/packages/quill/dist/index.js`
93.3 KiB	`../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js`
90.6 KiB	`../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js`

_{Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479}

✅ Toolbar bundle — eager 2.28 MiB within budget

What the toolbar ships to customer pages, measured from the esbuild output (minified, post-tree-shake). The eager set is the entry plus everything statically imported from it — fetched before any feature runs; deferred chunks load lazily. The eager guardrail is 5.72 MiB. Each output file must also stay below 10 MB, where CloudFront stops compressing it. The module boundary is enforced separately by check-toolbar-graph.

Metric	Size	Δ vs base	Budget
Eager (shipped) entry + static imports	2.28 MiB · 18 files	no change	`████░░░░░░` 39.8% of 5.72 MiB
Deferred (lazy)	2.11 MiB · 45 files	no change	n/a — loads on demand
Loader `dist/toolbar.js`	1.1 KiB	no change	`█░░░░░░░░░` 5.8% of 19.5 KiB

Largest eagerly-shipped chunks

Size	File
754.5 KiB	`dist/toolbar/toolbar-app-UB54KCUA.css`
599.5 KiB	`dist/toolbar/chunk-chunk-IP5OG3UU.js`
484.7 KiB	`dist/toolbar/chunk-chunk-EL6RQSMS.js`
135.6 KiB	`dist/toolbar/chunk-chunk-7UG5SHZR.js`
131.8 KiB	`dist/toolbar/chunk-chunk-FDH2IBXT.js`
71.3 KiB	`dist/toolbar/toolbar-app-P2YIMAVU.js`
69.0 KiB	`dist/toolbar/chunk-chunk-TSAL54PB.js`
35.6 KiB	`dist/toolbar/chunk-chunk-TX5RGCWJ.js`
21.0 KiB	`dist/toolbar/chunk-chunk-6WYJMFDF.js`
6.8 KiB	`dist/toolbar/chunk-chunk-DV7IWQNF.js`

_{Posted automatically by check-toolbar-size · sizes are toolbar output bytes (shipped, post-tree-shake) from the esbuild metafile}

✅ Dist folder size — no change

Total size of the built frontend/dist folder (all assets), compared against the base branch.

Total: 1458.90 MiB · no change

✅ Playwright — all passed

All tests passed.

View test results →

greptile-apps · 2026-09-04T10:34:47Z

_{Reviews (1): Last reviewed commit: "fix(usage-ingestion): survive GOAWAY on ..." | Re-trigger Greptile}

jose-sequeira

Just looking to double check the comment, otherwise LGTM. Probably best to manually deploy in dev to ensure that it doesn't affect anything in personhog

trunk-io · 2026-09-04T10:41:55Z

_{View Full Report ↗︎ ⋅ Docs}

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The service ends every connection with GOAWAY at max_connection_age. connect-node reports a flush that races that teardown as Canceled or Internal, neither of which the client retried, so it wrote the whole chunk off as dropped even though the service never saw the request. Ingest is idempotent by recordId, so retry both, and give the chunk three attempts with backoff instead of one immediate retry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…-callers

A retry that lands drops nothing, so it never touched the failed counter. Nothing showed it happened either. Mirror personhog's personhog_retries_total with usage_ingestion_retries_total, tagged with the producer and the error code the attempt failed on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

deployment-status-posthog · 2026-09-04T13:53:56Z

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-09-04 13:53 UTC	Run
prod-us	✅ Deployed	2026-09-04 14:33 UTC	Run
prod-eu	✅ Deployed	2026-09-04 14:23 UTC	Run

benjackwhite self-assigned this Sep 4, 2026

benjackwhite requested a review from jose-sequeira September 4, 2026 10:31

benjackwhite marked this pull request as ready for review September 4, 2026 10:31

jose-sequeira reviewed Sep 4, 2026

View reviewed changes

Comment thread nodejs/src/common/usage-ingestion/goaway.test.ts

jose-sequeira approved these changes Sep 4, 2026

View reviewed changes

benjackwhite and others added 4 commits September 4, 2026 12:43

chore(usage-ingestion): assert the request after GOAWAY is served

08e22c2

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/master' into goaway-crash-nodejs…

bb2e0ed

…-callers

benjackwhite mentioned this pull request Sep 4, 2026

fix(flags): retry usage records the connection dropped #95084

Merged

1 task

trunk-io Bot merged commit 54f84b7 into master Sep 4, 2026
225 of 229 checks passed

trunk-io Bot deleted the goaway-crash-nodejs-callers branch September 4, 2026 13:28

Navigation Menu

fix(usage-ingestion): survive GOAWAY on idle grpc connections - #95018

fix(usage-ingestion): survive GOAWAY on idle grpc connections#95018
trunk-io[bot] merged 5 commits into
masterfrom
goaway-crash-nodejs-callers

benjackwhite commented Sep 4, 2026

Uh oh!

trunk-io Bot commented Sep 4, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 4, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Sep 4, 2026

Uh oh!

Uh oh!

jose-sequeira left a comment

Uh oh!

trunk-io Bot commented Sep 4, 2026 •

edited

Loading

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Sep 4, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

benjackwhite commented Sep 4, 2026

Problem

Changes

How did you test this code?

Automatic notifications

Docs update

🤖 Agent context

Uh oh!

trunk-io Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 CI report

Uh oh!

greptile-apps Bot commented Sep 4, 2026

Uh oh!

Uh oh!

jose-sequeira left a comment

Choose a reason for hiding this comment

Uh oh!

trunk-io Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploy status

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

trunk-io Bot commented Sep 4, 2026 •

edited

Loading

github-actions Bot commented Sep 4, 2026 •

edited

Loading

trunk-io Bot commented Sep 4, 2026 •

edited

Loading

deployment-status-posthog Bot commented Sep 4, 2026 •

edited

Loading