Skip to content

feat(repo): trace the front door locally and in cloudflare - #45

Closed
systemfsoftware-maker wants to merge 38 commits into
lake1/delete-hellofrom
lake1/traces
Closed

systemfsoftware-maker wants to merge 38 commits into
lake1/delete-hellofrom
lake1/traces

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Lake 1, U10 — stacked on #43.

  • apps/site/src/worker.ts (KTD9): per request, OTLP_BASE_URL bound → Otlp.layerJson (effect/observability, service endgame-site, FetchHttpClient) to the local collector; unbound → alchemy's per-invocation CloudflareTracer layer into Workers tracing. The layer is built inside a scope that closes (and flushes) before the response resolves; the OTLP wiring is a module-scope memoized closure (effect-platform/runtime-construction-placement).
  • apps/site/alchemy.run.ts: Website observability on — logs (with invocation logs) and traces, persisted, sampled at 1; OTLP_BASE_URL=http://127.0.0.1:4318 only on non-cloud stages (cloud = prod, pr-<N>), read from Alchemy.Stage.
  • apps/site/tests/front-door.trace.test.ts (new, in-process under pnpm test/Stryker, no Tempo): the stimulus calls the front door through @endgame/site with the contract's traceparent; an ObservationWindow collects spans; the HTML port dies if reached. Cases: / with no Accept → route home, decision ServeMarkdownPage; /llms.txt → llms_txt, ServeLlmsTxt; unknown path → unknown, ServeMarkdownNotFound; each case also holds the span under the request that continues the contract traceparent.
  • @systemfsoftware/trace-spec 1.0.1 (exact catalog pin).

Deviations:

  • alchemy patch (Kiro ruling). alchemy 2.0.0-beta.80 has no working subpath for CloudflareTracer: the ./Cloudflare/* exports pattern maps to ./lib/Cloudflare/*/index.js and shadows ./* (beta.81 has the same map). patches/alchemy@2.0.0-beta.80.patch adds only that exports entry, via patchedDependencies; delete it in the Dependabot PR of the alchemy release that fixes the map.
  • Decision attribute on the declaration (Kiro-authorized). trace-spec decodes a declared span's attrs through its declaration, so the decision tag was invisible until ServePage declared app.front_door.serve_page.decision. Its literal set comes from serve-page.workflow.ts and is typed against the ServePageDecision union, so a new decision variant is a compile error until the attribute knows it.
  • No taxonomy facts yet. The taxonomy has one span, so there are no child/forbid facts to hold; U11 adds front_door.record_csp_violation.
  • src/mod.ts re-exports ServePage/frontDoorTaxonomy (tests import only the public API); the spec runs .live with NodeFileSystem because the kernel lane refuses the failure dump's real-fs write.
  • Workers freeze timers during CPU work, so local span durations read 0 µs.

Also on this stack, as fix commits on #41: the llms-txt property refuted a catalog whose path held ) (the Markdown link closed early); page titles, paths and the origin are now refined to valid link parts. And a TanStack generator temp file that #41 had committed is untracked and apps/site/.tanstack/ ignored. apps/site/artifacts/ (trace-spec failure dumps) is ignored here.

QA

$ pnpm check:ci
check=0
$ pnpm stack:ci
stack=0
# local stack: GET / with traceparent 00-38d24704a4226c6b8bd51a69c7291fca-…-01
200 text/markdown; charset=utf-8  "# Build with agents you can still trust"
# Tempo GET /api/v2/traces/38d24704a4226c6b8bd51a69c7291fca  (service.name endgame-site)
front_door.serve_page        app.front_door.route=home  app.front_door.serve_page.decision=ServeMarkdownPage
front_door.serve_page.read   (child)
front_door.serve_page.write  (child)
# Grafana Explore screenshot of that trace: /tmp/lake1-msg/u10-grafana.png (kept with the session evidence; gh cannot attach images)
# Sabotage: ServePageCommand's InstrumentationBrand path → 'app.front_door.route_sabotaged'
$ pnpm --filter @endgame/site exec vitest run tests/front-door.trace.test.ts
ContractDecodeError: Span "front_door.serve_page" … does not satisfy contract "front_door.serve_page": Missing key at ["app.front_door.route"]
Tests  3 failed (3)
# reverted → Tests  3 passed (3)

Restack onto the reviewed #41 (Kiro rulings, Lake 1 bottom review)

  • feat(repo): serve the starter site from one worker #41 now serves pnpm dev through vite dev in workerd, not alchemy dev, so alchemy no longer binds OTLP_BASE_URL locally. Without it the dev worker exported nothing to the local collector.
  • site-worker.ts holds the one local collector URL (localOtlpBaseUrl). The workers Vite plugin binds it for vite dev only, and alchemy.run.ts binds the same value for non-cloud stages. Builds and cloud stages never bind it (6cc0868).
  • Red, with the dev binding removed, on a fresh stack: the collector's otelcol_receiver_accepted_spans counter rose by 0 across three requests. Green: it rose by 9 (read, write and cell span per request).
  • Gate at 6cc0868: uncached pnpm check:ci exits 0.

macOS timeout fix

R110: Alchemy + distilled are the whole Cloudflare stack

Ryan 2026-10-06: Alchemy + distilled are the whole Cloudflare stack. This layer uses no Spectrum-to-Worker TCP, no Workers Issues automation, and no K2 or Basin. K2 and Basin arrive with the Alchemy bump that ships alchemy-run/alchemy#2010. Workers Issues stays on through Alchemy's Worker settings, and gRPC uses only the Worker's HTTP path (Connect/gRPC-web). The template holds no credentials.

check:ci no longer runs stryker. The release gate on push to main plans one
shard per workspace package that declares a mutation script, refuses an
empty set, and runs each shard at break 100 on the fleet with its
incremental report cached. The checker drops
prioritizePerformanceOverAccuracy, which stryker-js 15 removes. Operator
approval: Kiro, 2026-10-05 (GATE1)
The plan and per-package mutation jobs move from the self-hosted fleet,
which admits only private repositories, to ubuntu-latest. Mutation stays
one parallel job per package.

Operator approval: Kiro, 2026-10-05 (GATE1)
systemfsoftware-maker added a commit that referenced this pull request Oct 6, 2026
…and QA evidence

Findings for #43, #45, #49, #50 and #51 from ce-code-review (8 lenses, validator), the verifier's probes of every P0/P1 including those the merge dropped, and the real-browser, local-Tempo and live-preview QA. Nothing applied; each finding waits for a ruling
A package with a mutation script whose mutate globs match nothing
reports zero mutants and passes a break-100 threshold vacuously
(review finding #9). Each package now declares its globs once, as
stryker.mutate in package.json; its Stryker config reads them and the
release gate's shard planner expands them. The planner fails red at
plan time naming the package, its directory and its globs when they
match no file, including a package with no declared globs. Its Deno
tests are ordinary tests: a turbo root task, test:scripts, runs them
through the dev shell's deno, and both pnpm test and check:ci run it,
so a local check:ci covers the planner exactly as CI does
Every catalog entry is an exact version: effect 4.0.1, the
systemfsoftware toolchain on its Effect 4 stable majors (recommended preset
4.0.0, stryker-js 15.0.1, vitest 2.0.0, gherkin-spec 7.0.1) and current
tooling. minimumReleaseAge 1440 is explicit, so the policy is strict;
the exclude list holds the effect name patterns and one exact entry per
resolved systemfsoftware package. oxlint-tsgolint stays 7.0.2001, the
exact peer the presets require. turbo 2.11 writes an agent guidance block
into AGENTS.md unless agentGuidance is false, so turbo.json opts out.
Grader package majors move here because they require Effect 4 stable
(CONST-W3 declared)
oxlint.shared.ts, vitest.shared.ts and stryker.shared.ts hold the house
settings; each package config extends them and declares only its own
mutate set and aliases. Both lint overrides are gone: preset 4.0.0 already
accepts Gherkin step bodies and build-config imports. The root lint file is
named oxlint.shared.ts so lint-staged's nearest-config walk never treats
the repository root as a package. Turbo inputs track the shared files.
Operator approval: Kiro, 2026-10-05 (GATE1)
…root

Root tsconfig.base.json extends the bundler/dom preset and allows exactly
effect/http through the effect language service. effect 4.0.1 ships HTTP
only as unstable and global-fetch-in-effect bans the fetch alternative.
Operator approval: Kiro, 2026-10-05 (GATE1)
tsconfig.base.json now allows exactly effect/http and effect/observability.
effect 4.0.1 ships its OTLP exporter only as unstable, and the Effect-native
exporter keeps raw OpenTelemetry SDK wiring out of the Worker.
Operator approval: Kiro, 2026-10-05 (GATE1)
pnpm dev starts the OpenTelemetry collector, Tempo and Grafana under
process-compose with readiness probes. The binaries come from the locked
flake through a local-stack package, and bin/local-stack runs it from the
repo root the way bin/dprint runs dprint. Grafana provisions Tempo as its
default data source; stack state lives in the ignored local-stack/data
process-compose sends SIGKILL after a shutdown timeout only when one is
declared; without it a process that ignores SIGTERM keeps local-stack down
waiting forever. Tempo's live store does that after it has taken spans
(shutdown completing loop). Every process now declares a 10 second
timeout, after which its process group is killed; Tempo replays its WAL
on the next start
Tempo 3's live store, backend scheduler and block builder default to
/var/tempo. Only root can create that, so on a GitHub-hosted runner the
live store failed with "mkdir /var/tempo: permission denied", the
distributor that depends on it failed too, and Tempo stopped before it was
ready. Local runs as root had hidden it. All four paths now sit under
local-stack/data/tempo
`pnpm dev` (bin/local-stack) exits 1 with "TUI startup error: terminal
entry not found" whenever stdout is not a TTY and TERM is unset: an agent,
a CI job, `ssh host pnpm dev`. process-compose's TUI wants a terminal it
does not have. local-stack now sets PC_DISABLE_TUI when stdout is not a
TTY, so the same command renders plain logs and every process starts.

bin/check-local-stack proves it, and pnpm check:ci runs it as check:ci-run
on Linux. It starts the stack the way an agent does — stdout not a TTY,
TERM unset — inside an unprivileged network namespace, waits for every
readiness probe, stops the stack and checks every port is free.

Operator approval: Kiro, 2026-10-06 (QA Q1)
Grafana's built-in preinstall list (grafana-pyroscope-app,
grafana-exploretraces-app) downloads unpinned zips from grafana.com on
every fresh data dir; nothing pins that code and it runs outside the
sandbox. Nothing here needs those apps: the Tempo data source is
provisioned from a file and core Explore renders traces from it.
preinstall_disabled keeps the stack offline.

bin/check-local-stack now asserts Grafana never reaches for a plugin while
the stack runs with outbound networking denied, and kills a half-started
stack so a failed assertion cannot leak processes.

Operator approval: Kiro, 2026-10-06 (QA Q3)
check:ci runs bin/check-local-stack on Linux, but hosted Linux CI runs check:ci split into legs and only macOS runs it whole, where the check is skipped. The new leg runs it on every PR
…reads them

The release gate's shard planner now refuses a package whose stryker.mutate globs match no file, and reads them from package.json. The site declares its globs there (readme-opening-plugin.ts included) and its Stryker config reads them
The site's local runtime moves to workerd through @cloudflare/vite-plugin
1.62.5 and its wrangler 4.147.0 peer, with miniflare at the exact version
wrangler pins. Alchemy stays the only deploy path.

One workerd resolves for alchemy and the wrangler toolchain: 1.20261005.1,
the newest version both accept
…ugin

The site's vite config now registers @cloudflare/vite-plugin with the
framework's ssr environment as the worker environment, so `pnpm dev`,
`vite preview` and the local-stack site process serve the front door from
workerd instead of alchemy's node runtime. Alchemy stays the only deploy
path, and one shared module carries the worker name, entry and
compatibility posture for both
…ayer

Miniflare at wrangler's pin runs the production vite build output from
Node vitest, disposed when the layer scope closes. Readiness is a real
HTTP probe of /llms.txt held to a deadline, with typed failures; a
never-ready worker proves the timeout path red
A test-only worker entry built by the same vite pipeline composes the
real front door with a failing HtmlPort. The feature asserts the
browser's HTML 500 with Vary Accept, the agent's unaffected Markdown
page, no unhandled rejection, and the errored span (status error plus
exception event) read from an in-test OTLP receiver
A branch no request can reach does not ship. No real path fails on a
Markdown-preferring request: read cannot fail (the request source is
the Web Request itself, so toWeb cannot raise RequestParseError),
decide is infallible (S.Never), ServeMarkdownPage and the two other
Markdown writes are pure, llmsTxt is infallible so Result.getOrThrow
cannot throw, and renderHtml runs only for HTML-preferring requests.
The failure answer is therefore always the HTML 500 with Vary Accept
Alchemy's Website.Vite builds in a child that loads vite.config.ts and
injects its own Cloudflare plugin. It neutralises an official
@cloudflare/vite-plugin's hooks but cannot undo the factory's
construction-time side effects, so with the official plugin registered
every alchemy deploy failed in that child with exit 1 and no output.
The drift proof's real deploy went red on this (BundleError: Vite build
child exited with code 1).

Alchemy sets ALCHEMY_CLOUDFLARE_VITE_INJECTED=1 in that child and
documents it as the way to skip the official factory. sitePlugins now
skips it there and only there, so vite dev, vite preview, vite build and
the workerd harness keep running the site in workerd
A stack job on the fleet runs pnpm stack:ci inside the dev shell: it starts
the same process-compose definition as pnpm dev, waits until every service
is ready, runs the local e2e journeys and stops the stack. The dev shell now
carries local-stack and a pinned actionlint. AGENTS.md adds START-5.
Operator approval: Kiro, 2026-10-05 (GATE1)
The stack job's first hosted run timed out in is-ready, and `down` then
found no process-compose server: Tempo had stopped at startup, its
dependents never started, and process-compose exited with the project. The
job log said none of that. stack:ci moves into bin/stack-ci, which starts
the project with --keep-project so the server outlives a failed process,
and on any failure writes local-stack/data/stack-report.md (every process's
state and the last 40 lines of its log) before stopping the stack. The job
adds the report to the step summary and uploads it. A separate step
realises the dev shell first, so the 300 second readiness budget measures
startup rather than store downloads. The dev shell gains jq for the report.

Operator approval: Kiro, 2026-10-05 (GATE1)
packages/starter and every trace of it leave in one commit: the README
sections that describe it, the trusted-publisher note in .changeset, and
the catalog entries no remaining package uses (rimraf, tsdown).

npm publishing goes with it. The release job keeps capture, release-note
assertion, tagging and GitHub Releases and drops build, OIDC and pnpm
publish. A version in a manifest with no matching <name>@v<version> tag is
owed; no script queries registry.npmjs.org.
Operator approval: Kiro, 2026-10-05 (release.yml)
The Worker picks its span exporter per request: Otlp.layerJson to the local
collector when OTLP_BASE_URL is bound, otherwise alchemy's per-invocation
CloudflareTracer layer into Workers tracing. Two places bind it, both to
site-worker.ts's one local collector URL: the workers vite plugin for
vite dev, which is how pnpm dev serves the site, and alchemy for non-cloud
stages. Builds and cloud stages never bind it. The Website turns on logs and
traces, persisted, sampled at 1.

alchemy 2.0.0-beta.80 publishes no working subpath for CloudflareTracer: its
./Cloudflare/* exports pattern shadows ./*. patches/ adds only the missing
exports entry (Kiro ruling, 2026-10-05).

front-door.trace.test.ts proves the span graph in-process under an
ObservationWindow: route and decision attributes per case, parented under
the request that continues the contract traceparent. The ServePage
declaration now carries the decision attribute, tied to the decision union
The macOS check:ci leg failed at #45 and #49: "Importing the stack deploys
nothing" took 9.9 s and 7.3 s against vitest's default 5 s test timeout.
Importing alchemy.run.ts loads Alchemy and all of its Cloudflare providers,
and a cold import on a hosted macOS runner is slower than on Linux. The
scenario now carries a 60 s budget, as the repo's other features that do
real module or process work already do. What it asserts is unchanged
@systemfsoftware-maker
systemfsoftware-maker added this pull request to stack #54 October 6, 2026 19:52
@systemfsoftware-maker
systemfsoftware-maker removed this pull request from stack #54 October 6, 2026 21:56
@systemfsoftware-maker
systemfsoftware-maker added this pull request to stack #59 October 6, 2026 21:57
@systemfsoftware-maker
systemfsoftware-maker removed this pull request from stack #59 October 6, 2026 22:13
@systemfsoftware-maker

Copy link
Copy Markdown
Collaborator Author

Dropped by the 2026-10-06 starter spec: no tracing layer in the starter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant