Skip to content

build(repo): add the one-command local stack - #38

Closed
systemfsoftware-maker wants to merge 14 commits into
lake1/shared-configsfrom
lake1/local-stack
Closed

systemfsoftware-maker wants to merge 14 commits into
lake1/shared-configsfrom
lake1/local-stack

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Lake 1, U6 — stacked on #37.

pnpm dev (./bin/local-stack up) starts the local stack under process-compose with readiness probes and no cloud credentials (R63, without the site service that U7 adds):

Service Ready probe
OpenTelemetry collector (otelcol-contrib 0.155.0), OTLP on :4318 HTTP and :4317 gRPC :13133/
Tempo 3.0.3, monolithic, local storage under local-stack/data/tempo :3200/ready
Grafana 13.1.6, anonymous admin, Tempo provisioned as the default data source :3000/api/health
  • The binaries come from the locked flake: nix/local-stack.nix is a writeShellApplication over process-compose and the three services. bin/local-stack copies the bin/dprint wrapper and adds cd to the repo root, since every path in process-compose.yaml is root-relative.
  • Grafana's data, log, plugin and provisioning paths are passed as absolute cfg: overrides. Relative paths in grafana.ini resolve against Grafana's homepath in the Nix store; the first run showed can't read datasource provisioning files ... /nix/store/...-grafana-13.1.6/share/grafana/local-stack/grafana/datasources.
  • local-stack/data/ is gitignored.

QA

$ ./bin/local-stack up -D -t=false && timeout 300 ./bin/local-stack project is-ready --wait
ready exit=0
$ ./bin/local-stack process list -o wide
NAME             STATUS    HEALTH
grafana          Running   Ready
otel-collector   Running   Ready
tempo            Running   Ready
$ curl -s :3000/api/datasources | jq -c '.[]|{uid,type}'
{"uid":"tempo","type":"tempo"}
# POST one OTLP/JSON span (service local-stack-qa, name qa.roundtrip) to :4318/v1/traces, read it back from Tempo
POST /v1/traces -> 200
GET :3200/api/v2/traces/fd0a6202e8a47d6d4a8f3691e13b7445 -> found, span "qa.roundtrip"
# Grafana Explore (Tempo, TraceQL = the trace id) renders "local-stack-qa: qa.roundtrip", 1 span, 5ms
$ ./bin/local-stack down
$ pnpm check:ci
check:ci exit=0

Screenshot: Grafana Explore showing the trace local-stack-qa: qa.roundtrip (taken with headless Chromium; the image is kept with the session evidence, not uploaded, since the GitHub CLI cannot attach images to PR bodies).

Fix commits after opening

  • ed4b2cf build: local-stack down hung with Tempo stuck in Terminating. Once Tempo has taken spans, its live store loops in shutdown completing loop on SIGTERM, and process-compose escalates to SIGKILL only when shutdown.timeout_seconds is declared. Every process now declares 10 s. Reproduced first (stack up, spans sent, down still waiting after 5 min); after the fix, 20 OTLP spans then down → exit 0 in 10 s with Tempo killed. Tempo replays its WAL on the next start.

Review fixes (Kiro rulings, Lake 1 bottom review)

  • Q1, pnpm dev with no terminal (ea247d9, CI leg 4f8ea78).
    • nix/local-stack.nix sets PC_DISABLE_TUI=1 when stdout is not a TTY.
    • bin/check-local-stack (pnpm check:ci-run) does the following:
      1. Starts ./bin/local-stack up with stdout redirected and TERM unset.
      2. Waits for project is-ready, then checks every readiness probe (13133, 3200 /ready, 3000 /api/health).
      3. Runs down, then waits until all eight ports are free.
    • check:ci runs it on Linux. Hosted Linux CI splits check:ci into legs, so it also has its own local-stack leg.
    • Red with the TUI block removed: pnpm dev exited before the stack was ready … FTL TUI startup error error="terminal entry not found: term not set", exit 1. Green: exit 0.
  • Q3, Grafana downloads nothing at startup (c2767fa).
    • No Grafana plugin is needed: Explore and the Tempo datasource are built in. local-stack/grafana.ini sets [plugins] preinstall_disabled = true.
    • The check runs the whole stack in a network namespace with outbound traffic denied (unshare -rn, with bwrap --unshare-net as fallback). It proves the namespace is offline (curl: Could not resolve host: grafana.com) and fails if any log shows a plugin install.
    • Red with the [plugins] block removed: Grafana reached for a plugin with the network denied: Installing plugin, exit 1. Green: exit 0.

check:ci no longer runs stryker. The release gate on push to main plans one
shard per workspace package that declares a mutation script, refuses an
empty set, and runs each shard at break 100 on the fleet with its
incremental report cached. The checker drops
prioritizePerformanceOverAccuracy, which stryker-js 15 removes. Operator
approval: Kiro, 2026-10-05 (GATE1)
The plan and per-package mutation jobs move from the self-hosted fleet,
which admits only private repositories, to ubuntu-latest. Mutation stays
one parallel job per package.

Operator approval: Kiro, 2026-10-05 (GATE1)
@systemfsoftware-maker

Copy link
Copy Markdown
Collaborator Author

Fix 4ce06fb (CI run 37400878461 root cause): Tempo 3 defaults its live-store marker/WAL, backend-scheduler and block-builder paths to /var/tempo, which only root can create. On the hosted runner the live store failed (mkdir /var/tempo: permission denied), the distributor followed, and Tempo stopped before ready; local runs as root hid it (/var/tempo exists on this host). All four now live under local-stack/data/tempo. Proof: Tempo run as uid 65534 with the old config stops with module failed module=live-store; with the new config it answers /ready and creates its marker dir under the repo. CI stack green on #42, #43, #45, #49.

systemfsoftware-maker added a commit that referenced this pull request Oct 6, 2026
… and QA evidence

Findings for #35-#38 and #41 from ce-code-review (8 lenses, validator), the verifier's terminal probes of the confirmed findings, and the real-browser and real-CLI QA. Nothing applied; each finding waits for a ruling
A package with a mutation script whose mutate globs match nothing
reports zero mutants and passes a break-100 threshold vacuously
(review finding #9). Each package now declares its globs once, as
stryker.mutate in package.json; its Stryker config reads them and the
release gate's shard planner expands them. The planner fails red at
plan time naming the package, its directory and its globs when they
match no file, including a package with no declared globs. Its Deno
tests are ordinary tests: a turbo root task, test:scripts, runs them
through the dev shell's deno, and both pnpm test and check:ci run it,
so a local check:ci covers the planner exactly as CI does
Every catalog entry is an exact version: effect 4.0.1, the
systemfsoftware toolchain on its Effect 4 stable majors (recommended preset
4.0.0, stryker-js 15.0.1, vitest 2.0.0, gherkin-spec 7.0.1) and current
tooling. minimumReleaseAge 1440 is explicit, so the policy is strict;
the exclude list holds the effect name patterns and one exact entry per
resolved systemfsoftware package. oxlint-tsgolint stays 7.0.2001, the
exact peer the presets require. turbo 2.11 writes an agent guidance block
into AGENTS.md unless agentGuidance is false, so turbo.json opts out.
Grader package majors move here because they require Effect 4 stable
(CONST-W3 declared)
oxlint.shared.ts, vitest.shared.ts and stryker.shared.ts hold the house
settings; each package config extends them and declares only its own
mutate set and aliases. Both lint overrides are gone: preset 4.0.0 already
accepts Gherkin step bodies and build-config imports. The root lint file is
named oxlint.shared.ts so lint-staged's nearest-config walk never treats
the repository root as a package. Turbo inputs track the shared files.
Operator approval: Kiro, 2026-10-05 (GATE1)
…root

Root tsconfig.base.json extends the bundler/dom preset and allows exactly
effect/http through the effect language service. effect 4.0.1 ships HTTP
only as unstable and global-fetch-in-effect bans the fetch alternative.
Operator approval: Kiro, 2026-10-05 (GATE1)
tsconfig.base.json now allows exactly effect/http and effect/observability.
effect 4.0.1 ships its OTLP exporter only as unstable, and the Effect-native
exporter keeps raw OpenTelemetry SDK wiring out of the Worker.
Operator approval: Kiro, 2026-10-05 (GATE1)
pnpm dev starts the OpenTelemetry collector, Tempo and Grafana under
process-compose with readiness probes. The binaries come from the locked
flake through a local-stack package, and bin/local-stack runs it from the
repo root the way bin/dprint runs dprint. Grafana provisions Tempo as its
default data source; stack state lives in the ignored local-stack/data
process-compose sends SIGKILL after a shutdown timeout only when one is
declared; without it a process that ignores SIGTERM keeps local-stack down
waiting forever. Tempo's live store does that after it has taken spans
(shutdown completing loop). Every process now declares a 10 second
timeout, after which its process group is killed; Tempo replays its WAL
on the next start
Tempo 3's live store, backend scheduler and block builder default to
/var/tempo. Only root can create that, so on a GitHub-hosted runner the
live store failed with "mkdir /var/tempo: permission denied", the
distributor that depends on it failed too, and Tempo stopped before it was
ready. Local runs as root had hidden it. All four paths now sit under
local-stack/data/tempo
`pnpm dev` (bin/local-stack) exits 1 with "TUI startup error: terminal
entry not found" whenever stdout is not a TTY and TERM is unset: an agent,
a CI job, `ssh host pnpm dev`. process-compose's TUI wants a terminal it
does not have. local-stack now sets PC_DISABLE_TUI when stdout is not a
TTY, so the same command renders plain logs and every process starts.

bin/check-local-stack proves it, and pnpm check:ci runs it as check:ci-run
on Linux. It starts the stack the way an agent does — stdout not a TTY,
TERM unset — inside an unprivileged network namespace, waits for every
readiness probe, stops the stack and checks every port is free.

Operator approval: Kiro, 2026-10-06 (QA Q1)
Grafana's built-in preinstall list (grafana-pyroscope-app,
grafana-exploretraces-app) downloads unpinned zips from grafana.com on
every fresh data dir; nothing pins that code and it runs outside the
sandbox. Nothing here needs those apps: the Tempo data source is
provisioned from a file and core Explore renders traces from it.
preinstall_disabled keeps the stack offline.

bin/check-local-stack now asserts Grafana never reaches for a plugin while
the stack runs with outbound networking denied, and kills a half-started
stack so a failed assertion cannot leak processes.

Operator approval: Kiro, 2026-10-06 (QA Q3)
check:ci runs bin/check-local-stack on Linux, but hosted Linux CI runs check:ci split into legs and only macOS runs it whole, where the check is skipped. The new leg runs it on every PR
The leg's first hosted run failed before the stack started: "no
unprivileged network namespace (unshare -rn or bwrap --unshare-net)".
ubuntu-24.04 runners restrict unprivileged user namespaces through
AppArmor (actions/runner-images#10443), and the image maintainers
declined to lift it (actions/runner-images#11489). The leg now sets
kernel.apparmor_restrict_unprivileged_userns=0 before its gate, so the
check can start the stack with the network denied
@systemfsoftware-maker
systemfsoftware-maker removed this pull request from stack #34 October 6, 2026 16:04
@systemfsoftware-maker
systemfsoftware-maker added this pull request to stack #54 October 6, 2026 19:52
@systemfsoftware-maker

Copy link
Copy Markdown
Collaborator Author

Dropped by the 2026-10-06 starter spec: pnpm dev runs alchemy dev, no local trace stack.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant