README.md

services/testing

Shared helpers for the test suite.

This directory is on the Python path via pyproject.toml (pythonpath = ["services"]), so tests import as from testing.yaml_assertions import assert_yaml_equal_except.

Test tiers

The directory is the marker. A test in tests/<tier>/ is marked <tier> automatically by the repo-root conftest.py — there is nothing to declare. Pick the tier by what the test touches:

Tier	Directory	Touches	Runs
unit	tests/unit/	Nothing. Network, subprocess, DB and LLM-client construction all raise UnitTestViolation	Every PR push
service	tests/service/	Real service handlers, with the LLM/HTTP clients mocked	On merge
integration	tests/integration/	Real LLM, Pinecone, Postgres	Manual / nightly
acceptance	tests/acceptance/	Real everything, judged by an LLM (the .md specs below)	Manual / nightly

Two rules of thumb:

• Assert on bytes, not on wording, wherever you can. If the question is "did this content reach the model", that is a service-tier test with a scripted client — put an improbable canary in the input and assert it appears in the request. It is free, deterministic, and needs no human to read it. Reserve acceptance for the questions only a judge can answer: is the answer good.
• A guard you have not seen fail is not a guard. Break the code, watch the test go red, put it back.

Acceptance spec folders

Run one folder's own specs with poetry run pytest <folder>/*.md -s (the bare folder also collects its subfolders). Paths are under services/<service>/tests/acceptance/.

Folder	What it tests
workflow_chat/	The workflow agent alone: generating and editing workflow YAML
job_chat/	The job code agent alone: code for a single step
job_chat/quality_issues/	Whether generated code avoids known code-quality mistakes
job_chat/bugs/	Repros of real job_chat failures
global_chat/job_code/	Questions and code changes about one step, through the router
global_chat/multi_step_changes/	Edits to an existing workflow that the planner splits across agents
global_chat/one_shot_workflows/	A whole workflow with job code from one request
global_chat/gold_workflows/	Real-world workflow patterns, each phrased vaguely and verbosely
global_chat/bugs/	Repros of real global_chat failures

Modules

• yaml_assertions.py — pure-function YAML structural assertions, safe for every tier (unit included).

• judge.py — LLM-as-judge helper for acceptance tests. Evaluates chat-service responses against natural-language criteria under a named judge. Defaults to CLAUDE_SONNET from services/models.py and the general judge.

• judges.py — registry that loads judge configs from judges/<name>.md.

• judges/ — one markdown file per named judge. Each has a # role section (who the judge is and what it evaluates) and a # rules section (universal bullets that apply to every evaluation under this judge). Today: general and openfn_code_quality. Specs select judges via the judges: frontmatter field; default is [general].

• spec_parser.py — parses acceptance test markdown specs (services/<svc>/tests/acceptance/*.md) into Spec dataclasses.

• spec_collector.py — pytest plugin (registered via pytest_plugins in the repo-root conftest.py). Turns each MD spec into a pytest item that builds the service payload, calls the service via ApolloClient, and runs the judge. For inspection, three artifacts are written to a tmp/ folder next to the spec file:

  • <spec_id>.yaml — any workflow YAML in the response (response_yaml, workflow_yaml, content_yaml, or a workflow_yaml attachment).
  • <spec_id>.txt — the response text, prefixed with the agent path (e.g. agents: router -> planner -> job_code_agent).
  • <spec_id>.judges.txt — the judge verdict(s) in the same format printed during the run (per-judge PASS/FAIL header + criteria/flags summary).

  Filenames use __ as the metadata separator (the spec id and extension only use ./-), so they stay splittable. Multi-run specs append __run-N. Pass -E <label> / --experiment=<label> to append __<label> to every captured filename so runs with different settings or dates don't overwrite each other (e.g. tmp/<spec_id>__sonnet-2026-06-29.txt).

  Each spec spends real money, so a run of more than 5 spec runs asks for confirmation first (with the service and judge call counts, and a command per spec folder to run a subset instead), or refuses when there's no terminal to ask. Pass --max-specs=<n> to raise the limit, e.g. for a nightly job.

  Pass --no-judge to skip judging: the service is still called and its output captured, but nothing is graded and every spec passes.

• spec_context.py — prints a spec's context (# settings) as payload JSON, so the simulate-users skill can reuse spec setups without copying them: PYTHONPATH=services poetry run python -m testing.spec_context <spec.md>.

• apollo_client.py — ApolloClient for dispatching to a chat service. Currently a subprocess-based stub; the integration tier will replace its internals with a real HTTP client (same .call() signature, no test changes).

Why under services/ and not a top-level tests/?

Imports across the codebase resolve relative to services/ (see CLAUDE.md → "Python Import Patterns"). Putting helpers here means tests import without path-munging hacks, the same way services do from util import ….

The Bun service auto-discovery in platform/src/util/describe-modules.ts skips this directory because it has no testing.py index file.