bs-score

An LLM finds the defects and quotes the evidence. The bs-score CLI verifies every quote against the artifact it names, then scores what survived. Higher = more bullshit. A fabricated audit is NOT_VALID (not a clean 0), because none of its quotes verified.

Point it at a repo (README vs code), a PR, a PR review, a skill, an agent config, or a prompt.

Install

Two pieces: the skill (the recipe your agent follows) and the bs-score command (what verifies and scores what it emits).

1. The command — one line, every runtime:

uv tool install git+https://github.com/alexhawat/bs-score   # puts bs-score on PATH

2. The skill — clone this repository into your runtime's skills folder, so the root SKILL.md and the checklists/ it links to land together. The folder name is the skill's name, so keep it bs-score:

git clone --depth 1 https://github.com/alexhawat/bs-score ~/.claude/skills/bs-score

Runtime	Personal (every project)	Project (checked in)
Claude Code	~/.claude/skills/bs-score	.claude/skills/bs-score
Cursor	~/.cursor/skills/bs-score	.cursor/skills/bs-score
Codex	~/.codex/skills/bs-score	.agents/skills/bs-score
OpenCode	~/.config/opencode/skills/bs-score	.opencode/skills/bs-score
Grok Build	~/.grok/skills/bs-score	.grok/skills/bs-score
OpenClaw	~/.openclaw/skills/bs-score	<workspace>/skills/bs-score
Hermes	~/.hermes/skills/bs-score	skills/bs-score
Anything else	~/.agents/skills/bs-score	.agents/skills/bs-score

.agents/skills/ is the cross-runtime convention — Codex, OpenCode, Grok Build and OpenClaw all read it, so one clone there covers the four at once. Per-runtime wrapper folders live under agents/ for anyone who keeps the clone elsewhere; they point back at the root SKILL.md rather than duplicating it.

Or skip all of it and paste this at your agent:

Add the bs-score skill from https://github.com/alexhawat/bs-score and learn how to use it.

Other ways to get the command

• uvx --from git+https://github.com/alexhawat/bs-score bs-score … — no install, resolved per run. CI runs that exact command against every commit (uvx-install job), so it is proven, not promised.
• From a clone: uv sync, then uv run bs-score … — or python3 score.py …, which needs no uv and no network.
• uvx bs-score (no --from) will work after the planned PyPI release; today it has nothing to fetch.

LLM-facing context files: llms.txt (index) and llms-full.txt (the whole recipe, weights, and schema in one fetch).

30-second demo

An auditor that made everything up, scored from a clone (git clone https://github.com/alexhawat/bs-score bs-score && cd bs-score && uv sync):

$ bs-score examples/findings.hallucinated.json --repo-root examples/fixture-repo --format md
## Bullshit Score: **NOT_VALID** (NOT_VALID)

- review type: `repo` (profile `repo`)
- target: `examples/fixture-repo (fabricated audit)`
- kept 0 · rejected 4 · deduped 0 · baselined 0
- evidence: verified (path_not_found 1, quote_not_found 3)
- depth: **unreported** — no `audit.files_read` block, so depth is unverifiable
- blast radius: 0 file(s) affected across 10 scanned
- scoring `30ad8b532c34` · schema `749e202f7380`

### Rejected (not scored)

- `h1` — unverified_evidence: quote_not_found: quote is not present in src/api.py
- `h2` — unverified_evidence: path_not_found: src/config.py does not exist
- `h3` — unverified_evidence: quote_not_found: quote is not present in README.md
- `h4` — unverified_evidence: quote_not_found: quote is not present in src/users.py

Hallucinating earns nothing: a finding whose quote is not there is rejected, not scored — four confident fabrications add up to zero. For what a real report looks like, run bs-score examples/findings.valid.json --repo-root examples/fixture-repo --format md (it scores 17).

What it catches

Wrong information first — that is the focus:

• README lies — features, paths, flags, and version requirements the project does not have.
• Phantom features — capabilities asserted with no implementation behind them.
• Stale docs — quick-starts that cannot run as printed, drifted numbers, dead links.
• Fabricated review claims — "verified, no secrets" that nobody checked; praise for code that does the opposite.

Then the classics: logic and state bugs, injection and secrets, broken control flow, unsafe instructions in agent configs and prompts.

The bs-score claims <doc> subcommand catches the mechanical slice of this with no LLM at all — paths, entry points, CLI flags, version strings, Markdown links, and fenced code blocks are checked deterministically, and --emit-findings turns each failure into a pre-verified wrong_claim.

bs-score find --jev is an optional middle path: TypeSafe Jev (System One) judges semantic wrongness on deterministically collected markdown units and emits a scorable findings payload. Jev cannot invent prose — quotes are verbatim spans from the artifact and titles come from fixed templates. Claim-like sentences are filtered in code before any Jev call; each file gets one batched system_one request with keep/type/confidence questions for every unit in that file. The model is pinned to jev-1.13.0; routing applies both answer probabilities and confidence floors from src/bs_score/jev_questions.py. Discovery logs model id and token usage per call. Keep state lean — Jev accuracy drops as irrelevant context grows (see TypeSafe jaggedness guidance). Requires typesafe-sdk (uv sync --extra jev or pip install 'bs-score[jev]') and TYPESAFE_API_KEY.

What it audits

target_kind	What you point it at	Checklist
repo	README claims and a deep code pass	repo
pr / branch	A diff and its callers	pr
review	An existing PR review, graded against the diff	review
skill	A skill pack: SKILL.md + every script it names	skill
agent	Agent/persona configs (.cursor/, .claude/, AGENTS.md, …)	agent
prompt	One or many prompts — system, developer, tool, persona	prompt
docs	A single document — claims vs what exists, no code pass	docs

Why you can trust the number

Mechanism	What it stops
Evidence verification	A quote not present in the named artifact is rejected before scoring. If findings were submitted but nothing verified, the run is NOT_VALID — not a clean score of 0.
Content-addressed dedupe	(type, canonical file, quote fingerprint) is one finding, whether filed as src/a.py:10, ./src/a.py:11, or copy-pasted into seven files.
Pinned weights	Custom weights need --allow-custom-scoring; every report carries scoring_sha256 + schema_sha256, and receipts reproduce byte-for-byte in CI.
Full schema enforcement	Types, enums, minLength, numeric bounds — cross-checked against jsonschema; one malformed finding is rejected on its own.
Root-cause clustering	Findings sharing a cluster id collapse into one charge even when the wording differs in each place, so one defect cannot be filed three ways for triple the points.
Measured blast radius	Every verified quote is swept across the tree, so files_affected is counted by the scorer rather than claimed by the model.
Audit-depth check	audit.files_read is checked against the tree: a docs-only pass is stamped shallow, and a listed file that does not exist earns no credit.

Counting, clustering, and depth

Three mechanisms keep the score from depending on how hard the model looked. Every verified quote is swept across the tree, so files_affected is measured rather than claimed — file the defect once and the report says it is in twenty-six files. Findings sharing a cluster id collapse into one charge, so a root cause worded three ways costs its points once. And a payload's audit.files_read is checked against the tree, so a docs-only pass is stamped shallow and a listed file that does not exist earns no credit; --require-depth turns that into a non-zero exit.

Details, including the --require-depth truth table: docs/counting-and-depth.md.

Scoring

One score. Start at 0, unbounded sum, higher = worse. The five finding types are fixed; the weights depend on the review type, because the same defect does not cost the same everywhere.

type	meaning	repo	pr branch	review	skill	agent	prompt	docs
breaking_bug	Crash, wrong result, data loss, dead workflow	5	6	6	5	4	4	2
security_issue	Auth, injection, secrets, unsafe instructions	4	5	5	4	5	5	3
missing_feature	Asserted capability with no implementation	3	2	3	4	3	2	3
bug	Incorrect behaviour, not catastrophic	3	3	4	3	2	3	1
wrong_claim	A claim the artifact contradicts	2	2	3	3	3	3	5

Weights live in scoring.json, with the rationale for each column in its type_notes. confidence is display-only, and so is the band the report prints: 0 clean · 1–5 minor drift · 6–15 misleading · 16+ bullshit.

Examples

Every number here comes from a committed fixture and is asserted in CI; the artifacts the quotes point at live under examples/.

example	review type	score	shows
findings.valid.json	repo	17	README lies + real bugs; one duplicate merged, one blank-evidence finding rejected
findings.blast.json	repo	5	One root cause reported three ways — merged, then counted across every file it reaches
findings.shallow.json	repo	2	A docs-only pass that also claims to have read a file that does not exist
findings.hallucinated.json	repo	NOT_VALID	Four confident fabrications, all rejected

bs-score examples/findings.valid.json   --repo-root examples/fixture-repo
bs-score examples/findings.blast.json   --repo-root examples/fixture-repo
bs-score examples/findings.shallow.json --repo-root examples/fixture-repo --require-depth

Eight more — review, skill, agent, prompt, docs, two in-the-wild repositories, and a French/Japanese pair proving quotes verify verbatim in any language — are documented with their commands in examples/README.md. Each writes the same report as its committed examples/score.*.json receipt.

CLI

bs-score FINDINGS [--repo-root DIR] [--sources PATH ...] [--no-verify]
                  [--require-evidence] [--strict-lines] [--line-window N]
                  [--no-scan] [--require-depth]
                  [--fold-case] [--fail-over N] [--format json|md|sarif]
                  [--baseline FILE] [--write-baseline FILE] [-o FILE]
                  [--scoring FILE --allow-custom-scoring] [-v|-q]

bs-score claims DOC [--repo-root DIR] [--check-links] [--emit-findings]
                    [--ignore GLOB ...]

bs-score find --jev [--repo-root DIR] [--target-kind KIND] [--glob GLOB ...]
                    [--document PATH ...] [--score] [-o FILE]

Highlights:

flag	effect
--sources	Register non-file evidence (prompt files, review bodies) so their quotes verify too
--fold-case	Case-insensitive quote matching (str.casefold); exact is the default
--fail-over N	Exit 1 when the score exceeds N — the CI gate
--require-depth	Fail when a review type that needs a code pass cannot prove one
--no-scan	Skip the blast-radius sweep (the count is then unreported)
--format sarif	SARIF 2.1.0 for GitHub code scanning
--baseline / --write-baseline	Accept known findings once; gate only on new ones
claims	LLM-free claim checks on a document; --emit-findings for a scorable payload
find --jev	Jev-based finding discovery for repo / docs; optional --score chains the scorer
--ignore	Spans that are not claims about this tree — a file a command writes, a version a changelog records. Reported as skip naming the pattern, never dropped

Exit codes: 0 scored and within threshold · 1 over --fail-over (or claims found a provably false claim) · 2 unusable input.

Integrations

GitHub Action (composite, action.yml):

- uses: alexhawat/bs-score@main
  with:
    findings: findings.json
    fail-over: "10"

It writes the report as a file in the format you ask for and exposes score, band and report outputs. docs/github-action.md has the PR-comment and code-scanning recipes.

SARIF — bs-score findings.json --format sarif -o report.sarif, then the codeql-action upload-sarif step puts findings into code scanning.

pre-commit — hook id bs-score (see .pre-commit-hooks.yaml).

MCP server (experimental) — uv pip install 'bs-score[mcp]', then bs-score-mcp exposes audit and score tools over stdio.

Self-audit — the v1 audit of this repository (examples/findings.self.json) is replayed as a CI gate on every PR (scripts/check_dogfood.py, which asserts the score is 0): all eight defects it documented must stay fixed — reintroduce one, its quote verifies again, and CI goes red.

Development

uv sync --extra dev
uv run pytest
uv run ruff check .
uv run python scripts/gen_agent_wrappers.py --check   # wrappers are generated
uv run python scripts/check_receipts.py               # receipts reproduce exactly
uv run python scripts/gen_llms_txt.py --check         # llms.txt files are generated

Contributor rules for agents: AGENTS.md. Releasing (owner-only): RELEASING.md.

FAQ

Is the score objective? No — it's verifiable. An LLM still chooses what to look for and how bad it is. What bs-score removes is the unverifiable part: every point traces to a quote that provably exists in the artifact, under pinned weights, with a sha256 receipt for the weights, the schema, and the findings. You can replay any score and get the same number.

What stops the agent gaming it? Three things. Rejection: an invented quote is rejected, so padding the findings list with fabrications scores zero. Dedupe: the same defect filed five ways is one finding. Pinned weights: the agent can't propose its own points — findings JSON containing score or points is rejected outright, and custom weight files require an explicit --allow-custom-scoring flag that the report discloses.

Why does higher mean worse? Because it counts bullshit, not quality. A 0 means everything the auditor said checked out — or the auditor said nothing verifiable, which the report shows separately (rejected, not_verifiable). Read the band, then read the kept findings.

License

MIT