Integrations for TypeSafe Jev — the System One decision model — across fourteen packages spanning agent harnesses, CI actions, CLIs, and editor extensions.
Jev is not a chat model. You send it a state plus typed questions and it returns calibrated probabilities your code acts on directly. That makes it the right tool for routing, ranking, gating, and verification — anywhere you currently pay a chat model to emit JSON you immediately parse.
| Package | Harness | Transport | Capabilities |
|---|---|---|---|
@jev-harness/core |
— | — | Client, primitives, 23 patterns, redaction, budget guard, caches, decision log |
@jev-harness/omp |
Oh My Pi | Extension | 5 tools + 3 hooks (incl. verbatim compaction) |
@jev-harness/mcp |
Any MCP client | MCP over stdio | 7 tools |
@jev-harness/claude-code |
Claude Code | Plugin | PreToolUse gate + prompt skill routing |
@jev-harness/pi |
Pi | Extension | 5 tools + 2 hooks |
@jev-harness/eval |
— | jev-eval + jev-tune CLIs |
Labeled-dataset evaluation, calibration, threshold sweeps, golden baselines |
@jev-harness/github |
GitHub Actions | Action | jev-review: advisory PR review comment (fail-open) |
@jev-harness/jev-gate-action |
GitHub Actions | Action | Semantic acceptance gate: destructive + secret-leak + risk on a PR diff |
@jev-harness/pr-triage-action |
GitHub Actions | Action | PR triage: auth impact, risk score, review routing |
@jev-harness/kit |
— | — | Shared adapter foundation: env config, result envelope, core-pattern plumbing |
@jev-harness/playground |
— | npm run playground |
Local web playground: state + questions → live probabilities |
@jev-harness/cli |
CI / subagent workflows | jev + jev-gate CLIs |
jev ask/models/eval terminal access + jev-gate semantic acceptance gate |
jev-py |
Python | stdlib-only client | Async client, 12+ patterns, eval + tune (parity-tested against the TS metrics) |
@jev-harness/vscode |
VS Code | Extension | Destructive-change gate + claim verification, fail-open |
Cheaper than a chat model for closed-set decisions. Input tokens bill at $0.042/Mtok and output tokens are free, because Jev emits probabilities, not prose. One call can carry many questions at once — they are evaluated independently against the same state, so batching is nearly free.
Typed by construction. A choice question cannot return a string outside its criteria map. It can still be wrong — calibration is not correctness — so keep thresholds and side effects in your own code and validate on your own labeled data.
| Primitive | Asks | Returns |
|---|---|---|
noul |
Whether a condition holds | noul: P(yes), 0–1. No separate confidence field; ~0.5 means yes and no are equally likely. |
choice |
One of a defined set | choice (winner key) + probabilities + confidence |
score |
Degree along an ordered rubric | score (probability-weighted, can land between levels) + probabilities + confidence |
Mix them freely in one call:
import { askJev, choice, noul } from "@jev-harness/core";
const res = await askJev(
{ apiKey: process.env.TYPESAFE_API_KEY! },
{
diff: "Changed the login redirect URL and session cookie flags.",
},
{
touches_auth: {
type: "noul",
instructions: "Does this change affect authentication or session security?",
},
risk: {
type: "score",
instructions: "Security risk level",
criteria: ["None", "Low", "Moderate", "High", "Critical"],
},
route: {
type: "choice",
instructions: "Who should review this?",
criteria: {
auto: "No human needed",
peer: "Normal review",
security: "Needs a security reviewer",
},
},
},
);
noul(res, "touches_auth"); // 0.97
score(res, "risk"); // { score: 2.02, ... }
choice(res, "route").choice; // "security"@jev-harness/core ships the patterns we validated in production, so every harness behaves identically:
routeSkill— pick the right skill for a request. Sending descriptions matters: names alone route "test the login page in a browser" to a desktop-automation skill instead of the browser-testing one.judgeDestructive— gate a tool call on whether it destroys data. Tuned threshold 0.5 (see the golden baseline below).chooseBrowserAction— pick one browser action from a numbered element table. Advisory only; the caller validates the index against the live snapshot.pickTool— choose one tool from a candidate set and flag confirmation-worthy side effects.rankCandidates— score a list of strings against a task, best-first.gateInjection— gate untrusted tool results and fetched pages for prompt injection before they reach the model. Threshold 0.7.verifyStep— did the finished work actually satisfy the task? Jev as a cheap critic; loop only whendoneis false.needsClarification— detect a genuine fork (two materially different readings) so the agent asks before burning tokens on a wrong guess.isDuplicate— semantic dedup of memory entries and tool results in one batched call.routeEffort— decide whether a task deserves the expensive model or the cheap fast tier.
Safety and verification patterns (verifyClaim, detectPromptInjection, needsMoreContext, judgeRegression, triageUrgency, chooseSubagent, debateJudge) and devops patterns (commitGate, migrationSafety, testPrioritizer, secretLeak, dedupeItems, logSeverity) ship in the same package — every one is a single batched call with overridable thresholds.
- Redaction —
JevConfig.redact: truescrubs likely secrets and direct identifiers (AWS keys, JWTs, private keys, GitHub/Slack tokens,sk-keys, auth headers, env secret assignments, URL credentials, connection strings, emails) fromstatebefore the request leaves the machine. Fail-open, opt-in at the client level; the OMP and Claude Code hooks enable it by default (OMP_JEV_REDACT=0/JEV_REDACT=0to disable). - Budget guard —
createBudgetGuardcaps requests per rolling window and per lifetime, so a runaway hook loop becomes a loud error instead of a bill. - Persistent cache —
createPersistentCache+withPersistentCacheserve exact-repeat judgments from disk across restarts, with hit-rate stats.
createDecisionLog records one structured row per judgment — kind, model, a stable digest of (kind, state, questions), answers, threshold, action, latency — and compare() computes agreement/flip-rate between two logs over matching digests. That is the raw material for threshold and model-version tuning: set OMP_JEV_DECISION_LOG to a path and the OMP gate appends every verdict as JSONL.
Hooks fire per tool call, so core ships the cost controls that keep them cheap:
createCachedClient— TTL cache keyed on (model, state, questions). Repeated identical judgments — the destructive gate seeing the same call twice — stop costing requests.createCoalescer— merges concurrentaskcalls that share the same state into ONE batched request; Jev evaluates the merged questions independently, so answers are identical to separate calls.withFailMode— make the safety policy explicit at each call site:"open"(allow on error),"closed"(deny on error), or"throw".
@jev-harness/kit is the shared adapter foundation — the OMP and Pi adapters are both built on it. It owns the host-independent parts: env credential resolution, the { content, details } tool-result envelope, fail-open error rendering, core-pattern call plumbing, and the lexical skill prefilter. Bring your own host schemas (zod, typebox, …) and hook wiring; the kit does the rest.
Tuning question wording is the main design activity — @jev-harness/playground makes it fast:
npm run playground # → http://localhost:4173Paste a state, pick a preset (destructive gate, PR triage, injection gate, skill routing), edit the questions, and watch the probabilities move. When a question looks right, validate it on labeled data with eval.
The same workflow lives in the terminal via the jev CLI:
jev ask --questions '{"gate":{"type":"noul","instructions":"destructive?"}}' # inline questions
jev ask --state state.json --questions questions.json # files
jev eval --dataset cases.jsonl --out report.json # calibrationjev ask prints the raw JSON response — probabilities per question, usage included — so question wording can be iterated without an adapter.
Every threshold in this repo is a starting point, not ground truth. @jev-harness/eval runs a labeled dataset through your questions and reports accuracy, Brier, AUC, ECE, a reliability diagram, and a max-F1 threshold sweep per question:
TYPESAFE_API_KEY=... jev eval --dataset cases.jsonl --out report.jsonThe repo ships its own benchmark, in splits that are kept apart on purpose (packages/eval/golden):
| Split | File | Cases | Slices | Questions |
|---|---|---|---|---|
| dev / tuning | destructive-gate.json |
78 | core, obfuscation, false-positive-trap, paraphrase | destructive (tool call) |
| holdout | destructive-gate.holdout.json |
89 | core, obfuscation, steering, distractor, false-positive-trap, paraphrase | destructive (tool call) |
| dev / tuning | merge-gate.json |
41 | destructive, benign, secret, placeholder | destructive + secret_leak (PR diff) |
The dev split is what question wording may be iterated against. The holdout is the generalization number: AUC 0.998, Brier 0.018, precision 1.00, recall 0.980 (~$0.0017 per run). The dev split is deliberately easier and sits at 1.00 across the board — exactly why it is not the number to quote.
Slices exist because a healthy aggregate hides a broken slice. They cover clear cases, obfuscated commands (quoting, command substitution, wrappers, globs, variable indirection, encoded payloads — MITRE T1027.010), injected steering text that argues for its own classification, distractor context, false-positive traps (dry runs, kill -0, dd … of=/dev/null), placeholder-vs-real credentials, and paraphrase pairs whose probabilities must not drift apart.
A vitest gate re-derives every number from the committed baselines and enforces floors per slice (negative-only slices are gated on zero false positives); the manual live-eval workflow re-runs every dataset against the real API and fails on aggregate or slice regressions. Re-record with node packages/eval/scripts/record-baseline.mjs <dataset> <question> [out.json].
Two results worth quoting:
- Question wording is a design surface. The first recorded wording named only data-destruction examples, so system-abuse commands scored low (
chmod -R 777 /0.37, a fork bomb 0.17) and the 0.75 threshold missed 5 of the 20 destructive cases. Naming system abuse as well took the set to AUC 1.000 with no misses, and moved the threshold to 0.5 — the middle of the 0.4–0.6 plateau, which leaves margin for run-to-run variance. - Thresholds move a lot when you measure them. On
merge-gate.jsonthe PR-diff destructive question scored precision 1.00 / recall 0.42 at the shipped 0.75 (it missed 7 of 12 labeled destructive merges); the measured mid-gap default is 0.12 (precision 1.00, recall 0.92), and the secret-leak default moved to 0.07 (precision 1.00, recall 1.00). Both are still advisory unless a repo opts intofail_on_block.
Every adapter resolves credentials the same way:
| Variable | Required | Default | Purpose |
|---|---|---|---|
TYPESAFE_API_KEY |
yes | — | Bearer token from console.typesafe.ai |
TYPESAFE_BASE_URL |
no | https://api.typesafe.ai |
Override the API host |
TYPESAFE_DEFAULT_MODEL |
no | jev-latest |
Pin a model version once thresholds are tuned |
JEV_TIMEOUT_MS |
no | 15000 |
Per-request timeout |
JEV_REDACT / OMP_JEV_REDACT |
no | on (hooks) | "0" disables state redaction |
OMP_JEV_MAX_CALLS_PER_MIN |
no | 120 |
Rolling-window budget for the OMP adapter; 0 disables |
OMP_JEV_CACHE_DIR / OMP_JEV_CACHE_TTL_MS |
no | ~/.omp/cache/jev-harness, 24h |
Persistent judgment cache |
OMP_JEV_DECISION_LOG |
no | — | Path to a JSONL file for gate decision records |
Credentials are read from the environment and never logged.
- Fail open by default. A Jev outage must never block your agent. Every hook catches its own errors and resolves to "allow".
- Advisory, not authoritative. Tools report a decision; code executes it. The one exception is the destructive gate, which is explicitly a veto.
- State redaction before transport. Tool input, diffs, and history routinely embed secrets; they are scrubbed before any request leaves the machine.
- Budget-capped hooks. A runaway loop hits the budget guard and fails open loudly instead of quietly spending.
- No mid-conversation model switching. Measured: switching models mid-stream invalidates the provider prompt cache and cost one deployment
$19.53across 309 requests. - Append-only context injection. Skill hints are appended to the user turn, never used to rewrite the system prefix.
npm install # add --include=dev when NODE_ENV=production
npm run build # core first, then every adapter (+ committed bundles)
npm run typecheck
npm test
npm run lint # ESLint
npm run format # Prettier
npm run qa # build + typecheck + lint + format:check + testSee CONTRIBUTING.md for the release flow and the rules on committed build artifacts, and SECURITY.md to report a vulnerability.
Apache-2.0. See LICENSE.