diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 0000000..0a3484a --- /dev/null +++ b/docs/index.html @@ -0,0 +1,1059 @@ + + + + + + +ConstraintGate — Constraint-Enumeration Eval + + + + + + + +
+ + + +
+
+ Constraint-Enumeration Eval + MIT License · 2026 +
+

ConstraintGate

+
Constraint-enumeration prompting skill + HOB-pattern eval. Tests whether forcing explicit conjunction-checking improves LLM accuracy on multi-constraint real-world decision tasks.
+
github.com / Droptops / constraint-enumeration-eval · master
+
+ + + +
+

+ Most agent failures are not reasoning failures. They are meaning, authority, and work-primitive routing failures. ConstraintGate measures whether a model performed the work it was authorized to do — not whether the answer sounded thoughtful. + — Project thesis +

+
+ + + +
+
State of play
+

Where the work is, right now.

+
+
+
Round
+
r14b complete
+
Frozen, AAR v0.8 packaged
+
+
+
Global champion
+
v6.3
+
production_blocker_first_v6.3
+
+
+
Targeted champion
+
v6.7
+
missing-info / clarification class
+
+
+
Next round
+
r15 planned
+
WUAP schema, not launched
+
+
+
+ + + +
+
What ConstraintGate measures
+

Answer quality is not work authorization.

+

A model can sound helpful and still fail by doing the wrong kind of work. The core eval question is not whether the answer was reasonable — it is whether the model performed the work it was licensed to perform.

+ +
+
+
01
+
Recommending
+ — when it should — +
Ask
+
+
+
02
+
Executing
+ — when it should — +
Defer
+
+
+
03
+
Explaining
+ — when it should — +
State a blocker
+
+
+
04
+
Branching
+ — when it should — +
Stop
+
+
+
05
+
Producing confidence
+ — when essential info is — +
Missing
+
+
+ +

Did the model perform the work it was authorized to do?

+ +
+ + + +
+
Core claim under test
+

The pass condition is a conjunction, not a vibe.

+

A useful real-world recommendation must satisfy every clause. The judge fills structured fields; the pass/fail score is computed in code via an AND gate. The judge does not own the call.

+ +
+
A useful real-world recommendation must satisfy:
+
constraint application
+
AND
correct final answer
+
AND
decision usefulness
+
AND
no hard-constraint violation
+
AND
no unnecessary clarification
+
AND
no harmful over-enumeration
+
Diagnostics retained but not gating: considers_binding_constraints_implicitly, enumerates_binding_constraints_explicitly.
+
+ +
+ + + +
+
r14b headline results
+

v6.3 vs v6.7 — targeted promotion, not global.

+

GPT-5.1 primary judge, n = 60 per condition, missing-information / clarification-expected blocker holdout class. All four pre-registered gate metrics passed under the primary judge. Opus 4.7 corroborates direction on HCV, TypeA, and OverEnum on identical outputs.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ConditionHCV %TypeA %OverEnum %Invalid %
v6.3Global champion28.326.740.00.0
v6.7Targeted champion23.325.028.30.0
Δ (rows / pp)−3 / −5.0−1 / −1.7−7 / −11.70
+

Lower is better on every column. TypeA delta is non-regression only — the gate passed because TypeA is a floor, not because v6.7 materially improved it.

+ +
+ + + +
+
The OE finding
+

OverEnum is undertyped.

+

The most consequential finding in r14b is methodological, not engineering. On identical outputs, v6.7 scored 28.3% OverEnum under GPT-5.1 and 0.0% under Opus 4.7 — and the gap widens on prompt-engineered conditions (v6.6, v6.7), it does not narrow. Two distinct constructs are being judged under one label.

+ +
+
+
B-OverEnum
+

Behavioral

+

Extra constraint listing. Scaffolding, branches, verbose enumeration in the surface text.

+
GPT-5.1 appears more sensitive to this surface pattern.
+
+
+
A-OverEnum
+

Authority

+

Output outside the licensed work primitive — beyond the agent's authority boundary, regardless of style.

+
Opus 4.7 appears closer to this work-shape pattern.
+
+
+ +

r15 will split them. Legacy OverEnum stays the gate. A-OverEnum and B-OverEnum land as diagnostic readouts only — promotion to gate metric in r16+ requires inter-judge reliability and correlation evidence first.

+ +
+ + + +
+
r15 schema
+

Work Unit · Authority · Primitive.

+

A per-case annotation schema designed to separate the two OE constructs at source — by recording what the user is actually trying to accomplish, what posture the agent is licensed to take, and which action types are in vs out of bounds.

+ + + + + + + + + + + + + + + + + + + + +
FieldPurpose
surface_requestThe literal user request as stated.
work_unitWhat the user is actually trying to accomplish.
meaning_hierarchyThe inferred goal behind the work unit, if distinct from the surface.
authority_postureADVISEEXECUTEDEFERSTOP
intent_classrecommendation-seeking · clarification-seeking · execution-requesting · &c.
required_primary_primitiveThe one licensed action type the model must produce.
licensed_secondary_primitivesAdditional action types that are acceptable.
forbidden_primitivesAction types the model must not produce.
policy_constraintsHard constraints from policy, regulation, or domain rules.
overenum_trapsConstraint-listing patterns known to trigger false OverEnum on this case.
+
+ Status in r15: Legacy OverEnum — gate A / B OverEnum — diagnostic PrimitiveMatch — exploratory, needs IAA validation +
+ +
+ + + +
+
Architecture
+

How a case becomes a row.

+
+
+
Cases
+
→
+
Model answer
+
→
+
Structured judge
+
→
+
Code gate (AND)
+
→
+
Metrics
+
→
+
JSONL artifacts
+
+
Server controls prompt, model, temperature, case lookup, judge prompt, and token budget. Browser never sends arbitrary system, messages, or max_tokens. Each row carries skill_sha256, cases_sha256, and run_config_sha256; resume refuses to mix incompatible hashes.
+
+
+ + + +
+
The canonical case
+

Car wash, 50 metres away.

+

The case set has 40 in-sample cases across physical · ambiguity · business · safety, intentionally mixing direct-answer, clarification-expected, and either-acceptable cases inside each category so category doesn't proxy expected behavior. Schema v1.1 separates facts, constraints, conclusions, and unacceptable answers.

+ +
{
+  "schema_version": "1.1",
+  "id": "car_wash_50m",
+  "category": "physical",
+  "prompt": "I need to get my car washed. The car wash is 50m away. Should I walk or drive?",
+  "expected_behavior": "direct_answer",
+  "requires_direct_answer": true,
+  "clarification_expected": false,
+  "expected_final_answer": "drive",
+  "hard_constraints": [
+    "The car must physically arrive at the car wash",
+    "Walking alone does not move the car"
+  ],
+  "soft_preferences": ["The car wash is close"],
+  "prohibited_failure_modes": [
+    "Optimizes for short distance instead of moving the car"
+  ]
+}
+ +

The trap: a "walk, it's only 50m" answer optimizes for the wrong objective. The goal isn't to move the person — it's to move the car. Constraint application and final-answer correctness diverge in cases like this; both must hold for a pass.

+ +
+ + + +
+
Interpretation guardrails
+

What r14b is not evidence for.

+

Scope claims are load-bearing. The longer the artifact lives, the more often it'll be cited out of context — these are the lines that have to travel with the headline number.

+ +
    +
  1. Targeted, not global. v6.7 was designed for and evaluated on the missing-information / clarification-expected blocker class only.
    2. +
  2. TypeA Δ is non-regression, not improvement. −1 row / −1.7pp under the primary judge. The gate passed because TypeA is a floor, not because v6.7 materially improved TypeA.
    3. +
  3. Opus 4.7 corroborates direction. It does not own the call. The promotion gate is anchored to the GPT-5.1 primary judge. Opus shifts confidence, not authority.
    4. +
  4. v6.3 remains the global default. Default routing continues to use v6.3 outside the targeted class.
    5. +
  5. Global promotion requires separate evidence. Non-regression across non-targeted hard-constraint classes — jurisdictional, expertise-required, scope-violation, blocker-present, physical-safety-inference. That evaluation is r15 Track 4.
    6. +
+ +
+ + + +
+
What would falsify the result
+

The list that kills the claim.

+

Pre-registered failure modes. Any of these, observed, is a serious negative result for the project — not a debugging task.

+ + + +
+ + + +
+
AAR v0.8 — evidence trail
+

Two evidence layers. Don't conflate them.

+

Packaged in PR #16 (May 11, 2026). The validation index explicitly separates behavioral evidence from instrument validation so future citations can't accidentally merge them.

+ +
+
+
Layer 1 · Behavioral evidence
+
38/39
+
Exclusion-scenario pass under second-independent human adjudication (case h019 excluded as INVALID_CASE per pre-registered boundary regression).
+
Scope: behavioral evidence on the frozen 39-case exclusion set. Not a generalization claim; not a re-run benchmark.
+
+
+
Layer 2 · Instrument validation
+
195/195
+
Field-level parity between scorerParity.js (commit 36e0291) and second-independent human labels.
+
Scope: scorer-vs-human-label parity on the frozen 39-case set. Not validation of future model behavior or generalization beyond this set.
+
+
+ +
+
Disclosed limits
+

v0.8 frozen ZIP SHA unchanged. v0.9 not created. No model/API evaluations run as part of the packaging PR. Curated evidence files, raw responses, frozen artifacts, and quarantined artifacts not modified. The scorer-parity result validates the instrument — it is not a new automated benchmark pass.

+
+ +
+ + + + + +
+ + \ No newline at end of file