Repository navigation
run: add AAR v0.8 EXECUTE-boundary frozen-ZIP eval readout - #20
Merged
Merged
Conversation
Run identity: claude-sonnet-4-6 @ temperature 0, 25 fixtures (24 classes per PR #19 design; EB-24 is a paired class), 25/25 ok. System prompt: eval/SKILL.md (Constraint Enumeration Protocol), read-only. The AAR v0.8 frozen ZIP lives outside this repo; this run measures CE-skill behavior on AAR-style EXECUTE-boundary prompts, NOT a re-run against the AAR frozen skill bundle. Deterministic scorer (eval/lib/scorerParity.js, unmodified) summary on 24-case scored denominator (EB-06 excluded as INVALID_CASE): primitiveMatch: 19 / 24 = 79.2% bOverEnum (=0): 21 / 24 = 87.5% (3 fail: scope_extension on EB-14, EB-16, EB-24A) hallucinatedToolIO: 24 / 24 = 100.0% checklistExposed: 6 / 24 fired (proposed gate ≤ 1) executeBoundaryViolation NOT scored (not implemented; see design §9). The proposed gates in design §10 are reference-only — not pre-registered against this run, not currently active. Repo vs. local-only split: per .gitignore policy ("Frozen/failed eval result snapshots — not for publication"), raw responses, scored output, and the eval/results_frozen_aar_v0_8_execute_boundary_<runid>/ folder are on local disk only. SHA-256 values in the readout (and the manifest's cases_sha256) are the audit anchors, consistent with the existing v0.8 canonical-scorer-replay evidence pattern. Manifest moved into a subfolder (eval/cases/aar_v0_8_execute_boundary/) because eval/lib/loadCases.js strictly requires all top-level .json files in eval/cases/ to be case arrays — placing a manifest .json there breaks the test suite. Subfolder is invisible to the non-recursive readdirSync; test suite remains 66/66. This does NOT modify v0.8, does NOT create v0.9, does NOT supersede the v0.8 evidence trail, and is NOT a new automated benchmark pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…xtures, deterministic scorer System prompt: frozen AAR v0.8 ZIP (agent-authority-router-skill_v0_8.zip, SHA 55233ad67853748f0ad62d228d166f496c03e3955cac313be064d554cbfe1b0c), extracted in-memory at runtime via PowerShell Expand-Archive to an OS temp dir, read into Node, and torn down before any API call. No extracted skill content was written to the repo. Run identity: claude-sonnet-4-6 @ temperature 0, 25 fixtures (24 classes per PR #19 design; EB-24 is paired), 25/25 ok. Deterministic scorer (eval/lib/scorerParity.js, unmodified) summary on 23-case scored denominator (EB-06 and EB-07 excluded as INVALID_CASE): primitiveMatch: 19 / 23 = 82.6% bOverEnum (=0): 23 / 23 = 100.0% hallucinatedToolIO: 23 / 23 = 100.0% checklistExposed: 23 / 23 = 100.0% The 4 primitiveMatch misses (EB-08, EB-14, EB-20, EB-24B) are documented in the readout as scorer expressivity gaps, not AAR behavioral defects. executeBoundaryViolation NOT scored (not implemented; see design §9). The proposed gates in design §10 are reference-only — not pre-registered against this run, not currently active. Runner change (eval/scripts/run-execute-boundary.js): added optional SKILL_ZIP_PATH env-var override. When set, the runner extracts SKILL.md from the named ZIP into an OS temp dir using PowerShell Expand-Archive, reads it, and removes the temp dir before any API call. When unset, the runner falls back to eval/SKILL.md (prior behavior). No other behavioral change to the runner. Repo vs. local-only split: per .gitignore policy ("Frozen/failed eval result snapshots — not for publication"), raw responses, scored output, and the eval/results_frozen_aar_v0_8_execute_boundary_<runid>/ folder are on local disk only. SHA-256 values in the readout are the audit anchors, consistent with the existing v0.8 canonical-scorer-replay pattern. This does NOT modify v0.8 (frozen ZIP SHA unchanged), does NOT create v0.9, does NOT supersede the v0.8 evidence trail, and is NOT a new automated benchmark pass. ANTHROPIC_API_KEY was bridged from User-scope registry to process scope at run time and never written to any file or memo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The manifest's claim_boundary field was authored for the prior CE-skill run (commit 3dcde05) and stated the system prompt was eval/SKILL.md. PR #20's actual run (commit dd9aceb) used the frozen AAR v0.8 ZIP (SHA 55233ad6…b0c), as documented in the readout. The two artifacts in the same PR therefore contradicted each other. Updated claim_boundary to state the frozen ZIP was used, with full SHA-256, in-memory-extract handling, and a cross-reference to the readout. Also explicitly disclaims the prior CE-skill run as a separate, non-load-bearing comparison surface. No runner, scorer, case file, or readout content changed. No evals rerun. No v0.8/v0.9/curated/raw/frozen/quarantined artifacts touched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds the first frozen-ZIP EXECUTE-boundary regression run derived from the PR #19 design artifact.
Run summary
55233ad6…b0c(matches canonical evidence trail).primitiveMatchbOverEnum(target=0)hallucinatedToolIO(target=0)checklistExposed(target≤1)primitiveMatchmisses: EB-08, EB-14, EB-20, EB-24B.Claim boundary
executeBoundaryViolationremains future/proposed work and is not scored here.Security / hygiene
.gitignore:19. SHAs in the readout provide the audit trail, matching the existing v0.8 canonical-scorer-replay evidence pattern.55233ad6…b0c).Diff scope
Two commits, five files:
eval/cases/aar_v0_8_execute_boundary_cases.jsonl— 25 fixtures (24 classes; EB-24 paired)eval/cases/aar_v0_8_execute_boundary/manifest.json— manifest (subfolder placement avoids breakingloadCasesstrict-array invariant)eval/scripts/run-execute-boundary.js— minimal runner with optionalSKILL_ZIP_PATHenv-var support for in-memory ZIP extractioneval/scripts/score-execute-boundary.js— scorer glue usingeval/lib/scorerParity.jsunmodifiedeval/public_notes/aar_v0_8_execute_boundary_readout.md— readout memo