Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -333,6 +333,17 @@ record-zero replacements after review, merge, and a new exact-commit owner GO.
The successful seeds must not be rerun, and aggregation remains separately
unauthorized. See
`docs/f2_f3_fixed_checkpoint_batch_stability_repair.md`.
The separately authorized issue-#194 replacements then completed seeds 3407
and 3409 with exit code zero and zero restarts. Together with the three
issue-#192 completions, all five unselected-checkpoint seed pairs are now
available privately. No aggregate was computed during monitoring. Issue #196
prepares one separately gated CPU-only audit that will validate the 20 unique
selected/unselected prediction files and retained checkpoint identities before
assembling fixed-epoch-1, fixed-epoch-2, and dev-selected policy statistics.
Preparation and merge authorize no Kubernetes object, PVC or prediction read,
metric, inference, training, checkpoint selection or reselection, retry,
QALB-test access, diagnostic, prompt/parser change, or XG. One aggregate Job
requires a fresh owner GO naming the exact merged commit.
Issue #116's separately authorized, corpus-text-free runtime probe subsequently
completed on phone-verified Kaggle account `thgh15`. It confirmed exactly one
Tesla P100, CUDA 12.8, and importable preinstalled PyTorch 2.10.0+cu128, but
Expand Down
15 changes: 15 additions & 0 deletions PLANS.md
Original file line number Diff line number Diff line change
Expand Up @@ -910,3 +910,18 @@ or XG.

Issue #190's single authorization is consumed. Do not rerun the audit or use
the result to change the primary parser, predictions, or score.

## Issue #196 — prepare fixed-checkpoint sensitivity aggregation

- [x] Freeze the three completed source attempts and executable commits.
- [x] Reuse 20 unique selected/unselected prediction files across three policies.
- [x] Require exact training checkpoint, prediction, count, schema, and alignment gates.
- [x] Prepare one CPU-only, write-once, no-retry aggregate Job.
- [x] Pass focused and full repository validation plus Kubernetes dry-runs.
- [ ] Obtain review and merge the repository-only preparation.
- [ ] Obtain a fresh exact-commit owner GO before one aggregate Job.

Issue #196 authorizes repository preparation only. It authorizes no Kubernetes
object, PVC or prediction read, metric, inference, training, checkpoint
selection or reselection, retry, QALB-test access, diagnostics, prompt/parser
change, or XG.
49 changes: 49 additions & 0 deletions docs/f2_f3_fixed_checkpoint_aggregate_protocol.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# F2/F3 fixed-checkpoint aggregate protocol

Status: issue #196 preparation only; no execution is authorized.

## Purpose

This CPU-only audit assembles the post-hoc five-seed sensitivity table for
fixed epoch 1, fixed epoch 2, and the frozen common-development checkpoint
selection rule. It performs no inference and does not change any checkpoint.
The original seed-3407 P100 comparison remains the primary experiment.

## Frozen private sources

- selected checkpoints: issue #183 attempt `5155890101`, evaluation commit
`e004e625a00c9c1c6fac7e2dbc0e7bc450fbad17`;
- unselected checkpoints for seeds 3408, 3410, and 3411: issue #192 attempt
`5157509573`, executable commit
`6b77efafd53660d2b98557b93cff983e91dbbf27`;
- unselected replacement checkpoints for seeds 3407 and 3409: issue #194
attempt `5158062318`, executable commit
`3b2a30aa994071d5a51a51f62ee31df6cd13d958`.

The audit validates the retained training completion records and exact adapter
hashes without exposing development losses. It then validates 20 unique
prediction files: two arms, two epoch checkpoints, and five seeds. The three
reported policies reuse those files by exact checkpoint identity.

## Validation and output

Before computing any statistic, the audit requires exact source attempts and
commits, complete 511-row counts, the frozen test hash, prediction hashes,
schema, paired order, batch-16 RTX 3090 identity, pre-test gate success, and
the absence of retry, training, QALB-test use, prompt/parser change, or exposed
development values. It recomputes each source summary's counts and paired
statistics from private rows.

The public result contains only corpus-text-free per-seed counts, accuracies,
policy means, sample standard deviations, F3-minus-F2 differences, source
hashes, and validation flags. Predictions, record IDs, corpus text, model
responses, adapters, private logs, and development values remain on the PVC.

## Execution boundary

The manifest contains one write-once, no-GPU Job with `backoffLimit: 0` and a
one-hour deadline. Preparation and merge authorize no Kubernetes object, PVC
read, prediction read, metric, model loading, inference, training, checkpoint
selection or reselection, retry, QALB test, diagnostics, prompt/parser change,
or XG. One execution requires a fresh owner GO on issue #196 naming the exact
merged commit and confirmation `AGGREGATE_F2_F3_FIXED_CHECKPOINT_RESULTS`.
36 changes: 36 additions & 0 deletions results/f2_f3_fixed_checkpoint_aggregate_gate_audit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# F2/F3 fixed-checkpoint aggregate gate preparation audit

## Outcome

Issue #196 prepares, but does not execute, one CPU-only write-once aggregate
audit for the completed issue-#183, issue-#192, and issue-#194 sources. It
validates 20 unique private prediction files, the retained checkpoint
identities, source hashes, 511-row contracts, and record alignment before
assembling fixed-epoch-1, fixed-epoch-2, and dev-selected policies.

The proposed public output is corpus-text-free. It retains only aggregate
counts and statistics, source hashes, checkpoint names, and validation flags.
No private prediction, record ID, corpus text, response, adapter, log, or
development value may leave the PVC.

## Authorization

Preparation and merge authorize no Kubernetes object, PVC read, prediction
read, metric computation, GPU use, model load, inference, training, checkpoint
selection or reselection, retry, QALB test, diagnostic, prompt/parser change,
or XG. A fresh exact-commit owner GO on issue #196 is required before exactly
one CPU-only aggregate Job.

## Validation

- `python -m unittest tests.test_f2_f3_fixed_checkpoint_aggregate -v`: 3 passed.
- `python -m unittest discover -s tests -v`: 316 passed, 2 optional data skips.
- `python -m compileall scripts`: passed.
- focused Ruff and `git diff --check`: passed.
- generated manifest JSON parsing: passed.
- Kubernetes client and server dry-runs: one CPU-only Job passed both.
- read-only cluster queries before and after dry-run: zero persisted issue-#196
Jobs.

No Kubernetes object, PVC read, prediction read, metric, model load, inference,
or training occurred during preparation.
Loading
Loading