Make bot failure analysis resumable and group by leaf errors

[CodexVeritas]

Improves scripts/skills/analyze_bot_failures/analyze_bot_run_failures.py for large windows (e.g. --since 1w --max-runs 400).

Robustness

• Network retries: github_get retries requests.ConnectionError / requests.Timeout with 5s/15s/45s backoff, then raises. HTTP errors are unchanged (404s for jobs without logs are still logged and skipped).
• Resumable downloads: raw logs are cached in <output-dir>/raw_logs/run<id>_job<job id>_<bot>.log (moved out of the per-timestamp folder) and reused on rerun. Job id is in the name because re-run jobs get new ids under the same run id. Writes go through a .partial file so an interrupted write is never reused.
• UTF-8 decoding fix: the logs endpoint returns no charset, so requests decoded logs as latin-1 and the ❌/✅ summary lines never matched. Logs are now decoded and saved as UTF-8. Logs from older analyses are not reused (different path).
• Job listing stops on a partial page instead of fetching an extra empty page.

Grouping by root cause

• Each failed question is now one event, parsed from its Error while processing question url: '<url>': <nested message> line. The leaf errors nested in the repr'd error lists (e.g. litellm NotFoundError, BadRequestError, NumericDistribution ValidationError) are extracted. The event uses the most common leaf, and the group shows the full leaf mix.
• Wrapper tracebacks (the inner ValueError, the forecast_bot RuntimeError, the final run_bots RuntimeError) are only parsed for jobs with no question-level error lines, so one failure is no longer counted three times.
• When the report summary was logged, its ❌ lines set how many failures each URL had (timeout retries can repeat error lines, and subquestions of a group post share a URL). Without a summary (e.g. cancelled job), every error line counts.
• Signatures now ignore escaping depth, pydantic input_value/docs-link details, list contents, and req_... ids. Question URLs without a trailing slash are matched.
• New categories: model_not_found, content_filter, empty_llm_response.

Report

• Per-bot table: failed jobs / jobs in analyzed runs, and failed questions / questions attempted (from the Running bot X with N questions line). Failed jobs come from the job listing, so jobs whose log 404s still count; only completed jobs count toward the denominator. The report explains that the question ratio only covers failed jobs. If that line is missing, the job is left out of the ratio and a warning is logged; the script does not raise.

Testing

• poetry run pytest code_tests/unit_tests/test_analyze_bot_run_failures.py: 23 pass. Covers both nesting shapes, closing-quote handling, same signature at different escaping levels, retry/subquestion counting, the missing-summary case, the per-bot table, network retries, UTF-8 decoding, the cache, and analyze_runs end to end with GitHub calls stubbed.
• Ran the parser on 827 previously saved failed-job logs: 2449 failed-question events, 0 jobs falling back to tracebacks, no job with more failures than questions attempted, 37 groups (top: NotFoundError ×1166).
• Live read-only runs on 8 recent workflow runs. A second run into the same output dir reused every cached log.

🤖 Generated with Claude Code