Skip to content

Lower multiple choice probability floor from 1% to 0.1% - #367

Merged
CodexVeritas merged 4 commits into
mainfrom
claude/f-tools-min-value-constraints-118e14
Oct 14, 2026
Merged

CodexVeritas merged 4 commits into
mainfrom
claude/f-tools-min-value-constraints-118e14

Conversation

@CodexVeritas

@CodexVeritas CodexVeritas commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • PredictedOptionList clamped every option to [1%, 99%], which is stricter than the 0.1% minimum the Metaculus API accepts. Bot forecasts with options under 1% (or over 99%) were silently renormalized.
  • On questions with many options, raising each low option to 1% could move the largest option by more than 5 points, which tripped the existing max-change check and raised a ValueError.
  • The floor is now 0.1%. Options below it are pinned to exactly 0.1% and the excess is taken proportionally from the other options, so renormalizing can't push a pinned option back under the API minimum (plain clamp-then-divide would leave values like 0.000999). With 2+ options this also caps every option at 99.9%.
  • Float rounding is absorbed by setting the largest option to 1 minus the others, so a 2-option list lands on exactly 0.999 instead of one float step above it. Lists with 1000+ options, where every option cannot keep 0.1%, raise a ValueError.
  • The "Sum of option probabilities is not 1" warning is now only logged when the sum actually differs from 1; it previously fired on every PredictedOptionList.

Live API check

Posted multiple choice forecasts to a test question on www.metaculus.com:

Lowest option Sum Result
exactly 0.001 1 accepted
0.000999 (plain clamp-then-divide output) 1 400: "Probabilities for current options must be between 0.001 and 0.999"
0.0009 1 400 (same message)
exactly 0.001 1 + 1e-6 accepted

So changing only the clamp constants to 0.001/0.999 would get forecasts with a 0% option rejected.

Tests

  • Updated test_clamping_of_probabilities for the new 0.1% / 99.9% bounds.
  • Added tests for: sub-1% options passing through unchanged, a 40-option question that used to raise, and normalized outputs always landing in [0.001, 0.999].
  • Added a test that the sum warning is skipped for sums of 1 (including float noise like 0.1 * 10) and still logged otherwise.
  • Re-enabled a commented-out PredictionExtractor case with 0.1%–0.5% options.
  • Checked ~590k inputs (exhaustive 2- and 3-option grids plus seeded random forecasts with many near-0.1% and near-99.9% values, up to 1500 options) against an exact-fraction reference implementation and the old code: no outputs outside [0.001, 0.999], no sums off by more than 1e-12, no reordering of options, re-validating an output never changes it, and no input that the old code accepted with API-valid output is now rejected.
  • The new tests fail against the old code. Full unit suite passes (TZ=UTC).

🤖 Generated with Claude Code

CodexVeritas and others added 4 commits October 10, 2026 21:20
PredictedOptionList clamped every option to [1%, 99%], which is stricter than the 0.1% minimum the Metaculus API accepts. Sub-1% forecasts were silently renormalized, and questions with many options raised once the clamped options pushed the largest one more than 5 points from its original value.

Options below 0.1% are now pinned to exactly 0.1% and the excess is taken proportionally from the other options, so renormalizing cannot push a pinned option back under the API minimum.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 'Sum of option probabilities is not 1' warning was logged on every PredictedOptionList, including ones whose probabilities already summed to 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adding the float rounding correction on top of the largest option could leave it one ulp above 0.999 in 2-option lists (e.g. 0.9990000000000001). The largest option is now set to 1 minus the sum of the others, which is exactly 0.999 when the other option is pinned at 0.001.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With exactly 1000 options the guard (minimum * count > 1) passed because 0.001 * 1000 == 1.0, but float rounding then left one option at 0.0009999999999992, below the API minimum. The guard now uses >=.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CodexVeritas
CodexVeritas merged commit c0ce3a1 into main Oct 14, 2026
2 checks passed
@CodexVeritas
CodexVeritas deleted the claude/f-tools-min-value-constraints-118e14 branch October 14, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant