Repository navigation
Lower multiple choice probability floor from 1% to 0.1% - #367
Merged
Merged
Conversation
PredictedOptionList clamped every option to [1%, 99%], which is stricter than the 0.1% minimum the Metaculus API accepts. Sub-1% forecasts were silently renormalized, and questions with many options raised once the clamped options pushed the largest one more than 5 points from its original value. Options below 0.1% are now pinned to exactly 0.1% and the excess is taken proportionally from the other options, so renormalizing cannot push a pinned option back under the API minimum. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 'Sum of option probabilities is not 1' warning was logged on every PredictedOptionList, including ones whose probabilities already summed to 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adding the float rounding correction on top of the largest option could leave it one ulp above 0.999 in 2-option lists (e.g. 0.9990000000000001). The largest option is now set to 1 minus the sum of the others, which is exactly 0.999 when the other option is pinned at 0.001. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With exactly 1000 options the guard (minimum * count > 1) passed because 0.001 * 1000 == 1.0, but float rounding then left one option at 0.0009999999999992, below the API minimum. The guard now uses >=. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PredictedOptionListclamped every option to [1%, 99%], which is stricter than the 0.1% minimum the Metaculus API accepts. Bot forecasts with options under 1% (or over 99%) were silently renormalized.ValueError.ValueError.PredictedOptionList.Live API check
Posted multiple choice forecasts to a test question on www.metaculus.com:
So changing only the clamp constants to 0.001/0.999 would get forecasts with a 0% option rejected.
Tests
test_clamping_of_probabilitiesfor the new 0.1% / 99.9% bounds.0.1 * 10) and still logged otherwise.PredictionExtractorcase with 0.1%–0.5% options.TZ=UTC).🤖 Generated with Claude Code