Repository navigation
Add posterior CRPS, Brier scores and finite-action risk metrics - #9
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a reusable
skbel.metricsmodule for:Inputs have explicit shape, finite-value and weight contracts with no implicit
broadcasting. Weight normalization rescales before summation to avoid overflow
from large finite weights. Include API notes and a small executable example.
Verification and limits
48 focused tests cover hand-computed and brute-force score comparisons, scaling
invariance, large/tiny/zero weights, invalid inputs, tied actions and ordinary
package imports. Full local upstream tests and Ruff checks pass; repository CI
checks supported Python/OS combinations. No dependency changes.
These are general scoring and terminal-decision arithmetic, not an acquisition
policy or proof of posterior calibration. CRPS is marginal, not a joint score;
the U estimator requires iid unweighted draws; Brier's two-class result is twice
the scalar binary convention. Future-observation simulation/updating and domain
loss choices remain caller responsibilities.