Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 48 additions & 10 deletions docs/WHAT_IF.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,20 @@ From a v0.12.0 checkout, using the verified binary installed above:
--workload-map examples/what-if/workload-map.json --format json
```

The [long-prompt sample](../examples/what-if/long-prompt.csv) has one call on
each side of Haiku 5.5's 100,000-token boundary. It requires a build with
version 2 price tables; earlier releases cannot run it:

```sh
./fleetdiff-install/fleetdiff what-if --litellm-spend examples/what-if/long-prompt.csv \
--period 2026-10-07 --from claude-sonnet-5-5 --to claude-haiku-5-5 \
--fraction 1 --format json
```

The move's `target_tiers` lists one covered source request in each tier. These
counts precede the proposed move fraction; they are not a forecast of sent
requests.

## Scenarios and workload mappings

For several moves, use `--scenario` with a versioned file:
Expand Down Expand Up @@ -79,18 +93,43 @@ all three assumptions. Unmeasured pairs stay level 1 only, with

## Prices and arithmetic

[Bundled prices](../internal/reprice/data/prices.json) are dated USD per million
tokens, for the recorded standard/global tier and cache-write duration. Make a
copy with your applicable rates and date, then pass `--prices prices.json`.
[Bundled prices](../internal/reprice/data/prices-v2.json) are dated USD per million
tokens at direct-provider standard/global rates. Each request selects a price
tier from its complete prompt length, including cache reads and writes; all
its input, cache and output tokens then use that tier. Make a copy with your
applicable rates and date, then pass `--prices prices.json`.
Every known source and target used in the report needs all four prices. Unknown
model IDs or missing prices are refused for a move; other unpriced input rows
are excluded and counted. The report labels overrides explicitly.
LiteLLM rows may use `provider/name`: the prefix must exactly equal that row's
`custom_llm_provider` and the exact catalog entry's provider. Use the public
catalog ID for `--from` and `--to`. No case folding, trimming or alias lookup is
performed; investigation labels use the same match.
Omitted input-tier limits and cache-write durations retain the bundled values;
an applicable replacement tier must be stated explicitly in the price file.
Version 2 tables use ordered `tiers`, with `max_prompt_tokens: 0` meaning an
unbounded last tier. Each tier includes `input`, `output`, `cache_read` and a
`cache_write` map by duration. The model's `cache_write_ttl` chooses the
duration assumed for these records; the current reader does not observe a
per-request cache-write TTL. For example:

```json
{
"version": 2,
"date": "2026-11-19",
"models": [{
"id": "claude-haiku-5-5",
"provider": "anthropic",
"cache_write_ttl": "5m",
"tiers": [
{"max_prompt_tokens": 100000, "input": "0.10", "cache_read": "0.01", "cache_write": {"5m": "0.125", "1h": "0.20"}, "output": "0.50"},
{"max_prompt_tokens": 0, "input": "0.50", "cache_read": "0.05", "cache_write": {"5m": "0.625", "1h": "1"}, "output": "2.50"}
]
}]
}
```

Version 1 single-row tables remain readable with their prior output. A
replacement tier must be stated explicitly; an override cannot silently
extend a bundled model's priced range.

For inclusive input `I`, cache reads `R`, cache writes `W`, and billed output `O`:

Expand Down Expand Up @@ -139,11 +178,10 @@ earlier ones; replay never adds their totals twice. Conflicting contracts,
mixed enabled/disabled snapshots and regressions fail before a report is written.

The fixed input histogram retains request counts, token sums and per-request
extrema. The current price format has one applicable input tier per model.
Every request in a priced cohort must fit it: source cohorts crossing the tier
are excluded, and a move crossing the target tier is refused. Fractions do not
shrink request sizes. No average-based tier selection or higher-tier rate is
invented.
extrema. A summary cohort can be priced only when its minimum and maximum
select the same source and target tier. A cohort spanning tiers needs
request-level rows; its average cannot select a tier. Fractions do not shrink
request sizes. Missing higher-tier rates still refuse the move.

Summaries have no model-by-key/team intersections, so scoped moves and workload
maps require request-level exports. Legacy summaries without `model_usage`
Expand Down
2 changes: 1 addition & 1 deletion docs/calibration/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"schema": "fleetdiff-calibration-publication/v1",
"sha256": {
".github/workflows/ci.yml": "bba5ee8a30e8a9e11f812958526c8d59d265183d3400278f27e5c96a26745d48",
"docs/WHAT_IF.md": "29ca0eee0f40b968222e4dd4fc26670ffb0d02f0684b9b4b2b751668c3479449",
"docs/WHAT_IF.md": "153ad4c75dd32dbb1c2d3e5e28a6f3e0df48dfa3549e785088c151fccb2febb4",
"docs/WHAT_IF_CALIBRATION.md": "9e929d6793a8f243950896b7c4672b732e9579687e2ac40dfed9af38f44920da",
"docs/calibration/LICENSES.md": "fce7715504d899724474f0519b77ab0ddc5fabe95278ddb437c200b2ed2fdd7d",
"docs/calibration/README.md": "d6feb2af08b3fef21f1eca1f18e096cf9c15c8dc1e44c79d3fc5c44ed409c5f7",
Expand Down
35 changes: 35 additions & 0 deletions docs/prices/source-record-20261119.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"schema": "fleetdiff-price-check/v1",
"checked_date": "2026-11-19",
"currency": "USD",
"unit": "per_million_tokens",
"processing": "direct-provider standard/global synchronous text",
"selected_cache_write_classes": {"anthropic": "5m", "openai": "30m"},
"sources": [
{
"provider": "anthropic",
"url": "https://platform.claude.com/docs/en/about-claude/pricing",
"models": ["claude-sonnet-5-5", "claude-haiku-5-5", "claude-opus-5-5", "claude-fable-5-2"],
"rule": "Haiku 5.5 over 100000 input tokens, including cache reads and writes, pays higher rates on every component of that request; other listed models have standard rates through their full context."
},
{
"provider": "openai",
"url": "https://developers.openai.com/api/docs/models/gpt-6.1-sol",
"models": ["gpt-6.1-sol"],
"rule": "Above 272000 input tokens, input and cache rates are doubled and output is 1.5 times the short-context rate for the full request."
},
{
"provider": "openai",
"url": "https://developers.openai.com/api/docs/models/gpt-6-luna",
"models": ["gpt-6-luna"],
"rule": "Above 272000 input tokens, input and cache rates are doubled and output is 1.5 times the short-context rate for the full request."
},
{
"provider": "openai",
"url": "https://developers.openai.com/api/docs/pricing",
"models": ["gpt-6.1-sol", "gpt-6-luna"],
"rule": "Standard short-context cache-write rates for 30-minute and 12-hour retention are listed by model."
}
],
"scope": "No batch, fast, regional, tax, account-specific or negotiated adjustments; the selected write class is a table assumption, not a per-row observed TTL."
}
1 change: 1 addition & 0 deletions evidence/model-mix/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"evidence/model-mix/test_reproduce_model_mix.py": "0cacf17ac594e07fead744e907ff61f22437cfcbcc2c748703e20b21747c54e4",
"internal/reprice/data/factors.json": "cd1b087e294a3d9768cc52b8dd8a6d0b2ab169bcb8b979bc44957310869d2b99",
"internal/reprice/data/prices.json": "5d0e1af2d161af4823e92f00123b3b724b5b7f528d5984cfcc05242e189d252f",
"internal/reprice/data/prices-v2.json": "3c3a9e53380d07ec4958055a62fc5648961bdd42e50f5e252b4050b833906cd7",
"scripts/reproduce-model-mix.py": "7350792bb1444cc6e383dfe932ee7d3a41b2923879089e83a2da522f3f289671"
}
}
11 changes: 9 additions & 2 deletions examples/what-if-summaries/expected.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@
"Model usage summaries contain model-level cohorts only; model-by-key and model-by-team intersections are unavailable."
]
},
"price_date": "2026-10-14",
"price_date": "2026-11-19",
"price_source": "bundled",
"price_version": 1,
"price_version": 2,
"factor_version": 1,
"factor_date": "2026-10-14",
"covered_observations": 5,
Expand Down Expand Up @@ -85,6 +85,13 @@
],
"factor_sources": [
"https://raw.githubusercontent.com/llm-measurement/fleetdiff/main/evidence/model-mix/source-record.json#original-fit"
],
"target_tiers": [
{
"tier": 1,
"max_prompt_tokens": 272000,
"covered_requests": 4
}
]
}
],
Expand Down
2 changes: 1 addition & 1 deletion examples/what-if-summaries/expected.text
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Moving 50% of claude-sonnet-5-5 traffic to gpt-6.1-sol (covered usage): $0.798000 -> $0.659080 to $0.675154, depending on workload assumptions.
Period: Wed 2026-10-07 00:00:00 UTC -> Wed 2026-10-07 00:01:00 UTC (end exclusive). Prices: 2026-10-14 (bundled); measured factors: 2026-10-14.
Period: Wed 2026-10-07 00:00:00 UTC -> Wed 2026-10-07 00:01:00 UTC (end exclusive). Prices: 2026-11-19 (bundled); measured factors: 2026-10-14.

Level 1, same work and output verbosity: $0.675458
Level 2, qa: $0.660386 (propagated mean-interval endpoints $0.659080 to $0.661890)
Expand Down
11 changes: 9 additions & 2 deletions examples/what-if/expected.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
"outside_period": 0,
"gaps": []
},
"price_date": "2026-10-14",
"price_date": "2026-11-19",
"price_source": "bundled",
"price_version": 1,
"price_version": 2,
"factor_version": 1,
"factor_date": "2026-10-14",
"covered_observations": 3,
Expand Down Expand Up @@ -80,6 +80,13 @@
],
"factor_sources": [
"https://raw.githubusercontent.com/llm-measurement/fleetdiff/main/evidence/model-mix/source-record.json#original-fit"
],
"target_tiers": [
{
"tier": 1,
"max_prompt_tokens": 272000,
"covered_requests": 3
}
]
}
],
Expand Down
2 changes: 1 addition & 1 deletion examples/what-if/expected.text
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Moving 50% of claude-sonnet-5-5 traffic to gpt-6.1-sol (covered usage): $0.154400 -> $0.109398 to $0.147974, depending on workload assumptions.
Period: Wed 2026-10-07 00:00:00 UTC -> Thu 2026-10-08 00:00:00 UTC (end exclusive). Prices: 2026-10-14 (bundled); measured factors: 2026-10-14.
Period: Wed 2026-10-07 00:00:00 UTC -> Thu 2026-10-08 00:00:00 UTC (end exclusive). Prices: 2026-11-19 (bundled); measured factors: 2026-10-14.

Level 1, same work and output verbosity: $0.148703
Level 2, qa: $0.112531 (propagated mean-interval endpoints $0.109398 to $0.116140)
Expand Down
3 changes: 3 additions & 0 deletions examples/what-if/long-prompt.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
request_id,call_type,api_key,team_id,model,prompt_tokens,completion_tokens,total_tokens,status,startTime,endTime,cache_read_input_tokens,cache_write_input_tokens,reasoning_output_tokens
synthetic-tier-short,anthropic_messages,SYNTHETIC_TIER_KEY,SYNTHETIC_TIER_TEAM,claude-sonnet-5-5,100000,100,100100,success,2026-10-07T10:00:00Z,2026-10-07T10:00:01Z,20000,10000,0
synthetic-tier-long,anthropic_messages,SYNTHETIC_TIER_KEY,SYNTHETIC_TIER_TEAM,claude-sonnet-5-5,100001,100,100101,success,2026-10-07T10:01:00Z,2026-10-07T10:01:01Z,20000,10000,0
15 changes: 12 additions & 3 deletions internal/cli/whatif_failure_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -147,10 +147,13 @@ func TestWhatIfFailureDisclosureEligibility(t *testing.T) {
return []string{"--scope-key", "SENTINEL_SELECTED_KEY"}
}, prepriceCount: true},
{name: "source-price-tier", extra: func(t *testing.T) []string {
return []string{"--prices", whatIfFailurePrices(t, func(p *reprice.Prices) { p.Models[0].InputTierMax = 9999 })}
return []string{"--prices", whatIfFailurePrices(t, func(p *reprice.Prices) { p.Models[0].Tiers[0].MaxPromptTokens = 9999 })}
}, prepriceCount: true},
{name: "target-tier-rejection", extra: func(t *testing.T) []string {
return []string{"--prices", whatIfFailurePrices(t, func(p *reprice.Prices) { p.Models[1].InputTierMax = 1 })}
return []string{"--prices", whatIfFailurePrices(t, func(p *reprice.Prices) {
p.Models[1].Tiers[0].MaxPromptTokens = 1
p.Models[1].Tiers = p.Models[1].Tiers[:1]
})}
}, wantError: true},
{name: "zero-origin", change: func(_ int, r map[string]string) {
for _, k := range []string{"prompt_tokens", "completion_tokens", "total_tokens", "cache_read_input_tokens", "cache_write_input_tokens", "reasoning_output_tokens"} {
Expand All @@ -164,7 +167,13 @@ func TestWhatIfFailureDisclosureEligibility(t *testing.T) {
{name: "zero-price-nonzero-tokens", extra: func(t *testing.T) []string {
return []string{"--prices", whatIfFailurePrices(t, func(p *reprice.Prices) {
for i := range p.Models {
p.Models[i].Input, p.Models[i].Output, p.Models[i].CacheRead, p.Models[i].CacheWrite = "0", "0", "0", "0"
for j := range p.Models[i].Tiers {
tier := &p.Models[i].Tiers[j]
tier.Input, tier.Output, tier.CacheRead = "0", "0", "0"
for ttl := range tier.CacheWrite {
tier.CacheWrite[ttl] = "0"
}
}
}
})}
}, priced: true, prepriceCount: true},
Expand Down
8 changes: 7 additions & 1 deletion internal/cli/whatif_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,13 @@ func TestWhatIfPriceOverrideAndUnmatchedScope(t *testing.T) {
}
p.Date = "2026-10-15"
for i := range p.Models {
p.Models[i].Input, p.Models[i].Output, p.Models[i].CacheRead, p.Models[i].CacheWrite = "0", "0", "0", "0"
for j := range p.Models[i].Tiers {
tier := &p.Models[i].Tiers[j]
tier.Input, tier.Output, tier.CacheRead = "0", "0", "0"
for ttl := range tier.CacheWrite {
tier.CacheWrite[ttl] = "0"
}
}
}
b, err := json.Marshal(p)
if err != nil {
Expand Down
2 changes: 1 addition & 1 deletion internal/reprice/bundled.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ var bundled embed.FS
func Bundled() (Prices, Factors, error) {
var p Prices
var f Factors
a, e1 := bundled.ReadFile("data/prices.json")
a, e1 := bundled.ReadFile("data/prices-v2.json")
b, e2 := bundled.ReadFile("data/factors.json")
if e1 != nil || e2 != nil || json.Unmarshal(a, &p) != nil || json.Unmarshal(b, &f) != nil {
return p, f, errors.New("invalid bundled what-if evidence")
Expand Down
Loading
Loading