Incorrect estimates on KV Cache Calculator for some models

[strangiato]

I deployed gemma-4-13b-it and gemma-4-31b-it-nvfp4 on an H200 with the RHAII 3.5.1 image to compare the actual results to the estimates from ConfigIQ. Below is a comparison of the configiq estimates alongside the observed results:

gemma-4-31b-it
	ConfigIQ Estimates	Observed
KV Cache Memory (GiB)	74.95	64.68
KV Cache Tokens	972,134	626,949
KV Cache Size per Token (KiB)	880	N/A
KV Cache Memory / KV Cache Tokens	80.8	108.2
KV Cache Memory / KV Cache Size per Token	89307.7

gemma-4-31b-it-nvfp4
	ConfigIQ Estimates	Observed
KV Cache Memory (GiB)	102.81	103.92
KV Cache Tokens	2,684,804	1,007,303
KV Cache Size per Token (KiB)	440	N/A
KV Cache Memory / KV Cache Tokens	40.2	108.2
KV Cache Memory / KV Cache Size per Token	245009.3

Overall, the available KV Cache Memory estimate was pretty decent for both, however, the tokens estimate and the per token estimate seemed pretty dramatically wrong.

While the estimates did have issues, the biggest concern is that they are not consistent within the estimate.

For example, with the fp16 version, the "KV Cache Memory / KV Cache Tokens (KiB)" value is simply the 74.95 GiB divided by the estimated 972k tokens which comes out to only 80.8 KiB instead of the tools estimated 880. Conversely, if I take the 74.95 GiB and divide that by the 880 KiB, I would only expect to have a KV Cache size of 89307 tokens instead of the estimated 972k. The issue was similar for the nvfp4 model and a lot of other models I did some quick checks on.

In reality the KV Cache per token is 108 KiB per token for both model versions.

I suspect the tool assumes a different KV Cache dtype for the nvfp4 model. The nvfp4 model still registered as bfloat16 to vLLM, so it uses that as the default KV Cache Dtype, so both the fp16 and the nvfp4 version have the same KV Cache size per token (108 KiB).

With that in mind, the 108 KiB per token is very different from the estimate ConfigIQ came up with for both cases.

[strangiato]

vLLM logs
gemma-4-31b-it-logs.txt
gemma-4-31b-it-nvfp4-logs.txt

[ashishkamra]

Full triage report: KV cache estimates for Gemma 4 on H200

Summary

Reproduced the calculator outputs for both models. The issue involves more than misleading labels: the estimator and vLLM use different capacity definitions, KV dtype assumptions differ for the NVFP4 case, and the original comparison uses different GPU memory-utilization settings.

The attached runtime logs and upstream vLLM allocation code provide a strong explanation for the reported ~108.2 KiB/token. Fixing only the labels or adding a dtype selector would leave a significant capacity-model error.

This report incorporates the runtime logs and corrects the preliminary interpretation that the discrepancy was primarily presentation plus dtype selection.

1. Evidence and scope

• Replayed requests against https://configiq.dev/api/memory and https://aisimulators.dev/memory on September 18, 2026.
• Inspected ConfigIQ at commit 6f5fc22742dfdf570721cf468cd29e14e5fd14fa and its pinned AISimulate SDK build, 0.12.0+9de2b79e.
• Reviewed both runtime logs supplied by the reporter.
• Compared the SDK implementation with upstream vLLM v0.24.0 cache allocation and capacity-reporting code. The logged runtime is the downstream build 0.24.0+rhaiv.13; exact downstream parity remains to be verified.
• Reproduction was at the API level. No new H200 deployment or browser interaction test was performed.

The issue introduction says gemma-4-13b-it, but the tables and served model names in the logs identify 31B. This appears to be a typo. The production reproduction used the catalog IDs below; the runtime logs reference /mnt/models, so the exact deployed checkpoint revisions remain unverified.

2. Production reproduction

Inputs: H200 SXM, vLLM 0.24.0, TP=1, PP=1, maximum batched tokens=8192, maximum batch size=128, memory fraction of_total=1.0, no explicit KV dtype override.

Model	KV budget, bytes	Displayed bytes/token	Token capacity
google/gemma-4-31B-it	80,476,110,848	901,120	972,134
nvidia/Gemma-4-31B-IT-NVFP4	110,389,035,008	450,560	2,684,804

These reproduce the issue's 74.95 GiB / 880 KiB / 972,134 tokens and 102.81 GiB / 440 KiB / 2,684,804 tokens exactly. Dividing the budgets by the returned capacities gives 80.84 and 40.15 KiB/token, respectively.

Reproduction command:

curl --fail-with-body -sS https://configiq.dev/api/memory \
  -H 'Content-Type: application/json' \
  --data '{
    "model_path": "google/gemma-4-31B-it",
    "system": "h200_sxm",
    "backend": "vllm",
    "backend_version": "0.24.0",
    "max_num_tokens": 8192,
    "max_batch_size": 128,
    "tp_size": 1,
    "pp_size": 1,
    "memory_fraction_kind": "of_total",
    "memory_fraction_value": 1.0
  }' | jq '.kvCache'

Repeat with model_path set to nvidia/Gemma-4-31B-IT-NVFP4 for the second case.

3. Runtime configuration established by the logs

Setting	Both runs
Runtime	vLLM 0.24.0+rhaiv.13
Architecture	Gemma4ForConditionalGeneration
Model dtype	torch.bfloat16
KV cache dtype setting	auto
Maximum model/context length	262,144
Maximum batched tokens	8,192; chunked prefill enabled
GPU memory utilization	0.92
TP / PP	1 / 1
Attention backend	FlashAttention 4

Logged result	Unquantized 31B	NVFP4 31B
Available KV memory	64.68 GiB	103.92 GiB
GPU KV capacity	626,949 tokens	1,007,303 tokens
Maximum concurrency at 262,144 tokens/request	2.39×	3.84×
Model-loading memory	58.99 GiB	19.78 GiB
Estimated CUDA graph memory	0.79 GiB	0.76 GiB

The unquantized deployment is BF16, although the issue describes it as FP16. Both are two-byte formats, so this naming difference does not explain the size discrepancy.

4. Root-cause analysis

A. The SDK's metrics describe different parts of an idealized KV curve

For this BF16 model at TP=1, the SDK models:

KV(sequence length L)
  = 800 KiB × min(L, 1024) + 80 KiB × L

The first term represents sliding-window layers and the second represents global-attention layers. Consequently:

• The initial, all-layer cost is 880 KiB/token.
• After the window fills, the idealized marginal cost is 80 KiB/token.
• The SDK computes kv_size_per_token_bytes at sequence length 1, but computes capacity by inverting the window-capped curve.

That explains why division by the displayed 880 KiB does not reproduce the returned capacity. It does not establish that the returned capacity matches vLLM's runtime allocation semantics.

B. vLLM reserves chunked-prefill headroom and reports context-relative capacity

In upstream vLLM v0.24.0, SlidingWindowSpec reserves the last window - 1 tokens plus newly scheduled tokens, rounds to cache blocks, and adds a block for alignment. The reported GPU capacity is derived from maximum concurrency multiplied by maximum model length.

The SDK instead inverts an idealized single-sequence curve, charging the sliding window once. It does not reproduce that per-request reservation and context-relative capacity calculation.

Using the logged context and batching values, and assuming a 16-token sliding-window block:

Sliding reservation per request
  = (ceil((1024 - 1 + 8192) / 16) + 1) × 16
  = 9,232 tokens

Effective allocation per context token
  = 80 + 800 × 9,232 / 262,144
  = 108.173828125 KiB/token

This closely matches the reported 108.2 KiB/token for both deployments. Using the rounded logged budgets predicts approximately 626,971 and 1,007,341 tokens, within 0.004% of the reported values. Conversely, the reported capacities imply budgets of approximately 64.67769 GiB and 103.91600 GiB under this calculation; both round to the logged budgets.

Confidence: strong numerical and upstream-source evidence. The logs do not expose the exact cache-group layout, block sizes, or unrounded budgets, so this is not yet an exact reconstruction of the downstream allocator.

The observed 108.2 KiB is therefore an effective allocation ratio for this runtime configuration, not a universal architectural bytes-per-token constant. It must not be hard-coded as a Gemma 4 correction factor.

C. NVFP4 KV dtype resolution differs from the deployment

Calling AISimulators directly for the NVFP4 model with kvcache_quant_mode: "bfloat16", keeping memory utilization at 1.0, produces:

Metric	No override	Explicit BF16
KV budget	102.81 GiB	102.81 GiB
Initial KV cost	440 KiB/token	880 KiB/token
Capacity	2,684,804	1,337,282

This confirms a default KV-width mismatch consistent with one-byte KV storage in the estimate versus BF16 in the tested deployment. The API does not return its resolved dtype, so the exact default quantization mode should be verified rather than inferred solely from the halved byte count.

The backend accepts a KV dtype override, but ConfigIQ's strict request schema, forwarding client, and UI do not expose it. Weight quantization alone must not determine the vLLM KV dtype; backend behavior, checkpoint metadata relevant to KV storage, and explicit runtime overrides must be resolved separately.

To reproduce this diagnostic, use the same request against https://aisimulators.dev/memory, with the NVFP4 model and "kvcache_quant_mode": "bfloat16". This additional field is not currently accepted by ConfigIQ's /api/memory schema.

D. Memory budgets were compared with different utilization settings

The calculator reproduction uses 1.0, but both runtime logs show 0.92. Repeating the direct SDK-backed API requests with memory_fraction_value: 0.92 and explicit BF16 gives:

Model	Estimated KV budget	Logged KV budget	Estimated capacity
Unquantized 31B	63.67 GiB	64.68 GiB	824,285
NVFP4 31B	91.53 GiB	103.92 GiB	1,189,433

Matching utilization largely reconciles the unquantized budget, but reveals a substantial NVFP4 budget discrepancy. The original 102.81-versus-103.92 GiB agreement should not be treated as validation because mismatched assumptions masked other errors.

The SDK reports 31.94 GiB of NVFP4 weights, compared with vLLM's 19.78 GiB model-loading memory. These are not guaranteed to be identical measurements, but the gap warrants investigation of checkpoint identity, resident weights, quantization exclusions, and backend-specific storage. The logs also identify weight-only NVFP4 execution through Marlin on H200.

5. Implementation ownership and proposed fixes

1. AISimulate SDK: correct runtime capacity semantics. Implement the selected backend/version's per-request cache reservation and allocation model: maximum context length, maximum batched tokens, sliding-window reservations, cache groups, page/block rounding, and concurrency-based capacity. Keep any idealized single-sequence capacity explicitly distinguished from runtime-estimated capacity.
2. SDK and API: resolve and expose KV dtype. Match vLLM auto behavior for the tested checkpoint. Return the effective dtype and support validated explicit overrides through the service, ConfigIQ schema/client, and UI. Do not assume a generic fp8 label maps to every supported backend mode.
3. SDK: reconcile non-KV memory accounting. Compare at the same utilization and inspect NVFP4 resident-weight accounting before attributing the remaining gap to generic overhead. Account for graph/runtime reservations without double-counting existing allowances.
4. ConfigIQ: make assumptions and metric meanings visible. Distinguish maximum context length from maximum batched tokens; show utilization and effective KV dtype. Label runtime-estimated capacity and effective bytes/token consistently. Keep initial/marginal architectural costs as explanatory detail. Derive sizing metrics in AISimulators rather than adding model formulas to React.
5. API contract: preserve existing consumers. Add explicit capacity-basis and resolved-configuration metadata. Avoid silently redefining existing per-token fields; document or version semantic changes.

Simply replacing the displayed per-token value with budget / current_capacity would make the tiles look consistent while preserving an overestimated capacity. Likewise, exposing BF16 alone does not address the allocator-model mismatch.

6. Verification and acceptance criteria

Existing verification: npm test -- lib/api/__tests__/kv-cache-calc.test.ts passed 24/24 tests. These are adapter/schema tests, not validation of SDK sizing or runtime accuracy. No application source changes were made during triage.

Required regression coverage:

• Separate budget estimation from budget-to-capacity conversion, so compensating errors cannot hide each other.
• Validate the capacity conversion against the attached logs with recorded context, batching, dtype, and allocator assumptions; obtain exact blocks/budgets for an exact fixture.
• Cover changes to context length, batched-token limits, block boundaries, and multiple requests for hybrid models, plus a conventional full-attention model with linear KV growth as a control case.
• Verify automatic and explicit KV dtype resolution independently of weight quantization.
• Verify explicit override forwarding and effective-dtype metadata through ConfigIQ.
• Reconcile both model budgets at utilization 0.92, particularly NVFP4 weights.
• Ensure the UI identifies the capacity basis and does not present an idealized single-sequence inverse as backend-equivalent capacity.

Source references

• Reporter-provided runtime logs
• ConfigIQ result mapping, returned metrics, and request schema
• Calculator result labels
• AISimulators SDK passthrough
• Pinned SDK memory estimator and Gemma 4 KV curve
• vLLM v0.24.0 cache specs: SlidingWindowSpec
• vLLM v0.24.0 allocation and reporting: get_max_concurrency_for_kv_cache_config, get_kv_cache_capacity

[mazam-lab]

Hey Trevor, I'm actively working on this. Per Ashish's Triage Report and some digging through the AISimulator backend, the issue is with Gemma and other models that use sliding window attention (i.e. Deepseek), thought there's still several ideas and possible fixes I'm looking into. I'll give updates if there's any corrections or any changes made.

• Because Gemma uses a sliding window attention, the size that a token takes up in the KV Cache scales non-linearly. This means that 880 KB is basically the upper cap on the token size; for larger sequences you can expect most tokens to be closer to the 80 KB estimate that's given. This is a reporting issue in ConfigIQ we'll have to address
• As for the accuracy itself, you're right that AISimulate seems to have some issues in correctly choosing the KV Cache data type, and it seems some of the value discrepancies are caused by AISimulate not factoring in KV Cache-specific overhead that would cause the effective maximum number of tokens available for the KV Cache to be lowered when working with multiple sequences. Will be addressing this on the AISimulate side