Skip to content

Count each CombinedLlmArchetype call's cost once - #359

Merged
CodexVeritas merged 1 commit into
mainfrom
claude/archetype-cost-double-count
Oct 26, 2026
Merged

CodexVeritas merged 1 commit into
mainfrom
claude/archetype-cost-double-count

Conversation

@CodexVeritas

Copy link
Copy Markdown
Collaborator

Summary

CombinedLlmArchetype.invoke added response.cost to the active MonetaryCostManagers after its direct call. That direct call goes through GeneralLlm._mockable_direct_call_to_model, which already adds the same cost, so every call through the deprecated model classes (Gpt4o, Gpt4oVision, Perplexity, DeepSeekR1, ...) was counted twice.

This is separate from the litellm-callback double count addressed in #358. The two PRs touch different files.

Changes

  • invoke no longer adds the cost a second time. The raise_error_if_limit_would_be_reached() check before the call stays. It is the only pre-call limit check on this path, because litellm catches and logs exceptions raised from a callback's log_pre_api_call.
  • The hard-limit tests in test_models_incurring_cost.py mock one level lower for archetypes. They mocked the archetype's direct call, which also skips the cost counting inside GeneralLlm. For archetypes they now mock GeneralLlm's litellm call instead, via the new AiModelMockManager.mock_general_llm_litellm_call_with_value. The cost is then counted by the real code path. Other models (ExaSearcher) keep the direct-call mock.
  • New unit test test_invoke_counts_call_cost_once. It invokes Gpt4o with a mocked litellm call and checks the manager total equals one call's cost. Before this change it was double (0.001 vs 0.0005).

Testing

  • code_tests/unit_tests: the new test passes. The only failures are 14 order-dependent tests that also fail on main and pass when run alone.
  • The mocked hard-limit tests in test_models_incurring_cost.py (-k error_thrown) pass for all 5 models (15 cases). Removing the line without moving the mocks made the 12 archetype cases fail.

🤖 Generated with Claude Code

CombinedLlmArchetype.invoke added response.cost to the active cost managers
after its direct call. That direct call goes through
GeneralLlm._mockable_direct_call_to_model, which already adds the same cost,
so calls through the deprecated model classes (Gpt4o, Gpt4oVision,
Perplexity, DeepSeekR1, ...) were counted twice.

- Remove the extra addition from invoke. The hard-limit check before the call
  stays.
- The hard-limit tests in test_models_incurring_cost.py mocked the archetype's
  direct call, which also skips the cost counting inside GeneralLlm. For
  archetypes they now mock GeneralLlm's litellm call instead
  (AiModelMockManager.mock_general_llm_litellm_call_with_value), so the cost
  is counted by the real code path.
- Add a unit test that invoke counts one call's cost once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CodexVeritas
CodexVeritas merged commit 8bd2c6e into main Oct 26, 2026
2 checks passed
@CodexVeritas
CodexVeritas deleted the claude/archetype-cost-double-count branch October 26, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant