Skip to content

feat(pydantic-ai): upgrade to v2.60.0 and add conversation, tool judge, timeouts and error category examples - #95

Open
martimfasantos wants to merge 7 commits into
clean/update-pydantic-aifrom
clean/update-pydantic-ai-2.55.0
Open

martimfasantos wants to merge 7 commits into
clean/update-pydantic-aifrom
clean/update-pydantic-ai-2.55.0

Conversation

@martimfasantos

@martimfasantos martimfasantos commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Summary

Stacked on #75 (the 2.43.0 migration) — merge that first; this PR's diff is only the step from 2.43.0 to 2.60.0. It upgrades Pydantic AI to 2.60.0, fixes an intermittent failure in 25_agentic_evals.py that the upgrade exposed, and adds examples for Conversation (2.55), ToolCallJudge (2.53), run timeouts and deadlines (2.58) and model error categories (2.59).


What changed

Package upgrade

  • pydantic-ai[spec] upgraded from 2.43.0 → 2.60.0 (pulls openai 3.29.0 — 2.55 requires >=3.26)
  • pydantic-ai-harness>=0.60.0 added for 29_tool_call_judge.py
  • pydantic-evals / pydantic-graph floors raised from >=0.2.0 to >=2.60.0; genai-prices 0.1.13 and google-genai 2.33.0 are required transitive bumps
  • settings.py: the comment naming "Pydantic AI 2.43" now says 2.55

Existing examples updated

File Change
25_agentic_evals.py handle_ticket is now async and awaits agent.run. About half the runs reported RuntimeError: ... bound to a different event loop for one case inside the eval report, while still exiting 0.

pydantic-evals runs a plain def task in its own thread and event loop per case, and all cases share one agent's pooled HTTP connections, so a connection opened on one loop got reused on another. On 2.43 / openai 3.13 it passed 12 of 12 runs; on 2.55 it failed within 1–2. After the fix: 12 of 12. The OUTPUTS.md block is re-captured.

New examples added

File Feature Docs
28_conversation.py Conversation carried across runs and stored/restored with ConversationTypeAdapter; the same UsageLimits(request_limit=2) refuses turn 3 when carried as a conversation but not when rebuilt from message_history= Carrying a conversation whole
29_tool_call_judge.py ToolCallJudge as a capability — a judge model allows deleting a cache file and blocks deleting the production DB; the agent receives the custom denial message Tool call judge
30_timeouts_and_deadlines.py run(timeout=3) raises RunTimedOut (a RunCancelled and a TimeoutError) and keeps all_messages(); a tool reads ctx.deadline / remaining_time() across runs with no timeout, 30s and 8s; request_timeout with max_retries=0 raises ModelRequestTimeout on a real OpenAI call Timeouts
31_model_error_categories.py Simulated (FunctionModel) provider errors caught by category — rate limit, quota, overloaded, unavailable, server, context window — via ModelHTTPError.for_category; FallbackModel(fallback_on=(ModelRateLimitError,)) sends only rate limits to the real model (failed_attempts shown), and a quota error is re-raised without fallback Handling model API errors

Key changes in v2.43.0 → v2.60.0

  • Conversation / conversation= (2.55) — carries messages, running usage, conversation_id and pending deferred tool calls between runs. Caching capability with a unified cache setting (2.55).
  • ToolCallJudge in pydantic-ai-harness and repair_messages (2.53); the harness package is now built and released from the same repo (2.52).
  • Workspaces (ctx.workspace) (2.52) and OpenAILiveModel for GPT-Live realtime/voice (2.51).
  • TypeSafeModel / DecisionModel plus Choices, BoolCriteria, UseEnumMemberDocstrings (2.45–2.50).
  • wrap_* hooks cover each whole stage; run_stream() usage counts tools after the final result (2.54).
  • Python 3.15 support and harness durability (2.56); ApprovalReviewer (2.57).
  • Run timeouts (2.58) — run(timeout=), RunContext.deadline / remaining_time(), and request_timeout / max_retries in ModelSettings; also SystemPromptPart(scope='request').
  • Unified model error categories with ModelHTTPError.for_category (2.59); instructions accept on_change='append'.
  • FileUnderstanding capability and 'max' thinking effort (2.60); RetryPromptPart is deprecated. Harness 0.60.0 adds RelevanceCompaction, SmartSearch and DockerSandbox.
  • Security fixes for web_fetch_tool, OpenTelemetry content leaks, ConcurrencyLimitedModel and MCP credentials in traces (2.44, 2.52, 2.53, 2.58, 2.59). 2.55 requires Python ≥3.11 (this folder already requires ≥3.12).

Notes

  • pydantic-ai-harness is Alpha (0.x). Its API may change between minor releases; the README says so.
  • ToolCallVerdict carries no arguments or reason. 29 matches verdicts to calls by tool_call_id to print the arguments.
  • Judge calls count toward the run's usage and limits. 29 reports 4 requests: 2 agent, 2 judge.
  • 30 Example 3 always times out, on purpose. It gives a real OpenAI call a 50ms request_timeout to show ModelRequestTimeout.
  • 31 simulates provider failures. Rate limits and outages can't be triggered on demand, so a FunctionModel raises them; only the Example 2 fallback answer comes from OpenAI. The README says so.
  • 17_evals.py uses the same sync-task pattern as 25 but passed 6 of 6 runs and is left alone; it only failed when the evaluation was run twice in one process.
  • Two leftovers from feat(pydantic-ai): migrate to v2.43.0 and add new examples #75 are fixed here. The README now lists fasta2a 2.0.1 (was 0.6.1), and the 18_a2a.py block is re-captured in the A2A 1.0 response format (result.task, supportedInterfaces).

Configuration

  • All examples use OpenAI (gpt-4o-mini) via settings.py
  • Dependencies: pydantic-ai[spec]>=2.60.0, pydantic-ai-harness>=0.60.0, pydantic-evals>=2.60.0, pydantic-graph>=2.60.0, pydantic-settings
  • Managed with uv package manager

Testing

All 32 examples ran with exit code 0 against v2.60.0 (18_a2a.py served in the background). OUTPUTS.md updated for the five touched files (25, 28, 29, 30, 31), the re-captured 18, and the header; the other 26 sections are unchanged. uv lock --check, uv sync --frozen and py_compile pass locally.

@martimfasantos martimfasantos added size:M Reviewable diff of 150-299 lines (excludes uv.lock, OUTPUTS.md) framework:pydantic-ai Touches the pydantic-ai/ framework folder labels Oct 11, 2026
@martimfasantos martimfasantos changed the title feat(pydantic-ai): upgrade to v2.55.0 and add conversation and tool call judge examples feat(pydantic-ai): upgrade to v2.60.0 and add conversation, tool judge, timeouts and error category examples Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

framework:pydantic-ai Touches the pydantic-ai/ framework folder size:M Reviewable diff of 150-299 lines (excludes uv.lock, OUTPUTS.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant