Repository navigation
feat(pydantic-ai): upgrade to v2.60.0 and add conversation, tool judge, timeouts and error category examples - #95
Open
martimfasantos wants to merge 7 commits into
Conversation
…r categories examples
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #75 (the 2.43.0 migration) — merge that first; this PR's diff is only the step from
2.43.0to2.60.0. It upgrades Pydantic AI to2.60.0, fixes an intermittent failure in25_agentic_evals.pythat the upgrade exposed, and adds examples forConversation(2.55),ToolCallJudge(2.53), run timeouts and deadlines (2.58) and model error categories (2.59).What changed
Package upgrade
pydantic-ai[spec]upgraded from2.43.0→2.60.0(pullsopenai3.29.0 — 2.55 requires>=3.26)pydantic-ai-harness>=0.60.0added for29_tool_call_judge.pypydantic-evals/pydantic-graphfloors raised from>=0.2.0to>=2.60.0;genai-prices0.1.13 andgoogle-genai2.33.0 are required transitive bumpssettings.py: the comment naming "Pydantic AI 2.43" now says 2.55Existing examples updated
25_agentic_evals.pyhandle_ticketis nowasyncand awaitsagent.run. About half the runs reportedRuntimeError: ... bound to a different event loopfor one case inside the eval report, while still exiting 0.pydantic-evals runs a plain
deftask in its own thread and event loop per case, and all cases share one agent's pooled HTTP connections, so a connection opened on one loop got reused on another. On 2.43 / openai 3.13 it passed 12 of 12 runs; on 2.55 it failed within 1–2. After the fix: 12 of 12. TheOUTPUTS.mdblock is re-captured.New examples added
28_conversation.pyConversationcarried across runs and stored/restored withConversationTypeAdapter; the sameUsageLimits(request_limit=2)refuses turn 3 when carried as a conversation but not when rebuilt frommessage_history=29_tool_call_judge.pyToolCallJudgeas a capability — a judge model allows deleting a cache file and blocks deleting the production DB; the agent receives the custom denial message30_timeouts_and_deadlines.pyrun(timeout=3)raisesRunTimedOut(aRunCancelledand aTimeoutError) and keepsall_messages(); a tool readsctx.deadline/remaining_time()across runs with no timeout, 30s and 8s;request_timeoutwithmax_retries=0raisesModelRequestTimeouton a real OpenAI call31_model_error_categories.pyFunctionModel) provider errors caught by category — rate limit, quota, overloaded, unavailable, server, context window — viaModelHTTPError.for_category;FallbackModel(fallback_on=(ModelRateLimitError,))sends only rate limits to the real model (failed_attemptsshown), and a quota error is re-raised without fallbackKey changes in v2.43.0 → v2.60.0
Conversation/conversation=(2.55) — carries messages, running usage,conversation_idand pending deferred tool calls between runs.Cachingcapability with a unifiedcachesetting (2.55).ToolCallJudgeinpydantic-ai-harnessandrepair_messages(2.53); the harness package is now built and released from the same repo (2.52).ctx.workspace) (2.52) andOpenAILiveModelfor GPT-Live realtime/voice (2.51).TypeSafeModel/DecisionModelplusChoices,BoolCriteria,UseEnumMemberDocstrings(2.45–2.50).wrap_*hooks cover each whole stage;run_stream()usage counts tools after the final result (2.54).ApprovalReviewer(2.57).run(timeout=),RunContext.deadline/remaining_time(), andrequest_timeout/max_retriesinModelSettings; alsoSystemPromptPart(scope='request').ModelHTTPError.for_category(2.59); instructions accepton_change='append'.FileUnderstandingcapability and'max'thinking effort (2.60);RetryPromptPartis deprecated. Harness 0.60.0 addsRelevanceCompaction,SmartSearchandDockerSandbox.web_fetch_tool, OpenTelemetry content leaks,ConcurrencyLimitedModeland MCP credentials in traces (2.44, 2.52, 2.53, 2.58, 2.59). 2.55 requires Python ≥3.11 (this folder already requires ≥3.12).Notes
pydantic-ai-harnessis Alpha (0.x). Its API may change between minor releases; the README says so.ToolCallVerdictcarries no arguments or reason.29matches verdicts to calls bytool_call_idto print the arguments.29reports 4 requests: 2 agent, 2 judge.30Example 3 always times out, on purpose. It gives a real OpenAI call a 50msrequest_timeoutto showModelRequestTimeout.31simulates provider failures. Rate limits and outages can't be triggered on demand, so aFunctionModelraises them; only the Example 2 fallback answer comes from OpenAI. The README says so.17_evals.pyuses the same sync-task pattern as25but passed 6 of 6 runs and is left alone; it only failed when the evaluation was run twice in one process.18_a2a.pyblock is re-captured in the A2A 1.0 response format (result.task,supportedInterfaces).Configuration
gpt-4o-mini) viasettings.pypydantic-ai[spec]>=2.60.0,pydantic-ai-harness>=0.60.0,pydantic-evals>=2.60.0,pydantic-graph>=2.60.0,pydantic-settingsuvpackage managerTesting
All 32 examples ran with exit code 0 against v2.60.0 (
18_a2a.pyserved in the background).OUTPUTS.mdupdated for the five touched files (25,28,29,30,31), the re-captured18, and the header; the other 26 sections are unchanged.uv lock --check,uv sync --frozenandpy_compilepass locally.