Web search: Create tests and optimise tool use behaviour #694

We need to create tests that surface how the assistant uses the new search/fetch tools and how it uses their outputs in its answers.

We need to test the assistant's behaviour when using the new tools to:

• Assess if the tool is called at appropriate times, and not too often, as this could slow down model conversation turns. Prompts may need adjusting.
• Optimise max_content_tokens (together with max_uses). It is currently quite low and might truncate pages before the relevant information too often. If we can optimise the assistant's behaviour to only use the tools when necessary, and not speculatively, we can tolerate higher token costs from fetched content.
• Assess if the model knows how to navigate allowed/banned urls, and doesn't waste resources trying to open banned urls. Testing so far suggests it does respect the allowlist when the prompt names it.
• Assess the separate problem of url_not_in_prior_context. Anthropic's web fetch will only open a URL that has already appeared in the conversation, which means a URL in a user message, a client side tool result, or an earlier search or fetch result. A URL in the system prompt does not count, and neither does one the model produced itself. The model does not appear to know this, so it opens every turn by trying to fetch a URL it remembers, being refused, and only then searching. The observed sequence is fetch → refused → fetch → refused → search → fetch success.
  • Two possible fixes, worth testing both: prompting it to search before fetching, or injecting relevant URLs into the user turn, which satisfies the rule directly. Note that removing the example URL from the prompt was already tested and did not fix this, because the model reaches for URLs it knows from training. Injecting a URL in the user turn was tested and did work on the first attempt.
• Assess multi-turn conversations as well: is the assistant re-fetching the same content every turn? This may be slow and costly. A cheap patch for this could be to prompt it to state its findings in its answer to the user, the results would persist across conversation turns.
  • For background, return_history keeps only strings, so all fetched content is discarded at the end of every turn. The URL is discarded too, and a URL that appears only in Claude's own earlier output does not satisfy the prior context rule, so a follow-up question has to search and fetch again from scratch.