[IZO-Ong]

Short Description

Adds a test layer that records how the global chat planner uses web_search / web_fetch, uses it to run a staged experiment on the planner's prompt and config, and ships a prompt line telling the planner to search before fetching.

Fixes #694

Implementation Details

What ships to production

One bullet appended to planner_web_tools_prompt in services/global_chat/prompts.yaml:

web_fetch can only open a URL that already appears in this conversation: in the user's message, or in an earlier search or fetch result. URLs you remember, and URLs in these instructions, are refused. Search first, then fetch a URL from the results.

max_content_tokens, max_uses, the allowlist and planner code are unchanged.

Test layer (services/global_chat/tests/web_tools/)

The questions in #694 are about how the tools are used (order, URLs, refusals, re-fetches). As such, the checks assert on a recorded tool-call trace:

• trace.py: turns the planner's raw API responses into an ordered list of web calls, pairing calls to results by tool_use_id across rounds.
• recording.py: RecordingPlanner keeps each raw response, and variants are applied to config/prompts in memory.
• metrics.py: per-run metrics and the stage winner rule.
• scenarios.py: six scenarios, each with a trap (two controls that should never touch the web, a shallow and a deep FHIR question, an off-allowlist DHIS2 question, a 3-turn conversation).
• run_web_experiments.py: the staged runner. Runs are cached on disk keyed by a hash of the resolved model, planner config, prompts and scenario.
• calibrate.py: fetched the FHIR pages at 10k/25k/50k tokens to find where facts land, which is how the scenario facts were chosen.

Experiment results

The test suite does N=3 runs per cell against claude-opus-4-8. "Grounded" means the fact is in the answer and in a fetched page, suggesting it was read and not recalled.

Stage 0 (baseline): every FHIR first turn (9/9) opened by fetching patient.html from memory and being refused (url_not_in_prior_context).

Stage 0 table

Scenario	Web calls	Refused fetches	Grounded	Follow-up fetches
control (OpenFn concept)	0	0	–	–
control (code edit)	0	0	–	–
fhir_shallow	4.3	1.3	0.33	–
fhir_deep	3.3	1.0	0.67	–
off_allowlist (DHIS2)	0.3	0	–	–
multi_turn (3 turns)	6.7	2.7	–	1.7

Stage 1 (fetch strategy): in almost every run, the model's first move was to fetch a URL it remembered (patient.html), which web_fetch refuses. With the search-first line (1a), the model searched, fetched the right page and answered from it every time (6/6), against 3/6 for base.

Stage 1 table (FHIR scenarios)

Variant	Refused / run	Grounded (shallow / deep)	Input tokens	Seconds
base	1.17	0.33 / 0.67	106k	41
1a: search-first line	1.50	1.0 / 1.0	115k	46
1b: inject entry URLs	2.00	0.67 / 0.67	132k	53
1c: both	1.33	1.0 / 0.33	130k	47
1d: 1a minus the prompt's example URL	1.83	0.67 / 1.0	131k	57

Stage 2 (max_content_tokens, carrying 1a): keep 10k. fhir_deep never fetched the long Patient page at any size (0/9). The model is smart enough to find the short value-set page, so the model routes around truncation when a short page has the answer.

Stage 2 table

Size	Grounded	Input tokens	Seconds
10k (current)	1.0	115k	46
25k	0.83	125k	50
50k	1.0	153k	66

Stage 3 (findings across turns, carrying 1a): with 1a, no follow-up turn re-fetched, and 2/3 multi-turn runs opened with no refused fetch at all. The findings line only added costs.

Stage 3 table

Variant	Refused / run	Follow-up fetches	Input tokens	Seconds
base (Stage 0)	2.67	1.67	262k	123
1a	0.33	0	172k	99
1a + "state your findings"	1.33	0	295k	162

Answers to #694:

Question	Answer
Are the tools used only when needed?	Yes. Controls made 0 web calls in every variant (30/30 runs).
Does it respect the allowlist?	Yes. 0 attempts at banned URLs.
url_not_in_prior_context	The first refusal comes from training memory and no prompt change removes it in single-turn questions. The search-first line makes recovery reliable (grounded 6/6) and removed it in 2/3 multi-turn runs. URL injection works as a route but cost more.
max_content_tokens	Keep 10k. Bigger fetches cost more and had no measurable gains.
Re-fetching across turns	Fixed by the addition of the search-first line.

Test results on the shipped prompt

• Unit: 609 passed, 1 skipped.
• Strict suite: 6 passed.
• Acceptance (judged): 3/4 pass.
  • The one failure, fhir_shallow, gets the facts right; the judge marked it down for how the answer reads. Before searching and fetching, the model writes short notes to itself, like "I need to search first", and the planner glues all of them onto the start of the answer with no spaces. The user sees: "…I need to search first.Now I can fetch…"
  • This already happened before this PR (an old answer began "Now I can fetch the R4 page.Based on…"). The new line makes it more noticeable, because the notes now describe the search-first rule. The fix belongs in the planner, so it's in follow-up 1.

Deviations from the design

The strict suite allows up to 2 refused fetches on a first turn, where the design asked for 0. The model nearly always opens a FHIR question by trying a URL it remembers, which gets refused. No prompt change was able to stop this, although the new shipped line saw a smaller amount of 1–2 refusals per run, while the old prompt had runs with 4. In follow-up turns, any refused fetch still fails.

Follow-ups

1. Planner narration leaks into answers. Perhaps keep only the final round's text in response/history, or separate rounds with a delimiter.
2. The same page is often fetched 2–3 times within one turn at every content size.
3. Answers built from search snippets can't be checked by the grounding metric as search results come back encrypted.

Steps to run test suites

The live suites need an ANTHROPIC_API_KEY for api.anthropic.com.

export ANTHROPIC_API_KEY=sk-ant-...

poetry run pytest services/global_chat/tests/integration/test_web_tools_pass_fail.py -v   # strict suite, ~$1.50
poetry run pytest services/global_chat/tests/acceptance/web_tools -v -s                    # judged specs, ~$2

# re-run an experiment stage (free when cached; --scenario X --runs 1 for one scenario)
cd services && poetry run python -m global_chat.tests.web_tools.run_web_experiments --stage 1 --carry base

AI Usage

Please disclose whether you've used AI in this work (it's cool, we just want to
know!):

• Yes, I have used AI
• No, I have not used AI

You can read more details in our
Responsible AI Policy

[hanna-paasivirta]

Thank you for the great experiments and analysis @IZO-Ong! I appreciate the detailed work behind the final seemingly simple prompt change.

The experiment structure, where each stage carries the winning approach forward, is efficient, a good fit here and makes the results easy to follow. The experiments are carefully designed. E.g. calibrating where facts sit in each page, then requiring them in both the answer and a fetched page, is a neat way to test truncation and to tell reading from remembering. It looks tricky to fully optimise this, especially within the constraints of the narrow allowlist. The final minimal change is well supported and a great solution.

There are a couple of things we could tweak before merging:

• Experiment code: I would keep the strict suite, acceptance specs, scenarios and trace recording in the repo, but take the experiment runner, variants, calibration and winner logic out of this PR. They've done their work, and if we merge them, they become test code we need to maintain. I’ll tag your current commit so the full experiment stays reproducible and link it from Web search: Create tests and optimise tool use behaviour #694. That also means removing the SEARCH_FIRST import from test_planner.py.
• Langfuse: should we log web call result codes (queries, URLs, refusal codes) in the planner’s meta so we can monitor refusals in production? Right now only counts and hosts reach the trace.

A few other questions I would like to hear your views on, not requests for changes:

• First-turn refusals: are these worth trying to fix further? What might remove them, if anything, if we had lots more time to dedicate to this issue? Knowing what one costs in time and tokens could help decide if it's worth it.
• Wasted searches: the experiments count refused fetches and scenarios that don't require searches which is great. A small remaining gap is searches that succeed and find nothing useful. For example, with only our docsite allowed, the planner searched it for external API docs that will never be there. These show as successful in the trace but add time and cost. Did you see any in the off_allowlist runs? Its stage 0 baseline averaged 0.3 web calls. This is probably best left to us for a separate piece of experimentation though.
• More generally, do you think an allowlist will work here and earn its place for security, or does web search need to be open to be really useful? Even with a full curated list, do you foresee problems, e.g. the model reaching for sites we haven’t listed, or searching listed sites for things they don’t cover?

[IZO-Ong]

Hi Hanna! Sorry for the late reply. I agree with both changes. I'll take out the runner, variants, calibration and winner logic, and add per-call web details to the planner's meta. The full experiment is at b0b8f8f if you want to tag it.

First-turn refusals: I don't think they're a big concern now, as a refused fetch only returns an error code, usually within the same API response as the search that follows. As such, refused fetches only cost a few seconds and few tokens rather than a full planner round. Moreover, the fix I can think likely will cost more than they save. We can try injecting a URL index of every allowed site adds tokens to every turn, but injecting just the entry pages (expt 1b) added ~26k input tokens per run and didn't reduce refusals.

Wasted searches: I saw one in Stage 0, where one of the three off_allowlist runs made a single search, probably when it tried to access hl7.org / docs.openfn.org for DHIS2 tracker docs. I can't see what it found, as the trace only kept "ok" for searches. For this, I'll record search result URLs in the new meta in order to count searches whose results were never opened as well. Additionally, this off_allowlist wasn't rerun with the new prompt, which might search more often.

Allowlist vs open web: I would keep the allowlist as long as the planner reads web pages in the same context where it edits workflows and job code. Mainly, this boils down to the allowlist being quite effective, with 0 attempts at banned URLs, and for domains off the list, the planner answers from memory and says it couldn't check the source.

To make web search more useful, we would probably need more domains, and we should test each one to check that the web search tools can actually read it. One thing we could do is tell the model in the prompt what each domain covers might stop it searching sites for things they don't have. In the long run, we could give the web tools to a read-only research subagent with no workflow-editing tools that returns what it found and its sources. That way, the planner would never see the pages and injected text would have to survive the summary, thus preventing the subagent from acting on it.

[hanna-paasivirta]

Thanks @IZO-Ong, this all makes sense, and the trimmed PR and the new Langfuse logging look good. I’ll tag b0b8f8f and link it from #694 so the full experiment stays reproducible. Makes sense on first-turn refusals, it isn’t worth spending more turn tokens to remove it, especially since injecting entry URLs didn’t help. Recording search result URLs is a nice touch, and will help us count searches that didn’t lead anywhere. Your allowlist reasoning make sense too, and I love the read-only research subagent idea – I think we'll be implementing that! Keeping fetched pages out of the context that edits workflows is a much cleaner security boundary than any list of domains.

Thanks again for such careful, well-structured work. It’s been great to review.

[hanna-paasivirta]

Looks perfect. Thank you!