Add SerpAPI Google AI Mode search wrangle

[Copilot]

Linked issue

What changes

Adds search.ai_mode, combining cited web search and synthesized content into one normalized SerpAPI request while preserving existing search contracts.

• API: Supports direct Python and recipe/DataFrame entry points, scalar/list queries, row IDs, multiple query columns, and structured/readable dual output.
• Normalization: Maps references, quick results, shopping results, inline products, answer markdown, text blocks, metadata, errors, and structured pricing.
• Behavior: Deduplicates sources before applying n_results, preserves query order and blank rows, avoids ambiguous currency inference, and makes raw provider responses opt-in.
• Prompting: Provides an industrial-product research prompt by default; custom prompts support general research.
• Compatibility: Restores wrangles.search as the core module while retaining SerpApiWranglesClient; classic search and retrieval behavior is unchanged.
• Documentation: Adds parameter, output, error, caching, cost, and recipe guidance.

wrangles:
  - search.ai_mode:
      queries: Product Search Query
      id: ID
      output:
        - AI Mode Results
        - AI Mode Text
      n_results: 10
      country: us
      language: en

How it was verified

• Focused AI Mode and schema tests: 22 passed.
• Credential-safe suite: 1,492 passed, 3 skipped, 112 deselected.
• Recipe schema generated and validated offline.
• Package installation and public API imports verified.
• CodeQL: no alerts.
• Live SerpAPI smoke test not run because no API key was available.
• GitHub Actions awaits workflow approval (action_required; zero jobs started).

Compatibility and risk

No recipe migration is required. Existing search.find_links and search.retrieve_link_content contracts remain unchanged.

SERPAPI_API_KEY is required for nonblank requests unless api_key is supplied. Provider schema drift and request cost are the primary risks; normalization tests cover the documented response sections, and no_cache remains explicit. Safest rollback is reverting the two feature commits.

Ready-for-review checklist

• One human delivery owner is assigned
• The linked issue and intended milestone are correct
• The branch is current with main and has no merge conflicts
• Focused tests pass
• New or changed behavior has direct test coverage
• Documentation/schema/configuration is updated where applicable
• The PR contains no unrelated changes
• The PR description reflects the branch's current scope and latest validation
• One primary reviewer is requested only when this PR is ready

See the pull request workflow.

• Fixes New Search with AI Mode (serpAPI) #1073

[thomasstvr]

@copilot a few things that jump out at me immediately:

• Tests should be real. We want to actually hit SerpAPI, do not use a fake client. There is a GitHub secret called SERPAPI_API_KEY that can be used for these tests. Also, the point of tests/recipes/wrangles is to use recipes to test the wrangles, there is only one case of this in your tests. Non-recipe tests are fine as well, but they should be located in tests/test_wrangles.py.
• The tests should also be located within a class of test_search.py, ie "TestAiMode".
• Some parameters of search.ai_mode do not make sense for our use case or for ai mode in general. For example, n_results does not seem to do anything at all. I am under the impression that ai mode is meant to return only one result. Device should be removed or set to desktop since we do not support mobile or tablet. UULE is probably not needed when we can just keep it simple with location. The use of kwargs to pass these would be nice, but too many parameters in the schema can confuse users.
• The biggest issue I have found so far is that it does not work. SerpAPI does not return a dict, it returns a dict like object (<class 'serpapi.models.SerpResults'>) therefore the catch "if not isinstance(response, dict)" catches all responses and raises an error.

These are just my initial findings, with more tests and a working function, I will have another look.

[Copilot]

@copilot a few things that jump out at me immediately:

• Tests should be real. We want to actually hit SerpAPI, do not use a fake client. There is a GitHub secret called SERPAPI_API_KEY that ca...

Addressed in a084f13: SerpAPI SerpResults mappings are converted to dictionaries, AI Mode is fixed to desktop, n_results/device/uule were removed from the schema, and advanced direct options use **kwargs. The fake suite was replaced with live SerpAPI recipe coverage under TestAiMode in test_search.py plus a live direct test in tests/test_wrangles.py. Local credential-safe tests pass (1,471 passed); CI is awaiting workflow approval and has started zero jobs.

Recommended disposition: Comment only

Next steps

1. PR assignee: Approve CI run 32501806547, then re-request review after the live SerpAPI tests pass.
2. Reviewer: Verify a084f13, confirm required checks pass, then submit a fresh review.

[ebhills]

Superseded by #1188, which implements the revised search.ai_mode design: retrieve SerpAPI Markdown and metadata, apply opt-in markup cleanup, and delegate product-specific semantic structuring to a subsequent extract.ai step.

The updated design comment on #1073 and findings/usage guide explain the trial findings and decision.

Closing this earlier JSON-normalization proposal without merging it. #1073 remains open for the replacement PR.