Add Markdown AI Mode search with downstream extraction

[ebhills]

Linked issue

Closes #1073. Supersedes #1138 with the Markdown retrieval design.

What changes

Google AI Mode answers vary between headings, paragraphs, nested lists and tables. This implementation retrieves the answer as Markdown and moves product-specific interpretation into a subsequent extract.ai step.

• search.ai_mode requests SerpAPI output=md. One output contains the original Markdown body; an optional second output contains parsed frontmatter and request/status metadata. Blank queries skip the provider, and failed or empty answers are reported explicitly.
• Optional location, country and language controls follow the classic-search aliases. Raw responses can be retained for diagnosis and replay.
• standardize.clean adds opt-in Unicode-escape and simple LaTeX cleanup. The trial then removes the exact Google viewer-dialog instruction from Markdown link labels before extraction, preserving product wording, destinations and raw evidence. Shared content/source helpers validate extracted source associations. The known-glitch notes document recurrence checks.
• The trial puts product identity first in the query, builds Input Product Information with merge.to_dict, and uses extract.ai for Product Description, Match Confidence, Specifications, references and Pricing. Offers carry price, currency, unit of measure, source and reference IDs.
• The runner, recipe and small synthetic response fixture live together under tests/fixtures/search_ai_mode/. Generated captures and workbooks stay in ignored .data/.

See the design findings and usage guide and fixture instructions. The separate AI Overview research note preserves the two-request probe, source limitations and proposed follow-up. AI Overview implementation remains deferred.

How it was verified

• Latest focused offline run: 165 passed across tests/test_search-ai_mode.py, tests/test_search_ai_extraction.py and tests/test_standardize_clean_markup.py. Coverage includes the reported Renold link-label artifact, preserved URLs/code/prices/raw evidence, cleaned model input, stable replay, transport and source validation.
• Live SerpAPI format comparisons for INA, Renold and Belden: Markdown bodies matched reconstructed_markdown after excluding frontmatter and surrounding whitespace.
• Live extraction replays of saved Markdown responses using gpt-5.6-luna with low reasoning exercised all three samples; the Belden mismatch was classified Uncertain.
• git diff --check passed. The viewer-label repair was verified offline without new live provider/model calls. Full CI and deployment validation are not claimed.

Compatibility and risk

This replaces the experimental compact/complete/Markdown AI Mode contract. Recipes must use Markdown plus optional metadata and perform semantic extraction downstream; the guide documents removed AI Mode arguments. Review the shared search exports, formatting, URL-cleaning and nullable-definition changes alongside the new wrangle.

Markdown simplifies the pipeline; it does not improve source coverage or guarantee browser-equivalent answers. Semantic extraction remains model-dependent. Live trials require existing SerpAPI and OpenAI credentials and incur provider usage; no credentials or raw trial exports are committed. Rollback is to revert the feature merge and use the prior recipe/API contract.

Ready-for-review checklist

• One human delivery owner is assigned
• The linked issue and intended milestone are correct
• The branch is current with main and has no merge conflicts
• Focused tests pass
• New or changed behavior has direct test coverage
• Documentation/schema/configuration is updated where applicable
• The PR contains no unrelated changes
• The PR description reflects the branch's current scope and latest validation
• One primary reviewer is requested only when this PR is ready

See the pull request workflow.