epic(memory): improve private Mate recall with eval-gated hybrid retrieval

[robinbraemer]

Outcome

Improve private per-Mate recall quality and cost with eval-gated local lexical and optional semantic retrieval while keeping typed Markdown topics on the Mate PVC as the only memory authority.

Current baseline

AgentOS already provides typed topics, a concise MEMORY.md index, pinned recall, bounded attachments, an LLM relevance selector, automatic extraction, Dream consolidation, pause, correction, forget, routed proposals, redaction, and content-free telemetry. This epic improves retrieval; it does not replace those contracts.

Architecture boundaries

• Markdown topics remain authoritative, private, fallible context.
• Any lexical/vector index is a deletable, rebuildable derivative cache on the owning Mate PVC.
• No cross-Mate index, shared vector service, PostgreSQL memory mirror, or mandatory semantic indexing.
• Real memory does not leave the Mate Pod for embeddings by default.
• Pause and forget invalidate derived entries before acknowledgement.
• Missing, stale, corrupt, or resource-constrained indexing falls back without blocking the main turn.

Workstreams

• test(memory): benchmark current private-memory recall and define adoption gates #113 — benchmark current recall and define numerical adoption gates
• feat(memory): add a rebuildable local lexical index for Mate topics #111 — add a rebuildable local lexical/BM25 candidate
• spike(memory): evaluate local embeddings and vector search within Mate resource budgets #110 — evaluate local embeddings and vector search
• feat(memory): rank private recall with bounded hybrid signals and safe fallback #112 — rank with bounded hybrid signals and safe fallback
• feat(memory): harden index rebuild, pause, forget, and corruption recovery #116 — harden rebuild, pause, forget, and corruption recovery
• feat(memory): expose bounded private memory search and exact retrieval tools #115 — expose bounded private search and exact retrieval tools

Related observability work: #120 under #56.

Success criteria

• A privacy-safe evaluation measures recall, precision, downstream task success, latency, provider calls, token/cost, CPU/memory, index size, rebuild time, and forgetting correctness.
• Local lexical retrieval lands before optional vector retrieval unless evidence justifies another order.
• Hybrid recall is adopted only if it materially beats the current selector inside declared resource and privacy budgets.
• Every derivative state carries source hashes and schema/chunker/model/ranker versions and is safely rebuildable.
• Operators diagnose method and degradation without seeing memory content.

References

• https://github.com/openclaw/openclaw/blob/main/docs/concepts/memory-builtin.md
• https://agentgateway.dev/blog/2026-02-21-kill-switch/

Design record

docs/superpowers/specs/2026-07-31-resilient-delegation-access-effect-design.md