feat: keep several prompt cache entries (--prompt-cache-entries) and evict under memory pressure

[CodeAndCanvas728]

Problem

PromptCache holds a single entry. Any request that doesn't share the cached prefix replaces it, so a client that interleaves conversations loses its cache on every switch. Typical cases: an agent that fires a side request (title generation, a summariser, a sub-agent with a different prompt) between two turns of the main conversation, or several chat sessions against one server. Each switch costs a full re-prefill of the long conversation.

Change

• PromptCache becomes an LRU of up to --prompt-cache-entries N entries (default 1, so default behaviour and memory use are unchanged).
• save replaces any entry that is a prefix of the new one, so a single growing conversation still occupies one slot.
• restore (trimmable models) picks the entry with the longest usable match; the per-entry safety checks (excess vs. cached length, wrapped ring buffers) are unchanged, just factored into usableMatch.
• restoreExactPrefix (hybrid recurrent+attention models) picks the longest exact-prefix entry.
• A DispatchSource memory-pressure source evicts entries: all but the most recent on .warning, all on .critical. Cached KV is the largest evictable allocation, so it is returned before the OS has to compress or kill. PromptCache.evict(keepMostRecent:) is also the missing way to drop entries while idle.

No byte budget in this PR; that can follow if pressure-based eviction proves too late in practice.

Tests

PromptCacheLRUTests (7 tests): unrelated prompts both hit with 2 entries and not with 1; a hit refreshes recency and the least recent is dropped; an extension replaces its prefix entry; longest match wins; hybrid exact-prefix picks the longest; both evict modes. Existing cache tests pass unchanged. swift test --filter SwiftLMTests: 230 pass.

Measurements

Qwen3.5 35B-A3B (qwen3_5_moe, hybrid GatedDelta + attention, 4-bit), loaded text-only (--no-vision) with --stream-experts --ssd-prefetch --prefill-size 2048, on a 16 GB M2. Sequence: turn 1 (12.7k tokens), an unrelated 2.7k-token request, then turn 2 (14.7k tokens, extending turn 1). Greedy, max_tokens 16.

	turn 1	side request	turn 2	turn 2 cache
--prompt-cache-entries 4	145 s	37 s	29.8 s	hit, 12,681 / 14,657 tokens reused
default (1 entry)	148 s	42 s	209.5 s	miss (evicted by the side request)

Memory demand stayed about 8.4–8.7 GB in both runs. The pressure handler fired once (warning level) during the 4-entry run under swap on this machine and kept the newest entry.

Note

On hybrid models, --ctx-size currently disables the hybrid cache (attention layers become RotatingKVCache, which hybridCacheBoundary rejects); that is fixed separately in #207. The runs above did not use --ctx-size.

🤖 Generated with Claude Code

[solderzzc]

Thanks for this. I reviewed it and pushed one follow-up commit (4588b8b) on top of your branch rather than asking for another round trip. AI-assisted (Claude Code); tests run locally on origin/main + this branch (SwiftLMTests 262/0).

• Stale entries: save() now also drops an entry that diverges from the new prompt in at most 16 trailing tokens and shares more than it loses (re-rendered generation prompts, edited last message), so a conversation no longer pushes another one out of the cache. Hybrid snapshots and wrapped sliding-window rings keep the strict exact-prefix rule, because they can't be rewound.
• Memory pressure: evict() returns the dropped count; the handler logs only when something was dropped and calls Memory.clearCache() so the freed KV buffers actually return to the system. Note that with the default of 1, a critical event now empties the cache; code comment, help and README say so.
• Flag: README row, effective value in the startup Config line, values < 1 warn and are raised to 1.
• Tests: 7 → 36 (supersede rule, ping-pong, shared system prompt, ring buffers, longest-usable-match, rejected candidates, evict counts, flag parse).

Not changed: no byte budget, a short prompt within 16 tokens of a longer one replaces it, and no end-to-end run on a sliding-window model. The PR description is out of date ("unchanged", "7 tests") and could use a refresh before merge.