Repository navigation
Conversation
Add docs/research/opmem/PUBLICATION_READINESS.md with reproducible protocol, claims/evidence table, independent 13/13 vs 10/13 review, missing work, threats to validity, artifact instructions, manuscript outline, and PR #185-#187 overlap map. Fix provably stale 12-task wording in opmem-spec, public-bench-ladder, mem0-parity-gaps, and master-plan; banner July opmem post as pre-upd03. Co-authored-by: aria <aria@engramhq.com>
Add brainy-recall and brainy-supersede adapter variants for manuscript ablations. Introduce run_opmem_paper_experiments.py with run manifests and optional OPMEM_ALLOW_INFRA_EXIT for diagnostic harness runs. Co-authored-by: aria <aria@engramhq.com>
Co-authored-by: aria <aria@engramhq.com>
Co-authored-by: aria <aria@engramhq.com>
Co-authored-by: aria <aria@engramhq.com>
Paid vendor runs require cost estimate plus approval (not a spend ban). Block Mem0 OSS/Zep/Letta/LangMem on missing adapters and worksheets. Treat 9/10 with infra errors as invalid runs, not comparable scores. Replace placeholder bib entries with verified arXiv/URL records. Co-authored-by: aria <aria@engramhq.com>
Add mem0-oss, zep, letta, and langmem lane adapters with deterministic namespaces, per-task reset, settling hooks, and redacted response logs. Dry-run token instrumentation and spend worksheets use pinned gpt-4.1-mini and embedding-3-small rates; live runs gated by OPMEM_EXTERNAL_LIVE. Co-authored-by: aria <aria@engramhq.com>
Co-authored-by: aria <aria@engramhq.com>
Wire tiktoken payload budgets (mem0 extraction prompt included) into dry-run output. Add run_opmem_external_live.py with preflight ceilings, smoke/full modes, and live Mem0 OSS + LangMem adapters (Postgres/pgvector). Co-authored-by: aria <aria@engramhq.com>
Record mem0-oss 9/13 and langmem 8/13 live pins, blocked zep/letta lanes, payload worst-case worksheet, run manifests, and updated manuscript PDF. Co-authored-by: aria <aria@engramhq.com>
Implement Letta 0.11.7 pip server adapter (opmem_facts block mapping), start script, pinned requirements, and full 13-task run (5/13, 0 infra). Document Zep blocked artifact when ZEP_API_KEY is absent; update manuscript and external-lanes setup for independent reproduction. Co-authored-by: aria <aria@engramhq.com>
ZEP_API_KEY absent from VM env and CLOUD_AGENT_INJECTED_SECRET_NAMES; smoke/full Zep runs remain blocked with artifacts; no score claimed. Co-authored-by: aria <aria@engramhq.com>
Co-authored-by: aria <aria@engramhq.com>
Migrate adapter to thread/graph SDK; free-tier preflight; smoke+full artifacts opmem-phase2-zep-full (0 infra). Update manuscript matrix and Phase 2 summary. Co-authored-by: aria <aria@engramhq.com>
Publication scoring from opmem-phase2-zep-full artifact only; raw 3/13 in footnote. No harness re-run or scoring code changes. Co-authored-by: aria <aria@engramhq.com>
Remove stale blocked/pending Zep-Letta wording; limitations and future work match Table tab:external-lanes (9/13, 8/13, 5/13, 3/9*). PDF rebuild only. Co-authored-by: aria <aria@engramhq.com>
Update Table 4 caption to state each lane ran the full 13-task script and Zep is scored on nine supported tasks with the existing asterisk footnote. Document hand-maintained external-lanes.tex in generated/README.md. Rebuild main.pdf for publication. Co-authored-by: aria <aria@engramhq.com>
Add xurl \artifact macro and sloppy evidence block; use breakable paths for live-full JSON names and Zep fixture ids. Rebuild PDF. Co-authored-by: aria <aria@engramhq.com>
- README and PUBLICATION_READINESS executive table match frozen pins - Record ca4eec0 embedded re-verify (no Mem0 API) - Rebuild main.pdf after make pdf (no text TODOs) Co-authored-by: aria <aria@engramhq.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OpMem Paper 1 manuscript (LaTeX + PDF) grounded in PR #189 publication-readiness protocol and frozen empirical pins. Primary merge-pin at git
20533f2: Brainy 13/13, verbatim 10/13, Mem0 Platform 10/13 (opmem-pin-20260918.json). Phase~2 external lanes are pinned artifacts only in this PR (Mem0 OSS 9/13, LangMem 8/13, Letta 5/13, Zep 3/9*); no new paid API runs in the latest babysit pass.Changes
docs/research/opmem/paper/—main.tex,references.bib,Makefile, generated fragments,main.pdfdocs/benchmarks/artifacts/opmem-manuscript-20533f2/— run manifest, primary/lane/mechanism/stability JSON, logsdocs/benchmarks/artifacts/opmem-pin-20260918.json— combined three-system per-task pindocs/benchmarks/artifacts/opmem-external-lanes/— Phase~2 manifests + live JSON (historical runs; preserved)evals/run_opmem_paper_experiments.py,scripts/run-opmem-manuscript.sh,scripts/generate-opmem-paper-tex.pydocs/research/opmem/PUBLICATION_READINESS.md,README.md— aligned executive table + pin pointersdocs/benchmarks/artifacts/opmem-verify-ca4eec0-20261013.md— local embedded re-verify atca4eec0(no Mem0 key)Test plan
unset MEM0_API_KEY && go test ./internal/api/ -run 'TestOpMemBenchmarkAgainstHTTPServer|TestOpMemPaperExperiments' -count=1— Brainy 13/13, verbatim 10/13 matches20533f2pinmake -C docs/research/opmem/paper pdfpdftotext— noTODO/??in PDF textRisk / follow-ups
dev/mainwithout explicit approval (draft only).opmem-stability.json; manuscript documents invalid 9/10 aggregates when infra errors > 0./recallnot a substitute for search pin.47 API steps) and any new Phase2 vendor runs require worksheet + approval; Zep forget tasks remain not supported on pinned adapter.Related