widemem v1.5 on LoCoMo, measured in the open.
The standard long-term conversational memory benchmark from Snap Research (ACL 2024). Published reference numbers from Mem0, Zep, LangMem, A-Mem, OpenAI Memory, and a full-context baseline are included for context.
TL;DR, including the number we got wrong.
widemem v1.5 scores 55.15 overall J on the full 1,540-question LoCoMo set, at about 213 tokens per query where the reference systems use 1,700 to 26,000. One thing sets this number apart from every other bar on this page: it was graded by an independent judge (GPT-4o, separate from the answering model), while every reference number is self-graded per its own paper. Mid-pack accuracy at 1-10% of the token cost, held under the harsher standard. Per category (official mapping): temporal 60.02, single-hop 58.22, open-domain 45.14, multi-hop 43.85. Temporal is ahead of every reference system in our set (closest: Mem0^g at 58.13). This page has carried two public corrections. First, an earlier version reported 45.32 measured on a stale index; the fresh re-ingest gave 54.81 under the self-graded harness of the time. Second, our harness had the single-hop and multi-hop labels transposed, so we spent two months calling widemem a multi-hop leader when the number was measured on single-hop questions. That claim is retracted; multi-hop is now the weakest category. Both corrections, their causes, and a failed attempt to lift open-domain are documented below, because getting this wrong and fixing it in public is more useful than a clean story.
An earlier version of this page reported 45.32 and called widemem mid-pack. That number was measured against a memory index built by widemem v1.3.0 and never rebuilt, before the v1.4.0 extraction change shipped. We corrected it in public rather than quietly restating the number.
Re-ingesting all ten conversations with the then-current v1.4.1 code and re-scoring the full 1,540-question set gave 54.81 under the self-graded harness of the time. The current confirmed number, from the v1.5 rerun with an independent judge, is 55.15.
Second correction (2026-07-06): our LoCoMo harness had the single-hop and multi-hop category labels transposed. The official LoCoMo evaluation maps category 1 (282 questions) to multi-hop and category 4 (841 questions) to single-hop; our harness had it backwards. Every number this page published as “multi-hop”, including the 57.27 we called ahead of the field, was measured on the 841 single-hop questions. The multi-hop leadership claim is retracted.
The overall score and the token numbers were unaffected: they never depended on category labels. Under the correct labels that self-graded v1.4.1 run put single-hop at 57.27 and multi-hop at 46.69, trailing Mem0 on both; temporal became the strongest category. The harness mapping was fixed, and the full rerun under the official mapping completed on 2026-07-15 with an independent judge: multi-hop is 43.85, and those rerun numbers are what this page now reports. The retraction lives permanently in the public corrections log.
These widemem numbers are the full 1,540-question v1.5 run completed 2026-07-15: a single clean pass, no repair step, control mode (shipped defaults), graded by an independent GPT-4o judge. The reference systems' published full runs are self-graded per their papers, so on the judge axis the comparison tilts against us, not for us. Earlier versions of this page carried a six-conversation estimate of about 56, then a confirmed self-graded 54.81 on v1.4.1. The prior 45.32 figure was a stale-index measurement, and per-category labels published before 2026-07-06 had single-hop and multi-hop transposed. All of it is explained below.
The benchmark the whole field reports on.
LoCoMo (Long-term Conversational Memory) is the standard benchmark for evaluating AI memory systems, published by Snap Research at ACL 2024 (Maharana et al.). It is the benchmark used by Mem0, Zep, LangMem, A-Mem, and MemMachine in their own published numbers. We ran widemem against the same test set so the numbers sit alongside theirs on like-for-like terms.
- 10 extended conversations between pairs of people
- Each conversation: 19-35 sessions spanning weeks to months
- 5,882 total dialogue turns across all conversations
- 1,540 evaluable questions across 4 categories (single-hop, multi-hop, open-domain, temporal)
- Primary metric: LLM-as-a-Judge J score (an LLM judges correct vs wrong, averaged over multiple runs per question; our run uses an independent GPT-4o judge, detailed under Methodology)
Paper: Evaluating Very Long-Term Conversational Memory of LLM Agents. Dataset: github.com/snap-research/locomo.
Overall J score.
An earlier version of this page reported 45.32 and called widemem mid-pack. That number was measured against a memory index built by widemem v1.3.0 and never rebuilt. v1.4.0 changed extraction to resolve relative dates (“yesterday”, “last week”) to absolute dates at write time. That change shipped on PyPI and was never re-benchmarked, because every run reused the v1.3.0 index. Re-ingesting all ten conversations and re-scoring the full 1,540-question set gave 54.81 under the self-graded harness of the time. The v1.5 rerun under the official category mapping, graded by an independent GPT-4o judge, gives 55.15. This is the confirmed number, not an estimate, and it is the only bar in the chart above that was not graded by its own system's harness.
Ahead of every reference system in our set.
Temporal reasoning is where the v1.4.0 extraction change pays off: relative time references (“yesterday”, “last week”) are resolved to absolute dates at write time. widemem scores 60.02, ahead of every reference system in our set that publishes a temporal number, with Mem0^g closest at 58.13. One asymmetry worth stating plainly: those reference scores are self-graded per their papers, while widemem's 60.02 comes from an independent GPT-4o judge. The claim is scoped to this set of published numbers, not to every system in existence.
Multi-hop reasoning, after the label correction.
An earlier version of this section claimed widemem led every reference system on multi-hop at 57.27. That was wrong. Our harness had the LoCoMo category labels transposed, so the 57.27 was measured on the 841 single-hop questions. The corrected self-graded v1.4.1 figure was 46.69; under the v1.5 rerun with an independent judge widemem scores 43.85on the 282 multi-hop questions: the weakest category, behind Mem0 (51.15), LangMem (47.92), and Mem0^g (47.19), ahead of full-context, Zep, and A-Mem. The reference numbers above were always under the official mapping (they come from the systems' own papers), which is exactly why comparing our transposed row against them was misleading. Retraction details are in the corrections log.
Answer quality per token is what pays the bill.
widemem delivers about 213 tokens per query, roughly an order of magnitude leaner than the graph and large-context systems. This matters because every retrieved memory gets prepended to the answer-generation call, so it sets operating cost, rate-limit pressure, and latency in production. Compact context is a deliberate design choice: importance-weighted retrieval surfaces a few high-signal memories instead of a large pool of mediocre ones.
J per 1,000 tokens (efficiency)
| System | J score | Avg tokens | J / 1k tokens |
|---|---|---|---|
| LangMem | 58.10 | 127 | 457 |
| widemem v1.5 | 55.15 | 213 | 259 |
| Mem0 | 66.88 | 1,764 | 38 |
| A-Mem | 48.38 | 2,520 | 19 |
| Mem0^g | 68.44 | 3,616 | 19 |
| Zep | 65.99 | 3,911 | 17 |
| OpenAI Memory | 52.90 | 4,437 | 12 |
| Full-context | 72.90 | 26,031 | 3 |
For workloads where context cost is a real budget constraint (high-volume agents, rate-limited APIs, local-LLM deployments), answer quality per token is the number that decides the bill. The full field is shown so the trade-off is visible: systems that index more context tend to reach higher raw J at a large token cost.
Including the weak spot.
The full v1.5 run changes the per-category story sharply from the old v1.3.0 numbers. All labels below use the official LoCoMo mapping (category 1 = multi-hop, 282 questions; category 4 = single-hop, 841 questions). Per-category numbers this page showed before 2026-07-06 had single-hop and multi-hop transposed. The “now” numbers below are the v1.5 run graded by an independent GPT-4o judge; where the correction story needs them, the earlier self-graded v1.4.1 figures are named as such.
Temporal questions (J 30.53 then, 60.02 now)
This is the category the stale index hid. v1.3.0 extraction stored relative time references unresolved, so “Caroline went yesterday” never became a date and time questions were effectively unanswerable, scoring 30.53. v1.4.0 changed extraction to resolve relative dates to absolute dates at write time. On the full v1.5 run temporal is 60.02, widemem's strongest category and ahead of every reference system in our set, under an independent judge. This single fix, already shipped, is most of the gap between the old 45.32 and the confirmed 55.15.
Single-hop factual recall (J 53.31 then, 58.22 now)
Simple “where does Alice live” recall is 58.22on the full run, measured on the 841 single-hop questions. This is the category an earlier version of this page mislabeled as multi-hop. It still trails Mem0's published single-hop 67.13 by about 9 points, so it is a growth area, not a bragging right. Two-pass re-ranking (factual queries get a similarity boost toward the top pure-similarity match) is the path to closing the gap.
Multi-hop synthesis (J 41.25 then, 43.85 now)
Connecting facts across sessions scores 43.85on the 282 multi-hop questions: now the weakest category, up from the stale-index 41.25 but behind Mem0's published 51.15. The self-graded v1.4.1 run measured 46.69 here; the independent judge grades harder. The claim that this was widemem's strongest category is retracted; it rested on the transposed labels.
Open-domain questions (J 36.81 then, 45.14 now)
Open-domain scores 45.14, higher than the old number and no longer the weakest category (multi-hop now is). It remains a structural limit. Graph-backed systems traverse entity connections natively; Zep's temporal knowledge graph is a strong example of that approach. widemem stores facts flat and leans on the retrieval layer, which is weaker for broad relationship questions. We tried a lean, entity-aware re-rank to lift this; in a controlled gate it regressed the other categories without improving open-domain, so it was not shipped (the full negative result is in the repo issues). Becoming a graph database is not a goal. Open-domain at 45.14 is a deliberate, accepted limit of the lean flat architecture, and it stays on the page because it is an honest limit.
Sub-second at p95, alongside Mem0.
| Metric | widemem | Mem0 | Zep | LangMem |
|---|---|---|---|---|
| Search p50 | 0.138 | 0.148 | 0.513 | 17.99 |
| Search p95 | 0.232 | 0.200 | 0.778 | 59.82 |
| Total p50 | 0.935 | 0.708 | 1.292 | 18.53 |
| Total p95 | 1.565 | 1.440 | 2.926 | 60.40 |
widemem's numbers are measured on the v1.5 run; the others are published figures. widemem's search latency sits in the same range as Mem0's, within about 10ms at p50, and widemem and Mem0 are the two systems with sub-second search at p95.
Running a fair benchmark is harder than running any benchmark.
Here is what we did to keep the numbers comparable with the other systems' published results, and where the comparison is not yet apples-to-apples.
| Parameter | Value | Rationale |
|---|---|---|
| widemem version | v1.5 (re-ingested) | Rerun with the corrected harness |
| Answer LLM | GPT-4o-mini | Same as Mem0 paper |
| Judge LLM | GPT-4o, independent of the answerer | Reference numbers are self-graded per their papers; ours is not |
| Embeddings | text-embedding-3-small | Same as Mem0 paper |
| Vector store | FAISS local | widemem default |
| Retrieval mode | control (defaults, no hybrid or temporal flags) | Measures the shipped default path |
| Category mapping | official LoCoMo mapping | Fixed after the 2026-07-06 transposition correction |
| Answer prompts | allow complete counts and lists | The short-answer cap no longer truncates enumeration answers |
| Decay | exponential, rate 0.01 | widemem default |
| Scoring weights | sim 0.5 / imp 0.3 / rec 0.2 | widemem default |
| Top k per speaker | 10 | 20 memories total per question |
| Judge runs | 3 | Mem0 paper uses 10; we use 3 for cost |
| Scoring pass | single clean pass | No repair step |
| Coverage | all 10 conversations, 1,540 questions | Full confirmed run, like-for-like with the field |
Pipeline
- Phase 1 (ingestion): re-ingest the conversation turns into widemem v1.5. Each turn runs through extraction, conflict resolution, and FAISS storage. This is the step the original run skipped by reusing a v1.3.0 index, which is what produced the stale 45.32.
- Phase 2 (Q&A): run the questions. Search memories for each speaker, build the prompt, GPT-4o-mini generates the answer. Record latency and tokens.
- Phase 3 (judge): score predictions against ground truth. F1 and BLEU computed locally. The LLM-as-a-Judge is GPT-4o, independent of the answering model, run 3 times per question and averaged to produce J. Single clean pass, no repair step.
Where the comparison is not yet apples-to-apples.
- Judge asymmetry runs against us, not for us. Every reference number in our set is self-graded per its paper: the system's own harness grades its own answers. widemem's numbers are graded by an independent GPT-4o judge, separate from the GPT-4o-mini answerer. The one number we control is held under the harsher standard; the earlier self-graded v1.4.1 run scored 54.81 for comparison.
- Open-domain is an accepted limit. At 45.14 it is no longer the weakest category (multi-hop at 43.85 is), but it remains a structural limit. A lean entity-aware re-rank was built and gated to lift it; it regressed the other categories without improving open-domain, so it was not shipped. The full negative result is recorded in the widemem-ai repo issues. Open-domain is a deliberate trade of the lean, no-graph architecture, not an unfixed bug.
- Category labels were transposed until 2026-07-06. Our harness swapped the single-hop and multi-hop labels relative to the official LoCoMo evaluation. Per-category numbers published before that date carried the transposed names; overall, temporal, and open-domain scores were never affected. The harness mapping is fixed, this page uses the official labels, and the full rerun under the corrected harness completed on 2026-07-15; its numbers are what this page reports. The retraction is recorded permanently in the corrections log.
- Small revision from the earlier estimate. A prior version of this page carried a six-conversation estimate of about 56. The full 1,540-question self-graded run came in at 54.81, and the v1.5 independently judged rerun at 55.15. Stated plainly rather than quietly rounded.
- Re-baseline, not the original run. These numbers are widemem v1.5 re-ingested with current code. The earlier 45.32 was a v1.3.0-era measurement on an index that was never rebuilt after the v1.4.0 extraction change shipped.
- Repair-gap, resolved. An earlier run hit OpenAI rate limits and used a repair pass, which produced a large clean-vs-repaired score gap. A separate clean single-pass run confirmed the old 45.32 was a real measurement, not a repair artifact, and the re-baseline uses the clean single-pass method throughout. No repair step is involved in these numbers.
- Adversarial excluded. LoCoMo includes adversarial questions with no ground-truth answers. All systems exclude these from scoring.
- No hierarchy, no active retrieval. widemem's hierarchical memory and active retrieval are disabled here to compare like-for-like with flat-memory baselines. Enabling them may shift some categories at the cost of comparability.
Verify it yourself.
Full JSON results (benchmark/results/locomo_v15_control_final.json), benchmark runner, and evaluator are in the benchmark directory of the widemem-ai repo. The v1.5 control-run harness is benchmark/run_ws1.py; the original run used benchmark/run_locomo.py. The investigation that found the stale-index error is documented in the repo issues.
What's next
- Done (2026-07-15): full LoCoMo rerun under the corrected official category mapping, with the fixed harness and an independent GPT-4o judge. Those are the numbers this page now reports.
- LongMemEval pass (a second standard benchmark in the memory space)
- Memory-quality work: single-call ADD/UPDATE/DELETE consolidation and dedup
- Memory-footprint and throughput benchmarks at 100k, 500k, and 1M memories (for the self-hosting page)
Confirmed run: 2026-07-15, widemem v1.5, all 10 conversations (1,540 questions), control mode, clean single pass, GPT-4o-mini answers judged by an independent GPT-4o averaged over 3 runs. Earlier runs: May 2026, v1.4.1 self-graded re-baseline (54.81); March 2026, v1.3.0 original. Both superseded by this page. If you want to discuss the numbers, open an issue on GitHub or start at /enterprise.