We published our best benchmark number on the wrong questions

Our strongest LoCoMo result was multi-hop reasoning: 57.27, ahead of every system we compared against. We put it on the homepage. We put it in the README. It was measured on the wrong questions.

LoCoMo tags each of its 1,540 questions with a category ID. The official evaluator maps category 1 to multi-hop and category 4 to single-hop. Our harness mapped them the other way around. Every per-category number we published before July 6 had two labels swapped: the score we sold as multi-hop came from 841 single-hop questions, and our actual multi-hop score was hiding under the single-hop label, doing much worse.

The overall score never moved. 54.81 is 54.81 whatever you call the buckets. But the claim we led with, best-in-set multi-hop, was false. Under correct labels our multi-hop was 46.69, behind Mem0 and LangMem.

How a two-line bug survives four months

The transposition lived in one dictionary, copied across every runner we wrote. Copying is the point of the story. We validated new results by comparing them against old results, and the old results carried the same swapped labels, so everything always agreed with itself. Internal consistency feels like correctness. It is not. The bug only surfaced when we diffed our category counts against the official evaluator file, task_eval/evaluation.py, which we should have treated as the source of truth on day one.

We logged the retraction in the corrections log on July 6 and pulled the claim from the site and README on July 14. If you read our benchmarks page during that week, you saw a number our own repo had already retracted. That gap is ours to own.

What we fixed, and what it was worth

Relabeling forced us to redo the failure analysis, and the redo pointed somewhere unexpected: the answer prompt. Our harness told the model to answer in fewer than six words. Fine for "Where does Alice work?" Fatal for "How many countries has Alice visited?" when the true answer lists seven. The model would find every country, then truncate to fit the word budget, and the judge would mark it wrong.

The fix is one conditional: count and list questions may answer completely, everything else keeps the cap. On the full 1,540 questions, same stores, same models, that was worth +1.51 overall and +3.90 on multi-hop.

Then we changed the judge. Our numbers, like most published memory-system numbers, were self-graded: the same model answered the questions and scored the answers. Our own benchmark policy says a marketing number needs an independent judge, so the new runs use gpt-4o to grade gpt-4o-mini's answers. The stricter judge took back 1.17 points.

Both effects, isolated on the same data:

The headline we publish is 55.15, the harsher one. A bigger number graded by our own homework-checker is not a number we want on the homepage again.

Where v1.5 actually stands

Full 1,540-question LoCoMo, official category mapping, answers by gpt-4o-mini, judged independently by gpt-4o, three judge passes per question:

Mid-pack overall. Weak on multi-hop, and we say so. The reference numbers are self-graded, from each system's own paper; ours is the only column an independent judge signed off on.

Two things survive that level of scrutiny. Temporal reasoning: 60.02, ahead of every reference system in our set, under the harsher judge. And cost: 213 tokens of context per query against 1,764 for Mem0 and 26,031 for full-context stuffing, which means you can afford a much smarter answering model than your competitors for the same spend.

Full methodology, per-category data, and both result files are on the benchmarks page. The harness that produced them, official mapping and independent judge included, is now in the repo, so you can check our work. Somebody should. Last time it was us, four months late.