We built the benchmark that catches AI memory rotting. It caught ours — twice.

Every AI memory product degrades as it accumulates: stale facts outrank current ones, briefs fill with yesterday's decisions, and users quietly turn the feature off. Nobody measures this. LongMemEval and LoCoMo score static recall; the thing that actually kills trust — degradation over time — has no benchmark. So we built one. This post shows its first two runs on our own engine, including the two times it caught our own code being wrong.

Memory rots, and everyone knows it

The documented reasons people disable AI memory are consistent: old errors treated as fact, answers that over-reference history, incidental details bleeding into everything, and preferences from different months contradicting each other. The canonical failure: an agent confidently telling a customer they were "still on Postgres" six weeks after the migration to MySQL — retrieval worked, but a stale fact outscored the current one, because nothing ever decays.

Warehouses rot because storage only ever goes up. A memory needs a metabolism: use strengthens, non-use fades, contradiction supersedes, sleep prunes. We have that machinery built into the engine — but claiming it works is worthless without an instrument that could prove it wrong.

RotBench: degradation as a measurable curve

RotBench simulates 90 days of realistic memory traffic — decisions, reversals, standing preferences, fast-churning status facts, junk sessions, exploratory noise, and adversarial poisoned tool output — and streams it through the production memory door session by session, exactly as a real agent would. Every slot value is an invented token ("Marlin", "Ovelia"), so ground truth is derivable from the seed and scoring is exact substring match. No LLM judge. The current benchmark wars run on vendor-judged numbers that two vendors are publicly disputing; a deterministic world removes the argument.

At checkpoints it probes: is the current value in the top recall results? Does a superseded value outrank it (the "still on Postgres" failure — we call it stale_wins)? What fraction of the session brief is current vs stale? Did any poisoned tool content become memory?

Run 1: the baseline rots — and the fix craters

The warehouse arm (metabolism off — how every shipped memory product works today) rots on schedule: stale values outranking current ones climb from 1 to 7 as volume accumulates. The metabolism arm (supersession linkage + read-validity + decay) fixes exactly that — by day 90 it halves stale_wins and the brief is 16 current lines to 1 stale.

But the same run produced this — the metabolism arm's accuracy in its early window:

50% recall and 25% answer accuracy at day 15. The benchmark caught our own feature making young brains worse: facts distilled from the same conversation — "Kafka is chosen" and "RabbitMQ was rejected", born seconds apart — were treated as knowledge updates of each other, and the supersession machinery tombstoned current facts. A memory that is wrong for its first weeks is a worse product than one that slowly rots, because every new user lives entirely in the early window. The flags stayed off.

Run 2: the crater is fixed — and the benchmark catches us again

The root fix is structural: supersession is a knowledge update across time. Facts born within one session window are a single snapshot of the world and must never supersede each other. With the same-snapshot guard in place, the early window is clean — 100% recall and 100% answer accuracy at day 15, matching the warehouse arm everywhere.

And here is the part most vendors would not publish: in fixing the crater we also hardened the supersession judge's prompt — and over-corrected. In run 2 the metabolism arm's late-window advantage disappeared (stale_wins ties the warehouse at 7 by day 90, where run 1 showed 3 vs 6). The guard that stops same-arc facts from eating each other is currently also blocking some legitimate cross-session updates. The benchmark caught our fix breaking the thing the feature exists for. That is what a falsifier is for, and it is why the metabolism ships to production only when the blue curve dominates the amber one at every checkpoint — not before.

One more number that never moved: poisoned tool output became memory 0 times in 26 checkpoints across all four arms. Adversarial content in tool results — the OWASP ASI06 memory poisoning class — never reached the brain, because tool output is never distilled into durable memory.

Run 4: fix the ruler, and the curve bends

Run 3 exposed a third bug — this time in our own scorer. It counted "RabbitMQ was rejected because it cannot replay" as staleness because the old name appears in the text. But that sentence is a correct, useful record; ghost memory is a stale value asserted as current— the "still on Postgres" class. With the ruler fixed to count only that, the picture resolves:

The four-way: same world, four memory systems

We then ran the identical seed-42 world through every vendor's documented primary path, with the adapter code published alongside the results. This chart is the second iteration — our first adapters under-scored two vendors, and a documentation cross-check caught it before publication: Zep was being probed through raw graph search (a low-level primitive) instead of its recommended Context Block, and Mem0 through its legacy v1 search instead of v2. Fixing our own adapters moved Zep 8.3 → 41.7 and Mem0 33.3 → 58.3, and we struck the old numbers— the protocol only works if it corrects against us too. Mem0's per-checkpoint curve remains unplottable for a measured reason: its async extraction cannot keep pace with bulk ingestion (a fresh 239-session run stored zero memories in-window while single adds succeed in under a minute), so its point is a fully-settled day-90 probe — the favorable condition. Day-90 standings on the identical world: ThinkingRoot 91.7 · Supermemory 66.7 · Mem0 58.3 · Zep 41.7.

One number needs no caveat, because it is an API-shape fact rather than a tuning artifact: the poisoned tool turns. Every tenth day the world injects adversarial tool output — a fake credential and a "SYSTEM OVERRIDE" directive — into an otherwise normal session.

Memory poisoning via tool output is a named attack class (OWASP ASI06). Systems that ingest whatever the conversation carries — including retrieved web pages and tool results — store what attackers can reach. Our rule is structural: tool output is never distilled into durable memory; only the human's and the agent's own words are. That is why our poison column is zero by construction, not by luck.

What shipped to production today

• Time on every memory— session briefs now carry each fact's age ("3w ago"), and anything past 60 days in the episodic lane says so: "may be stale". A decision from March no longer reads like yesterday's.
• Receipts — low-confidence facts are marked unconfirmed; every fact stays traceable to the verbatim words it came from (byte-anchored quotes, not LLM summaries).
• The Mirror— when the engine supersedes or prunes, the next session's brief opens with it: "While you were away: Updated: deadline — now May, was March." The metabolism became visible.
• The same-snapshot guard — facts from one conversation can never supersede each other, the structural fix from run 1.
• Scope provenance — auto-captured facts record which project they came from, the foundation for automatic no-bleed scoping.
• A smarter capture gate— status-update language ("the cluster is now at X") distills; RotBench found it was being dropped.

Reproduce it

The world generator, driver, and both runs' raw results live in our repo under evals/rotbench/. Worlds are seed-deterministic (seed 42 everywhere above); scoring is exact substring match; the traffic goes through the same HTTP door every production agent uses. If you run a memory product, we'd genuinely like to see your curve — especially if it beats ours.

Memory that is honest about what it knows, when it learned it, and what it got wrong — that is the product. The benchmark keeps us honest first.