We live-tested 6 memory systems and tiered 7. Here's the scoreboard.

Memory/context-persistence is one of the most crowded categories in the agent-tooling space right now, and also one of the hardest to evaluate from the outside — every project reads well in its own README. So as part of our own equipment registry, we put 7 of them through the same evaluation pipeline: a bare-model floor (no memory at all) and a flat-file baseline (the simplest possible "just write it to disk" approach) as the two anchors every real system has to beat. Six were live-tested end-to-end; one (OpenAI's Conversations API) is tiered from source review, because it's a server-side storage primitive with no client-side algorithm to run — its row says so.

"Live" and "key-free" mean: real runs driven through a subscription-backed model shim, no paid API keys required to reproduce. Not every system ran every bench at full width — per-system scope (which benches, any sampling) is disclosed in each registry entry rather than averaged away.

No vendor claims taken at face value — every live-tested result below is from an actual run, not a docs page, and the one source-review tier is labeled as exactly that.

The scoreboard

SystemTierIn one line

Anthropic Memory Tool B Best result so far. A clean, well-specified six-command file-memory contract with a distinctive multi-session-recovery pattern and the cheapest live-testable setup we tried.

Cognee B-minus The real surprise: a knowledge-graph+vector engine that live-tested better than the flat-file baseline — most of the field didn't clear that bar.

LangMem B-minus Honestly documented and narrowly scoped, but its core write→read path has an open correctness bug we reproduced directly (langmem#140; a second report, #154, we could not reproduce).

CrewAI Memory C A genuinely real design — automatic contradiction consolidation, evidence-gap signaling — that clears the memoryless floor by a wide margin but doesn't beat flat-file on every task.

Graphiti C A genuinely distinctive temporal-knowledge-graph design (facts carry validity windows, nothing is deleted) undermined by a real gap between its documented API and what the graph actually knows.

OpenAI Conversations API C A real, working durable-state primitive — but there's no independent memory algorithm to evaluate; it's a thin wrapper over OpenAI's own storage. Tiered from source review, not live-scored — the only entry on this board that isn't.

Letta (formerly MemGPT) D Historically important, and the self-editing-memory design runs end-to-end — but at the version we could test (v0.6.7, pinned by the deprecated repo; current is v0.16.8) it lost to both the bare-model floor and the flat-file baseline on every fully-tested bench, sometimes dramatically. The vendor has deprecated that repository outright.

What this says about the category as a whole

Most of these didn't clearly beat a flat-file baseline on our small-corpus benches — Cognee explicitly did, in its own live run; the others were closer calls or trailed it. (Fair question: how does Anthropic's Memory Tool hold the top tier if the flat-file head-to-head is a closer call? Because a tier folds in more than that single comparison — spec quality, setup cost, failure behavior — and each registry entry carries the detail behind its letter.) That's not a knock on any one project — it's a genuinely useful signal for anyone deciding whether to reach for a memory framework at all versus starting with something boring that just writes files to disk. Sometimes boring wins, and you only find that out by actually running the comparison.

One thing every entry above has in common, credit where due: none of the underlying projects hid or disputed anything we found when we checked their own docs and source directly. The gaps we found were real, reproducible, and in every case explainable by reading the code — not vendor dishonesty, just the normal distance between a README and a live run.

Caveat we hold ourselves to: all of the above is small-scale (each bench's corpus fits in a single context window). Large-corpus, long-horizon performance is a different question we haven't answered yet — flagged as open in every entry, not swept under the tier number.