Measured end to end, on the official data.
Every conversation is written through the product’s own write path, exactly as an agent would write it. Every question is answered from what recall delivers, by a model, and graded with the benchmark’s own judge protocol. Datasets are pinned to a published revision and verified by SHA-256 before use.
Two kinds of number are reported. Retrieval uses no model and covers every question: the share of gold evidence turns that reach the delivered context, and what that context costs in tokens. Answer accuracy uses a model as reader and judge, on a stratified sample when model quota does not allow every question.
Result files record hashes of the dataset, harness, judges, reader prompt and product source. The harness refuses to drop a question that fails, rather than score only the survivors.
Methodology and resultsGold evidence reaches the reader far more often.
The baseline is a fixed 1,500-token recall. The improved arm is an adaptive recall of 6,000 tokens that grows to 16,000 for summaries, orderings, counts and lists. Store, embeddings, judge code and generation settings are identical between the arms.
The gain is not free. Delivered context grows from about 1,450 to between 6,000 and 10,000 tokens on average, which is 6 to 30 percent of each conversation history. Two cheaper levers, a larger embedding model and a graph walk, were measured and rejected because they did not pay for themselves.
| Benchmark | Questions | Baseline | Improved |
|---|---|---|---|
| LoCoMo, all categories | 1,540 | 0.823 | 0.950 |
| LongMemEval-S, stratified | 150 | 0.820 | 0.949 |
| BEAM 100K, all categories | 400 | 0.724 | 0.859 |
Answer accuracy rises with it.
On a 100-question stratified LoCoMo sample, graded with the binary judge used in published Mem0 and EverOS comparisons, overall accuracy rose from 68.0 to 87.0 percent. The paired bootstrap 95 percent interval for the difference is +11.0 to +27.0 points.
Most baseline misses were refusals: the reader said it had no information because the evidence never arrived. That is a retrieval failure, and the improved arm fixes much of it. Remaining misses are mostly loose date arithmetic and commonsense inferences the reader still declines.
| Category | Questions | Baseline % | Improved % |
|---|---|---|---|
| Single-hop | 26 | 76.9 | 96.2 |
| Multi-hop | 26 | 76.9 | 92.3 |
| Temporal | 25 | 76.0 | 84.0 |
| Open-domain | 23 | 39.1 | 73.9 |
| Overall | 100 | 68.0 | 87.0 |
The deployed gateway returns the same answers, fast.
A separate gateway process on a fresh store ingested all ten LoCoMo conversations over HTTP and answered all 1,540 questions through the HTTP recall endpoint, four clients at a time, on a shared 16-core CPU host. Every served context was byte-identical to in-process recall.
| Measure | Result |
|---|---|
| Served contexts identical to in-process recall | 1,540 of 1,540 |
| Recall latency, p50 / p95 | 152 ms / 283 ms |
| Recall throughput, 4 concurrent clients | 20.0 queries per second |
| Write latency, p50 / p95 | 780 ms / 1,149 ms |
| Gold evidence delivered | 0.958 |
What these numbers do not show.
The same model is reader and judge in the answer-accuracy runs. Published leaderboards mostly judge with other models, so absolute numbers are only approximately comparable to them. The two arms share the judge, so the difference between them is exact. Answer accuracy uses stratified samples; retrieval uses every question.
Benchmarks test recall of conversation facts. They cannot tell you whether a memory improved your agent’s outcomes on your tasks. That is a different, causal question, and it is why UniMem ships randomized per-memory holdouts. The example verdict tables in the project README come from a synthetic run and are labeled as such.
LoCoMo is licensed CC BY-NC 4.0. None of the datasets is redistributed with the product; a script downloads each from its publisher at the pinned revision.
We publish baselines, sample sizes and judge caveats on purpose. If a number here cannot be reproduced from the commands below, treat that as a bug and tell us.
Reproduce the retrieval result.
The retrieval measurement needs no model and no API key. Download the pinned datasets, install the CPU embedding extra, and run the harness over every LoCoMo question.
python scripts/fetch_memory_benchmarks.py --dest data/benchmarks --merge-beam-10m
pip install -e '.[fastembed]'
python benchmarks/conversation_bench.py --dataset locomo \
--data data/benchmarks/locomo10.json \
--budget 6000 --adaptive-budget --max-budget 16000
Run it on your own memory.
Try the local loop, or bring one workflow to a pilot conversation.
Technical details link to the inspected source. Check deployment requirements before enabling optional features.