LongMemEval-S Leaderboard

Leaderboard

All scores reported on LongMemEval-S 500-question evaluation, judged by the official gpt-4o-mini judge. Each row links to the system's source.

^* OMEGA reports 95.4% task-averaged (mean of per-category scores) as its headline figure; 93.2% is the raw count (466/500). We cite the raw count for direct comparability with other rows.

AgentMemory excluded: AgentMemory publishes 96.2% on LongMemEval-S and is a strong system worth evaluating. Its run uses GPT-4o as judge — not the LongMemEval official gpt-4o-mini judge this leaderboard enforces. Same-judge methodology is the comparability discipline here. See their result at github.com/JordanMcCann/agentmemory.

Beyond accuracy: the engineering matrix

Same systems, six axes that matter past raw accuracy. Each cell is sourced; cells silent in the published docs read “(not documented)” rather than guessing.

How to read this

The frontier systems above use Claude Opus 4.6 (Chronos) or GPT-5-mini (Mastra) as their reader, which sets a hard floor on cost and a hard ceiling on which models you can run. Engram's 64.6% comes from the gpt-4o-mini stack — open-source, MCP-native, runs against any OpenAI-compatible model (including local ones via Ollama), self-hosts on SQLite, and ships per-tenant isolation by default. If your constraint is “highest raw accuracy regardless of cost or deployment shape,” pick Chronos, Mastra, or OMEGA. If your constraint is “the open-source memory system that runs in our VPC, doesn't lock us into a frontier model, ships per-tenant isolation by default, and speaks MCP today,” Engram is the system on that axis. We publish our number with the official judge, the methodology, and the reproduction script — submit yours.

Cost footnote

Rough cost-per-question estimates (back-of-envelope; verify against your own usage). Frontier-model systems running Claude Opus 4.6 or GPT-5-mini typically cost $0.05–$0.20 per LongMemEval-S question end-to-end. Engram's gpt-4o-mini stack runs ~$0.005 per question. Multi-thousand-question evaluation runs with frontier-model stacks cost real money; budget accordingly.

Methodology

Benchmark. LongMemEval-S — the standard 500-question evaluation for long-form agent memory.

Judge. LongMemEval official gpt-4o-mini judge. Same judge as every entry above.

Engram configuration. v0.1.0 release stack: gpt-4o-mini reader, synthesis-mode reading for preference questions, recall-mode reading with verifier for everything else, query decomposition, 6-built-in-tool registry, two-stage retrieval, classifier-driven category budgets, today-anchored temporal grounding, hybrid retrieval (vector + BM25 + recency + graph + importance + temporal). Both reading modes documented in src/engram/read/.

Reproduction:

git clone https://github.com/jamjet-labs/engram && cd engram
uv sync
export OPENAI_API_KEY=sk-...   # for the official judge + reader
export LONGMEMEVAL_ORACLE=path/to/longmemeval_oracle.json
uv run python -m benchmarks.smoke_runner --n 500 --decompose --tools

Per-question result JSON lands in benchmarks/reports/. Synthesis-mode is auto-routed by the smoke_runner's question classifier — no separate flag needed.

Per-category breakdown

Where Engram v0.1.0 is competitive today, and where the roadmap work is. Honest disclosure — single-session and preference categories are strong; multi-session and temporal-reasoning are the open roadmap items.

Single-session and preference categories are where Engram's synthesis-mode reader and per-category routing pull their weight. Multi-session and temporal-reasoning are the open roadmap items — see Engram's roadmap for the planned work.

Engram's own progression

How v0.1.0's number evolved from the initial 100-question stratified result to the comparable 500-question full set.