Benchmarks
Saying "we have great memory" without a measured metric is not a claim, it's a vibe. The pages in this section record what we measure, how we measure it, which results are citable, and how results compare across the public AI memory systems landscape.
At A Glance
No citable LongMemEval-S number right now
The earlier headline run predates 1.0 and has been withdrawn as a public claim. The live eval harness still runs on the API path with SurrealDB-native graph and vector retrieval, OpenAI embeddings, no LLM extraction, no LLM reranking, and per-question physical tenant isolation. A new number is published only when a current run passes the benchmark gate and the manifest marks it citable.
Pages In This Section
- LongMemEval-S: how the live eval runs, what its metrics mean, reproduction commands, and the claim boundary.
- LongMemEval-V2: the official full-suite harness path, live Sibyl memory adapter contract, and honest-run requirements.
- AI Memory Landscape: honest competitive positioning. The retrieval-vs-QA-accuracy distinction, where Sibyl sits in the field, what we trail academic SOTA on.
- Benchmark Methodology: the full eval ladder, gate profiles, reporting rules, and the AI memory ledger format.
What We Measure
Sibyl runs a small ladder of evaluations, each with a specific scope:
| Eval | What it measures | When to cite |
|---|---|---|
moon run bench-live-smoke | Fast live health guard (latency, response shape) | Local sanity check |
moon run core:bench-context | Frozen context-pack fixtures (eight scenarios) | Retrieval & policy churn |
moon run bench-live | Canonical runtime benchmark against live API | Runtime evidence claims |
LongMemEval Live Smoke (CI) | 25-question live LongMemEval slice on every dispatch | Quick regression signal |
LongMemEval Live Full (CI) | Full 500-question LongMemEval against ephemeral stack | Public eval claims |
LongMemEval offline | Chroma-backed offline replay | Algorithm baseline |
Only a full LongMemEval live run that the manifest marks citable can back a public claim. Everything else is a guardrail or a baseline. Reporting rules and gate profiles are documented in Benchmark Methodology.
Why The Numbers Are Reproducible
Every full LongMemEval run uploads:
longmemeval_live_full.json: overall, per-type, and per-case results with ranked result IDs, answer ranks, latencies, ingest stats, readiness probes, and cross-question leakage counts.- A diagnostics summary covering warning counts, slow-query totals, SurrealDB resource usage at diagnostics time, and any timeout events.
- Run metadata: commit SHA, dataset corpus hash, embedding provider, HNSW settings, fusion backend, corpus text policy, projection settings, extraction settings, concurrency, repeat count.
To verify a published number, download the corresponding artifact, run jq against it, and compare it to the cited value. The LongMemEval-S page has the exact commands.