SurrealDB Graphiti Exit Benchmark Evidence

• Status: partial evidence, LongMemEval full offline artifacts present
• Last updated: 2026-05-13
• Related plan: docs/architecture/SURREALDB_V07_GRAPHITI_EXIT_AND_PURE_SURREAL_PLAN.md

Answer

We do not yet have a saved apples-to-apples pre-Graphiti versus post-Graphiti benchmark pair.

We do have strong post-native context-pack evidence and earlier local compare-run artifacts, but those are not a Graphiti baseline. Do not use them to claim native Surreal is faster or better than Graphiti until the paired live runtime artifacts exist.

Current Artifacts

Passing Native/Compare Context-Pack Run

Latest artifact:

• .moon/cache/evals/eval_context_pack_retrieval_compare_20260513_014405.json

Summary:

• search_type: context-pack
• retrieval_mode: compare
• repeat_count: 20
• cases: 160
• passed: 160
• failed: 0
• pass_rate: 1.0
• latency_ms: 16.0
• latency_p95_ms: 23.8
• source_metadata_coverage: 1.0
• facet_order_match_rate: 1.0
• leak_count: 0.0

This supports the v0.7 default-loop quality gate for the current native/compare retrieval path.

The earlier passing artifact, .moon/cache/evals/eval_context_pack_retrieval_compare_20260512_225828.json, also passed 160/160 with 23.7 ms p95 latency. The 20260513_014405 artifact is the latest release-gate receipt.

Earlier Failed Local Compare Run

Artifact:

• .moon/cache/evals/eval_context_pack_retrieval_compare_20260512_204628.json

Summary:

• search_type: context-pack
• cases: 8
• passed: 1
• failed: 7
• pass_rate: 0.125
• latency_ms: 622.7
• latency_p95_ms: 1187.0

This is useful debugging history. It is not a pre-Graphiti or legacy Graphiti baseline.

Offline LongMemEval-Style Full Artifacts

Artifacts:

• benchmarks/results/ai-memory/manifest.json
• benchmarks/results/ai-memory/longmemeval_sibyl_raw_20260513.json
• benchmarks/results/ai-memory/longmemeval_sibyl_hybrid_20260513.json

These are full offline component artifacts generated with longmemeval-offline-v2. They include case_results for all 500 questions with question IDs, answer session IDs, ranked session IDs, and per-case metrics. They do not measure the live API path, default app startup, Graphiti construction, or post-native Surreal runtime behavior.

Source data came from the Hugging Face xiaowu0162/longmemeval-cleaned longmemeval_s_cleaned.json file, downloaded to .moon/cache/benchmarks/longmemeval_s_cleaned.json for the local run. The exact local wrapper used for both runs was:

uv run --with chromadb python benchmarks/longmemeval_bench.py \
  .moon/cache/benchmarks/longmemeval_s_cleaned.json \
  --mode raw \
  --output benchmarks/results/ai-memory/longmemeval_sibyl_raw_20260513.json

uv run --with chromadb python benchmarks/longmemeval_bench.py \
  .moon/cache/benchmarks/longmemeval_s_cleaned.json \
  --mode hybrid \
  --output benchmarks/results/ai-memory/longmemeval_sibyl_hybrid_20260513.json

Both artifacts passed:

moon run bench-gate -- \
  benchmarks/results/ai-memory/longmemeval_sibyl_raw_20260513.json \
  --profile ai-memory

moon run bench-gate -- \
  benchmarks/results/ai-memory/longmemeval_sibyl_hybrid_20260513.json \
  --profile ai-memory

Overall Results

Mode	Questions	Elapsed	Recall@5	NDCG@5	Recall@10	NDCG@10
raw	500	312.3s	0.966	0.888	0.982	0.889
hybrid	500	309.4s	0.980	0.934	0.992	0.935

Results By Question Type

Mode	Question Type	Recall@5	NDCG@5	Recall@10	NDCG@10
raw	single-session-user	0.914	0.828	0.971	0.847
raw	multi-session	0.992	0.919	1.000	0.914
raw	single-session-preference	0.967	0.833	0.967	0.833
raw	temporal-reasoning	0.947	0.839	0.970	0.839
raw	knowledge-update	1.000	0.946	1.000	0.944
raw	single-session-assistant	0.964	0.953	0.964	0.953
hybrid	single-session-user	0.986	0.957	1.000	0.962
hybrid	multi-session	0.985	0.947	1.000	0.945
hybrid	single-session-preference	0.900	0.786	0.967	0.807
hybrid	temporal-reasoning	0.985	0.908	0.992	0.904
hybrid	knowledge-update	1.000	0.980	1.000	0.980
hybrid	single-session-assistant	0.964	0.954	0.964	0.954

The hybrid offline mode improves overall Recall@5, NDCG@5, Recall@10, and NDCG@10 relative to raw mode. The one quality regression in this saved pair is the single-session-preference slice, where raw Recall@5 and NDCG remain higher. Runtime is effectively flat for this offline Chroma path, with raw at 312.3s and hybrid at 309.4s.

benchmarks/results/ai-memory/manifest.json is the machine-checkable release ledger for this section. Its citable rows must point at committed full artifacts and match their summary metrics; its planned rows are intentionally non-citable until raw artifacts exist.

Retrieval Component And Mini-Memory Gate

Command:

• moon run core:bench-retrieval

Receipt:

• 25 passed in 0.42s

Coverage:

• Temporal boost correctness and throughput:
  • recent entities rank higher when relevance is equal
  • aggressive and gentle decay reorder results differently
  • high relevance can beat recency
  • entities without timestamps keep their original score
  • minimum boost prevents zeroing old records
  • per-type decay rates differ
  • 1,000 entities stay under the 500 ms test budget
  • 10,000 entities stay under the 2,000 ms test budget
• Reciprocal Rank Fusion correctness and throughput:
  • top shared results rank highest
  • list weights influence final ranking
  • disjoint lists merge successfully
  • 3 x 100 result fusion stays under the 100 ms test budget
  • 3 x 1,000 result fusion stays under the 500 ms test budget
• Mini-LongMemEval-style retrieval:
  • factual recall
  • recent temporal recall
  • entity-type filtering
  • aggressive versus gentle temporal reshuffling
  • Recall@5 >= 0.60 across known queries
• Reranking and hybrid-search wiring:
  • reranking defaults and disabled passthrough
  • graceful fallback when cross-encoder dependencies are unavailable
  • content extraction
  • hybrid config wiring
  • temporal and non-temporal hybrid ordering

This is a component correctness and budget gate. It does not write a saved JSON benchmark artifact.

AI Memory Benchmark Coverage

The release evidence bundle should include full results for every AI memory benchmark or competitor comparison we cite. A one-line headline is not enough; each suite needs the raw result artifact, overall metrics, per-slice metrics, corpus or dataset version, command, commit, runtime mode, and known caveats.

Every public benchmark claim must be traceable to a committed or archived artifact that a future release audit can re-open without guessing the corpus, mode, or code revision. If a suite has no artifact, it belongs in planned coverage only.

Current coverage:

Suite or comparison	Local artifact status	Current use
Context-pack frozen suite	Full current report in .moon/cache/evals/	v0.7 default-loop quality gate
LongMemEval-style offline suite	Full raw and hybrid JSON results in benchmarks/results/ai-memory/; both pass ai-memory gate	Citable offline retrieval baseline
Live Graphiti vs native Surreal	Missing paired bench-live artifacts	Required before public pre/post claims
LOCOMO-style long-memory suite	No harness or result artifact found in this checkout	Future external-suite positioning evidence
RULER-style long-context suite	No harness or result artifact found in this checkout	Future long-context stress evidence
Mem0 comparison	No committed result artifact found in this checkout	Future competitor comparison, if cited
Zep comparison	No committed result artifact found in this checkout	Future competitor comparison, if cited
LangMem comparison	No committed result artifact found in this checkout	Future competitor comparison, if cited

Until a suite has a raw artifact and full per-slice table, cite it only as planned coverage, not as evidence. If we decide to make benchmark claims in release notes, the missing rows above become release-blocking for that claim.

Full AI Memory Result Ledger

The table below is the release ledger for external AI memory benchmarks. Rows can move from planned to citable only after the raw artifact and the summarized table are both present in this document and benchmarks/results/ai-memory/manifest.json.

Suite or comparison	Required artifact pattern	Current status	Required summary before citation
LongMemEval-style offline suite	benchmarks/results/ai-memory/longmemeval_sibyl_<mode>_<date>.json	citable	Overall, per-question-type metrics, and case_results for every question
LOCOMO-style long-memory suite	benchmarks/results/ai-memory/locomo_<engine>_<timestamp>.json	planned	Overall score, per-category score, abstention/error rate, latency
RULER-style long-context suite	benchmarks/results/ai-memory/ruler_<engine>_<timestamp>.json	planned	Task-level score, context length, failure modes, latency
Mem0 comparison	benchmarks/results/ai-memory/mem0_<engine>_<timestamp>.json	planned	Overall quality, per-slice quality, ingestion latency, query latency
Zep comparison	benchmarks/results/ai-memory/zep_<engine>_<timestamp>.json	planned	Overall quality, per-slice quality, ingestion latency, query latency
LangMem comparison	benchmarks/results/ai-memory/langmem_<engine>_<timestamp>.json	planned	Overall quality, per-slice quality, ingestion latency, query latency

For citable rows, include both native Surreal and comparator artifacts when the claim is comparative. For non-comparative rows, include the native Surreal artifact and label the claim as Sibyl-only. The current LongMemEval files are the only citable AI-memory rows in this checkout. Every artifact must include enough metadata to reproduce the run without relying on local memory or chat history, and must pass:

moon run bench-gate -- <artifact>.json --profile ai-memory

External Benchmark Acceptance Checklist

Before any external AI memory result appears in release notes, README copy, launch notes, or a comparison table:

• store the raw artifact under benchmarks/results/ai-memory/ or another committed/archive path named in this document
• record the exact suite source, dataset split, version or commit, and any preprocessing
• record the Sibyl commit, runtime mode, graph engine, store, embedding model, generation model if used, and tokenizer or context budget
• include overall metrics and the complete per-slice table, even when a slice regresses
• include ingestion time, query latency, timeout count, error count, and skipped-case count when the suite exposes them
• keep prompts, evaluators, and scoring scripts versioned or archived with the result
• add a claim boundary explaining what the result does and does not prove
• pass moon run bench-gate -- <artifact>.json --profile ai-memory

This is deliberately stricter than a normal smoke benchmark. External memory benchmarks are positioning evidence, so they need the whole receipt.

Full Result Record Required Fields

Each cited suite needs a result record with:

• raw artifact path
• benchmark suite name and version
• corpus or dataset name and version
• Sibyl commit
• command
• runtime mode, including graph engine and store
• model and embedding configuration when the suite uses generated content or embeddings
• overall metrics
• per-slice metrics
• per-case result records with answer IDs, ranked result IDs, and case metrics
• elapsed time and latency metrics when available
• pass/fail gate and profile
• known caveats

For competitor comparisons, the record also needs competitor version, hosted versus self-hosted mode, data-ingestion path, and any tuning that changes retrieval quality or latency.

Release Notes Rule

Release notes can cite only the rows above that have full artifacts. For v0.7 today, that means the context-pack frozen suite and the offline LongMemEval-style raw/hybrid files are citable when their claim boundaries stay explicit. LOCOMO, RULER, Mem0, Zep, LangMem, and live Graphiti-vs-native claims stay out of release notes until their artifact rows exist.

Missing Live Runtime Artifacts

This checkout currently has no saved benchmarks/results/*.json live-runtime artifacts. That means there is no canonical bench-live legacy-vs-native pair to cite.

No committed full-result LOCOMO, Mem0, Zep, LangMem, RULER, or additional external AI-memory benchmark result files were found in this checkout. If we add those suites later, their raw result files must live next to the LongMemEval-style outputs, or in a named external archive manifest when they are too large for git, and be summarized here with the same artifact-first rule.

Claim Boundaries

Supported claims:

• Native/compare context-pack evaluation passes the frozen 20-run suite with 160/160 cases and 23.8 ms p95 latency.
• The default memory loop and selected entrypoints can run with graphiti_core imports blocked when moon run core:no-graphiti-smoke is green.
• Offline LongMemEval-style full artifacts show raw and hybrid component retrieval metrics across 500 questions, including per-question answer IDs and ranked session IDs.

Unsupported claims until the missing pair exists:

• Native Surreal is faster than Graphiti.
• Native Surreal has better retrieval quality than Graphiti.
• The failed 20260512_204628 context-pack artifact represents pre-Graphiti performance.
• Offline LongMemEval-style files represent production runtime behavior.

Required Pre/Post Pair

To make a public pre/post Graphiti claim, capture two bench-live artifacts against the same corpus, queries, auth scope, and release candidate.

Legacy Graphiti runtime:

moon run bench-live -- \
  --label graphiti-legacy \
  --metadata store=legacy \
  --metadata graph_engine=graphiti \
  --metadata corpus=<corpus-name>

Native Surreal runtime:

moon run bench-live -- \
  --label surreal-native \
  --metadata store=surreal \
  --metadata graph_engine=native_surreal \
  --metadata corpus=<corpus-name>

Gate and compare:

moon run bench-gate -- benchmarks/results/<legacy-artifact>.json \
  --profile acceptance \
  --require-metadata graph_engine=graphiti

moon run bench-gate -- benchmarks/results/<surreal-artifact>.json \
  --profile acceptance \
  --require-metadata graph_engine=native_surreal

uv run python benchmarks/compare_eval_reports.py \
  benchmarks/results/<legacy-artifact>.json \
  benchmarks/results/<surreal-artifact>.json

Keep the raw artifact paths in release notes or PR notes whenever citing any number.

Release Rule

v0.7 can ship on default-loop no-Graphiti proof plus the required context-pack gate. The refreshed LongMemEval-style artifacts can be cited as offline component retrieval evidence. Any release note, README, or announcement that compares native Surreal against Graphiti must wait for the paired bench-live artifacts above.