Sibyl v0.7 Graphiti Exit and Pure Surreal Cleanup Plan

• Status: Claude-reviewed execution plan
• Target release: v0.7 hardening wave
• Tracking epic: epic_564b41ff89d6
• 1.0 update: this plan is historical default-loop exit work. The active 1.0 roadmap supersedes "optional compatibility extra" as the endpoint: Graphiti should be deleted from the supported runtime and dependency graph before v1.0.
• Source docs:
  • docs/architecture/SURREALDB_NATIVE_MEMORY_CORE_SPEC.md
  • docs/architecture/SURREALDB_GRAPHITI_EXIT_INVENTORY.md
  • docs/architecture/SURREALDB_GRAPHITI_EXIT_BENCHMARK_EVIDENCE.md
  • docs/architecture/SURREALDB_PHASE3_BURNDOWN.md

This plan decomposes the next two roadmap lanes into executable work:

1. Exit Graphiti from the default memory loop.
2. Finish pure Surreal cleanup by removing or isolating legacy PostgreSQL, FalkorDB, Redis, and archive-only surfaces from default runtime paths.

The work is intentionally split. Graphiti exit proves Sibyl-native memory behavior. Pure Surreal cleanup removes leftover operational coupling. Either lane can land in slices, but the v0.7 release gate requires both to be boring, tested, and documented.

1. Release Definition

v0.7 is ready when a default install, CI run, local dev run, and production chart can operate on SurrealDB without constructing Graphiti or requiring legacy services.

Required outcomes:

• Default remember, recall, context, wake, reflect, API jobs, CLI commands, MCP tools, and prompt hooks use native Surreal services.
• Graphiti imports are either deleted or classified as named compatibility, migration, admin, or test-only surfaces with owners and removal conditions.
• Native Surreal owns entity lookup, deduplication, semantic search, relationship hydration, entity hydration, episode mentions, and embedding cache behavior needed by default memory flows.
• PostgreSQL, FalkorDB, and Redis are absent from default compose, charts, CI services, docs, package extras, and runtime configuration. Any retained Redis coordination backend must be explicit opt-in behavior, not a default data service.
• Archive import, migration, and rollback behavior is explicit. Nothing relies on ambient legacy services.
• Generated inventory and hand-authored inventory agree.

Required gates:

• moon run inventory-check inventory-typecheck inventory-test
• moon run core:test
• moon run api:test
• moon run cli:test
• moon run :check
• moon run baseline-seed
• moon run baseline-replay-runtime
• moon run core:bench-context -- --cases benchmarks/context_pack_cases.json --auth-manifest .moon/cache/baseline-runtime-manifest.json --label retrieval-compare --repeat 20 --metadata retrieval_mode=compare
• moon run core:no-graphiti-smoke
• CI green
• nightly regression green in .github/workflows/nightly-regression.yml

Benchmark release rule:

• v0.7 can ship on no-Graphiti default-loop proof plus the context-pack gate above.
• Do not make public performance or quality comparisons against Graphiti until a paired bench-live legacy Graphiti artifact and native Surreal artifact exist, are gated, and are compared with benchmarks/compare_eval_reports.py.
• Do not make public AI memory benchmark or competitor claims until the cited suite has a raw artifact under benchmarks/results/ai-memory/ or a named archive manifest, overall metrics, per-slice metrics, per-case result records, corpus or dataset version, command, commit, runtime mode, and caveats recorded in docs/architecture/SURREALDB_GRAPHITI_EXIT_BENCHMARK_EVIDENCE.md. benchmarks/results/ai-memory/manifest.json is the machine-checkable citable/planned ledger. Missing external suites stay planned coverage only. LOCOMO, RULER, Mem0, Zep, LangMem, and any future competitor suite need full result rows before they become release-note evidence. Every citable AI-memory artifact must pass moon run bench-gate -- <artifact>.json --profile ai-memory.

2. Non-Goals

• Do not rewrite every compatibility command before v0.7. Explicit migration and archive tools can remain if they are named, documented, and absent from default runtime paths.
• Do not delete user data or force destructive migrations. Archive policy must be decided before legacy backup and restore code is removed.
• Do not preserve Graphiti-shaped abstractions just because native code currently wraps them. Keep a compatibility boundary only while an active caller still needs it.
• Do not expand this lane into the post-v0.7 synthesis engine. Graph-guided synthesis stays after the native memory core is stable.

3. Epic 1: Graphiti Exit From Default Memory Loops

Goal: default memory behavior works without importing or constructing Graphiti.

Wave 1: Inventory Lock

Purpose: make the removal map enforceable before deleting runtime code.

Tasks:

• Regenerate runtime inventory and compare it against docs/architecture/SURREALDB_GRAPHITI_EXIT_INVENTORY.md.
• Classify each graphiti_core import as delete, compatibility, migration, admin, or test-only.
• Add owners and removal conditions for any newly discovered import.
• Expand the inventory check when a path can otherwise drift outside the hand-authored coverage rule.

Files:

• docs/architecture/SURREALDB_GRAPHITI_EXIT_INVENTORY.md
• docs/research/rust-port/INVENTORY.md
• tools/inventory/runtime_surface.py
• tools/tests/test_runtime_surface.py

Verify:

• moon run inventory-check inventory-typecheck inventory-test

Exit criteria:

• Every generated Graphiti import is classified.
• The generated and hand-authored inventories fail closed on drift.

Wave 2: Native Graph Read Parity

Purpose: remove Graphiti from entity and relationship reads used by recall, context, wake, and API graph surfaces.

Prerequisite:

• Spec Milestone A is green: W2.5 scoreboard plus the full Wave 1 memory policy helper set from SURREALDB_NATIVE_MEMORY_CORE_SPEC.md. That includes read decisions plus default-deny authorize_memory_write, authorize_memory_share, and authorize_memory_reflect decisions with stable reason codes.

Tasks:

• Add native exact entity lookup with deterministic normalization and tenant scoping.
• Authorize candidate hydration through the read-side memory policy helper before results can enter recall, context, wake, or API graph responses.
• Add native entity hydration for the typed models currently coerced from Graphiti nodes.
• Add native relationship hydration for relates_to, mentions, and explicit task-learning links.
• Move episode mention reads behind a native relationship manager.
• Preserve source IDs, confidence, temporal metadata, and policy metadata in hydrated records.

Files:

• packages/python/sibyl-core/src/sibyl_core/graph/entities.py
• packages/python/sibyl-core/src/sibyl_core/graph/relationships.py
• packages/python/sibyl-core/src/sibyl_core/tasks/workflow.py
• packages/python/sibyl-core/tests/test_graph_entities.py
• packages/python/sibyl-core/tests/test_graph_relationships.py
• packages/python/sibyl-core/tests/test_no_graphiti_default_loop.py

Verify:

• moon run core:test -- tests/test_graph_entities.py
• moon run core:test -- tests/test_graph_relationships.py
• moon run core:no-graphiti-smoke

Exit criteria:

• Native mode can hydrate entities and relationships without graphiti_core.
• The stale Procedure.category = None coercion class of failure is covered by a fixture or model normalization test.

Wave 3: Native Graph Write Parity

Purpose: remove Graphiti from default writes for sibyl add, task completion learnings, reflection promotion, and API entity creation.

Tasks:

• Route summarized remember/add writes through native entity, episode, and relationship services.
• Route task completion artifacts and learning links through native relationships.
• Route persisted reflection promotion through native graph writes by default, with SIBYL_NATIVE_WRITE=disabled as the explicit compatibility rollback.
• Call authorize_memory_write before any native write originating in tools/add.py or jobs/entities.py, and call authorize_memory_reflect before task-completion learning writes in tasks/workflow.py.
• Add at least one deny-case test for each native write and reflect surface touched by this wave.
• Remove fallback writes for structured entity types after native dedupe and exact lookup cover the same behavior.
• Keep compatibility writes available only through explicit compatibility or migration surfaces.

Files:

• apps/api/src/sibyl/jobs/entities.py
• packages/python/sibyl-core/src/sibyl_core/tools/add.py
• packages/python/sibyl-core/src/sibyl_core/tools/reflect.py
• packages/python/sibyl-core/src/sibyl_core/tasks/workflow.py
• packages/python/sibyl-core/tests/test_no_graphiti_default_loop.py
• apps/api/tests/test_jobs_entities.py
• apps/api/tests/test_tasks_workflow.py

Verify:

• moon run core:no-graphiti-smoke
• moon run api:test -- tests/test_jobs_entities.py
• moon run api:test -- tests/test_tasks_workflow.py

Exit criteria:

• Default writes create native Surreal records with source provenance and centralized policy decisions from sibyl_core.auth.memory_policy.
• Compatibility writes are no longer reachable by default CLI, MCP, API, or job flows.

Wave 4: Native Embedding Ownership

Purpose: stop carrying Graphiti-shaped embedder contracts in native retrieval.

Tasks:

• Introduce a native embedding service with cache ownership independent of Graphiti embedder types.
• Move Gemini embedding support behind the native service.
• Route vector writes and vector search through the native service.
• Keep deterministic embedder configuration surfaced in eval metadata.
• Leave cached_embedder.py and gemini_embedder.py as named compatibility code until Wave 5 removes Graphiti client construction. Native paths must stop importing them in this wave.

Files:

• packages/python/sibyl-core/src/sibyl_core/graph/cached_embedder.py
• packages/python/sibyl-core/src/sibyl_core/graph/gemini_embedder.py
• packages/python/sibyl-core/src/sibyl_core/graph/client.py
• packages/python/sibyl-core/src/sibyl_core/retrieval/native.py
• packages/python/sibyl-core/tests/test_graph_client.py
• packages/python/sibyl-core/tests/test_native_retrieval.py

Verify:

• moon run core:test -- tests/test_graph_client.py
• moon run core:test -- tests/test_native_retrieval.py
• moon run baseline-seed
• moon run baseline-replay-runtime
• moon run core:bench-context -- --cases benchmarks/context_pack_cases.json --auth-manifest .moon/cache/baseline-runtime-manifest.json --label retrieval-compare --repeat 20 --metadata retrieval_mode=compare

Exit criteria:

• Native retrieval and writes do not depend on Graphiti embedder interfaces.
• Eval reports include deterministic embedding model, dimensions, index settings, and tokenizer metadata.

Wave 5: Runtime Construction Cutover

Purpose: make Graphiti construction impossible on default paths.

Tasks:

• Replace default graph runtime construction with native Surreal managers.
• Move Graphiti client construction behind an explicit compatibility factory.
• Keep all graphiti_core imports inside compatibility factory functions so native app boot, CLI entrypoints, MCP tools, and prompt hooks do not import Graphiti at module import time.
• Remove Graphiti search-interface calls from compare mode once native retrieval is default and covered by seeded evals.
• Keep SIBYL_NATIVE_WRITE default enabled after Wave 3 contract tests and the no-Graphiti smoke are green. Document rollback as a flag flip plus a raw-capture rebuild or replay path.
• Keep admin and migration surfaces named so they cannot be mistaken for default runtime behavior.
• Expand no-Graphiti smoke coverage for CLI, MCP, API job, and prompt-hook entrypoints.

Files:

• apps/api/src/sibyl/persistence/graph_runtime.py
• packages/python/sibyl-core/src/sibyl_core/graph/client.py
• packages/python/sibyl-core/src/sibyl_core/graph/search_interface.py
• packages/python/sibyl-core/src/sibyl_core/graph/surreal/compat/ops/*
• packages/python/sibyl-core/tests/test_no_graphiti_default_loop.py

Verify:

• moon run inventory-check inventory-typecheck inventory-test
• moon run core:no-graphiti-smoke
• moon run api:test
• moon run cli:test

Exit criteria:

• No default path constructs Graphiti.
• Native writes are default-on with an explicit rollback envelope.
• Remaining Graphiti code is named compatibility, migration, admin, or test-only.

Wave 6: Dependency Boundary

Purpose: decide whether Graphiti remains as an optional compatibility dependency or is deleted from runtime packaging.

Tasks:

• Audit pyproject.toml, extras, lockfile, imports, and docs for Graphiti dependency ownership.
• Move Graphiti dependencies to an explicit compatibility extra if migration/admin surfaces still need them.
• Remove Graphiti from default runtime dependencies when no default import remains.
• Update release notes with exact compatibility status.

Files:

• packages/python/sibyl-core/pyproject.toml
• uv.lock
• docs/guide/surrealdb-migration-release-notes.md
• docs/architecture/SURREALDB_GRAPHITI_EXIT_INVENTORY.md

Verify:

• moon run inventory-check inventory-typecheck inventory-test
• moon run core:test
• moon run :check

Exit criteria:

• A default install does not need Graphiti for normal memory behavior.
• Any retained Graphiti dependency is opt-in and documented.

4. Epic 2: Pure Surreal Cleanup

Goal: SurrealDB is the only default data service. Legacy services exist only as explicit archive or migration inputs.

Wave 1: Legacy Surface Inventory

Purpose: identify every remaining default-path reference to PostgreSQL, FalkorDB, Redis, and legacy auth/RBAC.

Tasks:

• Regenerate runtime inventory after Graphiti-exit inventory updates.
• Search default runtime code, compose files, charts, CI, docs, and package extras for legacy services.
• Classify each reference as delete, archive, migration, compatibility, test-only, or historical docs.
• Add pure-Surreal checks for any default-path drift that inventory cannot currently detect.

Files:

• docs/architecture/SURREALDB_PHASE3_BURNDOWN.md
• docs/research/rust-port/INVENTORY.md
• tools/inventory/runtime_surface.py
• tools/tests/test_runtime_surface.py
• docker-compose.yml
• docker-compose.prod.yml
• docker-compose.quickstart.yml
• docker-compose.quickstart.test.yml
• compose.e2e.yml
• .devcontainer/docker-compose.yml
• .github/workflows/*
• charts/**

Verify:

• moon run inventory-check inventory-typecheck inventory-test
• targeted rg audit for postgres, falkor, redis, and legacy auth symbols

Exit criteria:

• Every remaining legacy reference has an owner, purpose, and deletion condition.

Wave 2: Archive And Rollback Policy

Purpose: settle what legacy data recovery means before deleting backup and restore branches.

Tasks:

• Define supported archive import formats.
• Define unsupported rollback paths and document why they are unsupported.
• Decide whether PostgreSQL archive support is migration-only or retained as a named import tool.
• Close the deferred graph export/import policy from Phase 3 Lane 5: drop it, keep it as a named import path, or replace it with a Surreal-native export.
• Make archive import commands require explicit input files and mode flags.
• Add tests for archive import refusing ambient database connections.

Files:

• docs/guide/surrealdb-migration-release-notes.md
• docs/architecture/SURREALDB_PHASE3_BURNDOWN.md
• archive or migration command modules under apps/api/src/sibyl/ and packages/python/sibyl-core/

Verify:

• targeted archive import tests
• moon run api:test
• moon run core:test

Exit criteria:

• Recovery is file-based or explicitly configured.
• Graph export/import policy is no longer deferred.
• No default command reaches for PostgreSQL, FalkorDB, or Redis as a live dependency.

Wave 3: Close Legacy Auth And RBAC Residue

Purpose: verify Phase 3 Lane 1 remains complete and remove only the residue that still appears in inventory, docs, or explicit migration commands.

Tasks:

• Confirm generated inventory still has no active legacy auth/RBAC runtime modules.
• Delete empty legacy auth/RBAC directories or shims when no import needs them.
• Decide whether the auth-readonly freeze/unfreeze command remains as a historical migration tool or is removed.
• Remove stale settings, environment variables, tests, and docs that imply a default legacy auth store.
• Confirm project-scoped recall and refresh-token behavior still use Surreal auth stores.
• Add regression coverage for revoked sessions and expired bearer tokens.

Files:

• apps/api/src/sibyl/cli/migrate.py
• apps/api/src/sibyl/persistence/legacy/**
• apps/api/src/sibyl/auth/**
• apps/api/tests/test_auth*.py
• packages/python/sibyl-core/src/sibyl_core/auth/**
• docs that mention legacy auth migration

Verify:

• moon run api:test -- tests/test_auth*.py
• moon run core:test -- tests/test_memory_policy.py

Exit criteria:

• Phase 3 Lane 1 remains true: legacy auth/RBAC is deleted or historical migration-only.
• Login, refresh, project-scoped recall, and owner-only debug routes remain green.

Wave 4: Close Legacy Content, Settings, And Backup Residue

Purpose: verify Phase 3 Lanes 2 and 3 remain complete, then remove or isolate the remaining archive, settings, backup, and coordination residue after policy is settled.

Tasks:

• Treat content and Surreal archive export/restore as already shipped. Audit only the residue.
• Remove settings branches that imply PostgreSQL, FalkorDB, or Redis are default data services.
• Decide whether Redis coordination remains explicit opt-in or moves out of the v0.7 default surface entirely.
• Remove backup and restore code that assumes live legacy services.
• Keep only explicit archive import or historical migration code.
• Update tests to assert default config forbids memory-mode production and legacy service fallbacks.

Files:

• apps/api/src/sibyl/persistence/**
• apps/api/src/sibyl/persistence/settings_runtime.py
• apps/api/src/sibyl/persistence/settings_types.py
• apps/api/src/sibyl/services/settings.py
• apps/api/src/sibyl/jobs/backup.py
• apps/api/src/sibyl/cli/migrate.py
• apps/api/src/sibyl/persistence/backups_common.py
• packages/python/sibyl-core/src/sibyl_core/migrate/archive.py
• packages/python/sibyl-core/src/sibyl_core/config*.py
• backup, restore, and migration modules
• related API/core tests

Verify:

• moon run api:test
• moon run core:test
• moon run inventory-check inventory-typecheck inventory-test

Exit criteria:

• Default config has no live PostgreSQL, FalkorDB, or Redis fallback.
• Backup and restore docs describe only supported Surreal archive flows.

Wave 5: Runtime, CI, And Chart Cleanup

Purpose: make the operational surface match the code.

Tasks:

• Remove legacy service containers from default compose and CI workflows.
• Keep SurrealDB as the only required database service in default CI.
• Remove legacy chart values, secrets, jobs, env vars, and docs.
• Ensure dev docs use Surreal container service in CI and Surreal local runtime by default.
• Keep legacy migration docs clearly marked as historical or archive-only.

Files:

• .github/workflows/*
• docker-compose.yml
• docker-compose.prod.yml
• docker-compose.quickstart.yml
• docker-compose.quickstart.test.yml
• compose.e2e.yml
• .devcontainer/docker-compose.yml
• charts/**
• README.md
• apps/api/README.md
• apps/cli/README.md
• docs/guide/why-surreal.md

Verify:

• CI green
• moon run :check
• moon run docs:lint
• chart render tests when available

Exit criteria:

• Default CI does not start PostgreSQL, FalkorDB, or Redis.
• Default docs do not instruct users to run legacy services.
• Charts render without legacy secrets or jobs.

Wave 6: Final Pure Surreal Release Audit

Purpose: prove the release surface is coherent.

Tasks:

• Run the full default install and local dev path against SurrealDB only.
• Run inventory and dependency checks from a clean checkout.
• Run the no-Graphiti smoke with legacy services absent.
• Audit docs for stale migration-era instructions.
• Audit benchmark evidence for full artifact-backed AI memory results. Missing external suites must stay explicit, not implied by context-pack or LongMemEval-style evidence. Check benchmarks/results/ai-memory/manifest.json and gate every citable artifact with moon run bench-gate -- <artifact>.json --profile ai-memory.
• Check release notes against the benchmark evidence table so only suites with full result records are cited.
• Append release notes with supported upgrade and archive import paths.

Verify:

• moon run inventory-check inventory-typecheck inventory-test
• moon run core:test
• moon run api:test
• moon run cli:test
• moon run :check
• CI green
• moon run baseline-seed
• moon run baseline-replay-runtime
• moon run core:bench-context -- --cases benchmarks/context_pack_cases.json --auth-manifest .moon/cache/baseline-runtime-manifest.json --label retrieval-compare --repeat 20 --metadata retrieval_mode=compare
• nightly regression green in .github/workflows/nightly-regression.yml

Exit criteria:

• Default development, CI, and release docs are all pure Surreal.
• Legacy references are only historical, archive, migration, compatibility, or test-only.

5. Suggested Execution Order

1. Graphiti Wave 1: lock inventory before deleting code.
2. Pure Surreal Wave 1: inventory legacy runtime references in parallel.
3. Pure Surreal Wave 2: settle archive and rollback policy.
4. Spec Milestone A: W2.5 scoreboard plus the full Wave 1 memory policy helper set are green.
5. Pure Surreal Wave 3: close legacy auth/RBAC residue.
6. Graphiti Wave 2: native graph read parity.
7. Graphiti Wave 3: native graph write parity.
8. Graphiti Wave 4: native embedding ownership.
9. Graphiti Wave 5: runtime construction cutover and native-write default flip.
10. Pure Surreal Wave 4: close legacy content, settings, backup, and coordination residue.
11. Pure Surreal Wave 5: clean runtime, CI, charts, and docs.
12. Graphiti Wave 6: dependency boundary and optional compatibility extra.
13. Pure Surreal Wave 6: final release audit.

This order keeps data recovery decisions ahead of destructive cleanup while moving Graphiti removal through measurable parity gates.

6. Risk Register

Risk	Why It Matters	Mitigation
Native retrieval quality regresses	Graphiti removal should not reduce context usefulness	Gate on seeded context evals and compare-mode evidence before default cutover
Legacy records lose visibility	Pre-v0.7 Graphiti-written memories may not have explicit scope/source metadata	Apply the Legacy Projection Rule from SURREALDB_GRAPHITI_EXIT_INVENTORY.md: project records with recoverable scope/source, organization-scope owner-less records with source IDs, and exclude records without recoverable source IDs
Archive cleanup deletes rollback options too early	Users need a clear recovery path after pure-Surreal cleanup	Decide archive policy before deleting backup/restore branches
Compatibility surfaces hide default dependencies	A named fallback can accidentally remain on the hot path	Use no-Graphiti smoke and inventory checks as release blockers
Embedding drift changes eval results	Native retrieval scores become noisy or unreproducible	Pin embedder metadata and tokenizer/index settings in eval reports
Auth cleanup breaks session refresh	Recent login failures showed auth/session behavior is easy to regress	Keep focused refresh, revocation, owner-debug, and project-recall tests in the release gate

7. Task Tracking Shape

Recommended Sibyl tracking:

• Epic: v0.7 Graphiti Exit
  • Task: lock Graphiti exit inventory
  • Task: finish native graph read parity
  • Task: finish native graph write parity
  • Task: move embeddings to native ownership
  • Task: cut over runtime construction
  • Task: decide Graphiti dependency boundary
• Epic: v0.7 Pure Surreal Cleanup
  • Task: inventory legacy runtime surfaces
  • Task: settle archive and rollback policy
  • Task: close legacy auth and RBAC residue
  • Task: close legacy content, settings, backup, and coordination residue
  • Task: clean runtime, CI, charts, and docs
  • Task: run final pure-Surreal release audit

Each task should carry the relevant verification command from this plan and complete with concrete evidence, not a summary-only note.