Sibyl Post-v0.8 Synthesis and Memory Workspace Plan
- Status: shipped as the v0.9 product foundation; still active as 1.0 source, synthesis, inspect, correction, import, and workspace design input.
- Released in:
v0.9.0 - Current roadmap:
SIBYL_1_0_ROADMAP.md - Depends on:
docs/architecture/SIBYL_V08_PURE_SURREAL_CLOSURE_AND_MEMORY_TRUST_PLAN.mddocs/architecture/SIBYL_V08_PURE_SURREAL_CLOSURE_EXECUTION_PLAN.mddocs/architecture/SIBYL_NORTHSTAR.md
- Primary outcome: turn the v0.8 trust substrate into a human-usable and agent-usable memory product with source-grounded synthesis, source inspection, correction, source adapters, and a memory workspace.
2026-05-15 update: read this document as the v0.9 execution plan plus the source of detailed contracts for the 1.0 roadmap. Any manual review language should now be interpreted as exception-only review. Safe, policy-backed memory decisions should become automatic; humans should handle sensitive, ambiguous, contradictory, destructive, or high-impact sharing cases.
v0.8 should leave Sibyl with a Surreal-only default runtime, policy-backed memory access, audit receipts, promotion and share previews, and release gates that prove Graphiti is no longer on the default path. The next large product move is to make that foundation useful: humans can understand and shape memory, and agents can produce large source-grounded artifacts without crossing privacy boundaries.
This plan intentionally keeps the same architecture center as the northstar: raw memory is source truth, graph records are derived explanation and retrieval surface, and authorization is part of retrieval rather than a final output filter.
1. Release Thesis
The post-v0.8 release made Sibyl feel like a real second-brain product. The winning shape is not more storage plumbing. It is a compact loop:
- Ingest source material into policy-scoped raw memory.
- Inspect what was remembered and why it is visible.
- Correct, hide, redact, delete, or promote memory with preview and audit.
- Synthesize grounded artifacts from authorized graph slices.
- Remember the generated artifact with provenance back to the source memory.
The release should be shippable even if broad sharing remains preview-only. The important behavior is that every source, derived fact, generated artifact, and human correction is inspectable, policy-filtered, and citable.
2. Definition Of Done
Post-v0.8 is ready when all of these are true:
synthesizecan generate Markdown and JSON artifacts from authorized memory spaces with section-level source IDs, hidden-context signals, redaction metadata, freshness metadata, and unresolved-claim lists.- Source inspection works across API, CLI, and web for raw source metadata, derived records, visibility, recent audit receipts, freshness, correction history, promotion state, and deletion or redaction status.
- Humans can correct, hide, redact, restore, export, or delete personal memory through policy-aware APIs and at least one friendly web flow.
- A source adapter contract exists and one mailbox-style import path proves resumable, deduplicated, source-preserving ingest into private memory.
- Import progress, skipped records, dedupe decisions, and extraction status are visible without requiring graph vocabulary.
- Promotion and share preview results are visible in CLI and web without enabling uncontrolled broad sharing.
- The memory workspace gives users one place to see recent captures, imports, reflections, recalls, agent access, source inspections, and suggested follow-up.
- MCP, CLI, API, jobs, and prompt hooks reuse the same policy context and audit receipts for read, write, inspect, correction, promotion, sharing preview, and synthesis.
- New release gates catch source-grounding regressions, permission leaks, synthesis citation gaps, adapter ingest drift, and UI trust-flow regressions.
The first shippable product cut does not need every adapter, every workspace panel, or broad sharing. It does need one vertical path that proves the loop end to end:
- A user can inspect an authorized source and see derived memory, visibility, policy, and audit context.
- A user can correct or hide memory and watch that correction change recall and synthesis.
- A user or agent can synthesize a source-backed architecture artifact from authorized project memory.
- The artifact can be remembered with section citations, unresolved claims, and provenance.
- A release gate can fail if citations, policy filtering, correction handling, or provenance drift.
3. Non-Goals
- Do not build arbitrary cross-organization sharing in this release. Keep broad sharing preview-only until real usage hardens the policy contract.
- Do not build a custom policy language. Extend the v0.8 policy primitives only where a concrete flow requires it.
- Do not make source adapters one-off importer branches. The first adapter should prove the contract rather than bypass it.
- Do not let synthesis invent unsupported claims. Missing sources are output gaps, not model creativity opportunities.
- Do not make the workspace a marketing dashboard. It is a working surface for inspection, correction, import progress, agent access, and memory confidence.
- Do not require live queries for the first workspace. Polling or normal query invalidation is acceptable until Surreal live-query behavior has explicit permission tests.
4. Product Stories
Source-Grounded Architecture Synthesis
Bliss asks Sibyl to synthesize an architecture overview for the current project. Sibyl proposes an outline from authorized project memories, materializes source packs per section, drafts the artifact, verifies citations and hidden-context signals, lists unresolved claims, and optionally remembers the final document as an artifact linked to the source memory.
Personal Memory Inspection
Bliss finds a remembered preference and opens its source. Sibyl shows where it came from, what derived facts were created, which agents read it, whether it was promoted, and how to correct, hide, redact, export, or delete it. Private text stays hidden from actors who can only see redacted metadata.
Large Mailbox Import
Bliss imports a mailbox archive into private memory. Sibyl preserves messages, thread IDs, timestamps, participants, attachments, source paths, dedupe keys, and import checkpoints. Lexical metadata search works before embeddings or reflection finish. Nothing enters project memory without an explicit promotion preview and apply action.
Agent Access Preview
Before granting a delegated coding agent access to a project or memory space, Bliss previews what the agent could recall. Sibyl reports visible source IDs, hidden-but-relevant counts, denied source IDs, and stable reason codes without leaking private content.
Correction And Supersession
Bliss marks an old decision stale and links the replacement decision. Future recall and synthesis prefer the newer decision, preserve the old source, expose the correction trail, and list supersession metadata in generated artifacts.
5. Architecture Principles
5.1 Source Truth First
Raw source records remain the durable truth. Ingest, reflection, synthesis, correction, and export must link back to source IDs. Derived graph records can be rebuilt, corrected, or superseded. Source records should preserve enough metadata to explain where they came from and how they were transformed.
5.2 Policy At Every Boundary
Read, write, inspect, correction, promotion, share preview, export, delete, and synthesis all accept the same policy context shape. UI affordances can make common cases pleasant, but the server and core services own decisions and reason codes.
5.3 Preview Before Sensitive Writes
Promotion, sharing, correction with broad effects, deletion, redaction, and synthesis artifact persistence all need previewable consequences. The preview should name source IDs, derived IDs, target scope, expected visibility, hidden counts, and stable allow or deny reasons.
5.4 Derived Artifacts Are Memory Too
Generated architecture docs, release notes, briefs, and reports are artifacts. When saved, they should carry source packs, section citations, unresolved gaps, policy metadata, author actor, target audience, and freshness timestamp.
5.5 Human Language Before Graph Vocabulary
The web flow should say "source", "visible to", "used by", "hidden", "redacted", "stale", "replacement", and "promoted" before it says edge, node, relation, or namespace. Power surfaces can expose graph details after the human has context.
5.6 Deterministic Before Generative
Synthesis planning, source selection, policy filtering, citation coverage, freshness checks, and verification should be deterministic. LLM drafting can turn verified section packs into prose, but it should not decide which hidden records are safe, which citations are mandatory, or which claims are supported.
5.7 Audit Metadata Is Not Source Content
Audit receipts should be rich enough to answer who, what, where, why, and under which policy decision. They should not copy private message bodies, document text, generated paragraphs, or raw attachment content. Receipts carry IDs, counts, reason codes, hashes, timestamps, and bounded metadata.
5.8 Jobs Carry Authority Explicitly
Background jobs must not infer actor authority from environment, default organization, queue name, or current server process. They carry a serialized policy context, reauthorize before writes, and fail closed when the payload is missing or stale.
5.9 Product Surfaces Share One Contract
The API is the contract owner. CLI, MCP, web, prompt hooks, and jobs may shape presentation, but they should not fork authorization, redaction, citation, correction, or synthesis semantics.
5.10 Full-Text Search Is An Accelerator, Not Truth
Surreal full-text indexes and vector search can accelerate recall and source discovery. They do not own lifecycle state, visibility, source truth, correction state, or citation guarantees. Those live in explicit records and policy-aware services.
6. Workstreams
Track A: Memory Space Productization
Purpose: turn v0.8 memory spaces and policy into a product surface rather than internal plumbing.
Build:
- Memory-space CRUD and membership APIs for private, delegated, project, team, organization, shared, and public scopes.
- Stable disabled-scope behavior for team, organization, shared, and public writes until explicit enablement.
- Graph projections for explanation and traversal, not authorization truth.
- CLI and web listing for spaces, membership, visibility, and disabled reason.
- Agent access preview using the same share-preview primitives from v0.8.
Exit criteria:
- A user can list memory spaces, see who or what can read them, and preview an agent's effective recall surface.
- Disabled scopes fail closed with stable reason codes.
- Project memory resolves through canonical control-plane records rather than graph-only inference.
Track B: Source Inspect, Correction, And Lifecycle
Purpose: make memory accountable and fixable.
Build:
- Source-centered inspect endpoint and CLI command.
- Derived-record, audit-receipt, correction, promotion, and visibility summaries.
- Correction actions: mark wrong, stale, sensitive, duplicate, superseded, hidden, redacted, restored, or deleted.
- Supersession links that retrieval and synthesis can understand.
- Export and deletion previews with source and derived-record counts.
- Audit receipts for every lifecycle action.
Exit criteria:
- Owners and admins can explain why memory exists and where it was used.
- Normal readers never receive hidden content through inspect.
- Corrections influence recall and synthesis without destroying source truth.
Track C: Source Adapter And Raw Ingest Pipeline
Purpose: make large-corpus import boring and resumable.
Build:
- Source adapter protocol with source identity, version token, metadata schema, privacy class, transformation behavior, dedupe key, checkpoint, and attachment behavior.
- Import records with status, counters, current checkpoint, skipped records, errors, dedupe stats, and extraction status.
- First mailbox-style adapter for MBOX or Maildir archives.
- Private-memory ingest default with explicit promotion as the only path into project or shared spaces.
- Background extraction hooks that can run after raw metadata search is ready.
Exit criteria:
- A mailbox-like archive can be imported, stopped, resumed, searched, inspected, and audited.
- Attachments and skipped records are represented without blocking the whole import.
- Imported private records cannot appear in project or organization recall without explicit policy-backed promotion.
Track D: Graph-Guided Synthesis
Purpose: build large-read output on top of recall, provenance, and policy.
Build:
- Synthesis request model with principal, delegated principal, organization, project, memory spaces, output type, audience, depth, seed query, source IDs, entity IDs, freshness policy, and optional outline.
- Outline planner that proposes sections from graph neighborhoods, active plans, decisions, artifacts, raw sources, and unresolved gaps.
- Section-pack materializer that reuses authorized context-pack retrieval.
- Draft artifact renderer for Markdown and JSON.
- Verification pass for source coverage, redactions, freshness, unsupported claims, hidden-but-relevant signals, and missing-source gaps.
- Optional artifact persistence through
rememberwith source links.
Exit criteria:
- Seeded synthesis produces an architecture overview with source IDs per section.
- Forbidden-scope memories are absent or represented only as redacted hidden signals.
- Unsupported claims are listed as gaps rather than written as facts.
- The generated artifact can be remembered with provenance.
Track E: Human Memory Workspace
Purpose: make memory visible, understandable, and controllable in the web UI.
Build:
- Memory home showing recent captures, imports, reflections, recalls, agent access, source inspections, and suggested review actions.
- Scope controls for private, delegated, project, team, organization, shared, and public memory.
- Source inspect panel with raw metadata, derived facts, visibility, audit, and correction actions.
- Import progress panel with counters, skipped records, dedupe, attachments, and extraction status.
- Promotion and share preview UI with visible, hidden, denied, and redacted counts.
- Synthesis runner UI with outline review, section source packs, draft preview, verification results, and remember action.
Exit criteria:
- A human can create or import private memory, find it, inspect it, correct it, preview promotion, synthesize from it, and understand agent access without graph vocabulary.
- UI tests cover the trust flows instead of only happy-path rendering.
Track F: Evaluation And Release Gates
Purpose: make trust and quality regressions block the release.
Build:
synthesis-gatefor citation coverage, hidden-scope absence, unresolved-gap reporting, output shape, and artifact provenance.adapter-ingest-gatefor resumability, dedupe, attachment accounting, private scope enforcement, and searchable metadata before embedding completion.- UI trust-flow tests for inspect, correction, promotion preview, share preview, import progress, and synthesis verification.
- Benchmark fixtures for agent behavior with and without synthesized project docs.
- Release evidence ledger that distinguishes local receipts from CI receipts.
Exit criteria:
- One command can prove the post-v0.8 product claims.
- Release notes cite gated artifacts, not manual anecdotes.
7. Data Model Additions
Prefer additive records with migration-safe defaults.
Control Plane
memory_spaces- ID, organization ID, scope, scope key, name, description, state, created by, created at, updated at.
memory_space_members- Space ID, principal type, principal ID, role, permissions, expires at, created by, created at.
memory_access_previews- Actor, target principal, spaces, requested action, visible counts, denied counts, reason codes, created at.
Source Ingest
source_imports- Source adapter, source identity, source version, privacy class, target memory scope, checkpoint, status, counters, dedupe stats, error summaries.
source_records- Adapter record ID, source import ID, raw memory ID, dedupe key, metadata, attachment refs, privacy class, transform version.
source_attachments- Source record ID, filename, media type, size, checksum, storage pointer, extraction state.
Lifecycle
memory_corrections- Source ID, derived ID, action, reason, replacement ID, actor, policy context, created at, reversible flag.
memory_redactions- Source ID, field path, redaction reason, replacement summary, actor, created at.
memory_deletions- Source ID, derived IDs, deletion mode, preview counts, actor, created at, retention metadata.
Synthesis
synthesis_runs- Request metadata, actor, memory spaces, output type, audience, status, verification summary, artifact ID, created at.
synthesis_sections- Run ID, title, outline path, source IDs, hidden counts, unresolved claims, freshness metadata, section text, verification status.
8. API Surface
Initial REST shape:
GET /api/memory/spacesPOST /api/memory/spacesGET /api/memory/spaces/{space_id}POST /api/memory/spaces/{space_id}/members/previewPOST /api/memory/spaces/{space_id}/membersGET /api/memory/inspect/{source_id}POST /api/memory/inspect/{source_id}/corrections/previewPOST /api/memory/inspect/{source_id}/correctionsPOST /api/memory/export/previewPOST /api/memory/delete/previewPOST /api/memory/deletePOST /api/sources/importsGET /api/sources/imports/{import_id}POST /api/sources/imports/{import_id}/resumePOST /api/sources/imports/{import_id}/cancelPOST /api/synthesize/planPOST /api/synthesize/sectionsPOST /api/synthesize/draftPOST /api/synthesize/verifyPOST /api/synthesize/remember
Every endpoint must return stable reason codes for policy denial and must avoid embedding hidden source text in error detail.
9. CLI And MCP Surface
CLI:
sibyl memory-space listsibyl memory-space inspect <space-id>sibyl memory-space preview-agent <agent-id>sibyl memory-inspect <source-id>sibyl memory-correct <source-id> --previewsibyl memory-correct <source-id> --applysibyl memory-delete <source-id> --previewsibyl source-import start <path> --adapter <name>sibyl source-import status <import-id>sibyl source-import resume <import-id>sibyl synthesize plansibyl synthesize draftsibyl synthesize verifysibyl synthesize remember
MCP tools:
memory_space_listmemory_inspectmemory_correct_previewsource_import_statussynthesize_plansynthesize_draftsynthesize_verify
MCP synthesis should prefer structured JSON output with concise Markdown rendering only when the caller explicitly requests it.
10. Web Surface
Routes:
/memory/memory/sources/[sourceId]/memory/imports/memory/imports/[importId]/memory/spaces/memory/spaces/[spaceId]/memory/synthesize/memory/synthesize/[runId]
Primary components:
MemoryHomeMemoryActivityFeedMemoryScopeSwitcherSourceInspectPanelSourceCorrectionDialogMemoryPromotionPreviewMemorySharePreviewSourceImportProgressSynthesisRunnerSynthesisOutlineEditorSynthesisSectionSourcesSynthesisVerificationPanel
The first screen should be the working memory workspace, not a landing page. It should show recent memory activity, pending review, import progress, and quick actions for inspect, import, and synthesize.
11. System Contracts
This section turns the plan into implementation contracts. Packets can adjust names while coding, but they should preserve these semantics unless the doc is intentionally revised first.
11.1 Memory Policy Contract
Every trust-sensitive operation receives or derives a MemoryPolicyContext before touching memory:
- actor user ID
- organization ID and organization role
- accessible project IDs
- accessible delegated authority IDs
- agent ID when an agent acts on behalf of a user
- requested memory scope and scope key
- requested project ID when project memory is involved
- source surface such as
api,cli,mcp,job,prompt_hook, orweb
Operations covered by this contract:
- raw memory read and write
- context pack render
- wake, recall, search, and reflection render
- reflection promotion
- task-learning episode and procedure creation
- source inspect
- correction, hide, redact, restore, supersede, export, and delete preview or apply
- promotion and share preview or apply
- source import start, resume, extraction, and promotion
- synthesis plan, section materialization, draft, verify, and remember
Stable deny reasons:
missing_actormissing_organizationmissing_memory_spacemissing_scope_keymissing_project_idscope_disabledunverified_membershipdelegation_requireddelegation_expiredunsupported_scopehidden_sourceredacted_sourcesource_deletedjob_policy_context_missingjob_policy_context_stale
Policy decisions return the same high-level shape everywhere:
{
"allowed": false,
"reason": "unverified_membership",
"memory_scope": "project",
"scope_key": "project_123",
"source_surface": "mcp",
"audit_id": "audit_..."
}The response may include redacted counts and denied IDs when the actor is allowed to know that hidden context exists. It must not include hidden source text.
11.2 Source And Derived Memory Contract
Raw source records are the root of trust. Derived graph entities, episodes, procedures, artifacts, facts, and relationships point back to source IDs. Retrieval may rank derived records, but inspect, correction, deletion, export, synthesis, and audit should all be able to walk from a derived record to its source.
Minimum source record fields:
idorganization_idmemory_scopescope_keysource_typesource_urior source-local identitysource_versioncontent_hashmetadataprivacy_classcreated_bycreated_atupdated_atlifecycle_state
Lifecycle states:
active: available to authorized recall and synthesishidden: excluded from normal recall, inspectable by owners/adminsredacted: content replaced or partially masked, metadata preservedsuperseded: retained but ranked below replacement recordsduplicate: retained for provenance, grouped under canonical sourcedeleted: content unavailable except retention metadata when policy requires it
Derived records carry:
- source IDs and source hashes
- transform version
- policy metadata from the write decision
- freshness timestamp
- correction and supersession pointers
- confidence or extraction quality when applicable
11.3 Inspect Contract
Source inspect answers four questions:
- What is this memory?
- Why is it visible or hidden to this actor?
- What did Sibyl derive from it?
- What actions can this actor safely take next?
Inspect responses include:
- source metadata and redacted content preview when allowed
- derived entity, relationship, episode, procedure, artifact, and fact IDs
- visibility summary by scope and principal type
- lifecycle state and correction history
- promotion or share state
- recent audit receipts
- freshness and transform version metadata
- available actions with preview-required flags
Inspect never leaks hidden content through nested audit details, derived snippets, error messages, or action labels.
11.4 Correction Contract
Correction actions are previewable before apply. The preview returns affected source IDs, derived IDs, relationship IDs, recall impact, synthesis impact, reversible flag, and audit action name.
Correction actions:
mark_wrongmark_stalemark_sensitivemark_duplicatesupersedehideredactrestoredelete
Retrieval and synthesis must understand these actions:
- Hidden and deleted records are excluded unless the actor explicitly requests an inspect/admin surface.
- Superseded records can be used as historical context only when the current replacement is also visible.
- Redacted records can contribute metadata and hidden-context counts, not redacted text.
- Duplicate records collapse under the canonical source for synthesis citations.
11.5 Source Adapter Contract
Adapters expose source records without owning memory policy. The import service owns target scope, policy context, persistence, audit, and job behavior.
Adapter responsibilities:
- identify a source corpus and version
- iterate source records deterministically
- emit source-local IDs, timestamps, participants, labels, metadata, body text, and attachments
- emit a stable dedupe key
- emit a checkpoint token after bounded batches
- classify privacy and transform requirements
- report skipped records with reasons
Import service responsibilities:
- verify target memory scope before start and resume
- persist import manifest, counters, checkpoint, errors, and dedupe stats
- write raw source records before extraction
- preserve attachments or attachment metadata according to storage policy
- enqueue extraction and embedding as follow-up work
- default all personal corpus imports to private memory
- require explicit promotion preview before project or shared visibility
Import states:
pendingrunningpausedcompletedfailedcancelled
Record extraction states:
metadata_readycontent_readyextraction_pendingextractedembedding_pendingindexedskippedfailed
11.6 Synthesis Contract
Synthesis runs as a staged pipeline:
plan: build an outline and expected evidence requirements.sections: materialize policy-filtered source packs per section.draft: render Markdown or JSON from section packs.verify: check citation coverage, hidden-context handling, freshness, and unsupported claims.remember: persist the generated artifact through normal memory write policy.
The first implementation should support a deterministic non-LLM path through plan, sections, and verify. Drafting can use templates or an LLM adapter, but it receives only authorized section packs.
Synthesis request fields:
- actor and organization context
- optional delegated principal or agent ID
- target project or memory-space set
- output type
- audience
- seed query
- source IDs and entity IDs
- freshness policy
- outline override
- output format
Section packs contain:
- section title and outline path
- source IDs
- source snippets or structured facts allowed for the actor
- derived IDs
- hidden-but-relevant counts
- denied source counts
- redaction metadata
- freshness metadata
- unresolved claim candidates
- policy reason codes
Verification statuses:
passedpassed_with_gapsblocked_missing_sourcesblocked_policy_denialblocked_unsupported_claimsblocked_stale_sources
Generated artifacts are memory. Persisted artifacts must include:
- source pack IDs or source IDs per section
- generated text hash
- unresolved claims
- verification summary
- actor and policy context
- freshness timestamp
- output type and audience
11.7 Workspace Contract
The workspace is a working surface for memory trust. The default screen prioritizes action over analytics:
- recent captures and imports
- pending corrections or review suggestions
- recent recalls and context renders
- agent access previews
- source inspections
- synthesis drafts and verification gaps
- import progress and skipped records
Every workspace action that changes visibility, lifecycle, promotion, sharing, deletion, or generated artifact persistence must use preview before apply. The UI can make happy paths fast, but the server remains the source of truth for policy and consequences.
11.8 Release Cut Lines
The post-v0.8 work should ship in visible cuts:
Cut 1: Trust Inspection
- Memory-space listing and access preview.
- Source inspect API, CLI, and first web panel.
- Correction preview for hide, redact, stale, and supersede.
memory-trust-gateremains green.
Cut 2: Source-Grounded Synthesis
- Synthesis plan, section materialization, verification, and artifact remember.
- CLI and MCP synthesis surfaces.
synthesis-gateblocks missing citations, hidden source text, and unsupported claims.
Cut 3: Memory Workspace
- Working
/memoryworkspace. - Inspect, correction, import progress, and synthesis UI.
- Component and browser checks cover the trust flows.
Cut 4: Large Source Import
- Source adapter contract.
- First mailbox adapter.
- Resumable import jobs and private-memory default.
adapter-ingest-gateblocks dedupe, checkpoint, and policy regressions.
Cut 5: Release Audit
- Clean final gates.
- CI, docs deploy, nightly, memory trust, synthesis, adapter, and benchmark receipts.
- Binary ship or hold recommendation.
12. Implementation Packets
Each packet should land as one atomic commit unless the implementation exposes a smaller natural boundary. Every packet needs focused tests, lint/typecheck for touched packages, git diff --check, and a receipt in this document or the release evidence ledger.
Packet 0.1: v0.8 Release Baseline
Depends on:
- v0.8 A6 pure Surreal release audit.
- v0.8 B6 memory trust gate.
Files:
docs/architecture/SIBYL_V08_PURE_SURREAL_CLOSURE_AND_MEMORY_TRUST_PLAN.mddocs/architecture/SIBYL_POST_V08_SYNTHESIS_AND_MEMORY_WORKSPACE_PLAN.md
Implementation:
- Record the final v0.8 release commit, CI run IDs, nightly run IDs, benchmark artifacts, and trust-gate receipts.
- Lock the starting state for this plan.
Verify:
moon run docs:lint- v0.8 release gates from the v0.8 plan.
Exit criteria:
- This plan starts from current evidence, not assumptions.
Receipt, 2026-05-14:
- Starting evidence baseline: pushed
maincommit4855ba8a. - Main CI run
25870913035, docs deploy run25877971558, and nightly regression run25877971585completed successfully on4855ba8a. - v0.8 claim boundary: Surreal-only default runtime, optional Graphiti compatibility, policy-backed inspectable memory, promotion preview, share preview, and the
memory-trust-gate. - Post-v0.8 owns persisted MemorySpace CRUD, source lifecycle and correction, source adapters,
synthesize, mailbox import, and the memory workspace.
Packet A1: Memory Space Records
Files:
apps/api/src/sibyl/persistence/surreal/auth_runtime.pyapps/api/src/sibyl/persistence/auth_runtime.pyapps/api/src/sibyl/api/routes/memory.pyapps/api/src/sibyl/api/schemas.pypackages/python/sibyl-core/src/sibyl_core/auth/memory_policy.pyapps/api/tests/test_surreal_auth_runtime.pyapps/api/tests/test_routes_memory.pypackages/python/sibyl-core/tests/test_memory_policy.py
Implementation:
- Add control-plane memory-space and membership records.
- Add owner/admin CRUD for space metadata.
- Keep disabled scopes closed with stable reason codes.
- Project memory spaces must map to canonical project records.
Verify:
moon run api:test -- tests/test_surreal_auth_runtime.py tests/test_routes_memory.pymoon run core:test -- tests/test_memory_policy.pymoon run api:lint api:typecheck core:lint core:typecheck
Exit criteria:
- Memory-space membership can be resolved without graph lookups.
- Disabled scopes fail closed.
Packet A2: Agent Access Preview
Files:
packages/python/sibyl-core/src/sibyl_core/services/native_memory.pyapps/api/src/sibyl/api/routes/memory.pyapps/api/src/sibyl/api/schemas.pyapps/cli/src/sibyl_cli/main.pyapps/cli/src/sibyl_cli/client.pyapps/api/tests/test_routes_memory.pyapps/cli/tests/test_main_capture.py
Implementation:
- Add preview for what a user, agent, or delegated principal can recall from a selected memory-space set.
- Return visible source IDs, hidden counts, denied IDs, reason codes, and audit receipt IDs when available.
- Reuse share-preview redaction and deny behavior.
Verify:
moon run api:test -- tests/test_routes_memory.pymoon run cli:test -- tests/test_main_capture.pymoon run api:lint api:typecheck cli:lint cli:typecheck
Exit criteria:
- Access preview is non-mutating and policy-equivalent to recall.
Packet B1: Source Inspect Surface
Files:
apps/api/src/sibyl/api/routes/memory.pyapps/api/src/sibyl/api/schemas.pyapps/api/src/sibyl/persistence/auth_runtime.pyapps/api/src/sibyl/persistence/surreal/auth_runtime.pypackages/python/sibyl-core/src/sibyl_core/services/native_memory.pyapps/cli/src/sibyl_cli/main.pyapps/cli/src/sibyl_cli/client.pyapps/api/tests/test_routes_memory.pyapps/api/tests/test_surreal_auth_runtime.pyapps/cli/tests/test_main_capture.py
Implementation:
- Add source inspect API and CLI.
- Return source metadata, derived IDs, correction history, promotion state, recent audit receipts, visibility metadata, and freshness metadata.
- Redact content unless the actor can read the source through normal memory policy.
Verify:
moon run api:test -- tests/test_routes_memory.py tests/test_surreal_auth_runtime.pymoon run cli:test -- tests/test_main_capture.pymoon run api:lint api:typecheck cli:lint cli:typecheck
Exit criteria:
- Inspect explains memory without leaking hidden content.
Packet B2: Correction And Lifecycle Actions
Files:
packages/python/sibyl-core/src/sibyl_core/services/native_memory.pypackages/python/sibyl-core/src/sibyl_core/retrieval/native.pypackages/python/sibyl-core/tests/test_native_memory.pypackages/python/sibyl-core/tests/test_native_retrieval.pyapps/api/src/sibyl/api/routes/memory.pyapps/api/src/sibyl/api/schemas.pyapps/api/tests/test_routes_memory.py
Implementation:
- Add preview and apply paths for stale, wrong, sensitive, duplicate, superseded, hidden, redacted, restored, and deleted states.
- Make retrieval prefer non-superseded and non-hidden records while preserving source history.
- Emit audit receipts for every lifecycle action.
Verify:
moon run core:test -- tests/test_native_memory.py tests/test_native_retrieval.pymoon run api:test -- tests/test_routes_memory.pymoon run core:lint core:typecheck api:lint api:typecheck
Exit criteria:
- Corrections affect recall and synthesis while preserving source truth.
Packet C1: Source Adapter Contract
Files:
packages/python/sibyl-core/src/sibyl_core/models/sources.pypackages/python/sibyl-core/src/sibyl_core/services/source_adapters.pypackages/python/sibyl-core/tests/test_source_adapters.pyapps/api/src/sibyl/api/routes/crawler.pyapps/api/src/sibyl/api/routes/memory.pyapps/api/src/sibyl/api/schemas.py
Implementation:
- Define adapter protocol, import manifest, source record, attachment record, checkpoint, dedupe key, and privacy-class behavior.
- Keep adapters source-preserving and policy-aware.
- Add deterministic fake adapter tests.
Verify:
moon run core:test -- tests/test_source_adapters.pymoon run api:test -- tests/test_routes_memory.pymoon run core:lint core:typecheck api:lint api:typecheck
Exit criteria:
- New adapters can be added without branching core import behavior.
Packet C2: Mailbox Import Adapter
Files:
packages/python/sibyl-core/src/sibyl_core/services/source_adapters.pypackages/python/sibyl-core/src/sibyl_core/services/mailbox_adapter.pypackages/python/sibyl-core/tests/test_mailbox_adapter.pyapps/api/src/sibyl/jobs/source_imports.pyapps/api/tests/test_jobs_source_imports.py
Implementation:
- Implement one mailbox-style adapter, preferably MBOX first unless Maildir is simpler against available fixtures.
- Preserve message IDs, thread IDs when available, timestamps, participants, subject, body, attachments, source path, and dedupe key.
- Default target scope to private memory.
Verify:
moon run core:test -- tests/test_mailbox_adapter.pymoon run api:test -- tests/test_jobs_source_imports.pymoon run core:lint core:typecheck api:lint api:typecheck
Exit criteria:
- A mailbox archive can be imported into private source records without extraction or embedding completing first.
Packet C3: Resumable Import Jobs And Progress
Files:
apps/api/src/sibyl/jobs/source_imports.pyapps/api/src/sibyl/api/routes/crawler.pyapps/api/src/sibyl/api/routes/memory.pyapps/api/src/sibyl/api/schemas.pyapps/api/tests/test_jobs_source_imports.pyapps/api/tests/test_routes_memory.py
Implementation:
- Add start, status, resume, cancel, and progress counters for source imports.
- Persist checkpoints after bounded batches.
- Record skipped records, dedupe counts, errors, attachments, and extraction pending counts.
- Make import jobs fail closed without actor and target-scope policy context.
Verify:
moon run api:test -- tests/test_jobs_source_imports.py tests/test_routes_memory.pymoon run api:lint api:typecheck
Exit criteria:
- Interrupted imports can resume without duplicate raw records.
- Import progress is readable and source-safe.
Packet D1: Synthesis Request And Plan Contract
Files:
packages/python/sibyl-core/src/sibyl_core/models/synthesis.pypackages/python/sibyl-core/src/sibyl_core/services/synthesis.pypackages/python/sibyl-core/tests/test_synthesis.pyapps/api/src/sibyl/api/routes/synthesis.pyapps/api/src/sibyl/api/app.pyapps/api/src/sibyl/api/schemas.pyapps/api/tests/test_routes_synthesis.py
Implementation:
- Add synthesis request, outline, section, source-pack, verification, and run models.
- Add a non-LLM outline planner from seed query, entity IDs, decisions, tasks, artifacts, and graph neighborhoods.
- Return gaps when no source supports a requested section.
Verify:
moon run core:test -- tests/test_synthesis.pymoon run api:test -- tests/test_routes_synthesis.pymoon run core:lint core:typecheck api:lint api:typecheck
Exit criteria:
- Synthesis planning is deterministic and source-aware before drafting exists.
Packet D2: Section Pack Materialization
Files:
packages/python/sibyl-core/src/sibyl_core/services/synthesis.pypackages/python/sibyl-core/src/sibyl_core/tools/context.pypackages/python/sibyl-core/tests/test_synthesis.pypackages/python/sibyl-core/tests/test_context_pack.pyapps/api/tests/test_routes_synthesis.py
Implementation:
- Reuse context-pack retrieval for each outline section.
- Track source IDs, hidden counts, redactions, freshness, and unresolved claims per section.
- Apply policy before materialization and again before render.
Verify:
moon run core:test -- tests/test_synthesis.py tests/test_context_pack.pymoon run api:test -- tests/test_routes_synthesis.pymoon run core:lint core:typecheck api:lint api:typecheck
Exit criteria:
- Section packs never contain unauthorized source text.
Packet D3: Draft, Verify, And Remember Artifact
Files:
packages/python/sibyl-core/src/sibyl_core/services/synthesis.pypackages/python/sibyl-core/src/sibyl_core/models/synthesis.pypackages/python/sibyl-core/tests/test_synthesis.pyapps/api/src/sibyl/api/routes/synthesis.pyapps/api/tests/test_routes_synthesis.py
Implementation:
- Render Markdown and JSON artifacts from section packs.
- Verify citation coverage, hidden-context handling, freshness, unsupported claims, and unresolved gaps.
- Add optional artifact persistence that writes generated artifacts through normal
rememberbehavior with source links.
Verify:
moon run core:test -- tests/test_synthesis.pymoon run api:test -- tests/test_routes_synthesis.pymoon run core:lint core:typecheck api:lint api:typecheck
Exit criteria:
- The first synthesis artifact is citable, policy-safe, and rememberable.
Packet D4: CLI And MCP Synthesis
Files:
apps/cli/src/sibyl_cli/main.pyapps/cli/src/sibyl_cli/client.pyapps/cli/tests/test_main_capture.pyapps/api/src/sibyl/server.pypackages/python/sibyl-core/src/sibyl_core/tools/synthesis.pypackages/python/sibyl-core/tests/test_tools.py
Implementation:
- Add CLI commands for plan, draft, verify, and remember.
- Add MCP tools for synthesis plan, draft, and verify.
- Prefer JSON output for agent calls and readable Markdown for humans.
Verify:
moon run cli:test -- tests/test_main_capture.pymoon run core:test -- tests/test_tools.pymoon run api:test -- tests/test_mcp_auth.py tests/test_server_accessible_projects.pymoon run cli:lint cli:typecheck core:lint core:typecheck api:lint api:typecheck
Exit criteria:
- Agents can synthesize through MCP without bypassing policy context.
Packet E1: Memory Workspace Foundation
Files:
apps/web/src/lib/api.tsapps/web/src/lib/constants/navigation.tsapps/web/src/app/(main)/memory/page.tsxapps/web/src/app/(main)/memory/memory-content.tsxapps/web/src/components/memory/memory-home.tsxapps/web/src/components/memory/memory-activity-feed.tsxapps/web/src/components/memory/memory-scope-switcher.tsxapps/web/src/components/layout/sidebar.tsxapps/web/src/lib/constants/navigation.test.tsapps/web/src/app/(main)/memory/page.test.tsx
Implementation:
- Add memory route and navigation.
- Render recent captures, imports, reflections, recalls, agent access, and pending review actions.
- Keep layout dense, calm, and task-oriented.
Verify:
moon run web:test -- src/app/\\(main\\)/memory/page.test.tsx src/lib/constants/navigation.test.tsmoon run web:lint web:typecheck
Exit criteria:
- Memory has a real working home in the app.
Packet E2: Source Inspect And Correction UI
Files:
apps/web/src/lib/api.tsapps/web/src/app/(main)/memory/sources/[sourceId]/page.tsxapps/web/src/components/memory/source-inspect-panel.tsxapps/web/src/components/memory/source-correction-dialog.tsxapps/web/src/components/memory/source-visibility-summary.tsxapps/web/src/components/memory/source-inspect-panel.test.tsx
Implementation:
- Render source metadata, derived records, audit summaries, visibility, correction history, and available lifecycle actions.
- Add correction preview and apply flow.
- Never render hidden text when API marks content redacted.
Verify:
moon run web:test -- src/components/memory/source-inspect-panel.test.tsxmoon run web:lint web:typecheck
Exit criteria:
- Human inspect and correction flows are usable without graph vocabulary.
Packet E3: Import And Synthesis UI
Files:
apps/web/src/lib/api.tsapps/web/src/app/(main)/memory/imports/page.tsxapps/web/src/app/(main)/memory/synthesize/page.tsxapps/web/src/components/memory/source-import-progress.tsxapps/web/src/components/memory/synthesis-runner.tsxapps/web/src/components/memory/synthesis-outline-editor.tsxapps/web/src/components/memory/synthesis-verification-panel.tsxapps/web/src/components/memory/synthesis-runner.test.tsx
Implementation:
- Show import status, counters, skipped records, dedupe, attachments, and extraction status.
- Add synthesis runner with outline review, section source packs, draft preview, verification gaps, and remember action.
- Keep share and promotion previews visible but do not enable broad share writes.
Verify:
moon run web:test -- src/components/memory/synthesis-runner.test.tsxmoon run web:lint web:typecheck
Exit criteria:
- A user can watch ingest progress and synthesize a source-backed artifact from the web UI.
Packet F1: Post-v0.8 Product Gates
Files:
moon.ymltools/trust/synthesis_gate.pytools/trust/adapter_ingest_gate.pytools/tests/test_synthesis_gate.pytools/tests/test_adapter_ingest_gate.pybenchmarks/context_pack_cases.jsondocs/testing/benchmark-methodology.md
Implementation:
- Add
synthesis-gateandadapter-ingest-gatemoon tasks. - Require source IDs per section, hidden-scope absence, unresolved-gap reporting, artifact provenance, import resumability, dedupe correctness, and private scope enforcement.
- Document benchmark and artifact locations.
Verify:
moon run synthesis-gate adapter-ingest-gatemoon run bench-gatemoon run docs:lint
Exit criteria:
- Product claims have repeatable local gates.
Packet F2: Release Audit
Files:
docs/architecture/SIBYL_POST_V08_SYNTHESIS_AND_MEMORY_WORKSPACE_PLAN.md- release notes draft
Implementation:
- Run the final local gates from a clean worktree.
- Record CI, docs deploy, nightly, synthesis, adapter, memory trust, and bench receipts.
- Write a binary ship or hold recommendation.
Verify:
moon run memory-trust-gatemoon run synthesis-gatemoon run adapter-ingest-gatemoon run bench-gatemoon run core:testmoon run api:testmoon run cli:testmoon run web:testmoon run docs:lintmoon run :check- CI green on
main - nightly regression green on
main
Exit criteria:
- The release can claim source-grounded synthesis, inspectable and correctable memory, source-preserving ingest, and a usable memory workspace.
Receipt, 2026-05-15:
- Local release-audit candidate: current local
main, ahead oforigin/mainby the v0.9 commit stack. The tracked worktree was clean after the gate repairs; local untracked scratch files were left untouched. - Product surface delivered:
- Source-grounded synthesis across service, CLI, MCP, API, and web with section source IDs, unresolved-gap reporting, hidden-source filtering, artifact provenance, and remember provenance.
- Inspectable and correctable memory through source inspection, correction preview/apply, audit receipts, and unified Memory workspace capture review.
- Source-preserving ingest through adapter contracts, mailbox import, resumable jobs, dedupe keys, private-memory defaults, and import progress UI.
- Unified workspace UX: Memory owns captures, imports, synthesis, review, and source inspection; the legacy Archive route redirects into Memory Captures with query parameters preserved.
- Local verification:
moon run memory-trust-gate-> PASS, 7 checks and 0 failed. Covered surfaces: audit, CLI, context pack, inspect, jobs, MCP, memory policy, promotion preview, prompt hook, raw memory, recall, reflect, share preview, task learning, and wake.moon run synthesis-gate-> PASS, 2 checks and 0 failed. Covered surfaces: artifact provenance, hidden-scope absence, remember provenance, source IDs per section, and unresolved-gap reporting.moon run adapter-ingest-gate-> PASS, 2 checks and 0 failed. Covered surfaces: dedupe correctness, import resumability, private scope enforcement, source adapter contract, and source-preserving ingest.moon run bench-gate-> Gate passed forbenchmarks/results/ai-memory/manifest.json.moon run core:test-> 932 passed, 14 skipped, 20 deselected in 7.19s.moon run api:test-> 1467 passed, 1 skipped, 16 deselected in 14.53s.moon run cli:test-> 174 passed in 8.70s.moon run web:test-> 26 files passed, 102 tests passed.moon run web:test-cov-> 26 files passed, 102 tests passed after the synthesis runner coverage test was stabilized.- Post-unified-UX update:
moon run web:lint web:typecheck web:test-> 3 tasks completed, 102 tests passed. moon run docs:lint-> all matched files use Prettier code style.moon run :check-> 40 tasks completed, 26 cache hits before the receipt docs were written; post-doc rerun completed 36 tasks with 33 cache hits.git diff --check-> clean.
- Gate repairs made during audit:
docs/.prettierignorenow ignores generateddocs/research/*dumps and local Gradial brainstorm scratch notes so docs lint evaluates tracked documentation rather than ignored local artifacts.docs/research/rust-port/INVENTORY.mdand its guard constants now include the synthesis REST router and the three MCP synthesis tools.tools/tests/test_dev_scripts.pynow uses non-login bash for the legacy detector test so the docker stub remains onPATHandmoon run :checkdoes not hang on real local Docker.
- GitHub receipts before the v0.8.1 rebase:
- Latest pushed
origin/mainwasf8d23e6450ec86dfd8251f0c94e6804cdbcc4f76. - CI run
25879991056succeeded onf8d23e6450ec86dfd8251f0c94e6804cdbcc4f76. - Docs deploy run
25879991026succeeded onf8d23e6450ec86dfd8251f0c94e6804cdbcc4f76. - Release run
25891107401succeeded onf8d23e6450ec86dfd8251f0c94e6804cdbcc4f76. - Latest nightly regression receipt is run
25877971585, successful on4855ba8ad8be6be958ba720e81b3459e727a973b. - CI-only PR #7 (
codex/v09-ci) succeeded on candidate headse944a1d3a81dc0f1c840a053394d59c9c61bce30in run25898456003andbc5bf7c33e5459c60819a7fa00880cf39e1cca0ein run25898597780. - Nightly Regression run
25898704879succeeded onbc5bf7c33e5459c60819a7fa00880cf39e1cca0e. - CI-only PR #7 (
codex/v09-ci) succeeded after the v0.8.1 rebase and unified captures polish one05a52c01a183876c4b9247203e329856edc293c; run25899235827passed Build, Static Checks, Package Tests, E2E, Storybook, and Detect Changes. - Nightly Regression run
25899328897succeeded one05a52c01a183876c4b9247203e329856edc293c. origin/mainlater advanced to the v0.8.1 release bumpf11bfede36b4a02df2b4a514bd2ffb2be555ebb2; the v0.9 candidate was rebased on top of that commit and rerun through CI-only plus nightly evidence.
- Latest pushed
- Binary recommendation: HOLD for tag or public release until the local 0.9 candidate reaches the final branch and docs deploy is green on the exact candidate head. The product and local verification are ready; release is not yet publishable without Bliss approval to push
mainand deploy docs.
13. Verification Matrix
| Surface | Gate | Required before |
|---|---|---|
| Memory spaces | moon run api:test -- tests/test_routes_memory.py | A2 |
| Policy context | moon run memory-trust-gate | every packet |
| Source inspect | moon run api:test -- tests/test_routes_memory.py | B2, E2 |
| Correction lifecycle | moon run core:test -- tests/test_native_memory.py | D1 |
| Adapter contract | moon run core:test -- tests/test_source_adapters.py | C2 |
| Mailbox ingest | moon run adapter-ingest-gate | F2 |
| Synthesis | moon run synthesis-gate | F2 |
| CLI | moon run cli:test | D4, F2 |
| MCP | moon run api:test -- tests/test_mcp_auth.py | D4, F2 |
| Web workspace | moon run web:test | E3, F2 |
| Full release | moon run :check | F2 |
14. Risk Register
| Risk | Why it matters | Mitigation |
|---|---|---|
| Synthesis writes unsupported claims | Users may trust generated docs too much | Require source IDs and unresolved-gap output |
| Inspect leaks hidden source text | Trust UI becomes a privacy bug | Server-side redaction and negative tests |
| Source adapters become one-offs | Ingest complexity fragments quickly | Contract-first fake adapter before mailbox adapter |
| Corrections delete source truth | Future audits lose provenance | Lifecycle metadata over destructive delete by default |
| Memory workspace becomes ornamental | Users still cannot fix memory | Build inspect/correct/import/synthesize flows first |
| Broad sharing ships too early | Policy surface expands before trust | Keep sharing preview-only until explicit enablement |
| Live UI outruns permission tests | Realtime can leak state changes | Defer live queries until permission fixtures pass |
| Import jobs bypass actor policy | Background workers become a side channel | Serialize policy context into job payloads |
15. Open Questions
- Should the first mailbox adapter be MBOX or Maildir?
- Should source attachments be stored inside SurrealDB records, content storage, or filesystem-backed blobs with metadata in SurrealDB?
- Which synthesis output ships first: architecture overview, release notes, onboarding guide, audit packet, or custom Markdown?
- Should generated artifacts be persisted automatically after verification, or only after explicit
remember? - What is the first web correction flow: mark stale, hide, redact, delete, or supersede?
- How much redacted hidden-context signal should synthesis expose without overemphasizing unavailable context?
- Should organization and team memory remain preview-only for the entire post-v0.8 release?
- What exact UI test runner should own workspace end-to-end flows if component tests are not enough?
- Which source adapter should follow mailbox import: repo snapshot, chat export, docs crawl, calendar export, or Haven event history?
16. Recommendation
Start with Track B and Track D in a narrow vertical slice:
- Build source inspect and correction primitives.
- Build deterministic synthesis planning and section packs.
- Add draft, verification, and artifact remember.
- Add the minimum workspace UI around those flows.
This sequence makes the product immediately more useful while preserving the trust model. Source inspect gives humans confidence. Synthesis gives agents a powerful large-read primitive. The workspace turns both into a visible product. Mailbox import should follow once inspect and lifecycle controls are solid, because large personal corpora are only safe after people can see and correct what Sibyl remembered.
