Helm Chart Reference

Complete reference for the Sibyl Helm chart (charts/sibyl).

Chart Info

apiVersion: v2
name: sibyl
description: Knowledge graph and task workflow for durable development memory
type: application
version: 1.4.10
appVersion: "1.4.10"

Release builds update version and appVersion from the repository VERSION file.

Installation

Release charts are published to the sibyl Helm repository on the gh-pages branch. The repository serves two charts: sibyl (the application) and sibyl-surrealdb (a SurrealDB wrapper).

Choose where completed validation receipts live before installation. An S3 bucket (backend.validationReceipts.url) lets API and worker run on any node with no shared volume. A persistent claim (backend.validationReceipts.existingClaim) must be shared by API and worker, and multi-node replicas then require ReadWriteMany storage. See Durable Validation Receipts. Organization backup archives follow the same choice: by default they live on the receipts claim, and with receipts in S3 they need their own bucket prefix or claim. See Backup Archives.

# From the published Helm repository (recommended)
helm repo add sibyl https://raw.githubusercontent.com/hyperb1iss/sibyl/gh-pages
helm repo update

helm upgrade --install sibyl sibyl/sibyl \
  -n sibyl \
  --create-namespace \
  -f values.yaml

To install from a source checkout instead:

# From local chart
helm upgrade --install sibyl ./charts/sibyl \
  -n sibyl \
  --create-namespace \
  -f values.yaml

# Dry run
helm template sibyl ./charts/sibyl -f values.yaml

Global Settings

global:
  # Image pull secrets for private registries
  imagePullSecrets: []

Schema Bootstrap

Sibyl bootstraps SurrealDB schemas at application startup. The chart runs no separate migration job.

Authentication Defaults

The chart defaults match the self-hosted single-user path. Local username/password login is enabled, the first setup signup creates the owner/admin user, post-setup account creation is invite-only, and OIDC is empty until an operator configures it.

auth:
  # Local username/password login is the default simple path.
  localAuthEnabled: true
  # Public account creation after setup stays invite-only by default.
  publicSignupsEnabled: false

oidc:
  providers: []
  silent_refresh_enabled: false
  extra_providers_enabled: false

breakGlass:
  enabled: false

For enterprise SSO deployments, configure a corporate OIDC provider and set auth.localAuthEnabled=false only after an owner has successfully signed in through OIDC. Break-glass access remains a separate, bounded opt-in.

Each configured provider requires an exact organization binding:

oidc:
  providers:
    - name: entra
      issuer: "https://login.microsoftonline.com/<tenant-id>/v2.0"
      client_id: "<app-client-id>"
      client_secret_env: "SIBYL_OIDC_ENTRA_CLIENT_SECRET"
      organization_slug: "acme"

The organization must already exist and must not be personal. Helm rejects providers without this binding, and Sibyl fails closed if the bound organization cannot be resolved.

Backend Configuration

Basic Settings

publicUrl: "https://sibyl.example.com"

backend:
  # Number of replicas (ignored if autoscaling is enabled)
  replicaCount: 1

  # Empty selects Recreate for a single fixed replica, RollingUpdate otherwise
  strategy: {}

  image:
    repository: ghcr.io/hyperb1iss/sibyl-api
    pullPolicy: IfNotPresent
    # Defaults to chart appVersion if empty
    tag: ""

Set publicUrl to the external application origin. The chart passes the value to the backend for callbacks, password resets, and redirects. It also derives the browser API and WebSocket origin. Use frontend.publicApiUrl only when the browser reaches the API through a different origin.

Service

backend:
  service:
    type: ClusterIP
    port: 3334
    annotations: {}

The backend needs no session affinity. MCP runs stateless, so any replica can answer any request.

Autoscaling

backend:
  autoscaling:
    enabled: false
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 80
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 300
        policies:
          - type: Percent
            value: 10
            periodSeconds: 60
      scaleUp:
        stabilizationWindowSeconds: 0
        policies:
          - type: Percent
            value: 100
            periodSeconds: 15
          - type: Pods
            value: 4
            periodSeconds: 15
        selectPolicy: Max

Pod Disruption Budget

backend:
  pdb:
    enabled: false
    # Minimum available pods (mutually exclusive with maxUnavailable)
    minAvailable: 1
    # maxUnavailable: 1

Pod Anti-Affinity

backend:
  podAntiAffinity:
    # Spreads pods across nodes
    enabled: false
    # "soft" (preferred) or "hard" (required)
    type: soft
    topologyKey: kubernetes.io/hostname

Resources

backend:
  resources:
    limits:
      cpu: 1000m
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 256Mi

Health Probes

backend:
  livenessProbe:
    httpGet:
      path: /api/health
      port: http
    initialDelaySeconds: 60
    periodSeconds: 30

  readinessProbe:
    httpGet:
      path: /api/health/ready
      port: http
    initialDelaySeconds: 5
    periodSeconds: 10

The readiness probe must hit /api/health/ready, the deep readiness endpoint that returns 503 when SurrealDB is unreachable. Liveness stays on /api/health, which only asserts the process is up.

Environment Variables

backend:
  env:
    SIBYL_SERVER_HOST: "0.0.0.0"
    SIBYL_SERVER_PORT: "3334"
    SIBYL_ENVIRONMENT: "production"
    SIBYL_LLM_PROVIDER: "anthropic"
    SIBYL_LLM_MODEL: "claude-haiku-5-5"
    SIBYL_EMBEDDING_MODEL: "text-embedding-3-small"
    SIBYL_EMBEDDING_DIMENSIONS: "1536"
    # BLAS/OpenMP thread caps keep native math libraries from oversubscribing pods
    OPENBLAS_NUM_THREADS: "1"
    OMP_NUM_THREADS: "1"
    MKL_NUM_THREADS: "1"
    NUMEXPR_NUM_THREADS: "1"

Trusted Proxies

Behind an ingress controller or Gateway, every request reaches the backend from the proxy pod's address. The login route allows five attempts per minute per client address, so until the backend trusts the proxy, all users share one bucket and the sixth login in a minute from anyone locks everyone out. The backend.forwardedAllowIps value names the proxies whose X-Forwarded-For header the backend believes. The chart renders it into SIBYL_FORWARDED_ALLOW_IPS.

Value	Default	Description
backend.forwardedAllowIps	""	IPs and CIDR ranges of trusted proxies, string or list

backend:
  # A range only the ingress controller pods draw their addresses from, never the pod CIDR.
  forwardedAllowIps: "10.250.0.0/28"
  # A list works too:
  # forwardedAllowIps:
  #   - 10.250.0.0/28
  #   - 10.250.1.0/28

On the command line, pass a list so Helm does not split on the comma: --set 'backend.forwardedAllowIps={10.250.0.0/28,10.250.1.0/28}'.

Empty (the default) keeps the backend's loopback-only trust. When backend.forwardedAllowIps is empty, a SIBYL_FORWARDED_ALLOW_IPS entry under backend.env still works; when both are set, the dedicated value wins.

Every trusted address must be a proxy, never something that could be a client. A trusted peer can name any client address it likes, and an ingress that appends to X-Forwarded-For instead of replacing it extends that power to everyone who calls through it from a trusted address. Envoy-based gateways, AWS ALB, and nginx's $proxy_add_x_forwarded_for all append. With the pod CIDR trusted, a pod at 10.244.9.9 that sends X-Forwarded-For: 100.101.102.103 through such an ingress arrives as 100.101.102.103, 10.244.9.9; the backend skips the pod's own address as trusted and resolves the forged one. A NetworkPolicy cannot stop this, because the request really does come from the controller. So never trust the cluster pod CIDR (it covers every pod, and with networkPolicy.enabled at its default of false any of them can also reach the backend directly), node ranges, or the VPC CIDR: each of them holds clients.

What to set:

• Trust only the ingress controller pods. Give them addresses nothing else draws from, such as a CNI IP pool selected by the controller's namespace or a node pool reserved for the controller with its own pod range, and list that range. Running kubectl get pods -n <controller-namespace> -o wide shows the addresses it has to cover.
• Make the controller replace X-Forwarded-For from untrusted peers rather than append to it. ingress-nginx does by default, as long as use-forwarded-headers and compute-full-forwarded-for stay off. With a controller that can only append, the trust list is all that keeps a forged entry out, so it must hold nothing but the controller.
• A load balancer in front of the controller: resolve the client at the controller rather than widening the backend's list. Have the controller trust only the load balancer's dedicated subnets (in ingress-nginx, use-forwarded-headers: true with proxy-real-ip-cidr set to them, because its default is 0.0.0.0/0), so it still hands the backend a single resolved address, and keep the backend trusting the controller alone.
• An L4 load balancer or a node hop that SNATs: do not trust node addresses. The controller then only ever sees node IPs, and trusting them hands the entry the client wrote the final say. Preserve the source address instead (externalTrafficPolicy: Local on the controller's Service, or PROXY protocol from the load balancer) and trust only the controller pods.
• Enable networkPolicy with networkPolicy.ingress.from naming the controller, so nothing can reach the backend directly and write the header itself.

Trusting a range lets anything inside it choose its client address

Any peer inside forwardedAllowIps can claim to be any client, dodge the per-address rate limits, and satisfy breakGlass.allowedIPs, and so can any caller behind an appending ingress whose own address is trusted. List the controller and nothing else. Keep /api and /mcp routed straight to the backend service (the default route table): the Next.js frontend passes a client-supplied X-Forwarded-For through unchanged, so it must never be the hop the backend trusts. A value of "*", or any range broader than an IPv4 /8 or an IPv6 /32, trusts every peer or nearly every one and makes the backend log forwarded_allow_ips_trusts_every_peer at startup; with every hop trusted, the leftmost header entry wins, and a client writes that one unless every proxy in front overwrites the header.

To confirm the setting took, sign in through the ingress and check that the backend's request log lines carry your own address in client, not the controller pod's. The environment reference explains how the backend resolves the address and which values it accepts.

Secrets

backend:
  # Reference to existing secret for sensitive env vars
  # Must contain: SIBYL_JWT_SECRET and SIBYL_SETTINGS_KEY.
  # Add provider and LLM API keys as needed; Amazon Bedrock needs none.
  existingSecret: ""

Production and read-only containers must set existingSecret. The chart enforces this at render time: with backend.env.SIBYL_ENVIRONMENT set to production and no backend.existingSecret, helm template and helm install both abort with an error naming the value to set. That guard exists because an unset SIBYL_JWT_SECRET makes the backend sign sessions with an empty key and makes MCP auth disable itself under the default mcp_auth_mode: auto. The chart deliberately does not auto-generate the secret in production, since a per-render value would rotate on every upgrade and diverge across the backend, worker, and bootstrap pods.

Passing the signing key as backend.env.SIBYL_JWT_SECRET fails the render as well, and so does the unprefixed backend.env.JWT_SECRET alias that the server falls back to. Every key under backend.env is written into the plaintext <release>-config ConfigMap, which is readable under broader RBAC than a Secret and routinely committed to GitOps repositories. Key names are matched case-insensitively, because pydantic-settings resolves environment variables that way.

Production also rejects backend.env.SIBYL_MCP_AUTH_MODE: off, which would serve every MCP tool unauthenticated no matter how well the JWT secret is provisioned. Use auto to enforce Bearer auth once a secret is set, or on to enforce it unconditionally. The server refuses to boot on the same condition, so the mode cannot be switched off in production by editing the ConfigMap after install.

The escape hatch for a trial install is backend.env.SIBYL_ENVIRONMENT: development, which is unsuitable for real traffic: the server then derives a JWT secret per process, and the chart's default readOnlyRootFilesystem: true blocks the write that would persist it, so sessions break on every restart and across replicas.

Both required keys are non-optional pod references, so a missing key fails before Sibyl starts instead of silently generating an ephemeral replacement. Keep SIBYL_SETTINGS_KEY stable across upgrades and pod replacements; changing it makes encrypted settings unreadable.

Storage Mode

The active persistence runtime is fixed to SurrealDB. Local and auto coordination run jobs in the API process and do not render a separate worker. Set Redis coordination explicitly for multi-pod deployments:

coordinationBackend: "redis"

See storage-modes.md for the mode matrix.

SurrealDB Connection (default)

backend:
  surreal:
    # ws:// or http:// URL to an external SurrealDB instance
    url: "ws://surrealdb:8000/rpc"
    username: "root"
    # Reference to a secret containing the password.
    # When empty, a password is auto-generated and stored in `<release>-surreal`.
    existingSecret: ""
    # Secret key holding the password (only used when existingSecret is set)
    secretKey: "password"
    # Inline password (ignored when existingSecret is set; auto-generated otherwise)
    password: ""
    namespacePrefix: "org_"
    database: "graph"

The chart renders backend.surreal.url into SIBYL_SURREAL_URL. Point it at a SurrealDB server (ws://, wss://, http://, or https://). The backend refuses an unsupported scheme at startup: rocksdb:// and tikv:// are storage arguments for the SurrealDB server, not client URLs. See SurrealDB URL forms.

Redis or Valkey Coordination

Used when coordinationBackend: "redis". This is recommended when running more than one backend or worker pod because it backs arq jobs, distributed locks, WebSocket pub/sub, and shared rate limits.

backend:
  redis:
    host: "valkey"
    port: "6379"
    jobsDb: "1"
    rateLimitDb: "4"
    # Reference to a secret containing the Redis/Valkey password
    existingSecret: ""
    secretKey: "password"
    # Inline password (not recommended; prefer existingSecret)
    password: ""
    # Shared rate-limit storage URL.
    # Leave empty to derive a redis:// URL from host/port/rateLimitDb.
    rateLimitStorage: ""

The chart emits a password-free SIBYL_RATE_LIMIT_STORAGE ConfigMap value and Sibyl injects the Redis password from SIBYL_REDIS_PASSWORD at runtime.

Security Contexts

The default UIDs match the packaged images for this chart version. If you intentionally pin older image tags, keep the image tag and security context in lockstep: pre-service-UID API/worker images use 1000:1000, and pre-service-UID web images use 1001:65533.

backend:
  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 10001
    runAsGroup: 10001
    fsGroup: 10001
    fsGroupChangePolicy: OnRootMismatch

  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL

Pod Placement

backend:
  nodeSelector: {}
  tolerations: []
  # Custom affinity (overridden by podAntiAffinity if enabled)
  affinity: {}
  podAnnotations: {}

Frontend Configuration

frontend:
  enabled: true
  replicaCount: 1

  image:
    repository: ghcr.io/hyperb1iss/sibyl-web
    pullPolicy: IfNotPresent
    tag: ""

  service:
    type: ClusterIP
    port: 3337

  autoscaling:
    enabled: false
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 80
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 300
        policies:
          - type: Percent
            value: 10
            periodSeconds: 60

  pdb:
    enabled: false
    minAvailable: 1

  podAntiAffinity:
    enabled: false
    type: soft
    topologyKey: kubernetes.io/hostname

  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 50m
      memory: 128Mi

  livenessProbe:
    httpGet:
      path: /
      port: http
    initialDelaySeconds: 10
    periodSeconds: 30

  readinessProbe:
    httpGet:
      path: /
      port: http
    initialDelaySeconds: 5
    periodSeconds: 10

  env:
    NODE_ENV: "production"
    NEXT_TELEMETRY_DISABLED: "1"

  # Defaults to http://<release>-backend:<port>/api when empty
  apiUrl: ""
  # Optional browser-visible API URL. When empty, publicUrl supplies <origin>/api.
  publicApiUrl: ""

  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 10002
    runAsGroup: 10002
    fsGroup: 10002

  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: false # Next.js needs write access
    capabilities:
      drop:
        - ALL

  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAnnotations: {}

Worker Configuration

The chart renders this deployment only when both worker.enabled is true and coordinationBackend is redis. Local and auto coordination keep job execution in the API process.

worker:
  enabled: true
  replicaCount: 1

  # Uses same image as backend
  # (worker is sibyl backend container with different entrypoint)

  autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 5
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 80
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 300
        policies:
          - type: Percent
            value: 10
            periodSeconds: 60

  pdb:
    enabled: false
    minAvailable: 1

  podAntiAffinity:
    enabled: false
    type: soft
    topologyKey: kubernetes.io/hostname

  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 50m
      memory: 128Mi

  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 10001
    runAsGroup: 10001
    fsGroup: 10001
    fsGroupChangePolicy: OnRootMismatch

  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL

  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAnnotations: {}

Ingress Configuration

Ingress is split into a shared route table (ingress.hosts) plus two independently toggled renderers: classic networking.k8s.io/v1 Ingress under ingress.classic, and Gateway API gateway.networking.k8s.io/v1 HTTPRoute under ingress.gatewayApi. Enable whichever your cluster uses. The legacy ingress.enabled flag still works as a compatibility toggle for classic Ingress.

ingress:
  # Shared route table consumed by both classic Ingress and Gateway API HTTPRoute.
  hosts:
    - host: sibyl.local
      paths:
        - path: /api
          pathType: Prefix
          service: backend
        - path: /mcp
          pathType: Prefix
          service: backend
        - path: /
          pathType: Prefix
          service: frontend
  classic:
    # Render a networking.k8s.io/v1 Ingress.
    enabled: false
    # Ingress class name. Empty leaves controller selection to cluster policy.
    className: ""
    annotations: {}
    tls: []
    # - secretName: sibyl-tls
    #   hosts:
    #     - sibyl.example.com
  gatewayApi:
    # Render a gateway.networking.k8s.io/v1 HTTPRoute.
    enabled: false
    annotations: {}
    parentRefs: []
    # - name: shared-gateway
    #   namespace: gateway-system
    #   sectionName: https
    # Optional hostnames override. Defaults to ingress.hosts[*].host.
    hostnames: []

Network Policy

Renders a default-deny NetworkPolicy plus explicit allows for frontend/backend ingress and SurrealDB (and optional Redis/Valkey) egress. Leave disabled unless your cluster enforces NetworkPolicies.

networkPolicy:
  # Enable default-deny plus explicit app allows.
  enabled: false
  ingress:
    # Sources allowed to reach frontend/backend, usually ingress controller pods.
    from: []
    # - namespaceSelector:
    #     matchLabels:
    #       kubernetes.io/metadata.name: ingress-nginx
  egress:
    # Allow DNS egress for selected Sibyl pods.
    allowDns: true
    dnsPorts:
      - 53
    # Extra egress rules appended to backend/worker policies.
    extra: []
  surrealdb:
    # Destination selectors for the SurrealDB service/pods.
    to: []
    ports:
      - 8000
  redis:
    # Enable Redis/Valkey egress allow rules.
    enabled: false
    to: []
    ports:
      - 6379

Pod Security

Labels the release namespace with Pod Security Admission enforce/audit/warn at the restricted level. Enable for hardened multi-tenant clusters.

podSecurity:
  # Label the release namespace with Pod Security restricted enforce/audit/warn.
  enforceRestricted: false
  version: "latest"

Bootstrap

A post-install/post-upgrade Job that seeds the first organization and an optional default memory space, so a fresh install lands ready to use. Disabled by default.

bootstrap:
  enabled: false
  organization:
    name: "Sibyl"
    slug: ""
  memorySpace:
    enabled: true
    name: "Default memory"
    scope: "private"
    scopeKey: ""
  job:
    annotations: {}
    backoffLimit: 3
    ttlSecondsAfterFinished: 300
    podAnnotations: {}

Break-Glass

Bounded emergency local-owner login for SSO outages. Keep disabled in normal operation; when enabled, both expiresAt (no more than four hours out) and allowedIPs are required. The allowlist is matched against the resolved client address, so behind an ingress set backend.forwardedAllowIps (see Trusted Proxies); without it the backend only ever sees the controller pod's address.

breakGlass:
  enabled: false
  # Source CIDRs allowed to use break-glass login when enabled.
  allowedIPs: []
  # UTC timestamp after which app-level break-glass login is denied.
  expiresAt: ""
  # Existing secret containing owner bootstrap fields.
  existingSecret: ""
  ownerEmailKey: "owner-email"
  ownerPasswordKey: "owner-password"

Service Account

serviceAccount:
  create: true
  name: ""
  annotations: {}

The backend, worker and bootstrap pods run as this service account, so an annotation here reaches every pod that calls a model provider or the S3 receipt store.

The frontend only calls the Sibyl API, so it runs under its own account, <fullname>-frontend, created with no annotations and no mounted API token:

frontend:
  serviceAccount:
    create: true
    name: ""
    annotations: {}
    automountServiceAccountToken: false

To keep the earlier shared account, set frontend.serviceAccount.create: false and frontend.serviceAccount.name to the backend account's name.

Amazon Bedrock

On EKS, Sibyl can run Claude and Cohere embeddings through Amazon Bedrock with no provider API keys. Bind an IAM role to the service account with IRSA, set a Region, and select the bedrock provider:

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/sibyl-bedrock

backend:
  # Still required in production, but it only needs SIBYL_JWT_SECRET and SIBYL_SETTINGS_KEY.
  existingSecret: sibyl-secrets
  env:
    AWS_REGION: "us-west-2"
    SIBYL_LLM_PROVIDER: "bedrock"
    SIBYL_LLM_MODEL: "claude-haiku-5-5"
    SIBYL_LLM_MEMORY_MODEL: "claude-opus-5-5"
    SIBYL_BEDROCK_INFERENCE_SCOPE: "us"
    SIBYL_EMBEDDING_PROVIDER: "bedrock"
    SIBYL_EMBEDDING_MODEL: "cohere.embed-v4:0"
    SIBYL_EMBEDDING_DIMENSIONS: "1536"
    SIBYL_GRAPH_EMBEDDING_PROVIDER: "bedrock"

The chart renders backend.env into the ConfigMap the worker also loads, and the worker loads the same backend.existingSecret into its environment and runs as the same service account, so a separate worker gets these settings and the role with no extra values.

The IRSA webhook injects AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE into each pod, and the AWS credential chain exchanges that token for role credentials and refreshes them before they expire. EKS Pod Identity works the same way through an association on the service account, with no annotation. Both paths leave SIBYL_ANTHROPIC_API_KEY and SIBYL_OPENAI_API_KEY unset: the chart references them from backend.existingSecret as optional keys, so the Secret does not need them.

The role needs bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream on the inference profiles and foundation models it routes to, such as us.anthropic.claude-*, anthropic.claude-* and cohere.embed-v4:0. See Amazon Bedrock for every setting, the inference scopes and the features Bedrock accepts per model. The admin AI settings page Test button proves the role works from inside the pod.

With networkPolicy.enabled, add an egress.extra rule that allows HTTPS (443) to the Bedrock runtime and STS endpoints, or through the proxy or VPC endpoints your cluster uses for AWS APIs. Switching an embedding provider changes the vector space, so plan a re-embed before trusting mixed results.

Production Example

Complete production-ready values:

global:
  imagePullSecrets:
    - name: ghcr-pull-secret

coordinationBackend: "redis"
publicUrl: "https://sibyl.example.com"

backend:
  replicaCount: 3
  image:
    repository: ghcr.io/hyperb1iss/sibyl-api
    tag: "1.4.10"
    pullPolicy: Always
  existingSecret: sibyl-secrets
  # Only the ingress controller pods' own range, so each user gets their own login
  # rate-limit bucket. Never the pod CIDR (see Trusted Proxies).
  forwardedAllowIps: "10.250.0.0/28"
  validationReceipts:
    existingClaim: sibyl-validation-receipts
    # Or keep receipts in S3 and leave existingClaim empty:
    # url: "s3://sibyl-receipts/prod?region=us-west-2"
  # Archives default to the receipts claim; with receipts in S3, name a
  # separate prefix (or a claim) for them:
  # backupArchives:
  #   url: "s3://sibyl-backups/prod?region=us-west-2"
  surreal:
    url: "ws://prod-surrealdb.internal:8000/rpc"
    username: "root"
    existingSecret: sibyl-surreal
    namespacePrefix: "org_"
    database: "graph"
  redis:
    host: "prod-valkey.internal"
    port: "6379"
    existingSecret: sibyl-redis
  env:
    SIBYL_ENVIRONMENT: "production"
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 20
  pdb:
    enabled: true
    minAvailable: 2
  podAntiAffinity:
    enabled: true
    type: hard
  resources:
    limits:
      cpu: 4000m
      memory: 4Gi
    requests:
      cpu: 1000m
      memory: 1Gi

frontend:
  enabled: true
  replicaCount: 2
  apiUrl: "http://sibyl-backend:3334/api"
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
  pdb:
    enabled: true
  podAntiAffinity:
    enabled: true

worker:
  enabled: true
  replicaCount: 2
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 8
  pdb:
    enabled: true
  podAntiAffinity:
    enabled: true

ingress:
  hosts:
    - host: sibyl.example.com
      paths:
        - path: /api
          pathType: Prefix
          service: backend
        - path: /mcp
          pathType: Prefix
          service: backend
        - path: /
          pathType: Prefix
          service: frontend
  classic:
    enabled: true
    className: "nginx"
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
      nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    tls:
      - secretName: sibyl-tls
        hosts:
          - sibyl.example.com

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/sibyl

Chart Templates

The chart includes these templates:

Template	Purpose
backend-deployment.yaml	Backend Deployment
backend-service.yaml	Backend ClusterIP Service
backend-hpa.yaml	Backend HorizontalPodAutoscaler
frontend-deployment.yaml	Frontend Deployment
frontend-service.yaml	Frontend ClusterIP Service
frontend-hpa.yaml	Frontend HorizontalPodAutoscaler
worker-deployment.yaml	Worker Deployment
worker-hpa.yaml	Worker HorizontalPodAutoscaler
pdb.yaml	PodDisruptionBudgets
configmap.yaml	Non-secret environment config
surreal-secret.yaml	Auto-generated Surreal secret (default)
redis-secret.yaml	Auto-generated Redis/Valkey secret
bootstrap-job.yaml	Post-install tenant bootstrap Job
ingress.yaml	Classic networking.k8s.io/v1 Ingress
httproute.yaml	Gateway API HTTPRoute
networkpolicy.yaml	Default-deny plus app-allow NetworkPolicies
podsecurity.yaml	Namespace Pod Security enforcement labels
serviceaccount.yaml	ServiceAccount

Debugging

# Render templates locally
helm template sibyl ./charts/sibyl -f values.yaml

# Debug with notes
helm install sibyl ./charts/sibyl -f values.yaml --debug --dry-run

# Get release values
helm get values sibyl -n sibyl

# Get all manifests
helm get manifest sibyl -n sibyl

Durable Validation Receipts

Validation writes each completed model result as an encrypted receipt before it commits the result to the database, so a database outage does not force Sibyl to pay for the same validation twice. Set exactly one of backend.validationReceipts.url (Amazon S3) or backend.validationReceipts.existingClaim (a shared volume) before installing or upgrading. The chart refuses to render with neither or both.

Amazon S3

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/sibyl

backend:
  validationReceipts:
    url: "s3://<bucket>/sibyl/receipts?region=us-west-2"

The URL names a bucket, an optional key prefix, and an optional region query parameter (auto is accepted for endpoints such as Cloudflare R2). The chart checks the URL's shape at render time (a 3-63 character bucket of lowercase letters, digits, dots and hyphens, no empty or dot segments in the prefix, and no query other than region), so a typo fails the install rather than crashlooping the pods. Without a region, the standard AWS_REGION and AWS_DEFAULT_REGION resolution applies (the EKS IRSA webhook sets AWS_REGION). Credentials come from the default AWS chain: IRSA, EKS Pod Identity, or an instance role, as described under Amazon Bedrock. The chart passes the URL to API and worker as SIBYL_VALIDATION_RECEIPT_URL through its ConfigMap and mounts no receipts volume, so pods may run on any node and single-replica rollouts use the Kubernetes RollingUpdate default. The frontend runs under its own service account (see Service Account), so the receipts role never reaches it. Give receipts a dedicated prefix, since backup export lists it.

Grant the pod role access to the prefix:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<bucket>/sibyl/receipts/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::<bucket>"
    }
  ]
}

s3:ListBucket makes a missing receipt answer 404 instead of 403. Sibyl must tell "absent" from "denied", and refuses to dispatch without it. Backup export also uses it to list the prefix once instead of reading every execution's receipt. If the bucket encrypts with a customer managed KMS key, also grant kms:GenerateDataKey and kms:Decrypt on that key. Sibyl sends no x-amz-server-side-encryption header, so a bucket policy that denies uploads without one refuses every receipt and the readiness check stops validation; use the bucket's default encryption instead.

The volume guarantees carry over:

• Encrypted by Sibyl. Each receipt is Fernet ciphertext under a per-execution key held in the content database, so the bucket never sees a result in the clear. Bucket encryption (SSE-S3 or SSE-KMS) is welcome but not relied on. Purging a source erases its key, which leaves any copy of the ciphertext unreadable.
• Create-only. Every write is a PutObject with If-None-Match: *. A 412 Precondition Failed answer means the receipt already exists: Sibyl compares the stored bytes and refuses any difference, so neither validation nor backup restore ever replaces a receipt. A 409 from a racing delete is retried, as S3 documents.
• Checked when each validation execution begins. The readiness check runs once per execution, before its first model call, not before every call. It deletes a never-written key (proving s3:DeleteObject without creating anything), confirms a second never-written key reads as absent, then creates a probe object under the prefix, confirms a second create is refused, reads it back, finds it in a listing, deletes it, and confirms it reads as absent. The listing check matters because backup export finds receipts by listing the prefix, so a store whose listings lag its writes would record a pending receipt as missing. A bucket that fails any step stops the execution before a provider is paid, including an S3-compatible store that ignores If-None-Match.
• Unsigned reads refused, best effort. The check also asks for the probe without credentials and stops validation if that read succeeds. A timeout, redirect, or any other non-success answer counts as refused, so this catches a public bucket but is not a full audit of bucket policy.

Each request has a 3 s connect and 10 s read timeout with up to three attempts, so an endpoint that accepts connections and never answers fails the check in about half a minute instead of holding a worker thread for minutes.

Keep S3 Block Public Access on (the default for new buckets), and do not grant Principal: "*" read access in the bucket policy, even behind a VPC endpoint condition. Versioning is not required. With networkPolicy.enabled, allow HTTPS egress to S3 and to STS (IRSA exchanges its token there), or to their VPC endpoints, through networkPolicy.egress.extra. For an S3-compatible endpoint, set backend.env.AWS_ENDPOINT_URL_S3; the readiness check verifies the store honors If-None-Match.

A successful readiness check leaves no object behind. A probe can still be stranded when a pod dies mid-check or the endpoint stops answering, and on a versioned bucket every check leaves a noncurrent probe version and delete markers, as discarded receipts do. Add a lifecycle rule for the prefix:

{
  "Rules": [
    {
      "ID": "sibyl-receipt-history",
      "Status": "Enabled",
      "Filter": { "Prefix": "sibyl/receipts/" },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 1 },
      "Expiration": { "ExpiredObjectDeleteMarker": true }
    },
    {
      "ID": "sibyl-receipt-probes",
      "Status": "Enabled",
      "Filter": { "Prefix": "sibyl/receipts/.probe-" },
      "Expiration": { "Days": 1 }
    }
  ]
}

A noncurrent receipt version is never needed: a receipt is only removed after the database holds its result.

To move an existing release from a claim to S3, stop validation work (scale the worker and backend to zero), then copy any remaining receipts from the claim's private directory into the prefix under the same names. Run the copy from a temporary pod that mounts the claim at /var/lib/sibyl-receipts and has the pod role's credentials and an AWS CLI whose s3api put-object accepts --if-none-match. Use that create-only upload so a receipt already in the bucket is never replaced:

for receipt in /var/lib/sibyl-receipts/private/*.receipt; do
  aws s3api put-object --bucket <bucket> --key "sibyl/receipts/$(basename "$receipt")" \
    --body "$receipt" --if-none-match '*'
done

A PreconditionFailed answer means that receipt is already there. Then upgrade with url set and existingClaim cleared. A content backup taken before the switch also carries pending receipts, and restoring it publishes them into the bucket.

Shared Volume

Provision a persistent claim and set backend.validationReceipts.existingClaim. API and worker mount the same claim. Multi-node replicas require a ReadWriteMany volume; provision it with the storage class supported by your cluster. The service user (UID/GID 10001) must be able to create a private child directory. No temporary volume or single-replica fallback is used.

Two chart defaults keep a ReadWriteOnce block volume (EBS, Persistent Disk, Azure Disk) working across restarts and rollouts:

• fsGroupChangePolicy: OnRootMismatch on the backend and worker pod security contexts. Block CSI drivers re-apply fsGroup on every mount by default, adding group permission bits to every file and directory, and Sibyl refuses a receipts directory or file with any group or other bits. Keep this key if you override podSecurityContext.
• An empty backend.strategy or worker.strategy rolls a single fixed replica with maxSurge: 0 and maxUnavailable: 1, because a surge pod scheduled onto another node cannot attach a ReadWriteOnce claim. The old pod stops before its replacement starts; if the replacement lands on another node, it waits for the volume to detach and then starts on its own. With more replicas or autoscaling the Kubernetes RollingUpdate default applies, which assumes ReadWriteMany storage. An explicit strategy passes through unchanged.

The single-replica default stays type: RollingUpdate rather than Recreate on purpose. The API server fills in rollingUpdate on every existing Deployment, no applier owns that field, and server-side apply (the Helm 4 default, Argo CD with ServerSideApply=true, Flux) can never remove it, so an upgrade that switches type to Recreate is rejected with spec.strategy.rollingUpdate: Forbidden. To opt into Recreate on an existing release, remove the field once first:

kubectl -n sibyl patch deploy sibyl-backend --type=json \
  -p '[{"op":"remove","path":"/spec/strategy/rollingUpdate"},{"op":"replace","path":"/spec/strategy/type","value":"Recreate"}]'

Downgrades

Content schema 40 retains a per-execution receipt key and erases it when a source is purged. Older servers cannot recover these pending receipts, and servers older than the S3 store cannot read a bucket at all. Schema repair must not replace the key-erasing purge event with the historical definition. Recover pending validation receipts with the current server before a planned downgrade, retaining the receipt volume or bucket and a coherent database backup. Removing the receipt store loses outage recovery for pending results; replay still refuses to repeat an unresolved provider dispatch.

Backup Archives

Organization backups (web settings, /api/backups, and the scheduled per-organization backups) are written as one .tar.gz archive per backup by whichever process runs the backup job: the worker under Redis coordination, or the API pod that took the job under local coordination. Any API replica then lists, downloads and deletes it, so every backend and worker pod must open the same archive store. The image's working directory is read-only under the chart and only per-pod storage could be written there, so the chart never leaves archives in the container.

Where archives live:

Values	Archive store
neither set (default)	/var/lib/sibyl-receipts/backups on the receipts claim
backend.backupArchives.existingClaim	/var/lib/sibyl-backups/archives on that claim
backend.backupArchives.url	the S3 prefix, through SIBYL_BACKUP_ARCHIVE_URL

Set at most one of url and existingClaim. With receipts in S3 there is no claim to share, so the chart refuses to render until one of them is set. The chart sets SIBYL_BACKUP_DIR or SIBYL_BACKUP_ARCHIVE_URL on API and worker alike, and refuses either variable in backend.env.

A claim is a shared directory only when every pod mounts it: a ReadWriteOnce claim works while API and worker run on one node (the single-replica default, which rolls with maxSurge: 0 for an archives claim exactly as for the receipts claim), and multi-node replicas need ReadWriteMany. Each archive is assembled in a hidden staging directory on the same volume and published with a hard link, so a listing never shows a half-written archive and a published archive is never replaced. The volume's filesystem must support hard links; each backup proves it with a probe before it exports anything, and fails with that reason on a filesystem that cannot. A new archive directory is created private (0700) and each archive 0600. The scheduled cleanup removes staging directories left by a killed job once they are a day old, since they hold the export in plaintext.

Sizing a claim

The receipts claim default suits a single replica and a small install: it keeps archives durable with no extra values, but they then share one volume with validation receipts. Receipts are small, but a receipt that cannot be written stops validation, so a full claim is an outage. At scale, give archives a dedicated claim (backend.backupArchives.existingClaim) or S3.

Size an archives volume, or the receipts claim when it holds archives, for:

• the archives kept: organizations backed up × backups per day × SIBYL_BACKUP_RETENTION_DAYS × archive size;
• staging for the largest running backup: its export is written uncompressed on the same volume before it is compressed, typically five to ten times the archive's size, plus the archive itself;
• a reserve for everything else on the volume, SIBYL_BACKUP_FREE_SPACE_RESERVE_MB (default 512; set it with backend.env).

Before each backup on a directory store, Sibyl claims room for it: eleven times that organization's last archive, checked against the volume's free space less the reserve and less what other running backups have claimed and not yet written. Before writing the archive it claims again, at the staged export's real size. A backup that would eat into the reserve fails with the free space, its estimate, the other backups' claims and the reserve in its error, before it writes anything more, so the reserve stays free for receipts.

Each claim is a hidden .reserve-<backup id> file beside the archives, so backups on different pods sharing the claim see each other; nothing is serialized. A backup writes its claim before it checks, so two backups starting at the same moment each count the other: neither can overcommit the volume, and at worst both are refused and the next scheduled run goes ahead. A claim untouched for a day belongs to a backup that died; it stops counting and the scheduled cleanup removes it. A claim that fails its check, for a full volume or an I/O error alike, is removed at once. Anything named like a claim that is not a small regular file (a link, a FIFO, an oversized file) is ignored and logged once rather than read.

That promise holds only where every pod sees a new file in the directory as soon as it is written. NFS-backed shared volumes, such as Amazon EFS, Google Filestore and most ReadWriteMany NFS provisioners, cache directory listings on each node for about 30 to 60 seconds. Backups on different nodes that start within that window can miss each other's claims. At worst they behave as if there were no claims: each still checks its own estimate, and its real staged size before writing the archive, but not the others'. On such volumes, keep headroom for concurrent backups, or use S3 for archives.

The estimate has two known gaps:

• An organization's first backup has no earlier archive to estimate from, so before its export only the reserve and other backups' claims are checked. Its staged size is checked before the archive is written, but the export itself can use space the reserve was meant to keep. Leave headroom beyond the reserve for the largest organization's first backup, or take it while the volume is roomy.
• The factor of eleven assumes the export is five to ten times its gzip archive, typical for JSON. For a large organization the claim is then large too, and an export that compresses less than that is over-estimated: it can be refused on a volume that would have held it. Free more space or lower the reserve rather than waiting for it to fit. An export that compresses better is under-estimated before it runs, and the second check, at its real staged size, still guards the archive.

Retried jobs

Under Redis coordination a backup job that was cancelled, or whose worker died, runs again. A backup already recorded as completed is reported as such and not rebuilt. When an earlier attempt published the archive but died before recording it, the retry adopts that archive instead of exporting again, once it has matched the digest recorded at publish time (S3), passed the same inventory, checksum and organization checks a restore applies, and so proved to belong to this organization. Any other archive under the backup's name fails the backup and is left untouched.

Archives in Amazon S3

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/sibyl

backend:
  validationReceipts:
    url: "s3://<bucket>/sibyl/receipts?region=us-west-2"
  backupArchives:
    url: "s3://<bucket>/sibyl/backups?region=us-west-2"

The URL has the same shape and render-time checks as backend.validationReceipts.url, and uses the same client: credentials from the pod's AWS identity, AWS_REGION when the URL names no region, backend.env.AWS_ENDPOINT_URL_S3 for an S3-compatible endpoint, and the same timeouts and retries. Archives and receipts may share a bucket but not a prefix, and neither prefix may contain the other; the chart and the server both refuse an overlap, because receipt export and archive retention each list their own prefix.

Grant the pod role access to the archives prefix, alongside the receipts statement:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::<bucket>/sibyl/backups/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::<bucket>"
    }
  ]
}

s3:ListBucket serves retention, which lists the prefix to find archives past SIBYL_BACKUP_RETENTION_DAYS, and lets a missing archive answer 404 instead of 403; each backup also asks whether an earlier attempt already published its archive, which needs that answer. s3:AbortMultipartUpload lets a failed upload clean up after itself. Downloads are streamed from the bucket through the API in 1 MiB chunks, so no replica holds a whole archive in memory, and the browser never needs bucket access. A single Range: bytes= request gets 206 Partial Content for that part, from either store, so an interrupted download resumes.

How archives keep their guarantees in S3:

• Written once. An archive up to 16 MiB is one PutObject, and a larger one a multipart upload in 16 MiB parts; both carry If-None-Match: *. Each object records its SHA-256 in user metadata, so a 412 Precondition Failed is resolved by comparing digests: the same digest is a retried write that already landed, a different one fails the backup and leaves the existing archive in place. A 409 from a racing request retries the whole upload, as S3 documents.
• Never half visible. S3 lists and serves a multipart upload only after CompleteMultipartUpload succeeds. A failed upload is aborted while the endpoint still answers.
• Checked as each backup begins. Before any organization data is exported, the job creates a probe object under the prefix, confirms a second create is refused, confirms an unsigned read of it fails, and deletes it. An S3-compatible store that ignores If-None-Match, or a bucket that serves that probe to anonymous requests, fails the backup with that reason. The unsigned read is best effort, as for receipts: it asks only for the probe key, and a timeout or any other non-success answer counts as refused, so it catches a public bucket but is not an audit of the bucket policy.
• Staged on the pod. An S3 backup assembles its export and archive in the pod's /tmp, the chart's tmp emptyDir, before uploading, which needs about eleven times the largest archive. Set ephemeral-storage requests and limits on the worker (and on the backend under local coordination) to match.

Archives hold secrets in the clear. Unlike receipts, an archive is not encrypted by Sibyl: it carries the organization's memory, and its content payload carries the recovery keys that decrypt its validation receipts. Keep the bucket's default encryption on (SSE-S3, or SSE-KMS with kms:GenerateDataKey and kms:Decrypt granted on the key; Sibyl sends no x-amz-server-side-encryption header, so rely on bucket default encryption rather than a policy that denies uploads without one), keep S3 Block Public Access on, grant read access to the prefix only to the Sibyl role and the people who restore backups, and consider a separate bucket or KMS key from receipts so read access to one does not imply the other. Sibyl logs archive names, sizes and locations, never their contents.

Add a lifecycle rule for the prefix so an upload stranded by a dead pod or endpoint, or a readiness probe, does not linger:

{
  "Rules": [
    {
      "ID": "sibyl-backup-uploads",
      "Status": "Enabled",
      "Filter": { "Prefix": "sibyl/backups/" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 1 },
      "Expiration": { "ExpiredObjectDeleteMarker": true }
    },
    {
      "ID": "sibyl-backup-probes",
      "Status": "Enabled",
      "Filter": { "Prefix": "sibyl/backups/.probe-" },
      "Expiration": { "Days": 1 }
    }
  ]
}

Retention stays Sibyl's: the scheduled cleanup removes archives older than SIBYL_BACKUP_RETENTION_DAYS along with their backup records, and an organization admin's cleanup removes only that organization's expired archives. Do not add an expiration rule for archives themselves, or the web settings will list backups whose archive is gone.

To move existing archives from a claim into the bucket, copy them under the same names with a create-only upload from a pod that mounts the claim (archives live in /var/lib/sibyl-receipts/backups on the default layout), then upgrade with backupArchives.url set:

for archive in /var/lib/sibyl-receipts/backups/sibyl_*.tar.gz; do
  aws s3api put-object --bucket <bucket> --key "sibyl/backups/$(basename "$archive")" \
    --body "$archive" --if-none-match '*' \
    --metadata "sha256=$(sha256sum "$archive" | cut -d' ' -f1)"
done

Upgrading

• Receipts already on a claim. Nothing to change: archives start landing in backups beside the receipts on that claim. Backups could not be written under earlier charts, so expect the claim to start filling once they succeed, and size it as above or move archives off it.
• Receipts in S3. helm upgrade refuses to render until backend.backupArchives.url or backend.backupArchives.existingClaim is set.
• Moving receipts to S3 and keeping their old claim for archives. Archives live in backups on the receipts claim but in archives on an archives claim. When backupArchives.existingClaim takes over the old receipts claim, move /var/lib/sibyl-backups/backups/* to /var/lib/sibyl-backups/archives/ from a pod that mounts it, or existing backups stay listed but answer 404 on download.
• Moving archives to S3. Use the create-only copy above, then set backupArchives.url.