Production Kubernetes Deployment

Deploy Sibyl to a production Kubernetes cluster using Helm.

Prerequisites

Kubernetes cluster (1.27+)
kubectl configured for your cluster
Helm 3.x
SurrealDB instance (in-cluster StatefulSet, Surreal Cloud, or another external host)

Architecture Overview

+---------------------------- Production Cluster -----------------------------+
|                                                                              |
|  +------------------+     +------------------+     +------------------+      |
|  |  Ingress/Gateway |     |    Backend (n)   |     |   Frontend (n)   |      |
|  |  (Kong/Nginx)    |---->|    HPA: 2-10     |     |   HPA: 2-10      |      |
|  +------------------+     +--------+---------+     +------------------+      |
|                                    |                                         |
|                          +---------+--------+                                |
|                          |    Worker (n)    |                                |
|                          |    HPA: 1-5      |                                |
|                          +------------------+                                |
|                                                                              |
+------------------------------------------------------------------------------+
                                    |
                         +----------+-----------+
                         |      SurrealDB       |
                         | graph + content +    |
                         | auth (unified)       |
                         +----------------------+

SurrealDB is the active runtime. See storage-modes.md for local archive rehearsal notes.

Quick Start

bash

# Add namespace
kubectl create namespace sibyl

# Create secrets (Surreal default)
kubectl create secret generic sibyl-secrets -n sibyl \
  --from-literal=SIBYL_JWT_SECRET=$(openssl rand -hex 32) \
  --from-literal=SIBYL_OPENAI_API_KEY=sk-... \
  --from-literal=SIBYL_ANTHROPIC_API_KEY=sk-ant-...

kubectl create secret generic sibyl-surreal -n sibyl \
  --from-literal=password=<your-surreal-password>

# Install with Helm
helm upgrade --install sibyl ./charts/sibyl \
  -n sibyl \
  -f values-production.yaml

The Helm chart deploys the SurrealDB-backed runtime only. Keep PostgreSQL archive rehearsal sidecars and preserved FalkorDB source deployments outside this release chart.

Values Configuration

Create a values-production.yaml (Surreal default):

yaml

# Coordination mode
coordinationBackend: "redis"

backend:
  replicaCount: 2

  image:
    repository: ghcr.io/hyperb1iss/sibyl-api
    tag: "1.0.0-rc.1"
    pullPolicy: Always

  # Reference pre-created secrets
  existingSecret: sibyl-secrets

  # SurrealDB connection
  surreal:
    url: "ws://your-surrealdb.example.com:8000/rpc"
    username: "root"
    existingSecret: sibyl-surreal
    namespacePrefix: "org_"
    database: "graph"

  # Valkey/Redis coordination for multi-replica deployments
  redis:
    host: "valkey.example.svc.cluster.local"
    port: "6379"
    jobsDb: "1"
    rateLimitDb: "4"
    existingSecret: sibyl-redis
    secretKey: password

  env:
    SIBYL_SERVER_HOST: "0.0.0.0"
    SIBYL_SERVER_PORT: "3334"
    SIBYL_ENVIRONMENT: "production"
    SIBYL_PUBLIC_URL: "https://sibyl.example.com"
    SIBYL_LLM_PROVIDER: "anthropic"
    SIBYL_LLM_MODEL: "claude-haiku-4-5"

  # Enable autoscaling
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 80

  # Enable PodDisruptionBudget
  pdb:
    enabled: true
    minAvailable: 1

  # Spread pods across nodes
  podAntiAffinity:
    enabled: true
    type: soft
    topologyKey: kubernetes.io/hostname

  resources:
    limits:
      cpu: 2000m
      memory: 2Gi
    requests:
      cpu: 500m
      memory: 512Mi

frontend:
  enabled: true
  replicaCount: 2

  image:
    repository: ghcr.io/hyperb1iss/sibyl-web
    tag: "1.0.0-rc.1"

  apiUrl: "http://sibyl-backend:3334/api"

  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10

  pdb:
    enabled: true
    minAvailable: 1

worker:
  enabled: true
  replicaCount: 2

  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 5

  pdb:
    enabled: true
    minAvailable: 1

# Ingress (adjust for your ingress controller)
ingress:
  enabled: true
  className: "kong"
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
  hosts:
    - host: sibyl.example.com
      paths:
        - path: /api
          pathType: Prefix
          service: backend
        - path: /mcp
          pathType: Prefix
          service: backend
        - path: /
          pathType: Prefix
          service: frontend
  tls:
    - secretName: sibyl-tls
      hosts:
        - sibyl.example.com

Database Setup

SurrealDB Requirements (default)

SurrealDB 3.x. Pin an explicit tested image tag for production instead of floating on latest.
TiKV-backed storage for replicated SurrealDB pods, or RocksDB-backed storage for a single pod
Root credentials set at start (--user / --pass)
Persistence and backups live in the selected SurrealDB storage engine

For a scalable in-cluster shape, deploy the TiDB Operator, create a small TiKV/PD cluster, and point the official SurrealDB Helm chart at tikv://<pd-service>:2379. For single-pod installs, a RocksDB-backed PVC is simpler.

Coordination Requirements

Set coordinationBackend: "redis" when running multiple backend or worker replicas. Use Valkey or Redis for arq jobs, distributed locks, WebSocket pub/sub, and shared rate limits. The local Tilt demo uses the official valkey/valkey Helm chart.

Archive Rehearsal Sidecars

PostgreSQL is no longer part of the active Kubernetes runtime. Keep PostgreSQL and preserved FalkorDB source deployments outside the release chart. Bring them up only for explicit legacy postgres.sql archive rehearsal or rollback validation during a write-freeze window.

Secrets Management

Option 1: Kubernetes Secrets

bash

# Create from literal values
kubectl create secret generic sibyl-secrets -n sibyl \
  --from-literal=SIBYL_JWT_SECRET=$(openssl rand -hex 32) \
  --from-literal=SIBYL_OPENAI_API_KEY=sk-... \
  --from-literal=SIBYL_ANTHROPIC_API_KEY=sk-ant-...

Option 2: External Secrets Operator

yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: sibyl-secrets
  namespace: sibyl
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: sibyl-secrets
  data:
    - secretKey: SIBYL_JWT_SECRET
      remoteRef:
        key: sibyl/jwt-secret
    - secretKey: SIBYL_OPENAI_API_KEY
      remoteRef:
        key: sibyl/openai-key

Option 3: Sealed Secrets

bash

# Create SealedSecret
kubeseal --format=yaml < sibyl-secrets.yaml > sibyl-secrets-sealed.yaml
kubectl apply -f sibyl-secrets-sealed.yaml

Schema Bootstrap

Sibyl bootstraps SurrealDB schema inline at startup. The Helm chart no longer runs an Alembic pre-upgrade hook for the active runtime.

Ingress Configuration

The chart can render a standard Kubernetes Ingress from values.yaml. For Gateway API deployments, leave ingress.enabled=false and apply an HTTPRoute like the Kong example below.

Kong Gateway

yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: sibyl
  namespace: sibyl
spec:
  parentRefs:
    - name: production-gateway
      namespace: kong
  hostnames:
    - sibyl.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
        - path:
            type: PathPrefix
            value: /mcp
      backendRefs:
        - name: sibyl-backend
          port: 3334
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: sibyl-frontend
          port: 3337

NGINX Ingress

yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sibyl
  namespace: sibyl
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - sibyl.example.com
      secretName: sibyl-tls
  rules:
    - host: sibyl.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: sibyl-backend
                port:
                  number: 3334
          - path: /mcp
            pathType: Prefix
            backend:
              service:
                name: sibyl-backend
                port:
                  number: 3334
          - path: /
            pathType: Prefix
            backend:
              service:
                name: sibyl-frontend
                port:
                  number: 3337

Health Checks

The chart configures liveness and readiness probes:

yaml

backend:
  livenessProbe:
    httpGet:
      path: /api/health
      port: http
    initialDelaySeconds: 10
    periodSeconds: 30

  readinessProbe:
    httpGet:
      path: /api/health
      port: http
    initialDelaySeconds: 5
    periodSeconds: 10

Scaling

Manual Scaling

bash

kubectl scale deployment sibyl-backend -n sibyl --replicas=5

HPA Configuration

With autoscaling enabled:

yaml

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300 # Wait 5min before scaling down
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15

Monitoring

Check Deployment Status

bash

# All resources
kubectl get all -n sibyl

# Pods status
kubectl get pods -n sibyl -o wide

# HPA status
kubectl get hpa -n sibyl

# Events
kubectl get events -n sibyl --sort-by='.lastTimestamp'

View Logs

bash

# Backend logs
kubectl logs -n sibyl -l app.kubernetes.io/component=backend -f

# Worker logs
kubectl logs -n sibyl -l app.kubernetes.io/component=worker -f

# All Sibyl logs
kubectl logs -n sibyl -l app.kubernetes.io/name=sibyl -f

Upgrades

bash

# Update values
helm upgrade sibyl ./charts/sibyl \
  -n sibyl \
  -f values-production.yaml \
  --set backend.image.tag=0.2.0

# Rollback if needed
helm rollback sibyl -n sibyl

Uninstall

bash

# Remove Helm release
helm uninstall sibyl -n sibyl

# Remove namespace (DELETES ALL DATA)
kubectl delete namespace sibyl

Next Steps

Helm Chart Reference - Complete values documentation
Environment Variables - All configuration options
Monitoring - Observability setup

Production Kubernetes Deployment ​

Prerequisites ​

Architecture Overview ​

Quick Start ​

Values Configuration ​

Database Setup ​

SurrealDB Requirements (default) ​

Coordination Requirements ​

Archive Rehearsal Sidecars ​

Secrets Management ​

Option 1: Kubernetes Secrets ​

Option 2: External Secrets Operator ​

Option 3: Sealed Secrets ​

Schema Bootstrap ​

Ingress Configuration ​

Kong Gateway ​

NGINX Ingress ​

Health Checks ​

Scaling ​

Manual Scaling ​

HPA Configuration ​

Monitoring ​

Check Deployment Status ​

View Logs ​

Upgrades ​

Uninstall ​

Next Steps ​

Production Kubernetes Deployment

Prerequisites

Architecture Overview

Quick Start

Values Configuration

Database Setup

SurrealDB Requirements (default)

Coordination Requirements

Archive Rehearsal Sidecars

Secrets Management

Option 1: Kubernetes Secrets

Option 2: External Secrets Operator

Option 3: Sealed Secrets

Schema Bootstrap

Ingress Configuration

Kong Gateway

NGINX Ingress

Health Checks

Scaling

Manual Scaling

HPA Configuration

Monitoring

Check Deployment Status

View Logs

Upgrades

Uninstall

Next Steps