Production Kubernetes Deployment

Deploy Sibyl to a production Kubernetes cluster using Helm.

Prerequisites

Kubernetes cluster (1.27+)
kubectl configured for your cluster
Helm 3.x
SurrealDB instance (in-cluster StatefulSet, Surreal Cloud, or another external host)

Architecture Overview

+---------------------------- Production Cluster -----------------------------+
|                                                                              |
|  +------------------+     +------------------+     +------------------+      |
|  |  Ingress/Gateway |     |    Backend (n)   |     |   Frontend (n)   |      |
|  |  (Kong/Nginx)    |---->|    HPA: 2-10     |     |   HPA: 2-10      |      |
|  +------------------+     +--------+---------+     +------------------+      |
|                                    |                                         |
|                          +---------+--------+                                |
|                          |    Worker (n)    |                                |
|                          |    HPA: 1-5      |                                |
|                          +------------------+                                |
|                                                                              |
+------------------------------------------------------------------------------+
                                    |
                         +----------+-----------+
                         |      SurrealDB       |
                         | graph + content +    |
                         | auth (unified)       |
                         +----------------------+

SurrealDB is the only data store. See Storage Modes for connection options.

Quick Start

Provision the sibyl-validation-receipts persistent claim before installation. API and worker share it, along with the organization backup archives kept in its backups directory; multi-node replicas require ReadWriteMany storage. With receipts in S3, give archives a bucket prefix (backend.backupArchives.url) or a claim of their own; see Backup Archives.

bash

# Add namespace
kubectl create namespace sibyl

# Create secrets (Surreal default)
kubectl create secret generic sibyl-secrets -n sibyl \
  --from-literal=SIBYL_JWT_SECRET=$(openssl rand -hex 32) \
  --from-literal=SIBYL_SETTINGS_KEY=$(openssl rand -base64 32 | tr '+/' '-_') \
  --from-literal=SIBYL_OPENAI_API_KEY=sk-... \
  --from-literal=SIBYL_ANTHROPIC_API_KEY=sk-ant-...

kubectl create secret generic sibyl-surreal -n sibyl \
  --from-literal=password=<your-surreal-password>

# Install with Helm from the published repository
helm repo add sibyl https://raw.githubusercontent.com/hyperb1iss/sibyl/gh-pages
helm repo update

helm upgrade --install sibyl sibyl/sibyl \
  -n sibyl \
  -f values-production.yaml

Installing from a source checkout works the same way with the local chart path: replace sibyl/sibyl with ./charts/sibyl.

Values Configuration

Create a values-production.yaml (Surreal default):

yaml

# Coordination mode
coordinationBackend: "redis"

backend:
  replicaCount: 2

  image:
    repository: ghcr.io/hyperb1iss/sibyl-api
    tag: "1.4.10"
    pullPolicy: Always

  # Reference pre-created secrets
  existingSecret: sibyl-secrets
  validationReceipts:
    existingClaim: sibyl-validation-receipts
    # Or keep receipts in S3 and leave existingClaim empty:
    # url: "s3://sibyl-receipts/prod?region=us-west-2"
  # Archives default to the receipts claim; with receipts in S3:
  # backupArchives:
  #   url: "s3://sibyl-backups/prod?region=us-west-2"
  # Only the ingress controller pods' own range, so logins key on each user's
  # address. Never the pod CIDR; see "Client Addresses Behind the Ingress".
  forwardedAllowIps: "10.250.0.0/28"

  # SurrealDB connection
  surreal:
    url: "ws://your-surrealdb.example.com:8000/rpc"
    username: "root"
    existingSecret: sibyl-surreal
    namespacePrefix: "org_"
    database: "graph"

  # Valkey/Redis coordination for multi-replica deployments
  redis:
    host: "valkey.example.svc.cluster.local"
    port: "6379"
    jobsDb: "1"
    rateLimitDb: "4"
    existingSecret: sibyl-redis
    secretKey: password

  env:
    SIBYL_SERVER_HOST: "0.0.0.0"
    SIBYL_SERVER_PORT: "3334"
    SIBYL_ENVIRONMENT: "production"
    SIBYL_PUBLIC_URL: "https://sibyl.example.com"
    SIBYL_LLM_PROVIDER: "anthropic"
    SIBYL_LLM_MODEL: "claude-haiku-5-5"

  # Enable autoscaling
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 80

  # Enable PodDisruptionBudget
  pdb:
    enabled: true
    minAvailable: 1

  # Spread pods across nodes
  podAntiAffinity:
    enabled: true
    type: soft
    topologyKey: kubernetes.io/hostname

  resources:
    limits:
      cpu: 2000m
      memory: 2Gi
    requests:
      cpu: 500m
      memory: 512Mi

frontend:
  enabled: true
  replicaCount: 2

  image:
    repository: ghcr.io/hyperb1iss/sibyl-web
    tag: "1.4.10"

  apiUrl: "http://sibyl-backend:3334/api"

  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10

  pdb:
    enabled: true
    minAvailable: 1

worker:
  enabled: true
  replicaCount: 2

  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 5

  pdb:
    enabled: true
    minAvailable: 1

# Ingress (adjust for your ingress controller)
# ingress.hosts is the shared route table. Enable ingress.classic for a classic
# networking.k8s.io/v1 Ingress, or ingress.gatewayApi for a Gateway API HTTPRoute.
ingress:
  hosts:
    - host: sibyl.example.com
      paths:
        - path: /api
          pathType: Prefix
          service: backend
        - path: /mcp
          pathType: Prefix
          service: backend
        - path: /
          pathType: Prefix
          service: frontend
  # Classic Ingress (NGINX and similar):
  classic:
    enabled: true
    className: "nginx"
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    tls:
      - secretName: sibyl-tls
        hosts:
          - sibyl.example.com
  # Gateway API HTTPRoute (Kong and similar): enable this instead of classic.
  gatewayApi:
    enabled: false
    parentRefs:
      - name: production-gateway
        namespace: kong

Database Setup

SurrealDB Requirements (default)

SurrealDB 3.x. Pin an explicit tested image tag for production instead of floating on latest.
TiKV-backed storage for replicated SurrealDB pods, or RocksDB-backed storage for a single pod
Root credentials set at start (--user / --pass)
Persistence and backups live in the selected SurrealDB storage engine

For a scalable in-cluster shape, deploy the TiDB Operator, create a small TiKV/PD cluster, and point the official SurrealDB Helm chart at tikv://<pd-service>:2379. For single-pod installs, a RocksDB-backed PVC is simpler. Either way, tikv:// and rocksdb:// configure the SurrealDB server only. Point Sibyl's backend.surreal.url at that server with a ws://, wss://, http://, or https:// URL. The backend rejects unsupported schemes such as rocksdb:// and tikv:// at startup; the embedded forms have their own rules and are not meant for a cluster (see SurrealDB URL forms).

Coordination Requirements

Set coordinationBackend: "redis" when running multiple backend or worker replicas. Use Valkey or Redis for arq jobs, distributed locks, WebSocket pub/sub, and shared rate limits.

Secrets Management

Option 1: Kubernetes Secrets

bash

# Create from literal values
kubectl create secret generic sibyl-secrets -n sibyl \
  --from-literal=SIBYL_JWT_SECRET=$(openssl rand -hex 32) \
  --from-literal=SIBYL_OPENAI_API_KEY=sk-... \
  --from-literal=SIBYL_ANTHROPIC_API_KEY=sk-ant-...

On AWS with the bedrock provider, leave the two provider keys out. Claude and Cohere embeddings then authenticate through the service account's IAM role; see Amazon Bedrock in the Helm chart guide.

Option 2: External Secrets Operator

yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: sibyl-secrets
  namespace: sibyl
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: sibyl-secrets
  data:
    - secretKey: SIBYL_JWT_SECRET
      remoteRef:
        key: sibyl/jwt-secret
    - secretKey: SIBYL_OPENAI_API_KEY
      remoteRef:
        key: sibyl/openai-key

Option 3: Sealed Secrets

bash

# Create SealedSecret
kubeseal --format=yaml < sibyl-secrets.yaml > sibyl-secrets-sealed.yaml
kubectl apply -f sibyl-secrets-sealed.yaml

Schema Bootstrap

Sibyl bootstraps SurrealDB schema inline at startup. The Helm chart runs no pre-upgrade migration hook.

Ingress Configuration

The chart renders from the shared ingress.hosts route table. Set ingress.classic.enabled=true for a classic networking.k8s.io/v1 Ingress, or ingress.gatewayApi.enabled=true (with parentRefs) for a Gateway API HTTPRoute. The standalone manifests below are equivalent hand-written forms if you prefer to manage routing outside the chart.

Client Addresses Behind the Ingress

Every request the backend receives comes from an ingress controller or Gateway pod, so on its own the backend sees one address for all users. The five-per-minute login limit is keyed on that address, which means the sixth sign-in in a minute from anyone locks everyone out. Tell the backend which proxies to believe and it reads the real client from X-Forwarded-For instead:

yaml

backend:
  # A range only the ingress controller pods draw their addresses from.
  forwardedAllowIps: "10.250.0.0/28"

Every trusted address must be a proxy, never something that could be a client. A trusted peer can name any client address, and an ingress that appends to X-Forwarded-For rather than replacing it (Envoy-based gateways, AWS ALB, and nginx's $proxy_add_x_forwarded_for all do) hands the same power to anyone who calls through it from a trusted address: the ingress appends the caller's own address, the backend skips it as trusted, and the caller's forged entry wins. A NetworkPolicy cannot close that path, because the request really does arrive from the controller. So do not trust the cluster pod CIDR, node ranges, or the VPC CIDR; each of them holds clients.

Recommended setup:

Give the controller pods addresses nothing else draws from (a CNI IP pool selected by the controller's namespace, or a node pool reserved for the controller with its own pod range) and trust only that range. kubectl get pods -n <controller-namespace> -o wide shows the addresses it has to cover.
Configure the controller to replace X-Forwarded-For from untrusted peers rather than append to it. ingress-nginx does by default, as long as use-forwarded-headers and compute-full-forwarded-for stay off.
Set networkPolicy.enabled: true with a networkPolicy.ingress.from that selects the controller's namespace, so nothing can reach the backend directly.

The backend walks X-Forwarded-For from the right and stops at the first address it does not trust. Handle any hop in front of the controller at the controller, not in the backend's list:

A cloud L7 load balancer in front of the controller: have the controller trust only the load balancer's dedicated subnets and resolve the client itself (in ingress-nginx, use-forwarded-headers: true with proxy-real-ip-cidr limited to them, since it defaults to 0.0.0.0/0), so the backend still receives one resolved address. Do not add the VPC CIDR to forwardedAllowIps.
An L4 load balancer or externalTrafficPolicy: Cluster, where traffic reaches the controller from node addresses: do not add the node range, because the entry to its left is one the client wrote. Set externalTrafficPolicy: Local on the controller's Service, or enable PROXY protocol, so the controller sees real client addresses.

Trusting a range lets anything inside it choose its client address, which also decides the break-glass allowlist, so list the controller and nothing else, and keep /api and /mcp routed straight to the backend service (the Next.js frontend passes a client's own X-Forwarded-For through untouched). After an upgrade, sign in and check that the backend's request log lines (kubectl logs -n sibyl deploy/sibyl-backend) carry your own address in client, not a controller pod's. The Helm reference covers the value's forms and the "*" escape hatch, and the environment reference explains how the address resolves.

Kong Gateway (standalone HTTPRoute)

yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: sibyl
  namespace: sibyl
spec:
  parentRefs:
    - name: production-gateway
      namespace: kong
  hostnames:
    - sibyl.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
        - path:
            type: PathPrefix
            value: /mcp
      backendRefs:
        - name: sibyl-backend
          port: 3334
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: sibyl-frontend
          port: 3337

NGINX Ingress (standalone manifest)

yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sibyl
  namespace: sibyl
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - sibyl.example.com
      secretName: sibyl-tls
  rules:
    - host: sibyl.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: sibyl-backend
                port:
                  number: 3334
          - path: /mcp
            pathType: Prefix
            backend:
              service:
                name: sibyl-backend
                port:
                  number: 3334
          - path: /
            pathType: Prefix
            backend:
              service:
                name: sibyl-frontend
                port:
                  number: 3337

Health Checks

The chart configures liveness and readiness probes:

yaml

backend:
  livenessProbe:
    httpGet:
      path: /api/health
      port: http
    initialDelaySeconds: 10
    periodSeconds: 30

  readinessProbe:
    httpGet:
      path: /api/health/ready
      port: http
    initialDelaySeconds: 5
    periodSeconds: 10

The readiness probe targets /api/health/ready, the deep readiness endpoint that returns 503 when SurrealDB is unreachable so the pod is pulled from service until it can serve. Liveness stays on /api/health.

Scaling

Manual Scaling

bash

kubectl scale deployment sibyl-backend -n sibyl --replicas=5

HPA Configuration

With autoscaling enabled:

yaml

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300 # Wait 5min before scaling down
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15

Monitoring

Check Deployment Status

bash

# All resources
kubectl get all -n sibyl

# Pods status
kubectl get pods -n sibyl -o wide

# HPA status
kubectl get hpa -n sibyl

# Events
kubectl get events -n sibyl --sort-by='.lastTimestamp'

View Logs

bash

# Backend logs
kubectl logs -n sibyl -l app.kubernetes.io/component=backend -f

# Worker logs
kubectl logs -n sibyl -l app.kubernetes.io/component=worker -f

# All Sibyl logs
kubectl logs -n sibyl -l app.kubernetes.io/name=sibyl -f

Upgrades

Keep image tags and security contexts in lockstep. Current Sibyl images run backend/worker as 10001:10001 and frontend as 10002:10002; older pinned images need matching overrides in backend.podSecurityContext, frontend.podSecurityContext, and worker.podSecurityContext.

bash

# Update values (use ./charts/sibyl instead when installing from source)
helm repo update
helm upgrade sibyl sibyl/sibyl \
  -n sibyl \
  -f values-production.yaml \
  --set backend.image.tag=1.4.10 \
  --set frontend.image.tag=1.4.10

# Rollback if needed
helm rollback sibyl -n sibyl

Uninstall

bash

# Remove Helm release
helm uninstall sibyl -n sibyl

# Remove namespace (DELETES ALL DATA)
kubectl delete namespace sibyl

Next Steps

Helm Chart Reference - Complete values documentation
Environment Variables - All configuration options
Monitoring - Observability setup

Production Kubernetes Deployment ​

Prerequisites ​

Architecture Overview ​

Quick Start ​

Values Configuration ​

Database Setup ​

SurrealDB Requirements (default) ​

Coordination Requirements ​

Secrets Management ​

Option 1: Kubernetes Secrets ​

Option 2: External Secrets Operator ​

Option 3: Sealed Secrets ​

Schema Bootstrap ​

Ingress Configuration ​

Client Addresses Behind the Ingress ​

Kong Gateway (standalone HTTPRoute) ​

NGINX Ingress (standalone manifest) ​

Health Checks ​

Scaling ​

Manual Scaling ​

HPA Configuration ​

Monitoring ​

Check Deployment Status ​

View Logs ​

Upgrades ​

Uninstall ​

Next Steps ​

Production Kubernetes Deployment

Prerequisites

Architecture Overview

Quick Start

Values Configuration

Database Setup

SurrealDB Requirements (default)

Coordination Requirements

Secrets Management

Option 1: Kubernetes Secrets

Option 2: External Secrets Operator

Option 3: Sealed Secrets

Schema Bootstrap

Ingress Configuration

Client Addresses Behind the Ingress

Kong Gateway (standalone HTTPRoute)

NGINX Ingress (standalone manifest)

Health Checks

Scaling

Manual Scaling

HPA Configuration

Monitoring

Check Deployment Status

View Logs

Upgrades

Uninstall

Next Steps