---
url: /sibyl/deployment/kubernetes.md
---
# Production Kubernetes Deployment

Deploy Sibyl to a production Kubernetes cluster using Helm.

## Prerequisites

* Kubernetes cluster (1.27+)
* kubectl configured for your cluster
* Helm 3.x
* **SurrealDB** instance (in-cluster StatefulSet, Surreal Cloud, or another external host)

## Architecture Overview

```
+---------------------------- Production Cluster -----------------------------+
|                                                                              |
|  +------------------+     +------------------+     +------------------+      |
|  |  Ingress/Gateway |     |    Backend (n)   |     |   Frontend (n)   |      |
|  |  (Kong/Nginx)    |---->|    HPA: 2-10     |     |   HPA: 2-10      |      |
|  +------------------+     +--------+---------+     +------------------+      |
|                                    |                                         |
|                          +---------+--------+                                |
|                          |    Worker (n)    |                                |
|                          |    HPA: 1-5      |                                |
|                          +------------------+                                |
|                                                                              |
+------------------------------------------------------------------------------+
                                    |
                         +----------+-----------+
                         |      SurrealDB       |
                         | graph + content +    |
                         | auth (unified)       |
                         +----------------------+
```

SurrealDB is the only data store. See [Storage Modes](../guide/storage-modes.md) for connection
options.

## Quick Start

Provision the `sibyl-validation-receipts` persistent claim before installation. API and worker share
it, along with the organization backup archives kept in its `backups` directory; multi-node replicas
require ReadWriteMany storage. With receipts in S3, give archives a bucket prefix
(`backend.backupArchives.url`) or a claim of their own; see
[Backup Archives](helm-chart.md#backup-archives).

```bash
# Add namespace
kubectl create namespace sibyl

# Create secrets (Surreal default)
kubectl create secret generic sibyl-secrets -n sibyl \
  --from-literal=SIBYL_JWT_SECRET=$(openssl rand -hex 32) \
  --from-literal=SIBYL_SETTINGS_KEY=$(openssl rand -base64 32 | tr '+/' '-_') \
  --from-literal=SIBYL_OPENAI_API_KEY=sk-... \
  --from-literal=SIBYL_ANTHROPIC_API_KEY=sk-ant-...

kubectl create secret generic sibyl-surreal -n sibyl \
  --from-literal=password=<your-surreal-password>

# Install with Helm from the published repository
helm repo add sibyl https://raw.githubusercontent.com/hyperb1iss/sibyl/gh-pages
helm repo update

helm upgrade --install sibyl sibyl/sibyl \
  -n sibyl \
  -f values-production.yaml
```

Installing from a source checkout works the same way with the local chart path: replace
`sibyl/sibyl` with `./charts/sibyl`.

## Values Configuration

Create a `values-production.yaml` (Surreal default):

```yaml
# Coordination mode
coordinationBackend: "redis"

backend:
  replicaCount: 2

  image:
    repository: ghcr.io/hyperb1iss/sibyl-api
    tag: "1.4.10"
    pullPolicy: Always

  # Reference pre-created secrets
  existingSecret: sibyl-secrets
  validationReceipts:
    existingClaim: sibyl-validation-receipts
    # Or keep receipts in S3 and leave existingClaim empty:
    # url: "s3://sibyl-receipts/prod?region=us-west-2"
  # Archives default to the receipts claim; with receipts in S3:
  # backupArchives:
  #   url: "s3://sibyl-backups/prod?region=us-west-2"
  # Only the ingress controller pods' own range, so logins key on each user's
  # address. Never the pod CIDR; see "Client Addresses Behind the Ingress".
  forwardedAllowIps: "10.250.0.0/28"

  # SurrealDB connection
  surreal:
    url: "ws://your-surrealdb.example.com:8000/rpc"
    username: "root"
    existingSecret: sibyl-surreal
    namespacePrefix: "org_"
    database: "graph"

  # Valkey/Redis coordination for multi-replica deployments
  redis:
    host: "valkey.example.svc.cluster.local"
    port: "6379"
    jobsDb: "1"
    rateLimitDb: "4"
    existingSecret: sibyl-redis
    secretKey: password

  env:
    SIBYL_SERVER_HOST: "0.0.0.0"
    SIBYL_SERVER_PORT: "3334"
    SIBYL_ENVIRONMENT: "production"
    SIBYL_PUBLIC_URL: "https://sibyl.example.com"
    SIBYL_LLM_PROVIDER: "anthropic"
    SIBYL_LLM_MODEL: "claude-haiku-5-5"

  # Enable autoscaling
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 80

  # Enable PodDisruptionBudget
  pdb:
    enabled: true
    minAvailable: 1

  # Spread pods across nodes
  podAntiAffinity:
    enabled: true
    type: soft
    topologyKey: kubernetes.io/hostname

  resources:
    limits:
      cpu: 2000m
      memory: 2Gi
    requests:
      cpu: 500m
      memory: 512Mi

frontend:
  enabled: true
  replicaCount: 2

  image:
    repository: ghcr.io/hyperb1iss/sibyl-web
    tag: "1.4.10"

  apiUrl: "http://sibyl-backend:3334/api"

  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10

  pdb:
    enabled: true
    minAvailable: 1

worker:
  enabled: true
  replicaCount: 2

  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 5

  pdb:
    enabled: true
    minAvailable: 1

# Ingress (adjust for your ingress controller)
# ingress.hosts is the shared route table. Enable ingress.classic for a classic
# networking.k8s.io/v1 Ingress, or ingress.gatewayApi for a Gateway API HTTPRoute.
ingress:
  hosts:
    - host: sibyl.example.com
      paths:
        - path: /api
          pathType: Prefix
          service: backend
        - path: /mcp
          pathType: Prefix
          service: backend
        - path: /
          pathType: Prefix
          service: frontend
  # Classic Ingress (NGINX and similar):
  classic:
    enabled: true
    className: "nginx"
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    tls:
      - secretName: sibyl-tls
        hosts:
          - sibyl.example.com
  # Gateway API HTTPRoute (Kong and similar): enable this instead of classic.
  gatewayApi:
    enabled: false
    parentRefs:
      - name: production-gateway
        namespace: kong
```

## Database Setup

### SurrealDB Requirements (default)

* SurrealDB 3.x. Pin an explicit tested image tag for production instead of floating on `latest`.
* TiKV-backed storage for replicated SurrealDB pods, or RocksDB-backed storage for a single pod
* Root credentials set at start (`--user` / `--pass`)
* Persistence and backups live in the selected SurrealDB storage engine

For a scalable in-cluster shape, deploy the TiDB Operator, create a small TiKV/PD cluster, and point
the official SurrealDB Helm chart at `tikv://<pd-service>:2379`. For single-pod installs, a
RocksDB-backed PVC is simpler. Either way, `tikv://` and `rocksdb://` configure the SurrealDB server
only. Point Sibyl's `backend.surreal.url` at that server with a `ws://`, `wss://`, `http://`, or
`https://` URL. The backend rejects unsupported schemes such as `rocksdb://` and `tikv://` at
startup; the embedded forms have their own rules and are not meant for a cluster (see
[SurrealDB URL forms](./environment.md#surrealdb-url-forms)).

### Coordination Requirements

Set `coordinationBackend: "redis"` when running multiple backend or worker replicas. Use Valkey or
Redis for arq jobs, distributed locks, WebSocket pub/sub, and shared rate limits.

## Secrets Management

### Option 1: Kubernetes Secrets

```bash
# Create from literal values
kubectl create secret generic sibyl-secrets -n sibyl \
  --from-literal=SIBYL_JWT_SECRET=$(openssl rand -hex 32) \
  --from-literal=SIBYL_OPENAI_API_KEY=sk-... \
  --from-literal=SIBYL_ANTHROPIC_API_KEY=sk-ant-...
```

On AWS with the `bedrock` provider, leave the two provider keys out. Claude and Cohere embeddings
then authenticate through the service account's IAM role; see
[Amazon Bedrock](./helm-chart.md#amazon-bedrock) in the Helm chart guide.

### Option 2: External Secrets Operator

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: sibyl-secrets
  namespace: sibyl
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: sibyl-secrets
  data:
    - secretKey: SIBYL_JWT_SECRET
      remoteRef:
        key: sibyl/jwt-secret
    - secretKey: SIBYL_OPENAI_API_KEY
      remoteRef:
        key: sibyl/openai-key
```

### Option 3: Sealed Secrets

```bash
# Create SealedSecret
kubeseal --format=yaml < sibyl-secrets.yaml > sibyl-secrets-sealed.yaml
kubectl apply -f sibyl-secrets-sealed.yaml
```

## Schema Bootstrap

Sibyl bootstraps SurrealDB schema inline at startup. The Helm chart runs no pre-upgrade migration
hook.

## Ingress Configuration

The chart renders from the shared `ingress.hosts` route table. Set `ingress.classic.enabled=true`
for a classic `networking.k8s.io/v1` Ingress, or `ingress.gatewayApi.enabled=true` (with
`parentRefs`) for a Gateway API HTTPRoute. The standalone manifests below are equivalent
hand-written forms if you prefer to manage routing outside the chart.

### Client Addresses Behind the Ingress

Every request the backend receives comes from an ingress controller or Gateway pod, so on its own
the backend sees one address for all users. The five-per-minute login limit is keyed on that
address, which means the sixth sign-in in a minute from anyone locks everyone out. Tell the backend
which proxies to believe and it reads the real client from `X-Forwarded-For` instead:

```yaml
backend:
  # A range only the ingress controller pods draw their addresses from.
  forwardedAllowIps: "10.250.0.0/28"
```

**Every trusted address must be a proxy, never something that could be a client.** A trusted peer
can name any client address, and an ingress that appends to `X-Forwarded-For` rather than replacing
it (Envoy-based gateways, AWS ALB, and nginx's `$proxy_add_x_forwarded_for` all do) hands the same
power to anyone who calls through it from a trusted address: the ingress appends the caller's own
address, the backend skips it as trusted, and the caller's forged entry wins. A NetworkPolicy cannot
close that path, because the request really does arrive from the controller. So do not trust the
cluster pod CIDR, node ranges, or the VPC CIDR; each of them holds clients.

**Recommended setup:**

* Give the controller pods addresses nothing else draws from (a CNI IP pool selected by the
  controller's namespace, or a node pool reserved for the controller with its own pod range) and
  trust only that range. `kubectl get pods -n <controller-namespace> -o wide` shows the addresses it
  has to cover.
* Configure the controller to replace `X-Forwarded-For` from untrusted peers rather than append to
  it. ingress-nginx does by default, as long as `use-forwarded-headers` and
  `compute-full-forwarded-for` stay off.
* Set `networkPolicy.enabled: true` with a `networkPolicy.ingress.from` that selects the
  controller's namespace, so nothing can reach the backend directly.

The backend walks `X-Forwarded-For` from the right and stops at the first address it does not trust.
Handle any hop in front of the controller at the controller, not in the backend's list:

* A cloud L7 load balancer in front of the controller: have the controller trust only the load
  balancer's dedicated subnets and resolve the client itself (in ingress-nginx,
  `use-forwarded-headers: true` with `proxy-real-ip-cidr` limited to them, since it defaults to
  `0.0.0.0/0`), so the backend still receives one resolved address. Do not add the VPC CIDR to
  `forwardedAllowIps`.
* An L4 load balancer or `externalTrafficPolicy: Cluster`, where traffic reaches the controller from
  node addresses: do not add the node range, because the entry to its left is one the client wrote.
  Set `externalTrafficPolicy: Local` on the controller's Service, or enable PROXY protocol, so the
  controller sees real client addresses.

Trusting a range lets anything inside it choose its client address, which also decides the
break-glass allowlist, so list the controller and nothing else, and keep `/api` and `/mcp` routed
straight to the backend service (the Next.js frontend passes a client's own `X-Forwarded-For`
through untouched). After an upgrade, sign in and check that the backend's `request` log lines
(`kubectl logs -n sibyl deploy/sibyl-backend`) carry your own address in `client`, not a controller
pod's. The [Helm reference](./helm-chart.md#trusted-proxies) covers the value's forms and the `"*"`
escape hatch, and the [environment reference](./environment.md#trusted-proxies) explains how the
address resolves.

### Kong Gateway (standalone HTTPRoute)

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: sibyl
  namespace: sibyl
spec:
  parentRefs:
    - name: production-gateway
      namespace: kong
  hostnames:
    - sibyl.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
        - path:
            type: PathPrefix
            value: /mcp
      backendRefs:
        - name: sibyl-backend
          port: 3334
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: sibyl-frontend
          port: 3337
```

### NGINX Ingress (standalone manifest)

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sibyl
  namespace: sibyl
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - sibyl.example.com
      secretName: sibyl-tls
  rules:
    - host: sibyl.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: sibyl-backend
                port:
                  number: 3334
          - path: /mcp
            pathType: Prefix
            backend:
              service:
                name: sibyl-backend
                port:
                  number: 3334
          - path: /
            pathType: Prefix
            backend:
              service:
                name: sibyl-frontend
                port:
                  number: 3337
```

## Health Checks

The chart configures liveness and readiness probes:

```yaml
backend:
  livenessProbe:
    httpGet:
      path: /api/health
      port: http
    initialDelaySeconds: 10
    periodSeconds: 30

  readinessProbe:
    httpGet:
      path: /api/health/ready
      port: http
    initialDelaySeconds: 5
    periodSeconds: 10
```

The readiness probe targets `/api/health/ready`, the deep readiness endpoint that returns `503` when
SurrealDB is unreachable so the pod is pulled from service until it can serve. Liveness stays on
`/api/health`.

## Scaling

### Manual Scaling

```bash
kubectl scale deployment sibyl-backend -n sibyl --replicas=5
```

### HPA Configuration

With autoscaling enabled:

```yaml
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300 # Wait 5min before scaling down
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
```

## Monitoring

### Check Deployment Status

```bash
# All resources
kubectl get all -n sibyl

# Pods status
kubectl get pods -n sibyl -o wide

# HPA status
kubectl get hpa -n sibyl

# Events
kubectl get events -n sibyl --sort-by='.lastTimestamp'
```

### View Logs

```bash
# Backend logs
kubectl logs -n sibyl -l app.kubernetes.io/component=backend -f

# Worker logs
kubectl logs -n sibyl -l app.kubernetes.io/component=worker -f

# All Sibyl logs
kubectl logs -n sibyl -l app.kubernetes.io/name=sibyl -f
```

## Upgrades

Keep image tags and security contexts in lockstep. Current Sibyl images run backend/worker as
`10001:10001` and frontend as `10002:10002`; older pinned images need matching overrides in
`backend.podSecurityContext`, `frontend.podSecurityContext`, and `worker.podSecurityContext`.

```bash
# Update values (use ./charts/sibyl instead when installing from source)
helm repo update
helm upgrade sibyl sibyl/sibyl \
  -n sibyl \
  -f values-production.yaml \
  --set backend.image.tag=1.4.10 \
  --set frontend.image.tag=1.4.10

# Rollback if needed
helm rollback sibyl -n sibyl
```

## Uninstall

```bash
# Remove Helm release
helm uninstall sibyl -n sibyl

# Remove namespace (DELETES ALL DATA)
kubectl delete namespace sibyl
```

## Next Steps

* [Helm Chart Reference](helm-chart.md) - Complete values documentation
* [Environment Variables](environment.md) - All configuration options
* [Monitoring](monitoring.md) - Observability setup