---
url: /sibyl/deployment/helm-chart.md
---
# Helm Chart Reference

Complete reference for the Sibyl Helm chart (`charts/sibyl`).

## Chart Info

```yaml
apiVersion: v2
name: sibyl
description: Knowledge graph and task workflow for durable development memory
type: application
version: 1.4.10
appVersion: "1.4.10"
```

Release builds update `version` and `appVersion` from the repository `VERSION` file.

## Installation

Release charts are published to the `sibyl` Helm repository on the `gh-pages` branch. The repository
serves two charts: `sibyl` (the application) and `sibyl-surrealdb` (a SurrealDB wrapper).

Choose where completed validation receipts live before installation. An S3 bucket
(`backend.validationReceipts.url`) lets API and worker run on any node with no shared volume. A
persistent claim (`backend.validationReceipts.existingClaim`) must be shared by API and worker, and
multi-node replicas then require ReadWriteMany storage. See
[Durable Validation Receipts](#durable-validation-receipts). Organization backup archives follow the
same choice: by default they live on the receipts claim, and with receipts in S3 they need their own
bucket prefix or claim. See [Backup Archives](#backup-archives).

```bash
# From the published Helm repository (recommended)
helm repo add sibyl https://raw.githubusercontent.com/hyperb1iss/sibyl/gh-pages
helm repo update

helm upgrade --install sibyl sibyl/sibyl \
  -n sibyl \
  --create-namespace \
  -f values.yaml
```

To install from a source checkout instead:

```bash
# From local chart
helm upgrade --install sibyl ./charts/sibyl \
  -n sibyl \
  --create-namespace \
  -f values.yaml

# Dry run
helm template sibyl ./charts/sibyl -f values.yaml
```

## Global Settings

```yaml
global:
  # Image pull secrets for private registries
  imagePullSecrets: []
```

## Schema Bootstrap

Sibyl bootstraps SurrealDB schemas at application startup. The chart runs no separate migration job.

## Authentication Defaults

The chart defaults match the self-hosted single-user path. Local username/password login is enabled,
the first setup signup creates the owner/admin user, post-setup account creation is invite-only, and
OIDC is empty until an operator configures it.

```yaml
auth:
  # Local username/password login is the default simple path.
  localAuthEnabled: true
  # Public account creation after setup stays invite-only by default.
  publicSignupsEnabled: false

oidc:
  providers: []
  silent_refresh_enabled: false
  extra_providers_enabled: false

breakGlass:
  enabled: false
```

For enterprise SSO deployments, configure a corporate OIDC provider and set
`auth.localAuthEnabled=false` only after an owner has successfully signed in through OIDC.
Break-glass access remains a separate, bounded opt-in.

Each configured provider requires an exact organization binding:

```yaml
oidc:
  providers:
    - name: entra
      issuer: "https://login.microsoftonline.com/<tenant-id>/v2.0"
      client_id: "<app-client-id>"
      client_secret_env: "SIBYL_OIDC_ENTRA_CLIENT_SECRET"
      organization_slug: "acme"
```

The organization must already exist and must not be personal. Helm rejects providers without this
binding, and Sibyl fails closed if the bound organization cannot be resolved.

## Backend Configuration

### Basic Settings

```yaml
publicUrl: "https://sibyl.example.com"

backend:
  # Number of replicas (ignored if autoscaling is enabled)
  replicaCount: 1

  # Empty selects Recreate for a single fixed replica, RollingUpdate otherwise
  strategy: {}

  image:
    repository: ghcr.io/hyperb1iss/sibyl-api
    pullPolicy: IfNotPresent
    # Defaults to chart appVersion if empty
    tag: ""
```

Set `publicUrl` to the external application origin. The chart passes the value to the backend for
callbacks, password resets, and redirects. It also derives the browser API and WebSocket origin. Use
`frontend.publicApiUrl` only when the browser reaches the API through a different origin.

### Service

```yaml
backend:
  service:
    type: ClusterIP
    port: 3334
    annotations: {}
```

The backend needs no session affinity. MCP runs stateless, so any replica can answer any request.

### Autoscaling

```yaml
backend:
  autoscaling:
    enabled: false
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 80
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 300
        policies:
          - type: Percent
            value: 10
            periodSeconds: 60
      scaleUp:
        stabilizationWindowSeconds: 0
        policies:
          - type: Percent
            value: 100
            periodSeconds: 15
          - type: Pods
            value: 4
            periodSeconds: 15
        selectPolicy: Max
```

### Pod Disruption Budget

```yaml
backend:
  pdb:
    enabled: false
    # Minimum available pods (mutually exclusive with maxUnavailable)
    minAvailable: 1
    # maxUnavailable: 1
```

### Pod Anti-Affinity

```yaml
backend:
  podAntiAffinity:
    # Spreads pods across nodes
    enabled: false
    # "soft" (preferred) or "hard" (required)
    type: soft
    topologyKey: kubernetes.io/hostname
```

### Resources

```yaml
backend:
  resources:
    limits:
      cpu: 1000m
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 256Mi
```

### Health Probes

```yaml
backend:
  livenessProbe:
    httpGet:
      path: /api/health
      port: http
    initialDelaySeconds: 60
    periodSeconds: 30

  readinessProbe:
    httpGet:
      path: /api/health/ready
      port: http
    initialDelaySeconds: 5
    periodSeconds: 10
```

The readiness probe must hit `/api/health/ready`, the deep readiness endpoint that returns `503`
when SurrealDB is unreachable. Liveness stays on `/api/health`, which only asserts the process is
up.

### Environment Variables

```yaml
backend:
  env:
    SIBYL_SERVER_HOST: "0.0.0.0"
    SIBYL_SERVER_PORT: "3334"
    SIBYL_ENVIRONMENT: "production"
    SIBYL_LLM_PROVIDER: "anthropic"
    SIBYL_LLM_MODEL: "claude-haiku-5-5"
    SIBYL_EMBEDDING_MODEL: "text-embedding-3-small"
    SIBYL_EMBEDDING_DIMENSIONS: "1536"
    # BLAS/OpenMP thread caps keep native math libraries from oversubscribing pods
    OPENBLAS_NUM_THREADS: "1"
    OMP_NUM_THREADS: "1"
    MKL_NUM_THREADS: "1"
    NUMEXPR_NUM_THREADS: "1"
```

### Trusted Proxies

Behind an ingress controller or Gateway, every request reaches the backend from the proxy pod's
address. The login route allows five attempts per minute per client address, so until the backend
trusts the proxy, all users share one bucket and the sixth login in a minute from anyone locks
everyone out. The `backend.forwardedAllowIps` value names the proxies whose `X-Forwarded-For` header
the backend believes. The chart renders it into `SIBYL_FORWARDED_ALLOW_IPS`.

| Value                       | Default | Description                                            |
| --------------------------- | ------- | ------------------------------------------------------ |
| `backend.forwardedAllowIps` | `""`    | IPs and CIDR ranges of trusted proxies, string or list |

```yaml
backend:
  # A range only the ingress controller pods draw their addresses from, never the pod CIDR.
  forwardedAllowIps: "10.250.0.0/28"
  # A list works too:
  # forwardedAllowIps:
  #   - 10.250.0.0/28
  #   - 10.250.1.0/28
```

On the command line, pass a list so Helm does not split on the comma:
`--set 'backend.forwardedAllowIps={10.250.0.0/28,10.250.1.0/28}'`.

Empty (the default) keeps the backend's loopback-only trust. When `backend.forwardedAllowIps` is
empty, a `SIBYL_FORWARDED_ALLOW_IPS` entry under `backend.env` still works; when both are set, the
dedicated value wins.

**Every trusted address must be a proxy, never something that could be a client.** A trusted peer
can name any client address it likes, and an ingress that appends to `X-Forwarded-For` instead of
replacing it extends that power to everyone who calls through it from a trusted address. Envoy-based
gateways, AWS ALB, and nginx's `$proxy_add_x_forwarded_for` all append. With the pod CIDR trusted, a
pod at `10.244.9.9` that sends `X-Forwarded-For: 100.101.102.103` through such an ingress arrives as
`100.101.102.103, 10.244.9.9`; the backend skips the pod's own address as trusted and resolves the
forged one. A NetworkPolicy cannot stop this, because the request really does come from the
controller. So never trust the cluster pod CIDR (it covers every pod, and with
`networkPolicy.enabled` at its default of `false` any of them can also reach the backend directly),
node ranges, or the VPC CIDR: each of them holds clients.

What to set:

* **Trust only the ingress controller pods.** Give them addresses nothing else draws from, such as a
  CNI IP pool selected by the controller's namespace or a node pool reserved for the controller with
  its own pod range, and list that range. Running
  `kubectl get pods -n <controller-namespace> -o wide` shows the addresses it has to cover.
* **Make the controller replace `X-Forwarded-For` from untrusted peers rather than append to it.**
  ingress-nginx does by default, as long as `use-forwarded-headers` and `compute-full-forwarded-for`
  stay off. With a controller that can only append, the trust list is all that keeps a forged entry
  out, so it must hold nothing but the controller.
* **A load balancer in front of the controller:** resolve the client at the controller rather than
  widening the backend's list. Have the controller trust only the load balancer's dedicated subnets
  (in ingress-nginx, `use-forwarded-headers: true` with `proxy-real-ip-cidr` set to them, because
  its default is `0.0.0.0/0`), so it still hands the backend a single resolved address, and keep the
  backend trusting the controller alone.
* **An L4 load balancer or a node hop that SNATs:** do not trust node addresses. The controller then
  only ever sees node IPs, and trusting them hands the entry the client wrote the final say.
  Preserve the source address instead (`externalTrafficPolicy: Local` on the controller's Service,
  or PROXY protocol from the load balancer) and trust only the controller pods.
* **Enable `networkPolicy`** with `networkPolicy.ingress.from` naming the controller, so nothing can
  reach the backend directly and write the header itself.

::: warning Trusting a range lets anything inside it choose its client address

Any peer inside `forwardedAllowIps` can claim to be any client, dodge the per-address rate limits,
and satisfy `breakGlass.allowedIPs`, and so can any caller behind an appending ingress whose own
address is trusted. List the controller and nothing else. Keep `/api` and `/mcp` routed straight to
the backend service (the default route table): the Next.js frontend passes a client-supplied
`X-Forwarded-For` through unchanged, so it must never be the hop the backend trusts. A value of
`"*"`, or any range broader than an IPv4 `/8` or an IPv6 `/32`, trusts every peer or nearly every
one and makes the backend log `forwarded_allow_ips_trusts_every_peer` at startup; with every hop
trusted, the leftmost header entry wins, and a client writes that one unless every proxy in front
overwrites the header.

:::

To confirm the setting took, sign in through the ingress and check that the backend's `request` log
lines carry your own address in `client`, not the controller pod's. The
[environment reference](./environment.md#trusted-proxies) explains how the backend resolves the
address and which values it accepts.

### Secrets

```yaml
backend:
  # Reference to existing secret for sensitive env vars
  # Must contain: SIBYL_JWT_SECRET and SIBYL_SETTINGS_KEY.
  # Add provider and LLM API keys as needed; Amazon Bedrock needs none.
  existingSecret: ""
```

Production and read-only containers must set `existingSecret`. The chart enforces this at render
time: with `backend.env.SIBYL_ENVIRONMENT` set to `production` and no `backend.existingSecret`,
`helm template` and `helm install` both abort with an error naming the value to set. That guard
exists because an unset `SIBYL_JWT_SECRET` makes the backend sign sessions with an empty key and
makes MCP auth disable itself under the default `mcp_auth_mode: auto`. The chart deliberately does
not auto-generate the secret in production, since a per-render value would rotate on every upgrade
and diverge across the backend, worker, and bootstrap pods.

Passing the signing key as `backend.env.SIBYL_JWT_SECRET` fails the render as well, and so does the
unprefixed `backend.env.JWT_SECRET` alias that the server falls back to. Every key under
`backend.env` is written into the plaintext `<release>-config` ConfigMap, which is readable under
broader RBAC than a Secret and routinely committed to GitOps repositories. Key names are matched
case-insensitively, because pydantic-settings resolves environment variables that way.

Production also rejects `backend.env.SIBYL_MCP_AUTH_MODE: off`, which would serve every MCP tool
unauthenticated no matter how well the JWT secret is provisioned. Use `auto` to enforce Bearer auth
once a secret is set, or `on` to enforce it unconditionally. The server refuses to boot on the same
condition, so the mode cannot be switched off in production by editing the ConfigMap after install.

The escape hatch for a trial install is `backend.env.SIBYL_ENVIRONMENT: development`, which is
unsuitable for real traffic: the server then derives a JWT secret per process, and the chart's
default `readOnlyRootFilesystem: true` blocks the write that would persist it, so sessions break on
every restart and across replicas.

Both required keys are non-optional pod references, so a missing key fails before Sibyl starts
instead of silently generating an ephemeral replacement. Keep `SIBYL_SETTINGS_KEY` stable across
upgrades and pod replacements; changing it makes encrypted settings unreadable.

### Storage Mode

The active persistence runtime is fixed to SurrealDB. Local and auto coordination run jobs in the
API process and do not render a separate worker. Set Redis coordination explicitly for multi-pod
deployments:

```yaml
coordinationBackend: "redis"
```

See [storage-modes.md](../guide/storage-modes.md) for the mode matrix.

### SurrealDB Connection (default)

```yaml
backend:
  surreal:
    # ws:// or http:// URL to an external SurrealDB instance
    url: "ws://surrealdb:8000/rpc"
    username: "root"
    # Reference to a secret containing the password.
    # When empty, a password is auto-generated and stored in `<release>-surreal`.
    existingSecret: ""
    # Secret key holding the password (only used when existingSecret is set)
    secretKey: "password"
    # Inline password (ignored when existingSecret is set; auto-generated otherwise)
    password: ""
    namespacePrefix: "org_"
    database: "graph"
```

The chart renders `backend.surreal.url` into `SIBYL_SURREAL_URL`. Point it at a SurrealDB server
(`ws://`, `wss://`, `http://`, or `https://`). The backend refuses an unsupported scheme at startup:
`rocksdb://` and `tikv://` are storage arguments for the SurrealDB server, not client URLs. See
[SurrealDB URL forms](./environment.md#surrealdb-url-forms).

### Redis or Valkey Coordination

Used when `coordinationBackend: "redis"`. This is recommended when running more than one backend or
worker pod because it backs arq jobs, distributed locks, WebSocket pub/sub, and shared rate limits.

```yaml
backend:
  redis:
    host: "valkey"
    port: "6379"
    jobsDb: "1"
    rateLimitDb: "4"
    # Reference to a secret containing the Redis/Valkey password
    existingSecret: ""
    secretKey: "password"
    # Inline password (not recommended; prefer existingSecret)
    password: ""
    # Shared rate-limit storage URL.
    # Leave empty to derive a redis:// URL from host/port/rateLimitDb.
    rateLimitStorage: ""
```

The chart emits a password-free `SIBYL_RATE_LIMIT_STORAGE` ConfigMap value and Sibyl injects the
Redis password from `SIBYL_REDIS_PASSWORD` at runtime.

### Security Contexts

The default UIDs match the packaged images for this chart version. If you intentionally pin older
image tags, keep the image tag and security context in lockstep: pre-service-UID API/worker images
use `1000:1000`, and pre-service-UID web images use `1001:65533`.

```yaml
backend:
  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 10001
    runAsGroup: 10001
    fsGroup: 10001
    fsGroupChangePolicy: OnRootMismatch

  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL
```

### Pod Placement

```yaml
backend:
  nodeSelector: {}
  tolerations: []
  # Custom affinity (overridden by podAntiAffinity if enabled)
  affinity: {}
  podAnnotations: {}
```

## Frontend Configuration

```yaml
frontend:
  enabled: true
  replicaCount: 1

  image:
    repository: ghcr.io/hyperb1iss/sibyl-web
    pullPolicy: IfNotPresent
    tag: ""

  service:
    type: ClusterIP
    port: 3337

  autoscaling:
    enabled: false
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 80
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 300
        policies:
          - type: Percent
            value: 10
            periodSeconds: 60

  pdb:
    enabled: false
    minAvailable: 1

  podAntiAffinity:
    enabled: false
    type: soft
    topologyKey: kubernetes.io/hostname

  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 50m
      memory: 128Mi

  livenessProbe:
    httpGet:
      path: /
      port: http
    initialDelaySeconds: 10
    periodSeconds: 30

  readinessProbe:
    httpGet:
      path: /
      port: http
    initialDelaySeconds: 5
    periodSeconds: 10

  env:
    NODE_ENV: "production"
    NEXT_TELEMETRY_DISABLED: "1"

  # Defaults to http://<release>-backend:<port>/api when empty
  apiUrl: ""
  # Optional browser-visible API URL. When empty, publicUrl supplies <origin>/api.
  publicApiUrl: ""

  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 10002
    runAsGroup: 10002
    fsGroup: 10002

  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: false # Next.js needs write access
    capabilities:
      drop:
        - ALL

  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAnnotations: {}
```

## Worker Configuration

The chart renders this deployment only when both `worker.enabled` is true and `coordinationBackend`
is `redis`. Local and auto coordination keep job execution in the API process.

```yaml
worker:
  enabled: true
  replicaCount: 1

  # Uses same image as backend
  # (worker is sibyl backend container with different entrypoint)

  autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 5
    targetCPUUtilizationPercentage: 70
    targetMemoryUtilizationPercentage: 80
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 300
        policies:
          - type: Percent
            value: 10
            periodSeconds: 60

  pdb:
    enabled: false
    minAvailable: 1

  podAntiAffinity:
    enabled: false
    type: soft
    topologyKey: kubernetes.io/hostname

  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 50m
      memory: 128Mi

  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 10001
    runAsGroup: 10001
    fsGroup: 10001
    fsGroupChangePolicy: OnRootMismatch

  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL

  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAnnotations: {}
```

## Ingress Configuration

Ingress is split into a shared route table (`ingress.hosts`) plus two independently toggled
renderers: classic `networking.k8s.io/v1` Ingress under `ingress.classic`, and Gateway API
`gateway.networking.k8s.io/v1` HTTPRoute under `ingress.gatewayApi`. Enable whichever your cluster
uses. The legacy `ingress.enabled` flag still works as a compatibility toggle for classic Ingress.

```yaml
ingress:
  # Shared route table consumed by both classic Ingress and Gateway API HTTPRoute.
  hosts:
    - host: sibyl.local
      paths:
        - path: /api
          pathType: Prefix
          service: backend
        - path: /mcp
          pathType: Prefix
          service: backend
        - path: /
          pathType: Prefix
          service: frontend
  classic:
    # Render a networking.k8s.io/v1 Ingress.
    enabled: false
    # Ingress class name. Empty leaves controller selection to cluster policy.
    className: ""
    annotations: {}
    tls: []
    # - secretName: sibyl-tls
    #   hosts:
    #     - sibyl.example.com
  gatewayApi:
    # Render a gateway.networking.k8s.io/v1 HTTPRoute.
    enabled: false
    annotations: {}
    parentRefs: []
    # - name: shared-gateway
    #   namespace: gateway-system
    #   sectionName: https
    # Optional hostnames override. Defaults to ingress.hosts[*].host.
    hostnames: []
```

## Network Policy

Renders a default-deny NetworkPolicy plus explicit allows for frontend/backend ingress and SurrealDB
(and optional Redis/Valkey) egress. Leave disabled unless your cluster enforces NetworkPolicies.

```yaml
networkPolicy:
  # Enable default-deny plus explicit app allows.
  enabled: false
  ingress:
    # Sources allowed to reach frontend/backend, usually ingress controller pods.
    from: []
    # - namespaceSelector:
    #     matchLabels:
    #       kubernetes.io/metadata.name: ingress-nginx
  egress:
    # Allow DNS egress for selected Sibyl pods.
    allowDns: true
    dnsPorts:
      - 53
    # Extra egress rules appended to backend/worker policies.
    extra: []
  surrealdb:
    # Destination selectors for the SurrealDB service/pods.
    to: []
    ports:
      - 8000
  redis:
    # Enable Redis/Valkey egress allow rules.
    enabled: false
    to: []
    ports:
      - 6379
```

## Pod Security

Labels the release namespace with Pod Security Admission enforce/audit/warn at the `restricted`
level. Enable for hardened multi-tenant clusters.

```yaml
podSecurity:
  # Label the release namespace with Pod Security restricted enforce/audit/warn.
  enforceRestricted: false
  version: "latest"
```

## Bootstrap

A post-install/post-upgrade Job that seeds the first organization and an optional default memory
space, so a fresh install lands ready to use. Disabled by default.

```yaml
bootstrap:
  enabled: false
  organization:
    name: "Sibyl"
    slug: ""
  memorySpace:
    enabled: true
    name: "Default memory"
    scope: "private"
    scopeKey: ""
  job:
    annotations: {}
    backoffLimit: 3
    ttlSecondsAfterFinished: 300
    podAnnotations: {}
```

## Break-Glass

Bounded emergency local-owner login for SSO outages. Keep disabled in normal operation; when
enabled, both `expiresAt` (no more than four hours out) and `allowedIPs` are required. The allowlist
is matched against the resolved client address, so behind an ingress set `backend.forwardedAllowIps`
(see [Trusted Proxies](#trusted-proxies)); without it the backend only ever sees the controller
pod's address.

```yaml
breakGlass:
  enabled: false
  # Source CIDRs allowed to use break-glass login when enabled.
  allowedIPs: []
  # UTC timestamp after which app-level break-glass login is denied.
  expiresAt: ""
  # Existing secret containing owner bootstrap fields.
  existingSecret: ""
  ownerEmailKey: "owner-email"
  ownerPasswordKey: "owner-password"
```

## Service Account

```yaml
serviceAccount:
  create: true
  name: ""
  annotations: {}
```

The backend, worker and bootstrap pods run as this service account, so an annotation here reaches
every pod that calls a model provider or the S3 receipt store.

The frontend only calls the Sibyl API, so it runs under its own account, `<fullname>-frontend`,
created with no annotations and no mounted API token:

```yaml
frontend:
  serviceAccount:
    create: true
    name: ""
    annotations: {}
    automountServiceAccountToken: false
```

To keep the earlier shared account, set `frontend.serviceAccount.create: false` and
`frontend.serviceAccount.name` to the backend account's name.

## Amazon Bedrock

On EKS, Sibyl can run Claude and Cohere embeddings through Amazon Bedrock with no provider API keys.
Bind an IAM role to the service account with IRSA, set a Region, and select the `bedrock` provider:

```yaml
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/sibyl-bedrock

backend:
  # Still required in production, but it only needs SIBYL_JWT_SECRET and SIBYL_SETTINGS_KEY.
  existingSecret: sibyl-secrets
  env:
    AWS_REGION: "us-west-2"
    SIBYL_LLM_PROVIDER: "bedrock"
    SIBYL_LLM_MODEL: "claude-haiku-5-5"
    SIBYL_LLM_MEMORY_MODEL: "claude-opus-5-5"
    SIBYL_BEDROCK_INFERENCE_SCOPE: "us"
    SIBYL_EMBEDDING_PROVIDER: "bedrock"
    SIBYL_EMBEDDING_MODEL: "cohere.embed-v4:0"
    SIBYL_EMBEDDING_DIMENSIONS: "1536"
    SIBYL_GRAPH_EMBEDDING_PROVIDER: "bedrock"
```

The chart renders `backend.env` into the ConfigMap the worker also loads, and the worker loads the
same `backend.existingSecret` into its environment and runs as the same service account, so a
separate worker gets these settings and the role with no extra values.

The IRSA webhook injects `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE` into each pod, and the AWS
credential chain exchanges that token for role credentials and refreshes them before they expire.
EKS Pod Identity works the same way through an association on the service account, with no
annotation. Both paths leave `SIBYL_ANTHROPIC_API_KEY` and `SIBYL_OPENAI_API_KEY` unset: the chart
references them from `backend.existingSecret` as optional keys, so the Secret does not need them.

The role needs `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` on the inference
profiles and foundation models it routes to, such as `us.anthropic.claude-*`, `anthropic.claude-*`
and `cohere.embed-v4:0`. See [Amazon Bedrock](./environment.md#amazon-bedrock) for every setting,
the inference scopes and the features Bedrock accepts per model. The admin AI settings page **Test**
button proves the role works from inside the pod.

With `networkPolicy.enabled`, add an `egress.extra` rule that allows HTTPS (443) to the Bedrock
runtime and STS endpoints, or through the proxy or VPC endpoints your cluster uses for AWS APIs.
Switching an embedding provider changes the vector space, so plan a re-embed before trusting mixed
results.

## Production Example

Complete production-ready values:

```yaml
global:
  imagePullSecrets:
    - name: ghcr-pull-secret

coordinationBackend: "redis"
publicUrl: "https://sibyl.example.com"

backend:
  replicaCount: 3
  image:
    repository: ghcr.io/hyperb1iss/sibyl-api
    tag: "1.4.10"
    pullPolicy: Always
  existingSecret: sibyl-secrets
  # Only the ingress controller pods' own range, so each user gets their own login
  # rate-limit bucket. Never the pod CIDR (see Trusted Proxies).
  forwardedAllowIps: "10.250.0.0/28"
  validationReceipts:
    existingClaim: sibyl-validation-receipts
    # Or keep receipts in S3 and leave existingClaim empty:
    # url: "s3://sibyl-receipts/prod?region=us-west-2"
  # Archives default to the receipts claim; with receipts in S3, name a
  # separate prefix (or a claim) for them:
  # backupArchives:
  #   url: "s3://sibyl-backups/prod?region=us-west-2"
  surreal:
    url: "ws://prod-surrealdb.internal:8000/rpc"
    username: "root"
    existingSecret: sibyl-surreal
    namespacePrefix: "org_"
    database: "graph"
  redis:
    host: "prod-valkey.internal"
    port: "6379"
    existingSecret: sibyl-redis
  env:
    SIBYL_ENVIRONMENT: "production"
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 20
  pdb:
    enabled: true
    minAvailable: 2
  podAntiAffinity:
    enabled: true
    type: hard
  resources:
    limits:
      cpu: 4000m
      memory: 4Gi
    requests:
      cpu: 1000m
      memory: 1Gi

frontend:
  enabled: true
  replicaCount: 2
  apiUrl: "http://sibyl-backend:3334/api"
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
  pdb:
    enabled: true
  podAntiAffinity:
    enabled: true

worker:
  enabled: true
  replicaCount: 2
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 8
  pdb:
    enabled: true
  podAntiAffinity:
    enabled: true

ingress:
  hosts:
    - host: sibyl.example.com
      paths:
        - path: /api
          pathType: Prefix
          service: backend
        - path: /mcp
          pathType: Prefix
          service: backend
        - path: /
          pathType: Prefix
          service: frontend
  classic:
    enabled: true
    className: "nginx"
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
      nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    tls:
      - secretName: sibyl-tls
        hosts:
          - sibyl.example.com

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/sibyl
```

## Chart Templates

The chart includes these templates:

| Template                 | Purpose                                     |
| ------------------------ | ------------------------------------------- |
| backend-deployment.yaml  | Backend Deployment                          |
| backend-service.yaml     | Backend ClusterIP Service                   |
| backend-hpa.yaml         | Backend HorizontalPodAutoscaler             |
| frontend-deployment.yaml | Frontend Deployment                         |
| frontend-service.yaml    | Frontend ClusterIP Service                  |
| frontend-hpa.yaml        | Frontend HorizontalPodAutoscaler            |
| worker-deployment.yaml   | Worker Deployment                           |
| worker-hpa.yaml          | Worker HorizontalPodAutoscaler              |
| pdb.yaml                 | PodDisruptionBudgets                        |
| configmap.yaml           | Non-secret environment config               |
| surreal-secret.yaml      | Auto-generated Surreal secret (default)     |
| redis-secret.yaml        | Auto-generated Redis/Valkey secret          |
| bootstrap-job.yaml       | Post-install tenant bootstrap Job           |
| ingress.yaml             | Classic networking.k8s.io/v1 Ingress        |
| httproute.yaml           | Gateway API HTTPRoute                       |
| networkpolicy.yaml       | Default-deny plus app-allow NetworkPolicies |
| podsecurity.yaml         | Namespace Pod Security enforcement labels   |
| serviceaccount.yaml      | ServiceAccount                              |

## Debugging

```bash
# Render templates locally
helm template sibyl ./charts/sibyl -f values.yaml

# Debug with notes
helm install sibyl ./charts/sibyl -f values.yaml --debug --dry-run

# Get release values
helm get values sibyl -n sibyl

# Get all manifests
helm get manifest sibyl -n sibyl
```

## Durable Validation Receipts

Validation writes each completed model result as an encrypted receipt before it commits the result
to the database, so a database outage does not force Sibyl to pay for the same validation twice. Set
exactly one of `backend.validationReceipts.url` (Amazon S3) or
`backend.validationReceipts.existingClaim` (a shared volume) before installing or upgrading. The
chart refuses to render with neither or both.

### Amazon S3

```yaml
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/sibyl

backend:
  validationReceipts:
    url: "s3://<bucket>/sibyl/receipts?region=us-west-2"
```

The URL names a bucket, an optional key prefix, and an optional `region` query parameter (`auto` is
accepted for endpoints such as Cloudflare R2). The chart checks the URL's shape at render time (a
3-63 character bucket of lowercase letters, digits, dots and hyphens, no empty or dot segments in
the prefix, and no query other than `region`), so a typo fails the install rather than crashlooping
the pods. Without a region, the standard `AWS_REGION` and `AWS_DEFAULT_REGION` resolution applies
(the EKS IRSA webhook sets `AWS_REGION`). Credentials come from the default AWS chain: IRSA, EKS Pod
Identity, or an instance role, as described under [Amazon Bedrock](#amazon-bedrock). The chart
passes the URL to API and worker as `SIBYL_VALIDATION_RECEIPT_URL` through its ConfigMap and mounts
no receipts volume, so pods may run on any node and single-replica rollouts use the Kubernetes
`RollingUpdate` default. The frontend runs under its own service account (see
[Service Account](#service-account)), so the receipts role never reaches it. Give receipts a
dedicated prefix, since backup export lists it.

Grant the pod role access to the prefix:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<bucket>/sibyl/receipts/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::<bucket>"
    }
  ]
}
```

`s3:ListBucket` makes a missing receipt answer 404 instead of 403. Sibyl must tell "absent" from
"denied", and refuses to dispatch without it. Backup export also uses it to list the prefix once
instead of reading every execution's receipt. If the bucket encrypts with a customer managed KMS
key, also grant `kms:GenerateDataKey` and `kms:Decrypt` on that key. Sibyl sends no
`x-amz-server-side-encryption` header, so a bucket policy that denies uploads without one refuses
every receipt and the readiness check stops validation; use the bucket's default encryption instead.

The volume guarantees carry over:

* **Encrypted by Sibyl.** Each receipt is Fernet ciphertext under a per-execution key held in the
  content database, so the bucket never sees a result in the clear. Bucket encryption (SSE-S3 or
  SSE-KMS) is welcome but not relied on. Purging a source erases its key, which leaves any copy of
  the ciphertext unreadable.
* **Create-only.** Every write is a `PutObject` with `If-None-Match: *`. A `412 Precondition Failed`
  answer means the receipt already exists: Sibyl compares the stored bytes and refuses any
  difference, so neither validation nor backup restore ever replaces a receipt. A `409` from a
  racing delete is retried, as S3 documents.
* **Checked when each validation execution begins.** The readiness check runs once per execution,
  before its first model call, not before every call. It deletes a never-written key (proving
  `s3:DeleteObject` without creating anything), confirms a second never-written key reads as absent,
  then creates a probe object under the prefix, confirms a second create is refused, reads it back,
  finds it in a listing, deletes it, and confirms it reads as absent. The listing check matters
  because backup export finds receipts by listing the prefix, so a store whose listings lag its
  writes would record a pending receipt as missing. A bucket that fails any step stops the execution
  before a provider is paid, including an S3-compatible store that ignores `If-None-Match`.
* **Unsigned reads refused, best effort.** The check also asks for the probe without credentials and
  stops validation if that read succeeds. A timeout, redirect, or any other non-success answer
  counts as refused, so this catches a public bucket but is not a full audit of bucket policy.

Each request has a 3 s connect and 10 s read timeout with up to three attempts, so an endpoint that
accepts connections and never answers fails the check in about half a minute instead of holding a
worker thread for minutes.

Keep S3 Block Public Access on (the default for new buckets), and do not grant `Principal: "*"` read
access in the bucket policy, even behind a VPC endpoint condition. Versioning is not required. With
`networkPolicy.enabled`, allow HTTPS egress to S3 and to STS (IRSA exchanges its token there), or to
their VPC endpoints, through `networkPolicy.egress.extra`. For an S3-compatible endpoint, set
`backend.env.AWS_ENDPOINT_URL_S3`; the readiness check verifies the store honors `If-None-Match`.

A successful readiness check leaves no object behind. A probe can still be stranded when a pod dies
mid-check or the endpoint stops answering, and on a versioned bucket every check leaves a noncurrent
probe version and delete markers, as discarded receipts do. Add a lifecycle rule for the prefix:

```json
{
  "Rules": [
    {
      "ID": "sibyl-receipt-history",
      "Status": "Enabled",
      "Filter": { "Prefix": "sibyl/receipts/" },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 1 },
      "Expiration": { "ExpiredObjectDeleteMarker": true }
    },
    {
      "ID": "sibyl-receipt-probes",
      "Status": "Enabled",
      "Filter": { "Prefix": "sibyl/receipts/.probe-" },
      "Expiration": { "Days": 1 }
    }
  ]
}
```

A noncurrent receipt version is never needed: a receipt is only removed after the database holds its
result.

To move an existing release from a claim to S3, stop validation work (scale the worker and backend
to zero), then copy any remaining receipts from the claim's `private` directory into the prefix
under the same names. Run the copy from a temporary pod that mounts the claim at
`/var/lib/sibyl-receipts` and has the pod role's credentials and an AWS CLI whose `s3api put-object`
accepts `--if-none-match`. Use that create-only upload so a receipt already in the bucket is never
replaced:

```bash
for receipt in /var/lib/sibyl-receipts/private/*.receipt; do
  aws s3api put-object --bucket <bucket> --key "sibyl/receipts/$(basename "$receipt")" \
    --body "$receipt" --if-none-match '*'
done
```

A `PreconditionFailed` answer means that receipt is already there. Then upgrade with `url` set and
`existingClaim` cleared. A content backup taken before the switch also carries pending receipts, and
restoring it publishes them into the bucket.

### Shared Volume

Provision a persistent claim and set `backend.validationReceipts.existingClaim`. API and worker
mount the same claim. Multi-node replicas require a ReadWriteMany volume; provision it with the
storage class supported by your cluster. The service user (UID/GID 10001) must be able to create a
private child directory. No temporary volume or single-replica fallback is used.

Two chart defaults keep a ReadWriteOnce block volume (EBS, Persistent Disk, Azure Disk) working
across restarts and rollouts:

* `fsGroupChangePolicy: OnRootMismatch` on the backend and worker pod security contexts. Block CSI
  drivers re-apply `fsGroup` on every mount by default, adding group permission bits to every file
  and directory, and Sibyl refuses a receipts directory or file with any group or other bits. Keep
  this key if you override `podSecurityContext`.
* An empty `backend.strategy` or `worker.strategy` rolls a single fixed replica with `maxSurge: 0`
  and `maxUnavailable: 1`, because a surge pod scheduled onto another node cannot attach a
  ReadWriteOnce claim. The old pod stops before its replacement starts; if the replacement lands on
  another node, it waits for the volume to detach and then starts on its own. With more replicas or
  autoscaling the Kubernetes `RollingUpdate` default applies, which assumes ReadWriteMany storage.
  An explicit strategy passes through unchanged.

The single-replica default stays `type: RollingUpdate` rather than `Recreate` on purpose. The API
server fills in `rollingUpdate` on every existing Deployment, no applier owns that field, and
server-side apply (the Helm 4 default, Argo CD with `ServerSideApply=true`, Flux) can never remove
it, so an upgrade that switches `type` to `Recreate` is rejected with
`spec.strategy.rollingUpdate: Forbidden`. To opt into `Recreate` on an existing release, remove the
field once first:

```bash
kubectl -n sibyl patch deploy sibyl-backend --type=json \
  -p '[{"op":"remove","path":"/spec/strategy/rollingUpdate"},{"op":"replace","path":"/spec/strategy/type","value":"Recreate"}]'
```

### Downgrades

Content schema 40 retains a per-execution receipt key and erases it when a source is purged. Older
servers cannot recover these pending receipts, and servers older than the S3 store cannot read a
bucket at all. Schema repair must not replace the key-erasing purge event with the historical
definition. Recover pending validation receipts with the current server before a planned downgrade,
retaining the receipt volume or bucket and a coherent database backup. Removing the receipt store
loses outage recovery for pending results; replay still refuses to repeat an unresolved provider
dispatch.

## Backup Archives

Organization backups (web settings, `/api/backups`, and the scheduled per-organization backups) are
written as one `.tar.gz` archive per backup by whichever process runs the backup job: the worker
under Redis coordination, or the API pod that took the job under local coordination. Any API replica
then lists, downloads and deletes it, so every backend and worker pod must open the same archive
store. The image's working directory is read-only under the chart and only per-pod storage could be
written there, so the chart never leaves archives in the container.

Where archives live:

| Values                                 | Archive store                                           |
| -------------------------------------- | ------------------------------------------------------- |
| neither set (default)                  | `/var/lib/sibyl-receipts/backups` on the receipts claim |
| `backend.backupArchives.existingClaim` | `/var/lib/sibyl-backups/archives` on that claim         |
| `backend.backupArchives.url`           | the S3 prefix, through `SIBYL_BACKUP_ARCHIVE_URL`       |

Set at most one of `url` and `existingClaim`. With receipts in S3 there is no claim to share, so the
chart refuses to render until one of them is set. The chart sets `SIBYL_BACKUP_DIR` or
`SIBYL_BACKUP_ARCHIVE_URL` on API and worker alike, and refuses either variable in `backend.env`.

A claim is a shared directory only when every pod mounts it: a ReadWriteOnce claim works while API
and worker run on one node (the single-replica default, which rolls with `maxSurge: 0` for an
archives claim exactly as for the receipts claim), and multi-node replicas need ReadWriteMany. Each
archive is assembled in a hidden staging directory on the same volume and published with a hard
link, so a listing never shows a half-written archive and a published archive is never replaced. The
volume's filesystem must support hard links; each backup proves it with a probe before it exports
anything, and fails with that reason on a filesystem that cannot. A new archive directory is created
private (`0700`) and each archive `0600`. The scheduled cleanup removes staging directories left by
a killed job once they are a day old, since they hold the export in plaintext.

### Sizing a claim

The receipts claim default suits a single replica and a small install: it keeps archives durable
with no extra values, but they then share one volume with validation receipts. Receipts are small,
but a receipt that cannot be written stops validation, so a full claim is an outage. At scale, give
archives a dedicated claim (`backend.backupArchives.existingClaim`) or S3.

Size an archives volume, or the receipts claim when it holds archives, for:

* the archives kept: organizations backed up × backups per day × `SIBYL_BACKUP_RETENTION_DAYS` ×
  archive size;
* staging for the largest running backup: its export is written uncompressed on the same volume
  before it is compressed, typically five to ten times the archive's size, plus the archive itself;
* a reserve for everything else on the volume, `SIBYL_BACKUP_FREE_SPACE_RESERVE_MB` (default 512;
  set it with `backend.env`).

Before each backup on a directory store, Sibyl claims room for it: eleven times that organization's
last archive, checked against the volume's free space less the reserve and less what other running
backups have claimed and not yet written. Before writing the archive it claims again, at the staged
export's real size. A backup that would eat into the reserve fails with the free space, its
estimate, the other backups' claims and the reserve in its error, before it writes anything more, so
the reserve stays free for receipts.

Each claim is a hidden `.reserve-<backup id>` file beside the archives, so backups on different pods
sharing the claim see each other; nothing is serialized. A backup writes its claim before it checks,
so two backups starting at the same moment each count the other: neither can overcommit the volume,
and at worst both are refused and the next scheduled run goes ahead. A claim untouched for a day
belongs to a backup that died; it stops counting and the scheduled cleanup removes it. A claim that
fails its check, for a full volume or an I/O error alike, is removed at once. Anything named like a
claim that is not a small regular file (a link, a FIFO, an oversized file) is ignored and logged
once rather than read.

That promise holds only where every pod sees a new file in the directory as soon as it is written.
NFS-backed shared volumes, such as Amazon EFS, Google Filestore and most ReadWriteMany NFS
provisioners, cache directory listings on each node for about 30 to 60 seconds. Backups on different
nodes that start within that window can miss each other's claims. At worst they behave as if there
were no claims: each still checks its own estimate, and its real staged size before writing the
archive, but not the others'. On such volumes, keep headroom for concurrent backups, or use S3 for
archives.

The estimate has two known gaps:

* **An organization's first backup** has no earlier archive to estimate from, so before its export
  only the reserve and other backups' claims are checked. Its staged size is checked before the
  archive is written, but the export itself can use space the reserve was meant to keep. Leave
  headroom beyond the reserve for the largest organization's first backup, or take it while the
  volume is roomy.
* **The factor of eleven** assumes the export is five to ten times its gzip archive, typical for
  JSON. For a large organization the claim is then large too, and an export that compresses less
  than that is over-estimated: it can be refused on a volume that would have held it. Free more
  space or lower the reserve rather than waiting for it to fit. An export that compresses better is
  under-estimated before it runs, and the second check, at its real staged size, still guards the
  archive.

### Retried jobs

Under Redis coordination a backup job that was cancelled, or whose worker died, runs again. A backup
already recorded as completed is reported as such and not rebuilt. When an earlier attempt published
the archive but died before recording it, the retry adopts that archive instead of exporting again,
once it has matched the digest recorded at publish time (S3), passed the same inventory, checksum
and organization checks a restore applies, and so proved to belong to this organization. Any other
archive under the backup's name fails the backup and is left untouched.

### Archives in Amazon S3

```yaml
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/sibyl

backend:
  validationReceipts:
    url: "s3://<bucket>/sibyl/receipts?region=us-west-2"
  backupArchives:
    url: "s3://<bucket>/sibyl/backups?region=us-west-2"
```

The URL has the same shape and render-time checks as `backend.validationReceipts.url`, and uses the
same client: credentials from the pod's AWS identity, `AWS_REGION` when the URL names no region,
`backend.env.AWS_ENDPOINT_URL_S3` for an S3-compatible endpoint, and the same timeouts and retries.
Archives and receipts may share a bucket but not a prefix, and neither prefix may contain the other;
the chart and the server both refuse an overlap, because receipt export and archive retention each
list their own prefix.

Grant the pod role access to the archives prefix, alongside the receipts statement:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::<bucket>/sibyl/backups/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::<bucket>"
    }
  ]
}
```

`s3:ListBucket` serves retention, which lists the prefix to find archives past
`SIBYL_BACKUP_RETENTION_DAYS`, and lets a missing archive answer 404 instead of 403; each backup
also asks whether an earlier attempt already published its archive, which needs that answer.
`s3:AbortMultipartUpload` lets a failed upload clean up after itself. Downloads are streamed from
the bucket through the API in 1 MiB chunks, so no replica holds a whole archive in memory, and the
browser never needs bucket access. A single `Range: bytes=` request gets `206 Partial Content` for
that part, from either store, so an interrupted download resumes.

How archives keep their guarantees in S3:

* **Written once.** An archive up to 16 MiB is one `PutObject`, and a larger one a multipart upload
  in 16 MiB parts; both carry `If-None-Match: *`. Each object records its SHA-256 in user metadata,
  so a `412 Precondition Failed` is resolved by comparing digests: the same digest is a retried
  write that already landed, a different one fails the backup and leaves the existing archive in
  place. A `409` from a racing request retries the whole upload, as S3 documents.
* **Never half visible.** S3 lists and serves a multipart upload only after
  `CompleteMultipartUpload` succeeds. A failed upload is aborted while the endpoint still answers.
* **Checked as each backup begins.** Before any organization data is exported, the job creates a
  probe object under the prefix, confirms a second create is refused, confirms an unsigned read of
  it fails, and deletes it. An S3-compatible store that ignores `If-None-Match`, or a bucket that
  serves that probe to anonymous requests, fails the backup with that reason. The unsigned read is
  best effort, as for receipts: it asks only for the probe key, and a timeout or any other
  non-success answer counts as refused, so it catches a public bucket but is not an audit of the
  bucket policy.
* **Staged on the pod.** An S3 backup assembles its export and archive in the pod's `/tmp`, the
  chart's `tmp` emptyDir, before uploading, which needs about eleven times the largest archive. Set
  `ephemeral-storage` requests and limits on the worker (and on the backend under local
  coordination) to match.

**Archives hold secrets in the clear.** Unlike receipts, an archive is not encrypted by Sibyl: it
carries the organization's memory, and its content payload carries the recovery keys that decrypt
its validation receipts. Keep the bucket's default encryption on (SSE-S3, or SSE-KMS with
`kms:GenerateDataKey` and `kms:Decrypt` granted on the key; Sibyl sends no
`x-amz-server-side-encryption` header, so rely on bucket default encryption rather than a policy
that denies uploads without one), keep S3 Block Public Access on, grant read access to the prefix
only to the Sibyl role and the people who restore backups, and consider a separate bucket or KMS key
from receipts so read access to one does not imply the other. Sibyl logs archive names, sizes and
locations, never their contents.

Add a lifecycle rule for the prefix so an upload stranded by a dead pod or endpoint, or a readiness
probe, does not linger:

```json
{
  "Rules": [
    {
      "ID": "sibyl-backup-uploads",
      "Status": "Enabled",
      "Filter": { "Prefix": "sibyl/backups/" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 1 },
      "Expiration": { "ExpiredObjectDeleteMarker": true }
    },
    {
      "ID": "sibyl-backup-probes",
      "Status": "Enabled",
      "Filter": { "Prefix": "sibyl/backups/.probe-" },
      "Expiration": { "Days": 1 }
    }
  ]
}
```

Retention stays Sibyl's: the scheduled cleanup removes archives older than
`SIBYL_BACKUP_RETENTION_DAYS` along with their backup records, and an organization admin's cleanup
removes only that organization's expired archives. Do not add an expiration rule for archives
themselves, or the web settings will list backups whose archive is gone.

To move existing archives from a claim into the bucket, copy them under the same names with a
create-only upload from a pod that mounts the claim (archives live in
`/var/lib/sibyl-receipts/backups` on the default layout), then upgrade with `backupArchives.url`
set:

```bash
for archive in /var/lib/sibyl-receipts/backups/sibyl_*.tar.gz; do
  aws s3api put-object --bucket <bucket> --key "sibyl/backups/$(basename "$archive")" \
    --body "$archive" --if-none-match '*' \
    --metadata "sha256=$(sha256sum "$archive" | cut -d' ' -f1)"
done
```

### Upgrading

* **Receipts already on a claim.** Nothing to change: archives start landing in `backups` beside the
  receipts on that claim. Backups could not be written under earlier charts, so expect the claim to
  start filling once they succeed, and size it as above or move archives off it.
* **Receipts in S3.** `helm upgrade` refuses to render until `backend.backupArchives.url` or
  `backend.backupArchives.existingClaim` is set.
* **Moving receipts to S3 and keeping their old claim for archives.** Archives live in `backups` on
  the receipts claim but in `archives` on an archives claim. When `backupArchives.existingClaim`
  takes over the old receipts claim, move `/var/lib/sibyl-backups/backups/*` to
  `/var/lib/sibyl-backups/archives/` from a pod that mounts it, or existing backups stay listed but
  answer 404 on download.
* **Moving archives to S3.** Use the create-only copy above, then set `backupArchives.url`.