Troubleshooting Guide ​

Common issues and solutions for Sibyl deployments.

Connection Issues ​

Cannot Connect to API ​

Symptoms:

  • Connection refused on port 3334
  • Frontend shows "Cannot connect to server"

Solutions:

  1. Check if server is running:

    bash
    # Docker Compose
docker compose --env-file ~/.sibyl/prod.env ps

# Kubernetes
kubectl get pods -n sibyl

  2. Check server logs:

    bash
    # Docker Compose
docker compose --env-file ~/.sibyl/prod.env logs backend

# Kubernetes
kubectl logs -n sibyl -l app.kubernetes.io/component=backend

  3. Verify port binding:

    bash
    # Check what's listening
lsof -i :3334
netstat -tlnp | grep 3334

  4. Check firewall rules (if applicable)

Database Connection Errors ​

SurrealDB Connection Refused:

surrealdb error: connection refused

Solutions:

  1. Verify SurrealDB is running:

    bash
    docker compose --env-file ~/.sibyl/prod.env ps surrealdb
kubectl get pods -n sibyl -l app=surrealdb

  2. Check connection settings:

    bash
    # Environment variables
echo $SIBYL_SURREAL_URL

  3. Test direct connection:

    bash
    curl http://localhost:8000/health

Authentication Issues ​

Forbidden After Login ​

Symptoms:

  • API response includes "error": "forbidden".
  • Response body says "Invalid request data." with remediation to check organization or project permissions.
  • Login or refresh succeeds, but the next project, search, admin, or MCP request fails.

What it usually means:

The client has a valid Sibyl session, but the request does not match the user's organization, project, memory-space, or admin permissions. This is different from an OIDC provider problem. In the default install, OIDC is off and local auth is expected.

Solutions:

  1. Confirm the intended auth mode:

    bash
    echo $SIBYL_LOCAL_AUTH_ENABLED
echo $SIBYL_PUBLIC_SIGNUPS_ENABLED
echo $SIBYL_OIDC

    Default local installs should have local auth enabled, public signups disabled, and no OIDC providers.

  2. For a new default install, complete setup first.

    The first setup signup creates the owner/admin user. After setup, additional users need an invite unless SIBYL_PUBLIC_SIGNUPS_ENABLED=true.

  3. Check the active CLI context and token owner:

    bash
    sibyl config context
sibyl auth status
sibyl whoami

  4. For project-scoped failures, switch to a project the user can access or ask an owner/admin to add the membership.

  5. For enterprise SSO, verify the IdP role claim emits Sibyl.Member, Sibyl.Admin, or Sibyl.Owner.

JWT Token Invalid ​

Symptoms:

  • 401 Unauthorized responses
  • "Invalid token" errors

Solutions:

  1. Check JWT secret is set:

    bash
    # Should be non-empty
echo $SIBYL_JWT_SECRET

  2. Verify token hasn't expired:

    • Default access token TTL: 60 minutes
    • Use refresh token to get new access token

  3. Check clock synchronization:

    • JWT validation requires synchronized clocks
    • Server and client should have correct time

GitHub OAuth Failing ​

Symptoms:

  • "OAuth error" after GitHub redirect
  • Missing callback URL

Solutions:

  1. Verify OAuth credentials:

    bash
    echo $SIBYL_GITHUB_CLIENT_ID
echo $SIBYL_GITHUB_CLIENT_SECRET

  2. Check callback URL in GitHub:

    • Must match SIBYL_PUBLIC_URL/api/auth/github/callback
    • Example: https://sibyl.local/api/auth/github/callback

  3. Verify public URL:

    bash
    echo $SIBYL_PUBLIC_URL

Schema Issues ​

"table ... does not exist" on a Fresh Store ​

Symptoms:

  • API requests return HTTP 500 internal_error with a generic message.
  • Server logs show NotFoundError: table projects does not exist (or team_projects, raw_captures, another core table) against a newly provisioned SurrealDB.
  • The instance was just installed, restored onto an empty database, or pointed at a new SIBYL_SURREAL_URL.

What it usually means:

The auth and content schemas were never applied to this store. The server applies both automatically at startup and refuses to start on failure outside development, so hitting this at request time means the server started in development mode against an empty store, or startup bootstrap failed and the log line was missed.

Solutions:

  1. Apply the schemas directly (safe to run repeatedly):

    bash
    sibyld db init

    The command applies pending auth and content migrations, verifies schema invariants, never drops tables, and skips migrations already recorded in schema_version.

  2. Check the startup bootstrap log for the failure that stopped the automatic path:

    bash
    sibyl logs tail -l error -s api

    Look for Surreal auth schema bootstrap failed or Surreal content schema bootstrap failed; each names the plane, target version, and error.

  3. Verify readiness before sending traffic. The readiness endpoint reports schema bootstrap state, and compose/Kubernetes health checks gate on it, so an unhealthy backend that keeps restarting is usually this failure loudly refusing to serve.

Performance Issues ​

Slow Queries ​

Symptoms:

  • API responses taking > 5 seconds
  • Timeouts on graph operations

Solutions:

  1. Check graph size:

    bash
    curl -H "Authorization: Bearer $TOKEN" \
  https://sibyl.local/api/admin/stats

  2. Limit query results:

    • Use limit parameter on list endpoints
    • Paginate large result sets

High Memory Usage ​

Symptoms:

  • OOMKilled pods in Kubernetes
  • Container restarts

Solutions:

  1. Increase resource limits:

    yaml
    backend:
  resources:
    limits:
      memory: 2Gi
    requests:
      memory: 512Mi

  2. Check for memory leaks:

    bash
    kubectl top pods -n sibyl

  3. Reduce worker or API concurrency for memory-constrained pods.

Worker Queue Backlog ​

Symptoms:

  • Jobs not completing
  • Crawl tasks stuck

Solutions:

  1. Check worker status:

    bash
    kubectl logs -n sibyl -l app.kubernetes.io/component=worker -f

  2. Scale workers:

    bash
    kubectl scale deployment sibyl-worker -n sibyl --replicas=3

  3. Check Redis when the opt-in coordination backend is enabled:

    bash
    redis-cli -h localhost -p 6381 -n 1 KEYS "*"

Port Conflicts ​

Local Data-Service Port Conflict ​

Error: SurrealDB or the optional Redis coordination profile cannot bind its port

Solutions:

  1. Find conflicting process:

    bash
    lsof -i :8000
lsof -i :6381

  2. Change port in compose:

    Set SIBYL_SURREAL_PORT or SIBYL_REDIS_PORT before starting Docker Compose.

  3. Update environment:

    bash
    SIBYL_SURREAL_URL=ws://127.0.0.1:<surreal-port>/rpc

Kubernetes-Specific Issues ​

Pods Stuck in Pending ​

Solutions:

  1. Check node resources:

    bash
    kubectl describe nodes
kubectl top nodes

  2. Check resource requests:

    bash
    kubectl describe pod <pod-name> -n sibyl

  3. Reduce resource requests:

    yaml
    resources:
  requests:
    cpu: 50m # Reduce from 100m
    memory: 128Mi # Reduce from 256Mi

Pods CrashLoopBackOff ​

Solutions:

  1. Check logs:

    bash
    kubectl logs <pod-name> -n sibyl --previous

  2. Check events:

    bash
    kubectl get events -n sibyl --sort-by='.lastTimestamp'

  3. Common causes:

    • Missing secrets
    • Database not ready
    • Port already bound

Historical Archive Rehearsal Database Unavailable ​

Solutions:

  1. Confirm the external rehearsal database is reachable:

    bash
    psql "postgresql://$SIBYL_POSTGRES_USER:$SIBYL_POSTGRES_PASSWORD@$SIBYL_POSTGRES_HOST:$SIBYL_POSTGRES_PORT/$SIBYL_POSTGRES_DB" -c 'select 1'

  2. Confirm the retained archive contains the database dump sidecar:

    bash
    tar -tzf migration-archive.tar.gz | grep postgres.sql

  3. Run restore only in explicit rehearsal mode with --restore-database-dump --source-type legacy-archive --target-mode postgres-rehearsal.

Kong Gateway Issues ​

Solutions:

  1. Check Kong operator:

    bash
    kubectl get pods -n kong-system

  2. Check gateway:

    bash
    kubectl get gateway -n kong
kubectl describe gateway sibyl-gateway -n kong

  3. Check dataplane:

    bash
    kubectl get pods -n kong
kubectl logs -n kong -l app=dataplane-sibyl-gateway

  4. Check HTTPRoutes:

    bash
    kubectl get httproute -n sibyl
kubectl describe httproute sibyl-api -n sibyl

Tilt-Specific Issues ​

Tilt Stuck on Resource ​

Solutions:

  1. Check Tilt logs:

    • Click on stuck resource in Tilt UI
    • Look for error messages

  2. Trigger manual rebuild:

    bash
    tilt trigger <resource-name>

  3. Full reset:

    bash
    tilt down
# If using Minikube:
minikube delete
minikube start --cpus=4 --memory=8192
# If using OrbStack: restart via OrbStack app or `orb restart`)
tilt up

Can't Access sibyl.local ​

Solutions:

  1. Check /etc/hosts:

    bash
    grep sibyl.local /etc/hosts
# Should show: 127.0.0.1 sibyl.local

  2. Check Caddy is running:

    • Look for caddy-proxy in Tilt UI

  3. Check Kong port-forward:

    • Look for kong-port-forward in Tilt UI

  4. Trust Caddy CA (macOS):

    bash
    sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain \
  ~/.local/share/caddy/pki/authorities/local/root.crt

Getting Help ​

If you're still stuck:

  1. Check logs thoroughly:

    bash
    # All logs with timestamps
kubectl logs -n sibyl --all-containers --timestamps -l app.kubernetes.io/name=sibyl

  2. Describe resources:

    bash
    kubectl describe pod <pod-name> -n sibyl
kubectl describe deployment <deploy-name> -n sibyl

  3. Check events:

    bash
    kubectl get events -n sibyl --sort-by='.lastTimestamp' | tail -50

  4. File an issue:

    • Include logs and error messages
    • Describe reproduction steps
    • Include environment details (OS, K8s version, etc.)

Released under the Apache-2.0 License.

Copyright © 2024-2026 Stefanie Jane