Troubleshooting Guide

Common issues and solutions for Sibyl deployments.

Connection Issues

Cannot Connect to API

Symptoms:

• Connection refused on port 3334
• Frontend shows "Cannot connect to server"

Solutions:

1. Check if server is running:

   # Docker Compose
   docker compose --env-file ~/.sibyl/prod.env ps
   
   # Kubernetes
   kubectl get pods -n sibyl

2. Check server logs:

   # Docker Compose
   docker compose --env-file ~/.sibyl/prod.env logs backend
   
   # Kubernetes
   kubectl logs -n sibyl -l app.kubernetes.io/component=backend

3. Verify port binding:

   # Check what's listening
   lsof -i :3334
   netstat -tlnp | grep 3334

4. Check firewall rules (if applicable)

Database Connection Errors

SurrealDB Connection Refused:

surrealdb error: connection refused

Solutions:

1. Verify SurrealDB is running:

   docker compose --env-file ~/.sibyl/prod.env ps surrealdb
   kubectl get pods -n sibyl -l app=surrealdb

2. Check connection settings:

   # Environment variables
   echo $SIBYL_SURREAL_URL

3. Test direct connection:

   curl http://localhost:8000/health

Authentication Issues

Forbidden After Login

Symptoms:

• API response includes "error": "forbidden".
• Response body says "Invalid request data." with remediation to check organization or project permissions.
• Login or refresh succeeds, but the next project, search, admin, or MCP request fails.

What it usually means:

The client has a valid Sibyl session, but the request does not match the user's organization, project, memory-space, or admin permissions. This is different from an OIDC provider problem. In the default install, OIDC is off and local auth is expected.

Solutions:

1. Confirm the intended auth mode:

   echo $SIBYL_LOCAL_AUTH_ENABLED
   echo $SIBYL_PUBLIC_SIGNUPS_ENABLED
   echo $SIBYL_OIDC

   Default local installs should have local auth enabled, public signups disabled, and no OIDC providers.

2. For a new default install, complete setup first.

   The first setup signup creates the owner/admin user. After setup, additional users need an invite unless SIBYL_PUBLIC_SIGNUPS_ENABLED=true.

3. Check the active CLI context and token owner:

   sibyl config context
   sibyl auth status
   sibyl whoami

4. For project-scoped failures, switch to a project the user can access or ask an owner/admin to add the membership.

5. For enterprise SSO, verify the IdP role claim emits Sibyl.Member, Sibyl.Admin, or Sibyl.Owner.

JWT Token Invalid

Symptoms:

• 401 Unauthorized responses
• "Invalid token" errors

Solutions:

1. Check JWT secret is set:

   # Should be non-empty
   echo $SIBYL_JWT_SECRET

2. Verify token hasn't expired:

   • Default access token TTL: 60 minutes
   • Use refresh token to get new access token

3. Check clock synchronization:

   • JWT validation requires synchronized clocks
   • Server and client should have correct time

GitHub OAuth Failing

Symptoms:

• "OAuth error" after GitHub redirect
• Missing callback URL

Solutions:

1. Verify OAuth credentials:

   echo $SIBYL_GITHUB_CLIENT_ID
   echo $SIBYL_GITHUB_CLIENT_SECRET

2. Check callback URL in GitHub:

   • Must match SIBYL_PUBLIC_URL/api/auth/github/callback
   • Example: https://sibyl.local/api/auth/github/callback

3. Verify public URL:

   echo $SIBYL_PUBLIC_URL

Schema Issues

"table ... does not exist" on a Fresh Store

Symptoms:

• API requests return HTTP 500 internal_error with a generic message.
• Server logs show NotFoundError: table projects does not exist (or team_projects, raw_captures, another core table) against a newly provisioned SurrealDB.
• The instance was just installed, restored onto an empty database, or pointed at a new SIBYL_SURREAL_URL.

What it usually means:

The auth and content schemas were never applied to this store. The server applies both automatically at startup and refuses to start on failure outside development, so hitting this at request time means the server started in development mode against an empty store, or startup bootstrap failed and the log line was missed.

Solutions:

1. Apply the schemas directly (safe to run repeatedly):

   sibyld db init

   The command applies pending auth and content migrations, verifies schema invariants, never drops tables, and skips migrations already recorded in schema_version.

2. Check the startup bootstrap log for the failure that stopped the automatic path:

   sibyl logs tail -l error -s api

   Look for Surreal auth schema bootstrap failed or Surreal content schema bootstrap failed; each names the plane, target version, and error.

3. Verify readiness before sending traffic. The readiness endpoint reports schema bootstrap state, and compose/Kubernetes health checks gate on it, so an unhealthy backend that keeps restarting is usually this failure loudly refusing to serve.

Performance Issues

Slow Queries

Symptoms:

• API responses taking > 5 seconds
• Timeouts on graph operations

Solutions:

1. Check graph size:

   curl -H "Authorization: Bearer $TOKEN" \
     https://sibyl.local/api/admin/stats

2. Limit query results:

   • Use limit parameter on list endpoints
   • Paginate large result sets

High Memory Usage

Symptoms:

• OOMKilled pods in Kubernetes
• Container restarts

Solutions:

1. Increase resource limits:

   backend:
     resources:
       limits:
         memory: 2Gi
       requests:
         memory: 512Mi

2. Check for memory leaks:

   kubectl top pods -n sibyl

3. Reduce worker or API concurrency for memory-constrained pods.

Worker Queue Backlog

Symptoms:

• Jobs not completing
• Crawl tasks stuck

Solutions:

1. Check worker status:

   kubectl logs -n sibyl -l app.kubernetes.io/component=worker -f

2. Scale workers:

   kubectl scale deployment sibyl-worker -n sibyl --replicas=3

3. Check Redis when the opt-in coordination backend is enabled:

   redis-cli -h localhost -p 6381 -n 1 KEYS "*"

Port Conflicts

Local Data-Service Port Conflict

Error: SurrealDB or the optional Redis coordination profile cannot bind its port

Solutions:

1. Find conflicting process:

   lsof -i :8000
   lsof -i :6381

2. Change port in compose:

   Set SIBYL_SURREAL_PORT or SIBYL_REDIS_PORT before starting Docker Compose.

3. Update environment:

   SIBYL_SURREAL_URL=ws://127.0.0.1:<surreal-port>/rpc

Kubernetes-Specific Issues

Pods Stuck in Pending

Solutions:

1. Check node resources:

   kubectl describe nodes
   kubectl top nodes

2. Check resource requests:

   kubectl describe pod <pod-name> -n sibyl

3. Reduce resource requests:

   resources:
     requests:
       cpu: 50m # Reduce from 100m
       memory: 128Mi # Reduce from 256Mi

Pods CrashLoopBackOff

Solutions:

1. Check logs:

   kubectl logs <pod-name> -n sibyl --previous

2. Check events:

   kubectl get events -n sibyl --sort-by='.lastTimestamp'

3. Common causes:

   • Missing secrets
   • Database not ready
   • Port already bound

Kong Gateway Issues

Solutions:

1. Check Kong operator:

   kubectl get pods -n kong-system

2. Check gateway:

   kubectl get gateway -n kong
   kubectl describe gateway sibyl-gateway -n kong

3. Check dataplane:

   kubectl get pods -n kong
   kubectl logs -n kong -l app=dataplane-sibyl-gateway

4. Check HTTPRoutes:

   kubectl get httproute -n sibyl
   kubectl describe httproute sibyl-api -n sibyl

Getting Help

If you're still stuck:

1. Check logs thoroughly:

   # All logs with timestamps
   kubectl logs -n sibyl --all-containers --timestamps -l app.kubernetes.io/name=sibyl

2. Describe resources:

   kubectl describe pod <pod-name> -n sibyl
   kubectl describe deployment <deploy-name> -n sibyl

3. Check events:

   kubectl get events -n sibyl --sort-by='.lastTimestamp' | tail -50

4. File an issue:

   • Include logs and error messages
   • Describe reproduction steps
   • Include environment details (OS, K8s version, etc.)