--- url: /sibyl/deployment/troubleshooting.md --- # Troubleshooting Guide Common issues and solutions for Sibyl deployments. ## Connection Issues ### Cannot Connect to API **Symptoms:** * Connection refused on port 3334 * Frontend shows "Cannot connect to server" **Solutions:** 1. **Check if server is running:** ```bash # Docker Compose docker compose --env-file ~/.sibyl/prod.env ps # Kubernetes kubectl get pods -n sibyl ``` 2. **Check server logs:** ```bash # Docker Compose docker compose --env-file ~/.sibyl/prod.env logs backend # Kubernetes kubectl logs -n sibyl -l app.kubernetes.io/component=backend ``` 3. **Verify port binding:** ```bash # Check what's listening lsof -i :3334 netstat -tlnp | grep 3334 ``` 4. **Check firewall rules** (if applicable) ### Database Connection Errors **SurrealDB Connection Refused:** ``` surrealdb error: connection refused ``` **Solutions:** 1. **Verify SurrealDB is running:** ```bash docker compose --env-file ~/.sibyl/prod.env ps surrealdb kubectl get pods -n sibyl -l app=surrealdb ``` 2. **Check connection settings:** ```bash # Environment variables echo $SIBYL_SURREAL_URL ``` 3. **Test direct connection:** ```bash curl http://localhost:8000/health ``` ## Authentication Issues ### Forbidden After Login **Symptoms:** * API response includes `"error": "forbidden"`. * Response body says `"Invalid request data."` with remediation to check organization or project permissions. * Login or refresh succeeds, but the next project, search, admin, or MCP request fails. **What it usually means:** The client has a valid Sibyl session, but the request does not match the user's organization, project, memory-space, or admin permissions. This is different from an OIDC provider problem. In the default install, OIDC is off and local auth is expected. **Solutions:** 1. **Confirm the intended auth mode:** ```bash echo $SIBYL_LOCAL_AUTH_ENABLED echo $SIBYL_PUBLIC_SIGNUPS_ENABLED echo $SIBYL_OIDC ``` Default local installs should have local auth enabled, public signups disabled, and no OIDC providers. 2. **For a new default install, complete setup first.** The first setup signup creates the owner/admin user. After setup, additional users need an invite unless `SIBYL_PUBLIC_SIGNUPS_ENABLED=true`. 3. **Check the active CLI context and token owner:** ```bash sibyl config context sibyl auth status sibyl whoami ``` 4. **For project-scoped failures, switch to a project the user can access or ask an owner/admin to add the membership.** 5. **For enterprise SSO, verify the IdP role claim emits `Sibyl.Member`, `Sibyl.Admin`, or `Sibyl.Owner`.** ### JWT Token Invalid **Symptoms:** * 401 Unauthorized responses * "Invalid token" errors **Solutions:** 1. **Check JWT secret is set:** ```bash # Should be non-empty echo $SIBYL_JWT_SECRET ``` 2. **Verify token hasn't expired:** * Default access token TTL: 60 minutes * Use refresh token to get new access token 3. **Check clock synchronization:** * JWT validation requires synchronized clocks * Server and client should have correct time ### GitHub OAuth Failing **Symptoms:** * "OAuth error" after GitHub redirect * Missing callback URL **Solutions:** 1. **Verify OAuth credentials:** ```bash echo $SIBYL_GITHUB_CLIENT_ID echo $SIBYL_GITHUB_CLIENT_SECRET ``` 2. **Check callback URL in GitHub:** * Must match `SIBYL_PUBLIC_URL/api/auth/github/callback` * Example: `https://sibyl.local/api/auth/github/callback` 3. **Verify public URL:** ```bash echo $SIBYL_PUBLIC_URL ``` ## Schema Issues ### "table ... does not exist" on a Fresh Store **Symptoms:** * API requests return HTTP 500 `internal_error` with a generic message. * Server logs show `NotFoundError: table projects does not exist` (or `team_projects`, `raw_captures`, another core table) against a newly provisioned SurrealDB. * The instance was just installed, restored onto an empty database, or pointed at a new `SIBYL_SURREAL_URL`. **What it usually means:** The auth and content schemas were never applied to this store. The server applies both automatically at startup and refuses to start on failure outside development, so hitting this at request time means the server started in `development` mode against an empty store, or startup bootstrap failed and the log line was missed. **Solutions:** 1. **Apply the schemas directly (safe to run repeatedly):** ```bash sibyld db init ``` The command applies pending auth and content migrations, verifies schema invariants, never drops tables, and skips migrations already recorded in `schema_version`. 2. **Check the startup bootstrap log** for the failure that stopped the automatic path: ```bash sibyl logs tail -l error -s api ``` Look for `Surreal auth schema bootstrap failed` or `Surreal content schema bootstrap failed`; each names the plane, target version, and error. 3. **Verify readiness before sending traffic.** The readiness endpoint reports schema bootstrap state, and compose/Kubernetes health checks gate on it, so an unhealthy backend that keeps restarting is usually this failure loudly refusing to serve. ## Performance Issues ### Slow Queries **Symptoms:** * API responses taking > 5 seconds * Timeouts on graph operations **Solutions:** 1. **Check graph size:** ```bash curl -H "Authorization: Bearer $TOKEN" \ https://sibyl.local/api/admin/stats ``` 2. **Limit query results:** * Use `limit` parameter on list endpoints * Paginate large result sets ### High Memory Usage **Symptoms:** * OOMKilled pods in Kubernetes * Container restarts **Solutions:** 1. **Increase resource limits:** ```yaml backend: resources: limits: memory: 2Gi requests: memory: 512Mi ``` 2. **Check for memory leaks:** ```bash kubectl top pods -n sibyl ``` 3. **Reduce worker or API concurrency for memory-constrained pods.** ### Worker Queue Backlog **Symptoms:** * Jobs not completing * Crawl tasks stuck **Solutions:** 1. **Check worker status:** ```bash kubectl logs -n sibyl -l app.kubernetes.io/component=worker -f ``` 2. **Scale workers:** ```bash kubectl scale deployment sibyl-worker -n sibyl --replicas=3 ``` 3. **Check Redis when the opt-in coordination backend is enabled:** ```bash redis-cli -h localhost -p 6381 -n 1 KEYS "*" ``` ## Port Conflicts ### Local Data-Service Port Conflict **Error:** SurrealDB or the optional Redis coordination profile cannot bind its port **Solutions:** 1. **Find conflicting process:** ```bash lsof -i :8000 lsof -i :6381 ``` 2. **Change port in compose:** Set `SIBYL_SURREAL_PORT` or `SIBYL_REDIS_PORT` before starting Docker Compose. 3. **Update environment:** ```bash SIBYL_SURREAL_URL=ws://127.0.0.1:/rpc ``` ## Kubernetes-Specific Issues ### Pods Stuck in Pending **Solutions:** 1. **Check node resources:** ```bash kubectl describe nodes kubectl top nodes ``` 2. **Check resource requests:** ```bash kubectl describe pod -n sibyl ``` 3. **Reduce resource requests:** ```yaml resources: requests: cpu: 50m # Reduce from 100m memory: 128Mi # Reduce from 256Mi ``` ### Pods CrashLoopBackOff **Solutions:** 1. **Check logs:** ```bash kubectl logs -n sibyl --previous ``` 2. **Check events:** ```bash kubectl get events -n sibyl --sort-by='.lastTimestamp' ``` 3. **Common causes:** * Missing secrets * Database not ready * Port already bound ### Kong Gateway Issues **Solutions:** 1. **Check Kong operator:** ```bash kubectl get pods -n kong-system ``` 2. **Check gateway:** ```bash kubectl get gateway -n kong kubectl describe gateway sibyl-gateway -n kong ``` 3. **Check dataplane:** ```bash kubectl get pods -n kong kubectl logs -n kong -l app=dataplane-sibyl-gateway ``` 4. **Check HTTPRoutes:** ```bash kubectl get httproute -n sibyl kubectl describe httproute sibyl-api -n sibyl ``` ## Getting Help If you're still stuck: 1. **Check logs thoroughly:** ```bash # All logs with timestamps kubectl logs -n sibyl --all-containers --timestamps -l app.kubernetes.io/name=sibyl ``` 2. **Describe resources:** ```bash kubectl describe pod -n sibyl kubectl describe deployment -n sibyl ``` 3. **Check events:** ```bash kubectl get events -n sibyl --sort-by='.lastTimestamp' | tail -50 ``` 4. **File an issue:** * Include logs and error messages * Describe reproduction steps * Include environment details (OS, K8s version, etc.)