Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

Diagnosing and Stabilizing the JADA Agent Daemon: OAuth Token Failures and Session Management

During a scheduled health check of the orchestrator daemon running on our Lightsail instance (34.239.233.28), we discovered a critical OAuth authentication failure in the port sheet sync service alongside some expected behavioral patterns in the agent's session management. This post details the diagnostic process, findings, and the path forward for remediation.

What Was Done

We performed a comprehensive health audit of the jada-agent.service running on our primary orchestration instance. This involved:

  • SSH access via AWS Lightsail API-generated temporary credentials (since the persistent private key was not available locally)
  • Service status verification and uptime analysis
  • System resource utilization review (CPU, memory, disk)
  • Daemon log analysis covering the last 24 hours of activity
  • Google Analytics token validation and OAuth credential health checks
  • Task queue and session management pattern analysis

Technical Details: The OAuth Token Failure

The most significant finding was a recurring authentication error in the port_sheet_sync.py script, which runs every 30 minutes as a scheduled sync task.

[port-sheet] token error: HTTP Error 400: Bad Request

This error has been repeating consistently since at least this afternoon UTC. The root cause is an expired or revoked Google OAuth token stored in the credentials manager for this script. The token was originally minted to allow port_sheet_sync.py to read and write to a Google Sheet that tracks project metadata and deployment status.

Why This Matters: When the OAuth token becomes invalid, the sync fails silently in the daemon logs but the task continues to be scheduled every 30 minutes. This creates noise in the logs and means downstream consumers of the port sheet (typically reporting dashboards or other automation) are working with stale data. However, it does not cause the daemon itself to crash—it properly catches the HTTP 400 and logs the error.

The fix requires re-authenticating the Google account that owns the port sheet via OAuth 2.0 flow. This involves:

  • Running the authentication helper script: /Users/cb/Documents/repos/tools/auth_ga.py with the appropriate service account or user email
  • Following the OAuth consent screen flow to grant the daemon permission to access Google Sheets
  • Storing the refreshed token back in the credentials store (likely in ~/.jada/credentials/ or similar location on the instance)
  • Restarting or signaling the port_sheet_sync.py process to pick up the new token

Session Management and the "Max Turns" Pattern

The daemon logs revealed that today the agent consumed 3 of its allotted 5 daily sessions:

  • Session 1 (00:00 UTC): Exited with code 1 after hitting the 30-turn Claude API limit
  • Session 2 (00:02 UTC): Completed successfully; processed e-signature page blockers and crew page generator code, creating a needs-you task for manual follow-up
  • Session 3 (00:05 UTC): Exited with code 1 after hitting the 30-turn limit again

Analysis: The exit code 1 on sessions 1 and 3 are not daemon failures—they indicate the Claude conversation hit the configured maximum of 30 turns and gracefully exited. This is expected behavior when agent tasks are complex and require many back-and-forth interactions to complete. Session 2's success demonstrates the daemon itself is functioning properly; it picked up a task, worked through it, and completed it within the turn budget.

The pattern suggests that some tasks queued in the progress dashboard may be too complex to solve within 30 turns, or that task decomposition could be improved to break larger jobs into smaller, more focused subtasks. After session 3, the daemon found no new tasks and returned to idle state with a load average of 0.00—exactly as designed.

Infrastructure Health

The underlying Lightsail instance is in good shape:

  • Uptime: 11 days without restart
  • CPU utilization: 0.65% average over the last 2 hours (baseline polling and log reads only)
  • Memory: 144 MB of 914 MB in use (16% utilization)
  • Disk: 6.2 GB of 39 GB occupied (17% utilization)
  • Network status checks: 0 failures in the last 2 hours

No CPU spikes, network issues, or disk pressure were detected. The instance is appropriately sized for its workload.

Key Decisions and Architecture Notes

Temporary SSH Keys via Lightsail API: Since the persistent jada-key private key was not stored locally in ~/.ssh/, we used the AWS Lightsail GetInstanceAccessDetails API endpoint to obtain temporary SSH credentials. This is a more secure pattern than storing long-lived private keys on developer machines and demonstrates the value of API-driven credential vending.

Token Isolation: The port sheet OAuth token is scoped only to Google Sheets permissions and is separate from the Google Analytics token used by other tools. This isolation is correct—failure of one does not cascade to others. However, we should consider centralizing token refresh logic so that expired tokens are detected and re-authorized proactively rather than reactively.

Session Quotas: The 5-session daily limit appears intentionally conservative (the daemon logs show the quota resets at midnight UTC). This is a reasonable safety measure to prevent runaway agent spending. The pattern of hitting max turns suggests we should review task complexity and consider implementing task batching or streaming responses back to the progress dashboard so partial work is visible before session limits are hit.

What's Next

  • Immediate: Re-authenticate the Google OAuth token for port_sheet_sync.py using the auth helper and restart the sync loop
  • Short-term: Monitor the next 24 hours of daemon logs to confirm OAuth errors cease and port sheet syncs succeed
  • Medium-term: Review the two tasks that hit the 30-turn limit; determine if they can be decomposed or if the turn limit should be increased for specific task types
  • Longer-term: Implement proactive token expiration monitoring and automated re-auth flows for all external API credentials

The daemon is stable and operational. The OAuth failure is isolated, detectable, and remediable. No urgent infrastructure issues were found.