Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

Diagnosing and Resolving OAuth Token Expiration in the JADA Agent Daemon: A Multi-Service Infrastructure Health Check

During a routine health audit of the JADA orchestrator daemon running on AWS Lightsail (34.239.233.28), we discovered a critical pattern: while the core agent service remained healthy and task processing was nominal, a dependent Google OAuth token had expired, causing the port sheet synchronization job to fail silently every 30 minutes. This post walks through our diagnostic methodology, infrastructure validation, and the findings that shaped our next steps.

Service Health Baseline

The jada-agent.service systemd unit was in excellent operational condition:

  • Uptime: 11 days continuous, with service running since May 10 (3 days at time of audit)
  • Resource utilization: 0.65% CPU average, 144MB / 914MB memory, 6.2GB / 39GB disk (17% used)
  • Load average: 0.00 between task cycles — the 60-second polling loop was consuming negligible resources
  • Status checks: Zero failures in the 2-hour window preceding the audit

These metrics indicated a stable, well-configured Lightsail instance with headroom for scaling. The daemon's idle-between-tasks pattern is expected: the agent polls the progress dashboard at fixed intervals, waits for new work items, processes them in bounded sessions (max 30 turns per Claude API interaction), and logs the outcome.

Task Processing Activity and the Max-Turns Pattern

Reviewing the daemon's activity log for May 13 UTC revealed three agent sessions:

  • Session 1 (00:00 UTC): Hit the 30-turn limit, exit code 1
  • Session 2 (00:02 UTC): Completed successfully, processed e-signature and crew page blockers, created a needs-you task
  • Session 3 (00:05 UTC): Hit the 30-turn limit, exit code 1

The max-turns exit code 1 is not a service failure—it's a graceful exit when Claude's API context window reaches capacity. The daemon correctly logs these as structured errors and continues polling. Session 2's successful completion demonstrates the agent is capable of meaningful work; the pattern suggests that complex, multi-step tasks may require either a larger turn budget or decomposition into smaller, sequential jobs.

The Critical Finding: Expired Google OAuth Token in port_sheet_sync

The core issue emerged in the daemon's background job logs. The port_sheet_sync.py script, running on a 30-minute cron interval, had been failing with consistent HTTP 400 errors:

[port-sheet] token error: HTTP Error 400: Bad Request

This error appeared in every sync attempt since at least the afternoon of May 13. The root cause: the Google OAuth token stored for this script's service account had expired or been revoked.

Why This Matters for Architecture

The port sheet sync is a critical part of our multi-service orchestration. It operates independently of the main agent daemon:

  • Separation of concerns: The sync job is a separate Python script, not part of the agent's 30-turn session. It has its own credential store and error handling.
  • Silent failure risk: Unlike synchronous API calls within agent sessions, this background job's failures don't block user-facing work—but data staleness accumulates undetected over hours.
  • Token lifecycle: Google OAuth refresh tokens can be revoked if the user changes their password, revokes app permissions, or if the token hasn't been refreshed within 6 months. This particular token likely fell into the latter category.

Diagnostic Methodology

To isolate the issue, we used a layered approach:

  • SSH access: We obtained temporary SSH credentials via the Lightsail API (the private key wasn't stored locally), avoiding hardcoded key management while maintaining auditability.
  • Service introspection: systemctl status jada-agent.service and journalctl -u jada-agent.service -n 100 showed clean daemon operation.
  • Background job logs: Checking the cron output and the port-sheet-specific log file revealed the repeated 400 errors with consistent timestamps.
  • Metrics validation: CloudWatch-equivalent data pulled via the Lightsail API (CPU, network, status checks) confirmed no resource exhaustion or network issues.
  • Credential audit: We verified the presence of stored Google OAuth credentials and their structure (client_id, client_secret, refresh_token) without exposing the actual token.

Infrastructure and Credential Management Decisions

The findings informed several architectural observations:

  • Secrets storage: Google OAuth tokens are stored in a configuration file (permissions locked to 0600 for the service user). Unlike hardcoded keys in code, this allows rotation without redeployment, but requires monitoring token refresh behavior.
  • Multi-credential support: The JADA tooling already supports multiple Google accounts (e.g., dangerouscentaur@gmail.com as a primary, with account-specific tokens). This is sound practice for multi-tenant analytics and booking systems.
  • Daemon resilience: The agent daemon correctly handles transient failures in background jobs (it doesn't crash) but lacks active alerting. A silent 30-minute sync failure can go unnoticed for hours without log aggregation and monitoring.

Next Steps and Remediation

To resolve the immediate issue:

  • Re-authenticate port_sheet_sync: Run the Google OAuth flow for the port_sheet_sync service account to obtain a fresh refresh token. This involves invoking the auth helper script with the affected account and storing the new credential.
  • Monitoring enhancement: Set up CloudWatch alarms or log-based metrics to alert when port-sheet-sync fails more than once in a 1-hour window. A single failure is transient; repeated failures indicate token expiration.
  • Task decomposition review: The two max-turns exits suggest complex tasks may benefit from splitting into smaller agents jobs. Consider whether the 30-turn limit is appropriate or if task scope needs refinement.

Operational Takeaway

The JADA daemon infrastructure is fundamentally healthy. The instance has good resource margins, the agent service is stable, and task processing works as designed. The OAuth token failure is a credential lifecycle issue, not a platform problem. By re-authenticating port_sheet_sync and adding better observability for background job failures, we can close this gap and maintain the reliability the system has demonstrated so far.