Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

Diagnosing and Remediating OAuth Token Failures in the JADA Agent Daemon Infrastructure

During a routine health check of the jada-agent orchestrator daemon running on AWS Lightsail (34.239.233.28), we discovered a critical OAuth token degradation affecting the port_sheet_sync.py script. This post documents the diagnostic approach, findings, and remediation strategy for token lifecycle management in distributed agent systems.

What Was Done

We performed a comprehensive health audit of the JADA daemon by:

  • Establishing secure SSH access to the Lightsail instance via AWS-generated temporary credentials
  • Collecting service status, resource metrics, and daemon logs from jada-agent.service
  • Analyzing 24-hour task execution history and session consumption patterns
  • Identifying and isolating the root cause of recurring Google OAuth failures
  • Documenting infrastructure state for remediation

Technical Details: Daemon Health Assessment

The jada-agent.service systemd unit on the Lightsail instance has been running continuously for 3 days with no crashes or restarts. Resource utilization is nominal:

  • CPU: 0.65% average across the measurement window (60-second poll loop); no thermal spikes detected
  • Memory: 144 MB resident set size against 914 MB available; well below pressure thresholds
  • Disk: 6.2 GB of 39 GB used (17%); healthy headroom for logs and task state
  • Network: Status checks passing; 0 failures in the preceding 2 hours
  • Uptime: 11 days for the instance, 3 days for the daemon service

The daemon operates on a 30-second task polling loop, querying the progress dashboard for pending work. Between active sessions, it enters an idle state with minimal CPU consumption, which is the expected behavior.

Session Consumption and Task Execution

The agent is configured with a 5-session daily quota (UTC midnight boundary). On May 13, the daemon consumed all 3 available sessions by 00:05 UTC:

  • Session 1 (00:00 UTC): Reached the 30-turn Claude API limit and exited with code 1. No completion.
  • Session 2 (00:02 UTC): Completed successfully. Processed e-signature and crew page blockers; created a needs-you task for human review.
  • Session 3 (00:05 UTC): Reached the 30-turn limit again; exit code 1.

After session 3, the daemon correctly detected no remaining tasks in the queue and resumed idle polling. This behavior is correct per the orchestration logic.

The recurring "max turns" exits are noteworthy. Two of three sessions hit the 30-turn limit, suggesting that either task complexity has increased or the turn budget is misaligned with typical work scope. While these exit codes are logged as errors, they do not represent daemon crashes—the systemd unit remains active and continues polling. However, this pattern warrants monitoring; if tasks consistently exceed the turn budget, the limit or task decomposition strategy should be revisited.

Critical Issue: Google OAuth Token Degradation

The primary finding is a systematic OAuth token failure in the port_sheet_sync.py synchronization script. Every 30-minute sync attempt since at least May 13 afternoon has failed with:

[port-sheet] token error: HTTP Error 400: Bad Request

This indicates one of three conditions:

  • The Google OAuth 2.0 refresh token has expired and is no longer valid
  • The token was revoked by the user or Google's security systems
  • The token's scopes no longer grant permission to the target Google Sheet API resource

The immediate impact is that port sheet syncs have stalled. Any downstream processes that depend on current port sheet data will consume stale state. Given that this script runs every 30 minutes, a full sync cycle (approximately 30 minutes of missed updates) has accumulated since the first failure.

Infrastructure and Credential Management

The daemon infrastructure is structured as follows:

  • Lightsail Instance: 34.239.233.28 (11-day uptime, 2GB RAM, 2vCPU)
  • SSH Access: Secured via AWS Lightsail temporary credential API (no persistent keys stored locally)
  • Secrets Storage: Credentials stored in ~/jada/repos.env with restricted file permissions (600)
  • Service Management: jada-agent.service systemd unit with automatic restart policy

The Google OAuth token for port_sheet_sync.py is stored in the secrets directory (path redacted for security). The token structure includes client_id, client_secret, and refresh_token fields, matching the standard Google OAuth 2.0 authorization code flow.

Key Decisions and Why

Why SSH via Lightsail temporary credentials instead of persistent keys: The Lightsail API's temporary credential mechanism (valid for 60 seconds) is ephemeral and auditable. Storing persistent SSH keys locally on the development machine increases surface area for compromise. Temporary credentials ensure that each access event can be logged and the credential window is narrow.

Why port_sheet_sync.py uses OAuth refresh tokens: Refresh tokens allow long-lived access to Google APIs without requiring manual re-authentication every few days. However, they carry risk: if revoked or expired, the entire sync pipeline breaks silently. The mitigation is to monitor sync logs and implement alert thresholds for consecutive failures.

Why we monitor both exit codes and log output: The daemon's systemd exit codes tell us if a process crashed, but they don't explain why. Parsing logs for HTTP error codes, token errors, and task state gives us the signal-to-noise distinction between transient failures (network blips) and structural failures (expired credentials).

What's Next

Immediate actions:

  • Re-authenticate the Google OAuth token: Run the auth_ga.py script (located at /Users/cb/Documents/repos/tools/auth_ga.py) with the --account dangerouscentaur@gmail.com flag. This will trigger the OAuth 2.0 consent flow and generate a fresh refresh token.
  • Update the secrets file: Replace the expired token in ~/jada/repos.env with the new one. Verify file permissions remain at 600.
  • Validate sync recovery: Wait for the next 30-minute sync cycle and confirm the HTTP 400 error has cleared from the logs.

Medium-term improvements:

  • Token expiration monitoring: Implement a pre-check in port_sheet_sync.py that validates token validity before attempting the sync. Log explicit "token expired" alerts.
  • Turn limit analysis: Review the two max-