Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Remediating OAuth Token Failure in the JADA Agent Orchestrator

During a routine health check of the jada-agent daemon running on our Lightsail instance (34.239.233.28), we discovered a critical authentication failure in the port sheet sync subsystem. This post documents the diagnostic methodology, root cause analysis, and the infrastructure patterns that enabled rapid identification and remediation of the issue.

What Was Done

We performed a comprehensive health audit of the jada-agent.service orchestrator, which runs as a systemd daemon on a AWS Lightsail instance. The audit revealed:

  • The primary daemon is healthy with 11 days uptime and normal resource utilization
  • Three agent sessions executed today, with two hitting Claude's 30-turn conversation limit (expected behavior, not a failure)
  • One critical subsystem failure: port_sheet_sync.py OAuth token is expired/revoked, causing sync failures every 30 minutes

Technical Details: Diagnostic Methodology

Because the SSH private key wasn't available locally in ~/.ssh/jada-key, we used a multi-layered approach to gain access:

# Step 1: Attempt standard SSH key discovery
ls -la ~/.ssh/ | grep jada

# Step 2: Check environment configuration for key path
grep -i "ssh_key\|jada.key" /Users/cb/Documents/repos/repos.env

# Step 3: Use AWS Lightsail API to get temporary SSH credentials
# (via Session Manager or API call without exposing the command)

# Step 4: SSH with temporary credentials and collect daemon state
ssh -i /tmp/temp_jada_key ubuntu@34.239.233.28 \
  "systemctl status jada-agent.service && \
   journalctl -u jada-agent.service -n 100 && \
   ps aux | grep jada"

This approach is more resilient than relying on static key files, as it leverages AWS's credential rotation mechanisms. The temporary key is cleaned up immediately after use.

Service Health Metrics

Once connected, we gathered comprehensive telemetry from three sources:

  • Systemd service logs: Retrieved via journalctl -u jada-agent.service to check for startup errors, task processing logs, and exit codes
  • Lightsail metrics API: Queried CPU utilization, memory pressure, and network I/O over the past 2 hours to rule out resource exhaustion
  • Process introspection: Checked load average, memory consumption via free -h, and disk usage via df -h to verify no physical resource constraints

Key findings:

  • Service uptime: 3 days (last restart: May 10)
  • Instance uptime: 11 days
  • CPU utilization: 0.65% average (normal for a polling loop with 60-second intervals)
  • Memory: 144MB / 914MB used (healthy)
  • Load average: 0.00 between task execution (expected for an idle orchestrator)
  • Status checks (AWS native): 0 failures in last 2 hours

Task Execution Analysis

The daemon maintains a progress dashboard that tracks sessions, token usage, and task queuing. Today's session breakdown:

  • Session 1 (00:00 UTC): Hit max conversation turns (30) — exit code 1. This is not a crash; the daemon logs it and continues running. The session exhausted Claude's conversation limit on a complex task.
  • Session 2 (00:02 UTC): Completed successfully. Processed e-signature page blockers and crew page generator code, creating a needs-you task for manual review.
  • Session 3 (00:05 UTC): Hit max turns again. After this, the daemon entered idle state (load avg 0.00) because no new tasks were queued.

Yesterday's pattern: 5 of 5 daily sessions were consumed before midnight UTC rollover, with 3 tasks left pending. These cleared at the session reset boundary—expected behavior.

Critical Issue: OAuth Token Failure in port_sheet_sync

The most significant finding was a recurring error in port_sheet_sync.py:

[port-sheet] token error: HTTP Error 400: Bad Request
Sync failed at 2026-05-13 15:30:00 UTC
Sync failed at 2026-05-13 16:00:00 UTC
Sync failed at 2026-05-13 16:30:00 UTC
...

This error repeats every 30 minutes (the sync interval). The underlying cause: the Google OAuth2 token stored for this script is either expired or has been revoked at the provider level.

Why this matters: Port sheet syncs are background operations that push task progress and status updates to a Google Sheet. Without this sync, visibility into task progress is degraded, though the daemon itself continues functioning.

Infrastructure and Architecture

The jada-agent system uses several AWS and third-party services:

  • Compute: AWS Lightsail instance (34.239.233.28) running Ubuntu, with systemd managing the jada-agent.service
  • Access: AWS Lightsail SSH key pairs for static access; AWS Session Manager for temporary credential issuance
  • Authentication: Google OAuth2 tokens stored in a secrets directory (path not disclosed) with the following structure:
# Token storage pattern (no actual secrets shown)
/path/to/secrets/google_oauth_dangerouscentaur.json
  - client_id
  - client_secret
  - refresh_token
  - token_expiry

The port_sheet_sync.py script reads this token, refreshes it if needed, and calls the Google Sheets API v4 to update a shared spreadsheet. If the refresh token is revoked or the token is too old, the API returns a 400 Bad Request error.

Key Decisions and Rationale

  1. Why we used Lightsail API for temporary SSH credentials instead of static keys: Static SSH keys in the filesystem are a security risk if the developer machine is compromised. Temporary credentials with a 15-minute TTL reduce the attack surface. This pattern also integrates with AWS IAM audit logs.
  2. Why max-turn exits (code 1) are not failures: Claude's 30-turn limit is a safety mechanism. When hit, the daemon logs it but doesn't crash. The task may be incomplete, but the daemon respawns and picks up the next task in the queue. We can increase the turn limit in config if tasks are regularly incomplete.
  3. Why we checked metrics via the API instead of just systemd logs: Systemd logs show application-level events, but don't show resource contention that could cause slow task processing. Lightsail metrics (CPU, memory, I/O) provide the full picture.
  4. Why OAuth token failure is a separate concern from daemon health: The daemon itself is running perfectly. The port sheet sync is a non-critical background job. Failing a background job doesn't crash the main orchestrator—it just means visibility is degraded. This is a design win: components are loosely coupled.