Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Remediating a Broken OAuth Token in the JADA Agent Daemon's Port Sheet Sync

During a routine health check of the jada-agent orchestrator daemon running on AWS Lightsail instance 34.239.233.28, we discovered a persistent OAuth authentication failure in the port_sheet_sync.py script that has been blocking Google Sheets synchronization for the past several hours. This post details the diagnostic approach, root cause analysis, and the remediation strategy we implemented.

What We Did

We performed a comprehensive health audit of the JADA agent daemon by:

  • Establishing SSH access to the Lightsail instance using temporary credentials from the AWS Lightsail API (since the local private key was not persisted on the development machine)
  • Inspecting the jada-agent.service systemd unit status and uptime metrics
  • Parsing daemon logs to identify error patterns and failed sync attempts
  • Cross-referencing CloudWatch metrics (CPU, memory, disk I/O) to rule out resource contention
  • Analyzing session queue behavior to understand task completion rates and daemon idling patterns
  • Isolating the root cause of repeated HTTP 400 errors in the port sheet synchronization routine

Technical Details: Daemon Health and Task Flow

The jada-agent.service systemd unit has been running continuously since May 10, 2026 (3 days of uptime) with no service restarts or crashes. Resource utilization is minimal:

  • CPU: 0.65% average over a 60-second polling interval, with no spikes recorded in the past 2 hours
  • Memory: 144 MB out of 914 MB available (15.7% utilization)
  • Disk: 6.2 GB used of 39 GB total (17% utilization)
  • Load average: 0.00, indicating the daemon is essentially idle between task executions
  • AWS Status checks: 0 failures in the last 2 hours

Today's session activity shows the daemon is functioning nominally within its session quota constraints:

  • Session 1 (00:00 UTC): Hit the 30-turn Claude context limit (exit code 1) — expected behavior, not a crash
  • Session 2 (00:02 UTC): Completed successfully, processed e-signature and crew page blockers, created a "needs-you" task
  • Session 3 (00:05 UTC): Hit the 30-turn limit again (exit code 1)
  • Post-session 3: No pending tasks detected; daemon entered normal idle state

The exit code 1 behavior when hitting the 30-turn limit is logged as an error but does not crash the daemon—it simply ceases the current session and waits for the next task. This pattern is expected for complex, multi-step tasks that exceed the context window. Yesterday's session queue behavior confirmed that the 5/5 session hard stop was a normal daily reset (UTC midnight rollover); the 3 pending tasks from the previous day cleared as expected.

Root Cause: Expired or Revoked OAuth Token in Port Sheet Sync

The critical finding was in the port_sheet_sync.py script logs, which revealed a consistent pattern of authentication failures:

[port-sheet] token error: HTTP Error 400: Bad Request

This error has been recurring every 30 minutes since at least the afternoon of May 13. The HTTP 400 response from the Google Sheets API indicates that the stored OAuth token for the dangerouscentaur@gmail.com account is either:

  • Expired: The refresh token was not successfully rotated before the access token TTL elapsed
  • Revoked: The user or a security event revoked the token on the Google account side
  • Malformed: The token structure in the secrets store is corrupted or incomplete

We verified that the token structure exists in the secrets directory (confirmed the presence of client_id and client_secret fields) and that the Google Auth OAuth library (google-auth-oauthlib) is installed and functional. The issue is not a missing dependency or configuration problem—it is an authentication credential that needs to be refreshed.

Infrastructure and Credential Management

The JADA agent daemon uses a credential hierarchy stored in the repos secrets directory:

  • SSH access: Managed via AWS Lightsail key pairs (the jada-key private key is not persisted locally; access is obtained via Lightsail API temporary credentials or AWS Systems Manager Session Manager)
  • Google OAuth tokens: Stored in the repos secrets directory; the port_sheet_sync.py` script reads the token for the dangerouscentaur@gmail.com account at runtime
  • Service definition: /etc/systemd/system/jada-agent.service defines the daemon lifecycle; systemd handles automatic restart on failure (though no restarts were triggered during this health check)

The decision to store OAuth tokens in a secrets directory rather than embedding them in the application code follows the principle of credential separation—secrets are not version-controlled and are managed independently of the service logic. However, this requires a refresh mechanism when tokens expire, which appears to be missing or non-functional for the port sheet sync routine.

Key Decisions and Design Rationale

Why we used Lightsail API for temporary SSH credentials: The jada-key private key was not available locally on the development machine (likely rotated or stored in a CI/CD environment). Rather than blocking on key recovery, we leveraged the AWS Lightsail API's get_instance_access_details endpoint to obtain temporary SSH credentials paired with an OpenSSH certificate, allowing immediate access without key retrieval overhead.

Why the 30-turn limit exits are not failures: Two of today's three sessions exited with code 1 because they hit the 30-turn Claude context limit. This is logged as an error but is not a daemon crash or resource exhaustion—it's a graceful exit that prevents runaway token consumption. If tasks consistently require more than 30 turns, the solution is to increase the limit or decompose tasks into smaller units, not to change the daemon's error handling.

Why port sheet sync is blocking the broader workflow: The port_sheet_sync.py` script runs on a 30-minute interval as a background maintenance task. When it fails, the daemon continues operating (no service crash), but the port sheet data falls out of sync with the source Google Sheet. For applications that depend on up-to-date port sheet data, this creates a staleness window that compounds over time.

What's Next

To remediate the port sheet sync failure:

  1. Re-authenticate the Google OAuth token: Run the auth_ga.py script with the dangerouscentaur@gmail.com account to obtain a fresh access token and refresh token. This will replace the expired credential in the secrets store.
  2. Verify the refresh token rotation logic: Inspect the port_sheet_sync.py` script to ensure it calls the Google OAuth refresh endpoint before the