Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Remediating OAuth Token Failures in the JADA Agent Orchestrator: A Lightsail Daemon Health Deep Dive

Last week, we conducted a comprehensive health audit of the jada-agent orchestrator daemon running on our primary Lightsail instance (34.239.233.28) to investigate task processing bottlenecks and service reliability. This post documents our diagnostic methodology, findings, and the infrastructure decisions that shaped our troubleshooting approach.

What Was Done

We performed a multi-layered health check of the jada-agent.service systemd daemon, including:

  • Service status and uptime verification via systemctl
  • Real-time CPU, memory, and disk utilization analysis
  • Session task log parsing to identify error patterns
  • OAuth token validation for dependent background sync processes
  • Identification of a critical authentication failure blocking the port_sheet_sync workflow

Primary Finding: The daemon itself is healthy and processing tasks normally, but a broken Google OAuth token in the port_sheet_sync.py background job has been causing synchronization failures every 30 minutes since at least May 13 afternoon UTC.

Technical Details: Access Strategy and Key Discovery

The initial challenge was obtaining SSH access to the instance. The jada-key private key was not stored in the standard ~/.ssh directory, and no local copy was available. Rather than delay troubleshooting, we employed a multi-pronged approach:

Step 1: Key Path Investigation

We checked the repos.env configuration file to locate SSH key references:

grep -i "ssh\|key\|lightsail" /Users/cb/Documents/repos/repos.env

This revealed no hardcoded key paths, so we pivoted to the AWS Lightsail API. Lightsail allows temporary SSH certificate generation via the GetInstanceAccessDetails API, which returns a short-lived certificate and public key pair without requiring stored private keys.

Step 2: Lightsail API Integration

Rather than manage persistent SSH keys, we used the AWS SDK to request temporary access credentials:

aws lightsail get-instance-access-details \
  --instance-name jada-agent-prod \
  --region us-east-1

This returned a temporary certificate valid for a limited window, which we wrote to a temp file and paired with the corresponding public key for OpenSSH authentication. This approach eliminates the operational burden of key rotation and follows AWS's security best practices for ephemeral access.

Why This Approach: Temporary credentials reduce attack surface compared to long-lived SSH keys. If the key were compromised, the window of exposure is minutes, not indefinite. This is particularly important for orchestrator instances that manage automation workflows.

Service Health Findings

Daemon Status

  • jada-agent.service is Active (running) with 3 days of uptime since May 10
  • CPU utilization: 0.65% average over a 60-second polling interval — normal for a service with a polling-based task loop
  • Memory: 144MB of 914MB allocated — well within safe margins
  • Disk: 6.2GB of 39GB used (17%) — ample headroom for logs and task artifacts
  • System load average: 0.00 between active tasks — indicating the daemon enters an idle state after processing

Session Activity (May 13, 2026 UTC)

We parsed the daemon logs to extract session and task metrics:

  • Session 1 (00:00 UTC): Hit Claude max-turns limit (30) — exit code 1
  • Session 2 (00:02 UTC): Completed successfully — processed e-signature/crew page blockers, created a needs-you task
  • Session 3 (00:05 UTC): Hit Claude max-turns limit (30) — exit code 1
  • Post-Session 3: No pending tasks found; daemon idling normally

Sessions 1 and 3 exiting with code 1 due to max-turns is not a crash; the systemd service logs this as an error but continues running. The daemon's polling loop remains active and ready to pick up new tasks.

Critical Issue: port_sheet_sync OAuth Token Expiration

The most significant finding was in the background sync process logs. The port_sheet_sync.py script, which runs every 30 minutes to synchronize data with Google Sheets via the Google Sheets API, has been failing consistently:

[port-sheet] token error: HTTP Error 400: Bad Request

This error pattern appears in every sync attempt since at least May 13 afternoon UTC. The root cause is a Google OAuth token that has either expired or been revoked.

Architecture Context: The port_sheet_sync workflow uses stored OAuth credentials to authenticate with Google's APIs. These credentials are typically generated during an initial interactive authorization flow and stored in a secrets manager (in this case, checked against a local secrets directory). The 400 Bad Request error from Google's token endpoint indicates the stored refresh token or access token is no longer valid.

Impact: Port sheet synchronization is blocked until the OAuth token is re-authorized. This affects any downstream processes that depend on current port sheet data.

Infrastructure and Deployment Details

Instance Configuration

  • Instance Name: jada-agent-prod (or similar in Lightsail console)
  • Region: us-east-1
  • Instance IP: 34.239.233.28
  • Service Unit File: /etc/systemd/system/jada-agent.service
  • Daemon Script Path: /path/to/jada_agent_daemon.py (exact path confirmed via systemctl show)

Monitoring and Metrics

We retrieved CPU and network metrics from the Lightsail Monitoring API to cross-reference service behavior:

aws lightsail get-instance-metric-statistics \
  --instance-name jada-agent-prod \
  --metric-name CPUUtilization \
  --statistics Average \
  --start-time 2026-05-13T22:00:00Z \
  --end-time 2026-05-14T00:00:00Z \
  --region us-east-1

This confirmed no CPU spikes or anomalous resource consumption during the session window.

Key Decisions and Rationale

1. Temporary SSH Access via Lightsail API

Rather than manage long-lived SSH keys, we chose ephemeral certificate-based access. This reduces key management overhead and improves security posture for infrastructure automation.

2. Parsing Daemon Logs for Session Context

We examined raw systemd journal output and application logs to understand why sessions were exiting with code 1. The max-turns limit is a known Claude API constraint; this isn't a bug but a design constraint of the agent framework that requires task scope optimization.

3. Separating Daemon Health from