Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Remediating OAuth Token Failures in the JADA Orchestrator Daemon

During a routine health check of the JADA agent daemon running on AWS Lightsail (34.239.233.28), we discovered a critical authentication failure in the port sheet synchronization pipeline. This post covers our diagnostic approach, root cause analysis, and the infrastructure decisions that shaped our remediation strategy.

What Was Done

We performed a comprehensive health audit of the jada-agent.service orchestrator daemon, including service status verification, log analysis, metrics collection, and task queue inspection. The audit revealed that while the daemon itself was healthy and actively processing tasks, a persistent OAuth token expiration was preventing the port_sheet_sync.py script from authenticating with Google APIs.

We identified that:

  • The daemon had been running for 11 days with 3 days of continuous uptime on the current boot cycle
  • Three agent sessions had executed in the past 24 hours, with two hitting the 30-turn Claude API limit
  • The port_sheet_sync job was failing every 30 minutes with HTTP Error 400: Bad Request
  • No critical infrastructure issues were present (CPU, memory, disk, and status checks all nominal)

Technical Details: Diagnostic Methodology

SSH Access and Credential Management

Since the private key material for jada-key was not available in the standard ~/.ssh/ directory, we used AWS Lightsail's temporary credential API endpoint to obtain short-lived SSH access. This approach—rather than storing long-lived private keys locally—follows the principle of least privilege:

aws lightsail get-instance-access-details \
  --instance-name jada-agent-prod \
  --region us-east-1

The API response provided a temporary private key and instance connection details. We wrote the key to a temporary file with restrictive permissions (chmod 600), established the SSH session, collected diagnostics, and immediately removed the temporary key material afterward.

Service Status and Log Analysis

Once connected to the Lightsail instance, we interrogated the systemd service directly:

systemctl status jada-agent.service
journalctl -u jada-agent.service --since "2 hours ago" --no-pager

The daemon logs revealed the OAuth token failure pattern in port_sheet_sync.py. Each 30-minute invocation (likely triggered by a cron job or systemd timer) was logging identical error messages, indicating a consistent authentication failure rather than a transient network issue.

Metrics Collection via Lightsail API

We pulled CPU, memory, network, and status check metrics directly from the AWS Lightsail monitoring API:

aws lightsail get-instance-metric-statistics \
  --instance-name jada-agent-prod \
  --metric-name CPUUtilization \
  --statistics Average \
  --start-time 2026-05-13T00:00:00Z \
  --end-time 2026-05-13T23:59:59Z \
  --period 300

Results showed CPU averaging 0.65% with no spikes, memory utilization at 144MB of 914MB available, and zero status check failures in the 2-hour window preceding the audit.

Task Queue and Session Accounting

We inspected the progress dashboard logs to understand task execution patterns. The daemon executed three agent sessions in the 24-hour period:

  • Session 1 (00:00 UTC): Hit the 30-turn Claude API limit and exited with code 1
  • Session 2 (00:02 UTC): Completed successfully; processed e-signature/crew page blockers and queued a needs-you task
  • Session 3 (00:05 UTC): Hit the 30-turn limit again; exit code 1

After session 3, the daemon found no pending tasks and entered an idle state—expected behavior for the polling loop with a 60-second check interval.

Root Cause: Expired Google OAuth Token

The port_sheet_sync.py script authenticates to Google's APIs using an OAuth 2.0 refresh token stored in the credentials backend. Based on the error pattern (consistent HTTP 400 responses every 30 minutes), the token had either expired due to age or been revoked through the Google Account security dashboard.

The script uses the google-auth-oauthlib library to manage token lifecycle. When the refresh token becomes invalid, the library cannot obtain a new access token, resulting in authentication failure for all downstream API calls (in this case, Google Sheets API for the port sheet sync).

Infrastructure and Architecture Decisions

Orchestrator Daemon Design

The JADA agent daemon is a long-running systemd service on a dedicated AWS Lightsail instance. It implements a polling model that periodically checks a task queue (the "progress dashboard") and spawns agent sessions to process work. This architecture provides:

  • Isolation: Agent workloads run on a dedicated instance, preventing resource contention with other services
  • Resilience: Service failures trigger systemd restart policies; the 11-day uptime reflects healthy recovery patterns
  • Observability: Systemd journal integration centralizes all daemon and agent logs
  • Cost efficiency: Lightsail provides fixed pricing ($5–$10/month tier) compared to EC2's variable cost model

OAuth Token Management

Sensitive credentials (OAuth tokens, API keys) are stored in a secrets directory managed outside the code repository. The port_sheet_sync.py script reads credentials at runtime from this secure location rather than embedding them in source code or environment variables.

However, the current implementation lacks automated token refresh detection and re-authentication workflows. When a token expires, manual intervention is required to re-authorize the script through the OAuth consent flow.

Temporary SSH Key Strategy

Rather than maintaining persistent SSH private keys on development machines, we use AWS Lightsail's temporary credential API. This reduces the attack surface: keys are short-lived (typically 60 seconds), automatically revoked after use, and never stored on disk long-term. The trade-off is an extra API call per SSH session, which is acceptable for ad-hoc diagnostic work.

Key Decisions and Rationale

Why we didn't escalate this as a critical issue: The daemon itself was healthy. The port sheet sync failure affected a secondary workflow (Google Sheets synchronization) but did not impair the primary agent orchestration loop. Task processing continued normally; only sheets-dependent automations were blocked.

Why we used Lightsail metrics API instead of CloudWatch: Lightsail instances emit basic metrics to CloudWatch, but the Lightsail API provides direct access to instance-level diagnostics without additional IAM configuration. For quick health checks, this reduces cognitive overhead.

Why the 30-turn limit matters: Two of three agent sessions hit the 30-turn Claude API limit. This isn't a bug; Claude conversations have explicit turn limits to manage latency and cost. However, recurring max-turn exits suggest that task scope should be decomposed into smaller units, or the turn limit should be tuned based on actual task complexity.

What's Next