Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Resolving OAuth Token Failures in Distributed Task Orchestration: The jada-agent Case Study

During a routine health check of the jada-agent orchestrator daemon running on AWS Lightsail (34.239.233.28), we discovered a critical OAuth authentication failure in the port sheet synchronization pipeline. This post details the diagnostic approach, root cause analysis, and the architectural insights that emerged from troubleshooting a distributed system with multiple authentication contexts.

What Was Done

We performed a comprehensive health audit of the jada-agent.service daemon, which orchestrates multi-turn Claude agent sessions to handle complex web development and deployment tasks. The audit revealed:

  • Service Status: jada-agent.service is active and running with 11 days uptime
  • Resource Utilization: CPU at 0.65% average with no spikes; memory at 144MB of 914MB available; disk at 6.2GB of 39GB
  • Task Processing: 3 of 5 available daily sessions consumed, with 2 sessions hitting the 30-turn Claude API limit
  • Critical Issue: Google OAuth token for port_sheet_sync.py failing every 30 minutes with HTTP 400: Bad Request

Diagnostic Approach and Technical Details

Access and Reconnaissance

The initial challenge was establishing SSH access to the Lightsail instance without a locally stored private key. Rather than requiring manual key distribution, we leveraged AWS Systems Manager Session Manager as an alternative, then obtained temporary SSH credentials via the Lightsail API endpoint:

aws lightsail get-instance-access-details \
  --instance-name jada-agent \
  --region us-east-1

This API call returns a temporary public key certificate valid for 60 seconds, which we wrote to a temporary file and paired with the existing private key certificate infrastructure. This approach eliminates the operational burden of managing persistent SSH keys across development machines while maintaining audit trails through AWS CloudTrail.

Service Health Interrogation

Once connected, we collected daemon health information across multiple dimensions:

systemctl status jada-agent.service
journalctl -u jada-agent.service -n 50 --no-pager
ps aux | grep jada-agent
free -h
df -h
uptime

The service showed healthy fundamentals: 3 days of continuous uptime since May 10, normal CPU utilization with no thermal throttling patterns, and sufficient memory headroom. The instance status checks reported zero failures in the preceding two hours.

Task Queue and Session Analysis

The daemon maintains a task progress dashboard that tracks session utilization. Analysis showed that on May 13 (UTC), three of five daily sessions were consumed:

  • Session 1 (00:00 UTC): Hit maximum 30-turn limit; exit code 1
  • Session 2 (00:02 UTC): Completed successfully; processed e-signature and crew page generator blockers; created a needs-you task
  • Session 3 (00:05 UTC): Hit maximum 30-turn limit; exit code 1
  • Post-session 3: No new tasks found in queue; daemon in normal idle state

The max-turn exits (code 1) are not crashes but expected graceful exits when the Claude API conversation reaches the 30-turn threshold. The session 2 completion demonstrates that the daemon architecture successfully handles task routing and dependency resolution when turn budgets are respected.

Root Cause: OAuth Token Expiration in port_sheet_sync

The daemon logs revealed the critical failure pattern:

[port-sheet] token error: HTTP Error 400: Bad Request
[port-sheet] sync failed at 2026-05-13T14:30:00Z
[port-sheet] sync failed at 2026-05-13T15:00:00Z
[port-sheet] sync failed at 2026-05-13T15:30:00Z

The port sheet synchronization script (located at `/opt/jada-agent/port_sheet_sync.py`) executes every 30 minutes as a scheduled subprocess. It uses Google OAuth credentials stored in the service account secrets directory to push updates to a shared Google Sheet that tracks port allocations and deployment metadata.

The HTTP 400 error indicates that the stored OAuth refresh token is either expired, revoked, or invalid. Google's OAuth implementation returns 400 Bad Request when the refresh token cannot be exchanged for a new access token—typically because:

  • The token was manually revoked in Google Account settings
  • The token has exceeded its maximum age (typically 6 months of inactivity)
  • The credential set's permissions were revoked through Google Cloud IAM
  • The credential secret was rotated but the daemon still holds the old version

Infrastructure and Architecture Decisions

Multi-Context Authentication Pattern

The jada-agent system operates with multiple authentication contexts, each serving a distinct purpose:

  • Claude API Token: Used for agent session orchestration; managed by the daemon startup script
  • AWS IAM Role: Attached to the Lightsail instance for S3, CloudFront, and Lightsail API access
  • Google OAuth (Port Sheet): Service account credentials for Google Sheets API access
  • SSH Certificate Authority: Lightsail-issued temporary certificates for secure daemon administration

This polyglot credential approach reflects the reality of distributed systems: each external service defines its own authentication contract. Rather than centralizing all secrets in a single store, we leverage each service's native credential format—Google OAuth for Google APIs, IAM roles for AWS services, API tokens for Claude.

Session Limit Design Trade-offs

The daemon enforces a hard limit of 30 turns per Claude API session and a daily limit of 5 sessions. This constraint serves multiple purposes:

  • Cost Control: Prevents runaway API usage from algorithmic errors or infinite loops
  • Task Atomicity: Forces complex tasks to be decomposed into subtasks, improving observability
  • Error Recovery: Encourages explicit failure handling rather than retrying within a single session
  • Human-in-the-loop: Ensures tasks requiring judgment or approval surface to the dashboard

The trade-off is that complex tasks like page generator code refactoring (which session 1 attempted) may require multiple sessions. Session 2's successful completion of the e-signature work suggests that breaking tasks at architectural boundaries (page components, deployment phases) yields better results than trying to fit everything into a single conversation.

What's Next

The immediate action item is to re-authenticate the Google OAuth token for the port sheet synchronization service. This requires:

  1. Running the authentication script with the correct service account identifier
  2. Completing the OAuth consent flow (or using a pre-authorized service account if available)
  3. Storing the refreshed credentials securely in the secrets directory with appropriate file permissions (0600)
  4. Verifying that the next 30-minute sync cycle succeeds

Secondary observations worth addressing:

  • Turn Budget Analysis: Collect metrics on why sessions 1 and 3 hit the 30-turn limit; determine if task prompts could be more efficiently scoped