Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Stabilizing the Jada Agent Daemon: OAuth Token Failures and Turn Limits in Production

Over the past development session, we performed a comprehensive health check on the jada-agent orchestrator daemon running on AWS Lightsail (34.239.233.28). While the service itself is stable and responsive, we uncovered a critical OAuth token degradation in the port sheet sync subsystem and identified a pattern of agent turn limits being reached during complex task runs. This post walks through the diagnostic approach, findings, and the infrastructure decisions that shaped our troubleshooting strategy.

What Was Done

  • Authenticated to the Lightsail instance using AWS SSM Session Manager and temporary SSH credentials from the Lightsail API (since the jada-key private key is not stored locally)
  • Pulled daemon status, systemd logs, and process metrics for the jada-agent.service
  • Correlated service logs with AWS CloudWatch metrics (CPU, memory, network, status checks) over the past 2 hours
  • Analyzed agent session logs and task queue state to understand task completion rates and failure modes
  • Identified and isolated the port_sheet_sync OAuth token failure occurring every 30 minutes
  • Documented the "max turns (30)" exit code 1 pattern occurring in 2 of 3 today's agent runs

Technical Details: Service Health and Metrics

Daemon Status

The jada-agent.service is healthy and stable:

  • Active (running) since May 10, 2026 — 3 days uptime
  • Instance uptime: 11 days
  • Load average: 0.00 (essentially idle between tasks)
  • CPU utilization: 0.65% average over 60-second poll intervals — well within normal bounds for a polling daemon
  • Memory: 144 MB / 914 MB available
  • Disk: 6.2 GB / 39 GB (17% utilization)
  • AWS Lightsail status checks: 0 failures in the past 2 hours

The daemon's 60-second event loop is working as designed. When no tasks are pending in the progress dashboard, the process sleeps and consumes minimal CPU. When tasks arrive, the agent wakes and begins processing.

Session Activity (UTC, May 13)

  • Session 1 (00:00): Reached max turns (30) — exit code 1
  • Session 2 (00:02): Completed successfully — processed e-signature link blockers and crew page generator code, created a needs-you task
  • Session 3 (00:05): Reached max turns (30) — exit code 1
  • After session 3, no additional tasks were found; daemon idling normally

Session quota is 5 sessions per day. After session 3, we have used 3 of 5. The 5/5 hard stop yesterday occurred before midnight UTC with 3 pending tasks queued; they cleared at midnight rollover (expected daily reset behavior).

Critical Issue: port_sheet_sync OAuth Token Degradation

The Problem

Every 30-minute sync cycle since at least May 13 afternoon has been failing with:

[port-sheet] token error: HTTP Error 400: Bad Request

The Google OAuth token for the port_sheet_sync.py script has expired or been revoked. Port sheet syncs have not successfully completed in at least 6+ hours.

Why This Matters

The port_sheet_sync.py daemon is responsible for syncing booking and crew availability data from Google Sheets to the production database. Without this sync:

  • Booking automation (Queen of San Diego, BookingAutomation.gs) cannot reliably read updated crew availability
  • Manual crew scheduling changes made in Google Sheets are not propagated downstream
  • Port sheet data becomes stale, introducing data consistency risk

Root Cause

Google OAuth tokens have a finite lifetime (typically 1 hour for access tokens, with a refresh token allowing renewal). The stored token for the dangerouscentaur@gmail.com service account has either:

  • Exceeded its access token lifetime and the refresh token has also expired or been revoked
  • Been revoked in the Google Cloud Console
  • Lost synchronization with the Google OAuth session (e.g., password change, security event, or browser-based revocation)

Immediate Action Required

Re-authenticate the Google OAuth token for port_sheet_sync.py using the auth flow in /Users/cb/Documents/repos/tools/auth_ga.py (or the equivalent port sheet auth script). The stored token is likely in a secrets directory referenced in /Users/cb/Documents/repos/repos.env. After successful re-auth, the port_sheet_sync daemon will resume 30-minute sync cycles on the next interval.

Secondary Issue: Agent Turn Limit (30 Turns) Exits

Observed Pattern

Two of today's three agent runs (sessions 1 and 3) hit the 30-turn Claude conversation limit and exited with code 1. This is not a crash or service failure — the daemon logs it and continues normally. However, it indicates that complex tasks are exceeding the conversation turn budget.

Why It Happens

Claude's API enforces a maximum conversation length. For long-running agentic tasks (file reads, code generation, multi-step problem solving), each back-and-forth exchange between the agent and Claude consumes turns. When a task is complex (e.g., refactoring booking automation code across multiple files, or generating new pages with SEO metadata), 30 turns can be exhausted before task completion.

Session 2 vs. Sessions 1 & 3

Session 2 completed successfully without hitting the turn limit, suggesting it was a well-scoped, lower-complexity task. Sessions 1 and 3 likely attempted larger refactoring or generation work. The daemon correctly exits on turn limit, and the task remains in the queue for a subsequent session — so work is not lost, but it's deferred.

Infrastructure and Access Decisions

Why SSM Session Manager Instead of SSH Key?

The jada-key private key is not stored in the local machine's ~/.ssh directory. Rather than storing long-lived SSH private keys locally (a security anti-pattern), we used AWS Systems Manager Session Manager combined with the Lightsail API's temporary credential endpoint. This approach:

  • Eliminates the need to rotate or store sensitive SSH keys on developer machines
  • Provides audit logging of all SSH sessions through CloudTrail
  • Uses short-lived, auto-expiring credentials (seconds to minutes)
  • Requires only IAM permissions, not SSH key material

Command Example (Secure SSH Access)

aws lightsail get-instance-access-details \
  --instance-name jada-agent \
  --region us-east-1 \
  > temp_creds.json

# Extract certificate and private key, write to temp file
# SSH with temp credentials
ssh -i temp_key.pem ubuntu@34.239.233.28