Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

Diagnosing and Remediating the JADA Agent Orchestrator: OAuth Token Expiry, Turn Limits, and Multi-Site Deployment Pipeline

This session involved comprehensive health diagnostics of the JADA agent daemon running on AWS Lightsail (34.239.233.28), remediation of a critical Google OAuth token failure in the port sheet sync subsystem, analysis of Claude API turn-limit behavior, and concurrent deployment of a new SEO landing page across the 86from.com property. Here's a detailed breakdown of what was done and why.

Infrastructure Diagnostics: Lightsail Instance Health Assessment

The primary objective was to verify daemon health without having the SSH private key locally available. The session demonstrates a practical pattern for remote diagnostics when standard SSH key access is unavailable:

  • Initial challenge: The jada-key private key was not present in ~/.ssh/, and ~/.ssh/config contained no reference to the Lightsail instance.
  • Solution pattern: Use AWS Lightsail's temporary SSH access credentials API rather than relying on pre-distributed key material. This sidesteps key management friction and provides audit trails.
  • Implementation: Called the Lightsail GetInstanceAccessDetails API, extracted the temporary certificate and private key, wrote them to a secure temp file with 0600 permissions, and established an SSH session.

Once connected, the diagnostics revealed a healthy baseline:

  • jada-agent.service status: Active and running continuously since May 10 (3 days uptime)
  • Instance uptime: 11 days; load average: 0.00 (essentially idle between task executions)
  • CPU utilization: ~0.65% average over 60-second poll cycles (normal for a daemon with a 60s main loop)
  • Memory footprint: 144 MB / 914 MB allocated (clean, no leaks)
  • Disk usage: 6.2 GB / 39 GB (17% utilization; plenty of headroom)
  • AWS status checks: 0 failures in the last 2 hours

Agent Session Activity and Turn-Limit Behavior

The daemon's session telemetry for May 13 revealed three distinct runs, with interesting failure modes:

Session 1 (00:00 UTC): Exit code 1 — hit max turns (30)
Session 2 (00:02 UTC): Exit code 0 — completed successfully
Session 3 (00:05 UTC): Exit code 1 — hit max turns (30)

Why this matters: Sessions 1 and 3 exited with non-zero codes because they reached Claude's 30-turn conversation limit. This isn't a service failure—the daemon logs it and continues idling normally. However, Session 2 (which completed within the turn budget) successfully processed blockers on the e-signature and crew page generator code, producing a needs-you task for manual follow-up.

Root cause analysis: Complex multi-step tasks exhaust the 30-turn limit before reaching terminal states. The pattern suggests that either task scope needs to be narrowed (breaking large tasks into subtasks), or the turn budget needs to be increased in the Claude API configuration. This is a design trade-off between cost per session and completion rates for complex workflows.

Critical Issue: Google OAuth Token Expiry in Port Sheet Sync

The most pressing finding was a recurring failure in the port_sheet_sync.py subsystem:

[port-sheet] token error: HTTP Error 400: Bad Request

This error has been firing every 30 minutes since at least the afternoon of May 13, causing all Google Sheets syncs to fail silently. The underlying cause: the OAuth token stored for the port sheet sync script has expired or been revoked.

Why this is critical: The port sheet is a source-of-truth document for booking and crew management. Without live sync, downstream systems (booking automation, crew scheduling) will operate on stale data.

Remediation path: Re-authenticate the Google OAuth flow for the port sheet service account. This requires:

  • Running the auth helper script (/Users/cb/Documents/repos/tools/auth_ga.py, recently refactored) with the port sheet service account email
  • Following the OAuth 2.0 consent flow to obtain a fresh access token
  • Storing the refreshed token in the encrypted secrets store (checked and confirmed to exist at ~/.repos-secrets/)
  • Restarting the port_sheet_sync daemon or triggering a manual sync to verify the new token is accepted

Concurrent Deployments: 86from.com SEO and Booking Widget

While diagnosing the daemon, parallel work proceeded on the 86from.com property (formerly 86dfrom, renamed to match the brand domain):

  • Directory rename: /repos/sites/86dfrom.com/ → /repos/sites/86from.com/ to align with the primary domain.
  • New SEO landing page: Created /repos/sites/86from.com/site/what-does-86d-mean to capture long-tail search traffic for "86d" terminology queries.
  • Booking widget refactoring: The index.html contained a booking widget with unescaped template syntax (double braces {{ }} appearing outside the widget's intended scope). This was causing parse failures in the JavaScript block.

Widget fix rationale: Double-brace syntax (Handlebars, Jinja2-style) is valid inside the booking widget's isolated context but invalid in global HTML. A detailed audit confirmed the issue was localized to the widget section. The fix involved replacing errant {{ and }} tokens with single braces within the widget block only, preserving the correct template syntax where needed.

Deployment pipeline:

  • Deployed corrected index.html to the staging S3 bucket
  • Invalidated the staging CloudFront distribution cache (specific distribution ID withheld for security)
  • Prepared production deployment to the main S3 bucket with a versioned comment tag embedding the booking widget model ID for tracking
  • Queued production CloudFront invalidation pending final QA

Key Architectural Decisions

  • Lightsail API for temporary SSH access: Rather than distributing and rotating long-lived SSH keys, we leveraged AWS's built-in temporary credentials mechanism. This reduces key sprawl and improves audit trails.
  • Modular authentication tooling: The auth_ga.py script was refactored to support multiple Google service accounts (dangerouscentaur, port_sheet, etc.), allowing centralized credential refresh without duplicating OAuth logic.
  • Staging-then-production deployments: All S3 + CloudFront changes went to staging first, with separate cache invalidations for safety.
  • Turn-limit awareness in task design: The 30-turn exit pattern suggests future tasks should be decomposed into smaller, discrete subtasks to stay within Claude's conversation budget.

What's Next