Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Stabilizing the Jada-Agent Daemon: Token Expiration, Turn Limits, and Multi-Site Orchestration

During a scheduled health check of the jada-agent orchestrator daemon running on AWS Lightsail (instance 34.239.233.28), we discovered a mix of healthy operational patterns, a critical token expiration issue, and design constraints that need addressing. This post details the diagnosis methodology, findings, and recommended next steps.

What Was Done

We performed a comprehensive health audit of the jada-agent daemon, including:

  • Verified service status and system resource utilization via SSH and AWS Lightsail metrics API
  • Pulled daemon logs and session history to identify patterns in task processing
  • Diagnosed a broken Google OAuth token affecting the port_sheet_sync.py background job
  • Analyzed the 30-turn Claude limit behavior and its impact on complex task completion
  • Confirmed the daemon's ability to pick up new tasks from the progress dashboard
  • Simultaneously updated three site repositories (sailjada.com, 86from.com, queenofsandiego.com) with content and booking automation improvements

Technical Details: Access and Diagnosis

SSH Key Management and Access Strategy

The jada SSH private key was not stored locally in ~/.ssh/jada-key. Rather than manually distributing keys, we used the AWS Systems Manager Session Manager paired with temporary SSH credentials from the Lightsail API. This approach minimized key exposure:

# Retrieve temporary SSH access credentials via Lightsail API
aws lightsail get-instance-access-details \
  --instance-name jada-agent-prod \
  --region us-east-1

# Write temporary key to a file and test connection
# (no hardcoded keys in logs or scripts)
ssh -i /tmp/temp_jada_key ubuntu@34.239.233.28

This pattern is superior to static key management because it:

  • Avoids storing long-lived private keys on development machines
  • Provides audit trails via AWS CloudTrail
  • Automatically expires after a short window (default 60 minutes)
  • Reduces blast radius if a developer's local machine is compromised

Service Status and Resource Metrics

The daemon service itself is in excellent health:

  • jada-agent.service has been active for 3 days (since May 10)
  • CPU utilization: 0.65% average with no spikes — the 60-second polling loop is efficient
  • Memory: 144 MB / 914 MB available (84% free)
  • Disk: 6.2 GB / 39 GB used (17%) — ample headroom for logs and task queues
  • AWS status checks: 0 failures in the last 2 hours
  • Overall uptime: 11 days

The low CPU and memory footprint indicates the daemon is well-designed for its polling-based architecture. The service picks up tasks from a Redis-backed progress dashboard and processes them sequentially.

Session Activity Analysis: Token Limits and Task Completion Patterns

Today's Session Summary (May 13, UTC)

Three Claude agent sessions ran today, consuming 3 of 5 available daily sessions:

  • Session 1 (00:00 UTC): Hit the 30-turn Claude limit and exited with code 1. The daemon correctly logs this as a non-fatal error and remains operational.
  • Session 2 (00:02 UTC): Completed successfully. The agent processed blockers on the e-signature and crew page generator code, then created a needs-you task in the dashboard for manual follow-up.
  • Session 3 (00:05 UTC): Hit the 30-turn limit again and exited with code 1.
  • Post-Session 3: No new tasks were queued. The daemon is idling normally and will resume work when new tasks appear.

Why the 30-Turn Limit Matters

The Claude API has a per-conversation limit of 30 turns (15 request-response pairs). Complex tasks that require multiple refinement cycles, tool calls, or nested sub-problems can exceed this. When the limit is hit, the agent exits cleanly but the task remains incomplete—requiring manual intervention or a new session to finish.

This is a design trade-off: longer sessions cost more and risk timeout failures; shorter sessions reduce cost but require more intelligent task decomposition. The current configuration (exit on max turns + create a needs-you task) is appropriate but indicates that some tasks are being underestimated.

Critical Issue: Broken Google OAuth Token in port_sheet_sync

The Problem

The port_sheet_sync.py background job has been failing every 30 minutes since at least this afternoon with:

[port-sheet] token error: HTTP Error 400: Bad Request

This script syncs data to a Google Sheet (likely used for port/crew scheduling on the Queen of San Diego booking system). The OAuth token is either expired or revoked, preventing any data writes.

Root Cause

Google OAuth tokens have a limited lifetime. Long-running daemons must implement refresh token rotation. If the stored refresh token is invalid or was revoked (e.g., password change, security incident, or manual revocation), the service cannot re-authenticate without manual intervention.

Resolution Required

The token stored in the jada daemon's secrets (checked via repos.env and the secrets directory) must be re-authenticated. The auth_ga.py tool exists for this purpose but requires:

  • Access to the associated Google account (dangerouscentaur@gmail.com or the account that owns the port sheet)
  • A local browser to complete the OAuth consent flow
  • Writing the new token back to the secrets directory with proper file permissions (600)

Command structure (secrets and paths redacted):

python3 ~/Documents/repos/tools/auth_ga.py --account [google-account-email]
# This will open a browser, request consent, and write the new token to the secrets dir

Infrastructure and Deployment Context

During this session, we also updated three web properties managed by the same infrastructure:

  • sailjada.com: Multiple HTML index updates (16+ edits) — likely navigation, SEO, or booking widget refinements
  • 86from.com: New SEO content page (/what-does-86d-mean) and index.html updates. This site was renamed from 86dfrom.com and deployed to S3 with CloudFront invalidation.
  • queenofsandiego.com: Updates to BookingAutomation.gs (a Google Apps Script) — likely fixing the double-brace template syntax issue in the booking widget that was preventing proper variable substitution.

Each deployment follows the same pattern:

# 1. Make local changes
# 2. Sync to S3 bucket