Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Resolving a Distributed Agent Orchestrator: Health Checks, OAuth Token Failures, and Task Queue Management

During a routine maintenance session on the jada-agent daemon infrastructure, we discovered a critical OAuth token failure in the port sheet synchronization pipeline, along with some edge cases in the agent's turn-limit handling. This post details the diagnostic approach, infrastructure patterns used, and the lessons learned from operating a distributed task orchestrator on AWS Lightsail.

Initial Challenge: SSH Access Without Stored Credentials

The jada-agent service runs on a Lightsail instance at 34.239.233.28. The initial request was to SSH in using a local jada-key private key, but the key wasn't present in the standard ~/.ssh/ directory. Rather than request manual key distribution, we used the AWS Lightsail API to temporarily generate SSH credentials:

aws lightsail get-instance-access-details \
  --instance-name jada-orchestrator \
  --region us-east-1

This returned a temporary private key valid for 60 minutes, eliminating the need to store long-lived SSH keys on the development machine. The approach follows the principle of ephemeral credentials—we generated credentials on-demand, used them immediately, then discarded them afterward.

Infrastructure and Architecture: Lightsail + Systemd + Python Daemon

The jada-agent is a Python-based task orchestrator deployed as a systemd service on AWS Lightsail. Here's what we found:

  • Service: jada-agent.service — Active and running for 3 days uptime
  • Instance specs: 914MB RAM, ~39GB disk, negligible CPU between task cycles (0.65% average)
  • Task loop: Polls every 60 seconds for pending tasks in a progress dashboard
  • Session limits: 5 concurrent sessions per day, 30 turns (Claude API calls) per session
  • Resource usage: 144MB RAM, 6.2GB disk (17% utilization)

The daemon is intentionally lightweight—it spends most of its time idle, waking only when tasks are queued. This design scales well for on-demand task processing without incurring constant compute costs.

Key Finding: OAuth Token Failure in Port Sheet Sync

The most critical issue discovered was in the port_sheet_sync.py script. Every 30-minute sync job since at least May 13 afternoon has been failing with:

[port-sheet] token error: HTTP Error 400: Bad Request

This indicates the Google OAuth token stored for the port sheet synchronization has expired or been revoked. The token is used to authenticate API calls to Google Sheets, and its failure silently broke the entire port sheet sync pipeline.

Why this matters: Port sheet syncs are critical infrastructure for keeping booking and resource data in sync. When the token fails, no errors propagate to alerting systems—the sync just stops, leaving stale data in downstream systems.

Resolution required: Re-authenticate the Google OAuth flow for port_sheet_sync.py. This involves:

  • Running the authentication script in the tools directory to refresh the token
  • Testing the sync with a manual trigger to verify the new token works
  • Adding token expiry alerting to catch this faster in the future

Agent Session Behavior: Max-Turns Exits

During the observation window (May 13, 00:00–00:05 UTC), the daemon executed three sessions:

  • Session 1 (00:00 UTC): Hit 30-turn limit, exited with code 1
  • Session 2 (00:02 UTC): Completed successfully, processed e-signature and crew page blockers, created a downstream task
  • Session 3 (00:05 UTC): Hit 30-turn limit, exited with code 1

Exit code 1 on max-turns is logged as an error but doesn't crash the daemon. The systemd service continues running and picks up new tasks at the next poll cycle. However, hitting the 30-turn limit means some tasks remain incomplete—they must be retried in a subsequent session.

Why this happens: Complex tasks (like debugging multi-file issues or coordinating changes across three sites) naturally require more than 30 Claude API calls. The agent explores code paths, makes hypotheses, tests them, and refines—each step is a turn.

Mitigation options:

  • Increase the turn limit if task complexity justifies it (trades off API costs)
  • Decompose large tasks into smaller, focused subtasks that fit within 30 turns
  • Add smarter task retry logic to resume incomplete work without re-querying context
  • Implement a "checkpoint" system where the agent saves intermediate state at turn 25, enabling resume-on-next-session

Session Quota Management: The Midnight Rollover

The daemon enforces a 5-session-per-day limit, rolling over at midnight UTC. On May 12/13, the daemon hit its session limit before midnight with 3 pending tasks queued. These tasks automatically cleared at the midnight boundary—exactly expected behavior for a quota-aware orchestrator.

This constraint prevents runaway costs and API abuse, but it means high-load days can cause task queue buildup. Current utilization (3 of 5 sessions on May 13) suggests comfortable headroom.

Infrastructure and Deployment Patterns

During this session, several site deployments occurred, showing the full stack:

  • 86from.com (formerly 86dfrom.com): Renamed directory, deployed index.html and new SEO content page to S3 bucket, invalidated CloudFront distribution cache
  • sailjada.com: Multiple HTML refinements deployed; numerous iterations suggest A/B testing or incremental feature rollout
  • queenofsandiego.com: BookingAutomation.gs script updated (Google Apps Script for booking logic)

The deployment pattern uses S3 for static hosting + CloudFront for edge caching. CloudFront cache invalidation ensures new content reaches users immediately rather than waiting for TTL expiry.

Key Decisions and Trade-Offs

  • Ephemeral SSH credentials over stored keys: Reduces key distribution burden and audit surface, at the cost of tighter time windows. For maintenance tasks, this is ideal.
  • Lightweight Lightsail instance with idle polling: Lower cost than always-on compute, suitable for periodic tasks. Trade-off: slight latency on new task pickup (up to 60 seconds).
  • Hard turn limits (30 per session): Prevents runaway costs and API quota exhaustion. Trade-off: some tasks require manual restart or scope reduction.
  • Silent token expiry (no alerting on OAuth failures): Common pattern in background sync jobs, but dangerous. Recommend adding token expiry checks and alerting.

What's Next

  • Re-authenticate port_sheet_sync.