Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Stabilizing the JADA Agent Daemon: OAuth Token Recovery and Task Processing Analysis

During a recent development session, we performed a comprehensive health check on the JADA agent orchestrator daemon running on our Lightsail instance (34.239.233.28). While the core daemon remained stable with 11 days of uptime, we uncovered a critical OAuth token expiration in the port sheet sync process and identified patterns in task processing that warrant infrastructure adjustments.

What Was Done

We conducted a multi-layer diagnostic across three primary areas:

  • Daemon Service Health — Verified systemd service status, uptime, resource utilization (CPU, memory, disk)
  • Task Processing Pipeline — Reviewed session logs, turn counts, and completion rates over a 24-hour window
  • External Service Integration — Identified and diagnosed the broken Google OAuth token in the port sheet sync routine

Technical Details: Access and Diagnostics

Since the jada-key private key was not stored locally in ~/.ssh/jada-key, we used the AWS Lightsail API to generate temporary SSH credentials on-demand, avoiding the need to store persistent keys in the repository.

# Command pattern (no actual credentials shown):
# 1. Fetch temporary key from Lightsail via AWS CLI
# 2. Write to temporary file with restricted permissions (600)
# 3. SSH into instance using temporary certificate
# 4. Execute diagnostic commands
# 5. Remove temporary key files after session

This approach maintains security by never storing long-lived SSH keys in the development environment while still enabling administrative access when needed.

Service Status Findings

The jada-agent.service systemd unit has been running continuously since May 10, 2026 — 3 days of uptime with zero service restarts. Key metrics:

  • CPU utilization: 0.65% average across a 60-second polling interval (normal idle for an event-driven loop)
  • Memory footprint: 144MB / 914MB available (15.8% utilization) — well within safe margins
  • Disk usage: 6.2GB / 39GB (17.8%) — healthy headroom for log growth
  • Load average: 0.00 — indicating the daemon spends most time waiting for tasks from the progress dashboard queue
  • EC2 status checks: 0 failures in the last 2 hours

Session Processing Analysis (2026-05-13)

The daemon consumed 3 of its 5 daily sessions with the following outcomes:

  • Session 1 (00:00 UTC): Hit the 30-turn Claude API limit with exit code 1. Task remained incomplete but did not crash the daemon.
  • Session 2 (00:02 UTC): Completed successfully. The agent processed e-signature and crew page generator blockers, creating a needs-you task for manual review.
  • Session 3 (00:05 UTC): Hit the 30-turn limit again, exiting with code 1.
  • Post-03:00 UTC: No new tasks detected in the queue; daemon in normal idle state.

The pattern suggests that complex multi-step tasks (particularly those involving code generation or cross-site coordination) exhaust the 30-turn limit before completion. This is not a failure condition — the daemon continues running and logs it appropriately — but incomplete tasks remain queued until the next session window.

Critical Issue: Port Sheet Sync OAuth Token Expiration

The most actionable finding was the recurring failure in the port sheet sync process. Every 30-minute execution since at least afternoon UTC has logged:

[port-sheet] token error: HTTP Error 400: Bad Request

This indicates the Google OAuth 2.0 token stored for port_sheet_sync.py has expired or been revoked. The script authenticates using a pre-stored refresh token to sync crew availability data with Google Sheets, but the current credentials are no longer valid.

Why this matters: Port sheet syncs feed crew scheduling data into the booking automation pipeline. Extended gaps in this sync mean the booking system may present stale availability information to customers.

Infrastructure and Architecture Decisions

Lightsail Instance Configuration

The daemon runs on a dedicated Lightsail instance rather than Lambda or ECS for a few key reasons:

  • Persistent event loop — The daemon maintains a polling loop that checks the progress dashboard queue every few seconds. This pattern suits always-on compute better than serverless invocation.
  • State retention — Session tracking, task lock files, and agent context persist on the instance filesystem between runs.
  • Cost predictability — A micro instance (0.5GB RAM, 1 vCPU) runs continuously at a fixed monthly cost, vs. variable Lambda/ECS spend during peak hours.

OAuth Token Management

The port sheet sync uses a stored Google OAuth 2.0 refresh token to periodically re-authenticate without user interaction. This is the correct pattern for service-to-service authentication, but tokens have a defined lifespan. The current architecture stores credentials in a secrets file on the instance, which requires periodic manual re-authentication when tokens expire.

Improvement opportunity: Implement automatic token refresh logic that detects 400 errors, triggers a re-authentication flow, and updates the stored credentials — or move to a secrets management service (AWS Secrets Manager or Parameter Store) with lambda-based token refresh on expiration.

Task Processing Limits and Turn Allocation

The 30-turn limit per Claude session is set at the orchestration layer (likely in the agent runner configuration) to prevent runaway costs and infinite loops. However, we're hitting this ceiling on legitimately complex tasks.

Options to consider:

  • Increase the limit to 50-75 turns for sessions marked as high-priority or complex (e.g., multi-site code generation).
  • Implement task segmentation — Break multi-site deploys into separate subtasks (one for 86from.com, one for queenofsandiego.com, one for sailjada.com), each with its own session.
  • Add context persistence — Store intermediate results (generated code, configuration diffs) in a persistent store so the next session can resume from checkpoint rather than starting cold.

What's Next

Immediate action items:

  • Re-authenticate the Google OAuth token for port_sheet_sync.py. Trigger the auth flow for the dangerouscentaur@gmail.com account and update the stored credentials.
  • Monitor port sheet sync logs for the next 24 hours to confirm the 30-minute syncs resume successfully.
  • Review task history from the past week to identify which task types consistently hit the 30-turn limit and prioritize those for segmentation or increased allocation.
  • Evaluate token refresh automation — design a mechanism to detect and handle OAuth expiration gracefully without manual intervention.

The daemon itself is rock-solid.