Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Resolving the jada-agent Orchestrator Health Check on Lightsail

During a routine health inspection of our jada-agent daemon running on AWS Lightsail (34.239.233.28), we identified a working orchestrator with one critical OAuth token failure and a recurring pattern of Claude API turn limits. This post documents the diagnostic approach, findings, and remediation path.

What Was Done

We performed a comprehensive health check on the jada-agent.service daemon by:

  • Establishing SSH access to the Lightsail instance using temporary credentials via the Lightsail API (since the private key wasn't stored locally)
  • Verifying service status, uptime, and resource utilization metrics
  • Analyzing daemon logs for the past 24 hours of agent activity
  • Cross-referencing task completion rates with CloudWatch metrics
  • Identifying and isolating the port_sheet_sync OAuth token failure

Technical Details: Access and Diagnostics

SSH Access via Lightsail API

The jada-key private key was not available in the standard ~/.ssh directory. Rather than manually hunting for it across repos or environment configs, we used the AWS Lightsail API to request temporary SSH credentials:

aws lightsail get-instance-access-details \
  --instance-name jada-agent \
  --region us-east-1

This returned a temporary certificate and protocol that allowed one-time SSH access without storing persistent keys locally. This approach is cleaner for CI/CD and temporary debugging scenarios.

Service Status Verification

Once connected, we confirmed the daemon was active and healthy:

systemctl status jada-agent.service
journalctl -u jada-agent.service -n 100

The service has been running continuously since May 10 with 3 days of uptime. Load average is 0.00 between task executions, indicating the daemon is properly idling rather than spinning or leaking resources.

Resource Metrics

CPU utilization averaged 0.65% over the measurement window—normal for a service with a 60-second polling interval checking for pending tasks. Memory usage was 144MB of 914MB available (15.7%), and disk usage was 6.2GB of 39GB (17%). All metrics indicate healthy, underutilized infrastructure.

We also pulled 2-hour CloudWatch metrics via the Lightsail API:

aws lightsail get-instance-metric-statistics \
  --instance-name jada-agent \
  --metric-name CPUUtilization \
  --statistics Average \
  --start-time 2026-05-13T00:00:00Z \
  --end-time 2026-05-13T02:00:00Z \
  --period 300 \
  --region us-east-1

Network traffic and status checks also showed no anomalies.

Session Activity Analysis

The daemon logs revealed three agent sessions executed on 2026-05-13 UTC:

  • Session 1 (00:00 UTC): Hit the Claude API 30-turn limit and exited with code 1. The daemon logged this as an error but continued normal operation.
  • Session 2 (00:02 UTC): Completed successfully. Processed e-signature and crew page blockers, created a needs-you task for manual review.
  • Session 3 (00:05 UTC): Hit the 30-turn limit again, exited with code 1.

After session 3, no new tasks were found in the progress dashboard, and the daemon returned to idle state. This is expected behavior—the daemon polls for pending tasks every 60 seconds and executes them when available.

Exit Code 1 Pattern: The recurring "max turns" exits are not crashes. The daemon continues running. However, they indicate that complex tasks are consuming the full Claude API turn budget before completion. This is a task-design concern, not a daemon failure.

Critical Issue: port_sheet_sync OAuth Token Failure

The most significant finding was a persistent OAuth authentication failure in the port_sheet_sync script.

Every 30-minute sync attempt since at least May 13 afternoon has failed with:

[port-sheet] token error: HTTP Error 400: Bad Request

The port_sheet_sync.py script handles syncing data to Google Sheets via the Google Sheets API. The failure pattern indicates the stored OAuth token is either expired or revoked. This script is likely called by a cron job or systemd timer (stored in /etc/cron.d/ or /etc/systemd/system/timers.target.wants/). Port sheet syncs have not completed successfully for the past several hours.

Root Cause: Google OAuth 2.0 refresh tokens have a default lifetime. If the refresh token was not rotated before expiration, or if the credentials were revoked in the Google Cloud Console, all subsequent API calls fail with HTTP 400.

Infrastructure and Architecture

Lightsail Instance Details

  • Instance: jada-agent at 34.239.233.28
  • Region: us-east-1
  • Service file: /etc/systemd/system/jada-agent.service
  • Daemon binary/script: Likely located in /opt/jada/ or similar (inferred from service management)
  • Agent logs: Accessible via journalctl -u jada-agent.service

Orchestration Pattern

The jada-agent daemon implements a polling-based task orchestrator:

  1. Daemon polls a progress dashboard or task queue every 60 seconds
  2. When a pending task is found, it invokes Claude API for execution
  3. Each session is limited to 30 turns (API calls) before exiting
  4. Session results are logged and task status is updated
  5. Daemon returns to idle until next polling interval

This is a robust pattern for serverless-adjacent workloads, though the 30-turn limit suggests task scope tuning may be needed.

Key Decisions and Why

Why Use Lightsail API for Temporary Credentials: Storing persistent SSH keys in source repositories or local development machines introduces credential exposure risk. Using the Lightsail API to generate temporary, one-time certificates reduces that attack surface and is appropriate for debugging sessions.

Why Check CloudWatch Instead of Just SSH Logs: CloudWatch metrics provide a full historical view. SSH allows us to see daemon-specific logs (systemd journal). Together, they give us both infrastructure-level and application-level visibility.

Why Isolate the port_sheet_sync Issue: It's a separate script failure, not a jada-agent daemon failure. By identifying it distinctly, we can treat it as a Google API credential management issue rather than a daemon architecture problem.

What's Next

Immediate Actions:

  • Re-authenticate the Google OAuth token for port_sheet_sync.py by following the Google