Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

Debugging a Multi-Layered Daemon Health Issue: SSH Access, Google OAuth Token Expiry, and Agent Turn Limits

During a routine infrastructure health check on the JADA orchestrator daemon (running on AWS Lightsail instance 34.239.233.28), we discovered a working daemon with two distinct failure modes: an expired Google OAuth token blocking port sheet syncs, and recurring agent turn limits causing incomplete task processing. Here's how we diagnosed the issue, what we found, and what needs fixing.

Challenge: Accessing a Lightsail Instance Without a Local Private Key

The initial blocker was straightforward but instructive: the jada-key private key wasn't stored in the expected location (~/.ssh/jada-key), and it wasn't discoverable via environment variables in repos.env. Rather than spending time hunting through backup locations, we used two parallel approaches:

  • AWS Systems Manager Session Manager: Opened a managed SSH session without requiring a local private key file, relying instead on IAM instance profile permissions.
  • Lightsail API temporary credentials: Called the GetInstanceAccessDetails API endpoint to request temporary SSH access credentials, which provided a time-limited key pair we could write to disk and use for traditional SSH.

The second approach was necessary because the daemon's health check required pulling detailed logs and metrics that work better over a persistent SSH connection. Here's the pattern we used:

aws lightsail get-instance-access-details \
  --instance-name jada-agent-prod \
  --region us-east-1

The response included a temporary private key (valid for 15 minutes) and a public key we added to the instance's authorized_keys. We then wrote the key to a temporary file with proper permissions (600) and connected via SSH. Critically, we deleted the temporary key file immediately after use to avoid leaving credentials on disk.

The Daemon Health Picture: 3 Days Uptime, Idle Load, and One Active Problem

The jada-agent.service itself is healthy:

  • Active and running since May 10 (3 days of uninterrupted uptime)
  • CPU utilization steady at ~0.65% average (normal for a 60-second polling loop)
  • Memory footprint minimal: 144 MB of 914 MB available
  • Disk usage reasonable: 6.2 GB of 39 GB (17%)
  • Lightsail status checks: zero failures in the last 2 hours
  • Load average: 0.00 between tasks, indicating the daemon properly yields CPU when idle

We pulled CPU and network metrics directly from the Lightsail monitoring API to rule out transient spikes:

aws lightsail get-instance-metric-statistics \
  --instance-name jada-agent-prod \
  --metric-name CPUUtilization \
  --start-time 2026-05-13T15:00:00Z \
  --end-time 2026-05-13T17:00:00Z \
  --period 300 \
  --statistics Average,Maximum

No anomalies. The daemon is stable.

Session Consumption and Agent Turn Limits: The Real Issue

The daemon ran three sessions today (UTC):

  • Session 1 (00:00 UTC): Hit max turn limit (30 turns) and exited with code 1. No task completion.
  • Session 2 (00:02 UTC): Completed successfully. Processed e-signature and crew page blockers, created a needs-you task for manual review.
  • Session 3 (00:05 UTC): Hit max turn limit again and exited with code 1.

After session 3, the daemon found no new tasks and has been idling normally—expected behavior. The exit code 1 on max-turn sessions is not a crash; the daemon logs it and continues its polling loop. However, it means some tasks are being abandoned mid-processing.

This is a design constraint worth understanding: the agent is Claude-based, and each session consumes turns (API calls). We've set a hard limit of 30 turns per session to control costs and prevent runaway loops. When a complex task needs more than 30 turns, the session terminates, the task stays pending, and the next daemon cycle picks it up—but no progress is saved between sessions.

Session 2's success shows the system can work; simpler tasks complete within the turn budget. Complex tasks (like generating boilerplate code or debugging multi-file issues) are hitting the limit.

The Critical Issue: Expired Google OAuth Token in port_sheet_sync.py

This is the blocker. The port_sheet_sync.py script runs every 30 minutes as a daemon subprocess and syncs data to a Google Sheet via the Google Sheets API. The OAuth token is expired or revoked, causing all syncs to fail with:

[port-sheet] token error: HTTP Error 400: Bad Request

We identified this by tailing the daemon's error log and cross-referencing with the cron/subprocess logs. The token file is stored at a path managed by the jada configuration, and it needs to be refreshed via OAuth re-authentication.

The auth script already exists at /Users/cb/Documents/repos/tools/auth_ga.py, which handles Google OAuth token refresh. However, note that the script was moved or deleted during our session work—we need to ensure it's restored and can be run in the Lightsail environment to re-auth the port_sheet_sync credentials.

Infrastructure and Deployment Context

The daemon runs on a single AWS Lightsail instance in us-east-1. Related infrastructure includes:

  • S3 buckets: Staging and production sites are deployed to S3 (e.g., 86from.com site files)
  • CloudFront distributions: Content served via CloudFront with cache invalidation on deploy
  • Route53: DNS for jada.sailjada.com and related domains
  • Google Analytics: Tracking via GA4 with credentials managed locally and synced to port sheets

During this session, we also updated site files for 86from.com (formerly 86dfrom.com), deployed to S3, and invalidated the CloudFront cache. The daemon was idle throughout, confirming it doesn't block on deployment tasks.

Key Decisions and Lessons

  • SSH access pattern: Using Lightsail's temporary credential API rather than hunting for keys is faster and more secure. Always clean up temporary keys immediately.
  • Turn limits: The 30-turn limit is a cost-control measure, but it's causing task abandonment. We should either increase the limit for production runs (if budget allows) or redesign complex tasks to fit within the budget by breaking them into smaller subtasks.
  • OAuth token management: Credentials stored on disk need refresh workflows. We need automated alerts or a dashboard to detect expired tokens before they break downstream processes.

What's Next

  • Restore auth_ga.py to the repos/tools directory and run it to refresh the port_sheet_sync OAuth token.
  • Monitor port_sheet_sync logs for the next 24 hours to confirm syncs resume.
  • Review the agent's turn-limit pattern. If turn-limit exits remain