Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Stabilizing the Jada Agent Daemon: OAuth Token Recovery and Turn-Limit Monitoring

Over the past development session, we performed a comprehensive health check on the jada-agent orchestrator daemon running on AWS Lightsail instance 34.239.233.28. The investigation revealed a healthy service core with one critical OAuth integration failure and a recurring pattern of Claude API turn-limit exhaustion that warrants architectural review.

What Was Done

  • Established SSH access to the Lightsail instance via temporary credentials from the Lightsail API (after discovering the private key wasn't stored locally)
  • Collected comprehensive service health telemetry: uptime, CPU/memory utilization, disk usage, and network status checks
  • Extracted and analyzed 24 hours of agent session logs from the progress dashboard
  • Identified a broken Google OAuth token in the port_sheet_sync.py integration
  • Documented the recurring pattern of max-turn (30-turn limit) exits in complex multi-step tasks
  • Verified that the daemon itself remains stable despite downstream integration failures

Technical Details: Daemon Health Status

Service Metrics (as of 2026-05-13 18:00 UTC):

  • jada-agent.service status: Active and running continuously since May 10 (3 days uptime)
  • Instance uptime: 11 days with a load average of 0.00 — indicating the daemon is properly idle between task polls
  • CPU utilization: 0.65% average across a 60-second poll cycle, with no observed spikes
  • Memory footprint: 144 MB of 914 MB available (15.8% utilization) — well within safe operating parameters
  • Disk usage: 6.2 GB of 39 GB (17%) — ample headroom for logs and temporary task artifacts
  • Status checks: Zero failures recorded in the past 2 hours via Lightsail's automated health checks

The daemon's polling loop—which runs every 60 seconds to fetch new tasks from the progress dashboard—is functioning nominally. There are no signs of memory leaks, CPU thrashing, or network connectivity issues.

Session Activity Analysis (May 13, UTC)

The daemon executed three agent sessions across the day:

  • Session 1 (00:00 UTC): Exited with code 1 after hitting the 30-turn Claude API limit. No work was persisted.
  • Session 2 (00:02 UTC): Completed successfully. The agent processed e-signature and crew page generation blockers, created a needs-you task for manual follow-up, and exited cleanly.
  • Session 3 (00:05 UTC): Again hit the 30-turn limit and exited with code 1. Partial work may have been committed.
  • Post-Session (00:05 onwards): Daemon transitioned to idle polling. No new tasks were queued in the dashboard, consistent with normal end-of-batch behavior.

Yesterday's behavior showed a similar pattern: the daemon consumed all 5 allocated daily sessions before midnight and hit the hard session limit. At midnight rollover, queued tasks were cleared as designed.

Critical Issue: Broken Google OAuth Token

The most significant finding is the systematic failure of the port_sheet_sync.py script, which is responsible for syncing data to Google Sheets.

Symptom: Every 30-minute sync cycle since at least this afternoon has logged:

[port-sheet] token error: HTTP Error 400: Bad Request

Root cause: The Google OAuth 2.0 token stored for this service account has either expired or been revoked. The token file is likely located in ~/.jada/secrets/google_oauth_token.json or a similar path on the Lightsail instance, and it no longer grants the required scopes (typically https://www.googleapis.com/auth/spreadsheets).

Impact: Port sheet syncs have been non-functional for an unknown duration (at minimum, since this afternoon). Any downstream processes depending on synchronized port data are operating on stale information.

Why this happened: Google OAuth tokens have a finite lifetime (typically 1 hour for access tokens, with refresh tokens valid for 6 months unless revoked). If the refresh token was invalidated—either by user action, a security event, or token rotation on Google's end—the daemon cannot obtain a new access token.

Secondary Pattern: Claude API Turn Limits

Two of the three agent sessions today (Sessions 1 and 3) exited with code 1 after exhausting the 30-turn conversation limit with Claude. This is not a daemon crash; the service correctly logs these as errors and continues polling:

systemctl status jada-agent.service
# Output: active (running) since Thu 2026-05-10 15:23:14 UTC

However, the recurring nature of this pattern suggests that task complexity is outpacing the allocated turn budget. A 30-turn limit is appropriate for simple, well-scoped tasks (e.g., "update the homepage CSS"). Complex tasks requiring multi-step reasoning, API calls, file manipulation, and error recovery routinely exceed this budget.

Why this matters: Session 1's work was lost entirely. Session 3 may have partially committed work before hitting the limit. Session 2, which completed successfully, produced tangible output (the needs-you task), suggesting that task decomposition and clear success criteria help the agent finish within the turn budget.

Infrastructure and Access Patterns

The investigation revealed an important operational detail: the jada-key private SSH key was not stored in the expected location (~/.ssh/jada-key). Instead, we obtained temporary SSH credentials via the AWS Lightsail API:

aws lightsail get-instance-access-details \
  --instance-name jada-agent-prod \
  --region us-east-1 \
  --query 'accessDetails.{cert:certKey,user:username,host:ipAddress}' \
  --output text

This retrieves a temporary certificate valid for a limited time window, paired with a private key. This is a more secure pattern than storing long-lived SSH keys on the local machine, as it leverages AWS IAM for key rotation and audit logging.

Key Decisions and Rationale

  • Lightsail API instead of stored keys: Since the jada-key wasn't available locally, using AWS Lightsail's temporary credential API ensured we didn't need to store or transmit sensitive key material. This also provides better audit trails via CloudTrail.
  • Metrics-first diagnostics: We pulled CPU, memory, disk, and network metrics before attempting SSH, allowing us to identify whether the issue was performance-related or application-specific. This narrowed the investigation scope.
  • Log-based task reconstruction: Rather than relying solely on real-time daemon state, we extracted the full session history from logs to understand the pattern of failures over time. This revealed that yesterday's HARD STOP at 5/5 sessions was expected behavior, not a regression.

What's Next

The immediate action items are:

  • Re-authenticate Google OAuth for port