Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Resolving Agent Daemon Health Issues: OAuth Token Expiration and Turn Limits

During a routine health check of the jada-agent orchestrator daemon running on our Lightsail instance (34.239.233.28), we discovered a critical OAuth token failure in the port sheet sync pipeline and identified a secondary pattern where complex tasks were hitting Claude's turn limits. This post details the diagnosis process, root cause analysis, and remediation strategy.

What Was Done

We performed a comprehensive health audit of the jada-agent.service daemon including:

  • SSH access via AWS Lightsail API-generated temporary credentials (primary key not stored locally)
  • Service status verification and uptime analysis
  • Real-time metrics collection (CPU, memory, disk, network) via CloudWatch/Lightsail APIs
  • Log analysis across daemon runs from the past 24 hours
  • Task queue inspection and session accounting
  • OAuth token validation for dependent services

The daemon itself is operationally healthy, but two distinct issues emerged requiring immediate attention.

Technical Details: The Port Sheet Sync Token Failure

The port_sheet_sync.py script, which runs on a 30-minute schedule, has been failing consistently since at least early afternoon on 2026-05-13. Every invocation returns:

[port-sheet] token error: HTTP Error 400: Bad Request

This is a Google OAuth 2.0 token expiration or revocation issue. The script uses a stored refresh token to maintain access to a Google Sheet (likely the crew/port scheduling sheet based on naming), but the token has become invalid.

Why this matters: Port sheet synchronization is upstream to several workflows. If crews or port information isn't syncing into the dashboard, downstream task generation and crew assignment may operate on stale data. This is a data freshness risk, not a service availability risk—the daemon continues running, but the sync pipeline is broken.

Root cause: Google OAuth tokens can expire or be revoked if:

  • The refresh token itself expired (typically 6 months if unused)
  • The user revoked access at accounts.google.com/connected-apps
  • Google revoked the token due to security policy changes
  • The token was never properly persisted after initial auth

Technical Details: Claude Turn Limit Hits

Of the three daemon sessions that ran today, two exited with code 1 after hitting the 30-turn maximum. The daemon logs these as errors, but they're not service failures—they're task incompleteness indicators.

Session breakdown (UTC, 2026-05-13):

  • 00:00: Hit max turns (30) → exit code 1 (task incomplete)
  • 00:02: Completed successfully → processed e-signature/crew page blockers, created needs-you task
  • 00:05: Hit max turns (30) → exit code 1 (task incomplete)

The successful session (00:02) did meaningful work despite the apparent rapid cycling. The turn limit exists to prevent runaway API spend and to enforce bounded execution in multi-agent systems. However, when complex tasks require more than 30 turns to resolve, they fail to complete and are likely re-queued for the next daemon cycle.

Why this pattern occurs: The jada-agent is a multi-step reasoning system. Tasks involving:

  • Code generation and testing (e.g., crew page generator code mentioned in logs)
  • Cross-service validation (checking e-signature link availability)
  • Schema transformation (converting between internal formats and external APIs)

...naturally consume more turns. At 30 turns per session, some classes of work simply won't fit in a single execution window.

Infrastructure and Service Architecture

Daemon configuration: The jada-agent.service is a systemd service running on the Lightsail instance with:

  • Uptime: 3 days (since 2026-05-10)
  • Load average: 0.00 between tasks (expected for event-driven daemon)
  • CPU utilization: 0.65% average (minimal baseline)
  • Memory footprint: 144MB / 914MB available (16% used)
  • Disk usage: 6.2GB / 39GB (17% utilization)
  • Status checks: 0 failures in last 2 hours (healthy infrastructure)

Session management: The daemon runs with a hard limit of 5 sessions per 24-hour UTC rollover period. Today's consumption breakdown:

  • Sessions used: 3 of 5
  • Remaining quota: 2 sessions
  • Expected reset: Midnight UTC (next calendar day)

This quota prevents accidental runaway execution and enforces deliberate task planning. After session 3, no new tasks were queued; the daemon correctly idles until new work appears on the progress dashboard.

Key Decisions and Remediation Path

OAuth token re-authentication: The port_sheet_sync.py script must be re-authenticated against the Google Sheets API. This requires:

  • Running the auth tool (e.g., /path/to/repos/tools/auth_ga.py or equivalent auth script for Sheets API)
  • Using the service account or user credentials that own the port sheet
  • Storing the refresh token securely (ensure file permissions are locked: chmod 600 on token files)
  • Testing the sync manually before re-enabling the cron schedule

Turn limit strategy: Rather than immediately raising the turn limit, we should:

  1. Decompose tasks: Split complex tasks (e.g., "fix e-sig and crew page blockers") into smaller, self-contained subtasks that fit within 30 turns.
  2. Instrument task metadata: Log turn consumption per task type to identify which classes of work consistently exceed the limit.
  3. Increase incrementally: If task decomposition proves unworkable, raise the limit in 10-turn increments (30 → 40 → 50) and monitor cost/performance impact.
  4. Monitor API cost: Each turn consumes Claude API tokens. Increasing limits will increase monthly spend proportionally.

Why decomposition first: A 30-turn limit on a single task is inefficient but manageable if the task is broken into subtasks. The successful session (00:02) completed its work and created a human-readable task card (needs-you task), demonstrating the system can emit intermediate checkpoints. Using those checkpoints to structure future work will improve observability and resilience.

What's Next

  • Immediate (next 2 hours): Re-authenticate the Google Sheets API token for port_sheet_sync.py and verify the 30-minute sync runs successfully.
  • Short-term (next 24 hours): Analyze task logs from the two turn-limit failures to identify which task types exceeded capacity