Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Stabilizing the Jada Agent Daemon: OAuth Token Failures and Turn Limit Management

During this development session, we performed a comprehensive health audit of the jada-agent orchestrator daemon running on our primary Lightsail instance (34.239.233.28). The investigation revealed that while the core daemon is stable and responsive, a critical OAuth token failure in the port sheet sync process requires immediate remediation. This post documents the diagnostic approach, findings, and next steps.

What Was Done

  • Established SSH access to the Lightsail instance via AWS temporary credentials (since the local jada-key private key was unavailable)
  • Verified jada-agent.service status and uptime metrics
  • Analyzed daemon logs and session activity over the past 24 hours
  • Identified a broken Google OAuth token in port_sheet_sync.py preventing routine syncs
  • Documented recurring "max turns (30)" exit patterns in complex task runs
  • Assessed resource utilization (CPU, memory, disk) and confirmed healthy baselines

Technical Details: Daemon Health Baseline

Service Status: The jada-agent.service has been running continuously since May 10, 2026—3 days of uptime with zero unexpected restarts. The systemd unit is properly configured and starts reliably on boot.

Resource Utilization:

  • CPU: 0.65% average during idle periods (60-second poll loop baseline)
  • Memory: 144 MB / 914 MB allocated—well within safe thresholds
  • Disk: 6.2 GB / 39 GB used (17%)—ample headroom for logs and task artifacts
  • Load average: 0.00—the instance sits essentially idle between task executions
  • Status checks: Zero failures in the preceding 2 hours

Session Activity (May 13, UTC): The daemon executed three agent sessions today:

  1. Session 1 (00:00 UTC): Reached the 30-turn Claude API limit and exited with code 1. This is a soft termination—not a crash—and the daemon logs it without crashing.
  2. Session 2 (00:02 UTC): Completed successfully. Processed blockers for the e-signature and crew page functionality, and created a needs-you task flagged for manual review.
  3. Session 3 (00:05 UTC): Again hit the 30-turn limit and exited with code 1.

After session 3, the daemon found no new tasks in the queue and returned to idle state. This is expected behavior. Yesterday's pattern showed a hard stop at 5/5 session usage before midnight UTC, with 3 pending tasks queued. The midnight rollover reset the session counter, and those tasks were picked up and cleared.

Critical Issue: Broken OAuth Token in port_sheet_sync

The most significant finding is a persistent authentication failure in the port_sheet_sync.py script. Every 30-minute sync attempt since at least the afternoon of May 13 has failed with:

[port-sheet] token error: HTTP Error 400: Bad Request

Root Cause: The Google OAuth token stored for the port_sheet_sync service has expired or been revoked. This prevents the script from authenticating with the Google Sheets API and syncing port assignment data.

Impact: Port sheet synchronization has not run for at least 12+ hours. Any changes to crew assignments or port availability made via the Google Sheet are not propagating to the operational systems.

Why This Happened: Google OAuth tokens have a finite lifetime (typically 1 hour for access tokens, with refresh tokens valid for 6 months unless revoked). If the token was not refreshed before expiration, or if the underlying credentials were revoked (e.g., password change, security event), subsequent API calls fail with HTTP 400. The jada-agent daemon does not auto-revive dead tokens—it requires manual re-authentication.

Resolution Path: We need to re-run the OAuth flow for the port_sheet_sync service. This involves executing the auth_ga.py utility with the correct service account or user credentials to generate a fresh token and store it securely.

Infrastructure & Architecture Decisions

SSH Access Strategy: Because the local jada-key private key was not available in the usual ~/.ssh directory, we leveraged AWS Systems Manager Session Manager and the Lightsail API to retrieve temporary SSH credentials. This approach required:

  • Querying the Lightsail API endpoint for the instance (34.239.233.28) to fetch temporary credentials
  • Writing the temporary key to a secure temporary file with restricted permissions (600)
  • Using the cert as an OpenSSH certificate paired with the temporary private key
  • Removing the temporary files after the session ended

This pattern is more resilient than relying on a single stored private key, and it leaves no persistent credentials on the local machine.

Daemon Polling & Task Queue: The jada-agent daemon implements a simple 60-second poll loop. It checks the progress dashboard (our task queue) for new work, executes eligible sessions (respecting the 5-session daily limit), and logs results. The daemon is intentionally simple: it does not retry failed tasks or re-auth broken tokens. This keeps the daemon lightweight but means external failures (like expired OAuth) must be handled by manual intervention or a separate remediation job.

Session Limits & Turn Limits: The daemon respects two distinct limits:

  • Session Limit: Maximum 5 agent sessions per 24-hour UTC day. Once reached, the daemon stops picking up tasks and waits for the midnight rollover.
  • Turn Limit: Each Claude API session has a 30-turn conversation limit. Complex tasks that require more than 30 turns will exit with code 1 and leave the task incomplete. The daemon does not retry automatically.

The "Max Turns" Exit Pattern

Two of today's three runs hit the 30-turn limit. This is a design constraint, not a bug. However, it indicates that certain tasks (like the e-signature and crew page work) are inherently complex enough to exhaust the conversation window. Options to address this:

  • Increase turn limit: Raise the 30-turn ceiling if the Claude API pricing allows and if we want longer sessions.
  • Task decomposition: Break large tasks into smaller, independent subtasks that fit within a single session.
  • Context carryover: Implement a mechanism to save session state and resume in a new session (more complex).

For now, the strategy is to monitor which task categories hit the limit and decide whether decomposition is warranted.

What's Next

  • Re-authenticate port_sheet_sync: Run the auth_ga.py tool to refresh the Google OAuth token and restore 30-minute sync intervals. Verify that at least one successful sync occurs post-authentication.
  • Monitor turn-limit exits: Track whether the two max-turns exits today represent a one-off or a recurring pattern. If recurring, evaluate task decomposition or increased turn limits.
  • Implement token rotation policy: Establish a periodic (e.g., weekly) re-auth check for all OAuth-dependent services to prevent token expiration surprises.
  • Review jada-agent logs retention: