Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Stabilizing the JADA Agent Daemon: OAuth Token Failures and Turn Limit Management

During a routine health check of the JADA orchestrator daemon running on AWS Lightsail (34.239.233.28), we discovered a critical OAuth token failure affecting the port sheet sync pipeline, alongside architectural insights into session management and turn-limit constraints. This post details the diagnostics performed, findings uncovered, and the remediation strategy now underway.

What Was Done

We conducted a comprehensive health audit of the jada-agent.service daemon, including:

  • Verified service uptime and process health via SSH
  • Analyzed CPU, memory, and disk utilization over the past 2 hours
  • Reviewed daemon logs for the past 24 hours to identify error patterns
  • Examined session counts, task completions, and error rates
  • Identified a persistent OAuth token failure in the port_sheet_sync script
  • Assessed the impact of Claude API turn-limit constraints on task completion

Technical Details: Service Health and Resource Metrics

Service Status Overview

The jada-agent.service has been running continuously since May 10, 2026—a 3-day uptime window with no service crashes or restarts. Instance-level metrics show:

  • CPU Utilization: 0.65% average over the polling window (60-second intervals). No spike events detected in the past 2 hours, indicating normal idle behavior between task executions.
  • Memory Usage: 144 MB / 914 MB allocated (15.8% utilization), well within acceptable bounds.
  • Disk Usage: 6.2 GB / 39 GB (17% utilization). Sufficient headroom for log rotation and future deployments.
  • Network Status Checks: Zero failures in the past 2 hours; instance networking is stable.
  • System Load: Load average of 0.00, reflecting expected idle state between task submissions.

Session Activity Log (May 13, 2026)

The daemon executed 3 sessions today with the following outcomes:

  • Session 1 (00:00 UTC): Hit the 30-turn Claude API limit; exited with code 1. This is a soft failure—the daemon logs it but does not crash or restart.
  • Session 2 (00:02 UTC): Completed successfully. The agent processed e-signature page blockers and crew page generator code, creating a task in the progress dashboard for manual follow-up.
  • Session 3 (00:05 UTC): Hit the 30-turn limit again; exited with code 1. No further tasks were queued after this run; the daemon entered idle state as expected.

This pattern suggests that complex tasks requiring iterative refinement are consuming the full turn budget before completion. Session 2's success demonstrates that the daemon can complete meaningful work within the constraint, but multi-faceted tasks may require session chaining or task decomposition.

Critical Issue: OAuth Token Failure in Port Sheet Sync

Problem Description

The port_sheet_sync.py script, which maintains synchronization between the booking automation system and Google Sheets, has been failing every 30 minutes with a consistent error:

[port-sheet] token error: HTTP Error 400: Bad Request

This error began appearing at least this afternoon (May 13) and likely earlier. All port sheet syncs executed after the failure point have been silently skipped, meaning the booking data is now out of sync with the canonical sheet.

Root Cause Analysis

The OAuth 2.0 refresh token stored for the Google Sheets API is either:

  • Expired: The refresh token has exceeded its lifetime (typically 6 months for offline-mode tokens if unused).
  • Revoked: The token was manually revoked via Google Account Security settings or an OAuth scope mismatch has been detected.
  • Scope Mismatch: The token's authorized scopes no longer include the required Google Sheets API permissions.

The error occurs during the token refresh step in the Google Auth library, preventing any downstream API calls to the Sheets API from executing.

Infrastructure Context

The port_sheet_sync.py script is deployed as part of the JADA agent's cron-driven auxiliary task suite on the Lightsail instance. The token is stored in a secrets file (path structure: /home/jada/[redacted]/port_sheet_token.json) and loaded at script startup. The script runs every 30 minutes via cron, pulling booking data from the automation system and pushing updates to a shared Google Sheet.

Infrastructure and Architecture Patterns

Daemon Architecture

The JADA agent uses a pull-based task execution model:

  • A REST endpoint (the "progress dashboard") maintains a queue of pending tasks.
  • The daemon polls this queue at a fixed interval (typically 60 seconds).
  • When tasks are present, the daemon spawns a new session with Claude, passing the task description and context.
  • The session has a hard turn limit of 30 turns, after which it terminates and logs the exit state.
  • Upon session completion, the daemon records results (success, failure, or turn-limit hit) and continues polling.

Turn-Limit Constraints

The 30-turn limit per session is a cost and latency control mechanism. Each turn corresponds to a request-response pair with the Claude API. Complex, multi-step tasks can exceed this budget, causing the session to terminate prematurely. The daemon does not automatically retry or requeue partial tasks; instead, it logs the exit code and waits for manual intervention or re-queuing via the dashboard.

Auxiliary Task Architecture

The port_sheet_sync.py` and similar auxiliary scripts run outside the main agent loop via system cron. They have independent error handling and logging but share the same secrets management layer (stored on the instance filesystem). A failure in one auxiliary script does not affect the main daemon's health or session execution.

Key Decisions and Remediation Strategy

Why This Matters

The port sheet sync failure is not immediately visible to end users but represents data drift over time. Booking records may diverge between the automation system and the canonical sheet, leading to reconciliation issues and potential double-bookings or lost data. The OAuth token failure must be addressed urgently.

Remediation Steps (In Progress)

  • Re-authentication: The Google OAuth flow for the port sheet service account must be re-run using auth_ga.py or an equivalent OAuth flow script. This will generate a fresh refresh token and update the stored credentials.
  • Scope Verification: Ensure the OAuth scopes include https://www.googleapis.com/auth/spreadsheets and any other required scopes for the target sheets.
  • Secrets Management Hardening: Lock down file permissions on the secrets directory to 0700 (read/write/execute for owner only). This prevents accidental exposure and enforces principle of least privilege.