Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

Remote Claude Runner Setup & Ticket-Lane Debugging on Lightsail: Restoring 11 Stuck Tickets

What Was Done

JADA's ticket-automation pipeline runs on a dedicated Lightsail instance (jada-agent, 34.239.233.28) as a systemd service that polls progress.queenofsandiego.com for card updates and executes Claude-based workflows. A misconfigured environment variable (TICKET_LANE=todo instead of TICKET_LANE=agent-work) had blocked the runner from seeing its actual work queue for 4+ days, leaving 11 tickets pending: Keeli email automation, unsubscribe cleanup, tech-blog indexing, crew dispatch tasks, and captain hour reconciliation.

This session verified the Lightsail runner's billing configuration, debugged the lane routing, and fixed the systemd service to resume processing.

Technical Details: Plan-Billed vs. API-Billed Runners

JADA runs two Claude accounts in parallel:

  • Mac laptop (jadasailing@gmail.com): plan-billed, interactive use only
  • Lightsail box (c.b.ladd@gmail.com): plan-billed, 24/7 service runner

The Lightsail instance is explicitly not API-metered. The box's ~/.bashrc contains a commented-out ANTHROPIC_API_KEY (marked #DISABLED-PLAN-BILLING) to prevent fallback to metered usage. All tokens route through plan billing: 

$ env -u ANTHROPIC_API_KEY claude -p "verify billing"
# Responds via plan, not API

This design prevents runaway API costs if the headless runner hits an infinite loop—the subscription simply absorbs tokens until the box account hits plan limits.

The Ticket-Runner Service & Lane Routing

The systemd service lives at: 

/etc/systemd/system/jada-ticket-runner.service

Key environment variables:

  • TICKET_LANE=agent-work — the runner only processes tickets in this Trello lane
  • RUNNER_MODE=auto — automatically sends irreversible actions (SMS, email) to a "needs-you" review queue
  • AWS_PROFILE=queenofsandiego — uses the shared DynamoDB/S3 credentials for state and artifact storage

The runner script polls every 4 hours, processing up to 4 tickets per cycle. State is persisted in ~/icloud-jada-ops/runner-state/ (synced via iCloud on the Mac) and DynamoDB for cross-device awareness.

The debug showed the unit's Environment=TICKET_LANE=todo, which meant the runner was watching the wrong Trello lane entirely. Fix: 

sudo sed -i "s/^Environment=TICKET_LANE=todo/Environment=TICKET_LANE=agent-work/" \
  /etc/systemd/system/jada-ticket-runner.service
sudo systemctl daemon-reload
sudo systemctl restart jada-ticket-runner

Immediate confirmation showed the runner staged its first ticket (Keeli email submission) into the "needs-you" lane, resuming the queue.

Infrastructure: progress.queenofsandiego.com

The progress site (Trello board wrapper) serves from:

  • CloudFront distribution: EJJKHNHYNK4Q5 (staging variant; prod is separate)
  • Origin S3 bucket: queenofsandiego-web
  • Route53 DNS: progress.queenofsandiego.com → CloudFront alias
  • Cache policy: no cache on /api/* and /runner/* routes (deterministic API responses)

The ticket-runner polls this site's /api/cards?lane=agent-work endpoint, which pulls live card state from a Trello integration layer. Notes added to cards trigger the runner's next cycle.

macOS Full Disk Access & Claude Code Self-Updates

Mid-session, the SMS read tool began failing on ~/Library/Messages/chat.db even though permissions were previously granted. Root cause: Claude Code self-updated from 2.1.199 → 2.1.200, and macOS's Full Disk Access grant is versioned per binary path. The new binary lost access.

Fix (30 seconds, no restart needed):

  • System Preferences → Security & Privacy → Full Disk Access
  • Remove the old Claude Code entry
  • Toggle new Claude Code entry on

This recurs on every self-update. The symptom (chat.db: permission denied) is now documented in the SMS utility memory for future session handoffs.

Key Decisions & Rationale

Why Lightsail, not Lambda? The ticket-runner needs persistent state, file access to iCloud sync, and a stable IP for Namecheap API calls (IP-whitelisted for domain operations). Lambda's ephemeral environment and cold-start latency would require redesign.

Why plan billing, not API? Fixed monthly cost is predictable for a 24/7 agent. API billing scales with usage and offers no protection against runaway loops. Plan billing also aligns the account with JADA's interactive use on the Mac, simplifying auth maintenance.

Why systemd env vars, not config files? Environment variables are version-controlled via the service unit itself, reducing sync friction. The runner can be redirected to a different Trello lane with a single systemctl restart, no code deploy needed.

What's Next

The runner will now drain the 11 pending tickets over the next 3–4 cycles, sending externally-visible actions (SMS confirmations, unsubscribe processing) to the "needs-you" lane for manual review. The Keeli email, tech-blog indexing, and Daniela trip-sheet tasks are already staged. Captain hour reconciliation (Darrell's time ledger) is queued behind them.

Ongoing: macOS Full Disk Access will need re-toggling on the next Claude Code self-update. Consider building a small launchd plist monitor to detect this and alert via SMS, rather than waiting for failures.