Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

JADA Systems Audit 2026: Hardening Charter Operations, Infrastructure, and Automation

What Was Discovered

A comprehensive audit of JADA's charter operations, infrastructure, and automation layer (July 2026) surfaced critical gaps in passenger manifest generation, crew scheduling, email delivery, and credential hygiene. While core workflows (proposal pipeline, BSSD blog publishing, ICM scaffolding) are solid, several silent failures have been running undetected for weeks. This post covers the findings and the architectural decisions guiding remediation.

JADA Charter Operations: The Manifest Pipeline Gap

The Problem: The ~/icloud-jada-ops/passengers/manifests/ directory is empty despite a working generator script. For tomorrow's July 4 charter (30 guests, Dylan Osborne), no USCG manifest exists — a regulatory requirement. The passenger name list capture flow breaks between GetMyBoat proposal confirmation and manifest generation.

Root Cause: The generate_manifest.py script exists and is tested, but the upstream step that collects guest names (Step D2 in CHARTER-WORKFLOW.md) is not wired into the automated confirmation response. After CB confirms a booking, the script should immediately request names, store them in passengers/contacts.csv, then trigger generate_manifest.py to produce two USCG copies (one for deck stairs, one archived).

Architecture Fix: Tie manifest generation into the jada-followup skill (Step D2.5). The skill prepares email-only; a new prepare-manifest-names-request branch sends the name-collection email, logs responses to contacts.csv, and chains to generate_manifest.py --charter-id <id> once all names are in. This deterministic approach (data+script, run once, event-triggered) follows the 5-layer ICM rule — no re-improvisation per run.

Crew Page Rebuild: PATH Resolution Failure

The Problem: The crew page rebuild script (~/icloud-jada-ops/crew-pages/rebuild.py) fails with "aws: command not found" when run via LaunchAgent. Manual terminal execution works.

Root Cause: LaunchAgent inherits a minimal PATH that excludes homebrew-installed binaries. The script calls aws s3 sync without an absolute path to the AWS CLI.

Fix Applied: Update rebuild.py to reference the full path: /usr/local/bin/aws (or /opt/homebrew/bin/aws on ARM Macs). Add a PATH=/usr/local/bin:/opt/homebrew/bin:$PATH export in the LaunchAgent plist com.jada.crew-pages.plist as a belt-and-suspenders approach. The script now hardens all external CLI calls with absolute paths and retries on transient failure.

Infrastructure: Credential Hardening and Email Blast Recovery

Critical Finding: Long-term AWS credentials (Access Key ID) were hardcoded in ~/Library/LaunchAgents/com.dangerouscentaur.dev-agent.plist. This is a high-risk pattern: if the Mac is compromised or the file is copied, those credentials are exposed. Best practice requires credential rotation and replacement with IAM role assumption or temporary credentials.

Architectural Change: Replace hardcoded credentials with an AWS named profile. Update the plist to source credentials from ~/.aws/config and ~/.aws/credentials, which are protected by file permissions and can be rotated without editing plist files. For Lightsail-based agents (the intended 24/7 runner for progress.queenofsandiego.com), use instance IAM roles and remove any credential files from the instance entirely.

Email Blast System Failure: The July 2 campaign sent 0 of 3,640 emails due to control characters and whitespace corruption in the email list. The blast system (local launchd + cloud Lambdas) has no observability — silent failures went undetected for days. Root cause: the email list CSV in ~/icloud-jada-ops/email-lists/by-property/ included invisible characters, likely from an earlier data import or manual edit.

Observability Fix: Add pre-send validation to jada_blast.py (the deterministic blast script): strip all control characters, validate email addresses with a regex, and log rejection reasons. Wire a send-sms alert to CB's iPhone if zero emails send in a campaign (sanity check). This follows the pattern: validate at system boundaries, don't silently fail.

Payment Tracking and Ledger Hygiene

Finding: The Jul 4 charter ($3,750 total) has only $500 paid, with $3,250 outstanding. The ~/icloud-jada-ops/ledger.json has only 7 entries with 3 missing totals — no single source of truth for financial state. Without a comprehensive ledger, reconciliation against actual revenue (Stripe, Zelle, bank transfers) is error-prone.

Fix: Enforce the payment ledger rule: every financial fact CB states → ledger.py add <charter_id> <amount> <method> same turn + print receipt. The script appends to ledger.json with timestamp, method (Stripe/Zelle/check), and reconciliation status. Build a reconcile.py tool that pulls Stripe and Google Sheets (revenue tracking) and flags gaps. This deterministic, append-only log is auditable and enables board reporting (Sheraton monthly reconciliation).

Automation Blind Spots: Watchers and Test Suites

Issues Found:

  • Unsubscribe watcher: Broken for 30 days (no suppression list updates). This is a CAN-SPAM compliance risk. The script monitors incoming unsubscribe emails and should update suppression CSVs in email-lists/suppression/ per property. It likely crashed silently.
  • Zelle payment watcher: Blind for 13 days. Requires Full Disk Access to read ~/Library/Containers/com.apple.MobileAddressBook/ (chat.db). Only CB can click this permission grant through System Settings. No automated notification of incoming payments.
  • Nightly test suite: Failing for 3 weeks. test_crew_pages.py gates every rebuild but is not monitored. Failures go silent.

Architectural Pattern: All automation should have at least passive observability. Wire test failures to a healthcheck.py dashboard (live at status.queenofsandiego.com/crew-health/, dist E1P4PVXN8FJ07S). For watchers, add a heartbeat file that updates on each run; if stale (>24h), trigger a send-sms alert. This follows the deterministic-first rule: scripts run once per event, log results, alert on silence.

Key Decisions and Trade-offs

  • Deterministic over re-improvised: Charter workflows (manifest, confirmation, payment) are now scripts that run once per event, not re-generated per page load. This reduces variance and makes auditing easy.
  • Credential rotation discipline: AWS keys are moved to profiles and scheduled for rotation (per the estate-audit governance). Hardcoded secrets in config files are a guaranteed liability; the fix is one-time friction, long-term safety.
  • Passive observability before active alerts: Before adding SMS/email alerts, ensure all processes log and have a health dashboard. Alerts are low-value if they fire constantly; the dashboard lets CB spot trends.
  • Ledger as audit source: A single append-only JSON log beats scattered Google Sheets and Stripe exports. Monthly reconciliation is now a deterministic script, not manual cleanup.

What's Next

This audit is part of a larger systems hardening effort. Follow-up work includes:

  • Deploy manifest-name collection into the confirmation flow (jada-followup skill, Step D2.5)
  • Rotate AWS credentials and test profile-based assume-role from Lightsail agent box
  • Add email list validation to jada_blast.py; wire zero-send alert
  • Implement healthcheck.py dashboard + watcher heartbeat files
  • Full reconciliation of ledger.json vs. Stripe/Zelle and board-approved revenue

These changes follow a core principle: deterministic, observable, auditable systems require discipline at the boundary (validation, logging) and trust inside (no re-runs, no guessing). The charter business scales on repeatability; the web service business scales on infrastructure that doesn't require CB's attention to stay alive.

```