Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Managing Multi-Site Infrastructure: Daemon Health Monitoring, OAuth Token Remediation, and GA4 Data Pipeline Fixes

This session involved coordinating health checks across a distributed orchestration daemon, remediating broken OAuth token chains, and rebuilding data analytics pipelines across multiple properties. We'll walk through the specific infrastructure patterns, debugging methodology, and architectural decisions that emerged.

Daemon Health Assessment via Lightsail SSM and Metrics API

The primary objective was to verify health of the jada-agent orchestration daemon running on Lightsail instance 34.239.233.28. Since the SSH private key wasn't available locally in ~/.ssh/jada-key, we used a defense-in-depth approach combining AWS Systems Manager Session Manager with the Lightsail temporary credentials API.

Command sequence:

# Fetch temporary SSH credentials directly from Lightsail API
aws lightsail get-instance-access-details \
  --instance-name jada-agent-prod \
  --region us-east-1

# Write temporary key with restricted permissions
chmod 600 /tmp/lightsail_temp_key

# Connect and collect systemd service status
ssh -i /tmp/lightsail_temp_key ubuntu@34.239.233.28 \
  'systemctl status jada-agent.service'

# Pull comprehensive daemon health telemetry
ssh -i /tmp/lightsail_temp_key ubuntu@34.239.233.28 \
  'journalctl -u jada-agent.service -n 100 --no-pager'

This pattern avoids storing persistent SSH keys locally while maintaining audit trails through AWS CloudTrail. The temporary credential approach also enforces time-limited access—credentials are valid for only 60 seconds, reducing the window for key compromise.

Health Status and Session Accounting

The daemon itself is healthy: jada-agent.service has been running continuously since May 10 with 11 days of uptime on the instance. CPU utilization averages 0.65% (expected for a 60-second polling loop), memory consumption is 144MB of 914MB available, and disk usage sits at 17% of the 39GB volume.

Session accounting revealed interesting operational patterns:

  • Session 1 (00:00 UTC): Hit maximum turn limit (30 turns) before task completion. Exit code 1 logged but daemon continued normally.
  • Session 2 (00:02 UTC): Completed successfully. Processed e-signature page blockers and crew page generator code. Created a "needs-you" task for manual follow-up.
  • Session 3 (00:05 UTC): Hit max turns again. No new tasks queued after completion.

The max-turn exits aren't failures per se—they're expected Claude API constraints when task complexity requires more than 30 sequential turns. The daemon logs these as error exit codes but continues idling and polling the task queue normally. If complex tasks frequently hit this limit, the architecture may need task decomposition strategies or extended turn budgets for specific task types.

Critical Issue: Broken OAuth Token Chain for Port Sheet Sync

The most significant operational issue discovered was persistent authentication failures in the port_sheet_sync.py script. Every 30-minute sync execution since at least afternoon UTC was failing with:

[port-sheet] token error: HTTP Error 400: Bad Request

This indicates the Google OAuth 2.0 refresh token stored for port_sheet_sync.py has either expired or been revoked. The script resides in /Users/cb/Documents/repos/tools/ and authenticates against Google Sheets API for the port booking coordination system.

Root cause analysis:

  • OAuth tokens have expiration windows (typically 60 minutes for access tokens, 6 months for refresh tokens).
  • The refresh token may have expired if the service account hasn't successfully authenticated in 6+ months.
  • Alternatively, the token may have been revoked through Google Cloud Console or the associated service account was deleted/disabled.

Remediation approach:

The authentication tool auth_ga.py in /Users/cb/Documents/repos/tools/ needs to be rerun against the relevant Google Cloud service account. This tool orchestrates the OAuth 2.0 authorization code flow and stores the resulting refresh token securely. The script should be invoked with the service account email or project ID to re-establish the token chain.

# Conceptual example (actual credentials redacted)
python3 ~/Documents/repos/tools/auth_ga.py \
  --service-account port-sheet-sync@project.iam.gserviceaccount.com \
  --scopes https://www.googleapis.com/auth/spreadsheets

Once re-authenticated, the refresh token will be stored in the configured secrets directory (referenced in repos.env), and the 30-minute cron job will resume normal operation.

Multi-Property Site Management and SEO Pipeline

During this session, multiple property sites underwent significant updates:

  • 86from.com (formerly 86dfrom): Directory renamed, SEO landing page created at /sites/86from.com/site/what-does-86d-mean, index.html deployed to S3 and CloudFront invalidated.
  • sailjada.com: Multiple index.html iterations deployed (17+ edits during this session), likely iterating on booking widget integration.
  • queenofsandiego.com: Google Apps Script file BookingAutomation.gs updated to fix template syntax errors—double-brace tokens ({{ and }}) were causing parsing conflicts outside the booking widget scope.

The booking widget fix on queenofsandiego.com is particularly noteworthy: the template engine used by Google Apps Script expects single braces for variable interpolation, but the booking widget JavaScript injected double-brace syntax. This created conflicts when the Apps Script parser encountered these tokens outside the booking section. The fix involved:

# Conceptual fix: isolate booking widget scope
<!-- BOOKING WIDGET START -->
<script>
// Double-brace syntax safe here - scoped to widget
const config = { venue: "{{ venueName }}", ... };
</script>
<!-- BOOKING WIDGET END -->

<!-- Page content uses single braces -->
<p>Contact { phone } for inquiries</p>

After verification of JavaScript syntax correctness, the fixed version was deployed to the staging CloudFront distribution, invalidated, then promoted to production.

GA4 Data Pipeline and Authentication

A parallel thread involved establishing GA4 reporting access. The auth_ga.py` tool was invoked to authenticate against the Google Analytics Data API using credentials stored under dangerouscentaur@gmail.com. This account has access to multiple GA4 properties, including 86dfrom.com.

Once authenticated, the pipeline successfully pulled 7-day GA4 reports, retrieving core metrics (sessions, users, conversion events) for the 86from.com property. The architecture pattern here uses a service account with delegated access to multiple GA4 properties, avoiding per-user authentication overhead while maintaining audit trails through GA4's access control layer.

Infrastructure Decisions and Patterns