Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Staging a Critical Deposit Widget Outage: Apps Script Access Control and Multi-Region Deployment Recovery

The Problem: Silent Booking Funnel Failure

During a routine operational briefing on June 2nd, we discovered that the "Reserve" deposit widgets across all 11 event pages in the sailjada and queenofsandiego.com properties had gone silent. End users attempting to book charters and submit deposits were hitting a wall: the Apps Script endpoints serving the reservation forms were returning either 403 Forbidden or 404 Not Found responses.

The critical insight: this wasn't a regional issue or a traffic spike. This was a total stoppage on the revenue funnel, and because the failures were silent to users (no error messaging, just broken buttons), we had no visibility into how many bookings were being lost in real time.

Root Cause Analysis: Deployment Access Control and Project Fragmentation

The investigation revealed two distinct failure modes:

  • Primary endpoint (10 event pages): https://script.google.com/macros/s/AKfycbxxxxxxxxxxxxxxxxxxxxxxx44Pme8wCA/exec returning 403 Forbidden — the deployment existed but access control was revoked
  • Secondary endpoint (worship page): https://script.google.com/macros/s/AKfycbxxxxxxxxxxxxxxxxxxxxx_AFsLWaO3/exec returning 404 Not Found — the deployment had been deleted entirely

Both endpoints corresponded to the same Google Apps Script project (ID: 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5), but with different deployment slots. The root cause was a mismatch between:

  • Who had deployment access in the Google Cloud Console
  • The access control settings on each deployment slot ("Specific people" vs. "Anyone")
  • The execution context (running as "Me" vs. running as the script owner)

Technical Diagnosis Process

We performed a systematic audit using the following approach:

  1. Live HTTP verification — tested both endpoints directly with curl and inspected response headers to confirm the exact failure modes
  2. Source repository scan — searched /Documents/repos and the jada-ops CloudDocs folder for any hardcoded endpoint references or clasp configuration files
  3. Apps Script project inventory — checked ~/.clasp/clasp.json and inspected appsscript.json metadata files to map project IDs to deployment slots
  4. S3 and CloudFront audit — verified that all 11 event pages were still serving the old endpoints (no client-side caching issues, just server-side access revocation)
  5. Google Cloud IAM audit — confirmed the Apps Script project was present and accessible but deployments lacked the required "Execute" role for external callers

The Fix: Redeployment with Correct Access Control

The staging solution involved a controlled redeployment with corrected access settings:

1. Navigate to script.google.com and open project ID: 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5
2. Click "Deploy" → "Manage deployments"
3. For each deployment slot:
   - Set "Who has access" to "Anyone"
   - Set "Execute as" to "Me" (the Apps Script owner/account)
   - Click "Deploy"
4. Verify the /exec URL remains unchanged (same deployment ID)

Why this matters: redeploying into the same deployment slot preserves the /exec URL that the client pages already reference. If we had created a new deployment slot, we'd have needed to update 11 separate HTML files across S3 and trigger a CloudFront cache invalidation — a much riskier operation.

Infrastructure Details

The event pages and their deposit widgets live in two primary S3 buckets:

  • sailjada.com bucket: Contains 7 event pages, each with embedded <form> elements that POST to the Apps Script endpoint
  • queenofsandiego.com bucket: Contains 4 event pages plus a dedicated worship donation page, each with similar widget integrations

Both buckets are distributed via CloudFront with custom domains (cdn.sailjada.com and cdn.queenofsandiego.com). The event pages themselves are deployed via a GitHub Actions workflow that syncs source content from /Documents/repos/sites/ to S3.

The dry-run process involved:

# Verify no hardcoded endpoint rewrites needed
grep -r "script.google.com/macros/s" /Documents/repos/sites/*/
grep -r "AKfycb" /Documents/repos/sites/*/

# Test endpoint responses before applying fixes
curl -I https://script.google.com/macros/s/AKfycbxxxxxxxxxxxxxxxxxxxxxxx44Pme8wCA/exec
curl -I https://script.google.com/macros/s/AKfycbxxxxxxxxxxxxxxxxxxxxx_AFsLWaO3/exec

Why This Outage Was Critical

Unlike a partial failure or a single broken page, this was a complete revenue funnel collapse across all properties:

  • No error logging on the client side meant ops had zero visibility
  • No monitoring alerts were configured on the Apps Script endpoints themselves
  • The failure was silent — users saw a non-functional button, not a clear error message
  • Every booking attempt in the past hours (or days) was lost with no recovery path

Cost of delay: each hour the funnel remains down is revenue evaporating with no tracking mechanism.

Key Decisions and Lessons

  1. Preserve deployment IDs: We explicitly chose to redeploy into existing slots rather than create new ones. This avoided forced client-side changes and CloudFront invalidations.
  2. Access control over code changes: The fix required zero code modification. It was purely an IAM/deployment configuration issue in Google Cloud.
  3. Monitoring gap identified: We now have a standing action to add synthetic monitoring to all Apps Script endpoints — hourly health checks that alert on non-200 responses.
  4. Deployment documentation: The project configuration and deployment settings are now documented in /jada-ops/DEPOSIT-OUTAGE-FIX-2026-06-02.md for future reference.

What's Next

Post-fix validation includes:

  • Live endpoint testing across all 11 pages (curl + browser)
  • End-to-end deposit submission test (full form submission → Apps Script → backend database)
  • CloudFront cache verification (confirm pages serve fresh content within 30 seconds)
  • Implement Apps Script