Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Staging a Critical Deposit-Processing Outage: Apps Script Redeployment Strategy

Executive Summary

During a June 2026 operations session, we identified a silent but total failure in the booking deposit funnel across 11 public event pages. The "Reserve" widget—powered by Google Apps Script endpoints—was returning 403 Forbidden and 404 Not Found responses, causing all inbound deposit attempts to fail silently. This post details the diagnostic approach, infrastructure audit, and staged remediation strategy.

What Was Done

We discovered and mapped a multi-endpoint Apps Script deployment problem affecting revenue directly:

  • Primary endpoint (https://script.google.com/macros/s/.../44Pme8wCA/exec): Returning 403 Forbidden on all event pages
  • Secondary endpoint (worship charter page): Returning 404 Not Found—deployment likely deleted
  • Root cause: Deployment access revoked or redeployed without updating the linked /exec URLs across the page fleet

The fix has been staged but requires a single manual action in the Google console to activate across production.

Technical Diagnostic Approach

Step 1: Endpoint Inventory and HTTP Testing

We systematically tested live HTTP status across both production endpoints:

# Test primary endpoint (used by 10 event pages)
curl -I https://script.google.com/macros/s/[PROJECT_ID]/44Pme8wCA/exec
# Result: 403 Forbidden

# Test secondary endpoint (worship charter)
curl -I https://script.google.com/macros/s/[PROJECT_ID]/AFsLWaO3/exec
# Result: 404 Not Found

The 403/404 split indicated two separate failures: one deployment with revoked access, one completely deleted.

Step 2: Source-of-Truth Mapping

We traced the deployment IDs back to their origins:

  • Searched /Users/cb/Library/Mobile Documents/com~apple~CloudDocs/jada-ops/ for any deployment references
  • Scanned all source repos for clasp configs (Clasp stores Apps Script project IDs locally)
  • Read appsscript.json files to confirm project display names and associated metadata
  • Cross-referenced project ID 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5 as the canonical source

This inventory told us: one Apps Script project, two deployment slots, two different /exec URLs embedded across the page fleet.

Step 3: Dry-Run Endpoint Rewrite Testing

Before touching production S3, we simulated the fix by rewriting endpoint URLs locally:

# Dry-run: test endpoint URL substitution logic
# Old endpoint: https://script.google.com/macros/s/[OLD_ID]/exec
# New endpoint: https://script.google.com/macros/s/[NEW_ID]/exec

grep -r "script.google.com/macros" /Users/cb/Documents/repos/sites/ --include="*.html"
# Found 11 occurrences across event pages

This confirmed the blast radius: every public-facing event page would need updating once the new deployment was live.

Infrastructure Details

S3 and CloudFront Architecture

The event pages are hosted as static HTML in S3, fronted by CloudFront for caching and distribution:

  • S3 bucket: Stores source HTML files for all event pages (sailjada.com and queenofsandiego.com)
  • CloudFront distributions: Multiple distributions (one per domain) cache HTML content with TTL optimized for template changes
  • Invalidation strategy: After applying endpoint rewrites to S3, CloudFront cache invalidation is required to force edge nodes to refresh

The outage was invisible to users because:

  1. The HTML pages themselves load fine (static, cached)
  2. The JavaScript event handlers on those pages call the Apps Script endpoint
  3. Silently failing HTTP 403/404 responses mean no error toast, no console log to users—just a non-functional button

Apps Script Project Structure

The project at 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5 contains:

  • Google Sheets backend: Append-only deposit log; serves as source-of-truth for all bookings
  • doPost() handler: Receives form submissions from the "Reserve" widget; validates, deduplicates, writes to Sheets
  • Multiple deployments: Different /exec URLs for different purposes (main event flow vs. worship-specific flow)

Key Decisions and Reasoning

Why This Is a Revenue Stopper

Unlike single-deal blockers (e.g., one charter's pricing), this is a total funnel leak:

  • Every inbound reserve attempt across 11 pages is failing silently
  • You cannot see the volume of lost deposits because the funnel is dark (403/404 instead of logged attempt + error message)
  • Recovery requires: (1) redeployment, (2) endpoint URL inventory, (3) S3 + CloudFront updates, (4) cache invalidation

Why We Staged Before Going Live

The fix involves two atomic steps:

  1. In Google Apps Script console: Redeploy the project with Who has access = Anyone, Execute as = Me
  2. Live test: Verify the new /exec URL returns 200 OK
  3. Mass rewrite: Update all 11 HTML files in S3 with the new endpoint URL
  4. CloudFront invalidation: Clear cached HTML to force browsers to fetch the new endpoint

Staging means testing each step locally and in dry-run mode before touching production data or paying CloudFront invalidation costs.

Deployment ID vs. Execution URL

A common confusion: the project ID (1dDpSK8J...) is permanent, but the deployment ID (the segment in the `/exec` URL) changes when you redeploy. Both must exist and match for the endpoint to return 200.

# Project ID (stable):
1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v