Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

# Article: Diagnosing and Staging a Multi-Region Google Apps Script Deployment Outage

Diagnosing and Staging a Multi-Region Google Apps Script Deployment Outage: Deposit Widget Failure Across 11 Event Pages

What Was Done

Over a 6-hour session, we diagnosed a complete revenue stoppage affecting all public "Reserve" deposit widgets across 11 event pages. The root cause was a Google Apps Script deployment misconfiguration that had revoked public access, causing silent booking failures. We traced the outage from live HTTP endpoints back to the source Apps Script project, mapped all dependent pages, and staged the remediation without requiring code changes or S3 redeployment.

  • Verified both Apps Script endpoints were returning 403 (access denied) and 404 (deployment deleted)
  • Located the source Google Apps Script project by cross-referencing deployment IDs with clasp configs
  • Identified all 11 event pages dependent on the broken endpoints
  • Staged a single-action fix (redeployment with "Anyone" access) that requires no code or S3 changes

Technical Details: Root Cause Analysis

The deposit system uses two Google Apps Script endpoints, each deployed to handle specific event cohorts:

  • Primary endpoint: https://script.google.com/macros/s/.../44Pme8wCA/exec — serves 10 event pages (sailjada.com main reserve widgets)
  • Secondary endpoint: https://script.google.com/macros/s/.../AFsLWaO3/exec — serves 1 page (worship/specialized events)

Both were returning hard failures:

$ curl -i https://script.google.com/macros/s/.../44Pme8wCA/exec
HTTP/1.1 403 Forbidden
$ curl -i https://script.google.com/macros/s/.../AFsLWaO3/exec
HTTP/1.1 404 Not Found

The 403 indicated the deployment exists but public access is revoked. The 404 suggested the secondary deployment had been deleted entirely. Neither state allows unauthenticated form submissions from the public-facing reserve widgets.

Discovery Process

We located the source by:

  • Searching the jada-ops cloud notes directory for references to the Apps Script project name
  • Checking all clasp configuration files (`.clasp.json`) in the source repos to find matching project IDs
  • Reading the appsscript.json manifest for the project display name and deployment metadata
  • Verifying AWS SES suppression lists were not the cause (SES account health checked separately)

The Apps Script project ID is 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5, stored in source control's .clasp.json file. This is distinct from the deployment ID (the string in the /exec URL) — a critical distinction because redeploying to the same slot preserves the endpoint URL, meaning no S3 changes are required.

Infrastructure: Endpoint Topology and Dependencies

The deposit system spans three layers:

Layer Component Status
Frontend 11 event pages (S3 static sites, distributed via CloudFront) Live, healthy, but serving broken form endpoints
Backend 2 Google Apps Script deployments (public-facing HTTP endpoints) Down (403/404)
Storage Google Sheets (deposit records) + DynamoDB (crew metadata) Not reachable due to endpoint failure

All 11 event pages are hosted in S3 buckets and served through CloudFront. Each contains an embedded HTML form that POSTs to one of the two Apps Script endpoints. Because the endpoints are down, form submissions silently fail — no error is surfaced to the user, and no deposits are recorded.

Key Decisions and Trade-Offs

Why Not Re-Deploy the Code?

The initial impulse is to re-run clasp push && clasp deploy to force a new deployment. However, this approach has a hidden cost: a fresh deployment generates a new URL slug, which requires updating all 11 S3 pages and flushing the CloudFront cache. Instead, we can redeploy to the same deployment slot by managing the deployment's access control directly in the Google Apps Script console, which preserves the URL and requires zero S3 changes.

Why Check DDB and SES First?

The outage could have been caused by downstream issues (e.g., SES delivery quota exceeded, DynamoDB throttling, or IAM permission regression). We verified:

  • SES account health in both regions (us-east-1 and eu-west-1)
  • SES suppression list size and recent unsubscribe rate
  • DynamoDB table access and scan counts

All were healthy, confirming the failure is at the entry point (Apps Script endpoint access), not downstream.

Staging the Fix

The remediation is a single manual action in the Google Apps Script console:

  1. Navigate to script.google.com and open project 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5
  2. Go to Deploy → Manage deployments
  3. For each live deployment:
    • Set "Who has access" to "Anyone"
    • Set "Execute as" to "Me" (the Apps Script project owner)
    • Click Deploy
  4. Verify both endpoints return 200 and serve the expected form handler response

This action:

  • Requires no code changes
  • Preserves the existing endpoint URLs
  • Requires no S3 or