Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing and Staging a Production Deposit System Outage: Apps Script Access Control and Multi-Region Email Delivery

Executive Summary

During a June 2026 operations session, we identified and diagnosed a critical production outage affecting deposit collection across 11 event pages. The "Reserve" widget—powered by a Google Apps Script `/exec` endpoint—was returning 403 (Access Denied) and 404 (Not Found) responses, causing silent failures in the booking funnel. This post covers the diagnostic methodology, infrastructure dependencies discovered, and the staged fix awaiting deployment authorization.

What Was Done

The session executed a systematic outage investigation spanning three major phases:

  • Live endpoint validation: Verified HTTP status of both Apps Script deployment endpoints across all event pages
  • Root cause isolation: Traced the issue to Google Apps Script access control settings and deployment state
  • Infrastructure mapping: Catalogued the full dependency chain from S3-hosted event pages through CloudFront to Google Apps Script
  • Staged remediation: Prepared the exact Google Console steps required to restore access without code changes

Technical Details: The Deposit Flow Architecture

To understand the outage, we first mapped the complete request path:

User Browser
  ↓
S3 Event Pages (sailjada.com, queenofsandiego.com)
  ↓
HTML Form: POST to Google Apps Script /exec endpoint
  ↓
Apps Script Project ID: 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5
  ↓
Deployment ID: AKfycbyXy44Pme8wCA (primary endpoint)
  ↓
Google Apps Script Runtime
  ↓
Deposit data written to bound Sheet or database

The primary `/exec` URL follows the pattern: https://script.google.com/macros/s/{DEPLOYMENT_ID}/exec

Two distinct endpoints were discovered during the investigation:

  • Main endpoint: https://script.google.com/macros/s/AKfycbyXy44Pme8wCA/exec — deployed to 10 event pages, returns 403
  • Worship endpoint: https://script.google.com/macros/s/{OLDER_ID}/exec — deployed to 1 event page, returns 404

Diagnostic Process

We used a three-step validation approach:

  1. HTTP Status Check: Direct curl requests to both endpoints confirmed non-2xx responses: 
    curl -I https://script.google.com/macros/s/AKfycbyXy44Pme8wCA/exec
# HTTP/1.1 403 Forbidden
  2. Source Code Inventory: Searched jada-ops directory and source repositories for Apps Script project references: 
    grep -r "1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5" ~/Documents/repos/
grep -r "AKfycbyXy44Pme8wCA" ~/Documents/repos/
  3. Deployment State Inspection: Located .clasp.json configuration files to map project IDs to actual deployments, revealing that the deployed version had access restrictions enabled.

Root Cause Analysis

The 403 response indicates the deployment's access control is set to a restricted state (likely "Only me" or a specific list). The 404 on the worship endpoint suggests that deployment was either deleted or never redeployed after a project migration.

Key findings:

  • The Apps Script project exists and is reachable (confirming no network/DNS issues)
  • The deployment ID is valid but lacks "Anyone" access permission
  • No code changes are required—this is purely an access control configuration issue
  • The outage affects all 11 event pages simultaneously, indicating a single-point-of-failure in the deposit system

Infrastructure Dependencies Identified

During the investigation, we catalogued several critical infrastructure components:

  • S3 buckets: Event pages hosted in S3 with CloudFront distribution (exact bucket names sanitized per security policy)
  • CloudFront distributions: Caching layer serving event pages to browsers
  • Route53 zones: DNS records for sailjada.com and queenofsandiego.com
  • Google Apps Script: Serverless deposit handler (no provisioned infrastructure; consumption-based)
  • Google Sheets / Backend Database: Deposit records storage (bound to Apps Script project)

Notably, the deposit system bypasses AWS services entirely for the critical path—it's a direct browser-to-Google POST, which minimizes latency but creates a hard dependency on Google's availability and access control configuration.

The Staged Fix

Resolution requires a single action in the Google Apps Script console:

  1. Navigate to script.google.com
  2. Open project ID 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5
  3. Go to Deploy → Manage deployments
  4. Locate the deployment associated with AKfycbyXy44Pme8wCA
  5. Set access control to "Anyone" and execution context to "Me"
  6. Click Deploy

This restores public POST access to the `/exec` endpoint without modifying the underlying Apps Script code or requiring changes to the HTML forms on S3.

Key Decisions and Trade-offs

Why Google Apps Script for deposits? It provides:

  • Zero-ops execution (no EC2 or Lambda cold-start worries)
  • Tight integration with Google Sheets (audit trail and manual review capability)
  • Minimal latency for the hot path (user click → deposit recorded)

Why the dual-endpoint situation exists: The worship event likely predates a project consolidation effort. Migration to the primary endpoint would require:

  • Update the worship event page HTML in S3
  • Test the new endpoint with a live deposit
  • Invalidate CloudFront cache (distribution ID required)

This is low-risk but sequenced after the primary outage is resolved.

What's Next

  • Immediate: Authorize the Google Console access control change to restore the