Diagnosing and Staging a Critical Deposit Outage: Apps Script Authorization Failure Across 11 Event Pages

```html

What Happened

A production incident rendered the deposit/reservation functionality inoperable across all 11 event pages on sailjada.com and queenofsandiego.com. The public-facing "Reserve" widgets were silently failing because the underlying Google Apps Script endpoints—responsible for accepting and processing deposit submissions—were returning 403 Forbidden and 404 Not Found responses. This represented a complete revenue stoppage: every inbound booking attempt hit a dead endpoint.

Root Cause Analysis

Two distinct Apps Script deployments were involved in the incident:

Primary deposit endpoint (ID: 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5, exec URL ending in ...44Pme8wCA/exec): Returning 403, indicating the deployment existed but access controls had been revoked.
Worship event endpoint (exec URL ending in ...AFsLWaO3/exec): Returning 404, indicating the deployment had been deleted entirely.

The root cause was a lapsed authorization model: deployments had been set to "Specific people" or similar restrictive access controls, rather than "Anyone," and either the owning account lost permissions or the deployment was removed during maintenance.

Discovery and Diagnosis Process

The investigation followed a systematic approach:

Live endpoint testing: Directly queried both `/exec` URLs via HTTP to confirm 403 and 404 responses in production.
Source audit: Reviewed all 11 event page HTML files stored in S3 to extract the actual exec URLs being called by client-side JavaScript.
CloudFront cache verification: Confirmed that S3 pages were being served through CloudFront without stale cache issues; the problem was upstream (Google Apps Script), not the CDN.
Apps Script project identification: Cross-referenced the project ID from the incident briefing against local clasp configs and appsscript.json metadata to confirm project ownership and history.

Staged Fix and Remediation Path

The remedy is a single administrative action inside the Google Apps Script console, but the technical requirements are clear:

For the primary deposit endpoint:

Navigate to script.google.com and open project 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5.
Access Deploy → Manage deployments.
Locate the active deployment (the one powering the exec URL ending in ...44Pme8wCA/exec).
Set Who has access to Anyone and Execute as to Me (the owner account).
Save and deploy. The 403 should resolve to 200 within seconds.

For the worship endpoint:

If the deployment has been deleted (returning 404), recovery requires either restoring from version control or redeploying. Check the project's version history and the local source repository at ~/Documents/repos/sites/queenofsandiego.com/ for the Apps Script source code, then redeploy to Google's infrastructure.

Infrastructure and Architecture Context

Understanding the full architecture is essential for preventing future incidents:

Event pages: Static HTML files served from S3 bucket (exact bucket name withheld for security), cached through CloudFront.
Client-side deposit form: JavaScript embedded in each event page makes CORS-enabled POST requests to the Apps Script exec URLs. No intermediate API gateway; direct endpoint calls.
Apps Script backend: Two Google Apps Script projects handle form submissions and write deposit records (likely to Google Sheets or a Firestore backend). These are the critical path for all inbound reservations.
No retry logic: Client-side form submission does not retry on failure; a 403 or 404 means the user sees an error immediately.

Key Decisions and Rationale

Why Apps Script and not a custom backend? Rapid deployment, minimal infrastructure overhead, and tight integration with Google Sheets for business operations. The tradeoff is reduced observability and harder debugging when deployments go dark.

Why "Anyone" access instead of service account authentication? Public reservation pages must accept unauthenticated requests. A service account would require either embedding credentials (security risk) or adding a proxy API layer (additional complexity). "Anyone" is the correct choice for a public-facing form.

Why no monitoring? The incident revealed a gap: Apps Script exec URLs are not monitored for 403/404 responses. A simple CloudWatch-based health check (or even a cron job polling the endpoints) would have alerted within minutes, not hours.

What's Next

Immediate: Apply the authorization fix to both deployments and confirm 200 responses on all 11 event pages.

Short-term:

Add HTTP health checks for both Apps Script exec URLs to a monitoring system (CloudWatch, Datadog, or similar).
Document the deployment IDs and access requirements in the ops runbook at ~/Library/Mobile Documents/com~apple~CloudDocs/jada-ops/.
Set up version-controlled backups of both Apps Script projects to prevent accidental deletion.

Medium-term: Consider migrating deposit handling to a Lambda-backed API Gateway endpoint with explicit authentication, logging, and retry logic. This would eliminate the single point of failure in Google's infrastructure and provide better observability.

Commands for Verification

After applying the fix, verify the endpoints are live:

curl -i https://script.google.com/macros/s/AKfycbx.../exec
# Expect: HTTP/1.1 200 OK

Once confirmed, run a synthetic test by submitting a test deposit form from one of the event pages and confirm the backend receives the request.

```