Diagnosing and Staging a Production Deposit System Outage: Google Apps Script Access Control & Multi-Region Endpoint Recovery

```html

What Happened

On June 2, 2026, the deposit/reservation system serving 11 event pages across two domains (sailjada.com and queenofsandiego.com) went silent. All "Reserve" buttons were live in the UI, but clicking them returned HTTP errors instead of booking flows. The root cause: two Google Apps Script deployments had lost their public access grants, one returning 403 (Forbidden) and one returning 404 (Deleted). This was a total funnel stoppage — every inbound booking attempt was failing silently, with no visibility into lost revenue.

Technical Diagnosis Process

The investigation followed a systematic approach to isolate the failure point:

Live HTTP Status Check: Hit both Apps Script endpoints directly from production to confirm the response codes.
Deployment ID Tracking: Cross-referenced the clasp configuration files in source repos to find which Google Apps Script project IDs matched the live endpoints.
Multi-Region Inventory: Searched the jada-ops directory for any historical references to the Apps Script project names, deployment IDs, and endpoint URLs.
Configuration Audit: Read appsscript.json files and .clasp.json configs to verify project metadata and deployment state.

The diagnosis revealed two distinct failure modes:

Endpoint 1 (10 event pages): Returns 403 Forbidden. The Apps Script deployment exists but has access control set to a restricted group (likely a specific Google Workspace or user list), blocking anonymous requests.
Endpoint 2 (worship event pages): Returns 404 Not Found. The deployment was deleted entirely, or the web app configuration was removed.

Infrastructure: The Deposit System Architecture

Before outlining the fix, understanding the architecture is essential:

Frontend Origin: Event pages hosted on S3 buckets (static HTML/CSS/JS) served through CloudFront distributions.
Booking Backend: Google Apps Script web apps acting as serverless HTTP endpoints, receiving form submissions and writing bookings to Google Sheets or backend APIs.
Deployment Pattern: Two separate Apps Script projects, each with a unique deployment ID and /exec endpoint URL.
Access Model: Web apps deployed "Execute as Me" (service account credentials embedded in the deployment) but with "Who has access" set to "Anyone" — critical for public-facing endpoints.

The event pages reference these endpoints via hardcoded URLs embedded in JavaScript form handlers. For example, a reservation form might fetch:

POST https://script.google.com/macros/s/<DEPLOYMENT_ID>/exec

If the deployment loses "Anyone" access, the browser receives a 403 before even reaching the script logic.

Staging the Fix

The fix required two separate actions, one per endpoint:

Action 1: Restore Access on the 403 Endpoint

Location: Google Apps Script console → Project ID 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5

Steps:

Navigate to Deploy → Manage deployments
Select the active deployment (the one ending in ...44Pme8wCA)
Verify Execute as = Me (preserves the original account's permissions for downstream operations)
Change Who has access from the current restricted setting to Anyone
Click Deploy to apply the change

Why this approach: The "Anyone" setting allows the public internet to invoke the web app's doPost() and doGet() handlers without authentication. "Execute as Me" ensures the script runs under the original creator's account context, preserving access to the Google Sheets, Drive files, or third-party APIs it depends on. This is a critical distinction: it separates *who can call the endpoint* from *what identity executes the code*.

Action 2: Restore the 404 Endpoint

Location: Google Apps Script console for the second project (worship event pages)

Context: The 404 response indicates the deployment was removed or never redeployed after a project update. The source code may still exist in the project, but no active web app deployment is serving it.

Steps:

Open the Google Apps Script editor for the second project
Verify the doPost() handler function exists in the main script file
Click Deploy → New deployment
Select Type = Web app
Set Execute as to the service account or original owner
Set Who has access = Anyone
Click Deploy
Copy the new deployment's /exec URL

Critical Decision: If redeploying generates a new endpoint URL (different deployment ID), the event pages referencing the old URL must be updated. However, many Google Apps Script projects support redeploying into the *same* deployment slot, keeping the URL stable. This is preferable because it avoids a simultaneous frontend code push.

Testing and Validation

After staging the fixes (but before pushing to production), validation steps include:

Direct HTTP Test: curl -X POST https://script.google.com/macros/s/<DEPLOYMENT_ID>/exec — expect 200 OK, not 403 or 404.
Response Payload: Inspect the JSON or HTML response to confirm the Apps Script is executing and returning expected data structures.
Form Submission Dry-Run: Load an event page in the browser, fill out the reservation form, and verify the submission is accepted without errors.
Backend Verification: Check the destination (Google Sheet, database, or webhook) to confirm the booking record was created.

Why This Outage Happened

While the immediate cause is access control changes, the underlying vulnerability is architectural:

No monitoring or alerting: The endpoints went 403/404 silently. There was no synthetic HTTP check, no CloudWatch alarm, no daily health dashboard.
Deployment fragility: Google Apps Script deployments are tightly coupled to the account that created them. Account permissions, OAuth token revocation, or organizational policy changes can cascade into downtime.
Tight coupling to URLs: Event pages have hardcoded endpoint URLs. A redeployment with a new ID breaks all pages simultaneously.

Key Decisions & Trade-offs

Execute as Me, not