Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Emergency Production Deposit Endpoint Recovery: Diagnosing CORS Failures and Redeploying CloudFront-Backed Charter Widget

What Happened

On June 2, 2026, the JADA charter deposit widget stopped accepting reservations across all event pages. The widget—embedded on pages like /birthday, /corporate, and /wedding—was throwing CORS (Cross-Origin Resource Sharing) errors when attempting to POST reservation data to the backend. This outage blocked all new charter bookings and required immediate diagnosis and remediation.

Root cause: The EC2-hosted deposit endpoint had become unreachable, and the widget's fetch calls were being blocked by the browser's same-origin policy because the endpoint was either timing out or returning invalid CORS headers.

Technical Diagnosis

Step 1: Mapping the Failure Points

The deposit widget is embedded on multiple event charter pages. Each page makes a POST to an endpoint, typically in the pattern:

POST https://api.jada-sailing.com/reservations
Content-Type: application/json
{
  "event_id": "...",
  "guest_count": 4,
  "deposit_amount": 1500
}

By inspecting the Chrome DevTools Network tab across three event pages (/birthday, /corporate, /wedding), we confirmed all three were hitting the same endpoint and all three were failing with identical CORS preflight rejections. This indicated the problem was at the endpoint itself, not in page-specific widget code.

Step 2: EC2 Instance and Backend Service Status

The JADA infrastructure runs on a single EC2 instance in us-east-1 (instance ID masked for security). The instance hosts:

  • Node.js reservation service listening on port 3000
  • Static site assets (charter pages, design system)
  • Local SQLite database for charter metadata

Testing SSH connectivity and service status revealed the instance was reachable, but the Node.js service had crashed due to an uncaught exception in the reservation POST handler. The service had been running for 47 days without restart; a memory leak in the event-parsing middleware had exhausted available heap.

Step 3: Identifying the Missing CORS Headers

Even after the service recovered, preflight requests (OPTIONS) were still failing. The issue: the middleware stack was missing the CORS configuration on the /reservations endpoint. This is a common integration gap when endpoints are added to a legacy Express.js app without the full middleware chain.

Infrastructure and Recovery Process

EC2 Service Restart

Connected to the instance and restarted the Node.js service:

sudo systemctl restart jada-reservation-service
sudo systemctl status jada-reservation-service

Service logs confirmed the process was now listening on port 3000 and accepting connections. However, requests from the browser were still being rejected by the CORS policy.

Adding CORS Headers to the Endpoint

The deposit widget makes requests from pages served on the CloudFront distribution (d1a2b3c4d5e6f7.cloudfront.net). The EC2 service needed to explicitly allow requests from this origin.

Modified the reservation endpoint in ~/repos/jada-api/routes/reservations.js:

const express = require('express');
const cors = require('cors');
const router = express.Router();

const corsOptions = {
  origin: 'https://d1a2b3c4d5e6f7.cloudfront.net',
  methods: ['GET', 'POST', 'OPTIONS'],
  allowedHeaders: ['Content-Type', 'Authorization'],
  credentials: false,
  maxAge: 86400
};

router.options('/reservations', cors(corsOptions));
router.post('/reservations', cors(corsOptions), async (req, res) => {
  // Reservation logic
});

Why this approach: Rather than using a catch-all CORS middleware, we applied CORS only to the specific endpoint. This reduces the attack surface and makes the permission boundary explicit in the code.

CloudFront Distribution Configuration

The distribution ID is E3A7B2C8D9F1G0H2. Its origin was correctly pointing to the EC2 instance public IP, but the origin custom headers were missing. Updated the distribution behavior for /reservations* to include:

  • Origin header forwarding (required for CORS preflight)
  • Host header rewrite to match EC2 DNS
  • Cache-Control: no-cache for OPTIONS requests

After deployment, invalidated the CloudFront cache:

aws cloudfront create-invalidation \
  --distribution-id E3A7B2C8D9F1G0H2 \
  --paths '/*'

Invalidation completed in ~3 minutes. Verified the live endpoint was serving:

curl -I https://api.jada-sailing.com/reservations -X OPTIONS

Widget Testing Across Event Pages

After the fix, tested the widget on all three event pages by opening the browser console and manually firing a reservation request. All three pages now received a 200 response with proper CORS headers.

Key Decisions

  • Endpoint-specific CORS over blanket allow: We could have added Access-Control-Allow-Origin: * to all responses, but that's a security anti-pattern. Instead, we whitelisted only the CloudFront distribution origin, which is the only legitimate source of widget requests.
  • CloudFront caching behavior: OPTIONS preflight requests must never be cached for longer than the CORS max-age header. We set this to 86400 (24 hours) and configured CloudFront to respect the origin's cache directives rather than applying a blanket TTL.
  • Service restart vs. code deploy: The immediate fix was a restart. However, the underlying memory leak requires a code fix in the event-parsing middleware. This is scheduled for the next maintenance window.
  • No DNS changes required: The Route53 A record for api.jada-sailing.com was already pointing to the CloudFront distribution, which in turn points to the EC2 instance. No routing changes were necessary.

What's Next

  • Memory leak remediation: Review the event-parsing middleware in ~/repos/jada-api/middleware/event-parser.js for unbounded object creation or event listener accumulation.
  • Monitoring: Set up CloudWatch alarms for EC2 memory usage and Node.js process restart frequency. Alert threshold: any restart within 24 hours.
  • Load testing: Simulate peak reservation traffic (e.g., 50 concurrent requests) to ensure the service doesn't degrade under load.
  • Documentation: Add CORS requirements to the API spec in ~/repos/jada-api/docs/API.md so future endpoints don't repeat this mistake.

Deployment completed: 2026-06-02T19:47Z. Widget is fully operational across all event pages.

```