Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Preventing State Drift in Multi-Environment S3 Deployments: A Case Study in Feature Regression

During a recent development session, a critical regression occurred when stale local files overwrote newer production state in S3, silently wiping three working features from a live booking site. This post documents the root cause, the infrastructure pattern that failed, and the hard rules now enforced to prevent it.

What Happened

A local index.html file—outdated relative to what was live in the sailjada-prod S3 bucket—was deployed directly to production CloudFront, erasing:

  • A hero section JADA → BOOK NOW crossfade animation
  • Stripe embedded checkout booking flow
  • A previously-removed "For Ranch & Coast readers…" hero line that was never meant to return

The deploy also violated staging-first discipline by pushing both staging and prod environments in a single command, and ignored a prior session warning about stale local artifacts.

Root Cause: No Pre-Deploy State Inspection

The deployment pipeline had no mandatory step to:

  • Fetch current S3 state before editing: Developers never pulled the live index.html from S3 to compare against their local working copy
  • Diff before deploy: No automatic diff output forced visibility of what would be overwritten
  • Snapshot production: S3 versioning was not enabled, so the live version had no backup once overwritten
  • Single-target rule: Staging and prod were deployed in one operation, preventing staged review before prod promotion

Technical Details: The Enforcement Pattern

To prevent recurrence, eight hard rules (D1–D8) were added to /Users/cb/Documents/repos/sites/queenofsandiego.com/CLAUDE.md and loaded automatically on every session:

D1: Pull S3 and Diff Before Any Edit

aws s3 cp s3://sailjada-prod/index.html ./index.html.prod
diff -u index.html.prod index.html | head -50

Before touching local index.html, retrieve the live version and display the first 50 lines of diff. This surfaces whether local is stale.

D2: Staging-Only Single-Target Deploys

Every deploy targets one environment. Staging first:

aws s3 cp index.html s3://sailjada-staging/index.html --cache-control "max-age=0"

Then, only after visual inspection on staging.sailjada.com, promote to prod:

aws s3 cp s3://sailjada-staging/index.html s3://sailjada-prod/index.html

CloudFront distribution E2ABC123XYZ (prod) is invalidated after the prod copy completes.

D3: One Logical Change Per Deploy

Never batch unrelated edits. If you're fixing the hero fade and adding referral logic, deploy the hero fix first, validate it on staging, promote, then tackle referral in a separate session.

D4: Obey Prior Session Warnings

If a prior session summary says "local index.html is behind S3," stop and sync before deploying. This is not optional.

D5: Snapshot Production Before Overwrite

Before any cp to prod, save the current live version:

aws s3 cp s3://sailjada-prod/index.html ./snapshots/index.html.$(date +%s).backup

S3 versioning should be enabled on both sailjada-prod and sailjada-staging buckets (currently not enabled—this is a secondary improvement). Until then, local snapshots are mandatory.

D6: Print Proof Block Before Deploy

Before running aws s3 cp, output exactly six lines to chat:

  • File size (local vs. S3 current)
  • MD5 hash (local vs. S3 current)
  • Target bucket and path
  • CloudFront distribution ID if applicable
  • Affected features (by feature token, see D7)
  • Staging verification status (yes/no)

Example:

Local index.html: 3,650 lines, 847 KB, md5=abc123
S3 prod current:  3,680 lines, 851 KB, md5=def456  ← STALE
Target: s3://sailjada-prod/index.html
CloudFront: E2ABC123XYZ (invalidate on success)
Features: JADA_HERO_FADE, STRIPE_CHECKOUT, REFERRAL_INPUT
Staging verified: YES

D7: Feature-Token Registry

Maintain a searchable registry in the codebase. Before deploy, grep S3-current for feature tokens:

grep -o "data-feature=\"[^\"]*\"" s3://sailjada-prod/index.html | sort -u

Compare against what you're about to deploy. Example tokens:

  • data-feature="JADA_HERO_FADE" — hero crossfade animation
  • data-feature="STRIPE_CHECKOUT" — embedded payment form
  • data-feature="REFERRAL_INPUT" — promo code field
  • data-feature="CREW_UNIFORM_REMINDER" — post-booking email tag

If your local version removes a feature token that S3 still has, escalate to CB.

D8: Escalate to CB When S3 Is Ahead

If aws s3 cp s3://sailjada-prod/index.html ./index.html.prod shows that S3 is newer or has features your local copy doesn't, do not deploy. Ping CB in Slack with the diff, wait for direction, and sync your local repo from git.

Infrastructure Changes

S3 Buckets:

  • sailjada-staging — staging environment, cache-control set to max-age=0 (no caching)
  • sailjada-prod — production environment, cache-control set to max-age=3600 (1 hour)

CloudFront Distribution:

  • Distribution ID: E2ABC123XYZ (prod)
  • Invalidation after each prod deploy: aws cloudfront create-invalidation --distribution-id E2ABC123XYZ --paths "