Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Implementing Comprehensive Infrastructure Snapshots: A v1.0 Disaster Recovery Strategy for Multi-Site AWS Deployments

When infrastructure work goes sideways, you need a complete point-in-time snapshot—not just databases or code, but everything. This post details how we built a v1.0 snapshot strategy for three interconnected Shopify/custom-stack sites (queenofsandiego.com, sailjada.com, salejada.com) spanning S3, CloudFront, Lambda, Route53, DynamoDB, Google Apps Script, and Lightsail, using parallel agents to capture the entire infrastructure footprint in one operation.

What Was Done

We created a comprehensive v1.0 snapshot infrastructure that captures:

  • 46 S3 buckets (production, staging, backups, CloudFront origin buckets)
  • 66 CloudFront distributions with full config exports (origin settings, cache behaviors, Lambda@Edge associations)
  • 21 Lambda functions with source code, environment variables, and runtime configurations
  • 16 Route53 hosted zones with complete DNS records and routing policies
  • 14 DynamoDB tables with schema and item exports
  • 4 Google Apps Script projects (main JADA, Rady Shell replacement, old Rady Shell, EYD)
  • 1 Lightsail instance snapshot for the jada-agent server
  • Local application code, configuration files, and deployment tools

Technical Architecture

The snapshot operation used a parallel agent strategy to avoid serialization bottlenecks. Instead of downloading each S3 bucket sequentially (which would take hours), we spun up four concurrent background agents:

  • Agent 1: S3 Sync — aws s3 sync all 45 buckets to local snapshot directories
  • Agent 2: Lambda Export — Pull function code via aws lambda get-function, extract ZIP files, export environment variables via aws lambda get-function-configuration
  • Agent 3: AWS Config Export — CloudFront (aws cloudfront list-distributions + get-distribution-config), Route53 (aws route53 list-hosted-zones + list-resource-record-sets), DynamoDB schema via aws dynamodb describe-table
  • Agent 4: Local File Capture — Copy application source from /Users/cb/Documents/repos/, tools, scripts, deployment configs

Directory Structure & File Organization

The snapshot was organized hierarchically under /snapshot/v1.0/:

/snapshot/v1.0/
├── s3_buckets/
│   ├── sailjada-prod/
│   ├── sailjada-staging/
│   ├── qos-prod/
│   ├── qos-staging/
│   ├── salejada-prod/
│   └── ... (41 more buckets)
├── lambda/
│   ├── [function-name]/
│   │   ├── source/
│   │   ├── config.json
│   │   └── environment_variables.json
├── cloudfront/
│   ├── distributions.json
│   ├── [dist-id]/
│   │   └── config.json
├── route53/
│   ├── hosted_zones.json
│   ├── [zone-id]/
│   │   └── records.json
├── dynamodb/
│   ├── [table-name]/
│   │   ├── schema.json
│   │   └── items.json
├── gas/
│   ├── main_jada/
│   ├── rady_replacement/
│   ├── rady_old/
│   └── eyd/
├── lightsail/
│   └── jada-agent-v1.0-20260509.snapshot
├── local_code/
│   ├── sites/
│   ├── tools/
│   └── configs/
└── MANIFEST.md

Key Commands & Technical Details

S3 Bucket Sync (Parallelized):

# List all JADA-related buckets
aws s3api list-buckets --query 'Buckets[?contains(Name, `jada`) || contains(Name, `qos`) || contains(Name, `sailjada`) || contains(Name, `salejada`)].Name'

# Sync each bucket in parallel batches
aws s3 sync s3://sailjada-prod /snapshot/v1.0/s3_buckets/sailjada-prod --region us-west-2
aws s3 sync s3://qos-prod /snapshot/v1.0/s3_buckets/qos-prod --region us-west-2
# ... repeated for all 45 buckets in parallel

Lambda Function Export:

# List all Lambda functions
aws lambda list-functions --query 'Functions[].FunctionName'

# For each function, download code and config
aws lambda get-function --function-name [name] --region us-west-2 --query 'Code.Location' | xargs curl -o function.zip
aws lambda get-function-configuration --function-name [name] --region us-west-2 > config.json

# Extract environment variables from config
jq '.Environment.Variables' config.json > environment_variables.json

CloudFront Distribution Snapshot:

# Export all distribution configs
aws cloudfront list-distributions --query 'DistributionList.Items[].Id' | while read dist_id; do
  aws cloudfront get-distribution-config --id "$dist_id" > /snapshot/v1.0/cloudfront/"$dist_id"/config.json
done

Route53 DNS Records (All Zones):

# List all hosted zones
aws route53 list-hosted-zones --query 'HostedZones[].Id' | sed 's|/hostedzone/||'

# Export all records for each zone
aws route53 list-resource-record-sets --hosted-zone-id [zone-id] > records.json

Google Apps Script Backup via Clasp:

# Pull each GAS project
clasp clone [script-id] --rootDir /snapshot/v1.0/gas/main_jada/
clasp clone [script-id] --rootDir /snapshot/v1.0/gas/rady_replacement/
# ... repeated for all 4 GAS projects

Lightsail Instance Snapshot:

aws lightsail create-instance-snapshot \
  --instance-name jada-agent \
  --instance-snapshot-name jada-agent-v1.0-20260509 \
  --region us-west-2

Key Decisions & Rationale

  • Parallel agents instead of sequential operations: Downloading 45 S3 buckets one at a time would take 2-3 hours. Running 4 concurrent agents reduced total time to ~30 minutes.
  • Export vs. backup: For Cloud