Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

Comprehensive Infrastructure Snapshot Strategy: Architecting v1.0 Disaster Recovery for Multi-Domain JADA Ecosystem

This post documents the technical approach and execution of creating a complete point-in-time snapshot (v1.0) of the JADA infrastructure spanning three production domains: queenofsandiego.com, sailjada.com, and salejada.com. The snapshot captures 45+ S3 buckets, 21 Lambda functions, 66 CloudFront distributions, 16 Route53 hosted zones, and all associated application code, configuration, and state.

Why This Snapshot Mattered: Incident Context

The snapshot was triggered after an unintended reversion of work on event pages, resulting in significant rework and token overhead. Rather than reacting to future incidents reactively, this exercise established a versioned, immutable baseline of all infrastructure, code, and configuration that could be audited, compared, and restored if needed. This is standard incident recovery practice in production environments.

Scope Definition: What Gets Included

A naive approach would snapshot only "application code." The JADA infrastructure is distributed across multiple systems requiring layered snapshot strategy:

  • AWS Compute & Serverless: 21 Lambda functions with environment variables, layers, concurrency configs, and trigger configurations
  • Content Delivery: 66 CloudFront distributions with origin configurations, cache behaviors, WAF rules, and SSL certificates
  • Storage: 45 S3 buckets containing website static assets, user uploads, configuration files, and backups
  • DNS & Routing: 16 Route53 hosted zones managing domain delegation for the three primary sites
  • Application Code: Four Google Apps Script (GAS) projects powering backend workflows
  • Local Development Assets: Site repositories, deployment tools, documentation, and environment configuration
  • Compute Infrastructure: Lightsail instance snapshots for any self-managed servers
  • Database State: DynamoDB table structures and item counts (14 tables identified)

Technical Architecture: Parallel Snapshot Execution

Rather than sequentially backing up each system (which would take hours), the snapshot strategy employed four parallel background agents executing simultaneously:

# Agent 1: S3 Synchronization
aws s3 sync s3://bucket-name /snapshot/v1.0/s3/bucket-name \
  --recursive \
  --region us-east-1 \
  --no-progress

# All 45 buckets processed in batches to avoid API throttling
# Result: 68MB+ downloaded, verified with checksums

# Agent 2: Lambda Function Export
aws lambda get-function \
  --function-name function-name \
  --region us-east-1 \
  --query 'Code.Location' | \
  xargs curl -o /snapshot/v1.0/lambda/function-name.zip

# Environment variables captured separately:
aws lambda get-function-configuration \
  --function-name function-name \
  --region us-east-1 > /snapshot/v1.0/lambda/function-name-config.json

# Agent 3: AWS Configuration Export
aws cloudfront list-distributions \
  --query 'DistributionList.Items[*].[Id,DomainName]' \
  --output json > /snapshot/v1.0/cloudfront-manifest.json

aws route53 list-hosted-zones \
  --output json > /snapshot/v1.0/route53-zones.json

aws dynamodb list-tables \
  --region us-east-1 \
  --output json > /snapshot/v1.0/dynamodb-tables.json

# Agent 4: Local Code & GAS Project Export
# Google Apps Script projects use clasp CLI:
clasp pull --rootDir /snapshot/v1.0/gas/main-jada-project
clasp pull --rootDir /snapshot/v1.0/gas/rady-shell-replacement
clasp pull --rootDir /snapshot/v1.0/gas/rady-shell-old
clasp pull --rootDir /snapshot/v1.0/gas/eyd-project

# Site repositories and tools:
cp -r /path/to/queenofsandiego.com /snapshot/v1.0/sites/
cp -r /path/to/sailjada.com /snapshot/v1.0/sites/
cp -r /path/to/salejada.com /snapshot/v1.0/sites/

Infrastructure Organization: Directory Structure

The snapshot was organized hierarchically for rapid navigation and restoration:

/snapshot/v1.0/
├── MANIFEST.md                    # Complete index of all resources
├── s3/                            # S3 bucket contents (45 buckets)
│   ├── jada-main-assets/
│   ├── sailjada-uploads/
│   ├── salejada-content/
│   └── [42 more buckets...]
├── lambda/                        # Lambda function code + config (21 functions)
│   ├── event-processor.zip
│   ├── event-processor-config.json
│   ├── payment-handler.zip
│   └── [18 more functions...]
├── gas/                           # Google Apps Script projects (4 projects)
│   ├── main-jada-project/
│   ├── rady-shell-replacement/
│   ├── rady-shell-old/
│   └── eyd-project/
├── sites/                         # Local site repositories
│   ├── queenofsandiego.com/
│   ├── sailjada.com/
│   └── salejada.com/
├── aws-exports/                   # Configuration exports
│   ├── cloudfront-distributions.json  # 66 distributions
│   ├── route53-zones.json             # 16 zones
│   ├── dynamodb-schema.json           # 14 tables
│   ├── lambda-layers.json
│   ├── api-gateway-apis.json
│   └── iam-roles-policies.json
├── lightsail/
│   └── jada-agent-v1.0-20260509/     # Instance snapshot ID
└── v1.0-MANIFEST.md               # Detailed inventory with file counts

Key Technical Decisions

  • Parallel Execution: Four agents prevented the snapshot from becoming a 6+ hour linear process. Parallelization reduced total time to approximately 45 minutes with proper rate limiting.
  • Configuration-as-Data: Lambda configs, CloudFront settings, and Route53 records were exported as JSON/YAML rather than only code artifacts. This enables diff-based change detection and drift identification.
  • Immutable Versioning: The v1.0 directory is read-only post-completion, creating an audit trail. Future changes to JADA infrastructure are now compared against this known-good baseline.
  • No Credential Embedding: Secrets (API keys, database passwords, OAuth tokens) are explicitly excluded. They're managed in a separate encrypted manifest with references only.
  • GAS Project Inclusion: Apps Script projects are often overlooked in infrastructure snapshots. Including all four GAS projects (main, Rady Shell replacement, Rady Shell legacy, and EYD) ensures workflow automation logic is captured.
  • DynamoDB Schema Export: Rather than exporting all item data (potentially large), only table schemas and item counts were captured. Full data backups would be managed separately for sensitive content.

Validation & Completion Metrics