Rejecting Heavy CI/CD Toolchains: Why We Chose Terraform Principles + Lightsail Micro-Deployments

\nAfter a formal board review of our infrastructure architecture, we made a deliberate decision: reject enterprise CI/CD platforms in favor of lightweight, shell-driven deterministic deploys paired with Terraform-like infrastructure-as-code principles. This post walks through the technical reasoning, what we actually built, and how it works in production.\n

What Was Done

\nOur estate runs ~15 static S3+CloudFront sites, single-function AWS Lambdas, and scheduled tasks via launchd/cron. The board verdict was clear: rather than adopt GitLab CI, GitHub Actions, or similar platforms, we'd lean into our existing strengths—shell scripts, git mirrors, and verified deterministic deployments.\n

\nWe deployed a Lightsail twin instance that acts as both a deployment peer and a stall-watchdog for nightly test automation. This gives us:\n

\n
• Deterministic shell-based deploys with zero external dependencies
\n
• A secondary instance for peer-sync failover and autonomous watchdog checks
\n
• Nightly verification tests that run against real infrastructure (not mocks)
\n
• A git mirror that serves as the source of truth for production code
\n

Technical Details: The Terraform Principle Without Terraform

\nWe didn't adopt Terraform itself (too heavy for our scale), but we adopted its core principle: infrastructure state declared once, idempotently applied, auditable in version control.\n

\nFor S3 + CloudFront sites, we use shell scripts that:\n

\n
• Read bucket names and distribution IDs from a version-controlled config
\n
• Apply CloudFront invalidations after S3 sync operations
\n
• Record deployment checksums in a state file (git-tracked)
\n
• Skip re-deployment if content hasn't changed (idempotency)
\n

\nExample pattern (no credentials shown):\n

\n#!/bin/bash\n# Deploy static site to S3 + CloudFront\n# Usage: ./deploy-site.sh site-name\n\nSITE=$1\nBUCKET=\"jada-${SITE}-prod\"\nDIST_ID=$(grep \"${SITE}\" config/cloudfront-dists.txt | cut -d',' -f2)\n\n# Sync with checksum tracking\nNEW_HASH=$(find src/${SITE} -type f | xargs sha256sum | sha256sum | cut -d' ' -f1)\nOLD_HASH=$(cat .deploy-state/${SITE}.hash 2>/dev/null || echo \"none\")\n\nif [ \"$NEW_HASH\" != \"$OLD_HASH\" ]; then\n  aws s3 sync src/${SITE}/ s3://${BUCKET}/ --delete --cache-control \"max-age=31536000\"\n  aws cloudfront create-invalidation --distribution-id ${DIST_ID} --paths \"/*\"\n  echo $NEW_HASH > .deploy-state/${SITE}.hash\n  echo \"Deployed ${SITE} (hash: $NEW_HASH)\"\nelse\n  echo \"${SITE} unchanged, skipping deploy\"\nfi\n

Infrastructure: The Lightsail Twin Architecture

\nA single Lightsail instance (Ubuntu 22.04, t2.micro equivalent) serves dual purposes:\n

\n
• Primary: Deployment peer — runs shell-driven deploys via cron, triggered by git changes in our mirror
\n
• Secondary: Stall-watchdog — monitors Lambda execution logs and S3 sync operations for hangs or failures
\n

\nThe instance mounts:\n

\n
• /mnt/git-mirror — bare git repo cloned from GitHub (source of truth)
\n
• /opt/jada-ops — deployment scripts and infrastructure code
\n
• /var/spool/cron/crontabs — nightly test suite that validates all deployments
\n

\nNightly deterministic tests run via cron and exercise:\n

\n
• HEAD resolution (ensuring git mirror is in sync)
\n
• S3 bucket policies and object ACLs
\n
• CloudFront cache behavior and certificate status
\n
• Lambda function code and environment variable consistency
\n
• Route53 DNS resolution and CNAME targets
\n

\nTest results are written to CloudWatch Logs and alerted on failure.\n

Key Decision: Shell Scripts Over Declarative Platforms

\nWhy reject GitHub Actions, GitLab CI, or AWS CodePipeline?\n

\n
• Vendor lock-in: Our scripts run anywhere (local, Lightsail, any Linux box). No dependency on GitHub's availability or pricing changes.
\n
• Auditability: Deployment logic is plain bash in git. No opaque YAML DSL or proprietary pipeline syntax.
\n
• Determinism: Same script runs locally during development and in production. Zero surprises.
\n
• Cost: We pay for Lightsail (~$10/month), not per-pipeline or per-minute compute charges.
\n
• Secrets management: No need for GitHub Secrets API, vault integration, or CI-specific credential injection. IAM instance profiles handle auth.
\n

\nThis works because our estate is small and predictable. We're not deploying microservices or handling thousands of concurrent jobs.\n

Failover and Autonomous Watchdog

\nThe stall-watchdog runs independently:\n

\n
• Polls Lambda CloudWatch Logs every 5 minutes for incomplete executions
\n
• Checks S3 sync state files for hung operations (staleness > 6 hours)
\n
• Sends alerts to CloudWatch Alarms (not email—avoiding alert fatigue)
\n
• Optionally rolls back the last deployment if corruption is detected
\n

\nThis is not a full orchestrator. It's a simple state monitor that catches obvious failure modes without manual intervention.\n

What's Next

\nFuture work includes:\n

\n
• Extending the git mirror to pull from multiple upstream repos (handling monorepo splits)
\n
• Adding Route53 health check integration to the watchdog (currently manual verification)
\n
• Documenting the exact Lambda deployment flow (currently single-file, but may grow)
\n
• Setting up a secondary Lightsail instance in a different region for true HA
\n

\nThe core principle stands: infrastructure as shell scripts, versioned in git, deployed idempotently, validated nightly. No magic, no surprises.\n