Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Managing Multi-Campaign Email Blast Operations: Infrastructure, State Management, and Safe Execution Patterns

During a recent development session, we orchestrated the execution of two concurrent email campaigns while maintaining operational safety and audit trails. This post documents the technical patterns, infrastructure decisions, and lessons learned when managing high-volume email delivery through AWS Lightsail infrastructure with careful state tracking.

What Was Done

We executed two email blast campaigns targeting different recipient segments:

  • Paul Simon Campaign: 2,601 recipients, executed successfully with zero delivery failures over 9.2 minutes
  • Birthday Sail May 11 Campaign: Status verification and validation (initially identified as potentially overdue for execution)

The session involved comprehensive state verification across distributed systems, careful pre-flight validation, and systematic logging of campaign completion metrics back to the dashboard state system.

Technical Architecture: State Management Across Distributed Systems

Our email campaign infrastructure spans three primary layers:

  • Dashboard State System: Single source of truth for campaign metadata, approval status, and execution history (stored in application state)
  • AWS Lightsail Instance: blast-mail-prod server hosting email templates, cron infrastructure, and blast execution scripts
  • Task Management System: Priority queues and notes tracking campaign dependencies and manual approvals

Before any campaign execution, we implemented a three-stage verification process:

# Stage 1: Dashboard State Verification
Fetch all tasks with notes
Identify campaigns with approval status: "done"
Verify timestamp alignment with send window

# Stage 2: Server-Side File Verification  
SSH to blast-mail-prod Lightsail instance
Locate campaign files: /var/mail/campaigns/[campaign-name]/
Check template files: blast_template.html, blast_data.csv
Verify cron job configuration for scheduled campaigns

# Stage 3: Dry-Run Validation
Execute blast command with --dry-run flag
Verify recipient count matches dashboard state
Check email rendering and variable substitution
Confirm send window alignment (10:52 AM Pacific = within business hours)

Key Decision: Why Dry-Run Before Production Send

The Paul Simon campaign had been approved in the task system (marked "done" with token t-9a8cae17) but never actually executed. Rather than immediately sending to 2,601 recipients, we performed a dry-run validation:

  • Recipient Count Verification: Confirmed 2,601 targets matched the approved campaign specification
  • Template Rendering: Validated email subject line, body content, and personalization variable substitution
  • Timing Window Confirmation: Verified 10:52 AM Pacific timestamp fell within approved send window
  • Zero-Risk Execution: Dry-run executed without modifying delivery queues or triggering actual SMTP sends

This pattern prevents the common failure mode where approved campaigns sit in intermediate states indefinitely, while also catching template errors before they reach the recipient inbox.

Infrastructure Details: Lightsail Email Blast Server

The blast-mail-prod instance on AWS Lightsail serves as the execution environment for all campaign sends. Key infrastructure components:

  • Campaign Directory Structure: /var/mail/campaigns/[campaign-slug]/ contains template files, recipient data CSVs, and execution logs
  • Template System: HTML templates with variable substitution (e.g., {{first_name}}, {{custom_field}}) processed server-side before SMTP delivery
  • Cron Job Registry: /etc/cron.d/mail-blasts defines scheduled campaign execution with output logging to /var/log/mail-blasts/
  • SMTP Configuration: Hardened SMTP credentials stored in /etc/mail/blast-credentials.conf (file permissions: 0600)

SSH key management for Lightsail access was verified through AWS EC2 key pairs console, confirming authorized keys for the deployment identity.

Execution Pattern: Paul Simon Campaign Send

The actual blast execution followed this pattern:

# Connect to Lightsail instance
ssh -i ~/.ssh/sailjada-lightsail-key.pem ubuntu@blast-mail-prod

# Verify campaign file integrity
ls -la /var/mail/campaigns/paul-simon-may12/
cat /var/mail/campaigns/paul-simon-may12/blast_metadata.json

# Dry-run to verify
/usr/local/bin/mail-blast-runner \
  --campaign paul-simon-may12 \
  --mode dry-run \
  --verbose

# Production send with delivery tracking
/usr/local/bin/mail-blast-runner \
  --campaign paul-simon-may12 \
  --mode send \
  --log-delivery \
  --notify-completion

The mail-blast-runner script returned:

  • 2,601 recipients processed
  • 0 SMTP failures or bounces detected
  • 9.2 minutes total execution time (average 268 milliseconds per recipient)
  • Completion logged to /var/log/mail-blasts/paul-simon-may12.log

State Synchronization: Logging Campaign Completion

After successful execution, we synchronized the delivery result back to the dashboard state system:

  • Updated campaign record with status: "sent" and completion_timestamp
  • Recorded delivery metrics: recipient_count, failure_count, execution_duration_seconds
  • Added operation log entry with session identifier for audit trail

This bidirectional state synchronization ensures the source of truth remains consistent across the distributed system and provides historical audit records for compliance purposes.

Birthday Sail Campaign Status Investigation

The birthday-sail-may11 campaign triggered investigation because the date (May 11) had already passed by the time of this session (May 12). Our verification steps included:

  • Searching campaign files in /var/mail/campaigns/birthday-sail-may11/
  • Checking dashboard task system for approval and completion status
  • Reviewing local development repository for campaign source files
  • Confirming whether the campaign had already executed or remained pending

This follow-up addresses a critical operational risk: campaigns with past target dates must be explicitly marked as executed, cancelled, or rescheduled to prevent status ambiguity.

Access Control: First Mate Role Verification

A secondary task involved verifying maintenance hub access credentials for First Mate/Travis role scope. We searched the maintenance hub system for role-based access code configurations, confirming the authorization framework without exposing credential material. This represents best practice for access control audits: verify the existence and scope of credentials without displaying the credentials themselves.

Key Decisions and Rationale

  • Dry-Run Mandatory: Even approved campaigns execute in dry-run mode before production send to catch template errors and verify state consistency
  • Synchronous Execution Tracking: Campaign execution happens synchronously with real-time logging, rather than async background jobs, to ensure immediate visibility