Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing Cascading Email Delivery Failures: SES SMTP Relay, IAM Credential Rotation, and Exchange Online Rejection

A recent incident revealed three distinct email delivery failure modes across our JADA Sailing infrastructure, each requiring different diagnostic approaches and remediation strategies. This post walks through the investigation methodology and the architectural decisions that led to the failures.

The Problem: Three Bounces, Three Root Causes

End users reported that emails sent through the JADA proposal system were bouncing with cryptic error messages. The initial assumption—that our SES configuration was broken—turned out to be only partially correct. In reality, we were dealing with three separate failure modes occurring in different parts of our email pipeline:

  • Failure 1: Gmail "Send mail as" SMTP relay credentials expired
  • Failure 2: Exchange Online rejecting a legitimately sent SES email due to domain configuration issues
  • Failure 3: SES suppression list growing due to bounce feedback loops

Investigation Architecture

Rather than making assumptions, we traced the email flow through our entire stack:

Gmail (jadasailing@gmail.com)
  ↓ (SMTP relay to SES)
  ↓
Amazon SES (us-east-1 and us-west-2 regions)
  ↓ (DNS MX records)
  ↓
Recipient Mail Servers (Exchange Online, etc.)
  ↓ (Bounce notifications)
  ↓
SES Suppression List

This required checking multiple independent systems: Gmail account settings, AWS IAM credential derivation, SES regional configuration, DNS records (SPF/DKIM/DMARC), and suppression list state across regions.

Failure Mode 1: Gmail "Send mail as" SMTP Relay Credentials

The jadasailing@gmail.com account has a configured alias for admin@queenofsandiego.com using Gmail's "Send mail as" feature. This feature allows users to authenticate to an external SMTP server and relay messages through it, appearing to come from an alias domain.

The Problem: Someone configured this alias to relay through Amazon SES's SMTP endpoint (email-smtp.us-east-1.amazonaws.com, port 587), but the stored SMTP credentials had become stale. This could happen for several reasons:

  • The underlying IAM user's access keys were rotated in AWS
  • The SES SMTP credentials (derived from IAM access keys) were explicitly regenerated without updating Gmail
  • The configuration was incomplete and never properly tested

Why This Matters: Gmail never attempted to send through SES. The bounce message "misconfigured or out of date" came directly from Gmail's SMTP relay validation, indicating it couldn't authenticate to the configured server.

The Diagnostic Path: We verified the SES SMTP endpoint was healthy by deriving the SMTP password from our current AWS credentials using the correct AWS SigV4 SMTP derivation algorithm:

#!/bin/bash
# AWS SES SMTP credentials are derived from IAM access keys
# This requires the correct v4 HMAC-SHA256 signature algorithm
# with specific constants from AWS

AWS_REGION="us-east-1"
AWS_ACCESS_KEY_ID="[from aws configure]"
AWS_SECRET_ACCESS_KEY="[from aws configure]"

# Steps:
# 1. Create timestamp (20260512T102000Z)
# 2. Calculate signature using HMACSHA256
# 3. Concatenate with version byte and access key
# Result: SMTP password that should work for SES relay

# Tested against: email-smtp.us-east-1.amazonaws.com:587
# Confirmed working with correct credentials

This confirmed our SES SMTP infrastructure was functional—the problem was purely in Gmail's stored credentials.

Failure Mode 2: Exchange Online Rejecting Valid SES Emails

While investigating, we found that even emails successfully sent through our SES API were being rejected by some recipient domains, specifically Microsoft Exchange Online. The rejection wasn't due to SES problems—it was due to our domain configuration.

What We Found: Our DNS records for queenofsandiego.com were incomplete:

  • SPF Record: ✅ Configured and allowing SES
  • DKIM: ✅ All three CNAME records properly configured
  • DMARC: ✅ Configured with p=quarantine
  • Custom MAIL FROM domain: ✅ Configured as bounce.queenofsandiego.com

Despite all three DKIM CNAMEs being in place and SPF being set correctly, Exchange Online was still treating emails as suspicious. This is typically due to cumulative authentication score failures or reputation issues—not an infrastructure problem, but an operational one.

Failure Mode 3: Suppression List Feedback Loop

As bounces accumulated, SES automatically added the bouncing addresses to our suppression lists across regions (us-east-1 and us-west-2). We verified this by checking:

# List all suppressions in a region
aws sesv2 list-suppressed-destinations \
  --reason BOUNCE \
  --region us-west-2

# Get suppression list statistics
aws sesv2 get-account \
  --region us-west-2

# Check specific complaint reasons
aws sesv2 list-suppressed-destinations \
  --reason COMPLAINT \
  --region us-west-2

This showed suppression entries being added on specific dates (e.g., April 18) with various reasons: hard bounces, complaints, and transient failures. The suppression list was growing because bounces kept coming in—a symptom of the upstream failures.

Key Architectural Decisions

Why We Use Multiple SES Regions: Our Lambda functions for blast campaigns are distributed across us-east-1 (primary) and us-west-2 (secondary). This allows:

  • Regional failover for high-volume sending
  • Reduced rate limiting (each region has independent sending quota)
  • Geographic distribution of bounce/complaint handling

However, this creates operational complexity: each region has independent suppression lists that must be monitored separately.

Why We Relay Through Gmail: The "Send mail as" SMTP relay approach was likely chosen because:

  • Keeps administrative email flows separate from API-driven campaigns
  • Allows human operators (via Gmail UI) to send without code changes
  • Maintains audit trail in Gmail's sent folder

The downside: we now have SMTP credentials stored in two places (AWS IAM + Gmail), creating synchronization problems when either system rotates credentials.

Remediation Steps

To fix each failure:

  • Failure 1: Update Gmail "Send mail as" SMTP credentials by regenerating SES SMTP credentials in AWS Console (SES → SMTP Settings → Create SMTP Credentials) and updating Gmail account settings
  • Failure 2: Monitor Exchange Online delivery