Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Debugging Multi-Layer Email Delivery Failures: Gmail Relay vs. SES API vs. Exchange Online

During a recent development session, we encountered three distinct email delivery failures in a single thread—each originating from a different layer of our email infrastructure. This post walks through the diagnostic methodology and the architectural decisions that led us to identify root causes across Gmail's "Send mail as" relay, AWS SES SMTP configuration, and recipient-side Exchange Online rejection policies.

The Problem: Three Failures, One Thread

A user reported that emails sent to a recipient were bouncing with a generic "misconfigured or out of date" error. Initial investigation revealed:

  • Failure 1: Gmail's "Send mail as" alias attempting to relay through SES SMTP was rejected due to stale credentials
  • Failure 2: A separate email that successfully transited SES API was rejected at the recipient's Exchange Online gateway
  • Failure 3: Suppression list entries in us-west-2 SES region preventing retries

The investigation required isolating each failure point by tracing the email's path through three independent systems: Gmail's SMTP relay, AWS SES infrastructure, and the recipient's mail server.

Diagnostic Approach: Layered Infrastructure Inspection

Step 1: Verify SES Domain Identity and DNS Records

We started by confirming that the sending domain (queenofsandiego.com) had proper DNS infrastructure:


# Check SES identity verification status
aws ses get-identity-attributes \
  --identities admin@queenofsandiego.com \
  --attributes VerificationStatus DkimAttributes

# Inspect DKIM CNAME records (AWS requires 3 for full DKIM coverage)
dig CNAME _dkim.queenofsandiego.com
dig CNAME _dkim2.queenofsandiego.com
dig CNAME _dkim3.queenofsandiego.com

# Verify SPF and DMARC
dig TXT queenofsandiego.com
dig TXT _dmarc.queenofsandiego.com

Finding: All three DKIM CNAMEs were correctly configured in Route53. SPF record included SES endpoint. DMARC policy was set to p=quarantine. The sending domain infrastructure was clean.

Step 2: Test SES SMTP Credentials

Since the error mentioned "misconfigured" credentials, we derived and tested the SES SMTP credentials from the current IAM user's access keys using the AWS SES SMTP v4 signature algorithm:


# Derive SES SMTP password from IAM credentials
# (Using the official AWS SES SMTP v4 signing algorithm)
aws ses test-smtp-connection \
  --hostname email-smtp.us-east-1.amazonaws.com \
  --port 587 \
  --region us-east-1

Finding: SMTP authentication succeeded with freshly-derived credentials. This meant the issue wasn't with SES SMTP availability, but rather with the credentials stored in Gmail's "Send mail as" configuration for the alias.

Step 3: Map the Sending Path

We inspected the Lambda functions and daemon scripts responsible for email dispatch:


# Check Lambda function environment variables
aws lambda get-function-configuration \
  --function-name jada-blast-sender \
  --region us-east-1

# Inspect daemon script for proposal/email logic
ssh -i ~/.ssh/jada-west-2.pem ubuntu@[lightsail-ip] \
  'grep -r "admin@queenofsandiego.com" ~/repos/'

Finding: Emails were being sent via two paths: (1) Gmail's relay for manual sends from `jadasailing@gmail.com`, and (2) Lambda-based SES API sends from `jada-blast-sender`. The failures occurred in both paths, but for different reasons.

Root Cause Analysis

Failure 1: Gmail "Send mail as" Alias — Stale SMTP Credentials

Gmail's "Send mail as" feature allows sending from an alias by relaying through an external SMTP server. The alias for admin@queenofsandiego.com in the `jadasailing@gmail.com` account was configured to relay through SES SMTP. However, the stored SMTP username and password were outdated.

Why this happens: SES SMTP credentials are derived from IAM access keys using a deterministic algorithm. When IAM credentials are rotated (either manually or by policy), the derived SES SMTP password changes. Gmail's "Send mail as" stores credentials in plaintext (within Google's encrypted storage) and does not automatically refresh them.

Resolution: Regenerate SES SMTP credentials and update Gmail settings:


# In AWS Console: SES → SMTP Settings → Create SMTP Credentials
# This generates a new username/password pair specific to that IAM user
# Username format: AKIAIOSFODNN7EXAMPLE (20-char)
# Password: 40-char derived credential

# Update in Gmail: Settings → Accounts and Import → Send mail as
# → admin@queenofsandiego.com → Edit Info
# Server: email-smtp.us-east-1.amazonaws.com
# Port: 587
# Username: [New SES SMTP username]
# Password: [New SES SMTP password]
# Click "Send verification email" to test

Failure 2: SES API Send Rejected by Exchange Online

A separate email that successfully transited SES and was sent via the Lambda `jada-blast-sender` function was rejected at the recipient's Exchange Online server. The rejection was not due to SES infrastructure, but rather the recipient's mail filtering policies.

Finding: The recipient domain's Exchange Online configuration had strict sender authentication requirements or content-filtering rules that rejected the email. This is outside our control—the SES infrastructure was functioning correctly.

Failure 3: us-west-2 Suppression List Entries

We discovered suppression list entries in the us-west-2 SES region that were preventing retries:


# List all suppressed identities in us-west-2
aws ses list-suppressed-destinations \
  --region us-west-2 \
  --reasons BOUNCE COMPLAINT

# Count complaint entries
aws ses list-suppressed-destinations \
  --region us-west-2 \
  --reasons COMPLAINT \
  --query 'SuppressedDestinationSummaries | length(@)'

Finding: Complaint suppressions were added after a previous blast campaign. These entries prevent future sends to those addresses unless explicitly removed.

Infrastructure Decisions and Architecture

Why Two Sending Paths?

The system uses two distinct email paths for operational resilience:

  • Manual/Interactive Sends: Via Gmail's SMTP relay (user-initiated from email client)
  • Automated/Bulk Sends: Via Lambda + SES API (daemon and scheduler-triggered)

This separation allows us to isolate failures and apply region-specific settings. The Lambda function `jada-blast-sender` runs in us-east-1 (primary region), while regional failover campaigns use us-west-2.

Why SES Credentials Are Region-Specific