Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Debugging Multi-Layer Email Delivery Failures: Gmail Relay vs. SES Infrastructure

We recently encountered a series of email delivery failures across the JADA sailing platform that exposed two distinct failure modes in our email architecture. This post covers the diagnostic approach, root causes, and the architectural decisions that led us here.

The Problem: Three Emails, Two Different Failures

Three emails sent from the JADA platform to steigerwald-dougherty.com recipients bounced. The user reported seeing "misconfigured or out of date" errors. However, the investigation revealed this wasn't a single failure—it was two separate systems failing in different ways:

  • Failure Mode 1: Gmail "Send mail as" alias SMTP relay broken (credentials stale/expired)
  • Failure Mode 2: SES API delivery succeeded but recipient Exchange Online rejected the message

Understanding the difference between these modes required tracing the email path through multiple systems.

Architecture Overview: Gmail Alias + SES Dual-Path

The JADA email system uses a hybrid approach:

  • Primary: AWS SES API (via Lambda functions, jada-blast-sender in us-west-2)
  • Fallback: Gmail "Send mail as" alias configured in jadasailing@gmail.com with external SMTP relay

The Gmail alias was configured to send as admin@queenofsandiego.com using an external SMTP server—in this case, SES SMTP endpoint at email-smtp.us-east-1.amazonaws.com (port 587, TLS).

This design allows human-initiated emails (from the Gmail web UI) to be sent under the domain identity while keeping the primary blast infrastructure in AWS Lambda.

Diagnostic Process

Step 1: Verify SES Domain Infrastructure

First, we verified the SES sending domain was properly configured. All infrastructure checks passed:


# Check SES verified identities and status
aws ses get-identity-verification-attributes --identities admin@queenofsandiego.com

# Verify SPF, DKIM, DMARC DNS records
dig queenofsandiego.com TXT  # Should show SPF v=spf1 include:amazonses.com ~all
dig mail._domainkey.queenofsandiego.com CNAME  # All 3 DKIM tokens
dig _dmarc.queenofsandiego.com TXT  # DMARC policy

Result: All three DKIM CNAME records present, SPF record configured, DMARC policy set, custom MAIL FROM configured. SES infrastructure was clean.

Step 2: Check Suppression Lists and Account Status


# List all suppressed addresses in SES account
aws ses list-suppressions --region us-west-2

# Check account bounce/complaint limits
aws ses get-account-sending-enabled --region us-west-2

# Inspect specific suppression reasons
aws ses list-suppressions --reason-filter Bounce --region us-west-2

Result: Account sending enabled, but several addresses on suppression list. Cross-checked recipient domains against list—not the issue here.

Step 3: Test SES SMTP Credentials

The critical finding: we tested whether the SMTP credentials stored in Gmail's "Send mail as" settings were still valid. SES SMTP credentials are derived from IAM access keys using an AWS-specific v4 derivation algorithm.


# Generate SES SMTP credentials from current IAM key
# AWS requires: SMTP username = AWS Access Key ID
# AWS requires: SMTP password = derived using SigV4 algorithm

# Derive password:
# 1. Create canonical string: "AWS4" + SECRET_KEY
# 2. Sign "11" (date in YYYYMMDD format) with that string
# 3. Sign "us-east-1" with resulting signature
# 4. Sign "ses" with resulting signature  
# 5. Sign "aws4_request" with resulting signature
# 6. Base64 encode final signature

# Then test SMTP connection
openssl s_client -starttls smtp \
  -connect email-smtp.us-east-1.amazonaws.com:587 \
  -crlf -ign_eof
  # EHLO example.com
  # AUTH LOGIN [base64(username)]
  # [base64(password)]

Result: SMTP credentials derived from the current IAM key were valid and authenticated successfully. This meant the credentials stored in Gmail were not the current ones—they had been rotated at some point, invalidating the alias configuration.

Step 4: Trace the SES API Path

For emails sent via the primary SES API path (Lambda jada-blast-sender), we checked:


# List Lambda function environment and recent invocations
aws lambda get-function-configuration \
  --function-name jada-blast-sender \
  --region us-west-2

# Check SES sending limits and bounce rates
aws ses get-send-statistics --region us-west-2

# Inspect Lambda execution logs
aws logs tail /aws/lambda/jada-blast-sender \
  --follow --region us-west-2

Result: Lambda function was properly configured with SES API calls. Emails were being sent successfully from SES's perspective (no bounces in SES bounce statistics). The bounces were coming back from the recipient's mail server.

Root Cause Analysis

Failure Mode 1: Gmail SMTP Relay (Stale Credentials)

When the user sent an email through Gmail's web UI using the "Send mail as" alias, Gmail attempted to relay it through the configured external SMTP server. The credentials stored in that Gmail alias configuration were outdated—they no longer matched any valid IAM access key.

Gmail's error message "misconfigured or out of date" was accurate: the SMTP credentials had expired or been rotated without updating the Gmail alias settings.

Why this happens: IAM credential rotation is a security best practice. When credentials are rotated (or new ones generated as primary), the old credentials become invalid immediately. If those old credentials were stored in external systems (like Gmail), those systems fail silently on the next send attempt.

Failure Mode 2: Exchange Online Rejection

Emails that did traverse the SES API path (from Lambda directly, not via Gmail alias) were rejected by the recipient's Exchange Online mail server. This was a separate issue unrelated to SES infrastructure.

The bounces showed that:

  • SPF passed (SES domain was properly authenticated)
  • DKIM passed (all three tokens verified)
  • The rejection was a policy decision at Exchange Online

Likely cause: Exchange Online may have flagged the message based on content, header patterns, or recipient domain policies—not authentication infrastructure.

Key Infrastructure Decisions

Why use both Gmail alias + SES API? The hybrid approach provides flexibility: human users can send emails from the web interface with domain branding, while automation uses the SES API for scale and reliability. However, this introduces maintenance complexity—two separate credential paths must stay in sync.

Why SES SMTP (vs. API)? Gmail only supports SMTP; it cannot call AWS APIs. The