Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Debugging Multi-Layer Email Delivery Failures: Gmail SMTP Relay vs. SES API Authentication

During a recent development session, we encountered a critical issue where outbound emails from the JADA sailing platform were failing to deliver. What appeared to be a single failure turned out to be two distinct problems occurring at different layers of the email stack. This post documents the systematic debugging approach and the architectural insights that emerged.

The Problem: Two Failures, One Thread

A user reported that emails sent through the admin workflow were bouncing. Initial inspection of the bounce message showed: "SMTP authentication failed: misconfigured or out of date" coming from Gmail's servers. However, digging deeper revealed the actual failure tree was more complex.

  • Layer 1 (Gmail): The alias admin@queenofsandiego.com configured as a "Send mail as" identity within jadasailing@gmail.com had stale SMTP relay credentials
  • Layer 2 (SES): Even emails that bypassed Gmail and went directly through the SES API were being rejected by recipient mail servers

This is why systematic debugging across the entire stack was essential—fixing one wouldn't have solved the complete problem.

Technical Investigation: The Methodology

The debugging process followed this sequence:

1. SES Infrastructure Validation

First, we verified that all SES sending infrastructure was correctly configured:

# Check verified identities and their verification status
aws ses list-identities --region us-east-1

# Validate domain identity details including SPF/DKIM
aws ses get-identity-verification-attributes \
  --identities admin@queenofsandiego.com \
  --region us-east-1

# Verify DKIM CNAME records exist in Route53
# All 3 DKIM tokens present and validated
# SPF record: v=spf1 include:amazonses.com ~all
# DMARC policy: p=none (permissive for monitoring)
# Custom MAIL FROM domain configured

Result: The SES infrastructure was clean. All DKIM CNAMEs were in Route53, SPF was correctly configured, and the domain had DMARC validation enabled.

2. SES Suppression List Analysis

We checked whether emails were being suppressed due to previous bounces or complaints:

# Check suppression list for the recipient address
aws sesv2 get-suppressed-destination \
  --email-address [recipient] \
  --region us-east-1

# Count total suppressions and categorize by type
aws sesv2 list-suppressed-destinations \
  --region us-east-1 \
  --query 'SuppressedDestinationSummaries[].Reason' \
  --output text | sort | uniq -c

This revealed that certain addresses had been added to the suppression list (either due to bounces or complaints), but the specific recipient in question was not suppressed.

3. AWS Credentials and SMTP Derivation

The critical issue emerged when testing SMTP authentication. SES SMTP credentials are not standard IAM credentials—they are derived from your current IAM access key using an AWS-specific algorithm:

# The SMTP username is your SES SMTP user (created in AWS console)
# The SMTP password must be derived from IAM credentials using v4 signature

# SES SMTP endpoint (varies by region):
# email-smtp.us-east-1.amazonaws.com (port 587 or 465)
# email-smtp.us-west-2.amazonaws.com (port 587 or 465)

# Test SMTP authentication
openssl s_client -starttls smtp \
  -connect email-smtp.us-east-1.amazonaws.com:587 \
  -crlf

# At EHLO, AUTH LOGIN with derived credentials

The derivation algorithm is non-trivial—it requires building an AWS Signature Version 4 hash from your IAM secret key. We confirmed the credentials were being generated correctly but needed to verify they matched what was actually stored in Gmail's configuration.

Infrastructure Architecture: Multi-Tenant Email Sending

Understanding the complete picture required mapping the email flow architecture:

Gmail Web UI (jadasailing@gmail.com)
  └─ "Send mail as" alias: admin@queenofsandiego.com
      └─ SMTP relay to: email-smtp.us-east-1.amazonaws.com
         └─ AWS SES (verified domain: queenofsandiego.com)
            └─ Recipient mail server (e.g., Exchange Online at steigerwald-dougherty.com)

Parallel path: API-driven email
  └─ Lambda function (jada-blast-sender)
      └─ boto3 SES client
         └─ AWS SES (us-east-1 or us-west-2)
            └─ Recipient mail server

This revealed why there were two failure modes—the Gmail alias and the SES API are completely independent paths. A failure in one wouldn't prevent the other from being used, but both needed to be functional.

Root Cause: Credential Rotation

The most likely cause was SMTP credential rotation. Here's why:

  • SES SMTP credentials in the AWS console can be regenerated (which invalidates old ones)
  • Gmail's "Send mail as" configuration stores these credentials in encrypted form within the Gmail account
  • If someone regenerated the SES SMTP credentials without updating Gmail, the alias would silently fail at send time
  • Gmail would return the error to the original account (jadasailing@gmail.com) rather than the alias, making it harder to diagnose

The bounce message "misconfigured or out of date" is Gmail's way of saying "the credentials I have don't work anymore."

Key Decision: Credential Management Strategy

We identified a need for better credential lifecycle management:

  • Store SES SMTP credentials in a single source of truth—Currently, they may be stored in multiple places (Gmail account, local config files, environment variables)
  • Document credential rotation procedure—When SES credentials are rotated, all dependent systems must be updated atomically
  • Use AWS Secrets Manager for non-web services—Lambda functions and backend services should retrieve credentials from AWS Secrets Manager rather than storing them in environment variables
  • For Gmail specifically—Consider whether the "Send mail as" SMTP relay is the right architecture, or whether the application should always use the SES API directly

What's Next

To fully resolve this issue and prevent recurrence:

  1. Regenerate SES SMTP credentials in the AWS console and store them in AWS Secrets Manager
  2. Update Gmail alias configuration with the new credentials (Settings → Accounts and Import → "Send mail as" → Edit)
  3. Verify SMTP connectivity using the test above to confirm credentials work
  4. Check recipient domain MX records to ensure mail is being accepted (not rejected at SMTP level)
  5. Review suppression list policies—Consider whether to implement bounce handling that automatically re-verifies addresses
  6. Implement credential rotation automation—Schedule regular credential regeneration with automatic updates to all dependent systems