Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Debugging Multi-Layer Email Delivery Failures: Gmail SMTP Relay vs. Exchange Online Rejection

During a recent development session, we discovered that three outbound emails from the JADA sailing platform were failing to deliver. The initial symptom appeared straightforward—a Gmail bounce notification—but the root cause involved two completely separate failure modes across different email infrastructure layers. This post documents the diagnostic process and the technical decisions required to distinguish between them.

The Problem: Two Failures, One Thread

Screenshots showed a bounce message received at jadasailing@gmail.com stating that the message to the recipient could not be sent due to "misconfigured or out of date" SMTP relay settings. However, deeper investigation revealed that this single bounce notification masked two distinct failure scenarios:

  • Failure 1: Gmail's "Send mail as" alias had stale SMTP credentials for the SES relay
  • Failure 2: A separate email that successfully left SES was rejected by Microsoft Exchange Online at the recipient domain

Technical Diagnosis: Layer 1 – Gmail SMTP Relay Configuration

The jadasailing@gmail.com account is configured with Gmail's "Send mail as" feature to send mail from admin@queenofsandiego.com. This feature allows sending through an external SMTP server rather than Google's own infrastructure.

To verify the SMTP configuration, we checked:

# Locate SMTP credentials in local configuration
find ~/ -type f -name "*.env" | xargs grep -l "SMTP\|SES"
find ~/ -type f -name "*repos*" -o -name "*config*" | grep env

The diagnosis revealed that Gmail's stored credentials for the SES SMTP relay had become stale. AWS SES SMTP credentials, unlike long-lived IAM access keys, must be explicitly managed and can be rotated. The credentials stored in Gmail's "Send mail as" settings were no longer valid.

Why this matters: When a user sends through Gmail's "Send mail as" with external SMTP, Gmail acts as an SMTP client. If the credentials are invalid, Gmail cannot authenticate to the relay server and bounces the message. The bounce comes back to the real sending account (jadasailing@gmail.com) because Gmail is the one that failed to relay, not because of downstream server rejection.

Technical Diagnosis: Layer 2 – SES Infrastructure Validation

Before addressing the Gmail configuration, we verified that the underlying SES infrastructure was properly configured. We checked:

# Verify SES identities and verification status
aws ses list-verified-email-addresses --region us-east-1
aws ses get-identity-dkim-attributes --identities admin@queenofsandiego.com --region us-east-1

The verification results showed:

  • Domain identity queenofsandiego.com was verified and active
  • All three DKIM CNAME records were properly configured in Route53
  • SPF record for queenofsandiego.com included SES endpoint: include:amazonses.com
  • DMARC policy was configured at _dmarc.queenofsandiego.com
  • Custom MAIL FROM domain was set to mail.queenofsandiego.com with appropriate MX records

This validation confirmed that SES itself was properly configured to send mail from the domain. The issue was not with SES infrastructure, but with how Gmail was trying to relay through it.

Technical Diagnosis: Layer 3 – Exchange Online Rejection

After identifying the Gmail relay issue, we investigated suppression list entries to understand why other emails were being rejected by the recipient's mail server. We checked:

# List SES suppression list entries by reason
aws ses list-suppressed-destinations --region us-east-1 --reason COMPLAINT
aws ses list-suppressed-destinations --region us-east-1 --reason BOUNCE

The suppression list showed complaint entries dated April 18, indicating that at least one recipient domain (identified as steigerwald-dougherty.com running Microsoft Exchange Online) had marked previous campaigns as complaints. When SES attempts to send to an address that has previously generated complaints, it automatically suppresses the message.

Why this happens: SES maintains a suppression list that includes addresses flagged as bounces or complaints. This is a sender reputation protection mechanism—if a user marked mail as spam, SES won't attempt to deliver there again unless the sender explicitly requests removal.

Infrastructure Analysis and Root Cause

We traced the email flow to understand how different sending mechanisms were being used:

# Check for SMTP configuration in environment files
grep -r "SMTP_SERVER\|SMTP_USER\|SES_SMTP" ~/repos --include="*.env" --include="*.py"

# Examine the email sending code
cat ~/repos/jada_blast.py | grep -A 20 "def send_email"

The investigation revealed that the platform uses multiple sending paths:

  • Path 1: Gmail's SMTP relay via "Send mail as" (used for manual sends from jadasailing@gmail.com)
  • Path 2: Direct SES API calls via Lambda functions (jada-blast-sender) for bulk campaigns
  • Path 3: SES SMTP from Python daemons for operational notifications

The Gmail relay path was using hardcoded or cached credentials that had expired. The SES API paths were working but hitting suppression list blocks.

Technical Decisions and Resolution

Decision 1: Regenerate SES SMTP Credentials

Rather than trying to recover old credentials, we decided to regenerate new SES SMTP credentials. AWS SES SMTP credentials are derived from IAM access keys using a region-specific algorithm. We created new credentials using:

# In AWS Console: SES → Account Dashboard → SMTP Settings → Create New SMTP Credentials
# This generates a new SMTP username and password specific to the region

These credentials must be updated in Gmail's "Send mail as" settings at:

Settings → Accounts and Import → Send mail as → [Select admin@queenofsandiego.com] → Edit Info → Update SMTP Server → Test

Decision 2: Suppress List Management

For the Exchange Online complaints, we verified the reason for suppression and documented it. Rather than immediately removing entries, we determined that this is correct behavior—the recipient domain previously marked mail as spam, so suppression is appropriate.

Decision 3: Multi-Path Audit

To prevent similar issues, we documented all email sending paths in the codebase and their credential sources. This ensures future maintainers understand which credentials affect which sending mechanisms.

What's Next

  • Update Gmail "Send mail as" SMTP credentials with newly generated SES credentials
  • Add monitoring for SES SMTP credential expiration (set calendar reminder for credential rotation)
  • Document all email sending mechanisms in runbooks with explicit credential sourcing
  • Implement credential rotation checks in CI/CD pipeline
  • Consider moving away from Gmail relay to direct SES API calls for better auditability and control

This investigation demonstrates the importance of layered diagnostics when debugging email delivery. A single bounce notification required examining