Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

```html

Diagnosing Multi-Layer Email Delivery Failures: Gmail Alias SMTP Relay vs. SES API Rejections

During a recent development session, we encountered a critical email delivery issue affecting the JADA sailing platform. What appeared to be a single failure was actually two distinct problems occurring at different layers of our email infrastructure. This post documents the systematic diagnostic approach and the architectural insights that emerged.

The Problem Statement

Three emails sent via the admin@queenofsandiego.com address were bouncing with the error message: "SMTP relay misconfigured or out of date." The bounces were arriving at jadasailing@gmail.com, not at the admin address, which was our first diagnostic clue.

What We Discovered: Two Separate Failure Modes

Failure 1: Gmail "Send mail as" SMTP Relay Credential Expiration

The admin@queenofsandiego.com address is configured as a "Send mail as" alias within the jadasailing@gmail.com Google Workspace account. Gmail's "Send mail as" feature allows sending from alternative addresses by relaying through an external SMTP server—in this case, Amazon SES.

Gmail stores SMTP credentials (username and password) for this relay configuration. When a user clicks "Send" from the alias, Gmail attempts authentication with the configured SES SMTP endpoint. The error message indicates those credentials are no longer valid.

Root cause: SES SMTP credentials are derived from IAM access keys. When an IAM access key is rotated or regenerated, the corresponding SES SMTP password becomes invalid. This can happen via:

  • Automated key rotation policies in IAM
  • Manual regeneration of credentials in the AWS console
  • Removal and re-creation of the SES SMTP user

The Gmail alias had no way of knowing credentials had changed—it simply failed on send and bounced back to the primary account.

Failure 2: SES API Rejections at Recipient Domain

A separate path exists in our architecture: direct SES API calls from backend services (documented in jada_blast.py and Lambda functions like jada-blast-sender). These emails are sent directly through SES API, bypassing Gmail entirely.

The SES configuration for queenofsandiego.com is clean:

  • DKIM: All three DKIM CNAME records properly configured in Route53 and verified in SES
  • SPF: Record includes SES endpoint: v=spf1 include:amazonses.com ~all
  • DMARC: Policy set to p=quarantine with proper forensics reporting
  • Custom MAIL FROM: Domain configured as mail.queenofsandiego.com with corresponding MX records

Despite correct sender authentication, emails were being rejected by the recipient's mail server (in this case, Exchange Online at steigerwald-dougherty.com). This indicates a content-level or policy-level rejection, not an authentication failure.

Technical Investigation Approach

Step 1: SSH to JADA Lightsail Instance

We began by inspecting the on-premises agent code:

ssh -i ~/.ssh/jada-west-2.pem ubuntu@[lightsail-ip]
cat ~/bin/jada_blast.py  # Email sending implementation
tail -f ~/logs/daemon.log  # Real-time activity monitoring

This revealed two separate code paths for email: the daemon process uses SES API directly, while the Gmail interface uses SMTP relay.

Step 2: SES Configuration Audit

We systematically verified SES setup across regions:

aws ses list-verified-email-addresses --region us-east-1
aws ses get-identity-dkim-attributes --identities admin@queenofsandiego.com --region us-east-1
aws route53 list-resource-record-sets --hosted-zone-id [zone-id] --query "ResourceRecordSets[?contains(Name, 'dkim')]"

SES identities were verified, DKIM was correctly signed, and suppression lists were checked for any blocking rules.

Step 3: SES SMTP Credentials Derivation

For the Gmail alias issue, we derived the correct SES SMTP credentials from current IAM keys using AWS's v4 signature algorithm:

aws iam list-access-keys --user-name [ses-smtp-user]
# Then derive SMTP password from the access key using SES SMTP v4 signature

We validated the derived credentials against SES SMTP endpoints to confirm they were currently active and not expired.

Step 4: Lambda and Campaign Configuration Review

We examined the serverless email sending infrastructure:

aws lambda get-function-configuration --function-name jada-blast-scheduler --region us-west-2
aws lambda get-function-code --function-name jada-blast-sender --region us-west-2 --query Location

This revealed campaign metadata stored in S3 (s3://jada-blast-assets/campaigns/) and scheduler state in DynamoDB. We cross-referenced Paul Simon campaign deliverability against the suppression list.

Infrastructure Architecture Insights

Our investigation exposed an important architectural pattern: dual email paths create dual failure modes.

  • Path 1 (Interactive): User sends from Gmail alias → Gmail SMTP relay to SES → Recipient
  • Path 2 (Automated): Lambda/Daemon → SES API → Recipient

Each path has independent credential and authentication requirements. Path 1 stores credentials in Gmail's security model; Path 2 uses IAM-based access. When either credential set is rotated, the corresponding path fails silently.

Key Decisions and Rationale

Why didn't we immediately see this as two problems? The bounce messages came back to the same inbox, but the error signature ("misconfigured relay") is unique to Gmail's SMTP relay handler, not SES API. This was the diagnostic breakthrough.

Why is SES configuration clean but delivery still fails? Authentication (DKIM/SPF/DMARC) passes the first hop; recipient domain policies (content filtering, external sender restrictions, DLP rules) are a separate layer entirely. A message can have perfect credentials but still be rejected by Exchange Online policy.

Why check suppression lists? SES maintains a bounce/complaint suppression list per region. If a recipient's address was previously marked as a complaint, SES refuses to send to it. This is a silent fail: the email never reaches the recipient's server.

What's Next

To resolve Failure 1, the Gmail alias requires updated SMTP credentials in Google Workspace settings. The fix requires:

  • Regenerating SES SMTP credentials in AWS IAM
  • Updating the "Send mail as" alias configuration in jadasailing@gmail.com settings
  • Running a test send to validate the new credentials work

For Failure