Sail JADA — Engineering Log

Technical work on sailjada.com and the JADA booking systems

Debugging DNS Shadowing: How a Wildcard CNAME Broke adamcherrycomics Subdomain Resolution

When http://adamcherrycomics.dangerouscentaur.com went down for hours despite CloudFront being healthy, the investigation revealed a subtle but critical DNS configuration issue: wildcard CNAME shadowing. This post walks through the debugging process, the root cause, and the fix.

Initial Symptoms

The site was returning no response. HTTP checks failed, DNS lookups failed, but CloudFront itself was healthy and serving content when accessed directly with the correct Host header. This pointed to a DNS resolution problem rather than an infrastructure issue.

The debugging workflow:

  • HTTP response check for the subdomain → timeout/no response
  • Verbose curl with explicit timeout → connection failed
  • CloudFront distribution E2Q4UU71SRNTMB status → healthy
  • CloudFront direct test with Host header → content served successfully
  • DNS lookup against Namecheap authoritative nameservers → missing records

The evidence pointed to DNS being the culprit.

Understanding the Configuration

The site infrastructure uses:

  • Domain: dangerouscentaur.com (registered at Namecheap)
  • DNS: Namecheap authoritative nameservers (pdns1.namecheap.com, pdns2.namecheap.com, etc.)
  • CDN: CloudFront distribution E2Q4UU71SRNTMB
  • Target subdomain: adamcherrycomics.dangerouscentaur.com
  • S3 origin: Static HTML and assets in S3, invalidated via CloudFront
  • Serverless checkout: Lambda function adam-cherry-checkout handling Stripe payments

The DNS strategy was to use a wildcard CNAME (*) to point all subdomains to CloudFront. This allowed any subdomain like adamcherrycomics.dangerouscentaur.com to resolve to the CDN.

The Root Cause: RFC 1034 § 4.3.3 Wildcard Shadowing

When a DNS query for adamcherrycomics.dangerouscentaur.com arrives, the nameserver looks for:

  1. An exact match for adamcherrycomics.dangerouscentaur.com
  2. A wildcard match for *.dangerouscentaur.com

The problem: a CNAME record for www.adamcherrycomics was present in the DNS zone. According to RFC 1034, this creates a DNS node at adamcherrycomics (the parent of www.adamcherrycomics). When a DNS node exists, even if it has no records, the wildcard cannot match. This is called wildcard shadowing.

The sequence was:

  • Query: adamcherrycomics.dangerouscentaur.com
  • Nameserver finds node adamcherrycomics (created by www.adamcherrycomics CNAME)
  • Node exists but has no CNAME, so query returns NODATA
  • Wildcard never consulted because a node exists
  • Resolution fails

To verify, queries were run directly against Namecheap authoritative servers:

dig adamcherrycomics.dangerouscentaur.com @pdns1.namecheap.com
dig adamcherrycomics.dangerouscentaur.com @pdns2.namecheap.com

These returned NODATA. Checking the full zone showed the wildcard * CNAME was present, but the adamcherrycomics node itself had no records.

The Fix: Explicit CNAME

The solution was to add an explicit CNAME record for adamcherrycomics pointing to the same CloudFront distribution as the wildcard.

Records in Namecheap after fix:

  • * → CNAME to CloudFront (for catch-all subdomains)
  • adamcherrycomics → CNAME to CloudFront (explicit, breaks wildcard shadowing)
  • www.adamcherrycomics → CNAME to CloudFront (existing record)

The Namecheap API was used to fetch all existing records, merge in the new adamcherrycomics CNAME, and update the zone. The new record received HostId 511341200 in the Namecheap system.

After adding the record:

  • Verified the record appeared in Namecheap's authoritative response
  • Polled pdns2.namecheap.com until resolution succeeded
  • Verified HTTPS response from the site
  • Tested DNS propagation across major resolvers (Google 8.8.8.8, Cloudflare 1.1.1.1, and Namecheap authoritative servers)

Parallel Site Updates

While diagnosing the DNS issue, several content updates were made:

  • Lambda checkout function: /Users/cb/Documents/repos/sites/adamcherrycomics.com/lambda/checkout.py was rebuilt multiple times with dependencies (typing_extensions) and configuration fixes (ui_mode parameter for Stripe hosted page mode)
  • HTML updates: Modified index.html, about-artist.html, contact.html, shipping.html for styling and content fixes (dropdown gap in navigation, artist photo replacement)
  • Deployment: Files pushed to S3 with CloudFront cache invalidations via distribution E2Q4UU71SRNTMB

The Lambda function was tested with direct invocation:

aws lambda invoke --function-name adam-cherry-checkout \
  --payload '{"body": "{\"product_name\": \"hells-lounge\"}"}' \
  response.json

This verified the checkout flow was working independently of the DNS issue.

Key Learnings

  • Wildcard shadowing is subtle: A DNS node blocks wildcard matching even if the node has no records. Always verify with direct authoritative nameserver queries, not just local resolvers.
  • Explicit records override wildcards: When wildcards are the primary routing mechanism, any parent node that exists must have an explicit record to avoid shadowing.
  • CloudFront health doesn't