Scaling Domain Availability Checks: From Whois Rate-Limiting to RDAP Registry Queries

```html

Overview

Ticket t-f860fe03 required determining how many port cities worldwide could support a "Queen of [City]" domain franchise — a tour operator concept where local boat operators could claim queenof[city].com domains. The challenge: checking availability across 130+ potential port cities while working around whois rate-limiting and designing a scalable, authoritative domain-lookup pipeline.

What Was Done

We built a multi-stage domain-availability verification system:

Initial whois approach: Created /Users/cb/icloud-jada-ops/ticket-runner/check_queenof_domains.py to batch-query the `.com` registry via whois CLI
Pivoted to RDAP: Switched from Verisign whois (throttled, returns "unknown" under load) to RDAP HTTP/JSON registry lookups (authoritative, rate-limit friendly)
Built master aggregator: Created master_check.py to orchestrate three concurrent domain-check jobs across different city categories (franchise ports, dream destinations, US cities)
Infrastructure probing: Built probe_taken.py to probe what's actually hosted on taken domains (CloudFront, S3, nginx, etc.) — key for understanding competitive landscape
Deployed web infrastructure: Stood up dragonbodyguards.com and related domains on AWS CloudFront + S3 using lambda routing, ACM certificates, and Namecheap DNS management

Technical Details: Registry Lookup Pipeline

Whois Approach (Initial, Rate-Limited)

The first implementation used Python's whois library wrapping the CLI:

#!/usr/bin/env python3
import whois
domains = ["queenofbangkok.com", "queenofdubai.com", ...]
for domain in domains:
    try:
        result = whois.whois(domain)
        status = "TAKEN" if result.domain_name else "AVAILABLE"
    except Exception as e:
        status = "UNKNOWN"  # Verisign rate-limit response

Problem: Across 130 domains, whois returned ~130 "unknown" statuses. Verisign (the `.com` registry operator) throttles by IP when queried rapidly. Even our control domain queenofsandiego.com (known to be registered) returned "unknown" — a clear signal that whois was unreliable at scale.

RDAP Solution (Authoritative, HTTP-Based)

Switched to RDAP (Registration Data Access Protocol), Verisign's HTTP/JSON endpoint. RDAP is:

Authoritative: Direct registry query, not cached or intermediated
Rate-limit friendly: HTTP 429 is explicit; no silent throttling
JSON responses: Structured data, easier to parse than whois text
Deterministic: 404 = available, 200 = registered, 429 = backoff and retry

#!/usr/bin/env python3
import requests
import time

RDAP_BASE = "https://rdap.verisign.com/com/v1/domain/"

def check_domain_rdap(domain):
    """Query Verisign RDAP for domain status."""
    url = f"{RDAP_BASE}{domain}"
    try:
        resp = requests.get(url, timeout=5)
        if resp.status_code == 404:
            return "AVAILABLE"
        elif resp.status_code == 200:
            return "TAKEN"
        elif resp.status_code == 429:
            time.sleep(2)  # Explicit backoff
            return check_domain_rdap(domain)  # Retry
    except requests.exceptions.Timeout:
        return "TIMEOUT"
    return "ERROR"

# Test control domain
print(check_domain_rdap("queenofsandiego.com"))  # Output: TAKEN ✓

Result: 0 unknowns, clean categorization. Control domain correctly identified as TAKEN.

Infrastructure: Multi-Check Orchestration

File Structure

/Users/cb/icloud-jada-ops/ticket-runner/
  ├── check_queenof_domains.py      # Franchise port cities
  ├── check_queenof_dream.py        # Dream destinations (scenic, romantic)
  ├── check_queenof_us.py           # US port cities only
  ├── master_check.py               # Orchestrator: runs all three, aggregates results
  └── probe_taken.py                # Probes hosting details for taken domains

Master Orchestrator Pattern

master_check.py uses subprocess + file aggregation:

#!/usr/bin/env python3
import subprocess
import json
from datetime import datetime

checks = [
    ("check_queenof_domains.py", "QUEEN-OF-FRANCHISE-DOMAINS"),
    ("check_queenof_dream.py", "QUEEN-OF-DREAM-DESTINATIONS"),
    ("check_queenof_us.py", "QUEEN-OF-US-CITIES"),
]

results = {}
for script, label in checks:
    print(f"Running {script}...")
    subprocess.run(["python3", script])
    
    # Aggregator expects output file named "{LABEL}-{DATE}.md"
    output_file = f"{label}-{datetime.now().strftime('%Y-%m-%d')}.md"
    with open(output_file, 'r') as f:
        results[label] = f.read()

# Write consolidated report
with open("CONSOLIDATED-REPORT.md", 'w') as f:
    for label, content in results.items():
        f.write(f"\n## {label}\n{content}\n")

Infrastructure Probing: What's Actually Hosted?

Created probe_taken.py to understand the competitive landscape — for domains we can't register, what's their current use case?

#!/usr/bin/env python3
import socket
import subprocess

def probe_domain(domain):
    """HTTP HEAD request to determine hosting provider."""
    try:
        # Resolve to IP
        ip = socket.gethostbyname(domain)
        
        # Check CloudFront (CNAME pattern)
        result = subprocess.run(["host", domain], capture_output=True, text=True)
        if "cloudfront" in result.stdout.lower():
            return "CloudFront"
        
        # Reverse-lookup to detect S3 / nginx / other
        result = subprocess.run(["host", ip], capture_output=True, text=True)
        if "amazonaws" in result.stdout.lower():
            return "AWS S3 / ELB"
        
        # HTTP header inspection
        import requests
        resp = requests.head(f"https://{domain}", timeout=3, allow_redirects=False)
        if "Server" in resp.headers:
            return resp.headers