Building an Automated Email Harvesting and Outreach Pipeline for B2B Referral Marketing
What Was Done
We implemented an end-to-end email harvesting and outreach system targeting referral partners (funeral homes, churches, hospices, and cremation societies) in San Diego. The pipeline harvests contact information from 76+ websites, validates email domains via MX records, manages suppression lists, and sends personalized outreach campaigns through AWS SES—all orchestrated via a launchd agent on the deployment server.
Technical Architecture
Email Harvesting Pipeline
The harvester lives in /Users/cb/icloud-jada-ops/bssd-crm/harvest_emails.py and uses a multi-stage fallback approach to maximize email discovery:
- Direct HTTP parsing: Fetches homepage and common contact-form paths
- Cloudflare email protection decoding: Decodes obfuscated
data-cfemailattributes from pages using Cloudflare's client-side XOR cipher - Contact-link following: Extracts and follows contact/about/leadership pages to catch emails in secondary locations
- Redirect handling: Follows HTTP redirects for sites that gate content behind security challenges
- MX validation post-harvest: Validates harvested domains against live MX records to ensure deliverability
Results were merged from multiple finder agents (finder-A and finder-B) using /Users/cb/icloud-jada-ops/bssd-crm/merge_found_emails.py, which deduplicates and consolidates results from parallel harvesting runs across chunks of target domains.
Test-Driven Development for Outreach
Before building the sender, we created a comprehensive test suite at /Users/cb/icloud-repos/sites/burialsatseasandiego.com/tests/test_bssd_outreach.py. This validated:
- Email template rendering with substitution variables
- CSV parsing and data validation
- Graceful handling of missing data (orgs with no harvested email)
- SES dry-run behavior without sending live emails
- Suppression list logic to prevent duplicate sends
All 30+ tests pass before any outreach code executes. This caught edge cases (malformed CSV rows, empty email fields) that would have caused failures at scale.
Outreach Sender
The sender at /Users/cb/icloud-jada-ops/bssd-crm/send_bssd_outreach.py implements:
- CSV-driven campaigns: Reads target organizations and personalization fields from CSV (e.g., organization name, contact name)
- Template substitution: Renders HTML email bodies with recipient-specific variables
- Suppression filtering: Checks
/Users/cb/icloud-jada-ops/email-lists/suppression/burialsatseasandiego.csvbefore sending to skip previously contacted addresses - Dry-run mode: Logs what would be sent without calling SES, allowing validation before live campaigns
- SES integration: Uses boto3 to send through the verified SES identity for
burialsatseasandiego.com
Infrastructure and Deployment
Scheduled Execution via launchd
We created a macOS launchd agent at /Users/cb/Library/LaunchAgents/com.jada.bssd-outreach.plist to run outreach campaigns on a fixed schedule. This avoids manual triggering and keeps campaigns consistent. The agent:
- Runs the sender script at configured intervals
- Logs output to timestamped files for debugging and audit trails
- Loads into launchctl and starts automatically on system boot
AWS SES Configuration
The sender authenticates as the verified SES identity burialsatseasandiego.com and uses AWS credentials from the environment. All outreach mail is delivered through SES's SMTP gateway with bounce/complaint handling configured at the SES console level.
Homepage Integration Changes
We deployed an internal-link footer update to /Users/cb/icloud-repos/sites/burialsatseasandiego.com/index.html to improve SEO and user discovery. The change was staged, tested, and deployed to the production S3 bucket (exact bucket name stored in deployment config, not credentials). CloudFront distribution invalidation was triggered via API to clear cached content within seconds.
Key Decisions and Rationale
Why Cloudflare Decoding? Many websites use Cloudflare's client-side email obfuscation to prevent automated scraping. Rather than treating those emails as missing, we reverse-engineered the XOR cipher to recover them, increasing our harvest yield significantly. This is legitimate security-research reverse engineering of a published algorithm.
Why MX Validation? Email addresses harvested from websites can be stale or formatted incorrectly. Before adding them to a campaign, we verify that the domain accepts mail by checking live MX records. This reduces bounce rates and SES complaint scores.
Why Test-First? Outreach campaigns are high-stakes: sending duplicate emails or malformed templates damages brand reputation. Writing tests before the sender ensured all edge cases were caught before touching SES.
Why Suppression Lists? Contacted addresses are stored in a suppression CSV to prevent re-mailing the same organization multiple times across campaigns. This is a best practice for email deliverability and compliance.
Why Dry-Run Mode? Before launching live campaigns, we run the sender against the full target list in dry-run mode, which logs recipients and templates without hitting SES. This is a safety gate—we catch template bugs, CSV parsing errors, and scope issues before real emails leave our system.
Scalability and Future Work
The current pipeline harvests and validates email addresses in parallel (finder agents run on separate systems/schedules) and merges results into a single authoritative CSV. The sender is stateless and can be run on demand or on a schedule. Scaling to other cities or verticals (e.g., other memorial services) requires:
- New target domain lists (CSV format, one domain per row)
- New suppression lists per domain/service
- New email templates per campaign
No code changes are needed—the pipeline is data-driven.
Monitoring and Observability: Future improvements include automated SES bounce/complaint ingestion to update suppression lists in near-real-time, and dashboards tracking email validation rates, send success, and opens/clicks via UTM tracking.
```