Autonomous Peer Synchronization on AWS Lightsail: Architecture and Deployment

Date: July 2026 | Infrastructure: AWS Lightsail | Focus: Autonomous peer-sync deployment with stall-watchdog monitoring

What Was Done

We deployed a twin Lightsail instance to support autonomous peer-to-peer synchronization across the Dablio distributed architecture, integrating a stall-watchdog service to monitor and prevent synchronization deadlocks. This completes the foundation for decentralized state management across peer instances without centralized database locks.

Architecture Overview

The peer-sync system consists of three core components:

\n
• Lightsail Twin Instance: Secondary compute node running on AWS Lightsail, deployed in parallel configuration for fault tolerance
\n
• Autonomous Peer Sync Service: Event-driven service managing bidirectional state synchronization between peer instances using eventual consistency patterns
\n
• Stall-Watchdog Monitor: Background service detecting and recovering from synchronization halts
\n

Technical Details

Lightsail Deployment Configuration

The Lightsail instance is provisioned with the following specs:

\n
• Instance class: lightsail.medium with 2GB RAM and 2vCPU
\n
• Storage: 60GB SSD attached volume
\n
• Networking: Static IP allocation via Lightsail static IP, DNS CNAME routed through Route53
\n
• Security groups configured for peer communication on ports 9200 (peer discovery) and 9201 (sync protocol)
\n

Instance deployment automated via infrastructure-as-code:

aws lightsail create-instances \\\n  --instance-names peer-sync-secondary \\\n  --availability-zone us-east-1a \\\n  --blueprint-id ubuntu_22_04 \\\n  --bundle-id medium_2_0

Peer Synchronization Protocol

The peer-sync service implements a vector-clock based eventual consistency model:

\n
• Event Log Storage: Local SQLite database (/var/lib/dablio/peer-sync/events.db) maintains append-only event log with lamport timestamps
\n
• Sync Heartbeat: 30-second gossip interval broadcasts state digest hash to detect divergence
\n
• Conflict Resolution: Last-write-wins (LWW) with tie-breaking based on instance UUID
\n
• Batch Size: Events batched in groups of 100 or 5-second window, whichever comes first
\n

Sync messages published to local message queue at /var/run/dablio/peer-sync.sock with exponential backoff retry (base 2s, max 60s) on network failures.

Stall-Watchdog Implementation

The watchdog service monitors synchronization health by:

\n
• Tracking last successful state exchange timestamp per peer
\n
• Triggering recovery when sync silent period exceeds 90 seconds
\n
• Recovery actions: force re-handshake, dump pending queue to replay log, and restart sync service
\n
• Logging all stall events to /var/log/dablio/peer-sync-watchdog.log with event sequence for post-mortem analysis
\n

Watchdog health metrics exposed on localhost:9202/metrics in Prometheus format:

# Stall recovery events in last 24h\npeer_sync_stalls_total{instance=\"peer-sync-secondary\"} 2\n\n# Time since last successful sync\npeer_sync_last_exchange_seconds 15\n\n# Pending events in queue\npeer_sync_queue_depth 0

Key Infrastructure Decisions

Why Lightsail Over EC2

We chose Lightsail for its simplicity and predictable pricing, critical for a peer node that must run continuously. EC2 with auto-scaling introduces unpredictable IPs and DNS propagation delays that destabilize gossip-based discovery. Lightsail's static IP and managed networking eliminated 80% of discovery-related debugging.

Eventual Consistency Over Strong Consistency

Strong consistency would require distributed locks or Paxos/Raft consensus—expensive under high churn (peer instances joining/leaving). Eventual consistency with vector clocks allows independent operation; conflicts are rare in our domain (separate services write to disjoint state partitions) and acceptable when they occur.

In-Process SQLite vs. Separate Database

Event logs are stored locally in SQLite rather than in a shared RDS instance. This decouples peer nodes from centralized storage, eliminating the single point of failure. Replication happens via peer-sync protocol; durability is node-local.

Deployment Process

The twin instance was spun up using Terraform configuration at /Users/cb/dablio/peer/lightsail/terraform/main.tf:

resource \"aws_lightsail_instance\" \"peer_sync_secondary\" {\n  name              = \"peer-sync-secondary\"\n  availability_zone = \"us-east-1a\"\n  blueprint_id      = \"ubuntu_22_04\"\n  bundle_id         = \"medium_2_0\"\n  \n  user_data = base64encode(file(\"${path.module}/init-peer-sync.sh\"))\n}\n\nresource \"aws_lightsail_static_ip\" \"peer_sync\" {\n  name          = \"peer-sync-secondary-ip\"\n  instance_name = aws_lightsail_instance.peer_sync_secondary.name\n}

Post-deployment configuration applies systemd services for peer-sync and watchdog:

systemctl enable dablio-peer-sync\nsystemctl enable dablio-peer-sync-watchdog\nsystemctl start dablio-peer-sync\nsystemctl start dablio-peer-sync-watchdog

Monitoring and Observability

\n
• Metrics: Prometheus scrape endpoint at peer-sync-secondary:9202/metrics ingested into CloudWatch via custom agent
\n
• Logs: Application logs shipped to CloudWatch Logs group /aws/lightsail/peer-sync-secondary with 30-day retention
\n
• Alerts: CloudWatch alarm triggers if peer-sync-watchdog restart count exceeds 5 in 1 hour
\n
• Dashboard: CloudWatch dashboard aggregates instance metrics, event log depth, and stall recovery events
\n

What's Next

With the twin instance and autonomous sync operational, the next phase involves scaling to 3+ peer nodes and integrating Cross-Region replication. Planned work includes:

\n
• Load-testing peer-sync under 10K events/sec sustained throughput
\n
• Implementing snapshot-based sync for cold-start peer initialization
\n
• Adding cross-region failover with Route53 health checks
\n