Cloud & Platform · 2025
Platform Recovery & Hardening
Recovered and hardened a compromised live charity platform, then handed it back with runbooks anyone could follow.
- My role
- Lead recovery engineer · Faith Associates
- When
- 2025
- Status
- Production
- Stack
- Cloudflare WAF, PM2, Ubuntu
01
Where it started
A live client platform for British Muslim Trust had been compromised and was running unstable: no hardened edge, no automated off-site backups and no process supervision, so every traffic spike risked downtime.
02
What I owned
Lead recovery engineer · Faith Associates
03
How I built it
- Locked the origin to Cloudflare's IP ranges at the UFW layer, behind WAF rules and Authenticated Origin Pulls.
- Replaced unmanaged Node processes with a PM2 cluster that has memory-bound restarts, zero-downtime reloads and log rotation.
- Scripted daily database and file snapshots to off-site storage, then wrote the disaster-recovery runbooks and handover.
[Client Traffic / Webhooks]
│
▼
[Cloudflare Edge Network] ── (DDoS Shield, WAF Rules, SSL Termination)
│ (Strict Origin IP Filtering)
▼
[Linux Production Host (Ubuntu 22.04 LTS)]
├── UFW Firewall (80/443 allowed from Cloudflare only; SSH key-only)
├── Fail2Ban (rate-limiting jail & brute-force defence)
├── Nginx Reverse Proxy (gzip & keepalive)
│ │
│ ▼
├── PM2 Process Manager (cluster mode, restart on memory >500MB)
│ ├── Worker Instance #1
│ └── Worker Instance #2
│
└── Cron (daily encrypted DB/file snapshots -> off-site S3-compatible storage)- Edge perimeter: host origin ports restricted to Cloudflare public CIDRs, stopping direct-to-IP scanning and brute-force attempts.
- Process supervision: PM2 clusters with max-memory restart boundaries and zero-downtime reloads.
- Snapshot automation: daily incremental file backups and database dumps with 30-day retention.
Cloudflare origin cloaking & WAF rules
Authenticated Origin Pulls plus drop rules for all non-Cloudflare traffic at the UFW layer removed the exposed host ports entirely.
PM2 cluster & self-healing runtime
ecosystem.config.js with two workers, zero-downtime `pm2 reload` and automated log rotation via pm2-logrotate.
Structured technical handover
Disaster-recovery documentation, secret rotation steps and verified runbooks so non-technical stakeholders keep full operational control.
Hardening: SSH password authentication disabled in favour of Ed25519 key pairs, with automated daily integrity scans and patch routines.
04
What changed
No unplanned outages after the rollout, direct-to-host scans shut out, and a verified 15-minute recovery time objective.
- ✓Zero unplanned outages following the hardening and PM2 cluster rollout.
- ✓Malicious direct-to-host scans eliminated through Cloudflare origin cloaking.
- ✓Verified 15-minute recovery time objective using the automated backup pipeline.