Infrastructure · Incident & Resilience
On 19 July 2026 the boom-hosting droplet ran out of memory and CPU and took nine client sites offline. Five days earlier the other droplet did the same thing. This is the post-mortem, the fix, and — the part that actually matters — an honest ladder of redundancy options with what each one costs and what it genuinely buys you. Includes the wedged-server runbook so the next one takes ten minutes instead of an hour.
Droplet ubuntu-boom-hosting-nyc1 · 24.199.95.101 · NYC1 · Ubuntu 24.04 LTS
Connection timed out during banner exchange. Kernel and NIC alive; sshd cannot fork or allocate to complete a login. That signature is memory exhaustion, not a network fault.159.65.235.140) went down the same way — a Coolify next build (a Rankee webhook deploy) ran on the box hosting the Coolify control plane, drove load average to 167, and took the shared Supabase down with it for about an hour.
Mitigations were applied that afternoon — concurrent builds cut 2→1, a NODE_OPTIONS=--max-old-space-size=1024 build cap on the three apps, and swap raised 2→4 GB. But they were applied to that droplet only. boom-hosting never got the same treatment, and the durable fix — getting builds off production entirely — remained open. Same class of failure, different box. See Coolify setup guide.
This is the idea the whole document hangs on. Getting it wrong means spending money on the wrong fix.
"Can I get my data back?"
DO intra-daily backups were enabled and the most recent snapshot was 32 minutes old when the box wedged. Zero data was lost. Zero data was ever at risk. The shared Supabase Postgres everything depends on lives on a different droplet and was never touched.
"Are my sites up right now?"
Nine client sites served errors for an unknown duration. No amount of additional backup would have prevented one second of that. Availability is bought with monitoring, headroom, and redundancy — three different things from backups.
You cannot reason about redundancy without knowing the blast radius of each box. Two droplets, one managed platform.
| Host | Address | Runs | If it dies | Redundancy today |
|---|---|---|---|---|
| boom-hosting | 24.199.95.101 |
9 client Next.js sites (bm-portal, bmbb / haw / rar / rlg-vercel, eric-doerr, haw-portal, posttee, untouchables) · Traefik · MediaMTX camera streaming · Wakapi · Coolify builds | 9 client sites + all camera streams offline | None |
| Supabase droplet | 159.65.235.140 |
Self-hosted Supabase (db.boommedia.us) · Coolify control plane · Rankee |
Every app's database. Ecosystem-wide outage. | None |
| Vercel | managed | 11 SaaS apps | Vercel's problem, not yours — no single box to fail | Built in |
-vercel: they were moved off a platform with no single point of failure onto one that has exactly one.What was undersized, what it went to, and the one checkbox worth knowing about.
The original 4 GB was carrying nine Next.js apps, Traefik, MediaMTX transcoding live camera streams, Wakapi, and Coolify builds. A single next build routinely wants 2–4 GB on its own. This was not a leak to hunt down — the box was genuinely oversubscribed from day one.
Why 4 vCPU and not just more RAM. The box was pinned at 99.3% CPU and 96.8% memory. The tempting $42/mo option is 2 vCPU / 8 GB — it fixes memory and leaves the CPU ceiling exactly where it was. MediaMTX transcoding is CPU-bound and bursty; next build pegs every core it can reach. Roughly $6/mo more buys double the CPU. Take it.
DigitalOcean resizes are permanent by default — grow the disk and you can never scale back down. Ticking "Downscale anytime by keeping the storage size fixed (80 GB)" preserves the ability to shrink later. It does not reduce the price, so it is free optionality. Tick it.
The one trade-off: you stay capped at 80 GB of disk. Across nine apps, Docker images and build caches accumulate fast, so pair this with a periodic docker system prune. Disk was not a factor in this outage, but it is the next thing to creep up on you.
You found out by looking at a dashboard. That is the failure to fix first, and it is nearly free.
Free · already available on the droplet · ~10 min
Droplet page → Monitoring → create alert policies. These catch the slow climb before the wedge:
UptimeRobot free tier or Better Stack · ~20 min
Monitoring that runs on the server cannot tell you the server is gone. External checks can:
Three changes, in descending order of value. The first one is the actual root cause of both outages.
This is the root cause of both incidents. A next build competing with live traffic for 2 shared vCPUs is what drove load to 167 on the Supabase droplet and 99.3% here. Resizing buys headroom; it does not stop a build from eating the box during the next deploy.
Two options: point Coolify at a dedicated build server, or — cheaper and better — build the image in GitHub Actions, push it to a registry, and have Coolify pull the finished artifact. Production then never compiles anything; it only runs containers.
Unverified on boom-hosting — SSH was unreachable during the incident so the swap configuration could not be checked. The Supabase droplet was raised to 4 GB on 14 July; whether boom-hosting ever received the same treatment is unknown. Run free -h and swapon --show first, then size to 4 GB if it's short.
Swap turns "the server is gone and I can't even SSH in" into "the server is slow but I can log in and fix it." The entire reason this became a control-panel emergency rather than a five-minute SSH fix is that no shell was reachable — on 14 July, with swap present, the same failure was resolved by SSHing in and killing the build PID.
# 4 GB swapfile, survives reboot fallocate -l 4G /swapfile chmod 600 /swapfile mkswap /swapfile swapon /swapfile echo '/swapfile none swap sw 0 0' >> /etc/fstab # prefer RAM, use swap only under real pressure sysctl vm.swappiness=10 echo 'vm.swappiness=10' >> /etc/sysctl.conf
Without limits, one runaway app starves the other eight and takes sshd down with it — which is exactly what locked you out. With limits, the offender gets OOM-killed by Docker and restarts; everything else stays up and you keep your shell.
# per service, in docker-compose / Coolify resource settings mem_limit: 768m memswap_limit: 1g cpus: 0.75 # plus a global cleanup cron — reclaims disk from old images docker system prune -af --filter "until=168h"
Assume it happens anyway. How fast can you be back, and do you actually know that number?
Intra-daily backups are enabled and working — 32 minutes old at the time of failure. Keep them. This is your floor and it held.
Per the existing DR plan, a restore drill has never been run. A backup you have not restored is not a backup — it is a hope. One afternoon: restore last night's snapshot to a throwaway droplet, confirm it boots, time it, destroy it.
DO snapshots live inside DO. If the problem is ever the account rather than the droplet, offsite is what saves you. Backblaze B2 is already chosen — bucket boom-media-backups — but scripts were never installed on the droplet.
"Redundant servers" is not one thing — it's a ladder, and each rung costs real money and real complexity. Here is the honest version, including which rungs are not worth it for Boom.
One droplet, no alerts, no swap, builds on production. Any failure is a full outage of unknown duration, discovered by luck. This is what produced two incidents in five days.
Same one box, plus: alerts, external uptime checks, swap, container limits, builds moved off. You still have a single point of failure — but you find out early, you can always SSH in, and the common causes are gone. This is where the vast majority of the value is, and it's essentially free.
Move the nine client Next.js sites back to Vercel. There is no box to run out of memory, no OOM, no 2am power cycle, and deploys don't compete with live traffic because builds happen on Vercel's infrastructure. For static/SSR Next.js sites the cost is frequently near zero.
Keep the droplet for what genuinely needs a persistent server: MediaMTX camera streaming, Wakapi, and anything holding long-lived processes or state. This shrinks the blast radius of a droplet failure from "nine client sites" to "cameras and internal tooling."
Two identical droplets behind a DO Load Balancer, health-checked, traffic drains automatically from a sick node. Real HA for self-hosted services. The catch: this only works cleanly for stateless workloads. Anything holding local state (Wakapi's DB, MediaMTX sessions) needs that state externalised first, which is the actual work.
A cheaper cousin: Reserved IP + a warm standby. One extra droplet, kept current, and you remap the IP manually when the primary dies. Minutes of downtime instead of seconds, at roughly half the cost and a fraction of the complexity.
Failover across regions, managed Postgres with standby and PITR. The one piece of this worth taking early is moving the shared Supabase Postgres off the self-hosted droplet onto DO Managed Postgres — that box is a single point of failure for the entire ecosystem and it already failed once. Full multi-region is overkill at your scale.
Everything above assumes a server fails. This section is about the provider cutting you off. It's rarer, far worse, and no amount of server redundancy helps.
A load balancer and two droplets protect you from hardware failure. They protect you from nothing if the DigitalOcean account itself is suspended — because both droplets, the load balancer, the snapshots and the reserved IP are all inside the account that just got locked. Account-level failure takes out every redundancy layer simultaneously.
| Cause | Realistic? | Warning | What survives |
|---|---|---|---|
| Card declines / expires | Most likely by far | Emails first, then grace period, then suspension | Everything — if you catch the emails |
| Free-tier / plan limit hit | Plausible on Vercel | Usage warnings, then throttle or overage | Data. Usually degrades before it stops. |
| Abuse / ToS flag | Uncommon but abrupt | Often little to none | Sometimes nothing accessible |
| Account compromise | Preventable | None | Whatever is offsite |
| Provider outage (region) | Happens | Status page | Everything — it comes back |
A backup payment method on both DO and Vercel. Billing alerts to an address you actually read plus Slack. Confirm the account contact email isn't one that forwards into a folder you ignore. This is fifteen minutes and it removes the single most probable cause of total account loss.
2FA on DO, Vercel, GitHub, Backblaze and the domain registrar. The registrar matters more than people expect — losing DNS control is functionally identical to losing hosting, and it's the hardest to undo. Turn on registrar lock while you're there.
If the DO account vanished tonight, could you rebuild on another host? That depends entirely on whether these four things live somewhere else:
boom-media-backups. Chosen, but scripts were never installed on the droplet. Gap.env.local files in repos, not backed up anywhere. Doppler is provisioned but unwired. Without secrets, code and data still don't get you a running system. GapThree of those four are incomplete. This is the highest-value work in this section — and note it's the same work as offsite backups generally, so it isn't a separate project.
You're already better placed than you might think: code on GitHub, backups going to Backblaze, DNS at the registrar, 11 SaaS apps on Vercel, media on Bunny. A DO account loss would hurt badly but would not be extinction-level, because DO doesn't hold your only copy of anything except the running servers.
Moving the nine client sites back to Vercel improves this again — the same change recommended at Level 2 for a completely different reason. It splits your hosting across two providers so that no single account holds everything customer-facing.
A parallel environment at Hetzner or similar, kept current. Genuine protection against total DO loss — and a permanent second copy of everything to maintain, patch and pay for. At your scale the cost and ongoing drag outweigh the risk. A tested rebuild-from-B2 procedure gets you most of the protection for a fraction of the burden.
100% uptime isn't purchasable at any budget. AWS, Google and Cloudflare all have multi-hour outages on their record, with engineering budgets larger than most countries'. Anyone selling you "it can never go down" is selling something.
What is achievable, and what's worth aiming at:
That is a realistic target and you're a few free afternoons from most of it. The gap between "goes down sometimes, briefly, and I know immediately" and "never goes down" costs more than the entire business spends on infrastructure — and buys less than fixing the alerting, the builds, and the offsite secrets.
Where this lands if you take Levels 1 and 2 plus the one Level 4 piece.
| Workload | Today | Target | Why |
|---|---|---|---|
| 9 client sites | boom-hosting droplet | Vercel | Stateless. No box to die, no build contention, near-zero cost. |
| 11 SaaS apps | Vercel | Vercel — no change | Already correct. |
| MediaMTX cameras | boom-hosting droplet | Droplet, monitored + capped | Needs persistent processes. Can't be serverless. |
| Wakapi | boom-hosting droplet | Droplet, monitored + capped | Internal tooling. Low stakes if briefly down. |
| Builds | On production droplets | GitHub Actions → registry | Root cause of both outages. |
| Shared Postgres | Self-hosted, single droplet | DO Managed Postgres + PITR | Single point of failure for the entire ecosystem. |
| Secrets | .env.local in repos | Doppler (already provisioned) | Not currently backed up anywhere. |
Follow this next time instead of improvising. Written from what actually worked on 19 July.
# does it answer at all? ping 24.199.95.101 # can you get a shell? ssh -o ConnectTimeout=15 root@24.199.95.101 \ "uptime; free -h"
banner exchange → memory exhaustion. The box is alive but starved. This was 19 July.The DO Web Console is out-of-band — serial/VNC, not SSH — so it can work when SSH is wedged. Worth 60 seconds before you hard-cut.
systemctl stop docker poweroff
Stopping Docker first lets containers flush and exit cleanly. If the console never gives you a prompt, close it and move on — the box is too starved.
Nothing is being damaged while it sits there. The box is too starved to write meaningfully to disk. Waiting is safe.
Monthly deltas against the pre-incident baseline.
| Item | Layer | Cost / mo | Buys you |
|---|---|---|---|
| DO Monitoring alerts | 1 · Detect | $0 | Warning before the wedge |
| UptimeRobot free tier | 1 · Detect | $0 | You learn it's down before a client does |
| Swap + container limits | 2 · Prevent | $0 | Degrade instead of lock out |
| GitHub Actions builds | 2 · Prevent | $0 | Removes the root cause of both outages |
| Droplet resize 4→8 GB, 2→4 vCPU | 2 · Prevent | +$20–28 | Real headroom Done |
| Restore drill | 3 · Recover | ~$1 | A real RTO number instead of a guess |
| B2 offsite sync | 3 · Recover | ~$1–5 | Survives losing the DO account itself |
| Move 9 sites to Vercel | 4 · Redundancy | ~$0 | Removes the largest single point of failure |
| DO Load Balancer + standby | 4 · Redundancy | +$60 | Seconds of downtime — only if state moves first |
| DO Managed Postgres | 4 · Redundancy | +$60–200 | Ecosystem DB stops being one box |
| Recommended path (Levels 1–2) | — | ≈ +$25/mo | ~90% of the available benefit |
In priority order. The first two are the ones that would have changed the outcome on 19 July.
.env.local. Secrets are currently not backed up anywhere.doctl and run doctl auth init locally rather than sharing a DO token that can destroy droplets.