A disaster-recovery design for a money-critical ordering platform: continuous point-in-time recovery, a redundant failover database, immutable offsite copies, and granular per-client / per-system restore — built on your DigitalOcean + Backblaze B2 stack.
Your 15 SaaS apps can tolerate losing an hour of data in a worst case — annoying, not catastrophic. Boom Online Ordering cannot. Every lost order is lost revenue for a restaurant client, a customer charged with no record, or a payout dispute. The bar is defined by two numbers — set them deliberately:
| Capability | Today (shared hourly dump) | BOO target |
|---|---|---|
| Max data loss (RPO) | Up to 60 minutes | ~Seconds (PITR) |
| Downtime on DB failure (RTO) | Hours (manual rebuild) | Minutes (auto-failover) |
| Node/droplet dies | Outage + possible loss | Standby promotes, stays up |
| Restore to an exact moment | No — only hourly snapshots | Yes — any second in the window |
| Restore ONE client | Manual, slow | Scripted extract from PITR |
| Ransomware / corruption | Dumps could be encrypted too | Immutable, locked offsite copy |
No single mechanism protects against everything. A node failure, a bad migration, a ransomware attack, and a provider outage are four different disasters needing four different answers. Stack them.
A second Postgres node continuously replicates the primary via streaming replication. If the primary droplet dies, the standby is promoted automatically. Keeps BOO online with near-zero downtime. Automated with Patroni, or handled for you by a managed database's standby node.
Postgres's Write-Ahead Log is shipped to B2 continuously (every few seconds). You can restore the database to any exact second — e.g. "the moment before the bad deploy at 14:32:07." This is what takes RPO from 60 minutes down to seconds. Tool: pgBackRest or WAL-G.
Full + incremental base backups (nightly full, hourly incremental) give WAL a starting point to replay from. pgBackRest manages retention automatically (e.g. keep 14 fulls). Stored in B2.
A copy of backups written with Object Lock can't be altered or deleted — even by a compromised key — until its retention expires. This is your insurance against an attacker who reaches your infrastructure and tries to destroy the backups too.
Primary DB on DigitalOcean, backups on Backblaze B2 = already two providers (good). Optionally replicate the B2 bucket to a second region so a regional B2 outage still leaves you recoverable.
Automated monthly restore drills into a scratch DB. Alerts if WAL archiving falls behind, a backup fails, or replication lag grows. An untested backup is not a backup.
Both deliver the six layers. The difference is how much database operations you run yourself. For a money system, my recommendation is to buy the hard parts (HA + failover) and self-run only the independent backups.
DigitalOcean Managed Postgres for the BOO database + pgBackRest → B2 for backups you control.
Patroni + etcd across 2–3 droplets + pgBackRest → B2.
"Restore one client to a point in time" needs a strategy, because in a shared database you can't PITR just one tenant. Three complementary techniques:
Restore the whole BOO database to a scratch instance at the target timestamp, then copy just that client's rows back into production, filtered by tenant_id. Scripted, this is a 10-minute operation for any single client, to any second.
# Restore BOO to the exact moment before an incident pgbackrest --stanza=boo --type=time \ --target="2026-07-09 14:32:00" restore # → scratch DB # Extract one client and re-import to production pg_dump --data-only \ -t orders -t transactions -t payments \ --where="tenant_id='rlg-bailbonds'" scratch_db \ | psql production_boo
Nightly, dump each active client's data to its own object in B2 (b2:boo-backups/tenants/<id>/<date>.sql.gz). When one client needs a quick rollback you restore just their file — no full-cluster restore needed. Cheap, and gives clean per-client restore points.
| Model | Per-client restore | Trade-off |
|---|---|---|
Shared DB, tenant_id column (current) | Via extract (above) | Simplest ops; restore needs filtering |
| Schema-per-tenant | Cleaner — dump/restore one schema | Moderate complexity; good middle ground |
| Database-per-tenant | Trivial — restore one DB | Best isolation; heaviest ops at scale |
For BOO specifically, schema-per-tenant is often the sweet spot: strong isolation and simple per-client restore, without a database explosion. Worth evaluating before the client count grows.
Backups protect the database; these protect the correctness of money even when systems hiccup. Layer them on top:
| Safeguard | What it buys you |
|---|---|
| Stripe as payment source-of-truth | Payment records also live in Stripe. If your DB is ever behind, reconcile against Stripe's ledger — you can rebuild payment state from it. Never rely solely on your own DB for money truth. |
| Append-only transaction ledger | Write every order/payment event as an immutable row (never UPDATE/DELETE). Corruption or a bad edit can be replayed and audited. This is the "black box recorder" for BOO. |
| Idempotency keys | Every order/charge carries a unique key so a retry after a network blip never double-charges or double-creates. Essential during failover moments. |
| Nightly reconciliation job | Automated compare of BOO orders ↔ Stripe charges ↔ payouts. Flags any drift the next morning, before a client notices. |
| Outbox pattern for integrations | Order → Rewardee points, BOO → external hooks: write the intent to an outbox table in the same transaction, deliver async. Nothing gets lost if a downstream is briefly down. |
| Scenario | Response | Layer |
|---|---|---|
| Primary DB droplet dies | Standby auto-promotes; verify app reconnects; rebuild a new standby | L1 |
| Bad migration / dropped table | PITR restore to the second before it; extract affected tables back | L2 |
| One client's data corrupted | Restore their nightly tenant export, or PITR-extract by tenant_id | L2/per-tenant |
| Ransomware on infrastructure | Rebuild from Object-Locked immutable B2 copy the attacker couldn't touch | L4 |
| DigitalOcean region outage | Restore from B2 to a droplet in another region/provider | L5 |
| Payment/DB mismatch | Reconcile against Stripe ledger; replay from append-only ledger | App |
Separate transaction data from the shared SaaS Postgres. Managed DO Postgres recommended for the primary (buys you L1 HA immediately).
Continuous WAL archiving + nightly full / hourly incremental to a dedicated boo-backups bucket with Object Lock. This is the biggest single RPO win (60 min → seconds).
Append-only ledger, idempotency keys, and the nightly Stripe reconciliation job. Protects correctness independent of the DB.
Nightly per-client dumps and a tested one-command per-client restore. Evaluate schema-per-tenant if isolation needs grow.
Alert on WAL lag / failed backups / replication lag. Automated monthly restore test into a scratch DB. Prove RTO/RPO are actually met.