A law firm that can be back in under two hours: immutable backups and a tested failover to a second region
A law firm of about 300 attorneys with offices in three U.S. cities, a leader in its segment.
- Scale
- ~450 users, 3 offices, 2 regions 1,000+ miles apart, RPO 15 minutes, RTO 2 hours for Tier 1
- Engagement
- Design and build over 5 months, then quarterly drills
The challenge
The firm kept its document management system, billing and mail on servers in one metropolitan area, and its backups in the same building as the servers. A hurricane, a flood or a ransomware attack that reached the backup copy would have stopped client work and put privileged files at risk. Nobody could say how long recovery would take because nobody had timed it.
A law firm's product is the matter file, and clients ask about its protection in every outside-counsel questionnaire. This firm, about 300 attorneys in three cities, kept its document management system, billing and mail on servers in one metropolitan area, with backups in the same building. A hurricane, a flood or a ransomware crew that reached the backup copy would have stopped client work. No one could say how long recovery would take, because it had never been timed.
We started with numbers, not products. Every system was placed in a tier with a written recovery point objective (how much data may be lost) and recovery time objective (how long it may be down). Tier 1 is email, document management, billing and identity: 15 minutes and 2 hours. Tier 2 follows within 8 hours, and the rest within 3 days. Those targets decided the design. Fifteen minutes meant continuous asynchronous replication of virtual machines and database logs to a second region more than 1,000 miles away, so one storm cannot reach both copies.
Replication is not a backup, because it copies mistakes and encryption as faithfully as it copies work. So the firm also has backups on the 3-2-1-1-0 pattern: three copies on two media, one offsite, one immutable, and zero errors after verification. The immutable copy is object storage with Object Lock in compliance mode, which stops anyone, including an administrator with stolen credentials, from deleting it before its retention date. The backup system has its own identity and MFA with no trust relationship to the production directory, and restores pass through a clean-room step that scans systems before they rejoin the network.
Failover needs more than data. The second region has a domain controller, a standby network with the same addressing, IPsec tunnels from all three offices and DNS records with 60‑second TTLs, so users reach the new site without waiting for caches to expire. Microsoft 365 is backed up separately from the platform, with the same immutability. The runbook names an owner for every step and a call tree for the partners; it is short enough to follow under stress.
We rehearse it. Each quarter the team restores a Tier 1 system into the second region and times it; once a year, attorneys work from the second region for a full day. In the last full drill the firm was running in 1 hour 40 minutes, with 11 minutes of data lost, both inside target. Rule 1.6(c) of the ABA Model Rules asks lawyers to make reasonable efforts to protect client information; a tested plan, with dates and timings, is the kind of evidence that supports that effort. Our article on backups that restore describes the pattern in more detail.
What our team delivered
- Systems sorted into tiers with written targets: Tier 1 (email, document management, billing, identity) with a recovery point objective (RPO) of 15 minutes and a recovery time objective (RTO) of 2 hours; Tier 2 within 8 hours; the rest within 3 days
- Asynchronous replication of virtual machines and database transaction logs to a second region more than 1,000 miles from the primary, so one storm cannot reach both
- Backups on the 3-2-1-1-0 pattern: three copies, two media, one offsite, one immutable, zero errors after verification. The immutable copy is object storage with Object Lock in compliance mode, so nobody, including an administrator, can delete it before its retention date
- A separate backup identity with its own MFA and no trust relationship to the production directory, and a clean-room restore step that scans restored systems before they rejoin the network
- A second domain controller, a standby network with the same addressing, IPsec tunnels from all three offices to both regions and DNS records with 60-second TTLs so cutover does not wait on caches
- Microsoft 365 mailboxes and files backed up separately from the platform, with the same immutability
- A runbook with named owners, a call tree and a checklist, rehearsed every quarter; one full failover each year with attorneys using the second region for a working day
Result
In the last full drill the firm ran from the second region in 1 hour 40 minutes, with 11 minutes of data lost, both inside target. Backups are out of reach of an attacker who holds production credentials, and the partners have a measured answer when a client asks about recovery.
A DNS change takes one TTL to settle
After you change a record, resolvers keep serving the old answer until their cached copy expires. Each cache was filled at a random moment within the last TTL, so the share still serving the old answer falls in a straight line from 100% to 0 over one TTL (the time axis is logarithmic, which bends the lines). Lower the TTL a full old-TTL before a planned change. Some resolvers ignore very low TTLs, so treat these as best case. Illustrative model.
Mail that keeps flowing
Incoming mail is authenticated and scanned before it reaches anyone, mailboxes are copied across three regions so a server failure goes unnoticed, and outgoing mail leaves signed and encrypted.
Backup tiers: data lost versus time to recover
Recovery point objective (RPO) is how much recent data a restore loses; recovery time objective (RTO) is how long the restore takes. Faster tiers sit close to the production system and share its fate; slower tiers survive ransomware and site loss. The scale is logarithmic. Illustrative figures.
Failover in under a second
ISP A fails at 300 ms. BFD (RFC 5880) with a 50 ms interval and multiplier 3 declares the session down 150 ms later, BGP withdraws the routes, and the forwarding table repoints to ISP B. Traffic is back at full rate in well under a second; without BFD the router would wait for the 90 s BGP hold timer. Illustrative figures.
Facing something similar?
Tell us about your environment. We will tell you plainly what we would do and what it would take.