Aventura, Florida · serving clients across the United States (305) 404-9404Toll-free (800) 767-8566Emergency? Get a call within 1 hour
Legal services Illustrative engagement

A law firm that can be back in under two hours: immutable backups and a tested failover to a second region

A law firm of about 300 attorneys with offices in three U.S. cities, a leader in its segment.

Scale
~450 users, 3 offices, 2 regions 1,000+ miles apart, RPO 15 minutes, RTO 2 hours for Tier 1
Engagement
Design and build over 5 months, then quarterly drills

The challenge

The firm kept its document management system, billing and mail on servers in one metropolitan area, and its backups in the same building as the servers. A hurricane, a flood or a ransomware attack that reached the backup copy would have stopped client work and put privileged files at risk. Nobody could say how long recovery would take because nobody had timed it.

A law firm's product is the matter file, and clients ask about its protection in every outside-counsel questionnaire. This firm, about 300 attorneys in three cities, kept its document management system, billing and mail on servers in one metropolitan area, with backups in the same building. A hurricane, a flood or a ransomware crew that reached the backup copy would have stopped client work. No one could say how long recovery would take, because it had never been timed.

We started with numbers, not products. Every system was placed in a tier with a written recovery point objective (how much data may be lost) and recovery time objective (how long it may be down). Tier 1 is email, document management, billing and identity: 15 minutes and 2 hours. Tier 2 follows within 8 hours, and the rest within 3 days. Those targets decided the design. Fifteen minutes meant continuous asynchronous replication of virtual machines and database logs to a second region more than 1,000 miles away, so one storm cannot reach both copies.

Replication is not a backup, because it copies mistakes and encryption as faithfully as it copies work. So the firm also has backups on the 3-2-1-1-0 pattern: three copies on two media, one offsite, one immutable, and zero errors after verification. The immutable copy is object storage with Object Lock in compliance mode, which stops anyone, including an administrator with stolen credentials, from deleting it before its retention date. The backup system has its own identity and MFA with no trust relationship to the production directory, and restores pass through a clean-room step that scans systems before they rejoin the network.

Failover needs more than data. The second region has a domain controller, a standby network with the same addressing, IPsec tunnels from all three offices and DNS records with 60‑second TTLs, so users reach the new site without waiting for caches to expire. Microsoft 365 is backed up separately from the platform, with the same immutability. The runbook names an owner for every step and a call tree for the partners; it is short enough to follow under stress.

We rehearse it. Each quarter the team restores a Tier 1 system into the second region and times it; once a year, attorneys work from the second region for a full day. In the last full drill the firm was running in 1 hour 40 minutes, with 11 minutes of data lost, both inside target. Rule 1.6(c) of the ABA Model Rules asks lawyers to make reasonable efforts to protect client information; a tested plan, with dates and timings, is the kind of evidence that supports that effort. Our article on backups that restore describes the pattern in more detail.

What our team delivered

  • Systems sorted into tiers with written targets: Tier 1 (email, document management, billing, identity) with a recovery point objective (RPO) of 15 minutes and a recovery time objective (RTO) of 2 hours; Tier 2 within 8 hours; the rest within 3 days
  • Asynchronous replication of virtual machines and database transaction logs to a second region more than 1,000 miles from the primary, so one storm cannot reach both
  • Backups on the 3-2-1-1-0 pattern: three copies, two media, one offsite, one immutable, zero errors after verification. The immutable copy is object storage with Object Lock in compliance mode, so nobody, including an administrator, can delete it before its retention date
  • A separate backup identity with its own MFA and no trust relationship to the production directory, and a clean-room restore step that scans restored systems before they rejoin the network
  • A second domain controller, a standby network with the same addressing, IPsec tunnels from all three offices to both regions and DNS records with 60-second TTLs so cutover does not wait on caches
  • Microsoft 365 mailboxes and files backed up separately from the platform, with the same immutability
  • A runbook with named owners, a call tree and a checklist, rehearsed every quarter; one full failover each year with attorneys using the second region for a working day

Result

In the last full drill the firm ran from the second region in 1 hour 40 minutes, with 11 minutes of data lost, both inside target. Backups are out of reach of an attacker who holds production credentials, and the partners have a measured answer when a client asks about recovery.

A DNS change takes one TTL to settle

After you change a record, resolvers keep serving the old answer until their cached copy expires. Each cache was filled at a random moment within the last TTL, so the share still serving the old answer falls in a straight line from 100% to 0 over one TTL (the time axis is logarithmic, which bends the lines). Lower the TTL a full old-TTL before a planned change. Some resolvers ignore very low TTLs, so treat these as best case. Illustrative model.

A DNS change takes one TTL to settleLine chart on a log time axis of the share of resolvers still serving the old DNS answer after a change. With a TTL of 300 seconds the old answer is gone after 5 minutes, with 3600 seconds after 1 hour, with 86400 seconds after 24 hours. Each line falls linearly from 100 percent to zero over one TTL.Resolvers still serving the old answer after a changeTTL 300 sTTL 3600 sTTL 86400 s (24 h)0%25%50%75%100%10 s1 min5 min1 h6 h24 htime since the record changed (log scale)gone at 5 mingone at 1 hgone at 24 hWorst case for a change you did not plan: a long TTL means a long tail of stale answers.

Mail that keeps flowing

Incoming mail is authenticated and scanned before it reaches anyone, mailboxes are copied across three regions so a server failure goes unnoticed, and outgoing mail leaves signed and encrypted.

Mail that keeps flowingDiagram of enterprise email flow. Inbound mail from an internet sender reaches the MX edge server running Postfix, where SPF, DKIM and DMARC checks light up in sequence, then passes a spam and malware filter; a spam message is diverted to quarantine. Clean mail is delivered to a mailbox cluster, a database availability group with copies in three regions. On a loop the active server in region A fails and the copy in region B activates automatically, so the next message is still delivered. Outbound mail from the user's mailbox is DKIM-signed and sent over an encrypted TLS connection to the recipient. INBOUNDOUTBOUNDSenderInternetMX edgePostfixSPFDKIMDMARCThreat filterSpam · malwareQuarantineMAILBOX CLUSTER · DAGRegion ARegion BRegion CActiveCopyActiveCopyMailboxUser inboxDKIM signingTLS encryptedRecipientSPFDKIMDMARCSPFDKIMDMARCOffline

Backup tiers: data lost versus time to recover

Recovery point objective (RPO) is how much recent data a restore loses; recovery time objective (RTO) is how long the restore takes. Faster tiers sit close to the production system and share its fate; slower tiers survive ransomware and site loss. The scale is logarithmic. Illustrative figures.

Backup tiers: data lost versus time to recoverPaired horizontal bars on a logarithmic time scale. Snapshot: RPO 1 hour, RTO 15 minutes. Replica: RPO 1 minute, RTO 5 minutes. Immutable object store: RPO 1 day, RTO 4 hours. Offline copy: RPO 7 days, RTO 1 day.RPO and RTO by backup tier (log scale)RPO: data lostRTO: time to recover1 min10 min1 h6 h1 day7 daysSnapshotsame array, same failure domain1 h15 minReplicasecond site, asynchronous1 min5 minImmutable object storewrite-once, cannot be altered1 d4 hOffline copyair-gapped, no network path7 d1 dKeep one tier of each kind: the fast one restores quickly, the immutable one survives an attacker with admin rights.

Failover in under a second

ISP A fails at 300 ms. BFD (RFC 5880) with a 50 ms interval and multiplier 3 declares the session down 150 ms later, BGP withdraws the routes, and the forwarding table repoints to ISP B. Traffic is back at full rate in well under a second; without BFD the router would wait for the 90 s BGP hold timer. Illustrative figures.

Failover in under a secondLine chart over 1.2 seconds. ISP A carries 8 gigabits per second until 300 milliseconds, then drops to zero. ISP B carries nothing until about 650 milliseconds and reaches 8 gigabits per second at 720 milliseconds. Roughly 400 milliseconds of loss in between.Traffic per upstream during a link failureISP A (primary)ISP B (standby)04812Gb/s0200 ms400 ms600 ms800 ms1000 ms1200 msabout 400 ms of loss1 Link to ISP A fails2 BFD: 3 missed hellos (3 × 50 ms), session down3 Routes withdrawn, FIB repointed to ISP BWithout BFD, BGP would wait for the 90 s hold timer (30 s keepalives).

Frequently asked questions

Direct answers to the questions we hear most. Something else on your mind?

Send us a message
What recovery targets did the firm set?

Systems were sorted into tiers. Tier 1 (email, document management, billing, identity) has a 15-minute recovery point objective and a 2-hour recovery time objective. Tier 2 recovers within 8 hours and the rest within 3 days. In the last full drill the firm ran from the second region in 1 hour 40 minutes with 11 minutes of data lost.

How are the backups protected from ransomware?

The immutable copy is object storage with Object Lock in compliance mode, so nobody, including an administrator, can delete it before its retention date. A separate backup identity has its own MFA and no trust relationship to the production directory. A clean-room step scans restored systems before they rejoin the network.

How does the firm know the failover works?

It rehearses. A runbook names owners and a call tree, with a drill every quarter and one full failover each year in which attorneys work from the second region for a day. The second region is more than 1,000 miles away, and DNS records use 60-second TTLs so cutover does not wait on caches.

Do you know if your company is safe?

Speak with a security engineer today. The first assessment call is free and confidential.

CallFree assessment