A regional ISP's edge refresh: dual 100G uplinks, BNG, carrier-grade NAT and sub-second failover

A regional internet provider with about 40,000 subscribers had outgrown its edge. One router pair with 10G uplinks carried the peak with little room, a failed uplink took minutes to recover, and subscribers ended on aging PPPoE concentrators. Public IPv4 addresses were running out, and the old NAT kept logs that were hard to search. The company could not afford an outage, so the refresh had to be built beside the old edge and cut over in stages.

The new edge has two routers, each with 100G QSFP28 uplinks, one toward each of two transit providers, plus a 100G port at an internet exchange. BGP runs on both routers (RFC 4271) with BFD for failure detection, so a dead link is noticed in well under a second instead of waiting out a hold timer measured in tens of seconds. Prefix lists and IRR data filter what neighbors may announce, RPKI origin validation (RFC 6811) drops announcements that claim someone else's prefix, and communities steer traffic between the transits.

Subscribers connect through two BNGs in active-active pairs. The new fiber footprint uses IPoE: a subscriber's device asks for an address with DHCP, and DHCP option 82 (RFC 3046) tells the BNG which port the request came from, with no PPPoE session to keep alive and no 8‑byte PPPoE overhead cutting the MTU to 1,492. PPPoE stays only on legacy wireless links that need it. IPv6 runs dual-stack with DHCPv6 prefix delegation, and because most large content networks speak IPv6, a large share of traffic never touches the NAT.

Carrier-grade NAT handles the subscribers that still need IPv4. The design follows RFC 6888, which requires a CGN to use endpoint-independent mapping and filtering behavior where possible, to limit ports per subscriber and to keep applications working, and uses the shared address space of RFC 6598 (100.64.0.0/10) between subscribers and the NAT. A /23 of public addresses serves the pool, and port-block allocation hands each subscriber a block of ports, such as 1,024, so a log line names a subscriber, an address and a port range rather than recording every flow. When a legal request arrives, the team can answer in minutes.

Protection and operations round it out. BGP FlowSpec and remotely triggered blackhole routing drop flood traffic at the edge, and a tested path to a scrubbing provider handles what the edge cannot. A peering map shows where traffic goes, so the next capacity decision follows the data. The cutover took several maintenance windows, each with a rollback, and the old routers stayed warm until the new ones had carried a full month. Subscriber counts, address sizes and failover times in this illustrative case are modeled for a refresh of this kind. Our articles on BGP with BIRD and on surviving a DDoS flood explain the techniques.