01 / Campus
Collapsed core with HSRP on the STP root
IOS XE, Catalyst 9000
One building, two core switches, layer 2 to the access closets. The most common campus design there is, and the one most often built with the gateway and the spanning-tree root on different switches.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- For every VLAN, the same switch is the spanning-tree root and the HSRP active gateway. If they differ, traffic crosses the inter-core link twice.
- Split the VLANs: odd VLANs on CORE-1, even on CORE-2, so both cores and both uplinks carry traffic.
- Set explicit STP priorities (4096 and 8192). Do not use the root primary macro.
- HSRP version 2, preempt with a delay so routing and the port-channel settle before the gateway moves back.
- Inter-core link is an LACP port-channel of at least two links on different line cards or stack members.
- Root guard on core downlinks, BPDU Guard and PortFast on every access edge port, explicit allowed VLAN lists on every trunk.
Addressing
| VLAN | Subnet | Gateway (VIP) | CORE-1 | CORE-2 | Root and active |
|---|---|---|---|---|---|
| 10 Users | 10.10.10.0/24 | .1 | .2 | .3 | CORE-1 |
| 20 Voice | 10.10.20.0/24 | .1 | .2 | .3 | CORE-2 |
| 30 Wireless | 10.10.30.0/23 | .1 | .2 | .3 | CORE-1 |
| 40 Printers | 10.10.40.0/26 | .1 | .2 | .3 | CORE-2 |
Config
CORE-1
spanning-tree mode rapid-pvst spanning-tree vlan 10,30 priority 4096 spanning-tree vlan 20,40 priority 8192 ! interface Vlan10 ip address 10.10.10.2 255.255.255.0 standby version 2 standby 10 ip 10.10.10.1 standby 10 priority 110 standby 10 preempt delay minimum 60 ! interface Vlan20 ip address 10.10.20.2 255.255.255.0 standby version 2 standby 20 ip 10.10.20.1 standby 20 priority 100 standby 20 preempt delay minimum 60 ! interface range TenGigabitEthernet1/1/1 - 2 channel-group 1 mode active interface Port-channel1 description CORE-2 switchport mode trunk switchport nonegotiate switchport trunk allowed vlan 10,20,30,40 ! interface range GigabitEthernet1/0/1 - 3 description Downlinks to access switchport mode trunk switchport nonegotiate switchport trunk allowed vlan 10,20,30,40 spanning-tree guard root
CORE-2
spanning-tree mode rapid-pvst spanning-tree vlan 20,40 priority 4096 spanning-tree vlan 10,30 priority 8192 ! interface Vlan10 ip address 10.10.10.3 255.255.255.0 standby version 2 standby 10 ip 10.10.10.1 standby 10 priority 100 standby 10 preempt delay minimum 60 ! interface Vlan20 ip address 10.10.20.3 255.255.255.0 standby version 2 standby 20 ip 10.10.20.1 standby 20 priority 110 standby 20 preempt delay minimum 60 ! port-channel and downlinks as CORE-1
Verify
show
show spanning-tree root show standby brief show etherchannel summary show interfaces trunk
Prove it before you sign it off
| Test | Expect |
|---|---|
| Compare the two outputs above | For each VLAN, the root bridge and the HSRP active router are the same switch. |
| Reload CORE-1 | CORE-2 becomes active for VLAN 10 and 30 when the 10 second hold time expires (default timers). Hosts keep the same gateway MAC. |
| CORE-1 returns | It takes back VLAN 10 and 30 after the 60 second preempt delay, not before. |
| Shut one Po1 member | Po1 stays up on the other link. No STP change. |
02 / Branch
Dual-WAN branch with a tracked default route
IOS XE, ISR and Catalyst 8000 routers
A branch router with a primary internet or WAN circuit and a cheaper backup. You want failover that notices when the provider is broken, not just when the cable is unplugged.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Probe something beyond the provider's router. Their router can answer while their network is down.
- Pin the probe address to ISP A with a host route, and add a Null0 host route behind it. Without the Null0 route the probe fails over to ISP B, succeeds, and the default route flaps back and forth.
- Damp the track with a delay: quick to fail (30 seconds), slow to return (60 seconds or more).
- The backup default route has a distance of 250 so it only appears when the tracked route is gone.
- NAT each provider with its own route-map matched on the outgoing interface, so traffic is translated to the address of the link it leaves on.
- Existing sessions break on failover because their NAT address changes. Tell the users, it is not a fault.
Addressing
| Item | Value | Note |
|---|---|---|
| ISP A link | 203.0.113.2/30 | next hop 203.0.113.1 |
| ISP B link | 198.51.100.2/30 | next hop 198.51.100.1 |
| Probe target | 192.0.2.53 | replace with a stable address you are allowed to ping |
| LAN | 10.20.0.0/24 | NAT inside |
Config
BRANCH-RTR
ip sla 10 icmp-echo 192.0.2.53 source-interface GigabitEthernet0/0/0 threshold 1000 timeout 1000 frequency 10 ip sla schedule 10 life forever start-time now ! track 10 ip sla 10 reachability delay down 30 up 60 ! ! pin the probe to ISP A, and drop it rather than reroute it ip route 192.0.2.53 255.255.255.255 203.0.113.1 ip route 192.0.2.53 255.255.255.255 Null0 250 ! ip route 0.0.0.0 0.0.0.0 203.0.113.1 track 10 ip route 0.0.0.0 0.0.0.0 198.51.100.1 250 ! ip access-list standard NAT-INSIDE permit 10.20.0.0 0.0.0.255 route-map NAT-ISPA permit 10 match ip address NAT-INSIDE match interface GigabitEthernet0/0/0 route-map NAT-ISPB permit 10 match ip address NAT-INSIDE match interface GigabitEthernet0/0/1 ! ip nat inside source route-map NAT-ISPA interface GigabitEthernet0/0/0 overload ip nat inside source route-map NAT-ISPB interface GigabitEthernet0/0/1 overload
Verify
show
show ip sla statistics 10 show track 10 show ip route 0.0.0.0 show ip nat translations | count .
Prove it before you sign it off
| Test | Expect |
|---|---|
| Normal running | track 10 is Up, the default route points at 203.0.113.1, SLA return code OK. |
| Block the probe upstream, or shut Gi0/0/0 | After about 30 seconds track 10 goes Down and the default route moves to 198.51.100.1. |
| ISP A recovers | The route returns 60 seconds after the probe succeeds again, not on the first reply. |
| Watch for flapping | show track 10 shows one change per event. Repeated changes mean the probe is escaping via ISP B. |
03 / Internet edge
Dual-homed internet edge with BGP to two providers
IOS XE, ISR and Catalyst 8000 routers
You hold your own address block and AS number and want to stay online when either provider fails. One provider is primary, the other is backup, and neither should ever be able to use you as a path to the other.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Send each provider your own prefix and nothing else: a prefix list plus an AS-path filter of ^$. Without both, one provider's routes leak to the other and you become their transit.
- Take only what you need from each provider, usually a default route, and set a maximum-prefix limit that restarts on its own.
- Originate the block with a network statement and a Null0 route at distance 254, so the static only holds the prefix in the table when nothing better exists.
- Outbound: local preference 200 on the primary provider's default. It is shared over iBGP, so both routers send internet traffic to ISP A.
- Inbound: prepend your AS three times on the backup. MED is no use between two providers, and never announce a more-specific on one link only, because the most specific route beats any prepend.
- Run iBGP between loopbacks with next-hop-self, and let OSPF carry the loopbacks, not the provider links.
- Put the default into OSPF without "always", with a better metric on EDGE-1, so it disappears when the BGP default does.
- Prepending moves most of the internet, not ISP B's own customers: providers prefer customer routes. For more control, use ISP B's documented communities to lower its local preference.
- A /24 is the smallest prefix most networks accept, so a single /24 cannot be split to share load. With a /23 or larger, announce the aggregate on both links and half on each.
Addressing
| Item | Value | Note |
|---|---|---|
| Our AS and prefix | AS 64500, 203.0.113.0/24 | documentation values, swap in your own |
| ISP A link | 192.0.2.0/30 | ISP A .1 (AS 64496), EDGE-1 .2 |
| ISP B link | 198.51.100.0/30 | ISP B .1 (AS 64511), EDGE-2 .2 |
| Loopbacks | 10.255.0.1, 10.255.0.2 | iBGP peering and router IDs |
| Inside transit | 10.0.0.0/29 | EDGE-1 .1, EDGE-2 .2, firewall .3, OSPF area 0 |
Config
EDGE-1
interface Loopback0 ip address 10.255.0.1 255.255.255.255 interface GigabitEthernet0/0/0 description ISP A ip address 192.0.2.2 255.255.255.252 interface GigabitEthernet0/0/1 description Inside transit ip address 10.0.0.1 255.255.255.248 ! router ospf 1 router-id 10.255.0.1 passive-interface default no passive-interface GigabitEthernet0/0/1 network 10.0.0.0 0.0.0.7 area 0 network 10.255.0.1 0.0.0.0 area 0 default-information originate metric 10 ! ip route 203.0.113.0 255.255.255.0 Null0 254 ip prefix-list OUR-PREFIXES seq 5 permit 203.0.113.0/24 ip prefix-list DEFAULT-ONLY seq 5 permit 0.0.0.0/0 ip as-path access-list 10 permit ^$ ! route-map ISP-A-IN permit 10 match ip address prefix-list DEFAULT-ONLY set local-preference 200 route-map ISP-A-OUT permit 10 match ip address prefix-list OUR-PREFIXES ! router bgp 64500 bgp router-id 10.255.0.1 bgp log-neighbor-changes neighbor 192.0.2.1 remote-as 64496 neighbor 192.0.2.1 description ISP A neighbor 192.0.2.1 password <agreed with ISP A> neighbor 10.255.0.2 remote-as 64500 neighbor 10.255.0.2 description EDGE-2 neighbor 10.255.0.2 update-source Loopback0 ! address-family ipv4 network 203.0.113.0 mask 255.255.255.0 neighbor 192.0.2.1 activate neighbor 192.0.2.1 route-map ISP-A-IN in neighbor 192.0.2.1 route-map ISP-A-OUT out neighbor 192.0.2.1 filter-list 10 out neighbor 192.0.2.1 maximum-prefix 100 80 restart 15 neighbor 10.255.0.2 activate neighbor 10.255.0.2 next-hop-self
EDGE-2
interface Loopback0 ip address 10.255.0.2 255.255.255.255 interface GigabitEthernet0/0/0 description ISP B ip address 198.51.100.2 255.255.255.252 interface GigabitEthernet0/0/1 description Inside transit ip address 10.0.0.2 255.255.255.248 ! router ospf 1 router-id 10.255.0.2 passive-interface default no passive-interface GigabitEthernet0/0/1 network 10.0.0.0 0.0.0.7 area 0 network 10.255.0.2 0.0.0.0 area 0 default-information originate metric 20 ! ip route 203.0.113.0 255.255.255.0 Null0 254 ip prefix-list OUR-PREFIXES seq 5 permit 203.0.113.0/24 ip prefix-list DEFAULT-ONLY seq 5 permit 0.0.0.0/0 ip as-path access-list 10 permit ^$ ! route-map ISP-B-IN permit 10 match ip address prefix-list DEFAULT-ONLY route-map ISP-B-OUT permit 10 match ip address prefix-list OUR-PREFIXES set as-path prepend 64500 64500 64500 ! router bgp 64500 bgp router-id 10.255.0.2 bgp log-neighbor-changes neighbor 198.51.100.1 remote-as 64511 neighbor 198.51.100.1 description ISP B neighbor 198.51.100.1 password <agreed with ISP B> neighbor 10.255.0.1 remote-as 64500 neighbor 10.255.0.1 description EDGE-1 neighbor 10.255.0.1 update-source Loopback0 ! address-family ipv4 network 203.0.113.0 mask 255.255.255.0 neighbor 198.51.100.1 activate neighbor 198.51.100.1 route-map ISP-B-IN in neighbor 198.51.100.1 route-map ISP-B-OUT out neighbor 198.51.100.1 filter-list 10 out neighbor 198.51.100.1 maximum-prefix 100 80 restart 15 neighbor 10.255.0.1 activate neighbor 10.255.0.1 next-hop-self
Verify
show
show ip bgp summary show ip bgp neighbors 192.0.2.1 advertised-routes show ip bgp neighbors 192.0.2.1 routes show ip bgp 0.0.0.0 show ip route 0.0.0.0 show ip ospf database external 0.0.0.0
Prove it before you sign it off
| Test | Expect |
|---|---|
| advertised-routes to each provider | Exactly one line: 203.0.113.0/24. Anything else is a leak. |
| show ip bgp 0.0.0.0 on EDGE-2 | Best path is via 10.255.0.1 with local preference 200; the ISP B default is a valid backup. |
| A looking glass or RIPEstat from outside | Your prefix via 64496 64500 as the usual path; via ISP B it shows 64500 four times. |
| Shut the ISP A session | Within the hold time both routers use ISP B for outbound, and inbound moves to ISP B as ISP A withdraws your prefix. |
| Shut both sessions in a lab | No default in OSPF at all, so the firewall knows the internet is gone rather than sending traffic into a black hole. |
Gotchas this design avoids
- Your dual-homed edge becomes a transit network
- iBGP routes are received but never used
- MED does nothing across two providers
- Local preference does not change inbound traffic
- The edge keeps advertising a default after the internet has gone
- A BGP network statement advertises nothing
- maximum-prefix tears the session down and leaves it down
04 / Data centre and campus
Firewall transit /29 per VRF
IOS XE core with an HA firewall pair
A core switch routes several security zones (for example corporate, guest and payment) and every zone must reach the others and the internet only through a firewall pair. Each zone is a VRF with its own small transit network to the firewall.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- One VRF per zone on the core, one transit VLAN and one /29 per VRF. Traffic between zones has no path except through the firewall.
- The same offsets in every /29: +1 core switch, +2 and +3 kept for a second core or HSRP later, +4 firewall virtual address, +5 FW-1, +6 FW-2. Anyone can find the firewall without a diagram.
- Each VRF has a default route to +4. The firewall routes the zone's subnets back to +1.
- Firewall-facing ports are trunks with negotiation off and only the transit VLANs allowed.
- Mark them as edge trunks. Firewalls do not run spanning tree, and a failover should not wait for STP to move a port to forwarding.
- Keep one numbering scheme for VLAN, VRF and subnet, so VLAN 902 is always the second zone and 10.255.1.8/29.
Addressing
| VRF | VLAN | Transit /29 | Core (+1) | FW VIP (+4) | FW-1, FW-2 |
|---|---|---|---|---|---|
| CORP | 901 | 10.255.1.0/29 | 10.255.1.1 | 10.255.1.4 | .5, .6 |
| GUEST | 902 | 10.255.1.8/29 | 10.255.1.9 | 10.255.1.12 | .13, .14 |
| PCI | 903 | 10.255.1.16/29 | 10.255.1.17 | 10.255.1.20 | .21, .22 |
Config
CORE
vrf definition CORP address-family ipv4 ! vlan 901 name TRANSIT-CORP ! interface Vlan901 description Transit to firewall, VRF CORP vrf forwarding CORP ip address 10.255.1.1 255.255.255.248 ! ip route vrf CORP 0.0.0.0 0.0.0.0 10.255.1.4 name FW-VIP ! ! users in VRF CORP interface Vlan10 vrf forwarding CORP ip address 10.10.10.1 255.255.255.0 ! ! repeat for GUEST (902, 10.255.1.8/29) and PCI (903, 10.255.1.16/29) ! interface range GigabitEthernet1/0/47 - 48 description FW-1 and FW-2 switchport mode trunk switchport nonegotiate switchport trunk allowed vlan 901-903 spanning-tree portfast edge trunk
Verify
show
show vrf show ip route vrf CORP 0.0.0.0 show ip arp vrf CORP 10.255.1.4 ping vrf CORP 10.255.1.4 source Vlan10
Prove it before you sign it off
| Test | Expect |
|---|---|
| ARP for the VIP | One MAC for 10.255.1.4, owned by the active firewall. |
| Fail the firewall over | The VIP moves to FW-2. The core ARP entry updates from gratuitous ARP and pings lose at most a few packets. |
| Traceroute from a GUEST host to a CORP host | The second hop is the firewall (10.255.1.12). The core has no direct route between zones. |
| show spanning-tree interface Gi1/0/47 detail | The port is in the portfast edge trunk mode, and forwarding. |
05 / Data centre
vPC pair with dual-attached access
NX-OS, Nexus 9000
Two Nexus switches act as one for the devices below them, so every server or access switch has an active-active port-channel with no blocked links.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- The peer keepalive never runs over the peer-link. Use mgmt0 in the management VRF or a dedicated routed link.
- Peer-link: at least two links on different modules, a trunk, spanning-tree port type network.
- peer-gateway, so hosts that reply to the physical MAC of the other switch are still routed locally.
- peer-switch with the same STP priority on both, so the pair appears as one root.
- auto-recovery and delay restore, so a single switch coming back on its own still brings its vPCs up, and routing converges before they forward.
- Use the same number for the vPC and the port-channel. Dual-attach everything, or orphan ports lose service when the peer-link fails.
Addressing
| Item | N9K-1 | N9K-2 | Note |
|---|---|---|---|
| mgmt0 (keepalive) | 192.168.0.1/24 | 192.168.0.2/24 | VRF management |
| Role priority | 10 | 20 | lower is primary |
| STP priority | 4096 | 4096 | same, with peer-switch |
| Vlan10 gateway | 10.30.10.2 | 10.30.10.3 | HSRP VIP .1, both forward |
Config
N9K-1
feature lacp feature vpc feature interface-vlan feature hsrp ! spanning-tree vlan 1-3967 priority 4096 ! vpc domain 10 role priority 10 peer-keepalive destination 192.168.0.2 source 192.168.0.1 vrf management peer-switch peer-gateway auto-recovery ip arp synchronize delay restore 150 ! interface port-channel1 description vPC peer-link switchport mode trunk spanning-tree port type network vpc peer-link interface Ethernet1/49-50 channel-group 1 mode active ! interface port-channel20 description ACCESS-1 switchport mode trunk switchport trunk allowed vlan 10,20 vpc 20 interface Ethernet1/1 channel-group 20 mode active
N9K-2
! as N9K-1, with these differences vpc domain 10 role priority 20 peer-keepalive destination 192.168.0.1 source 192.168.0.2 vrf management
Verify
show
show vpc brief show vpc consistency-parameters global show vpc orphan-ports show port-channel summary
Prove it before you sign it off
| Test | Expect |
|---|---|
| show vpc brief | Peer status: peer adjacency formed ok. Keepalive: peer is alive. Every vPC shows consistency success. |
| Shut one access uplink | Po20 stays up on the other switch. No STP topology change. |
| Fail the peer-link with the keepalive up | The secondary suspends its vPC ports and SVIs. Dual-attached devices carry on through the primary. |
| Reload N9K-1 | N9K-2 keeps forwarding and takes the primary role. vPCs come back on N9K-1 after delay restore. |
Gotchas this design avoids
- A peer-link failure cuts off single-attached devices
- Changing one vPC peer suspends the vPCs
- The vPC keepalive runs over the peer-link
- After a power cut only one Nexus comes back, and its vPCs stay down
- The vPC primary is not the switch you configured
- "no feature" deletes all of that feature's config
- Mode "on" at one end and LACP at the other
06 / Data centre
vPC pair with routed uplinks, a firewall pair and a backup path
NX-OS, Nexus 9000, with an ASA pair
The data centre core is a vPC pair. It routes to the LAN core over its own links, carries a firewall pair on vPCs, and keeps the protected VLANs reachable only through the firewall.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Uplinks to the LAN core are routed point-to-point links from each Nexus, not a vPC. Each switch runs its own OSPF adjacencies, and the core load-shares across all four links.
- A routed link between the two peers, with its own OSPF adjacency, so a switch that loses its uplinks routes through its partner instead of dropping the traffic hashed to it.
- The firewalls attach with a port-channel each, one link to each Nexus, as vPC trunks. The Nexus reaches the protected subnets with static routes to the firewall's outside address, and the firewalls point their default route at the HSRP address. No routing protocol runs over the vPC.
- Protected VLANs exist only at layer 2 on the Nexus, with no SVI, so the only way in or out is through the firewall.
- SVIs are passive in OSPF, HSRP keeps default timers, and there is no HSRP tracking. With vPC both switches forward for the gateway anyway.
- The data centre core and the LAN core meet at layer 3, which gives each its own change window and a clean line to move the data centre later.
Addressing
| Item | N9K-1 | N9K-2 | Note |
|---|---|---|---|
| Uplink to CORE-1 | 10.255.1.0/31 | 10.255.1.4/31 | Eth1/53, OSPF area 0 |
| Uplink to CORE-2 | 10.255.1.2/31 | 10.255.1.6/31 | Eth1/54, OSPF area 0 |
| L3 backup link | 10.255.0.0/31 | 10.255.0.1/31 | Eth1/48, peer to peer |
| Vlan10 SERVERS | 10.30.10.2 | 10.30.10.3 | HSRP VIP 10.30.10.1 |
| Vlan100 FW-OUTSIDE /29 | 10.30.100.2 | 10.30.100.3 | VIP .1, firewall .4 (standby .5) |
| VLAN 20 SECURE-APP | no SVI | no SVI | gateway 10.30.20.1 on the firewall |
Config
N9K-1
feature ospf
feature interface-vlan
feature hsrp
feature lacp
feature vpc
!
vlan 10
name SERVERS
vlan 20
name SECURE-APP
vlan 100
name FW-OUTSIDE
!
vpc domain 10
role priority 10
peer-keepalive destination 192.168.0.2 source 192.168.0.1 vrf management
peer-switch
peer-gateway
auto-recovery
ip arp synchronize
!
ip prefix-list FW-NETS seq 5 permit 10.30.20.0/24
route-map FW-NETS permit 10
match ip address prefix-list FW-NETS
ip route 10.30.20.0/24 10.30.100.4
!
router ospf 1
router-id 10.255.255.1
redistribute static route-map FW-NETS
!
interface Ethernet1/53
description To CORE-1
no switchport
ip address 10.255.1.0/31
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
no shutdown
interface Ethernet1/54
description To CORE-2
no switchport
ip address 10.255.1.2/31
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
no shutdown
interface Ethernet1/48
description L3 backup to N9K-2
no switchport
ip address 10.255.0.0/31
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
no shutdown
!
interface Vlan10
no shutdown
ip address 10.30.10.2/24
ip ospf passive-interface
ip router ospf 1 area 0.0.0.0
hsrp 10
priority 110
ip 10.30.10.1
interface Vlan100
no shutdown
ip address 10.30.100.2/29
ip ospf passive-interface
ip router ospf 1 area 0.0.0.0
hsrp 100
priority 110
ip 10.30.100.1
!
interface port-channel40
description FW-A
switchport mode trunk
switchport trunk allowed vlan 20,100
vpc 40
interface Ethernet1/10
channel-group 40 mode active
interface port-channel41
description FW-B
switchport mode trunk
switchport trunk allowed vlan 20,100
vpc 41
interface Ethernet1/11
channel-group 41 mode active
!
interface port-channel1
description vPC peer-link
switchport mode trunk
switchport trunk allowed vlan 10,20,100
spanning-tree port type network
vpc peer-linkN9K-2
! as N9K-1, with these differences vpc domain 10 role priority 20 peer-keepalive destination 192.168.0.1 source 192.168.0.2 vrf management router ospf 1 router-id 10.255.255.2 interface Ethernet1/53 ip address 10.255.1.4/31 interface Ethernet1/54 ip address 10.255.1.6/31 interface Ethernet1/48 ip address 10.255.0.1/31 interface Vlan10 ip address 10.30.10.3/24 interface Vlan100 ip address 10.30.100.3/29 ! HSRP priority stays at the default of 100
FW-A (ASA)
interface TenGigabitEthernet0/8 channel-group 1 mode active no shutdown interface TenGigabitEthernet0/9 channel-group 1 mode active no shutdown interface Port-channel1 no nameif no security-level no ip address interface Port-channel1.100 vlan 100 nameif outside security-level 0 ip address 10.30.100.4 255.255.255.248 standby 10.30.100.5 interface Port-channel1.20 vlan 20 nameif inside security-level 100 ip address 10.30.20.1 255.255.255.0 standby 10.30.20.2 route outside 0.0.0.0 0.0.0.0 10.30.100.1 1 ! failover and state links not shown
Verify
show
show vpc brief show ip ospf neighbors show ip route 10.30.20.0/24 show hsrp brief show port-channel summary
Prove it before you sign it off
| Test | Expect |
|---|---|
| show ip ospf neighbors on N9K-1 | FULL to both core switches and to N9K-2 over Eth1/48. No neighbours on any VLAN interface. |
| Shut both uplinks on N9K-1 | Servers keep working. Traffic hashed to N9K-1 is routed over Eth1/48 to N9K-2. A traceroute shows the extra hop. |
| Fail FW-A | FW-B becomes active and takes 10.30.100.4 and 10.30.20.1. The static routes on the Nexus do not change. |
| traceroute from a VLAN 20 host to the LAN core | The first hop is 10.30.20.1, the firewall. There is no path that avoids it. |
| Fail the peer-link with the keepalive up | N9K-2 suspends its vPCs and SVIs. Servers and both firewalls carry on through N9K-1. |
07 / Data centre
Double-sided vPC between an aggregation pair and an access pair
NX-OS, Nexus 9000
Two vPC pairs stacked on top of each other, joined by a single port-channel of four links. Every link forwards, and spanning tree has nothing to block.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- All four links between the pairs are one port-channel, configured as a vPC on both pairs. Build it as two port-channels and spanning tree sees a loop and blocks one.
- Different domain IDs on the two pairs. The domain ID becomes the LACP system ID, and two pairs with the same one make the bundle flap.
- The aggregation pair is the spanning-tree root for every VLAN, with peer-switch and the same priority on both. Root guard on its vPC towards the access pair keeps it that way.
- The access pair is layer 2 only and keeps default spanning-tree priorities. Gateways live on the aggregation pair.
- Each pair has its own peer-link and its own keepalive path.
- Spanning-tree port type network on the peer-links only. The vPC between the pairs is normal, and host ports are edge.
Addressing
| Item | AGG pair | ACC pair | Note |
|---|---|---|---|
| vPC domain | 10 | 20 | must differ |
| vPC system MAC | 00:23:04:ee:be:0a | 00:23:04:ee:be:14 | built from the domain ID |
| Keepalive (mgmt0) | 192.168.0.1 and .2 | 192.168.1.1 and .2 | VRF management |
| STP priority | 4096 on both | default | root on aggregation |
| Interconnect | Po100, vPC 100 | Po100, vPC 100 | same number both sides, for sanity |
Config
AGG-1
feature lacp feature vpc ! spanning-tree vlan 1-3967 priority 4096 ! vpc domain 10 role priority 10 peer-keepalive destination 192.168.0.2 source 192.168.0.1 vrf management peer-switch peer-gateway auto-recovery ip arp synchronize ! peer-link Po1 as in the vPC pair design ! interface port-channel100 description To access pair ACC-1 and ACC-2 switchport mode trunk switchport trunk allowed vlan 10,20 spanning-tree port type normal spanning-tree guard root vpc 100 interface Ethernet1/1 description To ACC-1 channel-group 100 mode active interface Ethernet1/2 description To ACC-2 channel-group 100 mode active
AGG-2
! as AGG-1, with these differences vpc domain 10 role priority 20 peer-keepalive destination 192.168.0.1 source 192.168.0.2 vrf management
ACC-1
feature lacp feature vpc ! vpc domain 20 role priority 10 peer-keepalive destination 192.168.1.2 source 192.168.1.1 vrf management auto-recovery ! peer-link Po1 as in the vPC pair design ! interface port-channel100 description To aggregation pair AGG-1 and AGG-2 switchport mode trunk switchport trunk allowed vlan 10,20 spanning-tree port type normal vpc 100 interface Ethernet1/49 description To AGG-1 channel-group 100 mode active interface Ethernet1/50 description To AGG-2 channel-group 100 mode active ! interface Ethernet1/1 description Server switchport access vlan 10 spanning-tree port type edge
ACC-2
! as ACC-1, with these differences vpc domain 20 role priority 20 peer-keepalive destination 192.168.1.1 source 192.168.1.2 vrf management
Verify
show
show vpc brief show vpc role show port-channel summary show lacp neighbor interface port-channel100 show spanning-tree vlan 10
Prove it before you sign it off
| Test | Expect |
|---|---|
| show port-channel summary on ACC-1 | Po100 (SU) with both members (P). |
| show vpc role on both pairs | Two different vPC system MACs, ending 0a and 14. |
| show spanning-tree vlan 10 on ACC-1 | Po100 is the root port and forwarding. Nothing is blocked. |
| Reload AGG-1 | The access pair keeps forwarding through AGG-2. Po100 loses two members and stays up. |
| Fail ACC-1 | Servers dual-attached to the access pair carry on through ACC-2. |
08 / Management
Out-of-band management in Mgmt-vrf
IOS XE, Catalyst 9000
You want to reach and manage every switch even when the production network is broken, and keep management traffic away from user VLANs.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Gi0/0 on Catalyst 9000 is fixed in Mgmt-vrf. It cannot carry user traffic or join a port-channel, so it is ideal for management and useless for anything else.
- Mgmt-vrf has its own default route. The global default route does not apply to it.
- Every service needs its VRF or source interface set: TACACS+, syslog, NTP, SNMP and file copies all default to the global table.
- The VTY access-class needs vrf-also, or SSH arriving through Mgmt-vrf is refused.
- Add a console server on the same out-of-band network for the day the switch will not boot.
- Put a firewall between the OOB network and everything else. It is the keys to the building.
Addressing
| Item | Value | Note |
|---|---|---|
| OOB subnet | 10.250.1.0/24 | gateway .1 on the management firewall |
| Switch Gi0/0 | 10.250.1.11 | one address per switch |
| Services | 10.250.0.0/24 | TACACS+ .5, syslog .20, NTP .10, SNMP .30 |
Config
CAT9K
interface GigabitEthernet0/0 vrf forwarding Mgmt-vrf ip address 10.250.1.11 255.255.255.0 no shutdown ! ip route vrf Mgmt-vrf 0.0.0.0 0.0.0.0 10.250.1.1 ! ip access-list standard MGMT-ONLY permit 10.250.0.0 0.0.255.255 line vty 0 15 access-class MGMT-ONLY in vrf-also transport input ssh ! tacacs server TAC1 address ipv4 10.250.0.5 key <key> aaa group server tacacs+ TACACS server name TAC1 ip vrf forwarding Mgmt-vrf ip tacacs source-interface GigabitEthernet0/0 ! logging host 10.250.0.20 vrf Mgmt-vrf logging source-interface GigabitEthernet0/0 vrf Mgmt-vrf ntp server vrf Mgmt-vrf 10.250.0.10 snmp-server host 10.250.0.30 vrf Mgmt-vrf version 3 priv NMS ip tftp source-interface GigabitEthernet0/0 ip ssh source-interface GigabitEthernet0/0
Verify
show
show ip route vrf Mgmt-vrf ping vrf Mgmt-vrf 10.250.0.5 show tacacs show ntp associations show logging | include Logging to
Prove it before you sign it off
| Test | Expect |
|---|---|
| SSH to 10.250.1.11 from the management network | Login works with a TACACS+ account. |
| Shut all production uplinks in a lab | SSH over Gi0/0 still works. That is the whole point. |
| test aaa group TACACS <user> <password> legacy | User successfully authenticated. |
| show logging | Shows logging to 10.250.0.20 (vrf Mgmt-vrf) with messages counted. |
09 / Campus
Campus with one logical distribution switch and stacked access
IOS XE, current Catalyst access stacks and a distribution pair that supports a virtual switch link
A building, or a few, with layer 2 to the closets and a wireless controller pair in the middle. Instead of two distribution switches with HSRP and spanning tree deciding which uplinks to block, the pair runs as one logical switch, so every uplink forwards and there is no first-hop protocol to keep in step.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Pick the shape by size. One closet: a single stack is the whole network. One building: access plus a distribution pair acting as a collapsed core. Three or more distribution blocks: add a core, because a full mesh needs N x (N-1) / 2 links and a core needs 2N.
- Run the distribution pair as one logical switch (a virtual switch link, or a stack). The gateway is one SVI, so there is no HSRP or VRRP, and failover is the port-channel losing a member, not a protocol timing out.
- Each access stack connects with one LACP port-channel, members on both distribution chassis. Never mode on.
- Keep each VLAN inside one closet where you can. Stretch a VLAN only for a real application need, and know you are doing it.
- In the access stack, make two members without uplinks the active and standby, and put the uplinks on two others, so losing the active never takes uplinks with it.
- Work out the uplink ratio twice: normal running and the worst single failure. Spread access points across stack members, since each can now push several gigabits.
- Above about four 10G uplinks per stack, move to 25G rather than add links: a port-channel tops out at eight members, and fibre pairs between rooms are harder to add than optics.
- Routed links from distribution to core, with the access subnets summarised at the distribution layer, so a flapping closet does not ripple across the campus.
- Wireless clients terminate on the controller pair in the middle (centralised mode) when the site has a distribution layer or more than about 100 access points. Each controller gets one port-channel with members on both chassis.
- Queue on the uplinks. At 7:1 or 20:1 there will be bursts, and voice should not be the traffic that drops.
Addressing
| Item | Value | Note |
|---|---|---|
| Closet 1A | VLAN 110 data, 210 voice | 10.1.110.0/24 and 10.1.210.0/24 |
| Closet 1B | VLAN 120 data, 220 voice | 10.1.120.0/24 and 10.1.220.0/24 |
| Wireless clients | VLAN 300 | 10.1.48.0/22, only on the WLC trunks |
| Infrastructure | VLAN 900 | 10.1.0.0/24, switch and WLC management |
| Distribution to core | 10.255.1.0/31, 10.255.1.2/31 | routed port-channel members, OSPF area 1 |
| Summary | 10.1.0.0/16 | advertised by distribution into area 0 |
Config
DIST (both chassis, before reload)
! Run on each chassis, then save and reload. ! Check the platform's StackWise Virtual guide for supported ports. stackwise-virtual domain 10 interface FortyGigabitEthernet1/0/1 stackwise-virtual link 1 interface FortyGigabitEthernet1/0/2 stackwise-virtual link 1 interface TenGigabitEthernet1/0/16 stackwise-virtual dual-active-detection
DIST (one logical switch)
ip routing vlan 110,120,210,220,300,900 ! interface Vlan110 ip address 10.1.110.1 255.255.255.0 ip helper-address 10.250.0.10 interface Vlan120 ip address 10.1.120.1 255.255.255.0 ip helper-address 10.250.0.10 ! ! Access stack 1A: one member on each chassis interface range TenGigabitEthernet1/0/11, TenGigabitEthernet2/0/11 description ACCESS-1A uplinks channel-group 11 mode active interface Port-channel11 switchport mode trunk switchport trunk native vlan 999 switchport trunk allowed vlan 110,210,900 ! ! WLC-1: one port-channel across both chassis interface range TenGigabitEthernet1/0/20, TenGigabitEthernet2/0/20 description WLC-1 LAG channel-group 20 mode active interface Port-channel20 switchport mode trunk switchport trunk allowed vlan 300,900 ! ! Routed to the core, summarise the campus interface range TenGigabitEthernet1/0/48, TenGigabitEthernet2/0/48 no switchport channel-group 48 mode active interface Port-channel48 no switchport ip address 10.255.1.0 255.255.255.254 router ospf 1 router-id 10.255.255.1 area 1 range 10.1.0.0 255.255.0.0 network 10.1.0.0 0.0.255.255 area 1 network 10.255.1.0 0.0.0.1 area 0 passive-interface default no passive-interface Port-channel48
ACCESS-1A stack
switch 1 priority 15 switch 2 priority 14 vlan 110,210,900 spanning-tree mode rapid-pvst ! ! Uplinks on members 3 and 4, never on active or standby interface range TenGigabitEthernet3/1/1, TenGigabitEthernet4/1/1 description Uplink to DIST channel-group 11 mode active interface Port-channel11 switchport mode trunk switchport trunk native vlan 999 switchport trunk allowed vlan 110,210,900 ! interface range GigabitEthernet1/0/1 - 48 switchport mode access switchport access vlan 110 switchport voice vlan 210 spanning-tree portfast spanning-tree bpduguard enable ! interface Vlan900 ip address 10.1.0.11 255.255.255.0 ip default-gateway 10.1.0.1
Verify
show
show stackwise-virtual show stackwise-virtual dual-active-detection show switch show redundancy show etherchannel summary show spanning-tree blockedports show ip ospf neighbor show ip route 10.1.0.0 255.255.0.0 longer-prefixes
Prove it before you sign it off
| Test | Expect |
|---|---|
| Normal running | Every port-channel member is (P) and show spanning-tree blockedports lists nothing. |
| Reload DIST chassis 2 | Under a second of loss. Every access and WLC port-channel stays up on its chassis 1 member. |
| Reload the active switch in ACCESS-1A | The standby takes over and Port-channel11 keeps both members, because neither uplink was on the failed switch. |
| Shut one uplink of ACCESS-1A | Traffic continues on the other member. Recalculate the ratio with half the bandwidth and check voice queue drops. |
| Reload a distribution chassis with a call running over Wi-Fi | The call survives and the controllers do not swap roles. |
Gotchas this design avoids
- Stack uplinks on the active switch
- Uplink ratio checked only with everything working
- Mode "on" at one end and LACP at the other
- The standby controller and the active controller on one switch each
- Moving uplinks from 10G to 25G on the same multimode
- Sub-second HSRP timers on a chassis with stateful switchover
- Spanning tree root on one switch, HSRP active on the other
10 / WAN
Broadband sites on a dual-hub DMVPN
IOS XE, ASR 1000 or Catalyst 8000 hubs, ISR spokes
Sites with nothing but an ordinary broadband line, and perhaps a 4G backup, that still need private, encrypted access to the data centre and must survive a whole data centre going dark. Every site builds tunnels to a hub in each of two data centres. Only the hubs need fixed addresses, and branches build tunnels to each other only when they have traffic to send.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Two hubs, one in each data centre, on the same mGRE cloud. Every spoke registers with both and keeps both IPsec sessions up all the time, so losing a hub costs a routing reconvergence, not a tunnel build.
- Front-door VRF on the hubs: the internet interface and the tunnel transport live in VRF INET, so the internet default never meets customer routes. Keyrings must be defined in that VRF.
- One tunnel, one public loopback and one keyring per customer VRF on each hub. Give them one number: Loopback10, Tunnel10, tunnel key 10, NHRP network ID 10.
- Phase 3: the hubs send spokes nothing but a default summary and set ip nhrp redirect. Spokes set ip nhrp shortcut and build direct tunnels to each other on demand. Spokes are EIGRP stubs.
- The default summary installs a Null0 route on the hub with distance 5. Raise it above the hub's real default (201 beats iBGP's 200).
- Spoke routes enter the core through a prefix list, so a spoke can never inject a default or another customer's range.
- ip mtu 1400 and MSS 1360 on every tunnel. Spokes on dynamic addresses add registration no-unique and a 600-second hold time.
- 4G backup is a second cloud with its own tunnel, key and hub loopbacks. Raise delay on the 4G tunnel at both ends (a tunnel's default is 5000) so it loses in both directions until broadband fails. If both clouds leave by the same interface, mark the IPsec profile shared.
- If one hub must be primary, make the backup hub advertise a worse metric (an outbound offset list on its tunnel) and set the return path to match, so traffic never leaves through one data centre and returns through the other.
- DH group 14 or better, one pre-shared key per customer, and nothing in the config that another customer's key could unlock.
Addressing
| Item | Value | Note |
|---|---|---|
| Broadband cloud | Tunnel10, 172.16.10.0/24 | HUB-1 .1, HUB-2 .2, spokes .11 upwards |
| 4G cloud | Tunnel20, 172.16.20.0/24 | HUB-1 .1, HUB-2 .2, spokes .11 upwards |
| Hub public loopbacks | 198.51.100.10 and .11 (HUB-1), .20 and .21 (HUB-2) | VRF INET, one per customer per cloud |
| Hub internet links | 203.0.113.0/30 (HUB-1), 203.0.113.4/30 (HUB-2) | VRF INET, default to the provider |
| Spoke LANs | 10.1.N.0/24 | N is the site number |
| Routing | EIGRP AS 100 on the tunnels, BGP AS 65000 in the core | redistributed through a prefix list |
Config
HUB-1
vrf definition INET address-family ipv4 ! vrf definition CUST-A rd 65000:10 address-family ipv4 route-target both 65000:10 ! crypto keyring CUST-A-DMVPN vrf INET local-address Loopback10 pre-shared-key address 0.0.0.0 0.0.0.0 key <CUST-A key> ! crypto isakmp policy 10 encryption aes 256 hash sha256 authentication pre-share group 14 ! crypto ipsec transform-set TS-DMVPN esp-aes 256 esp-sha256-hmac mode transport crypto ipsec profile CUST-A-DMVPN set transform-set TS-DMVPN set pfs group14 ! interface GigabitEthernet0/0/0 description Internet vrf forwarding INET ip address 203.0.113.2 255.255.255.252 ! interface Loopback10 description CUST-A DMVPN broadband, public vrf forwarding INET ip address 198.51.100.10 255.255.255.255 ! interface Tunnel10 description CUST-A DMVPN broadband vrf forwarding CUST-A ip address 172.16.10.1 255.255.255.0 no ip redirects ip mtu 1400 ip tcp adjust-mss 1360 ip nhrp authentication <8 chars> ip nhrp network-id 10 ip nhrp map multicast dynamic ip nhrp redirect ip summary-address eigrp 100 0.0.0.0 0.0.0.0 tunnel source Loopback10 tunnel mode gre multipoint tunnel key 10 tunnel vrf INET tunnel protection ipsec profile CUST-A-DMVPN ! ! Loopback11 and Tunnel20 (4G cloud) are built the same way: ! 198.51.100.11, 172.16.20.1, key 20, network-id 20, ! plus delay 7000 so the core's path back to a spoke also prefers broadband ! ip route vrf INET 0.0.0.0 0.0.0.0 203.0.113.1 ! router eigrp 100 address-family ipv4 vrf CUST-A autonomous-system 100 summary-metric 0.0.0.0/0 distance 201 network 172.16.10.0 0.0.0.255 network 172.16.20.0 0.0.0.255 passive-interface default no passive-interface Tunnel10 no passive-interface Tunnel20 exit-address-family ! ip prefix-list PL-CUST-A-SITES seq 10 permit 10.1.0.0/16 ge 24 le 24 route-map RM-CUST-A-EIGRP-TO-BGP permit 10 match ip address prefix-list PL-CUST-A-SITES ! router bgp 65000 address-family ipv4 vrf CUST-A redistribute eigrp 100 route-map RM-CUST-A-EIGRP-TO-BGP exit-address-family
HUB-2
! As HUB-1, with these values: interface GigabitEthernet0/0/0 vrf forwarding INET ip address 203.0.113.6 255.255.255.252 ! interface Loopback10 vrf forwarding INET ip address 198.51.100.20 255.255.255.255 ! interface Tunnel10 ip address 172.16.10.2 255.255.255.0 ! interface Loopback11 vrf forwarding INET ip address 198.51.100.21 255.255.255.255 ! interface Tunnel20 ip address 172.16.20.2 255.255.255.0 ! ip route vrf INET 0.0.0.0 0.0.0.0 203.0.113.5
SPOKE-3
crypto keyring CUST-A-DMVPN pre-shared-key address 0.0.0.0 0.0.0.0 key <CUST-A key> ! crypto isakmp policy 10 encryption aes 256 hash sha256 authentication pre-share group 14 ! crypto ipsec transform-set TS-DMVPN esp-aes 256 esp-sha256-hmac mode transport crypto ipsec profile CUST-A-DMVPN set transform-set TS-DMVPN set pfs group14 ! interface Tunnel10 description DMVPN broadband cloud ip address 172.16.10.13 255.255.255.0 no ip redirects ip mtu 1400 ip tcp adjust-mss 1360 ip nhrp authentication <8 chars> ip nhrp network-id 10 ip nhrp holdtime 600 ip nhrp nhs 172.16.10.1 nbma 198.51.100.10 multicast ip nhrp nhs 172.16.10.2 nbma 198.51.100.20 multicast ip nhrp registration no-unique ip nhrp shortcut tunnel source Dialer1 tunnel mode gre multipoint tunnel key 10 tunnel protection ipsec profile CUST-A-DMVPN ! interface Tunnel20 description DMVPN 4G cloud, backup ip address 172.16.20.13 255.255.255.0 no ip redirects ip mtu 1400 ip tcp adjust-mss 1360 delay 7000 ip nhrp authentication <8 chars> ip nhrp network-id 20 ip nhrp holdtime 600 ip nhrp nhs 172.16.20.1 nbma 198.51.100.11 multicast ip nhrp nhs 172.16.20.2 nbma 198.51.100.21 multicast ip nhrp registration no-unique ip nhrp shortcut tunnel source Cellular0/2/0 tunnel mode gre multipoint tunnel key 20 tunnel protection ipsec profile CUST-A-DMVPN ! interface GigabitEthernet0/1/0 description LAN ip address 10.1.3.1 255.255.255.0 ! router eigrp 100 network 10.1.3.0 0.0.0.255 network 172.16.10.0 0.0.0.255 network 172.16.20.0 0.0.0.255 passive-interface default no passive-interface Tunnel10 no passive-interface Tunnel20 eigrp stub connected
Verify
show
show dmvpn show crypto isakmp sa show crypto ipsec sa | include peer|encaps|decaps show ip eigrp neighbors show ip route 0.0.0.0 show ip route nhrp show ip nhrp nhs detail
Prove it before you sign it off
| Test | Expect |
|---|---|
| show dmvpn on a spoke | Both hubs on Tunnel10 (and both on Tunnel20 if 4G is fitted) in state UP, with the hubs' loopback addresses as the NBMA addresses. |
| show ip route 0.0.0.0 on a spoke | The default via the hubs on Tunnel10 only. The Tunnel20 paths stay in the topology table as backups. |
| traceroute from SPOKE-1's LAN to SPOKE-2's LAN, twice | The first goes through a hub. The second is one hop, and show ip route nhrp lists SPOKE-2's LAN. |
| Shut HUB-1's internet interface | Spokes lose one EIGRP neighbour and keep working through HUB-2 within the hold time. Nothing needs to rebuild. |
| Unplug a spoke's broadband | Its routes move to Tunnel20. The core still reaches its LAN, through the 4G cloud. |
| show ip route vrf CUST-A 0.0.0.0 on a hub | The real default from the core, not the summary's Null0 route. |
Gotchas this design avoids
- IPsec never comes up on a hub with a front-door VRF
- The DMVPN hub config has no multicast map, and works anyway
- Branch-to-branch traffic still goes through the hub
- An EIGRP default summary blackholes the hub's own default route
- A broadband spoke drops off the DMVPN after its public address changes
- Two DMVPN clouds from one WAN interface, and one of them breaks
- A DMVPN spoke encrypts fine but never registers
- IPsec tunnels still negotiating DH group 5
- Pings cross the GRE tunnel, file transfers stall
11 / WAN
Wholesale broadband straight into customer VRFs
IOS XE, ASR 1000 LNS pair, FreeRADIUS pair
Sites on business broadband bought from a wholesale provider that hands the PPP sessions to you over L2TP instead of to the internet. Each site logs in, RADIUS decides which customer it belongs to, and the session lands straight in that customer's VRF. The site router needs no tunnels and is never on the internet.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Two LNSs in two data centres. The provider's LACs keep L2TP tunnels to both, so if an LNS or a whole data centre fails, sites redial and land on the other.
- RADIUS decides everything about a session: it checks the CHAP login, gives the site its fixed address (Framed-IP-Address), and puts the session in the customer's VRF. One profile per customer VRF, one user per site.
- In the profile, the VRF attribute comes first and the unnumbered loopback second. Moving an interface into a VRF wipes its IP settings.
- Every customer VRF has a loopback on both LNSs with the same address, from 100.64.0.0/10. Sessions are unnumbered to it, so every site router is configured identically.
- Sites that need more than a default run BGP to that loopback, all with the same private AS number. Filtering lives on the LNS, not the site: a prefix list of the customer's ranges in, default only out, a maximum-prefix limit, one peer group per customer.
- Two RADIUS servers on separate hosts, with database replication in both directions so a site added on one exists on both. Each keeps its own log.
- Usernames carry a realm per customer, which the wholesale provider uses to route the session to you. Site passwords are generated and kept in the password vault, never in the design document.
- Remove Framed-Compression from FreeRADIUS's default PPP reply before the first session. Virtual-access sub-interfaces cannot apply it.
- Maintenance by draining: take one RADIUS server out of the group, or shut one LNS's link to the provider so sessions move to the other, before working on it.
Addressing
| Item | Value | Note |
|---|---|---|
| LNS loopback, customer A | Loopback3510, 100.64.1.254/32 | same on both LNSs, and the same address in every customer VRF |
| Site WAN addresses | 100.64.20.0/22 | one /32 per site, given by RADIUS |
| Site LANs, customer A | 10.20.N.0/24 | N is the site number |
| BGP | sites AS 65100, LNS AS 65000 | the same AS at every site |
| RADIUS servers | 10.0.10.11 (DC1), 10.0.20.11 (DC2) | UDP 1812 and 1813 from the LNSs only |
| Username | site-0042@cust-a.example.net | realm per customer |
Config
LNS-1
aaa new-model aaa group server radius PPP-RADIUS server name RADIUS-1 server name RADIUS-2 aaa authentication ppp PPP-AUTHEN group PPP-RADIUS aaa authorization network PPP-AUTHOR group PPP-RADIUS aaa accounting network PPP-ACCT start-stop group PPP-RADIUS ! radius server RADIUS-1 address ipv4 10.0.10.11 auth-port 1812 acct-port 1813 key <RADIUS key> radius server RADIUS-2 address ipv4 10.0.20.11 auth-port 1812 acct-port 1813 key <RADIUS key> ! vpdn enable vpdn-group WHOLESALE accept-dialin protocol l2tp virtual-template 1 terminate-from hostname <provider LAC name> local name lns-1 l2tp tunnel password <tunnel password> ! interface Virtual-Template1 description Wholesale broadband sessions no ip address no peer default ip address ip tcp adjust-mss 1452 keepalive 20 3 ppp authentication chap PPP-AUTHEN ppp authorization PPP-AUTHOR ppp accounting PPP-ACCT ! vrf definition CUST-A rd 65000:3510 address-family ipv4 route-target both 65000:3510 ! interface Loopback3510 description CUST-A PPP unnumbered and BGP vrf forwarding CUST-A ip address 100.64.1.254 255.255.255.255 ! ip prefix-list PL-CUST-A-SITES seq 10 permit 10.20.0.0/16 ge 24 le 24 ip prefix-list PL-DEFAULT-ONLY seq 10 permit 0.0.0.0/0 ! router bgp 65000 address-family ipv4 vrf CUST-A redistribute connected neighbor PG-CUST-A peer-group neighbor PG-CUST-A remote-as 65100 neighbor PG-CUST-A update-source Loopback3510 neighbor PG-CUST-A prefix-list PL-CUST-A-SITES in neighbor PG-CUST-A prefix-list PL-DEFAULT-ONLY out neighbor PG-CUST-A maximum-prefix 10 neighbor 100.64.20.42 peer-group PG-CUST-A neighbor 100.64.20.42 description site-0042 neighbor 100.64.20.42 activate exit-address-family
RADIUS profile
# Group (profile) CUST-A: sent with every site in customer A
# Order matters: the VRF first, then the unnumbered loopback
Cisco-AVPair := "ip:vrf-id=CUST-A"
Cisco-AVPair += "ip:ip-unnumbered=Loopback3510"
Service-Type = Framed-User
Framed-Protocol = PPP
Framed-IP-Netmask = 255.255.255.255
#
# User site-0042@cust-a.example.net
# check: Cleartext-Password := <generated, stored in the vault>
# reply: Framed-IP-Address = 100.64.20.42
# group: CUST-A
#
# /etc/freeradius/3.0/mods-config/files/authorize
DEFAULT Framed-Protocol == PPP
Framed-Protocol = PPP
# Framed-Compression = Van-Jacobson-TCP-IPSITE-0042
interface GigabitEthernet0/0/0.101 description FTTC, VLAN 101 to the modem encapsulation dot1Q 101 pppoe enable group global pppoe-client dial-pool-number 1 ! interface Dialer1 ip address negotiated mtu 1492 ip tcp adjust-mss 1452 encapsulation ppp dialer pool 1 ppp chap hostname site-0042@cust-a.example.net ppp chap password <generated> ! interface GigabitEthernet0/1/0 description LAN ip address 10.20.42.1 255.255.255.0 ! router bgp 65100 bgp router-id 10.20.42.1 bgp log-neighbor-changes neighbor 100.64.1.254 remote-as 65000 address-family ipv4 redistribute connected neighbor 100.64.1.254 activate exit-address-family
Verify
show
show vpdn tunnel show vpdn session show derived-config interface virtual-access 3.1 show ip route vrf CUST-A show bgp vpnv4 unicast vrf CUST-A summary show aaa servers sudo freeradius -X (on the RADIUS server, service stopped)
Prove it before you sign it off
| Test | Expect |
|---|---|
| Connect one site | show vpdn session lists it, and its /32 appears in show ip route vrf CUST-A as connected. |
| show derived-config on its virtual-access | vrf forwarding CUST-A and ip unnumbered Loopback3510, and nothing you did not ask RADIUS for. |
| Remove RADIUS-1 from the group and reconnect a test site | It authenticates through RADIUS-2. Put RADIUS-1 back. |
| Add a test user on RADIUS-2 | It appears on RADIUS-1 within seconds. |
| Shut LNS-1's link to the provider | Sessions redial onto LNS-2 within the provider's retry time, and BGP sites come back with the same routes. |
| Power off one RADIUS server, then power it on | Logins work again without anyone touching it. |
Gotchas this design avoids
- PPP sessions land in the global table, or drop, when RADIUS sets the VRF
- FreeRADIUS's default PPP reply breaks sessions on virtual-access sub-interfaces
- After a power cut, the RADIUS server is up but nobody can log in
- Two RADIUS servers share one database, but not one log
- Provider addresses inside a customer VRF clash with the customer's own
- maximum-prefix tears the session down and leaves it down
12 / Campus
Routed access: layer 3 to every access switch
IOS XE, Catalyst 9000
A campus that wants recovery in under a second without spanning tree or HSRP on the uplinks. Each access switch routes for its own VLANs. The price is that a VLAN can never span two access switches.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Every access uplink is a routed point-to-point /31 taken from the block. No trunks, no HSRP and no spanning tree on any uplink.
- Each access switch (or stack) is the only gateway for its own VLANs. A VLAN never spans two access switches.
- Each distribution block is a totally stubby OSPF area. The distribution switches summarise the whole block into area 0 with one area range.
- A routed link between the two distribution switches, inside the block's area, so a summary never attracts traffic the switch cannot deliver.
- OSPF network type point-to-point on every uplink: no DR election and faster adjacency.
- BFD on the uplinks. Cisco's tests took convergence from 5.68 seconds at defaults to 0.24 seconds with SPF and LSA throttles of 10, 100 and 5000 ms. Only tune timers that far once summarisation is in place.
- Management on a /32 loopback inside the block summary, not on a management VLAN.
- Multicast: PIM sparse mode everywhere, anycast RP on the core pair with MSDP between them, IGMP snooping at the access.
Addressing
| Use | Range | Example |
|---|---|---|
| Block summary | 10.10.0.0/16 | advertised into area 0 by both distribution switches |
| Access VLANs | 10.10.1.0 to 10.10.199.0, /24 each | ACC-1 VLAN 10: 10.10.1.0/24, gateway .1 |
| Loopbacks | 10.10.254.0/24 as /32 | DIST-1 .1, DIST-2 .2, ACC-1 .11 |
| Access uplinks | 10.10.255.0/24 as /31 | DIST-1 to ACC-1: .0 and .1; DIST-2 to ACC-1: .2 and .3 |
| Distribution link | 10.10.255.100/31 | DIST-1 .100, DIST-2 .101, area 10 |
| Core links | 10.0.255.0/24 as /31 | area 0 |
Config
DIST-1
ip routing interface Loopback0 ip address 10.10.254.1 255.255.255.255 ip ospf 1 area 10 ! interface TenGigabitEthernet1/0/1 description ACC-1 no switchport ip address 10.10.255.0 255.255.255.254 ip ospf network point-to-point ip ospf 1 area 10 bfd interval 250 min_rx 250 multiplier 3 ! interface TenGigabitEthernet1/1/8 description DIST-2, inside area 10 no switchport ip address 10.10.255.100 255.255.255.254 ip ospf network point-to-point ip ospf 1 area 10 bfd interval 250 min_rx 250 multiplier 3 ! interface TenGigabitEthernet1/1/1 description CORE, area 0 no switchport ip address 10.0.255.0 255.255.255.254 ip ospf network point-to-point ip ospf 1 area 0 bfd interval 250 min_rx 250 multiplier 3 ! router ospf 1 router-id 10.10.254.1 area 10 stub no-summary area 10 range 10.10.0.0 255.255.0.0 timers throttle spf 10 100 5000 timers throttle lsa 10 100 5000 timers lsa arrival 80 passive-interface default no passive-interface TenGigabitEthernet1/0/1 no passive-interface TenGigabitEthernet1/1/1 no passive-interface TenGigabitEthernet1/1/8 bfd all-interfaces
ACC-1
ip routing interface Loopback0 ip address 10.10.254.11 255.255.255.255 ip ospf 1 area 10 ! interface TenGigabitEthernet1/1/1 description DIST-1 no switchport ip address 10.10.255.1 255.255.255.254 ip ospf network point-to-point ip ospf 1 area 10 bfd interval 250 min_rx 250 multiplier 3 interface TenGigabitEthernet1/1/2 description DIST-2 no switchport ip address 10.10.255.3 255.255.255.254 ip ospf network point-to-point ip ospf 1 area 10 bfd interval 250 min_rx 250 multiplier 3 ! interface Vlan10 ip address 10.10.1.1 255.255.255.0 ip helper-address 10.0.50.10 ip ospf 1 area 10 ! router ospf 1 router-id 10.10.254.11 area 10 stub passive-interface default no passive-interface TenGigabitEthernet1/1/1 no passive-interface TenGigabitEthernet1/1/2 bfd all-interfaces
Verify
show
show ip ospf neighbor show ip route ospf show bfd neighbors show ip ospf | include Area|stub ! on the core show ip route 10.10.0.0 255.255.0.0 longer-prefixes
Prove it before you sign it off
| Test | Expect |
|---|---|
| show ip route ospf on ACC-1 | Intra-area routes for the block and a single inter-area default. Nothing from other blocks. |
| The same command on the core | One 10.10.0.0/16 summary from each distribution switch and nothing more specific. |
| Shut ACC-1's uplink to DIST-1 | BFD drops that adjacency at once and traffic carries on through DIST-2. Users lose a packet or two at most. |
| Shut DIST-1's link to ACC-2, then send traffic from the core to ACC-2 | Traffic that arrives at DIST-1 crosses the distribution link to DIST-2 instead of being dropped. |
| Reload DIST-1 | The core keeps the summary from DIST-2 and every access switch stays reachable. |
Running it on EIGRP instead
Plenty of campuses run EIGRP, and the design works just as well on it. The addressing, the /31 uplinks, the link between the distribution switches and BFD all stay the same. Four things take over the job the totally stubby area does in OSPF.
- Every access switch is a stub.
eigrp stub connectedmeans it only advertises its own networks and is never used as a path between the distribution switches. It also means they never query it when they lose a route, so a failure is dealt with in the distribution layer instead of spreading to every access switch. - The access layer gets a default route and nothing else. The distribution switches learn the default from the core and filter everything else out with a distribute-list on each access uplink. Don't make the default with
ip summary-address eigrp 100 0.0.0.0 0.0.0.0instead. That summary points at Null0 and stays up while any EIGRP route exists, so a distribution switch that has lost the core keeps offering a default it can only drop. - The block is summarised towards the core. One
ip summary-address eigrpon each core link, so the core sees 10.10.0.0/16 and nothing more specific. A change inside the block never reaches the core, and the core has nothing to query the block about. - BFD, not tight hello timers. Cisco's original guide set EIGRP hellos to 1 second and the hold time to 3. BFD on the uplinks finds a dead neighbour faster than that and leaves EIGRP on its default timers.
DIST-1, EIGRP
ip prefix-list DEFAULT-ONLY seq 5 permit 0.0.0.0/0 ! interface TenGigabitEthernet1/1/1 description CORE ip summary-address eigrp 100 10.10.0.0 255.255.0.0 ! router eigrp 100 eigrp router-id 10.10.254.1 network 10.0.255.0 0.0.0.255 network 10.10.0.0 0.0.255.255 passive-interface default no passive-interface TenGigabitEthernet1/0/1 no passive-interface TenGigabitEthernet1/1/1 no passive-interface TenGigabitEthernet1/1/8 ! one line per access uplink, never on the link to DIST-2 distribute-list prefix DEFAULT-ONLY out TenGigabitEthernet1/0/1 bfd all-interfaces
ACC-1, EIGRP
router eigrp 100 eigrp router-id 10.10.254.11 network 10.10.0.0 0.0.255.255 passive-interface default no passive-interface TenGigabitEthernet1/1/1 no passive-interface TenGigabitEthernet1/1/2 eigrp stub connected bfd all-interfaces ! show ip eigrp neighbors detail | include Stub show ip route eigrp
Details worth adding
- Give the summary a fixed cost.
area 10 range 10.10.0.0 255.255.0.0 cost 10on both distribution switches. Without it, the summary's cost follows the routes inside it, so it can change when a link in the block fails and the two distribution switches can end up advertising different costs. A fixed cost keeps the core load-sharing and stops changes inside the block reaching it. - Keep the distribution loopbacks out of area 0. An ABR sends a default into a stub area while it has any interface up in area 0, whether or not it can reach the backbone. A loopback never goes down, so a distribution switch that loses every core link keeps attracting traffic it can only drop, and it can't use its partner's default because an ABR ignores summary routes from a non-backbone area. That's why the config above puts Loopback0 in area 10. The gotcha, and the same fault as a Tick-Tock-TSHOOT ticket.
- Damp the uplinks, and check the link-down delay.
dampeningon each routed uplink holds a flapping link down instead of making the whole block recalculate every time it bounces. Cisco's guide also setcarrier-delay msec 0, so the switch acts on a lost link at once. Check what your platform does here, because any delay is added to every failover. - Know what you're buying. In Cisco's tests, a layer 2 access design with HSRP took 800 to 900 ms to recover from an uplink failure. Routed access took under 200 ms.
Gotchas this design avoids
- Every routed link between two switches elects a DR it does not need
- An area made stub on one router only
- passive-interface default drops every adjacency
- OSPF neighbours stuck in EXSTART or EXCHANGE
- OSPF advertises a loopback as /32 whatever its mask
- An ABR that has lost the backbone keeps sending a default into its stub area
- EIGRP access switches that aren't stubs slow every failure down
13 / Data centre
Nexus VXLAN EVPN fabric with an anycast gateway
NX-OS, Nexus 9000
A data centre that has outgrown one vPC pair. Racks need the same VLANs anywhere, routing at the first hop and tenants kept apart, without stretching spanning tree between them.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Underlay: routed point-to-point links, IP unnumbered from loopback 0, MTU 9216, OSPF point-to-point. Loopback 0 carries the router ID and BGP; loopback 1 is the VTEP.
- Overlay: iBGP L2VPN EVPN with both spines as route reflectors. Spines need no VXLAN configuration unless they are also borders.
- Broadcast and unknown unicast: a multicast underlay with both spines as anycast RP, or ingress replication if that traffic is light. Every VTEP uses the same mode and the same VNI-to-group mapping.
- Tenants: one L3 VNI per VRF, one L2 VNI per VLAN, and a distributed anycast gateway, so every leaf answers on the same gateway IP and MAC.
- ARP suppression on each L2 VNI. Some platforms need TCAM carving and a reload first.
- vPC leaf pairs: a shared secondary address on loopback 1 for bridged routes, with advertise-pip and advertise virtual-rmac so routed traffic reaches the leaf that actually has the route.
- Border leaf: no host /32 routes out to the edge, a default route injected into each tenant VRF, and that default filtered so it does not leak back out.
Addressing
| Use | Range | Example |
|---|---|---|
| Loopback 0, router ID and BGP | 10.254.0.0/24 as /32 | SPINE-1 .1, SPINE-2 .2, LEAF-3 .13 |
| Loopback 1, VTEP | 10.254.1.0/24 as /32 | LEAF-3 .13; LEAF-1 and LEAF-2 share .112 |
| Anycast RP | 10.254.2.1/32 | loopback 254 on both spines |
| Fabric links | IP unnumbered from loopback 0 | MTU 9216 |
| Multicast groups | 239.1.1.0/24 | VNI 30010 uses 239.1.1.10 |
| VRF TENANT-A | L3 VNI 50001 on VLAN 3001 | no address on VLAN 3001 |
| VLAN 10 | L2 VNI 30010, 10.10.10.0/24 | gateway 10.10.10.1 on every leaf |
Config
SPINE-1
nv overlay evpn
feature ospf
feature bgp
feature pim
!
interface loopback0
ip address 10.254.0.1/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback254
description Anycast RP, same on SPINE-2
ip address 10.254.2.1/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
ip pim rp-address 10.254.2.1 group-list 239.1.1.0/24
ip pim anycast-rp 10.254.2.1 10.254.0.1
ip pim anycast-rp 10.254.2.1 10.254.0.2
!
interface Ethernet1/1
description LEAF-1
no switchport
mtu 9216
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
no shutdown
router ospf UNDERLAY
router-id 10.254.0.1
!
router bgp 65000
router-id 10.254.0.1
template peer LEAVES
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.254.0.11
inherit peer LEAVES
neighbor 10.254.0.12
inherit peer LEAVES
neighbor 10.254.0.13
inherit peer LEAVESLEAF-3
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
fabric forwarding anycast-gateway-mac 0000.5e00.5310
ip pim rp-address 10.254.2.1 group-list 239.1.1.0/24
!
vlan 10
vn-segment 30010
vlan 3001
vn-segment 50001
vrf context TENANT-A
vni 50001
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
!
interface Vlan10
no shutdown
vrf member TENANT-A
ip address 10.10.10.1/24
fabric forwarding mode anycast-gateway
interface Vlan3001
no shutdown
mtu 9216
vrf member TENANT-A
ip forward
!
interface loopback0
ip address 10.254.0.13/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
interface loopback1
ip address 10.254.1.13/32
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
! fabric uplinks as on the spine
!
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
member vni 30010
suppress-arp
mcast-group 239.1.1.10
member vni 50001 associate-vrf
evpn
vni 30010 l2
rd auto
route-target import auto
route-target export auto
!
router bgp 65000
router-id 10.254.0.13
neighbor 10.254.0.1
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.254.0.2
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf TENANT-A
address-family ipv4 unicast
advertise l2vpn evpnLEAF-1 (vPC pair additions)
interface loopback1
ip address 10.254.1.11/32
ip address 10.254.1.112/32 secondary
!
vpc domain 10
peer-switch
peer-gateway
ip arp synchronize
delay restore 150
auto-recovery
!
router bgp 65000
address-family l2vpn evpn
advertise-pip
interface nve1
advertise virtual-rmacVerify
show
show nve peers show nve vni show bgp l2vpn evpn summary show l2route evpn mac all show ip route vrf TENANT-A show ip mroute
Prove it before you sign it off
| Test | Expect |
|---|---|
| show nve peers on LEAF-3 | LEAF-1 and LEAF-2 appear once, as their shared VTEP address, plus BORDER-1. All Up. |
| Ping a VLAN 10 host on LEAF-3 from one on LEAF-1 | It works. show l2route evpn mac all lists the remote MAC against LEAF-3's VTEP. |
| Move a virtual machine from LEAF-1 to LEAF-3 | The MAC mobility sequence number rises, the gateway does not change and traffic resumes within seconds. |
| Reload SPINE-1 | Every leaf keeps its routes through SPINE-2. Tenant traffic sees a brief blip, not an outage. |
| show ip route on the edge router | Tenant prefixes or summaries only, with no host /32 routes. |
14 / Data centre
EVPN Multi-Site between two data centres
NX-OS, Nexus 9000
Two data centres that need layer 2 and layer 3 between them without becoming one failure domain. Each site stays its own fabric, and border gateways join them.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Each site is a complete fabric with its own AS, its own replication mode and its own failure domain.
- The border gateways in a site share a Multi-Site virtual IP on a loopback, so the other site sees one next hop per site.
- Track both sides of every gateway: fabric-tracking on links into the local fabric, dci-tracking on links to the other site. A gateway that loses either side stops advertising.
- eBGP EVPN between sites from loopbacks, with peer-type fabric-external and rewrite-evpn-rt-asn, because each site's automatic route targets contain its own AS.
- A delay-restore on each gateway (300 seconds in Cisco's example) so a rebooted gateway does not attract traffic before it has converged.
- Between sites, broadcast and unknown unicast use ingress replication per VNI, whatever each site uses inside.
- Advertise only the gateway loopbacks into the inter-site underlay, by tag.
Addressing
| Item | Site A | Site B |
|---|---|---|
| Fabric AS | 65001 | 65002 |
| Gateway loopback 0, BGP | 10.1.254.31 and .32 | 10.2.254.41 and .42 |
| Gateway loopback 1, own VTEP | 10.1.255.31 and .32 | 10.2.255.41 and .42 |
| Loopback 100, Multi-Site VIP | 10.1.100.1 on both | 10.2.100.1 on both |
| DCI links | 10.255.0.0/31 (A1 to B1) | 10.255.0.2/31 (A2 to B2) |
| Shared VNIs | L2 30010, L3 50001 | the same |
Config
BGW-A1
nv overlay evpn
feature bgp
feature nv overlay
!
evpn multisite border-gateway 100
delay-restore time 300
!
interface loopback0
ip address 10.1.254.31/32 tag 54321
interface loopback1
ip address 10.1.255.31/32 tag 54321
interface loopback100
description Multi-Site VIP, same on BGW-A2
ip address 10.1.100.1/32 tag 54321
!
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
multisite border-gateway interface loopback100
member vni 30010
multisite ingress-replication
mcast-group 239.1.1.10
member vni 50001 associate-vrf
!
interface Ethernet1/1
description SPINE-A1
evpn multisite fabric-tracking
interface Ethernet1/49
description DCI to BGW-B1
no switchport
mtu 9216
ip address 10.255.0.0/31
evpn multisite dci-tracking
no shutdown
!
route-map RM-BGW-LOOPBACKS permit 10
match tag 54321
router bgp 65001
router-id 10.1.254.31
address-family ipv4 unicast
redistribute direct route-map RM-BGW-LOOPBACKS
neighbor 10.255.0.1
remote-as 65002
address-family ipv4 unicast
neighbor 10.2.254.41
remote-as 65002
update-source loopback0
ebgp-multihop 5
peer-type fabric-external
address-family l2vpn evpn
send-community
send-community extended
rewrite-evpn-rt-asn
! repeat for 10.2.254.42; fabric-side underlay and iBGP as on any leafVerify
show
show nve interface nve1 detail show nve multisite fabric-links show nve multisite dci-links show nve peers show bgp l2vpn evpn summary
Prove it before you sign it off
| Test | Expect |
|---|---|
| show nve interface nve1 detail on each gateway | Multi-Site state up, with fabric and DCI tracking both up. |
| show nve peers on BGW-A1 | Site B appears by its Multi-Site virtual address. |
| Shut every DCI link on BGW-A1 | BGW-A1 withdraws itself as a gateway and traffic to site B carries on through BGW-A2. |
| Reload BGW-A1 | It stays out of the path until the 300-second delay-restore has expired. |
| Break the underlay inside site B | Site A's fabric is unaffected apart from the loss of reachability to site B. |
15 / Data centre
ACI Multi-Pod across two rooms or sites
ACI, with Nexus 9000 as the inter-pod network
One ACI fabric spread across two rooms or two sites, with one policy domain. The inter-pod network carries VXLAN between the pods, and the APIC cluster spans both.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Inter-pod network MTU at least 50 bytes above the largest endpoint frame. The control-plane MTU on APIC defaults to 9000.
- Spines connect to the IPN with VLAN 4 sub-interfaces. The IPN runs OSPF with the spines (eBGP from 5.2(3)) and relays DHCP to the APICs, so new pods can be discovered.
- PIM Bidir in the IPN carries flooded traffic for bridge domains stretched between pods. Use a phantom RP: an address nobody owns, inside a /30 on one IPN router and a /29 on the other.
- Up to 50 ms round trip between pods.
- Three APICs: two in Pod 1, one in Pod 2, plus a standby APIC in Pod 2 for the loss of Pod 1. With five or more, never more than two in one pod.
- Lower the fabric-wide IS-IS metric for redistributed routes below 63 before the first upgrade.
- Keep external traffic local: an L3Out in each pod, with host-route advertisement or local preference per bridge domain. By default every pod's border leaves look equal.
- Two pods back to back with no IPN is supported from 5.2(3), but rules out Multi-Site, Remote Leaf and APIC over a routed network.
Addressing
| Item | Pod 1 | Pod 2 |
|---|---|---|
| TEP pool | 10.1.0.0/16 | 10.2.0.0/16 |
| Data-plane TEP on the spines | 10.1.255.254/32 | 10.2.255.254/32 |
| IPN links on VLAN 4 | 10.255.1.0/24 as /30 | 10.255.2.0/24 as /30 |
| Phantom RP | 10.255.255.1: IPN-1 has the /30, IPN-2 the /29 | the same |
| Bridge-domain multicast | 225.0.0.0/15 (default) | the same |
| APICs | APIC-1, APIC-2 | APIC-3, standby APIC |
Config
IPN-1
feature ospf feature pim feature dhcp feature lldp service dhcp ip dhcp relay ! interface Ethernet1/1 description SPINE-1A mtu 9216 no shutdown interface Ethernet1/1.4 description SPINE-1A on VLAN 4 mtu 9216 encapsulation dot1q 4 ip address 10.255.1.1/30 ip ospf network point-to-point ip router ospf IPN area 0.0.0.0 ip pim sparse-mode ip dhcp relay address 10.1.0.1 ip dhcp relay address 10.1.0.2 ip dhcp relay address 10.1.0.3 no shutdown ! interface loopback1 description Phantom RP: /30 here, /29 on IPN-2 ip address 10.255.255.2/30 ip ospf network point-to-point ip router ospf IPN area 0.0.0.0 ip pim sparse-mode ip pim rp-address 10.255.255.1 group-list 225.0.0.0/15 bidir ip pim rp-address 10.255.255.1 group-list 239.255.255.240/28 bidir ! router ospf IPN router-id 10.255.0.1
APIC settings
Pod Fabric Setup Policy Pod 1 TEP pool 10.1.0.0/16 Pod 2 TEP pool 10.2.0.0/16 Multi-Pod L3Out (tenant infra) Spine sub-interfaces VLAN 4, OSPF area 0, point-to-point Data-plane TEP, Pod 1 10.1.255.254/32 Data-plane TEP, Pod 2 10.2.255.254/32 System Settings > ISIS Policy ISIS metric for redistributed routes 32 System Settings > Control Plane MTU 9000 ! menu paths vary by APIC release
Verify
show
show ip ospf neighbors show ip pim rp show ip mroute ping 10.255.1.2 packet-size 9150 df-bit ! on an APIC acidiag avread
Prove it before you sign it off
| Test | Expect |
|---|---|
| acidiag avread on each APIC | All three active APICs are fully fit and agree on the cluster. |
| show ip pim rp on both IPN routers | Both use 10.255.255.1, reached through IPN-1's /30. |
| A 9150-byte ping with DF set across every IPN link | Replies everywhere. A link with a smaller MTU shows up here, not in production. |
| Shut IPN-1 | IPN-2's /29 becomes the path to the RP and flooded traffic between pods continues. |
| Cut Pod 2 off from the IPN | Pod 1 keeps the APIC majority and policy can still be changed there. Pod 2 forwards on what it already has. |
16 / Campus
SD-Access fabric site with a fusion pair for shared services
IOS XE, Catalyst 9000, Catalyst Center
A campus moving to SD-Access, where segmentation follows the user rather than the VLAN. Catalyst Center provisions most of the fabric. What goes wrong is role placement, the handoff to shared services and the move to default deny.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Pick the site model by size: Fabric in a Box (every role on one switch, stack or pair) for under about 1,000 endpoints and 50 APs; a collapsed core with access for under about 10,000.
- Co-locate the control plane with the border for the simplest design. Use dedicated control-plane nodes where users roam a lot, and no more than two control-plane nodes when they are co-located with fabric wireless.
- Routed underlay (IS-IS from LAN automation, or OSPF) with ECMP, BFD and NSF with SSO. The underlay is IPv4 only; the overlay can be dual stack. VXLAN cannot be fragmented, so raise the MTU end to end.
- Shared services stay outside the fabric. A VRF-aware fusion pair (StackWise Virtual or vPC) with each border connected to both members, one sub-interface and BGP session per virtual network, and leaking by prefix list and route-map.
- The WLC sits in the shared-services block with its management address in the global table. Fabric APs live in INFRA_VN, so AP join needs no leaking.
- Configure a critical VLAN, so new hosts get limited access when ISE cannot be reached.
- Before default deny, turn off TrustSec enforcement on underlay links and on AP and extended-node VLANs.
- Several sites: an SD-Access transit with dedicated transit control-plane nodes, using LISP Pub/Sub.
Addressing
| Use | Range | Example |
|---|---|---|
| Underlay loopbacks | 10.200.0.0/24 as /32 | BORDER-1 .1, EDGE-1 .11 |
| Underlay links | 10.200.255.0/24 as /31 | LAN automation can allocate these |
| Border to fusion | 10.200.250.0/24 as /30, one VLAN per VN | CORP on VLAN 3001, INFRA_VN on 3002 |
| CORP pool | 10.101.0.0/16 | user subnets allocated by Catalyst Center |
| GUEST pool | 10.102.0.0/16 | handed to the guest firewall |
| Shared services | 10.50.0.0/24 | DHCP .5, DNS .6, ISE .21 and .22, WLC .30 |
| Fabric MTU | 9100 | end to end, including the borders |
Config
FUSION
vrf definition CORP rd 65100:101 address-family ipv4 import ipv4 unicast map SHARED-TO-CORP export ipv4 unicast map CORP-TO-GLOBAL ! interface TenGigabitEthernet1/1/1.3001 description BORDER-1, CORP encapsulation dot1Q 3001 vrf forwarding CORP ip address 10.200.250.2 255.255.255.252 interface TenGigabitEthernet1/1/1.3002 description BORDER-1, INFRA_VN in the global table encapsulation dot1Q 3002 ip address 10.200.250.6 255.255.255.252 ! ip prefix-list SHARED-SERVICES seq 5 permit 10.50.0.0/24 ip prefix-list CORP-POOLS seq 5 permit 10.101.0.0/16 le 32 route-map SHARED-TO-CORP permit 10 match ip address prefix-list SHARED-SERVICES route-map CORP-TO-GLOBAL permit 10 match ip address prefix-list CORP-POOLS ! router bgp 65100 bgp router-id 10.200.0.21 neighbor 10.200.250.5 remote-as 65001 address-family ipv4 network 10.50.0.0 mask 255.255.255.0 neighbor 10.200.250.5 activate address-family ipv4 vrf CORP neighbor 10.200.250.1 remote-as 65001 neighbor 10.200.250.1 activate
Fabric nodes, before default deny
! Catalyst Center 2.3.7 and later does this for new links interface range TenGigabitEthernet1/1/1 - 2 description Underlay uplinks no cts role-based enforcement ! ! detect a dead ISE quickly, so the critical VLAN takes over radius-server dead-criteria time 5 tries 3 radius-server deadtime 5
Verify
show
show lisp session show ip route vrf CORP show bgp vpnv4 unicast all summary show cts role-based permissions show cts interface brief
Prove it before you sign it off
| Test | Expect |
|---|---|
| show lisp session on an edge node | Sessions up to both control-plane nodes. |
| From a CORP host, reach DHCP and DNS | Works through a border and the fusion pair. The fusion shows the CORP pool in the global table and 10.50.0.0/24 in VRF CORP, nothing else. |
| Reload BORDER-1 | Edges use BORDER-2 for external routes, and their LISP sessions to the second control-plane node stay up. |
| Block RADIUS to ISE, then connect a new host | It lands in the critical pool with limited access instead of being denied. |
| Turn on default deny at a lab site first | Underlay adjacencies and BFD stay up. Only user traffic without a permit is blocked. |
17 / WAN
SD-WAN controllers that survive losing a data centre
Catalyst SD-WAN, IOS XE SD-WAN
A WAN with controllers in two data centres or cloud regions, where losing a whole site must not take the branches with it. The defaults look resilient and are not quite.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- By default each edge builds two control connections per transport and two OMP sessions in total, one per controller however many transports reach it. An edge below that count holds a standing connection to the Validator.
- Put the controllers in groups by data centre or region. Give each site a controller-group-list with its local group first, and keep the same number of controllers in every group.
- Check on paper that every site still reaches two allowed controllers with either data centre gone.
- If every controller is lost, edges keep forwarding on their last known state for the OMP graceful-restart timer, 12 hours by default. The OMP hold time is 60 seconds.
- Several Validators behind one DNS name, in different regions. A Manager cluster scales management but is not disaster recovery, and must not be stretched: its members need 4 ms or less between them.
- Controllers in a colocation site behind a hub edge: the hub's implicit ACL drops branch control traffic, so permit the DTLS and TLS ports explicitly on the hub tunnel interface.
- Port hopping off at data centres, hubs and on private transports.
- Dual-router branches: VRRP tracks a prefix list, not OMP, and tloc-change increase-pref pulls return traffic to the VRRP active router.
Addressing
| Item | DC1 | DC2 | Branch |
|---|---|---|---|
| Site ID | 100 | 200 | 1101 |
| Controller group | 1 | 2 | list 1 then 2 |
| System IPs | 10.255.0.1, .2 | 10.255.0.3, .4 | 10.255.1.11, .12 |
| Transports | mpls, biz-internet | mpls, biz-internet | mpls on EDGE-1, biz-internet on EDGE-2 |
| Validator | validator.example.net | the same | the same |
Config
Controller in DC1
system system-ip 10.255.0.1 site-id 100 controller-group-id 1 organization-name "EXAMPLE-ORG" vbond validator.example.net
EDGE-1
system system-ip 10.255.1.11 site-id 1101 organization-name "EXAMPLE-ORG" vbond validator.example.net controller-group-list 1 2 max-omp-sessions 2 ! omp no shutdown graceful-restart timers holdtime 60 graceful-restart-timer 43200 ! sdwan interface GigabitEthernet1 tunnel-interface encapsulation ipsec color mpls restrict max-control-connections 2 no port-hop
Verify
show
show sdwan control connections show sdwan control local-properties show sdwan omp peers show sdwan bfd sessions
Prove it before you sign it off
| Test | Expect |
|---|---|
| show sdwan control connections on EDGE-1 | Two control connections on its transport, both to controllers in group 1. |
| show sdwan omp peers | Two OMP peers, both in DC1. |
| Block DC1's controllers | The edges connect to group 2. User traffic keeps flowing on the last known state while they do. |
| show sdwan control local-properties after recovery | No standing connection to the Validator, so the edge is back in equilibrium. |
| Power off EDGE-1 | EDGE-2 takes the LAN gateway in seconds, not after the 60-second OMP hold time. |
18 / Security
Firewall cluster on a Nexus vPC pair
ASA, Firepower 4100 and 9300, Nexus 9000
One firewall pair can no longer carry the throughput or the connection count. A cluster scales out, but only if the switches and the cluster control link are built for it, and by less than the data sheet suggests.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Expect partial scaling. Cisco's figures: throughput at 70 to 80 percent of the sum, concurrent connections at about 60 percent, new connections per second at about 50 percent.
- Some features stay on the control node (site-to-site VPN on most platforms, several inspection engines, non-per-session PAT), and their failover is not seamless.
- Cluster control link: dedicated interfaces on every member, never shared. A per-member LACP EtherChannel sized to that member's forwarding capacity, split across both switches, on an isolated VLAN, never cabled back to back.
- Control link MTU at least 100 bytes above the largest data MTU, at most 10 ms one way, and no loss or reordering.
- Data: one spanned EtherChannel across every member (the only mode on Firepower 4100 and 9300), with a fixed virtual MAC on it.
- The same EtherChannel hash on every switch connected to the cluster, so both directions of a flow land on the same member. On ASA appliances, turn off LACP graceful convergence on the Nexus side.
- Tune the health-check hold time, leave management interfaces out of interface monitoring, and set automatic rejoin after cluster, data and system failures.
- Clustering between sites is possible, with up to 10 ms one way, but avoid it unless there is no alternative.
Addressing
| Item | Value | Note |
|---|---|---|
| Cluster control link | 10.99.99.0/24 on VLAN 999 | isolated: no gateway, never routed |
| Control link addresses | FW-1 .1, FW-2 .2, FW-3 .3 | |
| Inside | 10.0.10.1/24 on Port-channel10.100 | one address for the whole cluster |
| Outside | 203.0.113.10/24 on Port-channel10.200 | |
| Data EtherChannel | vPC 10 on the Nexus pair | spanned across every member |
| MTU | inside 9084, control link 9184 | control link 100 bytes larger |
Config
FW-1 (ASA)
cluster interface-mode spanned force ! interface TenGigabitEthernet0/6 channel-group 48 mode active no shutdown interface TenGigabitEthernet0/7 channel-group 48 mode active no shutdown ! cluster group FW-CLUSTER local-unit FW-1 cluster-interface Port-channel48 ip 10.99.99.1 255.255.255.0 priority 1 key <shared-secret> health-check holdtime 3 no health-check monitor-interface Management0/0 health-check data-interface auto-rejoin 3 5 2 health-check cluster-interface auto-rejoin unlimited 5 1 enable ! ! the rest is configured on the control node and replicates interface TenGigabitEthernet0/8 channel-group 10 mode active no shutdown interface TenGigabitEthernet0/9 channel-group 10 mode active no shutdown interface Port-channel10 port-channel span-cluster mac-address 0000.5e00.5320 interface Port-channel10.100 vlan 100 nameif inside security-level 100 ip address 10.0.10.1 255.255.255.0 interface Port-channel10.200 vlan 200 nameif outside security-level 0 ip address 203.0.113.10 255.255.255.0 mtu inside 9084 mtu cluster 9184
N9K-1
vlan 999 name FW-CCL ! interface port-channel10 description Firewall data, spanned across every member switchport mode trunk switchport trunk allowed vlan 100,200 mtu 9216 no lacp graceful-convergence vpc 10 interface port-channel11 description FW-1 cluster control link switchport access vlan 999 mtu 9216 vpc 11 interface port-channel12 description FW-2 cluster control link switchport access vlan 999 mtu 9216 vpc 12 ! port-channel13 for FW-3 the same; N9K-2 mirrors this ! interface Ethernet1/10 description FW-1 data switchport mode trunk switchport trunk allowed vlan 100,200 mtu 9216 channel-group 10 mode active interface Ethernet1/20 description FW-1 cluster control link switchport access vlan 999 mtu 9216 channel-group 11 mode active
Verify
show
show cluster info show cluster info health show cluster info conn-distribution show port-channel summary ! on the Nexus show vpc brief show port-channel summary
Prove it before you sign it off
| Test | Expect |
|---|---|
| show cluster info | One control node and every other member in the cluster as a data node. |
| show cluster info conn-distribution under real load | Connections spread roughly evenly across members. |
| Pull one cluster control link cable from FW-2 | FW-2 stays in the cluster on its other link. |
| Power off FW-3 | Its flows continue on the other members and the cluster carries on. |
| A large file transfer through the cluster | No stalls. The control link counters show no drops. |
19 / Security
ISE deployment that survives a node failure
ISE, IOS XE access switches
802.1X and MAB for a campus or several sites, where ISE nodes, profiling probes and the switches' RADIUS settings have to ride out a failure without overloading anything or locking users out.
Layer 2 trunkPort-channelRoutedBlocked by STPKeepalive, tunnel or out-of-band
The rules
- Size from the current data sheets. Capacity depends on persona layout (standalone, hybrid or dedicated), and every dedicated TACACS+ node reduces the RADIUS node count. Run device administration on at least two nodes.
- Every switch has two policy nodes, the nearest first, with dead-criteria, deadtime and an automated test user so a dead node is noticed and retested without user traffic.
- Behind a load balancer: keep each client on one node (persist on Calling-Station-ID or Audit Session ID), send DHCP copies to the same virtual address, fail the health probe over before the switch's RADIUS timeout, reassemble fragmented RADIUS, and return web redirects to the node that handled the session.
- Profiling: DHCP helpers and URL redirects rather than SPAN. Avoid SNMP traps and NetFlow probes, poll SNMP no more than every 8 hours from the nearest node, and run pxGrid on two nodes at most.
- Node groups, so policy nodes in one place share endpoint ownership.
- Critical authentication when every node is dead: new and re-authenticating hosts go to a limited critical VLAN, and are re-initialised with a recovery delay when a node returns.
- Policy nodes keep authenticating while the primary admin node is down. Only changes wait for it, or for the standby to be promoted.
Addressing
| Node | Address | Role |
|---|---|---|
| PAN-1 | 10.50.0.11 | primary admin and monitoring |
| PAN-2 | 10.50.0.12 | secondary admin and monitoring |
| PSN-1 | 10.50.0.21 | policy service, node group SITE-A |
| PSN-2 | 10.50.0.22 | policy service, node group SITE-A |
| DHCP server | 10.50.0.5 | |
| Critical VLAN | 999 | DHCP, DNS and the path to ISE only |
Config
ACCESS-1
aaa new-model ! radius server ISE-PSN-1 address ipv4 10.50.0.21 auth-port 1812 acct-port 1813 key <shared-secret> automate-tester username radius-probe probe-on radius server ISE-PSN-2 address ipv4 10.50.0.22 auth-port 1812 acct-port 1813 key <shared-secret> automate-tester username radius-probe probe-on aaa group server radius ISE server name ISE-PSN-1 server name ISE-PSN-2 ! radius-server dead-criteria time 5 tries 3 radius-server deadtime 5 aaa authentication dot1x default group ISE aaa authorization network default group ISE aaa accounting dot1x default start-stop group ISE aaa server radius dynamic-author client 10.50.0.21 server-key <shared-secret> client 10.50.0.22 server-key <shared-secret> dot1x system-auth-control authentication critical recovery delay 1000 dot1x critical eapol ! interface GigabitEthernet1/0/10 switchport mode access switchport access vlan 10 authentication event server dead action authorize vlan 999 authentication event server alive action reinitialize authentication host-mode multi-auth authentication port-control auto mab dot1x pae authenticator spanning-tree portfast ! interface Vlan10 ip helper-address 10.50.0.5 ip helper-address 10.50.0.21
ISE settings
Deployment PAN-1 primary admin, primary monitoring PAN-2 secondary admin, secondary monitoring PSN-1 policy service, node group SITE-A PSN-2 policy service, node group SITE-A Profiling probes on PSN-1 and PSN-2 RADIUS, DHCP, HTTP on SNMP query every 8 hours or more SPAN, NetFlow, SNMP trap off pxGrid two nodes at most Network device: ACCESS-1 shared secret, CoA port 1700
Verify
show
show aaa servers test aaa group ISE radius-probe <password> new-code show access-session interface GigabitEthernet1/0/10 details show authentication sessions
Prove it before you sign it off
| Test | Expect |
|---|---|
| show aaa servers | Both policy nodes UP, with the automated test user probing each. |
| Stop PSN-1 | ACCESS-1 marks it dead within the dead-criteria time and new authentications go to PSN-2. |
| Block RADIUS to both policy nodes, then connect a new host | It lands in VLAN 999 with limited access. Existing sessions stay as they are. |
| Bring PSN-1 back | Hosts in the critical VLAN are re-authenticated after the recovery delay, not all at once. |
| Power off PAN-1 | Users still authenticate. Admin changes wait until PAN-1 returns or PAN-2 is promoted. |
20 / WAN
Replacing the WAN after a run of acquisitions
Catalyst SD-WAN (IOS XE edges), Catalyst 9000 core
Sixty-odd sites, many of them bought in, with clashing LAN ranges, every internet packet hairpinning through one colocation suite, and the main application about to move to SaaS. The case behind it: a veterinary group with 64 practices and two 24-hour hospitals, a five-person IT team, an 18-month deadline to integrate the acquisitions and a board target to cut WAN cost by a fifth. Practices can work on paper for four hours; the hospitals cannot be down for more than 15 minutes.
Internet or routed linkSD-WAN overlay tunnel
The rules
- Plan one address block for every site before the first migration: 10.100.0.0/16, a /22 per practice. Renumber each acquired practice in its own migration window. Temporary NAT only for sites that must reach the centre before their turn, removed afterwards.
- Two internet underlays from different providers at every practice, one fixed and one cellular. No MPLS: the main application is SaaS and only reachable over the internet.
- Break out SaaS and cloud telephony locally. Send other web traffic through the cloud security service. Only traffic for systems that live in the colocation suite goes to the hubs.
- Card terminals in their own service VRF that can reach only the payment provider. It keeps the rest of the practice out of the payment card scope.
- Hospitals get two fixed circuits from different carriers with separate building entries, plus cellular. A four-hour repair time is not a 15-minute outage limit.
- eBGP between the hubs and the core, one summary in each direction. No redistribution into the IGP and no static route per site.
- Cloud-hosted SD-WAN management, with every alert sent to whoever is on call together with a written action. No controllers to run in a suite that is shrinking.
- Migrate each site alongside its old service. Keep the old firewall and line for a week as the rollback, then remove them.
Addressing
| Range | Used for | Segment | Notes |
|---|---|---|---|
| 10.100.0.0/16 | All practices | summary | The only practice route the core sees |
| 10.100.68.0/22 | Practice 17 (site number x 4) | per site | Same pattern at every practice |
| 10.100.68.0/24 | Staff, VLAN 10 | service VPN 10 | DHCP from the colocation server |
| 10.100.69.0/24 | Voice, VLAN 20 | service VPN 10 | Cloud telephony breaks out locally |
| 10.100.70.0/26 | Card terminals, VLAN 30 | service VPN 30 | Static addresses, payment provider only |
| 10.100.71.0/24 | Guest Wi-Fi, VLAN 40 | service VPN 40 | Internet only |
| 10.101.0.0/21 | The two hospitals | summary | A /22 each |
| 10.110.0.0/30, 10.110.0.4/30 | Core to HUB-1, core to HUB-2 | transit | Core .1 and .5 |
| 10.120.0.0/16 | Colocation services | summary | Imaging, identity, DHCP |
| 192.168.1.0/24 as 10.199.17.0/24 | Practice 17 before it migrates | temporary NAT | On its old firewall and IPsec tunnel, gone at migration |
| 203.0.113.64/27 | Payment provider (example) | internet | Use the provider's published ranges |
Config
PRACTICE-17 edge (service side)
vrf definition 10 rd 1:10 address-family ipv4 exit-address-family vrf definition 30 rd 1:30 address-family ipv4 exit-address-family ! interface GigabitEthernet0/0/1.10 description Staff encapsulation dot1Q 10 vrf forwarding 10 ip address 10.100.68.1 255.255.255.0 ip helper-address 10.120.0.10 interface GigabitEthernet0/0/1.20 description Voice encapsulation dot1Q 20 vrf forwarding 10 ip address 10.100.69.1 255.255.255.0 ip helper-address 10.120.0.10 interface GigabitEthernet0/0/1.30 description Card terminals encapsulation dot1Q 30 vrf forwarding 30 ip address 10.100.70.1 255.255.255.192 ip access-group PAYMENTS-ONLY in ! ! VPN 30 has its own local internet exit and no route to anything else ip access-list extended PAYMENTS-ONLY remark card terminals reach the payment provider and nothing else permit tcp 10.100.70.0 0.0.0.63 203.0.113.64 0.0.0.31 eq 443 deny ip any any
HUB-1 (service VPN 10)
router bgp 65020 address-family ipv4 vrf 10 neighbor 10.110.0.1 remote-as 65010 neighbor 10.110.0.1 description CORE neighbor 10.110.0.1 activate redistribute omp aggregate-address 10.100.0.0 255.255.0.0 summary-only aggregate-address 10.101.0.0 255.255.248.0 summary-only exit-address-family ! ! carry the colocation routes learned from the core into the overlay sdwan omp address-family ipv4 vrf 10 advertise bgp ! HUB-2 identical, neighbour 10.110.0.5
CORE (colocation)
router bgp 65010 bgp log-neighbor-changes neighbor 10.110.0.2 remote-as 65020 neighbor 10.110.0.2 description HUB-1 neighbor 10.110.0.6 remote-as 65020 neighbor 10.110.0.6 description HUB-2 address-family ipv4 network 10.120.0.0 mask 255.255.0.0 neighbor 10.110.0.2 activate neighbor 10.110.0.2 prefix-list FROM-SDWAN in neighbor 10.110.0.2 prefix-list TO-SDWAN out neighbor 10.110.0.6 activate neighbor 10.110.0.6 prefix-list FROM-SDWAN in neighbor 10.110.0.6 prefix-list TO-SDWAN out ! ip prefix-list FROM-SDWAN seq 10 permit 10.100.0.0/16 ip prefix-list FROM-SDWAN seq 20 permit 10.101.0.0/21 ip prefix-list TO-SDWAN seq 10 permit 10.120.0.0/16 ! ! anchor so the network statement has an exact match ip route 10.120.0.0 255.255.0.0 Null0
Verify
show
show sdwan control connections show sdwan bfd sessions show sdwan omp routes vpn 10 ! on the core show ip bgp summary show ip route 10.100.0.0 255.255.0.0
Prove it before you sign it off
| Test | Expect |
|---|---|
| Unplug the fixed line at a practice | Traffic moves to cellular within seconds. The PMS stays usable and calls in progress continue or reconnect. |
| From a card terminal, try any address other than the payment provider | Blocked. A test payment still completes. |
| From a staff PC, open a test page in a blocked category | The cloud security service's block page appears. The PMS goes direct. |
| Shut HUB-1 | Practices still reach imaging through HUB-2. The core keeps one route to 10.100.0.0/16. |
| show ip route on the core | One route each for 10.100.0.0/16 and 10.101.0.0/21, not 64 separate /22s. |
| Pull one carrier at a hospital | Traffic moves in seconds, well inside the 15-minute limit. |
21 / Security
Separating operational technology from IT with a DMZ between them
IOS XE routers, Catalyst switches, any firewall vendor
SCADA servers, PLCs and remote outstations sharing a network with corporate IT, so anything that compromises IT can reach the plant. The case behind it: a water company with six treatment works and 380 remote sites, a regulatory audit in nine months, budget for one firewall pair this year, and a board instruction that if the corporate network is compromised the water must keep flowing.
Routed, through a firewall where shownIPsec tunnel
The rules
- Three zones: corporate IT, a DMZ, operational technology. Nothing passes straight from IT to OT or back. Every service the two share lives in the DMZ.
- Data leaves OT one way. The historian replicates out to a copy in the DMZ, and reporting reads only the copy.
- Remote access lands in the DMZ: multi-factor authentication, jump hosts, recorded sessions and access approved for a set time. No VPN into the plant network and no vendor remote tools on operator screens.
- Outstations use public cellular with private APNs from two operators, and IPsec with certificates from each site router to both control centres. Tunnels end in the OT zone, never on the corporate WAN.
- Keep the OT firewall pair together at the main control centre. A pair split across two sites leaves each with one firewall and puts failover traffic on the WAN.
- Build the rules from recorded flows. Watch for a few weeks, log before you block, then enforce.
- Until the DR control centre has its own firewalls, apply strict access lists on its router and record the gap and its end date in the risk register.
- Replace unmanaged switches at the treatment works. Lock PLC ports with port security set to restrict, so an unknown device is logged and dropped without stopping the plant.
- Watch OT with passive monitoring that understands DNP3 and Modbus, fed from mirror ports. Never scan PLCs actively.
Addressing
| From | To | Allowed | Why |
|---|---|---|---|
| OT historian (10.200.10.20) | DMZ historian copy (10.150.0.30) | Replication, outbound only | Reporting data |
| Corporate reporting | DMZ historian copy | SQL, read only | Five-minute finance reports |
| Engineers (anywhere) | DMZ remote access gateway (10.150.0.10) | HTTPS with MFA | Out-of-hours access |
| DMZ jump hosts (10.150.0.21, .22) | SCADA and PLCs (10.200.10.0/24, 10.200.20.0/22) | RDP and the PLC vendor's tool ports, approved sessions | Engineering work |
| OT monitoring sensor | DMZ collector (10.150.0.40) | Syslog and HTTPS | Security alerts |
| Outstations (10.210.0.0/16) | SCADA (10.200.10.0/24) | DNP3 TCP 20000, Modbus TCP 502, inside IPsec | Telemetry and control |
| Corporate IT (10.0.0.0/12) | OT zone (10.200.0.0/16, DR 10.201.0.0/16) | Nothing | The point of the design |
Config
DR-ROUTER (interim, until its firewalls arrive)
ip access-list extended CORP-IN remark only the DMZ jump hosts reach DR SCADA from the corporate side permit tcp host 10.150.0.21 10.201.10.0 0.0.0.255 eq 3389 permit tcp host 10.150.0.22 10.201.10.0 0.0.0.255 eq 3389 deny ip any any ip access-list extended CORP-OUT permit tcp 10.201.10.0 0.0.0.255 eq 3389 host 10.150.0.21 established permit tcp 10.201.10.0 0.0.0.255 eq 3389 host 10.150.0.22 established deny ip any any ! interface GigabitEthernet0/0/1 description To corporate WAN ip access-group CORP-IN in ip access-group CORP-OUT out
OT-CORE (mirror for passive monitoring)
monitor session 1 source vlan 110 both monitor session 1 source vlan 120 both monitor session 1 destination interface GigabitEthernet1/0/48 ! interface GigabitEthernet1/0/48 description OT monitoring sensor (receive only)
WORKS-SW1 (PLC access ports)
interface range GigabitEthernet1/0/1 - 12 description PLC ports switchport mode access switchport access vlan 120 switchport port-security switchport port-security maximum 1 switchport port-security mac-address sticky ! restrict, not shutdown: log the stranger, keep the plant running switchport port-security violation restrict spanning-tree portfast spanning-tree bpduguard enable no cdp enable
Verify
show
show access-lists CORP-IN show monitor session 1 show port-security show port-security interface GigabitEthernet1/0/1 show crypto session
Prove it before you sign it off
| Test | Expect |
|---|---|
| From a corporate PC, try to reach a SCADA server or PLC on any port | No response. Nothing in IT reaches OT. |
| Run the finance report | It reads the DMZ copy, with data no more than five minutes old. |
| An engineer logs in from home | MFA prompt, approval needed, session recorded, access ends at the approved time. |
| Pull one operator's SIM at a test outstation | Telemetry carries on over the second operator with no gap longer than the polling interval. |
| Plug a laptop into a PLC port | Port security logs the new MAC and drops its traffic. The PLC keeps running. |
| Send a test DNP3 write from an unexpected address | The passive monitoring raises an alert in the security team's console. |
22 / Data centre
Disaster recovery between two data centres without stretching a VLAN
IOS XE data centre cores (NX-OS has the same building blocks), SD-WAN to the sites
Two data centres, an application that must be back within an hour with no more than 15 minutes of data lost, and a request to stretch VLANs between the sites so servers keep their addresses. The case behind it: a logistics firm with 14 warehouses whose disaster recovery test took nine hours, because the database restore was slow, the second site's firewall rules were two years old, DNS answers were cached for a day and the runbook named servers that no longer existed.
RoutedStandby path, unused until failoverWitness connection
The rules
- No stretched layer 2. Each data centre is its own failure domain, and the service moves by routing.
- Give the application its own service range. Only the active site has that VLAN up, so only that site advertises it. Failover means changing which site advertises it.
- Keep the DNS time-to-live at a few minutes as a second safeguard, not the main mechanism.
- Use continuous asynchronous database replication: seconds of lag against a 15-minute RPO, with nothing added to each write. Use synchronous only if the business asks for zero loss and accepts the extra latency.
- Two interconnect paths on physically separate routes, from different carriers, with the routes in writing.
- A witness in a third location, so a broken interconnect never produces two primary databases.
- Active/standby, because the database accepts writes in one place. Use the standby for reporting, testing and a failover practice every quarter.
- Build both sites' firewall and network configuration from one source, and check the standby in every change.
- Runbook order: confirm the primary is really lost and not just cut off, promote the database, start the application, move the route, test one warehouse, then release all of them.
Addressing
| Range | Where | Notes |
|---|---|---|
| 10.50.10.0/24 | WMS service range, VLAN 50 | Vlan50 up at the active site only |
| 10.50.10.20 | WMS service address | The same in both sites, live in one at a time |
| 10.51.0.0/16 | DC1, everything else | Always advertised |
| 10.52.0.0/16 | DC2, everything else | Always advertised |
| 10.53.0.0/30, 10.53.0.4/30 | DCI paths A and B | Routed, no VLANs across |
| wms.example.net | DNS name | A record 10.50.10.20, time-to-live 300 seconds |
Config
DC1-CORE
vlan 50 name WMS-SERVICE interface Vlan50 description WMS service range - up only where WMS is active ip address 10.50.10.1 255.255.255.0 no shutdown ! router bgp 65101 neighbor 10.51.255.2 remote-as 65200 neighbor 10.51.255.2 description SD-WAN hub DC1 address-family ipv4 network 10.50.10.0 mask 255.255.255.0 network 10.51.0.0 mask 255.255.0.0 neighbor 10.51.255.2 activate ! ip route 10.51.0.0 255.255.0.0 Null0
DC2-CORE
vlan 50 name WMS-SERVICE interface Vlan50 description WMS service range - no shutdown only in the failover runbook ip address 10.50.10.1 255.255.255.0 shutdown ! router bgp 65102 neighbor 10.52.255.2 remote-as 65200 neighbor 10.52.255.2 description SD-WAN hub DC2 address-family ipv4 network 10.50.10.0 mask 255.255.255.0 network 10.52.0.0 mask 255.255.0.0 neighbor 10.52.255.2 activate neighbor 10.52.255.2 route-map DC2-OUT out ! ! belt and braces if both sites ever advertise the range at once. ! If your SD-WAN redistributes BGP into its overlay, check the prepend ! survives, or set the preference in the SD-WAN policy instead. ip prefix-list WMS seq 10 permit 10.50.10.0/24 route-map DC2-OUT permit 10 match ip address prefix-list WMS set as-path prepend 65102 65102 route-map DC2-OUT permit 20 ! ip route 10.52.0.0 255.255.0.0 Null0
Verify
show
show interfaces Vlan50 show ip bgp 10.50.10.0 show ip route 10.50.10.0 ! on a warehouse SD-WAN edge show sdwan omp routes 10.50.10.0/24
Prove it before you sign it off
| Test | Expect |
|---|---|
| Fail DCI path A | Replication carries on over path B. Nothing changes at the warehouses. |
| Fail both DCI paths with both sites running | The witness keeps DC1 primary. DC2 does not promote. The warehouses are unaffected. |
| Run the failover runbook in a planned window | The WMS is working from DC2 inside one hour, with replication lag at the moment of failover in seconds. |
| Check a warehouse's route to 10.50.10.0/24 | Via DC2 only. The DNS answer has not changed. |
| Compare the two sites' firewall policies | No differences apart from site-specific objects. |
| Fail back | The same steps in reverse, in a planned window. |
23 / Campus and WAN
Two sites: a three-tier HQ, a collapsed-core branch, two ISPs and GRE over IPsec
Catalyst 9500 core, 9300 distribution, 9200 access, ISR 4000 or Catalyst 8200 edge (IOS XE)
A classic brief: 250 staff at HQ (Sales, IT, HR and executives) and 50 at a branch (Sales and Support), all in 10.10.0.0/16, with three tiers at HQ, a collapsed core at the branch, two ISPs at HQ tracked with IP SLA, GRE over IPsec between the sites, OSPF areas 0, 10 and 20, HSRP, port security and DHCP snooping. Department sizes are assumed (Sales 110, IT 60, HR 50, executives 30; branch Sales 35, Support 15), and every subnet is sized for 20 percent growth plus its gateway addresses.
Routed, OSPF area 0 unless markedPort-channel, layer 2Layer 2 trunkGRE over IPsec
The rules
- Put the gateways and HSRP on the distribution pair, where the access VLANs end, and keep the core routed only. The brief asked for HSRP on the core; in a three-tier design that would stretch every access VLAN through the distribution layer and make it pointless.
- Split the VLANs: DIST-1 is spanning-tree root and HSRP active for 10, 30 and 50, DIST-2 for 20, 40 and 99. Root and active gateway on the same switch for every VLAN.
- Size each subnet from head count plus 20 percent plus its gateway addresses, and keep each site in one block that summarises: HQ LAN 10.10.0.0/22 (area 10), branch 10.10.8.0/24 (area 20), links and loopbacks 10.10.254.0/23. The rest of the /16 stays free.
- OSPF area 0 holds the core, the edge, the distribution uplinks and both tunnels. The ABRs summarise their areas. Use passive-interface default and the same reference bandwidth on every router. The two distribution switches also peer in area 10 over VLAN 99, so the area cannot split behind the summary.
- Track ISP1 with an IP SLA probe to an address beyond the ISP, pinned to ISP1 with a host route. ISP2 is a floating default at distance 250. NAT uses one route-map per ISP, so traffic is translated to the address of the link it actually leaves by.
- Two GRE tunnels protected by IKEv2 and IPsec in transport mode, one from each HQ ISP to the branch, with OSPF cost 10 and 1000. The backup tunnel only comes up once traffic is leaving through ISP2, so expect failover in tens of seconds. If you need both tunnels up all the time, give each ISP its own front-door VRF.
- The branch breaks out to the internet locally. Its static default beats the OSPF default from HQ, so the tunnels carry only company traffic. Branch DHCP is local, so it keeps working when the tunnels are down.
- Access ports: DHCP snooping with only the uplinks trusted and option 82 insertion turned off (the distribution relay would drop it), port security with a maximum of two addresses set to restrict, BPDU guard, and no DTP.
- Set ip mtu 1400 and MSS 1360 on both tunnels, which leaves room for GRE and IPsec.
Addressing
| Subnet | Use | Needed with growth | Gateway and notes |
|---|---|---|---|
| 10.10.0.0/24 | VLAN 10 Sales (HQ) | 132 + 3 | VIP .1, DIST-1 .2, DIST-2 .3 |
| 10.10.1.0/25 | VLAN 20 IT | 72 + 3 | VIP .1, .2, .3 |
| 10.10.1.128/25 | VLAN 30 HR | 60 + 3 | VIP .129, .130, .131 (a /26 would be one short) |
| 10.10.2.0/26 | VLAN 40 Executives | 36 + 3 | VIP .1, .2, .3 |
| 10.10.2.64/27 | VLAN 50 Servers | 24 + 3 | VIP .65; DHCP and DNS at .70 |
| 10.10.2.96/27 | VLAN 99 HQ management | 15 + 3 | VIP .97; area 10 link between the distribution pair |
| 10.10.2.128/25, 10.10.3.0/24 | HQ spare | Inside the area 10 summary | |
| 10.10.8.0/26 | VLAN 110 Branch Sales | 42 + 1 | Gateway .1 on BR-SW1 |
| 10.10.8.64/27 | VLAN 120 Branch Support | 18 + 1 | Gateway .65 |
| 10.10.8.96/29 | VLAN 199 Branch management | 5 + 1 | Gateway .97 |
| 10.10.8.104/30 | BR-R1 to BR-SW1 | 2 | BR-R1 .105, BR-SW1 .106, area 20 |
| 10.10.254.0/30 to .24/30 | HQ point-to-point links | 2 each | EDGE-CORE1 .0, EDGE-CORE2 .4, CORE1-CORE2 .8, CORE1-DIST1 .12, CORE1-DIST2 .16, CORE2-DIST1 .20, CORE2-DIST2 .24 |
| 10.10.254.28/30, .32/30 | Tunnel0 and Tunnel1 | 2 each | HQ-EDGE .29 and .33, BR-R1 .30 and .34 |
| 10.10.255.0/24 | Loopbacks, /32 each | EDGE .1, CORE1 .2, CORE2 .3, DIST1 .4, DIST2 .5, BR-R1 .11, BR-SW1 .12 | |
| 203.0.113.0/30, 198.51.100.0/30 | HQ ISP1 and ISP2 links | ISP .1, HQ-EDGE .2 (documentation addresses) | |
| 192.0.2.0/30 | Branch ISP link | ISP .1, BR-R1 .2 |
Config
DIST-1 (gateway, HSRP, OSPF ABR)
spanning-tree mode rapid-pvst spanning-tree vlan 10,30,50 priority 4096 spanning-tree vlan 20,40,99 priority 8192 ! vlan 10 name SALES vlan 20 name IT vlan 30 name HR vlan 40 name EXEC vlan 50 name SERVERS vlan 99 name MGMT ! interface range TenGigabitEthernet1/1/3 - 4 channel-group 1 mode active interface Port-channel1 description Layer 2 to DIST-2 switchport mode trunk switchport nonegotiate switchport trunk allowed vlan 10,20,30,40,50,99 ! interface Loopback0 ip address 10.10.255.4 255.255.255.255 interface TenGigabitEthernet1/1/1 description CORE-1 no switchport ip address 10.10.254.14 255.255.255.252 ip ospf network point-to-point interface TenGigabitEthernet1/1/2 description CORE-2 no switchport ip address 10.10.254.22 255.255.255.252 ip ospf network point-to-point ! interface Vlan10 ip address 10.10.0.2 255.255.255.0 ip helper-address 10.10.2.70 standby version 2 standby 10 ip 10.10.0.1 standby 10 priority 110 standby 10 preempt delay minimum 60 interface Vlan20 ip address 10.10.1.2 255.255.255.128 ip helper-address 10.10.2.70 standby version 2 standby 20 ip 10.10.1.1 standby 20 preempt delay minimum 60 ! Vlan30 and Vlan50 as Vlan10 (active here) ! Vlan40 and Vlan99 as Vlan20 (standby here) ! router ospf 1 router-id 10.10.255.4 auto-cost reference-bandwidth 100000 passive-interface default no passive-interface TenGigabitEthernet1/1/1 no passive-interface TenGigabitEthernet1/1/2 no passive-interface Vlan99 network 10.10.254.12 0.0.0.3 area 0 network 10.10.254.20 0.0.0.3 area 0 network 10.10.255.4 0.0.0.0 area 0 network 10.10.0.0 0.0.3.255 area 10 area 10 range 10.10.0.0 255.255.252.0
ACC-1 (port security, DHCP snooping)
ip dhcp snooping ip dhcp snooping vlan 10,20,30,40 no ip dhcp snooping information option ! interface range GigabitEthernet1/0/1 - 46 switchport mode access switchport access vlan 10 switchport nonegotiate switchport port-security switchport port-security maximum 2 switchport port-security mac-address sticky switchport port-security violation restrict spanning-tree portfast spanning-tree bpduguard enable ip dhcp snooping limit rate 15 ! interface range TenGigabitEthernet1/1/1 - 2 description Uplinks to DIST-1 and DIST-2 switchport mode trunk switchport nonegotiate switchport trunk allowed vlan 10,20,30,40,99 ip dhcp snooping trust ! interface Vlan99 ip address 10.10.2.110 255.255.255.224 ip default-gateway 10.10.2.97
HQ-EDGE (dual ISP, NAT, GRE over IPsec)
interface Loopback0 ip address 10.10.255.1 255.255.255.255 interface GigabitEthernet0/0/0 description ISP1 ip address 203.0.113.2 255.255.255.252 ip nat outside interface GigabitEthernet0/0/1 description ISP2 ip address 198.51.100.2 255.255.255.252 ip nat outside interface GigabitEthernet0/0/2 description CORE-1 ip address 10.10.254.1 255.255.255.252 ip nat inside ip ospf network point-to-point interface GigabitEthernet0/0/3 description CORE-2 ip address 10.10.254.5 255.255.255.252 ip nat inside ip ospf network point-to-point ! ! --- ISP1 health and the default routes --- ip sla 1 icmp-echo 203.0.113.250 source-interface GigabitEthernet0/0/0 frequency 5 ip sla schedule 1 life forever start-time now track 1 ip sla 1 reachability delay down 10 up 30 ! pin the probe to ISP1; use a stable address beyond the ISP ip route 203.0.113.250 255.255.255.255 203.0.113.1 ip route 0.0.0.0 0.0.0.0 203.0.113.1 track 1 name ISP1 ip route 0.0.0.0 0.0.0.0 198.51.100.1 250 name ISP2-BACKUP ! ! --- translate to the link the traffic leaves by --- ip access-list standard NAT-USERS permit 10.10.0.0 0.0.255.255 route-map NAT-ISP1 permit 10 match ip address NAT-USERS match interface GigabitEthernet0/0/0 route-map NAT-ISP2 permit 10 match ip address NAT-USERS match interface GigabitEthernet0/0/1 ip nat inside source route-map NAT-ISP1 interface GigabitEthernet0/0/0 overload ip nat inside source route-map NAT-ISP2 interface GigabitEthernet0/0/1 overload ! ! --- IKEv2 and IPsec protecting GRE --- crypto ikev2 proposal P-AES256 encryption aes-cbc-256 integrity sha256 group 19 crypto ikev2 policy POL-1 proposal P-AES256 crypto ikev2 keyring KR-BRANCH peer BRANCH address 192.0.2.2 pre-shared-key <long random key> crypto ikev2 profile IKEV2-BRANCH match identity remote address 192.0.2.2 255.255.255.255 authentication remote pre-share authentication local pre-share keyring local KR-BRANCH crypto ipsec transform-set TS-GRE esp-aes 256 esp-sha256-hmac mode transport crypto ipsec profile PROT-GRE set transform-set TS-GRE set ikev2-profile IKEV2-BRANCH ! interface Tunnel0 description BR-R1 via ISP1 (primary) ip address 10.10.254.29 255.255.255.252 ip mtu 1400 ip tcp adjust-mss 1360 ip ospf network point-to-point ip ospf cost 10 tunnel source GigabitEthernet0/0/0 tunnel destination 192.0.2.2 tunnel protection ipsec profile PROT-GRE interface Tunnel1 description BR-R1 via ISP2 (backup) ip address 10.10.254.33 255.255.255.252 ip mtu 1400 ip tcp adjust-mss 1360 ip ospf network point-to-point ip ospf cost 1000 tunnel source GigabitEthernet0/0/1 tunnel destination 192.0.2.2 tunnel protection ipsec profile PROT-GRE ! router ospf 1 router-id 10.10.255.1 auto-cost reference-bandwidth 100000 passive-interface default no passive-interface GigabitEthernet0/0/2 no passive-interface GigabitEthernet0/0/3 no passive-interface Tunnel0 no passive-interface Tunnel1 network 10.10.254.0 0.0.0.7 area 0 network 10.10.254.28 0.0.0.7 area 0 network 10.10.255.1 0.0.0.0 area 0 default-information originate
BR-R1 (branch edge, ABR)
interface Loopback0 ip address 10.10.255.11 255.255.255.255 interface GigabitEthernet0/0/0 description Internet ip address 192.0.2.2 255.255.255.252 ip nat outside interface GigabitEthernet0/0/1 description BR-SW1 ip address 10.10.8.105 255.255.255.252 ip nat inside ip ospf network point-to-point ! local breakout: this static default beats the OSPF default from HQ ip route 0.0.0.0 0.0.0.0 192.0.2.1 ip access-list standard NAT-BRANCH permit 10.10.8.0 0.0.0.255 ip nat inside source list NAT-BRANCH interface GigabitEthernet0/0/0 overload ! ! same proposal, policy and transform-set as HQ-EDGE crypto ikev2 keyring KR-HQ peer HQ-ISP1 address 203.0.113.2 pre-shared-key <long random key> peer HQ-ISP2 address 198.51.100.2 pre-shared-key <long random key> crypto ikev2 profile IKEV2-HQ match identity remote address 203.0.113.2 255.255.255.255 match identity remote address 198.51.100.2 255.255.255.255 authentication remote pre-share authentication local pre-share keyring local KR-HQ crypto ipsec profile PROT-GRE set transform-set TS-GRE set ikev2-profile IKEV2-HQ ! interface Tunnel0 ip address 10.10.254.30 255.255.255.252 ip mtu 1400 ip tcp adjust-mss 1360 ip ospf network point-to-point ip ospf cost 10 tunnel source GigabitEthernet0/0/0 tunnel destination 203.0.113.2 tunnel protection ipsec profile PROT-GRE interface Tunnel1 ip address 10.10.254.34 255.255.255.252 ip mtu 1400 ip tcp adjust-mss 1360 ip ospf network point-to-point ip ospf cost 1000 tunnel source GigabitEthernet0/0/0 tunnel destination 198.51.100.2 tunnel protection ipsec profile PROT-GRE ! router ospf 1 router-id 10.10.255.11 auto-cost reference-bandwidth 100000 passive-interface default no passive-interface Tunnel0 no passive-interface Tunnel1 no passive-interface GigabitEthernet0/0/1 network 10.10.254.28 0.0.0.7 area 0 network 10.10.255.11 0.0.0.0 area 0 network 10.10.8.104 0.0.0.3 area 20 area 20 range 10.10.8.0 255.255.255.0
BR-SW1 (collapsed core, two-member stack)
ip routing vlan 110 name SALES vlan 120 name SUPPORT vlan 199 name MGMT ! interface Loopback0 ip address 10.10.255.12 255.255.255.255 interface GigabitEthernet1/1/1 description BR-R1 no switchport ip address 10.10.8.106 255.255.255.252 ip ospf network point-to-point interface Vlan110 ip address 10.10.8.1 255.255.255.192 interface Vlan120 ip address 10.10.8.65 255.255.255.224 interface Vlan199 ip address 10.10.8.97 255.255.255.248 ! ! local DHCP keeps the branch working without the tunnels ip dhcp excluded-address 10.10.8.1 10.10.8.5 ip dhcp excluded-address 10.10.8.65 10.10.8.69 ip dhcp pool SALES network 10.10.8.0 255.255.255.192 default-router 10.10.8.1 dns-server 10.10.2.70 ip dhcp pool SUPPORT network 10.10.8.64 255.255.255.224 default-router 10.10.8.65 dns-server 10.10.2.70 ip dhcp snooping ip dhcp snooping vlan 110,120 ! access ports as ACC-1 ! router ospf 1 router-id 10.10.255.12 auto-cost reference-bandwidth 100000 passive-interface default no passive-interface GigabitEthernet1/1/1 network 10.10.8.0 0.0.0.255 area 20 network 10.10.255.12 0.0.0.0 area 20
Verify
show
show standby brief show spanning-tree root show ip ospf neighbor show ip route ospf show track 1 show ip sla statistics 1 show crypto ikev2 sa show crypto ipsec sa | include peer|encaps|decaps show ip dhcp snooping binding show port-security
Prove it before you sign it off
| Test | Expect |
|---|---|
| Compare show spanning-tree root and show standby brief on both distribution switches | For every VLAN, the root bridge and the HSRP active router are the same switch. |
| Reload DIST-1 | DIST-2 takes over VLANs 10, 30 and 50 after the hold time. When DIST-1 returns, it takes them back only after the 60-second preempt delay. |
| Block the probe target at ISP1, or unplug ISP1 | Track 1 goes down after 10 seconds. The default moves to ISP2, users are translated to 198.51.100.2, and Tunnel1 comes up. The branch reaches HQ again within a minute. |
| Restore ISP1 | After 30 seconds the default and NAT move back, Tunnel0 returns, and OSPF prefers it again on cost. |
| show ip route on BR-SW1 | One route for 10.10.0.0/22 for the whole HQ LAN, not one per VLAN, plus the default from BR-R1. |
| Plug a rogue DHCP server into an access port | Its offers are dropped. Clients still get addresses from 10.10.2.70. |
| Connect a third device through a small switch on one access port | Port security drops the third address and logs it. The port stays up for the other two. |
Gotchas this design avoids
- Spanning tree root on one switch, HSRP active on the other
- Snooping adds option 82 and the IOS DHCP server throws the request away
- On two ISPs, traffic leaves one link with the other link's address
- A static route stays up when the next hop is dead
- Pings cross the GRE tunnel, file transfers stall
- Sticky MACs vanish after a reload
24 / Data centre
Routing that follows an active/passive firewall pair across two EVPN sites
Nexus 9000 border gateways (NX-OS), EVPN Multi-Site, any active/passive firewall pair (shown with Palo Alto)
Two VXLAN EVPN fabrics joined by EVPN Multi-Site, and one active/passive firewall pair with a member in each data centre, inspecting traffic between the APP and DB VRFs. Hosts in both sites must always send that traffic to whichever firewall is active, and swap sites when the firewall fails over. The trap is that a passive firewall often looks perfectly healthy to the switch beside it.
Routed: DCI and firewall transitFabric linksFirewall HA links
The rules
- The firewall pair is the only path between the APP and DB VRFs. Each border gateway holds static routes in each VRF pointing at the firewall's address on that VRF's transit subnet: in APP, the DB prefixes; in DB, the APP prefixes.
- Track the firewall's data address with an IP SLA probe, not the interface. A passive firewall may keep its links up (Palo Alto's passive link state "auto", or any virtual lab), so the switch sees up/up whichever unit is active. Only the active unit answers on the shared address.
- The transit subnets are local to each border gateway: the same addressing at both sites, never stretched across the DCI and never advertised into EVPN. The probe can then only succeed through the local firewall.
- Redistribute the tracked statics into BGP in each VRF, and from there into EVPN as type 5 routes. The active site advertises them; the passive site's statics are withdrawn, so it learns the same prefixes from the other site over Multi-Site.
- Set route targets by hand in each VRF. The sites use different AS numbers, and automatic route targets include the AS, so they would never match.
- Keep the firewall HA links off the DCI, on their own path, with a backup heartbeat over management. If the DCI and HA links fail together, both firewalls go active and both sites advertise the same routes.
- Delay the track coming back up (30 seconds here) so the routes do not return before the newly active firewall has its sessions and routes in place.
- Expect traffic between sites to cross the DCI twice when a host and the active firewall are in different data centres. Size the DCI for it, or keep each application's tiers in the same site as the active firewall.
Addressing
| Range | Use | Where | Notes |
|---|---|---|---|
| 10.10.0.0/16 | APP VRF hosts (L3 VNI 50010) | Both sites | Anycast gateway on every leaf |
| 10.20.0.0/16 | DB VRF hosts (L3 VNI 50020) | Both sites | Anycast gateway on every leaf |
| 172.16.10.0/29 | APP transit, VLAN 2010 | Local to each BGW | Firewall .1 (follows the active unit), BGW .2 |
| 172.16.20.0/29 | DB transit, VLAN 2020 | Local to each BGW | Firewall .1, BGW .2 |
| 65100, 65200 | Fabric AS numbers | DC1, DC2 | Different AS per site |
| 65000:50010, 65000:50020 | Route targets for APP and DB | Both sites | Set by hand, identical in both sites |
Config
BGW-DC1 (NX-OS)
feature sla sender
!
vlan 2010
name APP-FW-TRANSIT
vlan 2020
name DB-FW-TRANSIT
!
interface Vlan2010
no shutdown
vrf member APP
ip address 172.16.10.2/29
interface Vlan2020
no shutdown
vrf member DB
ip address 172.16.20.2/29
!
! probe the firewall's data address, not the link
ip sla 10
icmp-echo 172.16.10.1 source-interface Vlan2010
vrf APP
frequency 5
ip sla schedule 10 life forever start-time now
ip sla 20
icmp-echo 172.16.20.1 source-interface Vlan2020
vrf DB
frequency 5
ip sla schedule 20 life forever start-time now
track 10 ip sla 10 reachability
delay up 30 down 5
track 20 ip sla 20 reachability
delay up 30 down 5
!
vrf context APP
vni 50010
ip route 10.20.0.0/16 172.16.10.1 track 10
rd auto
address-family ipv4 unicast
route-target both 65000:50010
route-target both 65000:50010 evpn
vrf context DB
vni 50020
ip route 10.10.0.0/16 172.16.20.1 track 20
rd auto
address-family ipv4 unicast
route-target both 65000:50020
route-target both 65000:50020 evpn
!
ip prefix-list FW-DB-ROUTES seq 10 permit 10.20.0.0/16
ip prefix-list FW-APP-ROUTES seq 10 permit 10.10.0.0/16
route-map RM-FW-APP permit 10
match ip address prefix-list FW-DB-ROUTES
route-map RM-FW-DB permit 10
match ip address prefix-list FW-APP-ROUTES
!
router bgp 65100
vrf APP
address-family ipv4 unicast
redistribute static route-map RM-FW-APP
advertise l2vpn evpn
vrf DB
address-family ipv4 unicast
redistribute static route-map RM-FW-DB
advertise l2vpn evpnBGW-DC2 (NX-OS)
! identical to BGW-DC1, including the transit addresses ! and the route targets, except: router bgp 65200 ! the transit VLANs 2010 and 2020 are NOT mapped to VNIs ! and NOT carried across the DCI
FW pair (settings to check)
HA mode: active/passive
HA1 and HA2: dedicated links on their own path, not the DCI
HA1 backup: over the management network
Link monitoring: both data interfaces
Path monitoring: the local BGW transit address (.2) in each zone
Passive link state: shutdown on hardware; in a virtual lab expect
the links to stay up and rely on the probes
Preemption: off, so a recovered unit does not flap traffic
between sites a second timeVerify
show
show track brief show ip sla statistics 10 show ip route vrf APP 10.20.0.0/16 show bgp l2vpn evpn 10.20.0.0 show bgp vrf APP ipv4 unicast 10.20.0.0/16 ! on the firewall pair show high-availability state
Prove it before you sign it off
| Test | Expect |
|---|---|
| Baseline, FW-01 active | BGW-DC1 has static routes in both VRFs, tracks up. BGW-DC2 has its tracks down and learns 10.20.0.0/16 and 10.10.0.0/16 over EVPN from DC1. |
| Ping from an APP host in DC2 to a DB host in DC2 | The traffic crosses the DCI to FW-01 and back. Firewall logs show the session on FW-01. |
| Suspend FW-01 (or fail it over) | Within the probe and down-delay time, BGW-DC1 withdraws its statics and BGW-DC2's tracks come up. Every host now reaches the other VRF through FW-02. |
| Check the route on a DC1 leaf after failover | The APP to DB route now points across Multi-Site to DC2. |
| Restore FW-01 as passive | Nothing moves. With preemption off, FW-02 stays active and the routes stay in DC2. |
| Fail the DCI while both firewalls stay healthy | The HA links on their own path keep one firewall active, and routing in the passive site stops at its own BGW. Record which hosts lose service: that is the cost of splitting the pair across sites. |
25 / WAN
Three-hub internet overlay: IPsec tunnels, BGP route reflectors and BFD
IOS XE: Catalyst 8200/8300 or ISR 4000 at branches and data centre hubs, Catalyst 8000V as the cloud hub
Twenty to a hundred sites on ordinary internet circuits, two data centres and a cloud region, and no appetite for an SD-WAN platform. Every branch has to survive losing any one hub or any one tunnel without anybody noticing, and you want config you can read line by line and generate from a list of sites. This is hub-and-spoke done properly: three hubs, a tunnel from every branch to every hub, BGP choosing the path, and BFD spotting a dead path in about three seconds.
IPsec tunnel (VTI) with iBGP and BFD inside
The rules
- Three hubs in three places: one in each data centre and one in the cloud. Each hub is a BGP route reflector, and the three hubs peer with each other in a full iBGP mesh over their own tunnels.
- Every branch builds a tunnel to every hub and runs iBGP over each one. Losing any hub, or any single tunnel, still leaves the branch two working paths.
- Static IPsec tunnel interfaces (VTIs) with IKEv2, and a /31 per tunnel worked out from the site: third octet is the hub, fourth is twice the branch number. Anyone can work out a tunnel address from a site number, and so can a template.
- Hubs set next-hop-self all towards the branches and towards each other. Plain next-hop-self leaves reflected routes alone, so branches would learn each other's prefixes with next hops they can never reach.
- The branch decides the path. A data centre's own prefixes, tagged with that hub's community, get local preference 200 from that hub. Everything else gets 150 from HUB-1, 140 from HUB-2 and 130 from HUB-3. Traffic for a data centre goes straight to its door; branch-to-branch traffic uses HUB-1 and falls back in order.
- BFD on every tunnel at 1 second x 3, echo off, with BGP timers of 10/30 as a backstop. Internet paths jitter: timers that suit dark fibre flap here and take the BGP session down with them.
- Branches break out to the internet locally and accept only 10.0.0.0/8 from the overlay: never a default route, never a public address. Host routes pin the hub addresses to the internet, so a tunnel can never be routed through itself.
- ip mtu 1400 and adjust-mss 1360 on every tunnel. That covers IPsec with AES-GCM and NAT-T, even on a PPPoE circuit.
- Hubs accept only the branch ranges from a branch and cap each one at 20 prefixes. A mistake at one branch cannot flood the others.
- Each data centre's hub is that data centre's only door into the overlay. If losing the hub router must not cut the data centre off, give it two (see How to grow it).
Addressing
| Range | Used for | Where | Notes |
|---|---|---|---|
| 198.51.100.10, .20 and 203.0.113.30 | Public addresses of HUB-1, HUB-2 and HUB-3 | Internet | HUB-3's is a 1:1 NAT in front of its private address |
| 192.0.2.12 (and so on) | Branch public address | Internet | Must be static for static VTIs |
| 10.1.0.0/16, 10.2.0.0/16 | DC1 and DC2 | Behind HUB-1, HUB-2 | Originated with communities 65000:101 and 65000:102 |
| 10.3.0.0/16 | Cloud VPC | Behind HUB-3 | Community 65000:103 |
| 10.20.N.0/24 | Branch N LAN (up to 127 branches) | Each branch | 10.20.12.0/24 for branch 12 |
| 10.254.0.0/29 | Hub mesh tunnels | Hubs | HUB-1/2 .0/31, HUB-1/3 .2/31, HUB-2/3 .4/31 |
| 10.254.H.(2N)/31 | Tunnel from branch N to hub H | Hub .even, branch .odd | BR-12 to HUB-2 is 10.254.2.24/31 |
| 10.255.0.H/32, 10.255.1.N/32 | Hub and branch loopbacks | Router ID and management | Branch loopbacks are advertised to the hubs only |
| AS 65000 | Whole overlay, iBGP | Everywhere | Hubs are reflectors, branches are clients |
Config
HUB-1 (DC1)
ip bgp-community new-format ! crypto ikev2 proposal OVERLAY encryption aes-gcm-256 prf sha384 group 19 crypto ikev2 policy OVERLAY proposal OVERLAY crypto ikev2 keyring OVERLAY peer HUB-2 address 198.51.100.20 pre-shared-key <unique-key> peer HUB-3 address 203.0.113.30 pre-shared-key <unique-key> peer BR-12 address 192.0.2.12 pre-shared-key <unique-key> crypto ikev2 profile OVERLAY match identity remote address 0.0.0.0 authentication remote pre-share authentication local pre-share keyring local OVERLAY dpd 10 3 on-demand ! crypto ipsec transform-set OVERLAY esp-gcm 256 mode tunnel crypto ipsec profile OVERLAY set transform-set OVERLAY set ikev2-profile OVERLAY ! interface Loopback0 ip address 10.255.0.1 255.255.255.255 interface GigabitEthernet0/0/0 description INTERNET ip address 198.51.100.10 255.255.255.248 interface GigabitEthernet0/0/1 description DC1 core ip address 10.1.255.1 255.255.255.252 ! interface Tunnel902 description HUB-2 (hub mesh) bandwidth 100000 ip address 10.254.0.0 255.255.255.254 ip mtu 1400 ip tcp adjust-mss 1360 bfd interval 1000 min_rx 1000 multiplier 3 no bfd echo tunnel source GigabitEthernet0/0/0 tunnel mode ipsec ipv4 tunnel destination 198.51.100.20 tunnel protection ipsec profile OVERLAY interface Tunnel903 description HUB-3 (hub mesh) bandwidth 100000 ip address 10.254.0.2 255.255.255.254 ip mtu 1400 ip tcp adjust-mss 1360 bfd interval 1000 min_rx 1000 multiplier 3 no bfd echo tunnel source GigabitEthernet0/0/0 tunnel mode ipsec ipv4 tunnel destination 203.0.113.30 tunnel protection ipsec profile OVERLAY interface Tunnel1012 description BR-12 bandwidth 50000 ip address 10.254.1.24 255.255.255.254 ip mtu 1400 ip tcp adjust-mss 1360 bfd interval 1000 min_rx 1000 multiplier 3 no bfd echo tunnel source GigabitEthernet0/0/0 tunnel mode ipsec ipv4 tunnel destination 192.0.2.12 tunnel protection ipsec profile OVERLAY ! ip route 0.0.0.0 0.0.0.0 198.51.100.9 ip route 10.1.0.0 255.255.0.0 10.1.255.2 ! ip prefix-list BRANCH-IN seq 10 permit 10.20.0.0/16 ge 24 le 24 ip prefix-list BRANCH-IN seq 20 permit 10.255.1.0/24 ge 32 route-map FROM-BRANCH permit 10 match ip address prefix-list BRANCH-IN route-map DC1-ORIGIN permit 10 set community 65000:101 ! router bgp 65000 bgp router-id 10.255.0.1 bgp log-neighbor-changes neighbor HUBS peer-group neighbor HUBS remote-as 65000 neighbor HUBS fall-over bfd neighbor HUBS timers 10 30 neighbor BRANCHES peer-group neighbor BRANCHES remote-as 65000 neighbor BRANCHES fall-over bfd neighbor BRANCHES timers 10 30 neighbor 10.254.0.1 peer-group HUBS neighbor 10.254.0.3 peer-group HUBS neighbor 10.254.1.25 peer-group BRANCHES ! address-family ipv4 network 10.1.0.0 mask 255.255.0.0 route-map DC1-ORIGIN neighbor HUBS send-community neighbor HUBS next-hop-self all neighbor BRANCHES route-reflector-client neighbor BRANCHES send-community neighbor BRANCHES next-hop-self all neighbor BRANCHES route-map FROM-BRANCH in neighbor BRANCHES maximum-prefix 20 80 restart 5 neighbor 10.254.0.1 activate neighbor 10.254.0.3 activate neighbor 10.254.1.25 activate exit-address-family
BR-12
ip bgp-community new-format ! ! same IKEv2 proposal, policy, profile and IPsec profile as the hubs, ! with keyring peers HUB-1, HUB-2 and HUB-3 ! interface Loopback0 ip address 10.255.1.12 255.255.255.255 interface GigabitEthernet0/0/0 description INTERNET ip address 192.0.2.12 255.255.255.248 ip nat outside interface GigabitEthernet0/0/1 description LAN ip address 10.20.12.1 255.255.255.0 ip nat inside ! interface Tunnel1 description HUB-1 bandwidth 50000 ip address 10.254.1.25 255.255.255.254 ip mtu 1400 ip tcp adjust-mss 1360 bfd interval 1000 min_rx 1000 multiplier 3 no bfd echo tunnel source GigabitEthernet0/0/0 tunnel mode ipsec ipv4 tunnel destination 198.51.100.10 tunnel protection ipsec profile OVERLAY ! Tunnel2: 10.254.2.25/31 to 198.51.100.20 ! Tunnel3: 10.254.3.25/31 to 203.0.113.30 ! ip route 0.0.0.0 0.0.0.0 192.0.2.9 ! the hubs are always reached over the internet, never the overlay ip route 198.51.100.10 255.255.255.255 192.0.2.9 ip route 198.51.100.20 255.255.255.255 192.0.2.9 ip route 203.0.113.30 255.255.255.255 192.0.2.9 ! ip access-list standard NAT-LAN permit 10.20.12.0 0.0.0.255 ip nat inside source list NAT-LAN interface GigabitEthernet0/0/0 overload ! ip prefix-list OVERLAY-IN seq 10 permit 10.0.0.0/8 le 24 ip prefix-list OWN seq 10 permit 10.20.12.0/24 ip prefix-list OWN seq 20 permit 10.255.1.12/32 ip community-list standard SITE-HUB1 permit 65000:101 ip community-list standard SITE-HUB2 permit 65000:102 ip community-list standard SITE-HUB3 permit 65000:103 ! route-map FROM-HUB1 permit 10 match ip address prefix-list OVERLAY-IN match community SITE-HUB1 set local-preference 200 route-map FROM-HUB1 permit 20 match ip address prefix-list OVERLAY-IN set local-preference 150 route-map FROM-HUB2 permit 10 match ip address prefix-list OVERLAY-IN match community SITE-HUB2 set local-preference 200 route-map FROM-HUB2 permit 20 match ip address prefix-list OVERLAY-IN set local-preference 140 route-map FROM-HUB3 permit 10 match ip address prefix-list OVERLAY-IN match community SITE-HUB3 set local-preference 200 route-map FROM-HUB3 permit 20 match ip address prefix-list OVERLAY-IN set local-preference 130 route-map TO-HUBS permit 10 match ip address prefix-list OWN ! router bgp 65000 bgp router-id 10.255.1.12 bgp log-neighbor-changes neighbor HUBS peer-group neighbor HUBS remote-as 65000 neighbor HUBS fall-over bfd neighbor HUBS timers 10 30 neighbor 10.254.1.24 peer-group HUBS neighbor 10.254.2.24 peer-group HUBS neighbor 10.254.3.24 peer-group HUBS ! address-family ipv4 network 10.20.12.0 mask 255.255.255.0 network 10.255.1.12 mask 255.255.255.255 neighbor 10.254.1.24 activate neighbor 10.254.1.24 route-map FROM-HUB1 in neighbor 10.254.1.24 route-map TO-HUBS out neighbor 10.254.2.24 activate neighbor 10.254.2.24 route-map FROM-HUB2 in neighbor 10.254.2.24 route-map TO-HUBS out neighbor 10.254.3.24 activate neighbor 10.254.3.24 route-map FROM-HUB3 in neighbor 10.254.3.24 route-map TO-HUBS out exit-address-family
HUB-3 (cloud 8000V), differences from HUB-1
interface GigabitEthernet1 description VPC subnet, public 203.0.113.30 by 1:1 NAT ip address 10.3.255.10 255.255.255.240 ! tunnels use GigabitEthernet1 as the source; IKEv2 moves ! to UDP 4500 (NAT-T) on its own because of the cloud NAT ! ip route 0.0.0.0 0.0.0.0 10.3.255.1 ip route 10.3.0.0 255.255.0.0 10.3.255.1 route-map CLOUD-ORIGIN permit 10 set community 65000:103 router bgp 65000 bgp router-id 10.255.0.3 address-family ipv4 network 10.3.0.0 mask 255.255.0.0 route-map CLOUD-ORIGIN ! ! in the cloud console, not on the router: ! - security group: UDP 500 and 4500 in, from the hub and branch addresses only ! - turn off source/destination checking on the 8000V's interface ! - VPC route table: 10.0.0.0/8 to the 8000V's interface
Verify
show
show crypto ikev2 sa show crypto ipsec sa | include interface|encaps|decaps show bfd neighbors show bgp ipv4 unicast summary ! on BR-12: three paths, best via HUB-1 at local-pref 200 show bgp ipv4 unicast 10.1.0.0/16 ! branch to branch: best via HUB-1 at 150, next hop 10.254.1.24 show bgp ipv4 unicast 10.20.13.0/24 ! on a hub: what a branch is sending show bgp ipv4 unicast neighbors 10.254.1.25 routes
Prove it before you sign it off
| Test | Expect |
|---|---|
| Baseline, all up | On every branch: three IKEv2 SAs, three BFD sessions, three BGP sessions. DC1, DC2 and cloud prefixes each go to their own hub at local-pref 200. Other branches go via HUB-1. |
| Pull HUB-1's internet cable during a continuous ping between BR-12 and BR-13 | About 3 seconds of loss. BFD drops the HUB-1 sessions on every branch, and branch-to-branch moves to HUB-2. DC2 and cloud traffic never moves. DC1 is unreachable from the branches: write that down, it is the price of one door. |
| Shut only Tunnel1 on BR-12 | BR-12 reaches DC1 through HUB-2 and across the hub mesh. No other branch notices anything. |
| Add 150 ms of jitter and 2% loss on one branch circuit for an hour (a WAN emulator, or a lab router in the path) | No BFD or BGP flaps. If there are any, the timers are too tight for that circuit. |
| Leave it running past an IPsec rekey (one hour by default) | No BFD or BGP flap in the log during the rekey. |
| Advertise a default route and a 10.20.99.0/24 from a test router behind BR-12 | The hubs drop the default and the off-plan /24. The branches never install a default from the overlay. 21 prefixes from one branch trips maximum-prefix. |
| From a branch PC, ping a DC server with DF set at 1372 bytes, then 1400 | 1372 works, 1400 fails cleanly with a fragmentation message. A large file copy does not stall. |
Failover, second by second
| When | What happens |
|---|---|
| 0 s | HUB-1's internet circuit fails. Anything sent towards HUB-1 is lost. |
| 1 to 3 s | Each branch misses three BFD packets in a row on Tunnel1, as do HUB-2 and HUB-3 on their mesh tunnels to HUB-1. |
| + a few ms | BFD tells BGP. Every session to HUB-1 drops, and every path learnt through it is withdrawn. |
| about 3 s | Branches use the next best path: other branches via HUB-2 at 140. Traffic to DC2 and the cloud never moved. |
| Without BFD | You wait for IPsec dead peer detection or the BGP hold timer: 30 seconds with the timers here, three minutes with the defaults. Traffic is dropped the whole time. |
What it is good for
- Twenty to about a hundred sites on ordinary internet circuits, with a small team and no SD-WAN budget.
- Leaving MPLS without buying a new platform. It runs on the ISR 4000 and Catalyst 8000 routers you probably already have.
- Making the cloud a proper hub, not a VPN bolted onto one data centre.
- Teams that want to read every line of config, and generate it from a list of sites.
When to use something else
- A lot of branch-to-branch traffic, such as site-to-site video. It all goes through a hub. DMVPN phase 3 or SD-WAN builds direct paths.
- Branches on dynamic addresses or behind carrier NAT. Static tunnels need fixed peers (see dynamic VTIs below).
- Well over a hundred sites. Every hub carries one tunnel interface per branch, and that gets unwieldy.
- Choosing a circuit per application by measured loss and delay. That is what SD-WAN is for.
Why this, and not something else
| Option | Choose it when | What it costs you |
|---|---|---|
| This design: static VTIs, BGP, BFD | Fixed branch addresses, up to about 100 sites, you want plain standard config | Branch-to-branch traffic goes through a hub |
| DMVPN phase 3 | Lots of branch-to-branch traffic, or branches on dynamic addresses | mGRE and NHRP to learn and troubleshoot |
| FlexVPN with dynamic VTIs | Many branches, one hub config, dynamic branch addresses | Virtual templates and dynamic BGP neighbours |
| Catalyst SD-WAN | Application-aware paths, central policy, zero-touch builds | Licences, controllers and a new platform to run |
| Carrier MPLS | You need an SLA on latency and loss | Price, lead times and one carrier's reach |
How to grow it
- A second circuit at each branch. Add three more tunnels from the second circuit to the same three hubs, and lower their local preference by 50 so they carry traffic only when the first circuit fails. Put each internet interface in its own front-door VRF (vrf forwarding on the WAN interface, tunnel vrf on the tunnel) so each tunnel always leaves by its own circuit. Do this from day one if a second circuit is likely, because it also takes away any chance of recursive routing.
- Two hubs in a data centre that cannot be cut off. Add HUB-1B on its own circuit, run OSPF or iBGP from both hubs to the data centre core, and give the branches a fourth tunnel. Both DC1 hubs originate 10.1.0.0/16 with the same community.
- Direct paths between branches. Keep the hubs for everything else and add DMVPN phase 3 for branch-to-branch traffic, or give a few large sites direct tunnels and peer them over BGP with a local preference above the hubs.
- Dynamic VTIs on the hubs. Replace the per-branch tunnels with an IKEv2 profile pointing at a virtual template, and use bgp listen range with a peer group so branches can come and go. Branches can then be on dynamic addresses.
- Segmentation. For two or three VRFs, run one tunnel per VRF per hub. Beyond that, run MPLS over the tunnels and use VPNv4 between hubs and branches, so one set of tunnels carries every VRF.
- QoS per branch. Shape each branch tunnel at the hub to that branch's circuit speed with a child LLQ policy, so a 1 Gbps hub cannot flood a 50 Mbps branch. Mark at the branch LAN edge; the tunnel policy sees the inner DSCP.
- IPv6. Switch the tunnels to GRE over IPsec (tunnel mode gre ip with tunnel protection) so they carry both protocols, and add an IPv6 address family to BGP with the same policy.
- Generate it rather than type it. Keep one record per site: number, public address, LAN, circuit speed. A template renders the hub and branch config from it, because every tunnel address and route-map follows from the site number. Check the rendered config into git and diff it against the device before each push.
- Watch it. Send BFD and BGP adjacency changes to your syslog collector and alert on any flap, not just on long outages. Add an IP SLA probe from each branch to each hub loopback to build a history of delay and loss per circuit.
Gotchas this design avoids
- next-hop-self on a route reflector does nothing to reflected routes
- BFD timers copied from the data centre flap across internet tunnels
- A GRE tunnel that flaps every few seconds
- Pings cross the GRE tunnel, file transfers stall
- Your routing protocol thinks the tunnel runs at 100 kbit/s
- maximum-prefix tears the session down and leaves it down