Cisco Gotchas: Config Mistakes and How to Fix Them

The commands that do something other than what you meant. Each one has what you will see, why it happens and the config that stops it happening to you. Written for change windows on IOS, IOS XE and NX-OS.

Platform

Impact

Every gotcha, with the fix

Causes an outageIOS / IOS XE and NX-OS

Adding a VLAN to a trunk removes every other VLAN

What you see: You add VLAN 30 to an uplink and every other VLAN on that trunk goes down at once.
Why: "switchport trunk allowed vlan 30" replaces the allowed list. It does not add to it.
Prevent it: Always use the add or remove keyword, and check the current list first.

show interfaces Gi1/0/49 trunk
interface Gi1/0/49
 switchport trunk allowed vlan add 30

Causes an outageIOS / IOS XE

A spare switch wipes the VLANs on the whole site

What you see: You plug in a switch from the store room and VLANs vanish across the domain. Access ports go inactive.
Why: In VTP version 1 or 2, a switch in server or client mode with the same domain name and a higher configuration revision overwrites everyone else's VLAN database.
Prevent it: Set every new or reused switch to transparent (or off) before it touches the network. On VTP version 3 only the primary server can change VLANs.

show vtp status
vtp mode transparent

Causes an outageIOS / IOS XE

Two switches that should trunk come up as access ports

What you see: The link is up, but only one VLAN passes and "show interfaces trunk" shows nothing.
Why: Many Catalyst models default to "dynamic auto". Two auto ends both wait for the other to ask, so the link forms as access.
Prevent it: Never rely on DTP. Hard-set both ends and turn negotiation off.

interface Gi1/0/49
 switchport mode trunk
 switchport nonegotiate

Quietly wrongIOS / IOS XE and NX-OS

Native VLAN mismatch quietly joins two VLANs

What you see: CDP logs %CDP-4-NATIVE_VLAN_MISMATCH and hosts in one VLAN can reach hosts in another.
Why: Untagged frames land in whatever each end calls its native VLAN, so a mismatch bridges two VLANs together.
Prevent it: Use the same unused native VLAN on both ends of every trunk, and prune it from the allowed list where you can.

show interfaces trunk
interface Gi1/0/49
 switchport trunk native vlan 999

Causes an outageIOS / IOS XE and NX-OS

PortFast everywhere without BPDU Guard invites a loop

What you see: A user plugs in a small switch or bridges two wall ports and the VLAN melts down.
Why: PortFast skips listening and learning. Without BPDU Guard nothing stops a port that receives BPDUs from forwarding straight away.
Prevent it: Pair PortFast with BPDU Guard on every edge port, or set both as defaults.

! IOS 15.2 and IOS XE
spanning-tree portfast edge default
spanning-tree portfast edge bpduguard default
! older IOS: spanning-tree portfast default and portfast bpduguard default
! NX-OS
spanning-tree port type edge default
spanning-tree port type edge bpduguard default

Quietly wrongIOS / IOS XE

Error-disabled ports stay down for ever

What you see: A port tripped BPDU Guard or port security weeks ago and is still down.
Why: Automatic recovery is off by default. The port waits for someone to shut and no shut it.
Prevent it: Turn on recovery for the causes you are happy to retry. The default interval is 300 seconds.

show interfaces status err-disabled
errdisable recovery cause bpduguard
errdisable recovery cause psecure-violation

Quietly wrongIOS / IOS XE and NX-OS

A new SVI stays down after no shutdown

What you see: interface Vlan50 shows down/down even though you typed no shutdown.
Why: Autostate keeps an SVI down until the VLAN exists and at least one port in it is up and forwarding, including a trunk that allows it.
Prevent it: Create the VLAN, then make sure a port or trunk carries it. On NX-OS, also check "feature interface-vlan".

show vlan id 50
show interfaces trunk | include 50

Quietly wrongIOS / IOS XE

"write erase" does not remove the VLANs

What you see: A switch you reset to factory still has old VLANs and a VTP domain.
Why: Catalyst switches keep VLANs and VTP state in vlan.dat, not in the startup configuration.
Prevent it: Delete vlan.dat as well when you wipe a switch for reuse.

write erase
delete flash:vlan.dat
reload

Causes an outageIOS / IOS XE and NX-OS

Changing the spanning-tree mode drops every VLAN briefly

What you see: You move from PVST+ to Rapid PVST+ and the whole switch stops forwarding for a few seconds.
Why: Changing the mode restarts spanning tree on every VLAN at once.
Prevent it: Treat it as an outage. Do it in a window, starting at the edge.

spanning-tree mode rapid-pvst

Quietly wrongIOS / IOS XE

"spanning-tree vlan 10 root primary" is a one-off

What you see: Your chosen root lost the election after someone added a switch.
Why: "root primary" is a macro. It sets a priority once (24576, or lower than the current root) and never adjusts again.
Prevent it: Set explicit priorities on the root and backup root.

spanning-tree vlan 10 priority 4096
! backup root
spanning-tree vlan 10 priority 8192

Quietly wrongIOS / IOS XE

Port security with maximum 1 breaks IP phones

What you see: The phone works, the PC behind it does not, or the port error-disables.
Why: A phone and a PC are two MAC addresses, and some platforms see the phone in both VLANs.
Prevent it: Allow 2 for a phone and PC, or 3 where the phone is learnt in both VLANs.

interface Gi1/0/10
 switchport port-security maximum 3

Quietly wrongIOS / IOS XE

Turning on QoS strips the voice markings

What you see: After "mls qos" on an older Catalyst (2960, 3560, 3750) call quality drops.
Why: Once mls qos is on, every port is untrusted, so incoming DSCP and CoS values are rewritten to 0.
Prevent it: Trust the ports that should keep their markings before you enable QoS.

interface Gi1/0/10
 mls qos trust device cisco-phone
 mls qos trust cos

Causes an outageIOS / IOS XE and NX-OS

Mode "on" at one end and LACP at the other

What you see: The port-channel is up on one switch and down or individual on the other. MAC flapping or a loop follows.
Why: Mode on sends no LACP. The LACP end hears nothing and runs or suspends its ports individually, while the "on" end sends down every link.
Prevent it: Use LACP active on both ends. On Catalyst, EtherChannel guard error-disables the "on" end when it spots this.

interface range Gi1/0/49 - 50
 channel-group 1 mode active
show etherchannel summary

Causes an outageIOS / IOS XE and NX-OS

LACP passive at both ends never forms a bundle

What you see: The port-channel stays down and the member ports show as individual. Spanning tree blocks all but one link, so the new uplink delivers a fraction of its bandwidth.
Why: A passive port only answers LACP. It never starts the negotiation, so two passive ends wait for each other forever.
Prevent it: Set active at both ends, or at least one. Check the flags in show etherchannel summary after every change.

interface range GigabitEthernet1/0/1 - 2
 channel-group 1 mode active
show etherchannel summary
show lacp neighbor

Causes an outageIOS / IOS XE and NX-OS

A VLAN added to one member suspends the port-channel

What you see: Right after adding a VLAN, a member shows (s) and the log says "is not compatible with Po100 and will be suspended (vlan mask is different)". VLANs that only use that trunk go down.
Why: Every member must match the port-channel exactly. Configuring a member on its own makes it different from the bundle, so the switch suspends it.
Prevent it: Configure the port-channel interface, never the members. IOS pushes the change to every member. Remove anything added to a member directly.

interface Port-channel100
 switchport trunk allowed vlan add 104
show etherchannel summary

Causes an outageIOS / IOS XE

BPDU filter on a port switches spanning tree off

What you see: Someone plugs a small switch into two wall ports and the VLAN melts, on a port that had BPDU Guard configured.
Why: spanning-tree bpdufilter enable on an interface stops the port sending and ignoring received BPDUs, so it takes no part in spanning tree at all. It also takes precedence over BPDU Guard on the same port. The global form, spanning-tree portfast bpdufilter default, is different: a PortFast port that hears a BPDU drops back to normal spanning tree.
Prevent it: Remove interface-level BPDU filter from user ports. Use BPDU Guard, and the global filter only where you have a reason.

show running-config | include bpdufilter
show spanning-tree interface Gi1/0/10 detail | include filter|guard
interface Gi1/0/10
 no spanning-tree bpdufilter
 spanning-tree bpduguard enable

Quietly wrongIOS / IOS XE and NX-OS

Root guard and loop guard on the wrong ports

What you see: An uplink goes root-inconsistent and stops forwarding, or a one-way fibre still causes a loop despite the guards.
Why: Root guard belongs on designated ports facing switches that must never become root, usually distribution downlinks. Loop guard belongs on root and alternate ports, the uplinks that receive BPDUs. They cannot run on the same port, and configuring loop guard turns root guard off there.
Prevent it: Root guard on downlinks to access switches, loop guard (or the global default) on uplinks, and UDLD aggressive on fibre.

show spanning-tree inconsistentports
interface Gi1/0/1
 description Downlink to access
 spanning-tree guard root
interface Te1/1/1
 description Uplink to distribution
 spanning-tree guard loop

Quietly wrongIOS / IOS XE and NX-OS

UDLD enabled, and a one-way fibre stays up

What you see: A fibre uplink loses one direction, spanning tree unblocks the other end, and UDLD never shut anything.
Why: In normal mode UDLD marks a port undetermined when it stops hearing its neighbour and leaves it forwarding. Aggressive mode retries and then err-disables the port, which is what stops the loop.
Prevent it: Use aggressive mode on fibre uplinks at both ends, and turn on err-disable recovery for udld if you want it back automatically.

udld aggressive
show udld neighbors
errdisable recovery cause udld

Causes an outageIOS / IOS XE and NX-OS

One MST change splits the region

What you see: After adding a VLAN to an instance on one switch, traffic takes odd paths and ports block on links that were forwarding.
Why: Switches are in the same MST region only if the name, revision number and the whole VLAN-to-instance table match exactly. Change the mapping on one switch and it becomes a separate region, and the boundary changes how every instance is calculated.
Prevent it: Make MST changes identically on every switch in the region in one window, and bump the revision everywhere at the same time.

show spanning-tree mst configuration digest
spanning-tree mst configuration
 name CAMPUS
 revision 7
 instance 1 vlan 10,20,30

Causes an outageIOS / IOS XE

Turning on DAI cuts off every static-IP device

What you see: Printers, servers and sometimes the gateway drop off the VLAN the moment Dynamic ARP Inspection is enabled. A busy port goes err-disabled.
Why: DAI checks ARP against the DHCP snooping table. A device with a static address has no binding, so its ARP is dropped unless an ARP ACL allows it or its port is trusted. Untrusted ports are also rate-limited to 15 ARP packets a second by default, and going over err-disables the port.
Prevent it: Trust uplinks, add ARP ACLs for static devices, raise the rate limit on busy ports, and enable DAI one VLAN at a time.

arp access-list STATIC-HOSTS
 permit ip host 10.10.10.20 mac host 0050.56aa.0020
ip arp inspection filter STATIC-HOSTS vlan 10
ip arp inspection vlan 10
interface Te1/1/1
 ip arp inspection trust
show ip arp inspection statistics vlan 10

Causes an outageIOS / IOS XE

A SPAN session pointed at the wrong port takes it out of service

What you see: Someone sets up a capture and an uplink or a server stops passing traffic.
Why: A SPAN destination port only sends the mirrored traffic. It stops forwarding normal traffic and leaves spanning tree while the session is active, whatever was configured on it before.
Prevent it: Check the destination is a spare port before you press enter, and remove the session when the capture is done.

show monitor session all
monitor session 1 source interface Gi1/0/10 both
monitor session 1 destination interface Gi1/0/47
no monitor session 1

Quietly wrongIOS / IOS XE

Private VLANs and VTP do not mix

What you see: Private VLANs work on the switch you built them on and nowhere else, or the commands are refused.
Why: VTP versions 1 and 2 do not carry private VLAN configuration, so Cisco requires transparent mode for them. VTP version 3 can carry them.
Prevent it: Set VTP transparent (or use version 3) and configure the primary and secondary VLANs on every switch that carries them.

vtp mode transparent
vlan 201
 private-vlan isolated
vlan 200
 private-vlan primary
 private-vlan association 201

Quietly wrongIOS / IOS XE and NX-OS

A 4 x 10G port-channel that carries 10G

What you see: One member link runs hot while the others idle, usually towards a single router, firewall or backup server.
Why: Each flow is pinned to one member by a hash of addresses or ports. If the hash only uses MAC addresses, which some older access switches do by default, everything towards one gateway MAC lands on one link. A single large flow can never use more than one member.
Prevent it: Hash on IP addresses and ports where the platform supports it, and check how the far end hashes its side.

show etherchannel load-balance
port-channel load-balance src-dst-ip
show interfaces port-channel 1 etherchannel

Quietly wrongIOS / IOS XE

Windows says "address conflict 0.0.0.0" after you enable 802.1X

What you see: PCs randomly report a duplicate IP address of 0.0.0.0 and get no address until they reconnect or reboot.
Why: Features such as 802.1X turn on IP device tracking, which sends ARP probes from 0.0.0.0 to check each host. If one lands while Windows is running its own duplicate-address check during DHCP, Windows decides the address is taken.
Prevent it: Delay the probes so they miss the host check, or source them from a real address in the subnet. On newer IOS XE the same settings live in device-tracking policies, so check your release.

ip device tracking probe delay 10
ip device tracking probe auto-source fallback 0.0.0.253 255.255.255.0 override
show ip device tracking all

Quietly wrongIOS / IOS XE

Output drops on a port that is barely used

What you see: A Catalyst 9000 port averages 5% utilisation, yet the output drop counter climbs and users complain about slow transfers.
Why: Drops come from microbursts lasting milliseconds, which 30-second and 5-minute averages never show. Each port only gets a slice of the shared ASIC buffer by default, so a burst towards a slower or busy port overflows its queue.
Prevent it: Confirm which queue drops, then let ports borrow more of the shared buffer. It assumes not every port bursts at once, so change it in a window and watch.

show platform hardware fed switch active qos queue stats interface Gi1/0/48
qos queue-softmax-multiplier 1200

Quietly wrongIOS / IOS XE and NX-OS

Hard-coding speed and duplex at one end only

What you see: The link works but is slow, with late collisions at one end and CRC or runt errors at the other, worst under load.
Why: When one end is fixed at 100 full and the other is left on auto, the auto end can sense the speed but not the duplex, so it falls back to half duplex. At gigabit, auto-negotiation is part of the standard and forcing one end often stops the link coming up at all.
Prevent it: Leave both ends on auto, or fix both ends to the same values. Never just one.

show interfaces Gi1/0/10 | include duplex|collisions|CRC
interface Gi1/0/10
 speed auto
 duplex auto

Quietly wrongIOS / IOS XE

The replacement stack member never joins

What you see: A like-for-like replacement is cabled into the stack, powers up, and show switch lists it as V-Mismatch with none of its ports working.
Why: A stack member must run the same software as the rest of the stack. A spare from the shelf usually has whatever version it shipped with. Unless auto-upgrade is configured, it waits to be upgraded.
Prevent it: Check the spare's version before the visit, or enable auto-upgrade so the stack copies its own software onto the new member and reloads it.

show switch
show version | include Version
software auto-upgrade enable

Causes an outageIOS / IOS XE

DHCP snooping switched on, and nobody gets an address

What you see: Minutes after enabling DHCP snooping, clients in the snooped VLANs stop getting addresses. The DHCP server logs requests and sends offers.
Why: Snooping treats every port as untrusted until told otherwise, and drops server messages (OFFER, ACK) arriving on untrusted ports. That includes your own uplink towards the real server.
Prevent it: Trust the uplinks and any port facing a legitimate server or relay, before you enable snooping on the VLANs. Then check the drop counters.

interface GigabitEthernet1/0/49
 ip dhcp snooping trust
show ip dhcp snooping
show ip dhcp snooping statistics

Causes an outageIOS / IOS XE

Snooping adds option 82 and the IOS DHCP server throws the request away

What you see: With snooping on and the uplink trusted, clients still get nothing. The router serving DHCP sees the DISCOVER but never sends an OFFER, and debug shows relay information with a zero giaddr.
Why: Snooping inserts option 82 by default, with the relay address (giaddr) left at 0.0.0.0 because the switch is not a relay. A Cisco relay or server receiving that on an untrusted interface treats it as inconsistent and drops it.
Prevent it: Stop the access switch inserting option 82 if nothing upstream uses it. Or tell the router the interface is trusted for relay information.

no ip dhcp snooping information option
! or, on the router facing the switch
interface GigabitEthernet0/0.10
 ip dhcp relay information trusted
debug ip dhcp server packet

Quietly wrongIOS / IOS XE

Sticky MACs vanish after a reload

What you see: Port security with sticky learning worked for months. After a power cut, a different laptop on a locked-down port is accepted without complaint.
Why: Sticky addresses are written into the running config, not the startup config. Unless someone saves, a reload forgets them and the port learns whatever plugs in first.
Prevent it: Save the config after the sticky addresses are learned, and check they appear in show running-config.

show port-security address
show running-config interface FastEthernet0/1 | include sticky
copy running-config startup-config

Quietly wrongIOS / IOS XE and NX-OS

Spanning tree root on one switch, HSRP active on the other

What you see: Everything works, but the link between the two distribution switches is busier than either uplink, and one uplink sits idle for a VLAN.
Why: Access switches forward towards the spanning tree root. If the gateway is active on the other distribution switch, every frame to the gateway crosses the inter-switch link first.
Prevent it: For each VLAN, make the same switch both the root and the HSRP active, and the other one the backup for both. Check them together after every change.

show spanning-tree root
show standby brief
show mac address-table address 0000.0c07.ac0a

Causes an outageIOS / IOS XE

Router-on-a-stick subinterfaces down, or on the wrong port

What you see: Every subinterface shows down, or up but nothing reaches the gateways, although the trunk on the switch looks fine.
Why: Subinterfaces inherit the state of their parent, so a parent left shut down takes them all down. A template written for another model can also build them on a port that is not the one cabled.
Prevent it: no shutdown on the parent, and check the subinterfaces sit on the interface that CDP says faces the switch. The dot1Q number must match the VLAN.

show ip interface brief | include \.
show cdp neighbors
show interfaces GigabitEthernet0/0.10 | include Encapsulation

Causes an outageIOS / IOS XE

Stack uplinks on the active switch

What you see: One switch in an access stack dies and the whole closet drops for a minute or more, even though half the uplinks were on another member.
Why: When the active switch fails, the stack has to elect a new active while also losing every uplink on that switch: two failures at once. With the uplinks elsewhere, the standby takes over and the port-channel never loses a member.
Prevent it: Make two members without uplinks the active and standby with stack priorities, and put the uplinks on two other members.

switch 1 priority 15
switch 2 priority 14
! uplinks on members 3 and 4
interface range TenGigabitEthernet3/1/1, TenGigabitEthernet4/1/1
 channel-group 11 mode active
show switch

Quietly wrongIOS / IOS XE and NX-OS

Uplink ratio checked only with everything working

What you see: The access stack is fine for months. The day one uplink switch fails, voice breaks up and wireless crawls in that closet.
Why: A ratio worked out with every uplink up says nothing about the failure case. Lose the member carrying half the uplinks and a stack running at about 7:1 can jump past 10:1, with all the access points still pushing traffic.
Prevent it: Work out the ratio for normal running and for the worst single failure. Spread access points and uplinks evenly across members, and queue voice so it survives short bursts.

show etherchannel summary
show interfaces Port-channel11 | include rate|drops
show platform hardware fed switch active qos queue stats interface TenGigabitEthernet3/1/1

Quietly wrongIOS / IOS XE

"ip default-gateway" does nothing on a layer 3 switch

What you see: The switch cannot reach anything off-subnet even though the default gateway is set.
Why: ip default-gateway is only used when ip routing is off. Once routing is on, the switch uses its routing table.
Prevent it: Add a default route instead.

ip route 0.0.0.0 0.0.0.0 10.0.0.1

Quietly wrongIOS / IOS XE

A static route out of an Ethernet interface with no next hop

What you see: The ARP table grows huge and the CPU climbs.
Why: Without a next hop every destination looks directly connected, so the router ARPs for each one and relies on the far end answering with proxy ARP.
Prevent it: Always give a next hop on multi-access interfaces.

ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0/0 203.0.113.1

Causes an outageIOS / IOS XE and NX-OS

A static route stays up when the next hop is dead

What you see: The ISP router is down, but your link is up, so the default route stays and traffic is black-holed.
Why: A static route only checks that the next hop is reachable, not that it works.
Prevent it: Track the route with IP SLA so it is withdrawn when the far end stops answering.

ip sla 1
 icmp-echo 203.0.113.1 source-interface Gi0/0/0
 frequency 5
ip sla schedule 1 life forever start-time now
track 1 ip sla 1 reachability
ip route 0.0.0.0 0.0.0.0 203.0.113.1 track 1

Quietly wrongIOS / IOS XE and NX-OS

The backup static route wins over the primary

What you see: All traffic uses the backup link even though OSPF or BGP is up.
Why: A floating static needs an administrative distance higher than the route it backs up. eBGP 20, EIGRP 90, OSPF 110, IS-IS 115, iBGP 200.
Prevent it: Pick a distance above the protocol it is backing up.

ip route 0.0.0.0 0.0.0.0 198.51.100.1 250

Causes an outageIOS / IOS XE and NX-OS

OSPF neighbours stuck in EXSTART or EXCHANGE

What you see: The adjacency never reaches FULL and keeps restarting.
Why: The interface MTU differs at each end, so database description packets are rejected.
Prevent it: Match the MTU. ip ospf mtu-ignore is a workaround, not a fix.

show ip ospf neighbor
show interfaces Gi0/0/1 | include MTU

Quietly wrongIOS / IOS XE and NX-OS

OSPF costs disagree between IOS and NX-OS

What you see: Traffic takes a different path in each direction, or a 1G link looks as good as a 100G link.
Why: IOS uses a 100 Mbps reference bandwidth, so 1G, 10G and 100G all cost 1. NX-OS defaults to 40 Gbps. Mixed platforms calculate different costs.
Prevent it: Set the same reference bandwidth on every OSPF router.

router ospf 1
 auto-cost reference-bandwidth 100000

Causes an outageIOS / IOS XE and NX-OS

passive-interface default drops every adjacency

What you see: You add a security best practice and all OSPF neighbours go down.
Why: It makes every interface passive, uplinks included.
Prevent it: Put "no passive-interface" for each uplink in the same change, or better, in the same paste.

router ospf 1
 passive-interface default
 no passive-interface Gi0/0/1

Quietly wrongIOS / IOS XE and NX-OS

A new OSPF router ID does not take effect

What you see: You changed router-id but show ip ospf still shows the old one.
Why: The router ID only changes when the process restarts, which drops every adjacency.
Prevent it: Plan the process restart as a short outage.

router ospf 1
 router-id 10.255.0.1
clear ip ospf process

Quietly wrongIOS / IOS XE

Two OSPF processes learn the same prefix and the first one wins

What you see: During a WAN migration, traffic for a remote site uses the old provider on one switch and the new one on another, and it changes after a reload.
Why: Routes from two OSPF processes both have a distance of 110 and their metrics are not compared, so the route installed first stays until it is withdrawn.
Prevent it: Make the preference deliberate: set a lower distance on the process you want, or filter the prefix inbound on the other.

router ospf 900 vrf SITE_WAN
 distance 109
show ip route vrf SITE_WAN 10.20.0.0

Quietly wrongIOS / IOS XE and NX-OS

A BGP network statement advertises nothing

What you see: The prefix is configured but not in "show ip bgp".
Why: BGP only advertises a network statement when an exact match, prefix and mask, is in the routing table.
Prevent it: Add a matching route, often a static to Null0 for an aggregate.

ip route 198.51.100.0 255.255.255.0 Null0
router bgp 65000
 network 198.51.100.0 mask 255.255.255.0

Causes an outageIOS / IOS XE and NX-OS

maximum-prefix tears the session down and leaves it down

What you see: A peer sends a few too many routes and the session stays idle until someone clears it.
Why: Without restart or warning-only, exceeding the limit shuts the session until a manual clear.
Prevent it: Add a restart interval, or warning-only if you only want the alert.

router bgp 65000
 neighbor 203.0.113.1 maximum-prefix 1000 90 restart 15

Causes an outageIOS / IOS XE and NX-OS

"clear ip bgp *" resets every session

What you see: You wanted a new route-map to apply and dropped every peer at once.
Why: A hard clear tears down the TCP sessions. Policy changes only need a soft refresh.
Prevent it: Clear the one peer, soft, in the direction you changed.

clear ip bgp 203.0.113.1 soft in
clear ip bgp 203.0.113.1 soft out

Causes an outageIOS / IOS XE and NX-OS

Your dual-homed edge becomes a transit network

What you see: Traffic between two providers starts flowing through your internet links, or one provider drops your session for sending them the internet.
Why: Without an outbound filter, routes learned from provider A are advertised to provider B. You are now offering transit, usually over the smallest links either provider has.
Prevent it: Send each provider only your own prefixes, with a prefix list and an AS-path filter that allows only routes you originate.

ip prefix-list OUR-PREFIXES permit 203.0.113.0/24
ip as-path access-list 10 permit ^$
router bgp 64500
 neighbor 192.0.2.1 prefix-list OUR-PREFIXES out
 neighbor 192.0.2.1 filter-list 10 out

Causes an outageIOS / IOS XE and NX-OS

iBGP routes are received but never used

What you see: show ip bgp on the second edge router lists the prefixes, but they are not installed and there is no best path.
Why: eBGP next hops are passed to iBGP peers unchanged. If the provider link subnet is not in your IGP, the next hop is unreachable and BGP ignores the path.
Prevent it: Set next-hop-self on iBGP sessions from the edge routers, rather than carrying provider link subnets in the IGP.

show ip bgp 0.0.0.0
router bgp 64500
 neighbor 10.255.0.2 next-hop-self

Quietly wrongIOS / IOS XE and NX-OS

MED does nothing across two providers

What you see: You set a higher MED towards the backup provider, and inbound traffic still arrives on the backup link.
Why: MED is only compared between paths learned from the same neighbouring AS. With two different providers, the rest of the internet never compares your two MEDs.
Prevent it: Use AS-path prepend on the backup link for inbound preference between providers. Keep MED for two links to the same provider.

route-map TO-ISP-B permit 10
 set as-path prepend 64500 64500 64500
router bgp 64500
 neighbor 198.51.100.1 route-map TO-ISP-B out

Quietly wrongIOS / IOS XE and NX-OS

Local preference does not change inbound traffic

What you see: Outbound traffic moves to the link you wanted, inbound stays exactly where it was.
Why: Local preference stays inside your AS and steers outbound traffic only. Inbound is decided by other networks: AS-path length, MED (same provider only) and, above all, the most specific prefix. A more specific route beats any prepend.
Prevent it: Local preference for outbound, prepend or more-specific announcements for inbound. Never leak a more-specific out of the backup link.

route-map FROM-ISP-A permit 10
 set local-preference 200
route-map TO-ISP-B permit 10
 set as-path prepend 64500 64500 64500

Quietly wrongIOS / IOS XE

eBGP multihop finds a path you did not intend

What you see: A loopback-to-loopback eBGP session stays up after its link fails, now running over some other path.
Why: ebgp-multihop lets the session form over any route to the peer loopback. If that route is learned dynamically, or the static route points at a next-hop address, the session reroutes instead of dropping.
Prevent it: Point the static routes for the peer loopback out of the specific interfaces, use the smallest hop count that works, and prefer direct eBGP where you can.

ip route 192.0.2.250 255.255.255.255 GigabitEthernet0/0/0
ip route 192.0.2.250 255.255.255.255 GigabitEthernet0/0/1
router bgp 64500
 neighbor 192.0.2.250 ebgp-multihop 2
 neighbor 192.0.2.250 update-source Loopback0

Quietly wrongIOS / IOS XE and NX-OS

A private AS number leaks onto the internet

What you see: Your provider or a route collector shows your prefix with 645xx or 65xxx in the AS path, and some networks drop it.
Why: 64512 to 65534 and 4200000000 to 4294967294 are private, like RFC 1918 addresses. Providers that accept a private-AS customer are expected to strip it, and multi-homed sites with their own public AS must strip it from anything they pass on.
Prevent it: Remove private AS numbers on sessions to the internet. In examples and labs, use the documentation ranges 64496 to 64511 and 65536 to 65551.

router bgp 64500
 neighbor 192.0.2.1 remove-private-as
show ip bgp 203.0.113.0

Causes an outageIOS / IOS XE and NX-OS

The edge keeps advertising a default after the internet has gone

What you see: An ISP session drops, but the inside still sends all internet traffic to that edge router, which drops it.
Why: default-information originate always injects a default into OSPF whether or not the router has one. Without always, the default is only advertised while one exists in the routing table, so it follows the BGP session.
Prevent it: Drop always on edge routers that learn their default from BGP. Give the backup edge a worse metric.

router ospf 1
 default-information originate metric 10
show ip ospf database external 0.0.0.0

Quietly wrongIOS / IOS XE

The provider's path beats your own backup link

What you see: Two sites have an MPLS VPN and a direct internal link, and traffic between them goes over the provider even though the internal link is faster.
Why: eBGP routes have an administrative distance of 20, lower than EIGRP (90) or OSPF (110). The same prefix learned over the internal link never gets into the routing table while the eBGP copy exists.
Prevent it: Mark the prefix as a backdoor network on the router that should prefer the IGP. BGP then treats it with distance 200 and does not advertise it.

router bgp 64500
 network 10.20.0.0 mask 255.255.0.0 backdoor
show ip route 10.20.0.0

Quietly wrongIOS / IOS XE and NX-OS

aggregate-address sends the summary and every route inside it

What you see: You add a summary to tidy up what you send a provider or peer, and they now receive the summary plus all the specifics.
Why: On its own, aggregate-address adds the summary and leaves the more-specific routes in place. It also only appears while at least one more-specific route is in the BGP table.
Prevent it: Add summary-only to suppress the specifics, or a suppress-map to keep some. Check what the neighbour actually receives.

router bgp 64500
 aggregate-address 10.20.0.0 255.255.0.0 summary-only
show ip bgp neighbors 192.0.2.1 advertised-routes

Quietly wrongIOS / IOS XE and NX-OS

Every routed link between two switches elects a DR it does not need

What you see: A routed uplink takes about 40 seconds to bring OSPF up after a reload, and the database fills with network LSAs for /30 and /31 links.
Why: OSPF treats Ethernet as a broadcast network even with only two routers on it, so it waits to elect a designated router. Point-to-point skips the wait and the extra LSAs. Both ends must match: a broadcast end and a point-to-point end can reach FULL and still be missing routes.
Prevent it: Set the network type to point-to-point at both ends of every routed link between two devices.

interface TenGigabitEthernet1/1/1
 ip ospf network point-to-point
show ip ospf interface brief

Quietly wrongIOS / IOS XE

OSPF advertises a loopback as /32 whatever its mask

What you see: A loopback configured as a /24 for a service shows up everywhere else as a /32, and the rest of the subnet is unreachable.
Why: OSPF treats a loopback as a host and advertises it as a /32 by default, whatever mask you set.
Prevent it: If the loopback stands for a whole subnet, set its network type to point-to-point so the real mask is advertised.

interface Loopback10
 ip address 10.99.0.1 255.255.255.0
 ip ospf network point-to-point

Quietly wrongIOS / IOS XE and NX-OS

The bandwidth command does not limit anything

What you see: Someone sets bandwidth 10000 on a 1 Gbps link to cap a customer at 10 Mbps, and they still pull a gigabit.
Why: bandwidth is only information for other features: OSPF and EIGRP metrics, QoS percentages and monitoring. Traffic still runs at line rate. It also changes routing costs, which can move traffic you did not mean to.
Prevent it: Shape or police to limit traffic, and use bandwidth only to tell routing and QoS the real circuit speed.

policy-map SHAPE-10M
 class class-default
  shape average 10000000
interface Gi0/0/1
 bandwidth 10000
 service-policy output SHAPE-10M

Quietly wrongIOS / IOS XE and NX-OS

redistribute static leaves the default route behind

What you see: Every static route appears in BGP except the one that matters: 0.0.0.0/0. Downstream sites have no way out.
Why: Redistribution into BGP deliberately skips the default route. It needs default-information originate as well, or a network 0.0.0.0 statement while a default exists in the routing table.
Prevent it: Use network 0.0.0.0 for a default you want everyone to have, or default-originate towards one neighbour only.

router bgp 64500
 address-family ipv4
  redistribute static
  default-information originate
show ip bgp 0.0.0.0

Causes an outageIOS / IOS XE and NX-OS

neighbor default-originate keeps attracting traffic after the uplink dies

What you see: The core-facing link on a PE or edge router fails, but sites still send everything to it, and it drops the lot.
Why: default-originate sends a default to that neighbour whether or not the router can reach anything. It does not even need a default in its own table.
Prevent it: Make it conditional on a route that proves the uplink works, matched exactly with a prefix list.

ip prefix-list CORE-UP permit 10.255.0.8/32
route-map DEFAULT-IF-CORE permit 10
 match ip address prefix-list CORE-UP
router bgp 64500
 neighbor 10.0.7.6 default-originate route-map DEFAULT-IF-CORE

Quietly wrongIOS / IOS XE and NX-OS

Accepting paths prepended twenty times

What you see: Your BGP table carries paths with the same AS repeated over and over, and memory and convergence suffer for no benefit.
Why: More than five or six prepends almost never changes anyone's path choice, but every router that accepts them stores the longer path. A regular expression with a back-reference spots them.
Prevent it: Drop paths where one AS repeats five or more times in a row, on sessions from customers and peers. Test the expression against your own table first.

ip as-path access-list 90 deny _([0-9]+)_\1_\1_\1_\1_
ip as-path access-list 90 permit .*
router bgp 64500
 neighbor 192.0.2.1 filter-list 90 in
show ip bgp quote-regexp "_([0-9]+)_\1_\1_\1_\1_"

Causes an outageIOS / IOS XE

A GRE tunnel that flaps every few seconds

What you see: The tunnel and the routing adjacency over it come up, go down, and come up again. The log shows %TUN-5-RECURDOWN.
Why: The router has learned the route to the tunnel destination through the tunnel itself, usually because the WAN subnet was advertised into the routing protocol running over the tunnel. Reaching the destination now depends on the tunnel being up.
Prevent it: Never advertise the tunnel endpoints into the protocol that runs over the tunnel. A more specific static route to the destination via the physical path also stops it.

show ip route 198.51.100.2
show interfaces Tunnel0 | include line protocol
router ospf 1
 no network 203.0.113.0 0.0.0.3 area 0

Quietly wrongIOS / IOS XE

The GRE tunnel says up/up while the far end is dead

What you see: Monitoring shows the tunnel up, but nothing crosses it, and static routes pointing into it keep attracting traffic.
Why: GRE has no keepalives by default. The tunnel line protocol is up as long as the source interface is up and the destination is routable, which says nothing about the other end.
Prevent it: Turn on keepalives at both ends, or run a routing protocol over the tunnel and route by that rather than by static routes into it.

interface Tunnel0
 keepalive 10 3
show interfaces Tunnel0 | include line protocol|Keepalive

Quietly wrongIOS / IOS XE

Pings cross the GRE tunnel, file transfers stall

What you see: Small packets and the routing protocol work over the tunnel. Web pages half-load, and SMB and SFTP transfers hang.
Why: GRE adds 24 bytes, so a 1500-byte packet no longer fits a 1500-byte path. If the packet has DF set and the ICMP "fragmentation needed" message is filtered, it is dropped silently.
Prevent it: Set the tunnel IP MTU and clamp TCP MSS to match. Add more headroom if IPsec protects the tunnel.

interface Tunnel0
 ip mtu 1476
 ip tcp adjust-mss 1436
ping 10.2.0.10 size 1476 df-bit

Causes an outageIOS / IOS XE

An EIGRP default summary blackholes the hub's own default route

What you see: You add a 0.0.0.0 summary to the hub's tunnel so spokes get only a default. The spokes are fine, but the hub loses its own route to the internet or the core and drops anything it has no specific route for.
Why: A summary installs a discard route to Null0 on the router doing the summarising, with administrative distance 5. For a default summary, that discard route beats the real default the hub learned from BGP (200 for iBGP) or a static.
Prevent it: Give the summary a distance above the real default's. On IOS XE that is summary-metric 0.0.0.0/0 distance 201 under the EIGRP address family. Older IOS took the distance as the last argument of ip summary-address eigrp.

show ip route vrf CUST-A 0.0.0.0
router eigrp 100
 address-family ipv4 vrf CUST-A autonomous-system 100
  summary-metric 0.0.0.0/0 distance 201

Quietly wrongIOS / IOS XE and NX-OS

HSRP never fails back

What you see: The primary router reboots and stays standby. Interface tracking lowers priority but nothing moves.
Why: HSRP preemption is off by default. VRRP preemption is on by default, which catches people moving between the two.
Prevent it: Turn preempt on, with a delay so routing converges first.

interface Vlan10
 standby 10 preempt delay minimum 60

Quietly wrongIOS / IOS XE

Printers and screens vanish when users move to another VLAN

What you see: Staff on the wireless network cannot see the meeting room screen or the printer, which work fine from the wired VLAN they sit on.
Why: Bonjour and other mDNS discovery use link-local multicast (224.0.0.251 and FF02::FB on UDP 5353) with a TTL of 1. It never leaves the VLAN, and no routing or multicast setting changes that.
Prevent it: Run an mDNS gateway on the controller or the distribution switch, and allow only the services each VLAN needs. Do not stretch VLANs across the site just to fix discovery.

mdns-sd gateway
show mdns-sd cache
show mdns-sd service-list

Causes an outageIOS / IOS XE

Sub-second HSRP timers on a chassis with stateful switchover

What you see: A supervisor fails over cleanly, but the gateway moves to the other switch anyway and some traffic drops while it comes back.
Why: Stateful switchover keeps forwarding while the new supervisor takes over the control plane, which takes longer than millisecond hello timers allow. The peer stops hearing hellos and takes the gateway, creating the outage the redundant supervisor was meant to prevent.
Prevent it: Leave HSRP at sensible timers on SSO platforms, or remove the need for a first-hop protocol by making the distribution pair one logical switch.

show standby | include Hello|Active|Standby
interface Vlan10
 standby 10 timers 1 3
show redundancy

Quietly wrongIOS / IOS XE and NX-OS

Provider addresses inside a customer VRF clash with the customer's own

What you see: A new customer VRF goes live and part of the customer's network is unreachable. Their LAN uses the same 10.x or 172.16.x range as your transit links or management loopbacks.
Why: Everyone uses RFC 1918 space first, so provider-side addressing taken from it eventually overlaps a customer.
Prevent it: Take provider-side addresses inside customer VRFs (transit links, management subnets, PPP loopbacks) from 100.64.0.0/10, the shared address space in RFC 6598. It is not routed on the internet and few enterprises use it, but check first: carrier-grade NAT and some VPN products use it too.

show ip route vrf CUST-A 100.64.0.0 255.192.0.0 longer-prefixes

Causes an outageIOS / IOS XE

Changing HSRP version flaps the gateway

What you see: Hosts lose their gateway for a few seconds during a "tidy-up".
Why: Version 1 and version 2 use different virtual MAC addresses and multicast groups, so the group restarts. Both ends must match.
Prevent it: Change both routers in a window. Groups above 255 need version 2.

interface Vlan10
 standby version 2

Security holeIOS / IOS XE

ip helper-address forwards more than DHCP

What you see: DNS, TFTP and NetBIOS broadcasts are forwarded to your DHCP server.
Why: By default it forwards eight UDP services: DHCP/BOOTP 67 and 68, TFTP 69, DNS 53, Time 37, NetBIOS 137 and 138, TACACS 49 and IEN-116 42.
Prevent it: Turn off the ones you do not need.

no ip forward-protocol udp tftp
no ip forward-protocol udp domain
no ip forward-protocol udp netbios-ns
no ip forward-protocol udp netbios-dgm
no ip forward-protocol udp tacacs
no ip forward-protocol udp time
no ip forward-protocol udp nameserver

Causes an outageIOS / IOS XE

NAT works for the first few users, then nobody

What you see: Internet access is "intermittent": some people browse, others cannot, and it changes through the day.
Why: A dynamic NAT pool without overload gives each inside host its own public address. When the pool runs out, new hosts fail until old translations time out.
Prevent it: Add overload, so hosts share addresses by port, or translate to the outside interface address.

show ip nat statistics
ip nat inside source list NAT-INSIDE interface GigabitEthernet0/0/0 overload

Causes an outageIOS / IOS XE

EIGRP authentication fails because of the clock

What you see: Neighbours drop after you add key lifetimes, or never come up on a newly installed router.
Why: accept-lifetime and send-lifetime are checked against the router's clock. A router without NTP thinks it is 1993, so no key is valid.
Prevent it: Get NTP working before you use key lifetimes, and overlap the lifetimes when you roll keys.

show clock detail
show ntp associations
show key chain

Causes an outageIOS / IOS XE

The DMVPN hub config has no multicast map, and works anyway

What you see: You copy a working hub's tunnel config to a new hub or an older router. The spokes register with NHRP, but no EIGRP or OSPF neighbours form over the tunnel.
Why: From IOS XE 16.3, ip nhrp map multicast dynamic is on by default on an mGRE tunnel and is hidden from show running-config. Classic IOS, including IOSv in a lab, still needs it typed. Without it the hub has nowhere to send multicast hellos, so the routing protocol never hears from it.
Prevent it: Put ip nhrp map multicast dynamic in the hub template explicitly. It does no harm where it is already the default, and it documents the intent. On IOS XE, show running-config all shows the hidden default.

show running-config all interface Tunnel10 | include multicast
show ip nhrp multicast
interface Tunnel10
 ip nhrp map multicast dynamic

Causes an outageIOS / IOS XE

IPsec never comes up on a hub with a front-door VRF

What you see: Spokes can ping the hub's public address, but show crypto isakmp sa sits at MM_NO_STATE, and debug on the hub says it has no pre-shared key for the peer.
Why: With tunnel vrf INET, IKE runs in the INET VRF. A keyring or crypto isakmp key defined in the global table does not match a peer in that VRF, so the hub cannot find a key for it.
Prevent it: Define the keyring in the front-door VRF: crypto keyring NAME vrf INET. If each customer has its own tunnel source loopback, give each keyring a local-address so the hub picks the right key for the right tunnel.

show crypto isakmp sa
debug crypto isakmp
crypto keyring CUST-A-DMVPN vrf INET
 local-address Loopback10
 pre-shared-key address 0.0.0.0 0.0.0.0 key <key>

Causes an outageIOS / IOS XE

A broadband spoke drops off the DMVPN after its public address changes

What you see: After the line resyncs or the router reloads, the spoke's tunnel is up but nothing reaches it. Hours later it starts working again by itself.
Why: NHRP registrations are unique by default. The hub keeps the spoke's old public address and refuses a registration from a new one until the old entry expires, and the default hold time is two hours. Broadband with a dynamic address makes this routine.
Prevent it: On every spoke with a dynamic address, add ip nhrp registration no-unique, and bring the hold time down to between 300 and 600 seconds.

show ip nhrp detail
interface Tunnel10
 ip nhrp registration no-unique
 ip nhrp holdtime 600

Causes an outageIOS / IOS XE

Two DMVPN clouds from one WAN interface, and one of them breaks

What you see: A spoke with a primary cloud and a backup cloud, for example a second hub pair or a 4G cloud, brings up one tunnel. The other flaps or carries nothing.
Why: Both mGRE tunnels have the same tunnel source, so IPsec cannot tell which tunnel a packet belongs to unless the protection profile is marked shared. Without it the two sets of crypto sessions trample each other. Each tunnel also needs its own tunnel key.
Prevent it: Use tunnel protection ipsec profile NAME shared on both tunnels, with a different tunnel key on each. Or source each cloud from a different interface.

show crypto socket
show dmvpn
interface Tunnel10
 tunnel key 10
 tunnel protection ipsec profile DMVPN shared
interface Tunnel20
 tunnel key 20
 tunnel protection ipsec profile DMVPN shared

Causes an outageIOS / IOS XE

A DMVPN spoke encrypts fine but never registers

What you see: IKE and IPsec are up and the encrypt counters on the spoke climb, but the hub has no NHRP entry for it and no routing neighbour forms.
Why: The tunnel key travels in the GRE header. If the spoke's key differs from the hub's, the hub decrypts the packet and then drops it, because none of its tunnels uses that key.
Prevent it: Set tunnel key to the hub's value for that cloud. Build spoke configs from the hub's values, not by copying another spoke that may belong to a different cloud.

show interfaces Tunnel10 | include key
show crypto ipsec sa | include encaps|decaps
show dmvpn

Quietly wrongIOS / IOS XE

Branch-to-branch traffic still goes through the hub

What you see: Branches reach each other, but traceroute from one of them always shows the hub's tunnel address in the middle, and every branch-to-branch call crosses the data centre.
Why: DMVPN phase 3 needs both halves: ip nhrp redirect on the hub, which tells the sender there is a better path, and ip nhrp shortcut on the spoke, which acts on it. A spoke without shortcut ignores the redirect, so its half of the conversation keeps going through the hub even when the other spoke's half goes direct.
Prevent it: Add ip nhrp shortcut to every spoke and ip nhrp redirect to the hubs. The hubs can then send the spokes nothing but a default, and NHRP builds the direct paths when they are needed.

show dmvpn detail
show ip route nhrp
traceroute 10.1.2.10 source 10.1.1.1

Security holeIOS / IOS XE

IPsec tunnels still negotiating DH group 5

What you see: The tunnels work. Then an audit flags them, or a new firewall at the far end rejects the proposal.
Why: Group 5 is a 1536-bit key exchange, below the 2048-bit minimum Cisco lists as acceptable. Templates written years ago often still use it for both IKE and PFS.
Prevent it: Move IKE and PFS to group 14 at least, preferably 19 or 20, and change both ends in the same window. For a new build, use IKEv2.

show crypto isakmp policy
show crypto ipsec sa | include PFS|transform
crypto isakmp policy 10
 group 14
crypto ipsec profile DMVPN
 set pfs group14

Causes an outageIOS / IOS XE

"aaa new-model" locks you out of SSH

What you see: The next SSH login asks for a username you never created.
Why: With aaa new-model and no default login list, VTY lines check the local user database. The console stays open, so you only notice remotely.
Prevent it: Create a local user first, then enable AAA, then test from a second session before you log out.

username admin privilege 15 secret <strong>
aaa new-model
aaa authentication login default local

Causes an outageIOS / IOS XE and NX-OS

Local fallback only works when TACACS+ is unreachable

What you see: TACACS+ rejects you and the local account does not work either.
Why: The next method is only tried on an error (no answer). A reject from the server is final.
Prevent it: Always end the list with local, and keep the local password where your team can find it.

aaa authentication login default group TACACS local
aaa authorization exec default group TACACS local if-authenticated

Security holeIOS / IOS XE

Securing "line vty 0 4" leaves lines 5 to 15 open

What you see: An audit finds Telnet still allowed on some VTY lines.
Why: Most current devices have 16 VTY lines. Old templates only configure the first five.
Prevent it: Configure all of them.

show running-config | section line vty
line vty 0 15
 transport input ssh
 access-class MGMT in

Causes an outageIOS / IOS XE

Removing the RSA key turns SSH off

What you see: You regenerate keys and your SSH session drops with no way back in.
Why: "crypto key zeroize rsa" removes the key SSH uses, which disables the SSH server.
Prevent it: Generate the new key over the old one instead, or do it from the console.

crypto key generate rsa general-keys label SSH modulus 3072
ip ssh rsa keypair-name SSH
ip ssh version 2

Causes an outageIOS / IOS XE

A restored config gives you a router with SSH switched off

What you see: A replacement router has the old config restored from backup, but SSH connections are refused and show ip ssh says SSH is disabled.
Why: The RSA key pair is not part of the running or startup config, so no backup carries it. Lines such as ip ssh version 2 fail on restore because there is no key yet.
Prevent it: Generate the keys before you hand the box back, re-apply the SSH lines, and test a login from the jump host. Keep the console cable handy until you have.

crypto key generate rsa general-keys modulus 2048
ip ssh version 2
show ip ssh
show crypto key mypubkey rsa

Causes an outageIOS / IOS XE

The first ACL line blocks everything else

What you see: You apply an ACL name to an interface, nothing changes, then you add the first permit and the link stops passing traffic, including your session.
Why: An ACL that does not exist yet permits everything. The moment it has one line, the implicit deny at the end applies.
Prevent it: Write the complete ACL first, then apply it, under a revert timer.

configure terminal revert timer 5
ip access-list extended EDGE-IN
 permit tcp 10.0.0.0 0.0.255.255 any eq 22
 permit ip any any
interface Gi0/0/0
 ip access-group EDGE-IN in
end
configure confirm

Causes an outageIOS / IOS XE

"no access-list 101 permit ..." deletes the whole list

What you see: You remove one line from a numbered ACL and the whole ACL disappears.
Why: In global configuration, "no access-list 101" followed by anything removes ACL 101.
Prevent it: Edit numbered ACLs by sequence number.

show access-lists 101
ip access-list extended 101
 no 30

Quietly wrongIOS / IOS XE

NTP authentication configured but the clock never syncs

What you see: show ntp associations shows the server but never a sync.
Why: A key is only used once it is listed as trusted.
Prevent it: Add the trusted-key line as well as the key and server.

ntp authenticate
ntp authentication-key 1 hmac-sha2-256 <key>
! older releases offer md5 only
ntp trusted-key 1
ntp server 10.0.0.10 key 1

Security holeIOS / IOS XE

SSH is locked down for IPv4 and wide open over IPv6

What you see: An audit or a scan reaches the router login over IPv6 from somewhere your access class should block.
Why: access-class on the VTY lines only filters IPv4. Once interfaces have IPv6 addresses, the router accepts SSH over IPv6 from anywhere unless the lines also have an IPv6 access class.
Prevent it: Add an IPv6 access class to every VTY line, or do not put IPv6 addresses on management-reachable interfaces until you have one.

ipv6 access-list MGMT-V6
 permit ipv6 2001:db8:100::/64 any
 deny ipv6 any any log
line vty 0 15
 access-class MGMT-V4 in
 ipv6 access-class MGMT-V6 in

Security holeIOS / IOS XE

Command authorisation that skips configuration mode

What you see: TACACS+ command sets block show commands for junior staff, yet they can still type anything in configure terminal or on the console.
Why: IOS does not send configuration-mode commands for authorisation unless aaa authorization config-commands is set, and the console is not authorised unless aaa authorization console is set.
Prevent it: Add both, and test with a restricted account before you log out of the admin session.

aaa authorization config-commands
aaa authorization console
aaa authorization commands 15 default group TACACS local

Causes an outageIOS / IOS XE

"log" on a busy ACL line burns the CPU

What you see: CPU jumps after someone adds log to an ACL for troubleshooting, routing neighbours flap, and the logs scroll too fast to read.
Why: Packets matching a log entry are process-switched by the CPU so the router can write the message. On a busy deny any any log, that is a lot of packets. For IPv6 ACLs every matching packet is process-switched.
Prevent it: Log only where you need it, remove it afterwards, and rate-limit both the process switching and the messages.

ip access-list logging interval 10
logging rate-limit 100 except 4
show processes cpu sorted | exclude 0.00

Causes an outageIOS / IOS XE

802.1X pointed at a TACACS+ server group

What you see: Clients never authenticate, and ISE shows no RADIUS requests from the switch at all.
Why: 802.1X and network authorisation (VLAN and ACL assignment) use RADIUS. TACACS+ is for device administration: who logs in to the switch and which commands they can run. A dot1x method list that names a TACACS+ group has nothing to talk to.
Prevent it: Use separate server groups: RADIUS for dot1x and network authorisation, TACACS+ for login, exec and commands.

aaa group server radius ISE-RADIUS
 server name ISE-1
aaa authentication dot1x default group ISE-RADIUS
aaa authorization network default group ISE-RADIUS
aaa authentication login default group ISE-TACACS local

Security holeIOS / IOS XE

Command accounting is not command authorisation

What you see: The design says ISE controls which commands people can run, yet anyone with level 15 can run anything. ISE just logs it.
Why: aaa accounting commands only records what was typed. Stopping a command takes aaa authorization commands, plus config-commands for configuration mode.
Prevent it: Configure authorisation and accounting, and test with a restricted account before closing your own session.

aaa authorization commands 15 default group ISE-TACACS local
aaa authorization config-commands
aaa accounting commands 15 default start-stop group ISE-TACACS

Quietly wrongIOS / IOS XE

TACACS+, TFTP and syslog ignore the management port

What you see: The management port pings but TACACS+, file copies or syslog time out.
Why: On Catalyst 9000 the management port sits in Mgmt-vrf, and each service needs the VRF and source interface set separately.
Prevent it: Set the VRF and source interface per service.

ip route vrf Mgmt-vrf 0.0.0.0 0.0.0.0 10.0.0.1
aaa group server tacacs+ TACACS
 ip vrf forwarding Mgmt-vrf
 ip tacacs source-interface GigabitEthernet0/0
ip tftp source-interface GigabitEthernet0/0
logging host 10.0.0.20 vrf Mgmt-vrf
logging source-interface GigabitEthernet0/0 vrf Mgmt-vrf

Causes an outageIOS / IOS XE

PPP sessions land in the global table, or drop, when RADIUS sets the VRF

What you see: Broadband sessions authenticate on the LNS but come up outside the customer's VRF, or are torn down as soon as they connect.
Why: The LNS applies the Cisco-AVPair values to the session's virtual-access interface in order. Putting an interface into a VRF removes its IP settings, so a VRF applied after ip:ip-unnumbered wipes what was just set.
Prevent it: Send ip:vrf-id first, then ip:ip-unnumbered. Check the order RADIUS actually sends, not the order the admin screen shows, and test on one user before changing a profile every site uses.

Cisco-AVPair = "ip:vrf-id=CUST-A"
Cisco-AVPair += "ip:ip-unnumbered=Loopback100"
show derived-config interface virtual-access 3.1
show vrf

Causes an outageIOS / IOS XE

FreeRADIUS's default PPP reply breaks sessions on virtual-access sub-interfaces

What you see: A new FreeRADIUS server authenticates PPP users, but on the LNS the sessions fail, or come up on full virtual-access interfaces instead of sub-interfaces.
Why: The default users file adds Framed-Compression = Van-Jacobson-TCP-IP to every PPP reply. Header compression cannot be applied on a virtual-access sub-interface, so the LNS has to build a full interface or refuse the session.
Prevent it: Comment that line out of the DEFAULT Framed-Protocol == PPP entry, restart FreeRADIUS and reconnect a test session.

sudo nano /etc/freeradius/3.0/mods-config/files/authorize
#	Framed-Compression = Van-Jacobson-TCP-IP
sudo systemctl restart freeradius
show vpdn session

Causes an outageIOS / IOS XE

After a power cut, the RADIUS server is up but nobody can log in

What you see: The server is back, the database and the web admin page work, but every authentication fails until someone restarts FreeRADIUS by hand.
Why: FreeRADIUS reads users and clients from SQL when it starts. If it starts before the database is ready, it fails to start and nothing tries again. The hypervisor may not have restarted the VM on its own either.
Prevent it: Make FreeRADIUS wait for the database and retry, set the VM to start automatically with its host, and include a power-off test in the handover.

systemctl status freeradius
sudo systemctl edit freeradius
[Unit]
After=mariadb.service
Requires=mariadb.service
[Service]
Restart=on-failure

Quietly wrongIOS / IOS XE

Two RADIUS servers share one database, but not one log

What you see: A user's login failed. The log on the server you checked shows nothing for that user, so you conclude the request never arrived.
Why: Database replication copies users, profiles and accounting between the pair. The FreeRADIUS log is a plain file on each server, so it only shows the requests that server answered.
Prevent it: Find out which server the NAS used, then read that server's log. Better, send both logs to the same syslog server.

show aaa servers
sudo tail -f /var/log/freeradius/radius.log
sudo freeradius -X   (stop the service first)

Quietly wrongIOS / IOS XE

Nobody can tell who changed what

What you see: Something changed overnight and the only clue is a CONFIG_I message with a username and no detail.
Why: By default IOS logs that configuration happened, not what was typed. The configuration change logger records each command with the user, and can send it to syslog.
Prevent it: Turn on the configuration logger with hidekeys, and send it to your syslog server.

archive
 log config
  logging enable
  logging size 500
  hidekeys
  notify syslog contenttype plaintext
show archive log config all

Quietly wrongIOS / IOS XE and NX-OS

Log timestamps you cannot line up

What you see: Two devices log the same outage seconds or hours apart, or show uptime instead of a time.
Why: Without NTP and datetime timestamps with milliseconds and the time zone, logs from different devices cannot be put in order, which is exactly what you need in an incident.
Prevent it: Sync NTP everywhere and set the same timestamp format on every device.

service timestamps log datetime msec localtime show-timezone
service timestamps debug datetime msec localtime show-timezone
show ntp associations

Quietly wrongIOS / IOS XE

A licence change waits for the next reload, whenever that is

What you see: Weeks after someone changed the licence level, an unrelated reload brings the switch up with different features, or with features missing.
Why: On Catalyst 9000 and other IOS XE platforms, license boot level only takes effect after a reload. Until then the change sits pending, and the next person to reload the box applies it without knowing.
Prevent it: Check for a pending licence change before every reload, and reload in the same window as any licence change.

show version | include license|Next reload
license boot level network-advantage addon dna-advantage

Causes an outageIOS / IOS XE and NX-OS

Copying a config file to running-config merges it

What you see: You restore a "known good" config and the device is a mix of old and new.
Why: copy file running-config adds and overwrites lines. It never removes lines that are not in the file.
Prevent it: Use configure replace on IOS XE, or rollback to a checkpoint on NX-OS.

configure replace flash:known-good.cfg list
! NX-OS
rollback running-config checkpoint pre-change

Quietly wrongIOS / IOS XE

The revert timer needs an archive

What you see: configure terminal revert timer refuses to start, mid-window.
Why: Rollback confirmed change saves a copy of the running config to the archive path. With no archive, there is nothing to roll back to.
Prevent it: Set the archive path as part of your baseline, not on the night.

archive
 path flash:archive
 maximum 14

Causes an outageIOS / IOS XE

An IOS XE upgrade rolls itself back hours later

What you see: The new version was running at the end of the window. In the morning the old one is back.
Why: After install activate, the device waits for install commit. If the auto-abort timer runs out first it rolls back and reloads.
Prevent it: Commit once checks pass, and confirm nothing is left uncommitted.

install commit
show install summary

Quietly wrongIOS / IOS XE

The switch boots in bundle mode after an upgrade

What you see: install commands fail and show version reports BUNDLE.
Why: The boot variable still points at a .bin file rather than packages.conf.
Prevent it: Point the boot variable at packages.conf and save before the reload.

show boot
no boot system
boot system flash:packages.conf
write memory

Causes an outageIOS / IOS XE

A debug locks up the device

What you see: You turn on a debug and the console and SSH stop responding.
Why: Console logging runs at 9600 baud and every message interrupts the CPU. A busy debug floods it.
Prevent it: Turn console logging off, log to the buffer, and set a way out before you start.

no logging console
logging buffered 256000 debugging
reload in 15
! when finished
undebug all
reload cancel

Quietly wrongIOS / IOS XE

"show logging | include CONFIG_I" finds nothing

What you see: You know the config changed but the filter returns no lines.
Why: In IOS regex an underscore means a space or boundary, so CONFIG_I never matches the real text.
Prevent it: Use a dot to match the underscore. The CLI Filter Builder warns about this for you.

show logging | include CONFIG.I

Quietly wrongIOS / IOS XE and NX-OS

Clearing counters before you capture them

What you see: After the change you cannot tell whether errors are new.
Why: Counters are your only before-and-after evidence for errors and drops.
Prevent it: Capture first, then clear, and note the time. Change Checker compares the two captures for you.

show interfaces | include line protocol|errors|drops
clear counters

Causes an outageIOS / IOS XE and NX-OS

The backups were safe. Nobody could reach them

What you see: A change goes badly and you go to restore the config, but the backup server, the cloud storage or the jump host is on the other side of the outage.
Why: Backups kept only on a server or in the cloud depend on the network, DNS and logins that the outage has just taken away. You can never have enough backups, but the ones that count are the ones you can reach with everything down.
Prevent it: Before every window, copy the configs (and the images, for an upgrade) to your own laptop and to the device's own flash. Then go through the what-ifs, even the ones that sound stupid, and write down how you would get back in.

copy running-config flash:CHG0012345-pre.cfg
dir flash: | include CHG0012345
! NX-OS
copy running-config bootflash:CHG0012345-pre.cfg

Causes an outageIOS / IOS XE and NX-OS

Nobody has called the vendor, and the MD has just asked

What you see: Hours into an outage, the MD walks in and asks what Cisco have to say about it. Everyone goes quiet, because nobody has opened a case.
Why: Engineers put off calling support because it feels like admitting defeat, and then lose hours that a TAC engineer could have saved. When the call does happen, it starts at the back of the queue, often at 2am.
Prevent it: Call as soon as you are stuck; forget being the hero. For a change where an outage is guaranteed, open the case before the window with the change reference and time, and make it a step in the implementation plan. If cost is questioned, compare an engineer on the line straight away with a support queue at 2am. Capture the show tech early, because it is the first thing they will ask for.

show tech-support | redirect flash:tac-CHG0012345.txt
! NX-OS
show tech-support details > bootflash:tac-CHG0012345.txt

Quietly wrongIOS / IOS XE and NX-OS

The outage that was only going to be a few seconds

What you see: The change request said a few seconds, or no outage at all. It took twenty minutes, nobody had been warned, and the post-incident review is about your estimate rather than the change.
Why: Reloads, reconvergence, ARP and MAC relearning, and the one device that does not come back cleanly all add time that never appears in the lab. Optimistic estimates feel professional until they are wrong in front of everyone.
Prevent it: Estimate the outage, then double it, and write the doubled figure in the MOP and the change request. Finishing early never upset anyone. Overrunning a time you promised is a second incident.

Quietly wrongIOS / IOS XE and NX-OS

[^255] does not mean "not 255"

What you see: A filter meant to catch reliability below 255/255 never matches, so poor links look healthy.
Why: Square brackets match one character. [^255] means one character that is not a 2 or a 5, and reliability is always three digits over 255.
Prevent it: Write the numbers you want as a range. The CLI Filter Builder writes ranges for you.

show interfaces | include line protocol|reliability ([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-4])/255

Quietly wrongIOS / IOS XE and NX-OS

"[^0] CRC" misses 10, 20 and 100 errors

What you see: An interface with 10 or 20 CRC errors does not show up in your error filter.
Why: [^0] checks only the last digit before the space. Any count ending in 0 fails, and those are often the ones that matter.
Prevent it: Match a number that does not start with 0: a boundary, then 1 to 9, then any digits.

show interfaces | include line protocol|_[1-9][0-9]* (input errors|CRC|late collision)

Quietly wrongIOS / IOS XE

A second "| exclude" excludes nothing

What you see: "show run | include interface | exclude Vlan" runs without an error, and the Vlan lines are still there.
Why: IOS and IOS XE take one filter per command on most platforms. Everything after the first keyword is the regex, so "| exclude Vlan" becomes "or the text exclude Vlan".
Prevent it: Put every match into one include, or anchor it so the unwanted lines cannot match. NX-OS can chain filters.

show running-config | include ^interface (Gi|Te)
! NX-OS
show running-config | include ^interface | exclude Vlan

Quietly wrongIOS / IOS XE

Spaces inside a filter are part of the match

What you see: A filter copied from someone's notes matches less, or more, than it should.
Why: IOS drops the spaces between the pipe keyword and the regex, but every space after that is literal. "a | b" means "a " or " b".
Prevent it: Leave spaces out of alternatives. Where you need to match a gap of any width, use " +".

show vlan brief | include active +[A-Z]

Quietly wrongIOS / IOS XE and NX-OS

Filters only ever see one line

What you see: A regex that matches an interface, its description and its VLAN together never returns anything on the device.
Why: include, exclude and begin test one line at a time. \r, \n and multi-line patterns only work in an editor or a script.
Prevent it: Use section. It prints the whole block, header and children, when any line in it matches.

show running-config | section access vlan 30$

Quietly wrongIOS / IOS XE and NX-OS

_65[100-199]_ does not match AS 65100 to 65199

What you see: An AS-path filter meant for a block of private AS numbers lets the wrong routes through, or none.
Why: Square brackets match one character. [100-199] is the set 0, 1 and 9, so the expression matches AS 650, 651 and 659.
Prevent it: Write the digits position by position, and prove it against the BGP table before it goes into a policy.

ip as-path access-list 10 permit _651[0-9][0-9]_
show ip bgp regexp _651[0-9][0-9]_

Quietly wrongPhysical layer

Cheap "Cat 6" that is not copper

What you see: Links fail insertion loss, conductors snap at the jack, PoE devices reboot or run hot.
Why: Copper-clad aluminium (CCA) cable is sold with category markings, but no category standard allows it. Aluminium has higher resistance and breaks when flexed.
Prevent it: Buy from a known supplier, ask for the CPR declaration of performance, and scrape a conductor on any reel that seems light or cheap.

Quietly wrongPhysical layer

A split pair passes the cheap tester

What you see: The continuity tester shows all eight lights in order, but the link errors or only runs at 100 Mb/s.
Why: Each pin is wired to the right pin, but the signal pairs are made from wires of two different twisted pairs. Continuity is fine, and crosstalk is terrible.
Prevent it: Certify with a proper field tester, which fails it on NEXT. Re-terminate both ends with one colour code.

Quietly wrongPhysical layer

Green and blue fibre connectors do not mix

What you see: High loss and high reflectance on a single-mode link, sometimes after someone borrowed a patch cord.
Why: APC connectors (green) are polished at an angle and UPC (blue) are flat. Mated together they leave an air gap, and the end faces can be damaged.
Prevent it: Match the connector colour at every mating point, and inspect any end face that has been mated wrongly.

Causes an outagePhysical layer

Most fibre faults are a dirty connector

What you see: A new link has high loss, errors under load, or works on one strand and not the other.
Why: A speck of dust on a 9 or 50 micron core blocks light, and plugging a dirty connector in transfers the dirt to the other side.
Prevent it: Inspect, clean, inspect again, then connect, every time, including brand new patch cords.

Quietly wrongPhysical layer

A link that runs 10G fails at 40G or 100G

What you see: An upgrade to SR4 optics on the same fibre gives errors or no link, though the run is short.
Why: SR4 budgets are 1.5 to 1.9 dB. Two standard-grade connector pairs can use 1.5 dB on their own, before any fibre.
Prevent it: Run the loss budget before ordering optics. Use low-loss connectors and cords, or remove a patch point.

Causes an outageIOS / IOS XE and NX-OS

Moving uplinks from 10G to 25G on the same multimode

What you see: The new 25G optics link up on the bench but take errors or stay down on the real run between the main and floor comms rooms.
Why: Multimode reach falls as speed rises. At 25G, short-reach optics typically need RS-FEC to reach about 70 to 100 m on OM3/OM4, some extended modules stretch further, and without FEC the reach can drop to tens of metres. OM1 may not carry more than 10G at all. Patch leads count towards the loss.
Prevent it: Measure the run end to end, check the optic and fibre grade against the data sheet, and make FEC match at both ends. Dual-rate 10/25G optics let you move one end first.

show interfaces transceiver detail
show interfaces TwentyFiveGigE1/1/1 | include FEC|errors
interface TwentyFiveGigE1/1/1
 fec cl108

Quietly wrongPhysical layer

The 100 m limit includes the patch cords

What you see: A link passes the permanent link test but errors once it is patched, or runs at a lower speed.
Why: The channel is 100 m at most: 90 m of fixed cable plus 10 m of cords in total. Long cords at both ends push it over.
Prevent it: Keep fixed runs under 90 m, and cords short. Test the channel as well as the permanent link when in doubt.

Security holePhysical layer

Plastic ties alone fail the fire rule

What you see: An inspection fails a ceiling full of data cable hung on plastic ties or in plastic trunking.
Why: BS 7671 regulation 521.10.202 requires wiring systems to be supported so they will not collapse prematurely in a fire. Most cabling specifications hold data cable on the same route to the same rule.
Prevent it: Use metal fixings or ties at intervals, or fire-resistant supports, wherever cable runs at high level.

Causes an outageIOS / IOS XE

The DHCP pool exists, but nobody gets an address

What you see: Clients on one subnet get nothing. show ip dhcp binding has no leases from that range.
Why: An IOS DHCP server picks the pool whose network statement contains the address the request arrived on: the receiving interface for local clients, or the relay's giaddr for relayed ones. With a typo in the network line no pool matches, and the request is dropped with nothing in the log unless you are debugging.
Prevent it: Make the pool network match the client subnet exactly, and check the relay SVI address sits inside it.

show ip dhcp pool
debug ip dhcp server packet
ip dhcp pool USERS
 network 192.168.10.0 255.255.255.0

Quietly wrongIOS / IOS XE

DHCP exclusions are not part of the pool

What you see: A pool copied to a new router starts handing out the gateway or printer addresses, or show ip dhcp conflict fills up.
Why: ip dhcp excluded-address is a global command. It applies to every pool on the router and does not travel with the pool, so a template that copies only the pool loses them. IOS pings each address before offering it and records a conflict if something answers, which is often the first clue.
Prevent it: Keep the exclusion lines next to the pool in your templates and MOPs, and check them after any rebuild.

show running-config | include excluded-address
show ip dhcp conflict
ip dhcp excluded-address 192.168.10.1 192.168.10.20

Quietly wrongIOS / IOS XE

Unicast flooding that looks like a broadcast storm

What you see: Every port in a VLAN sees traffic meant for one server. Access ports run busy, usually where two gateways share the load.
Why: Routers keep ARP entries for four hours by default; switches age MAC entries after five minutes. With asymmetric routing, the router sending to a host still has the ARP entry, but the switch has forgotten which port the host is on, so it floods every frame to the whole VLAN.
Prevent it: Make the MAC aging time at least as long as the ARP timeout, or bring the ARP timeout down on the SVIs. Better still, keep the return path symmetric.

mac address-table aging-time 14400
! or, on each SVI
interface Vlan10
 arp timeout 300

Causes an outageIOS / IOS XE and NX-OS

Interface tracking that never moves the gateway

What you see: The active router's uplink goes down, its priority drops, and it stays active.
Why: The default decrement is 10. A router on 110 drops to 100, level with a standby on 100, and an equal priority is not enough to take over. The standby also needs preempt, which is off by default.
Prevent it: Decrement by enough to fall below the standby, and configure preempt on both routers.

track 1 interface GigabitEthernet0/0/0 line-protocol
interface Vlan10
 standby 10 priority 110
 standby 10 preempt delay minimum 60
 standby 10 track 1 decrement 20

Quietly wrongIOS / IOS XE

GLBP has two jobs, and preempt is different for each

What you see: After a reboot the primary forwards traffic again, but a different router is still answering ARP for the gateway.
Why: The active virtual gateway (AVG) answers ARP and hands out virtual MACs; up to four active virtual forwarders (AVFs) carry the traffic. AVG preemption is off by default, while forwarder preemption is on with a 30-second delay. Priority elects the AVG; weighting sets each router's share of forwarding.
Prevent it: Set glbp preempt on the router you want as AVG, and read show glbp for both roles, not just the first line.

interface Vlan20
 glbp 20 ip 192.168.20.1
 glbp 20 priority 110
 glbp 20 preempt
show glbp brief

Causes an outageIOS / IOS XE

The NAT ACL is written back to front

What you see: NAT is configured, the interfaces are marked, and show ip nat translations stays empty.
Why: The NAT ACL is checked against packets travelling from inside to outside, so the inside subnet must be the source. An extended ACL with the inside subnet as the destination never matches. A deny line means do not translate, which is how VPN traffic is exempted.
Prevent it: Put the inside subnet first, then watch the ACL hit counts climb while a client tests.

ip access-list extended NAT-INSIDE
 deny ip 192.168.10.0 0.0.0.255 10.0.0.0 0.255.255.255
 permit ip 192.168.10.0 0.0.0.255 any
show access-lists NAT-INSIDE

Causes an outageIOS / IOS XE

Translations appear, but nothing comes back

What you see: show ip nat translations fills up and the hit counts rise, yet the far end never replies.
Why: The translated address has to be routed back to the router that did the translation. A pool from a range that is not on the outside interface needs a route from upstream, and with two edge routers the reply can come back through the one that holds no translation.
Prevent it: Check that the provider routes the pool to you, or translate to the outside interface address. Keep each flow on one edge router.

show ip nat translations verbose
show ip nat statistics
! on the upstream router
ip route 203.0.113.64 255.255.255.248 203.0.113.1

Quietly wrongIOS / IOS XE

Someone else is translating your traffic

What you see: A server log or firewall shows a source address you never configured, or an ACL keyed on client subnets stops matching.
Why: NAT on any router in the path changes what everything beyond it sees. A forgotten ip nat rule on an intermediate router does it quietly, and the hop that sees the wrong address is not the hop that changed it.
Prevent it: Check for translations on every router in the path, not just the edge, and follow the source address hop by hop.

show ip nat translations
show running-config | include ip nat|^interface

Quietly wrongIOS / IOS XE

Testing policy routing with a ping from the router

What you see: You ping from the router, the traffic ignores the route-map, and you decide PBR is broken.
Why: ip policy route-map on an interface acts only on packets arriving on that interface. Traffic the router generates itself is not policy-routed unless you add ip local policy route-map.
Prevent it: Test from a host behind the interface the policy is on, or use local policy deliberately.

show ip policy
show route-map
ip local policy route-map LOCAL-PBR

Quietly wrongIOS / IOS XE

set ip next-hop and set ip default next-hop are not the same

What you see: Policy routing works in the lab and does nothing in production, or keeps sending traffic somewhere after you expected it to fall back.
Why: set ip next-hop overrides the routing table whenever the next hop is reachable. set ip default next-hop is used only when there is no explicit route to the destination, and a default route does not count as one. If the next hop disappears, set ip next-hop falls back to normal routing.
Prevent it: Pick the one that matches the intent, and track next hops that are not directly connected.

route-map PBR permit 10
 match ip address GUEST
 set ip next-hop verify-availability 10.20.0.1 1 track 10
show route-map PBR

Quietly wrongIOS / IOS XE and NX-OS

The router cannot ping a host that can ping it

What you see: The server pings the router interface. From the router, the same ping to the server fails.
Why: If the interface is in a VRF, a plain ping uses the global routing table, which has no route to the server. The server's pings work because they arrive on the VRF interface and the replies use the VRF table.
Prevent it: Ping and traceroute inside the VRF, sourced from the right interface.

show vrf
ping vrf BRANCH 10.9.9.100 source GigabitEthernet0/0/1
traceroute vrf BRANCH 10.9.9.100

Causes an outageIOS / IOS XE

Hosts only get a link-local address

What you see: The VLAN interface has a global IPv6 address and pings from the router work, but PCs on the VLAN never get an address from SLAAC and cannot leave the subnet.
Why: Without "ipv6 unicast-routing", IOS treats itself as an IPv6 host: it does not forward IPv6 and does not send router advertisements, so hosts have no prefix and no default gateway.
Prevent it: Turn on IPv6 routing once per device, then check the interface is advertising.

ipv6 unicast-routing
show ipv6 interface Vlan10

Causes an outageIOS / IOS XE and NX-OS

A LAN prefix longer than /64 breaks SLAAC

What you see: Someone "saved addresses" by giving a user VLAN a /80 or /96. Hosts that rely on SLAAC, which includes every Android phone, never get a global address.
Why: SLAAC builds a 64-bit interface ID, so it only works when the prefix is exactly /64. Android does not support DHCPv6 for addresses at all.
Prevent it: Use /64 for every LAN and VLAN, however few hosts it has. Save the longer prefixes for point-to-point links, where /127 is the norm.

interface Vlan10
 ipv6 address 2001:db8:acad:10::1/64

Causes an outageIOS / IOS XE

Your own "deny ipv6 any any" breaks neighbour discovery

What you see: An inbound IPv6 ACL with an explicit deny at the end works for a few minutes after it is applied, then the neighbour on that link stops responding.
Why: IPv6 ACLs end with hidden lines that permit neighbour solicitations and advertisements before the final deny. Your explicit deny comes before those hidden lines, so ND is dropped and the neighbour entry ages out.
Prevent it: Permit nd-ns and nd-na above your own deny. On an interface that must receive router advertisements, permit router-advertisement too.

ipv6 access-list EDGE-IN
 permit icmp any any nd-ns
 permit icmp any any nd-na
 deny ipv6 any any log

Causes an outageIOS / IOS XE and NX-OS

Blocking all ICMPv6 breaks large transfers

What you see: Pings and small web pages work over IPv6, but downloads, VPNs and some sites hang part way through.
Why: IPv6 routers never fragment. A router with a smaller MTU drops the packet and sends ICMPv6 "packet too big" back to the sender. If a firewall or ACL drops that message, the sender never shrinks its packets.
Prevent it: Always permit ICMPv6 packet-too-big through ACLs and firewalls, in both directions.

ipv6 access-list EDGE-IN
 permit icmp any any packet-too-big

Quietly wrongIOS / IOS XE

A static route to a link-local next hop is refused

What you see: You point a route at the neighbour's fe80:: address and IOS rejects the line.
Why: Every interface has a link-local address in the same fe80::/10 range, so the address alone does not say which link to use.
Prevent it: Name the exit interface as well as the link-local next hop. Setting simple link-local addresses such as fe80::1 and fe80::2 makes these routes, and the routing tables, far easier to read.

interface GigabitEthernet0/0
 ipv6 address fe80::1 link-local
ipv6 route 2001:db8:acad:2::/64 GigabitEthernet0/0 fe80::2

Causes an outageIOS / IOS XE and NX-OS

OSPFv3 will not start on an IPv6-only router

What you see: You enable OSPFv3 on a router with no IPv4 addresses and no neighbours ever form. The log complains that there is no router ID.
Why: OSPFv3 still uses a 32-bit router ID, normally taken from an IPv4 address. With no IPv4 address on the box, there is nothing to take it from.
Prevent it: Set the router ID by hand on every OSPFv3 router. It is good practice even when IPv4 exists.

ipv6 router ospf 1
 router-id 10.255.0.1

Quietly wrongIOS / IOS XE

A 2960 rejects IPv6 commands until you change the template and reload

What you see: ipv6 address on the switch SVI is refused or does nothing, and IPv6 management never works.
Why: On many Catalyst access models, IPv6 needs a different SDM template, and a template change only takes effect after a reload. LAN Lite models do not support it at all.
Prevent it: Check the current template, set the dual-stack or routing template, and plan the reload into the change.

show sdm prefer
sdm prefer dual-ipv4-and-ipv6 default
! or on newer releases
sdm prefer lanbase-routing
reload

Quietly wrongIOS / IOS XE

A DHCPv6 address pool that hands out no addresses

What you see: The DHCPv6 pool has an address prefix, but hosts still build their own SLAAC addresses and show ipv6 dhcp binding stays empty.
Why: Hosts follow the router advertisement. The O flag (other-config) means "get DNS from DHCPv6, build your own address". Only the M flag (managed-config) tells them to request an address, and even then the prefix still allows SLAAC unless you turn autoconfig off.
Prevent it: Decide stateless or stateful. For stateful, set the M flag and stop advertising the prefix for SLAAC.

interface GigabitEthernet0/0.10
 ipv6 nd managed-config-flag
 ipv6 nd prefix default no-autoconfig
 ipv6 dhcp server VLAN10-V6
show ipv6 dhcp binding

Quietly wrongIOS / IOS XE

Every access point left on the default site tag

What you see: Clients roam badly and coverage hole alerts look random, on a controller that is nowhere near its limits.
Why: Current controllers split their work across several internal processes, and site tags decide which process handles each access point. On the default tag they are spread round robin, so neighbouring access points land in different processes, and features that need neighbours together (802.11k and v roaming help, coverage hole detection) lose sight of each other.
Prevent it: Create a custom site tag per roaming area, such as a building, keep each under about 400 access points, and never leave production access points on the default.

wireless tag site BLDG-A
 description Building A
ap 00a2.eeb1.2c34
 site-tag BLDG-A
show ap tag summary

Quietly wrongIOS / IOS XE

Branch calls drop when people walk to the other end of the building

What you see: At a branch using local switching, Wi-Fi calls drop and laptops re-authenticate when users move between two parts of the office.
Why: Access points in local-switching (FlexConnect) mode only share client keys within their site tag. Crossing to an access point in another site tag forces a full re-authentication. Seamless roaming within one tag also has a size limit, around 100 access points.
Prevent it: Put every access point a user can roam between in the same site tag, and split only where nobody walks between the areas.

wireless tag site BRANCH-12
 no local-site
 flex-profile BRANCH-FLEX
show wireless tag site detailed BRANCH-12

Causes an outageIOS / IOS XE

The standby controller and the active controller on one switch each

What you see: A distribution switch is rebooted for maintenance and every wireless client drops while the controllers swap roles.
Why: If the active controller is cabled only to one switch of the distribution pair, losing that switch also takes the active controller off the network, so the pair has to fail over. Spread across both switches, the active keeps half its links and nothing moves.
Prevent it: Bundle each controller's ports into one port-channel with members on both switches of the virtual pair or stack, for both the active and the standby.

interface range TenGigabitEthernet1/0/20, TenGigabitEthernet2/0/20
 description WLC-1 LAG
 switchport mode trunk
 channel-group 20 mode active
show etherchannel 20 summary

Quietly wrongIOS / IOS XE

80 or 160 MHz channels everywhere for speed

What you see: Single-client speed tests look great, but with the office full, throughput drops and retries climb on every access point.
Why: Wide channels bond several 20 MHz channels into one, leaving far fewer separate channels to hand out. In a dense deployment neighbouring access points end up on the same spectrum and spend their time waiting for each other.
Prevent it: Let the controller choose width per access point from the clients it sees (dynamic bandwidth selection), and use 20 or 40 MHz in dense areas.

ap dot11 5ghz rrm channel dca chan-width best
show ap dot11 5ghz channel

Quietly wrongIOS / IOS XE

A new install that never settles on a good channel plan

What you see: Weeks after the access points went in, neighbouring ones still share channels and the plan barely changes.
Why: Once running, automatic channel assignment is damped on purpose so it does not churn. If it never had a sensitive start-up pass after the last access point was installed, it can sit on a poor plan.
Prevent it: When all access points are installed, restart channel assignment in start-up mode: about 100 minutes of sensitive runs, then normal operation.

ap dot11 5ghz rrm dca restart
ap dot11 24ghz rrm dca restart
show ap dot11 5ghz channel

Security holeIOS / IOS XE

WPA2 Enterprise, and anyone can still kick users off

What you see: Users are thrown off Wi-Fi in one area at random. The logs show a flood of deauthentication frames.
Why: WPA2 encrypts data frames but not management frames, so deauthentication and disassociation frames can be forged. Protected management frames (802.11w) sign them, and WPA3 makes them mandatory.
Prevent it: Turn PMF on as optional now, required where every client supports it, and plan the move to WPA3.

wlan CORP 1 CORP
 shutdown
 security pmf optional
 no shutdown
show wlan id 1 | include PMF

Causes an outageNX-OS

"no feature" deletes all of that feature's config

What you see: You disable a feature to reset it and every related line is gone, with nothing to warn you on re-enable.
Why: Removing a feature on NX-OS removes all of its configuration.
Prevent it: Take a checkpoint first, and never use no feature as a quick reset.

checkpoint pre-change
show running-config ospf

Causes an outageNX-OS

A peer-link failure cuts off single-attached devices

What you see: The vPC peer link fails and servers attached to only one switch lose their uplink.
Why: With the keepalive still up, the secondary suspends its vPC member ports and SVIs to avoid a split brain. Orphan ports on the secondary are left without a gateway path.
Prevent it: Dual-attach everything. If you cannot, put orphans on the primary or suspend them with the vPC.

show vpc orphan-ports
interface Ethernet1/20
 vpc orphan-port suspend

Causes an outageNX-OS

Changing one vPC peer suspends the vPCs

What you see: You change the MTU or spanning-tree settings on one switch and its vPC member ports go down.
Why: Type-1 parameters must match on both peers. With graceful consistency check, the secondary suspends its vPCs until they match.
Prevent it: Check consistency before and after, and change both peers back to back.

show vpc consistency-parameters global
show vpc consistency-parameters vpc 10

Causes an outageNX-OS

The Nexus 3000 has no room for the new image

What you see: The copy fails part-way, or the upgrade cannot start, on an older Nexus 3000 or 3100.
Why: Early Nexus 3000, 3100 and 3500 models have about 1.4 to 1.6 GB of bootflash, and a full NX-OS image is over 1 GB, so two will not fit. A compacted image is much smaller. Its checksum no longer matches Cisco's published one, which is expected.
Prevent it: Use Cisco's pre-compacted image where one exists, or compact during the install or copy. Delete the old image once the new one is verified.

dir bootflash: | include free
install all nxos usb1:nxos.9.3.12.bin compact
copy scp://user@10.0.0.5/nxos.9.3.12.bin bootflash: compact vrf management

Causes an outageNX-OS

Jumping straight from an old 7.0(3) release to 9.3

What you see: The install refuses, reports incompatible configuration, or the switch comes back on the old release.
Why: From 7.0(3)I7(6) or earlier, 9.3(x) needs a hop through 7.0(3)I7(9) or later first. Some features also block the jump until they are removed. The install can sit silent for minutes and look frozen.
Prevent it: Plan the hops, run the incompatibility check against each target image, and never interrupt an install in progress.

show version
show incompatibility-all nxos bootflash:nxos.9.3.12.bin
install all nxos bootflash:nxos.7.0.3.I7.10.bin
! verify, then the next hop
install all nxos bootflash:nxos.9.3.12.bin

Causes an outageNX-OS

The vPC keepalive runs over the peer-link

What you see: The peer-link fails and both switches carry on as primary. MAC addresses flap, spanning tree disputes appear and traffic drops across the pair.
Why: The keepalive is how each switch tells a dead peer from a dead peer-link. If it shares the peer-link's cables or VLANs, both fail together, neither switch can tell which failure it is, and neither stands down.
Prevent it: Run the keepalive over mgmt0 in the management VRF, or over a dedicated routed link in its own VRF. Never over a VLAN that crosses the peer-link.

show vpc peer-keepalive
vpc domain 10
  peer-keepalive destination 192.168.0.2 source 192.168.0.1 vrf management

Quietly wrongNX-OS

The keepalive has been down for weeks and nothing told you

What you see: show vpc says the peer is not alive, but everything forwards normally. You only find out when the peer-link fails, or both switches reload.
Why: The keepalive carries no user traffic, so losing it changes nothing until a second failure. Then the pair cannot tell a dead peer-link from a dead switch. After a reload the domain cannot form without the keepalive, and with auto-recovery both switches can bring their vPCs up with no peer sync at all.
Prevent it: Alert on vPC keepalive syslog messages and include show vpc in every pre-change health check. Treat a dead keepalive as a fault to fix that day.

show vpc | include keep-alive
show vpc peer-keepalive
show logging logfile | include VPC

Causes an outageNX-OS

Two vPC pairs with the same domain ID

What you see: In a double-sided vPC, or a vPC between two data centres, the bundle between the pairs keeps flapping and member ports go suspended.
Why: Each pair builds its vPC system MAC from the domain ID (00:23:04:ee:be followed by the ID in hex) and uses it as its LACP system ID. Two pairs with the same ID each see their own system ID coming back from the partner.
Prevent it: Give every vPC domain that can see another one a unique domain ID. Plan the IDs per site, the way you plan VLAN numbers.

show vpc role | include system-mac
show lacp neighbor interface port-channel1
vpc domain 20

Causes an outageNX-OS

A VLAN added to a vPC but not to the peer-link

What you see: You add VLAN 30 to a server's vPC on both switches. The server's other VLANs work, VLAN 30 does not, and show vpc brief does not list 30 as active on that vPC.
Why: A VLAN only forwards on a vPC when it is allowed on the peer-link as well, because the peers sync its state and carry its flooded and failover traffic over that link. Adding it to the member port-channel alone is not enough.
Prevent it: Add new VLANs to the peer-link first, on both switches, with the add keyword. Then add them to the vPCs.

show vpc brief
interface port-channel1
  switchport trunk allowed vlan add 30
interface port-channel30
  switchport trunk allowed vlan add 30

Causes an outageNX-OS

A router or firewall peering with both Nexus switches over a vPC

What you see: OSPF or BGP comes up from the firewall to both switches, but some flows drop and others work, depending on source and destination.
Why: The firewall's port-channel hash and its choice of next hop are separate decisions. A packet addressed to one switch's MAC can hash to the other, cross the peer-link, and then be dropped, because traffic that has crossed the peer-link is not allowed out of a vPC member port.
Prevent it: Connect routed devices with separate layer 3 links to each switch, or use a non-vPC VLAN. On Nexus 9000 from 7.0(3)I5(1), routing adjacencies over a vPC are supported once both switches have peer-gateway and layer3 peer-router.

vpc domain 10
  peer-gateway
  layer3 peer-router
show running-config vpc

Causes an outageNX-OS

HSRP tracking on a vPC pair blackholes traffic

What you see: One switch loses its uplinks, tracking takes its gateway down, and some servers lose the network instead of failing over.
Why: With vPC both switches forward for the gateway MAC, and servers keep hashing to both whatever HSRP thinks. Traffic the tracked switch bridges over the peer-link to its partner can then fall foul of the rule that peer-link traffic may not leave by a vPC member port.
Prevent it: Do not use HSRP or VRRP object tracking on a vPC pair. Build a routed backup path between the two switches, so the one with dead uplinks routes through its partner.

show track
show hsrp brief
show running-config hsrp | include track

Causes an outageNX-OS

No routed path between the vPC peers

What you see: One Nexus loses all its uplinks to the core. About half the traffic from servers still arrives at it, and that half is dropped.
Why: Access switches and servers hash across both peers regardless of the uplinks behind them. The switch with no uplinks still answers as the gateway for whatever it receives, and has nowhere to send it unless it can route to its partner.
Prevent it: Add a routed point-to-point link between the peers with a routing adjacency. A dedicated SVI on a separate trunk will do. A VLAN over the peer-link is the last choice.

interface Ethernet1/48
  description L3 backup to N9K-2
  no switchport
  ip address 10.255.0.0/31
  ip router ospf 1 area 0.0.0.0
  no shutdown
show ip ospf neighbors

Causes an outageNX-OS

After a power cut only one Nexus comes back, and its vPCs stay down

What you see: Both switches lost power. One boots, the other does not, and every vPC on the healthy switch stays down.
Why: A vPC switch that boots and never hears its peer does not bring its vPCs up on its own unless auto-recovery is configured. With it, the switch waits for the reload delay (240 seconds by default) and then takes the primary role. Whether auto-recovery is on by default depends on platform and release.
Prevent it: Configure auto-recovery explicitly on both switches and check it in show vpc. Then the healthy switch carries on alone after the timer.

vpc domain 10
  auto-recovery reload-delay 240
show vpc | include Auto-recovery

Causes an outageNX-OS

Bridge Assurance on vPC member ports

What you see: During a vPC peer reload or failover, access switch uplinks go Bridge Assurance inconsistent and stop forwarding.
Why: Spanning-tree port type network turns on Bridge Assurance, which blocks a port as soon as BPDUs stop. During a failover the surviving peer can take a moment to start sending BPDUs on the vPCs, and the port blocks. vPC member ports never block anyway, so Bridge Assurance gains nothing there.
Prevent it: Keep port type network on the peer-link only. Leave vPC member ports as normal, and host ports as edge.

show spanning-tree inconsistentports
interface port-channel20
  spanning-tree port type normal

Quietly wrongNX-OS

The vPC primary is not the switch you configured

What you see: Single-attached devices were put on N9K-1 because it is primary. The peer-link fails and N9K-1 suspends its vPCs and SVIs instead.
Why: vPC roles do not pre-empt. After N9K-1 reloads, N9K-2 stays operational primary, even though N9K-1 has the lower role priority. Forcing the role back means bouncing the peer-link, which is disruptive.
Prevent it: Check the operational role, not the configuration, before every change. Better still, dual-attach everything so the role stops mattering.

show vpc role
show vpc brief | include role

Causes an outageNX-OS

A server on a port-channel cannot PXE boot

What you see: A new server cannot get an address or reach the build server during installation. Once the OS is installed and bonding is up, it works.
Why: Before the OS loads, the NIC sends no LACP. By default NX-OS suspends port-channel members that receive no LACP, so the installer has no network at all.
Prevent it: On server port-channels that must PXE boot, configure no lacp suspend-individual, and make the build VLAN the access or native VLAN. Keep the default on links between switches.

interface port-channel30
  no lacp suspend-individual
show port-channel summary

Quietly wrongNX-OS

LACP fast rate turns a hitless upgrade into a reload

What you see: The plan said non-disruptive, but the install reports a disruptive upgrade, or bundles drop while it runs.
Why: With fast rate, LACP sends a PDU every second and gives up after three. That is shorter than the control-plane restart an in-service upgrade needs, so the upgrade falls back to a reload.
Prevent it: Use the normal LACP rate unless something truly needs fast. Before an upgrade, check which ports are not ready and what the install will do.

show lacp issu-impact
show install all impact nxos bootflash:nxos.bin
interface port-channel30
  no lacp rate fast

Causes an outageNX-OS

A small switch plugged into a FEX port err-disables it

What you see: Someone patches a desk switch or lab switch into a Nexus 2000 port and the port goes err-disabled straight away.
Why: Fabric extender host ports are for hosts. BPDU Guard is on for every host port, so the first BPDU shuts it.
Prevent it: Connect switches to the parent Nexus, never to a FEX host port. If a hypervisor runs a virtual switch, make sure it does not send BPDUs.

show interface status err-disabled
show spanning-tree interface Ethernet101/1/10 detail

Causes an outageNX-OS

One fabric link fails and a block of FEX ports goes down

What you see: A single uplink between a Nexus 2000 and its parent fails, and a block of host ports goes down with it while the rest stay up.
Why: With static pinning each host port is tied to one fabric link, so losing the link takes its ports down, and servers survive only if their NIC teaming fails over. Setting max-links higher than the number of links connected also leaves some ports with nothing to pin to.
Prevent it: Bundle the fabric links into a port-channel, so host ports use whichever links are left. If you must pin, set max-links to match the links you have.

show fex 101 detail
interface Ethernet1/1-2
  switchport mode fex-fabric
  fex associate 101
  channel-group 101

Quietly wrongNX-OS

Jumbo frames on the ports, 1500 on the SVI

What you see: Jumbo-frame storage or backup traffic works between servers in the same VLAN, but fails or crawls when it is routed to another VLAN.
Why: On Nexus 9000 the MTU of layer 2 ports and of the VLAN interface are set separately. mtu 9216 on the port-channels lets jumbo frames switch within the VLAN, but routed traffic goes through the SVI, which is still 1500. On Nexus 5000 and 5500, layer 2 jumbo frames are set in a network-qos policy, not on the interface.
Prevent it: Set the MTU end to end: member ports, port-channels, the peer-link and the SVIs on both peers. Port-channel MTU is a type-1 vPC parameter, so change both peers back to back.

interface port-channel30
  mtu 9216
interface Vlan30
  mtu 9216
show interface Vlan30 | include MTU
ping 10.30.30.20 packet-size 8972 df-bit

Causes an outageNX-OS

Keepalive cabled mgmt0 to mgmt0 on chassis with two supervisors

What you see: After a supervisor switchover on one chassis, the vPC keepalive dies, although both switches are healthy.
Why: Only the active supervisor's mgmt0 is live. Cabled back to back, supervisor 1 to supervisor 1, a switchover leaves the active supervisor on one side talking to a standby on the other.
Prevent it: Connect every supervisor's mgmt0 to a management switch, or use a dedicated routed keepalive link on line card ports.

show vpc peer-keepalive
show module

Causes an outageNX-OS

VLAN 4000 cannot be created on the Nexus

What you see: The campus carries VLAN 4000 or 4010, and the Nexus refuses to create it, so the trunk between them cannot carry it.
Why: NX-OS reserves a block of 128 VLANs for internal use, 3968 to 4095 by default. Catalyst switches reserve a different range, so a VLAN that is fine on one side can be impossible on the other.
Prevent it: Keep your VLAN plan below 3968. If you must use those numbers, move the reserved block with system vlan <start> reserve, save, and reload.

show system vlan reserved
system vlan 3000 reserve
copy running-config startup-config
! takes effect after a reload

Causes an outageNX-OS

The switch breathes the hot aisle

What you see: A new top-of-rack switch runs hot, its fans sit at full speed and temperature alarms appear, while the servers beside it are fine.
Why: Nexus top-of-rack switches come in port-side intake and port-side exhaust versions, set by the fan and power supply modules. Mount the ports facing the rear to meet the server cabling, and the wrong version pulls its air from the hot aisle. All the fans and power supplies must blow the same way.
Prevent it: Decide which way the ports will face before you order, and order the airflow to match. If you must change it, swap every fan and power supply together with the switch powered off.

show environment fan
show environment temperature
show inventory

Causes an outageIOS / IOS XE

Routes redistributed into EIGRP never appear

What you see: You redistribute OSPF or BGP into EIGRP. There is no error, and none of the routes reach the EIGRP neighbours.
Why: EIGRP cannot turn another protocol's metric into its own. Without a seed metric, routes from OSPF or BGP are treated as unreachable and never advertised. Connected routes, static routes and other EIGRP processes are the exceptions.
Prevent it: Give every redistribute into EIGRP a metric, or set default-metric under the process. Base the values on the links the traffic really uses.

router eigrp 100
 redistribute ospf 1 metric 100000 10 255 1 1500 route-map OSPF-TO-EIGRP
show ip route eigrp | include EX

Causes an outageIOS / IOS XE

Two routers redistributing both ways send traffic the long way round

What you see: Redistribution works, but traceroutes show traffic crossing into the other protocol and back, and some routes flip between EIGRP and OSPF every few seconds.
Why: With two redistribution points, a route can leave EIGRP at one router, enter OSPF, and arrive at the other router as an OSPF external route. At 110 it beats the same route as an EIGRP external at 170, so the second router prefers the long way round, stops redistributing it, and the cycle starts again.
Prevent it: Tag routes as you redistribute them and deny that tag on the way back. Raise the OSPF external distance on the redistribution routers above 170. Better still, redistribute at one point only, or advertise a summary.

route-map EIGRP-TO-OSPF deny 10
 match tag 110
route-map EIGRP-TO-OSPF permit 20
 set tag 90
route-map OSPF-TO-EIGRP deny 10
 match tag 90
route-map OSPF-TO-EIGRP permit 20
 set tag 110
router ospf 1
 redistribute eigrp 100 subnets route-map EIGRP-TO-OSPF
 distance ospf external 171

Quietly wrongIOS / IOS XE

variance does nothing for the path you wanted

What you see: You set variance 2 so EIGRP also uses a slower second link. All the traffic still takes one path.
Why: Variance only adds paths that are feasible successors: the neighbour's reported distance must be lower than your best feasible distance. A path that fails that test is never used, however high you set variance.
Prevent it: Look for the second path in the topology table and compare its reported distance with the feasible distance. If it is not a feasible successor, change the metrics or the design, not the variance.

show ip eigrp topology 10.30.0.0/24
show ip eigrp topology all-links
router eigrp 100
 variance 2

Causes an outageIOS / IOS XE

Two EIGRP stub routers at one site stop backing each other up

What you see: A site has two WAN routers, both configured as EIGRP stubs. One loses its WAN link and the users behind it are cut off, although the other router is fine.
Why: A stub router does not pass routes it learned from one neighbour on to another. The router with the dead link never hears the WAN routes from its partner, and the hub never hears its LAN through the partner.
Prevent it: Use a stub leak-map so the two routers pass each other what they need, or do not make them stubs. Test the failure of each WAN link before sign-off.

route-map LEAK permit 10
 match ip address prefix-list SHARE
router eigrp 100
 eigrp stub connected summary leak-map LEAK
show ip eigrp neighbors detail | include Stub

Quietly wrongIOS / IOS XE and NX-OS

A distribute-list in under OSPF does not stop the route spreading

What you see: You filter a prefix with distribute-list in on one router. It leaves that router's table, but every router beyond it still has the route and sends traffic towards it.
Why: Every router in an OSPF area must hold the same database. A distribute-list in only stops the route being installed locally. The LSA is still flooded, so the filtering router can become a black hole for traffic the others send it.
Prevent it: Filter between areas with area filter-list on the ABR, or at redistribution on the ASBR. Use distribute-list in only when you want one router to ignore a route.

ip prefix-list NO-LAB deny 10.99.0.0/16 le 32
ip prefix-list NO-LAB permit 0.0.0.0/0 le 32
router ospf 1
 area 2 filter-list prefix NO-LAB in
show ip ospf database summary 10.99.0.0

Causes an outageIOS / IOS XE and NX-OS

An area made stub on one router only

What you see: You make area 1 a stub on the ABR and its neighbours in area 1 drop off and do not come back.
Why: The stub flag travels in every OSPF hello and must match. A router ignores hellos whose flag differs from its own, so the adjacency times out. no-summary is different: it only belongs on the ABR.
Prevent it: Configure area stub on every router in the area in one change window, and expect a short outage while it goes in. Add no-summary on the ABR only.

router ospf 1
 area 1 stub
! on the ABR only, for a totally stubby area
 area 1 stub no-summary
show ip ospf | include Area|stub

Causes an outageIOS / IOS XE

A virtual link pointed at an interface address

What you see: You configure a virtual link to join a stranded area to the backbone, and it never comes up.
Why: A virtual link names the router ID of the ABR at the far end, not an address, and the area in the command is the transit area between the two. The transit area cannot be a stub.
Prevent it: Read the far router's ID from show ip ospf and name the transit area. Treat a virtual link as a temporary fix, and plan the redesign that removes it.

show ip ospf | include ID
router ospf 1
 area 1 virtual-link 10.255.255.3
show ip ospf virtual-links

Causes an outageIOS / IOS XE and NX-OS

iBGP between loopbacks never establishes

What you see: Two routers in the same AS peer to each other's loopback. The session sits in Active or Idle, although the loopbacks ping.
Why: BGP sources its TCP session from the outgoing interface address. The far router only accepts the session from the address it has configured as the neighbour, the loopback. A plain ping also leaves from the interface address, which is why it looks fine.
Prevent it: Add update-source Loopback0 on both routers, make sure the IGP carries both loopbacks, and test with a ping sourced from the loopback. eBGP to a loopback also needs ebgp-multihop or disable-connected-check.

router bgp 65000
 neighbor 10.255.255.2 remote-as 65000
 neighbor 10.255.255.2 update-source Loopback0
ping 10.255.255.2 source Loopback0
show ip bgp summary

Quietly wrongIOS / IOS XE and NX-OS

A new BGP route-map changes nothing

What you see: You apply a route-map to a neighbour to set local preference or filter prefixes. show ip bgp still shows the old attributes.
Why: BGP applies policy to updates as they are sent or received. Routes already in the table are not looked at again until the neighbour sends them again.
Prevent it: Ask for the routes again with a soft clear, which keeps the session up. Never hard-clear a production peer to apply policy.

clear ip bgp 192.0.2.1 soft in
clear ip bgp 192.0.2.1 soft out
show ip bgp 203.0.113.0

Quietly wrongIOS / IOS XE and NX-OS

BGP uses one link out of two equal ones

What you see: Two links to the same provider, two eBGP sessions, identical paths. All outbound traffic leaves by one link.
Why: BGP installs a single best path by default. Unlike OSPF and EIGRP, it does not share load across equal paths until you allow it.
Prevent it: Set maximum-paths (maximum-paths ibgp for iBGP). The paths must tie on every attribute the best-path choice compares, including an identical AS path.

router bgp 65000
 maximum-paths 2
show ip bgp 0.0.0.0
show ip route 0.0.0.0

Quietly wrongIOS / IOS XE

Two match lines you meant as AND become OR

What you see: You write match ip address prefix-list A, and on the next line match ip address prefix-list B, expecting a route to need both. Routes in either list are caught.
Why: Different kinds of match on separate lines are combined with AND. Two matches of the same kind are merged by IOS into one line, and the lists on one line are combined with OR.
Prevent it: To require two conditions of the same kind, build one prefix-list or ACL that expresses both, or use two route-map entries. Always read show route-map, not what you typed.

route-map EXAMPLE permit 10
 match ip address prefix-list A
 match ip address prefix-list B
show route-map EXAMPLE
! shows: match ip address prefix-lists: A B

Causes an outageIOS / IOS XE and NX-OS

The end of a route-map means something different in each place

What you see: A route-map that worked for policy routing drops routes when reused for redistribution, or a BGP route-map filters out everything you did not mention.
Why: Every route-map ends with an implicit deny. For redistribution, denied routes are not redistributed. On a BGP neighbour, they are not accepted or not sent. For policy routing, a packet that is denied or matches nothing is simply routed normally.
Prevent it: End redistribution and BGP route-maps with a permit entry with no match when you mean "everything else unchanged". Write policy-routing maps knowing that unmatched traffic follows the routing table.

route-map FROM-ISP permit 10
 match ip address prefix-list PREFER
 set local-preference 200
route-map FROM-ISP permit 20
! no match here: everything else passes unchanged

Quietly wrongIOS / IOS XE

Your routing protocol thinks the tunnel runs at 100 kbit/s

What you see: Traffic avoids a GRE or IPsec tunnel with plenty of capacity, or EIGRP takes ages to exchange a large table over it.
Why: A tunnel interface cannot know the speed of the path beneath it, so IOS gives it a default bandwidth, 100 kbit/s on current releases. OSPF and EIGRP build their metrics from that figure, and EIGRP limits its own traffic to half of it.
Prevent it: Set bandwidth (and, for EIGRP, delay) on every tunnel to match the real path, the same at both ends. It changes metrics and protocol pacing only, not throughput.

show interfaces Tunnel0 | include BW
interface Tunnel0
 bandwidth 50000
show ip ospf interface Tunnel0 | include Cost

Causes an outageNX-OS

NX-OS will not redistribute without a route-map

What you see: redistribute static under OSPF on a Nexus is rejected as incomplete, and a migration script copied from IOS fails part way through.
Why: NX-OS requires a route-map on every redistribute command, so there is always an explicit policy. IOS lets you redistribute everything with no filter at all.
Prevent it: Write the prefix-list and route-map first, matching exactly what you mean to redistribute. Translate IOS redistribution lines by hand during a migration.

ip prefix-list STATIC-OUT seq 5 permit 10.50.0.0/16 le 24
route-map STATIC-TO-OSPF permit 10
  match ip address prefix-list STATIC-OUT
router ospf 1
  redistribute static route-map STATIC-TO-OSPF

Causes an outageIOS / IOS XE

The tracked route vanished the moment you added tracking

What you see: You add track 10 to the primary default route and the route disappears straight away, although the link is fine.
Why: An IP SLA operation does nothing until it is scheduled. An unscheduled probe has no result, the track reports Down, and the tracked route is withdrawn.
Prevent it: Schedule the probe and see it return OK before you attach the track to a route. Put the backup route in first, so a mistake fails over instead of failing.

ip sla schedule 10 life forever start-time now
show ip sla statistics 10
show track 10
ip route 0.0.0.0 0.0.0.0 203.0.113.1 track 10

Causes an outageIOS / IOS XE

The failover flaps every minute

What you see: When the primary circuit fails, the default route moves to the backup, then back, then away again, over and over.
Why: The probe target is reached by the default route. Once the backup takes over, the probe goes out of the backup, succeeds, and the track brings the primary route back, which fails the probe again.
Prevent it: Pin the probe target to the primary next hop with a host route, with a Null0 host route behind it so the probe is dropped rather than rerouted. Damp the track with delay down and up.

ip route 192.0.2.53 255.255.255.255 203.0.113.1
ip route 192.0.2.53 255.255.255.255 Null0 250
track 10 ip sla 10 reachability
 delay down 30 up 60

Security holeIOS / IOS XE and NX-OS

A read-write SNMP community is a way to replace the config

What you see: An old snmp-server community line with RW sits in the config, often "private", with no ACL on it.
Why: With write access, SNMP can change settings and, through the config-copy MIB, tell the device to load a configuration from a TFTP server. SNMPv1 and v2c send the community in clear text.
Prevent it: Remove RW communities. Use SNMPv3 with authentication and encryption, read-only, restricted by an ACL to the monitoring servers.

show running-config | include snmp-server community
no snmp-server community private RW
snmp-server group MON v3 priv access 99
snmp-server user nms MON v3 auth sha <auth-key> priv aes 128 <priv-key>

Quietly wrongIOS / IOS XE

After a config restore, monitoring fails: the SNMPv3 users are gone

What you see: A replacement router is loaded from the backup config. Everything works except monitoring, which reports authentication failures.
Why: SNMPv3 users are not shown in the running configuration, so they are never in a text backup. The groups and views come back; the users do not.
Prevent it: Keep the SNMPv3 user commands in the build template or the password vault, and re-enter them after any restore or replacement. Check with show snmp user.

show snmp user
show snmp group
snmp-server user nms MON v3 auth sha <auth-key> priv aes 128 <priv-key>

Causes an outageIOS / IOS XE

A VLAN map that drops one thing drops everything

What you see: You add a VLAN access map to block one server, and everything in the VLAN stops talking.
Why: A VLAN access map ends with an implicit drop, like an ACL. Unless the last entry forwards traffic with no match clause, anything not explicitly forwarded is dropped, including traffic you never mentioned.
Prevent it: End every VLAN access map with an entry that has action forward and no match. Try it on a quiet VLAN first.

vlan access-map BLOCK-SRV 10
 match ip address BLOCK-SRV
 action drop
vlan access-map BLOCK-SRV 20
 action forward
vlan filter BLOCK-SRV vlan-list 10
show vlan access-map

Causes an outageIOS / IOS XE

After a reload, some phones and access points stay dark

What you see: A switch comes back from a reload or power cut and a handful of PoE ports do not power their devices. show power inline shows the budget used up.
Why: At connection the switch reserves power by each device's class, which is often more than the device draws, until CDP or LLDP agrees a lower figure. A budget sized on measured use can run out when everything powers up at once.
Prevent it: Size the PoE budget on allocated power, not measured use. Give critical ports a static allocation, and add the second power supply where the platform takes one.

show power inline
interface GigabitEthernet1/0/10
 power inline static max 30000

Quietly wrongIOS / IOS XE

"debug ip packet" shows nothing for the traffic you are chasing

What you see: Users are sending traffic through the router, and debug ip packet stays silent, so it looks as if nothing arrives.
Why: Transit traffic is switched by CEF or in hardware, and debug only sees packets that reach the CPU: traffic to and from the router itself, and anything punted. Turning CEF off to see it can take the router down.
Prevent it: Prove traffic is arriving with an on-box packet capture, NetFlow, or ACL counters. Keep debug for control-plane problems, filtered, with console logging off.

monitor capture CAP interface GigabitEthernet0/0/1 both match ipv4 host 10.20.0.15 any
monitor capture CAP start
show monitor capture CAP buffer brief
monitor capture CAP stop

Quietly wrongIOS / IOS XE and NX-OS

"The router can ping it, so the network is fine"

What you see: The far server answers a ping from the router, but users behind the router cannot reach it.
Why: A ping from the router is sourced from its outgoing interface. The return route to that link can exist when the route back to the user LAN does not, and a firewall may allow one and not the other.
Prevent it: Ping and traceroute from the router with the user LAN as the source, so the test takes the same return path as user traffic.

ping 10.50.0.10 source GigabitEthernet0/0/1
traceroute 10.50.0.10 source GigabitEthernet0/0/1

Causes an outageIOS / IOS XE

On two ISPs, traffic leaves one link with the other link's address

What you see: After failover to the second ISP, nobody can browse. show ip nat translations shows inside hosts translated to the first ISP's address, on packets leaving the second ISP's link.
Why: Two overload rules that share one ACL do not choose by exit interface: the first one matches and wins. The second ISP drops traffic sourced from someone else's address. Translations built before the failover also stay until they time out.
Prevent it: Tie each overload rule to a route-map that matches the inside ACL and the exit interface. After a failover or failback, clear the old translations if users cannot wait for them to age out.

route-map NAT-ISP-A permit 10
 match ip address PAT
 match interface GigabitEthernet0/0/1
route-map NAT-ISP-B permit 10
 match ip address PAT
 match interface GigabitEthernet0/0/2
ip nat inside source route-map NAT-ISP-A interface GigabitEthernet0/0/1 overload
ip nat inside source route-map NAT-ISP-B interface GigabitEthernet0/0/2 overload
clear ip nat translation *

Quietly wrongIOS / IOS XE

Policy routing fails open when its next hops are down

What you see: Guest traffic is meant to use only the guest ISP. When that link fails, guests carry on browsing out of the corporate link.
Why: When none of the next hops or interfaces in a policy is available, the packet is routed normally by the routing table. A policy route is a preference, not a fence.
Prevent it: If the traffic must never take the normal path, end the set list with Null0. Keep an ACL on the corporate path as well if this is a security boundary.

route-map GUEST-ONLY permit 10
 match ip address GUEST
 set ip next-hop verify-availability 198.51.100.1 1 track 20
 set interface Null0
show route-map GUEST-ONLY

Causes an outageIOS / IOS XE

Local policy brings the tunnel up on the right link, and nothing passes

What you see: You steer the router's own IKE out of the backup ISP with ip local policy. The tunnel shows up, but traffic through it fails or the far end sees it arriving from the wrong address.
Why: Local policy routing acts on the packet before encryption. The encrypted ESP packet is routed again by the normal routing table, which still points at the primary link.
Prevent it: Add a host route to the VPN peer through the same link, tracked like the policy, so the encrypted packets follow the key exchange.

ip route 192.0.2.50 255.255.255.255 198.51.100.1 track 20
show crypto ipsec sa peer 192.0.2.50 | include encaps|decaps

Causes an outageIOS / IOS XE

The conditional backup announcement never happens

What you see: The primary link fails, and the prefix that should then be announced to the backup provider is never sent. show ip bgp neighbors says the advertise map is withdrawn.
Why: The non-exist-map watches for a prefix in the BGP table. If that prefix can still be learned another way, for example from the backup provider via the wider internet, it never disappears.
Prevent it: Add a match as-path to the watch map, anchored on the primary provider's AS, so only that provider's copy of the prefix counts.

ip as-path access-list 30 permit ^64500_
route-map WATCH-PRIMARY permit 10
 match ip address prefix-list WATCH
 match as-path 30
show ip bgp neighbors 192.0.2.9 | include Condition-map

Causes an outageIOS / IOS XE

NAT between VRFs works one way only

What you see: VRF users send traffic out through the VASI pair and the translations appear, but no replies come back.
Why: The VRF traffic is translated to the WAN interface's own address. Replies to that address are treated as traffic for the router itself and are never forwarded back through the VASI pair.
Prevent it: Translate to a separate address or pool routed to the router, not to the WAN interface address.

ip nat pool CUST-OUT 203.0.113.10 203.0.113.10 prefix-length 24
ip nat inside source list CUST-NETS pool CUST-OUT vrf WAN overload
show ip nat translations vrf WAN

Quietly wrongIOS / IOS XE

Policy routing picks the VRF, then the router drops the packet

What you see: Transit traffic selected into a VRF by policy routing works. Pings, DHCP relay or management traffic to the router's own address from the same hosts fail.
Why: With Multi-VRF selection, the interface's connected addresses exist only in the global table unless you tell the router to accept them in each VRF. Packets selected into a VRF and addressed to the router find no receive entry and are dropped.
Prevent it: Add ip vrf receive for every VRF the policy can select on that interface.

interface GigabitEthernet0/0/0
 ip policy route-map VRF-SELECT
 ip vrf receive FINANCE
show ip route vrf FINANCE connected

Causes an outageIOS / IOS XE and NX-OS

eBGP EVPN sends traffic to the spine instead of the leaf

What you see: In an eBGP-only EVPN fabric, VXLAN tunnels point at spine addresses. Traffic between leaves is dropped.
Why: eBGP changes the next hop to itself by default. For EVPN, the next hop must stay the originating leaf's VTEP address.
Prevent it: On the spines, apply an outbound route-map that leaves the next hop unchanged for the L2VPN EVPN address family.

route-map NH-UNCHANGED permit 10
 set ip next-hop unchanged
router bgp 65000
 neighbor 10.0.0.11
  address-family l2vpn evpn
   route-map NH-UNCHANGED out

Causes an outageIOS / IOS XE and NX-OS

eBGP spines throw away every EVPN route

What you see: Leaves peer with the spines and the sessions are up, but each leaf receives no EVPN routes from the others.
Why: A BGP speaker keeps only VPN routes whose route targets match a local VRF. Spines in an eBGP fabric hold no VRFs, so they discard everything. iBGP route reflectors are exempt, which is why this only bites in eBGP designs.
Prevent it: Tell the spines to keep every route target.

! IOS XE
router bgp 65000
 no bgp default route-target filter
! NX-OS
router bgp 65000
 address-family l2vpn evpn
  retain route-target all

Causes an outageACI

A spine coming back from an upgrade attracts traffic too early

What you see: In an ACI Multi-Pod fabric, a spine reloads for an upgrade and traffic between pods or to external networks drops while it comes back, although the other spines are healthy.
Why: While a spine boots, it advertises the maximum IS-IS metric, 63, so other switches route around it. Routes the spines redistribute into IS-IS from the inter-pod network also carry 63 by default, so the booting spine looks no worse than the rest.
Prevent it: Lower the fabric-wide IS-IS metric for redistributed routes below 63 (32 is common) before the first upgrade. It is a single setting in the fabric's IS-IS policy.

APIC: System > System Settings > ISIS Policy
  ISIS metric for redistributed routes: 32

Causes an outageASA and Firepower

A firewall cluster drops large packets that are redirected between members

What you see: Small flows work. Large transfers stall when the cluster has to send packets from one member to the owner of the connection.
Why: Packets forwarded over the cluster control link carry an extra trailer. If the control link MTU is the same as the data MTU, those packets are too big.
Prevent it: Set the cluster control link MTU at least 100 bytes above the largest data interface MTU, and the switches carrying it to jumbo frames.

! data interfaces at 9084, so the cluster control link needs 9184 or more
mtu cluster 9184
show cluster info
show running-config mtu

Quietly wrongIOS / IOS XE

The replaced core switch's MAC address lives on in the StackWise Virtual pair

What you see: You replace the failed switch in a StackWise Virtual pair and redeploy the old one elsewhere. Strange ARP and MAC flaps appear wherever the old switch went.
Why: The pair keeps using the MAC address of the switch that formed it, even after that switch has gone, until a full reload or a forced update.
Prevent it: After replacing the switch that owned the stack MAC, run stack-mac update force in a quiet window, or reload the pair. Wipe a returned switch before reusing it.

show switch
stack-mac update force

Quietly wrongIOS / IOS XE

"vlan-id 1" on a 9800 policy profile does not mean VLAN 1

What you see: Clients on a new WLAN get addresses from the controller's management VLAN, or at a FlexConnect site from the AP's native VLAN.
Why: On the 9800 a policy profile left at VLAN 1 means the management VLAN in local mode and the AP native VLAN in FlexConnect. It never means VLAN 1 itself.
Prevent it: Set an explicit VLAN name or number on every policy profile.

wireless profile policy CORP
 vlan CORP-USERS
show wireless profile policy detailed CORP | include VLAN

Causes an outageIOS / IOS XE

Mobility tunnels drop after a 9800 SSO switchover

What you see: The standby controller takes over cleanly, and roaming to and from other controllers stops working until the tunnels are rebuilt.
Why: The mobility peers know the pair by its wireless mobility MAC. If it is not set, the pair uses the active unit's MAC, which changes on switchover.
Prevent it: Set the wireless mobility MAC explicitly on SSO pairs, and use that MAC in every peer's mobility configuration.

wireless mobility mac-address 0000.5e00.5301
show wireless mobility summary

Causes an outageIOS / IOS XE

SD-WAN VRRP takes a minute to fail over

What you see: The active router at a dual-router branch loses its WAN connections, and the LAN gateway stays on it for about 60 seconds.
Why: VRRP tracking OMP waits for the OMP hold timer, 60 seconds by default, before it decides the overlay has gone.
Prevent it: Track a prefix list instead, for a route that disappears as soon as the overlay fails, such as the data-centre summary. Use tloc-change increase-pref so return traffic follows the VRRP active router.

show vrrp
show sdwan omp summary

Causes an outageIOS / IOS XE

Switching an SD-Access fabric to default deny cuts the underlay

What you see: You change the default group policy to deny. Routing adjacencies, BFD and management to fabric nodes start failing.
Why: Default deny applies to every untagged packet, and underlay links carry routing, BFD and management without a group tag.
Prevent it: Disable TrustSec enforcement on the layer 3 underlay links, and on AP and extended-node VLANs, before turning on default deny. Catalyst Center 2.3.7 and later does it for new links.

interface TenGigabitEthernet1/1/1
 no cts role-based enforcement
show cts role-based permissions default

Quietly wrongIOS / IOS XE

Two QoS classes match the same traffic and the first one takes it all

What you see: Video calls are counted in the voice class, the video class shows zero packets, and voice starts dropping when the priority queue fills.
Why: A policy map checks its classes from the top and stops at the first match. Voice and video both use RTP, often on the same UDP port range, so a voice class written on ports alone catches the video too.
Prevent it: Classify on more than the port range: the voice VLAN or subnet, trusted markings from the phones, or NBAR where the platform supports it. Then check every class is counting with show policy-map interface.

show policy-map interface GigabitEthernet0/1
show class-map

Quietly wrongIOS / IOS XE

One "permit tcp any any" in a QoS class remarks every TCP session

What you see: The business application crawls during busy hours. show policy-map shows almost everything landing in the bulk or scavenger class.
Why: The access list behind a low-priority class has a broad line in it, often added to catch ICMP or "everything else". Any class above it in the policy gets the traffic it matches; everything else falls into the catch-all.
Prevent it: Read the access lists in a QoS policy as carefully as a security access list. Never put "any any" lines in a class: let class-default catch what is left.

show access-lists
show policy-map interface GigabitEthernet0/1

Causes an outageIOS / IOS XE

The priority queue drops voice when it is too small

What you see: One call is fine. Several calls during a busy period all break up together, even though voice has priority.
Why: "priority <kbps>" gives strict priority up to that rate, and polices the class to it whenever the link is congested. Traffic above the rate is dropped, and it is dropped across every call, not just the newest.
Prevent it: Size the priority queue for the most calls you expect, using about 90 kbps per G.711 call over Ethernet. Keep total priority traffic to roughly a third of the link, and use call admission control where it is available.

show policy-map interface GigabitEthernet0/1 | section Q-VOICE

Causes an outageIOS / IOS XE

A QoS policy on a sub-rate circuit never does anything

What you see: The WAN port is 1 Gbps, the contract is 50 Mbps, and calls still break up at busy times. show policy-map shows no drops and no queueing at all.
Why: The router only queues when its own port is congested, and a 1 Gbps port carrying 50 Mbps never is. The carrier polices to the contract and drops the excess, whichever class it was in.
Prevent it: Shape to just under the contract rate in a parent policy, and put the queueing policy inside it as a child. The queue then forms in your router, where your priorities apply.

policy-map WAN-SHAPE
 class class-default
  shape average 47500000
  service-policy WAN-CHILD
interface GigabitEthernet0/1
 service-policy output WAN-SHAPE

Quietly wrongIOS / IOS XE

Shaping is not a way to protect voice

What you see: Someone shapes the voice class "so it is never dropped", and call quality gets worse.
Why: A shaper holds packets back to keep under a rate, which adds delay and jitter. Voice needs the opposite: to be sent first. Policing and shaping limit traffic; only a priority queue speeds it up.
Prevent it: Put voice in a priority queue. Shape the whole interface to the contract rate as the parent, and keep shaping off the voice class itself.

show policy-map interface GigabitEthernet0/1

Causes an outageIOS / IOS XE

The inbound ACL on a port forward matches the inside address and blocks everything

What you see: A server is published with a static NAT, and an inbound access list on the outside interface permits traffic to the server's private address. Nothing gets through from outside.
Why: For traffic arriving from outside, IOS checks the inbound access list before NAT translates the destination. The access list sees the public address, never the private one.
Prevent it: Write the inbound access list against the public address and port. Remember a line for replies to inside users' sessions, or the final deny breaks their browsing too.

ip access-list extended OUTSIDE-IN
 permit tcp any host 203.0.113.2 eq 443
 permit tcp any host 203.0.113.2 established
 deny   ip any any

Causes an outageIOS / IOS XE

Storm control with "action shutdown" takes down an uplink

What you see: A backup job or software rollout runs, and an uplink or a server port goes err-disabled. The site loses connectivity until someone recovers the port.
Why: Storm control acts on the rate of traffic it measures, not on whether it is a real storm. A low level with "action shutdown" treats a busy minute as an attack. On some older Catalyst models the unicast threshold counts all unicast traffic, not only unknown unicast.
Prevent it: Use storm control on access ports, with levels based on measured traffic. On uplinks, drop the excess or send a trap rather than shutting the port. Check what your platform counts for unicast before setting a unicast level.

interface GigabitEthernet1/0/10
 storm-control broadcast level 5.00
 storm-control action trap
show storm-control

Causes an outageIOS / IOS XE

OSPF area authentication is on, but the key is set per interface

What you see: After "area 0 authentication message-digest", some neighbours drop and do not come back.
Why: The area command turns authentication on for every interface in the area, including virtual links. The key itself lives on each interface. Any interface in the area without a matching key loses its neighbours.
Prevent it: Put the key on every interface in the area before you enable authentication for the area, or authenticate per interface instead. Change both ends of a link in the same window.

interface GigabitEthernet0/0
 ip ospf message-digest-key 1 md5 <key>
router ospf 1
 area 0 authentication message-digest
show ip ospf interface GigabitEthernet0/0 | include auth

Causes an outageIOS / IOS XE

An NSSA has no default route

What you see: A branch area is changed from stub to NSSA so a partner network can be redistributed. Its routers lose the way out to everything else.
Why: A stub area gets a default route from its ABR automatically. A plain NSSA does not: it blocks the external routes without replacing them.
Prevent it: Add "area <n> nssa default-information-originate" on the ABR, or make it totally NSSA with "no-summary", which brings an automatic inter-area default.

router ospf 1
 area 2 nssa default-information-originate
show ip route 0.0.0.0

Quietly wrongIOS / IOS XE

PPPoE: pings work, big pages stall

What you see: A broadband router on PPPoE is up, pings and small pages work, but some HTTPS sites and large downloads hang.
Why: PPPoE adds 8 bytes, so the dialer can carry 1,492-byte packets, not 1,500. Hosts on the LAN still send 1,500. If ICMP is filtered anywhere on the path, path MTU discovery cannot correct it.
Prevent it: Set the dialer MTU to 1492 and clamp TCP MSS to 1452 on the dialer or the LAN interface.

interface Dialer1
 mtu 1492
 ip tcp adjust-mss 1452

Security holeIOS / IOS XE

The native VLAN is set to a user VLAN

What you see: The native VLAN was moved off VLAN 1 to "something else", and the something else was the users' VLAN.
Why: Untagged frames on a trunk join the native VLAN. A misconfigured device, a double-tagging attack, or a port that falls back to untagged then lands straight in a VLAN full of users.
Prevent it: Use a dedicated native VLAN that carries nothing else and is not allowed on access ports, or tag the native VLAN as well where the platform supports it.

vlan 999
 name NATIVE-UNUSED
interface range GigabitEthernet1/1/1 - 2
 switchport trunk native vlan 999
vlan dot1q tag native

Quietly wrongIOS / IOS XE

Log messages never show up in an SSH session

What you see: You expect to see interface and routing messages while you work over SSH, and nothing appears.
Why: Messages go to the console by default. An SSH or Telnet session only receives them after "terminal monitor", which is an exec command, applies to that session only and is lost when you disconnect. Monitor logging also has its own severity level.
Prevent it: Run "terminal monitor" in each session, check "logging monitor" is set to the level you want, and stop it with "terminal no monitor" before you leave a debug running.

terminal monitor
show logging | include Monitor

Causes an outageNX-OS and IOS / IOS XE

The passive firewall's links stay up, so routing never follows a failover

What you see: The firewall pair fails over, but traffic keeps going to the site or interface where the old active unit was. Static routes and tracked interfaces on the switch side show nothing changed.
Why: Many firewalls keep the passive unit's data links up so failover is quicker, and in virtual labs link state is never passed through at all. The switch sees up/up on both sides, so a route tied to the interface, or tracking of line protocol, never moves.
Prevent it: Track the firewall's data address with an IP SLA probe from the switch, so only the side with the active unit keeps its routes. On hardware, set the passive link state to shutdown if you want the switch to see it as well.

ip sla 10
  icmp-echo 172.16.10.1 source-interface Vlan2010
    vrf APP
    frequency 5
ip sla schedule 10 life forever start-time now
track 10 ip sla 10 reachability
vrf context APP
  ip route 10.20.0.0/16 172.16.10.1 track 10

Causes an outageIOS / IOS XE

The day-0 template works on old switches and fails on new ones

What you see: A new switch running IOS XE 26.1 or later rejects lines from the standard template: Telnet on the VTY lines, ip http server, SNMP communities, TFTP settings, type 7 passwords. The same template applies cleanly to switches that were upgraded to the same release. A switch built by zero-touch provisioning never appears in monitoring.
Why: From 26.1, restricted insecure commands are refused unless the device is in insecure mode. An upgraded switch that already had those commands put itself into insecure mode automatically. A factory-fresh switch has no startup config, so it starts in secure mode and refuses them.
Prevent it: Fix the template rather than adding system mode insecure to it: SSH only, HTTPS or no web server, SNMPv3 with SHA and AES, SCP or SFTP for file transfer, scrypt secrets and SHA-256 NTP keys. Test the template on a switch with no startup config before you roll it out.

show system security mode
show system insecure configuration
show system insecure profile

Security holeIOS / IOS XE

The upgrade quietly put the switch in insecure mode

What you see: After upgrading to IOS XE 26.1 or later, the running config contains system mode insecure. Nobody added it. Everything still works, so nothing gets raised.
Why: If a device boots 26.1 with restricted insecure config, it moves itself into insecure mode so the upgrade does not cause an outage. Telnet, plain HTTP, SNMP communities and the rest keep working until a later release removes them, and insecure mode itself is due to be removed too.
Prevent it: Treat the line as a to-do list. Run show system insecure configuration on every upgraded device, migrate each item to its secure replacement, then remove insecure mode. Add a compliance check that flags system mode insecure, so a device cannot sit in it unnoticed.

show running-config all | include system mode insecure
show system insecure configuration
! after migrating
configure terminal
 no system mode insecure

Causes an outageIOS / IOS XE

next-hop-self on a route reflector does nothing to reflected routes

What you see: Branches behind a hub that is a route reflector learn each other's prefixes, but show bgp marks them inaccessible, or they point at another hub's tunnel address that the branch cannot reach. Branch-to-hub traffic works; branch-to-branch fails.
Why: On IOS XE, neighbor next-hop-self does not change the next hop on routes the router reflects from one iBGP client to another. They keep the original next hop, usually the other branch's tunnel address.
Prevent it: Use next-hop-self all on the reflector, towards the clients and towards the other reflectors, so every route it passes on points back at the hub. On a branch, check a prefix from another branch: the next hop should be the hub's tunnel address.

router bgp 65000
 address-family ipv4
  neighbor BRANCHES next-hop-self all
  neighbor HUBS next-hop-self all
!
show bgp ipv4 unicast 10.20.13.0/24

Causes an outageIOS / IOS XE and NX-OS

BFD timers copied from the data centre flap across internet tunnels

What you see: BGP sessions over IPsec tunnels drop for a few seconds several times a day, often at busy times or during a rekey, then come straight back. The circuits and the tunnels never went down.
Why: Timers like 50 ms x 3 suit dark fibre, where three missing packets really mean a failure. Across the internet, a burst of jitter or loss, or a busy CPU during an IPsec rekey, can lose three packets in 150 ms. BFD declares the neighbour down and takes BGP with it.
Prevent it: Over the internet, start at 1 second x 3 with echo mode off, and only go faster after a week of clean history. Keep BGP timers as a slower backstop, and alert on every BFD flap rather than only on long outages.

interface Tunnel1
 bfd interval 1000 min_rx 1000 multiplier 3
 no bfd echo
!
show bfd neighbors details
show logging | include BFD

Quietly wrongIOS / IOS XE and NX-OS

Traffic that leaves by the SVI it arrived on loses 1% under load

What you see: Small, steady packet loss through a core switch, worse at busy times. Voice breaks up while data seems fine. Pings to one address in a subnet are clean while a neighbouring address loses packets. The interfaces show no errors.
Why: When a switch routes a packet back out of the interface it arrived on, IOS sends the sender an ICMP redirect. On many Catalyst platforms that means the packet is forwarded by the CPU, not in hardware. Firewalls and routers ignore redirects, so every packet in the flow is punted, and under load the CPU queue overflows.
Prevent it: Add no ip redirects to every SVI where traffic can come in and go out the same way: transit VLANs between firewalls and routers, HSRP pairs, router-on-a-stick. Put it in the template, then check the redirect counter stops rising.

interface Vlan100
 no ip redirects
!
show ip traffic | include redirects
show ip interface Vlan100 | include redirects
show platform cpu packet statistics | include L3 Fwd

Quietly wrongIOS / IOS XE

A time-range in a redistribution route-map never takes effect

What you see: Office-hours and out-of-hours BGP communities, or a redistribution meant to run only in a maintenance window, stay stuck on whatever applied when the route last changed.
Why: Redistribution is event-driven (for BGP since IOS 12.3(17)T). A route is checked against the route-map again only when it is added, withdrawn or changed in the routing table. The time-range flipping is not such an event.
Prevent it: Turn time into reachability: an IP SLA probe through a time-based ACL, two tracked static routes and redistribution by tag. Or have EEM or kron run clear ip route for the prefix just after each boundary.

show time-range
show ip bgp 198.51.100.0/24 | include Community
! the full IP SLA version is on Routing Tricks

Causes an outageIOS / IOS XE

An EIGRP route says "FD is Infinity" and never reaches the routing table

What you see: In named-mode EIGRP the neighbour advertises the prefix and the topology table lists it with FD is Infinity, but show ip route has nothing.
Why: Wide metrics scale bandwidth and delay by 65,536, which gives a 64-bit metric. The routing table holds 32 bits. IOS divides by the RIB scale (128 by default) before installing, and if the result is still above 4,294,967,295 the route is unusable. A very slow link in the path, or a low seed bandwidth on redistribution, gets there.
Prevent it: Fix the cause first: a realistic seed metric on redistribution and no tiny bandwidth values. If that is not possible, raise metric rib-scale (1 to 255) until the scaled metric fits.

router eigrp CORE
 address-family ipv4 unicast autonomous-system 100
  metric rib-scale 160
!
show eigrp address-family ipv4 100 topology 198.51.100.0/24

Causes an outageIOS / IOS XE

BGP through NAT never establishes

What you see: Two eBGP routers peer through a router or firewall doing static NAT. The session sits in Active, and every fix reveals another fault.
Why: TCP MD5 signs the IP addresses, which NAT rewrites, so authentication always fails. The peers also no longer look directly connected to BGP, and the translated address needs a route.
Prevent it: Remove the MD5 password on this session or carry it in a tunnel that NAT never touches, configure ebgp-multihop on both ends, and add add-route to the outside static NAT.

ip nat outside source static 203.0.113.3 192.0.2.3 add-route
!
router bgp 64500
 neighbor 192.0.2.3 remote-as 64501
 neighbor 192.0.2.3 ebgp-multihop 2

Quietly wrongIOS / IOS XE

The lowest MED loses, and a session reset changes the winner

What you see: A neighbouring AS sends a prefix over several links with different MEDs to steer you. Sometimes you use the link they asked for and sometimes not, and clearing a session changes the answer.
Why: IOS compares paths in pairs, newest first, and compares MED only between two paths from the same AS. A path from a different AS in the middle of the list stops MED being compared across the whole group, so arrival order decides.
Prevent it: Enable bgp deterministic-med on every router in your AS, so paths are grouped by AS and compared on MED first. Add bgp bestpath compare-routerid so the last tie-break is the router ID rather than the oldest path.

router bgp 64500
 bgp deterministic-med
 bgp bestpath compare-routerid
!
show ip bgp 198.51.100.0/24

Causes an outageIOS XR

IOS XR with no loopback: BGP never starts

What you see: Neighbours are configured on an IOS XR router and none of them come up. show bgp summary shows a router ID of 0.0.0.0.
Why: IOS XR takes the BGP router ID only from bgp router-id or a loopback address. Unlike IOS, it never falls back to a physical interface address.
Prevent it: Set bgp router-id on every XR router, and give it a loopback in the saved configuration as well.

router bgp 64500
 bgp router-id 192.0.2.1
!
show bgp summary

Quietly wrongIOS / IOS XE

An extended ACL in a distribute-list means two different things

What you see: You filter routes with an extended ACL, expecting it to match prefix and mask, and the wrong routes are dropped, or all of them.
Why: In BGP, and in route-maps, an extended ACL's source field is the network and its destination field is the mask. In an IGP distribute-list in, the source field is the router that sent the update and the destination field is the network.
Prevent it: Use prefix-lists for prefix and mask. Keep extended ACLs in an IGP distribute-list for what they are good at: accepting a route from one neighbour and not another on a shared segment.

! BGP or route-map: exactly 192.0.2.0/24
access-list 101 permit ip host 192.0.2.0 host 255.255.255.0
! IGP distribute-list in: 192.0.2.0 only from 198.51.100.2
access-list 102 permit ip host 198.51.100.2 host 192.0.2.0

Quietly wrongIOS / IOS XE and NX-OS

Two ASBRs, and every router uses the far one

What you see: Two routers redistribute the same external routes into OSPF. You expected each site to use its nearest exit, and the whole domain funnels to one ASBR.
Why: E2 routes are compared on the external metric alone. The cost to reach the ASBR (the forward metric) only breaks a tie, so a seed metric lower by one beats any internal distance.
Prevent it: For nearest-exit routing, set the same E2 metric on every ASBR so the forward metric decides. Use different metrics only for a deliberate primary and backup, or use E1.

router ospf 1
 redistribute bgp 64500 subnets metric 20 metric-type 2
!
show ip route 198.51.100.0
! metric 20, type extern 2, forward metric N

Quietly wrongIOS / IOS XE

A default static mroute overrides every unicast route for RPF

What you see: Multicast from some sources fails the RPF check or arrives on the wrong link, although the unicast table has a more specific route the right way.
Why: The router finds the best match in each RPF source separately (unicast, static mroutes, MBGP), then chooses between them by administrative distance, not by prefix length. Static mroutes default to distance 0, so a default mroute beats everything. MBGP takes BGP's distances, so an iBGP multicast route loses to an OSPF default.
Prevent it: Add a distance-255 mroute for any range that must keep using unicast RPF, give other static mroutes a distance when they should lose, and check each source with show ip rpf.

ip mroute 198.51.100.0 255.255.255.0 Null0 255
ip mroute 0.0.0.0 0.0.0.0 Tunnel0
!
show ip rpf 198.51.100.10

Quietly wrongIOS / IOS XE

IPv6 redistribution leaves the connected links out

What you see: RIPng or EIGRP for IPv6 is redistributed into OSPFv3. The routes it learned arrive at the far end; the links it runs on do not.
Why: In IPv4, redistributing a protocol includes the connected networks it runs on. In IPv6 it does not: only the routes the protocol learned are taken.
Prevent it: Add include-connected to the redistribute command. It takes only the connected prefixes that protocol runs on, unlike a blanket redistribute connected.

router ospfv3 1
 address-family ipv6 unicast
  redistribute rip RIPNG include-connected

Quietly wrongIOS / IOS XE

IPsec transport mode quietly becomes tunnel mode

What you see: mode transport is configured to save overhead on GRE over IPsec, and the SAs come up in tunnel mode anyway. Your MTU sums are 20 bytes out.
Why: Transport mode is used only when the crypto ACL covers traffic between the two peer addresses and nothing else. Any other address in the ACL makes the peers negotiate tunnel mode.
Prevent it: Match only the GRE traffic between the two peers in the crypto ACL, and add require so phase 2 fails loudly instead of changing mode.

crypto ipsec transform-set TS esp-aes 256 esp-sha256-hmac
 mode transport require
ip access-list extended GRE-ONLY
 permit gre host 192.0.2.1 host 198.51.100.1
!
show crypto ipsec sa | include settings

Security holeIOS / IOS XE

Any certificate from your CA can bring up a tunnel

What you see: Routers authenticate IKE with certificates from the internal CA. A laptop, a printer or a lab box holding a valid certificate from the same CA completes IKE too.
Why: By default the router accepts any valid, unrevoked certificate from a trusted CA. Nothing checks what the certificate was issued for.
Prevent it: Add a certificate map to the trustpoint that matches fields such as subject-name or issuer-name, so only your routers' certificates are accepted.

crypto pki certificate map VPN-PEERS 10
 subject-name co ou=network-devices
crypto pki trustpoint CORP-CA
 match certificate VPN-PEERS

Security holeIOS / IOS XE

A MAC ACL does not stop IP traffic

What you see: You apply a MAC access list to keep one host away from another, test it, and IP traffic still flows.
Why: On Catalyst switches MAC ACLs match only non-IP frames. IPv4 and IPv6 packets are checked against IP ACLs only.
Prevent it: Use an IP port ACL or a VLAN map for IP traffic, and keep MAC ACLs for non-IP protocols.

ip access-list extended BLOCK-HOST
 deny ip host 192.0.2.10 host 192.0.2.20
 permit ip any any
interface GigabitEthernet1/0/10
 ip access-group BLOCK-HOST in

Quietly wrongIOS / IOS XE

NTP access control locks the master out of its own clock

What you see: After restricting who may use the router for time, the router reports itself unsynchronised and its clients stop trusting it.
Why: Once any ntp access-group has an ACL, every source not permitted by one of them is denied. A router running ntp master synchronises to an internal reference clock with a 127.127.x.x address, and that is denied too.
Prevent it: Permit the internal reference clock in the peer ACL along with any real peers, and give clients serve-only. Read the clock's address from show ntp associations on your release.

show ntp associations
! permit the 127.127.x.x address it lists
access-list 10 permit <reference-clock-address>
access-list 10 permit 192.0.2.2
access-list 20 permit 198.51.100.0 0.0.0.255
ntp access-group peer 10
ntp access-group serve-only 20

Causes an outageASA and Firepower

ASA ARP inspection with no-flood cuts off everything you did not list

What you see: ARP inspection is turned on in a transparent ASA to stop gateway spoofing. A few minutes later hosts lose each other as their ARP entries age out, and the ASA logs %ASA-3-322003 ARP inspection check failed.
Why: ARP inspection checks ARP only against the static ARP entries on the ASA. With flood (the default), ARP that matches no entry is forwarded. With no-flood, it is dropped.
Prevent it: Use flood, with static entries for the gateways and servers worth protecting. Use no-flood only when every device on the segment has a static entry.

arp inside 192.0.2.1 0000.5e00.5301
arp-inspection inside enable flood
arp-inspection outside enable flood
!
show arp-inspection

Quietly wrongIOS / IOS XE

Spanning-tree port-priority set on the wrong switch

What you see: You lower port-priority on an access switch's preferred uplink, and it keeps using the other one.
Why: Port priority is a tie-breaker read by the neighbour. With two equal paths to the root through the same upstream switch, the downstream switch compares the upstream bridge ID, then the upstream port ID. Its own port-priority is not part of the comparison.
Prevent it: Set port-priority on the upstream switch, on the port facing the switch you want to influence. To steer the local switch's own choice, change the port cost instead.

! on the upstream (designated) switch
interface GigabitEthernet1/0/1
 spanning-tree vlan 10 port-priority 64
!
show spanning-tree vlan 10

Quietly wrongIOS / IOS XE

Shaped to the contract, policed anyway

What you see: You shape to the provider's CIR, and their ingress policer still drops traffic in bursts.
Why: A shaper sends its committed burst, plus any excess burst, at the start of each interval. The policer measures over its own interval and burst sizes. If the shaper's interval is longer, or its bursts are bigger than the policer allows, the policer drops the difference.
Prevent it: Ask the provider for the policer's CIR, Bc and Be. Keep the shaper's interval no longer than the policer's, and size the shaper so its Bc plus Be fits within the policer's Be.

policy-map WAN-SHAPE
 class class-default
  shape average 50000000 500000 0
  service-policy WAN-QOS
!
show policy-map interface GigabitEthernet0/0/1 output

Causes an outageIOS / IOS XE

An ABR that has lost the backbone keeps sending a default into its stub area

What you see: A distribution switch loses every link to the core. The access switches keep sharing their default route across both distribution switches, and about half the traffic leaving the block is dropped.
Why: An ABR sends a default into a stub or totally stubby area while it has any interface up in area 0, whether or not it can reach the backbone. A loopback in area 0 never goes down. The switch can't fall back on its partner's default either, because an ABR ignores summary routes it hears from a non-backbone area.
Prevent it: Keep loopbacks and any other always-up interfaces on the distribution switches out of area 0, in the block's own area. Test it before sign-off: shut every core link on one distribution switch and check the default route on an access switch.

interface Loopback0
 ip ospf 1 area 10
!
show ip ospf interface brief
! on an access switch
show ip route 0.0.0.0

Quietly wrongIOS / IOS XE

EIGRP access switches that aren't stubs slow every failure down

What you see: In a routed access campus on EIGRP, a lost route makes the distribution switches query every access switch before they converge. Recovery takes longer than it should, and an access switch can end up carrying traffic between the two distribution switches.
Why: Without eigrp stub, every access switch is a possible path to anywhere. It gets queried for every lost route and can be used for transit. With hundreds of access switches that's a lot of queries, and a lot of chances for one to get stuck in active.
Prevent it: Make every access switch eigrp stub connected, send it a default route and nothing else, and summarise the block towards the core.

router eigrp 100
 eigrp stub connected
! on the distribution switch
ip prefix-list DEFAULT-ONLY seq 5 permit 0.0.0.0/0
router eigrp 100
 distribute-list prefix DEFAULT-ONLY out TenGigabitEthernet1/0/1
!
show ip eigrp neighbors detail | include Stub

Quietly wrongIOS / IOS XE and NX-OS

Communities you set are never sent to the neighbour

What you see: You tag routes with no-export, a blackhole community or the provider's traffic-engineering communities. The route map matches and sets them, but the far end behaves as if nothing was tagged.
Why: IOS, IOS XE and NX-OS don't send communities to a BGP neighbour unless you tell them to, neighbour by neighbour. The route map does its job and the community is stripped on the way out.
Prevent it: Add send-community (standard, extended or both) to every neighbour that should receive them, or to the peer group or template. Check what the neighbour actually receives, not what your table shows.

router bgp 64500
 address-family ipv4 unicast
  neighbor 198.51.100.1 send-community both
  neighbor 198.51.100.1 route-map TO-ISP out
!
show bgp ipv4 unicast neighbors 198.51.100.1 advertised-routes
show bgp ipv4 unicast 192.0.2.0/24 | include Community

Causes an outageIOS / IOS XE

A switch reloads and DAI locks out everyone who already had an address

What you see: After a power cut or an upgrade, users on one access switch can't reach their gateway. People who restart their PCs get back on; everyone else stays off until their lease renews.
Why: DHCP snooping keeps its bindings in memory. After a reload the table is empty, so Dynamic ARP Inspection and IP Source Guard drop every host that didn't send a fresh DHCP request.
Prevent it: Save the bindings with the DHCP snooping database agent, to flash or a server, so they come back after a reload. Include it in your DAI build standard and test a reload before sign-off.

ip dhcp snooping database flash:dhcp-snoop.db
ip dhcp snooping database write-delay 60
!
show ip dhcp snooping database
show ip dhcp snooping binding | count dhcp

Quietly wrongIOS / IOS XE

DHCP snooping drops requests from devices behind a bridge or repeater

What you see: Clients behind a wireless bridge, a workgroup bridge or some relay devices never get an address once DHCP snooping is on. Directly connected PCs on the same VLAN are fine.
Why: Snooping checks that a request's source MAC matches the client hardware address inside the DHCP packet, and that check is on by default. A device that forwards the request with its own MAC fails it.
Prevent it: Find out what's in the path first. If the bridge is legitimate, turn the check off for the switch rather than disabling snooping, and watch the drop counters.

no ip dhcp snooping verify mac-address
!
show ip dhcp snooping statistics detail

Quietly wrongIOS / IOS XE

An interface is in a different OSPF area from the one its network statement says

What you see: The network statement puts an interface in area 0, but show ip ospf interface puts it in area 10 and the neighbour never comes up.
Why: An ip ospf area command on the interface beats any network statement under the process, without a warning. It's often left behind from an earlier change or a template.
Prevent it: Pick one method for a router and stick to it. When an adjacency is stuck, check the interface config as well as the process.

show running-config interface GigabitEthernet0/1 | include ospf
show ip ospf interface brief

Causes an outageIOS / IOS XE

The EIGRP keys look identical on both routers, and authentication still fails

What you see: Both key chains show the same key string and key number, yet the neighbour never comes up and debug shows authentication failures.
Why: Everything after key-string is part of the key, including a trailing space. Paste the key with a space on the end and the hash never matches, and show running-config doesn't make it obvious.
Prevent it: Type key strings by hand or check the pasted line for trailing spaces. Re-enter the key on one side if in doubt.

key chain EIGRP-KEYS
 key 1
  key-string <no trailing space>
!
show key chain
debug eigrp packets terse

Quietly wrongIOS / IOS XE

Changing bandwidth to steer EIGRP moved OSPF and QoS as well

What you see: You lowered the bandwidth on a link to make EIGRP prefer the other path. Afterwards OSPF costs on that link changed, and QoS classes set in percent got smaller.
Why: The bandwidth command is read by OSPF's cost calculation and by any QoS policy written in percentages, not just EIGRP. Delay is only used by EIGRP.
Prevent it: Steer EIGRP with the delay command and leave bandwidth at the real circuit speed.

interface GigabitEthernet0/2
 delay 2000
!
show interfaces GigabitEthernet0/2 | include BW|DLY
show ip eigrp topology 192.0.2.0/24

Causes an outageIOS / IOS XE

Strict uRPF on a router with two ISPs drops good traffic

What you see: After uRPF goes on the internet interfaces, some sites stop working, and which ones changes when BGP changes its mind.
Why: Strict mode drops a packet unless the route back to its source points out of the interface it arrived on. With two providers, return traffic often arrives on the link that isn't your best path to the sender.
Prevent it: Use loose mode on multihomed edges, or strict mode only where routing is symmetric or every path is equal cost.

interface GigabitEthernet0/0
 ip verify unicast source reachable-via any
!
show ip interface GigabitEthernet0/0 | include verify|drop

Causes an outageIOS / IOS XE and ISE

ISE rejects a switch it knows, because the RADIUS packets arrive NATed

What you see: 802.1X and MAB fail from one site. ISE logs show the request from an unknown network device, although the switch is defined with the right NAS-IP-Address.
Why: ISE identifies a network device by the source address of the RADIUS packet, not the NAS-IP-Address attribute inside it. Source NAT on the way changes the address. CoA from ISE also goes to the address it knows, so it fails too.
Prevent it: Don't NAT RADIUS between network devices and ISE. Source RADIUS from a fixed interface on the switch and define that address in ISE.

ip radius source-interface Vlan99
!
show aaa servers
test aaa group ISE-GROUP testuser <password> new-code

Quietly wrongISE

Guests log in on the portal and keep their old address

What you see: After central web authentication, ISE moves the guest to a different VLAN. The guest shows as authorised but can't reach anything.
Why: The client got its address before it logged in. Nothing tells it to renew after the VLAN changes, so it keeps an address from the old subnet.
Prevent it: Don't change the VLAN after web authentication. Change access with a downloadable ACL or an SGT instead.

show access-session interface GigabitEthernet1/0/10 details
show ip dhcp snooping binding interface GigabitEthernet1/0/10

Security holeIOS / IOS XE and ISE

A downloadable ACL from ISE quietly replaces the port ACL

What you see: Ports have a static ACL to block a sensitive subnet. After 802.1X goes live, authenticated users can reach it.
Why: When ISE sends a downloadable ACL, it replaces the ACL configured on the port. Anything the port ACL blocked is now up to the dACL.
Prevent it: Put every restriction you need in the dACL, or enforce it further upstream. Check the ACL applied to a live session, not the port config.

show access-session interface GigabitEthernet1/0/10 details | include ACL
show ip access-lists interface GigabitEthernet1/0/10

Causes an outageISE

An ISE upgrade goes wrong because the admin nodes swap roles halfway through

What you see: During an ISE upgrade the secondary admin node promotes itself, and the deployment ends up split between versions.
Why: Automatic PAN failover sees the primary admin node go quiet during the upgrade and promotes the secondary.
Prevent it: Turn off PAN auto-failover before an upgrade and back on after it. Put it in the upgrade plan.

Quietly wrongIOS / IOS XE

An EEM applet runs, but every command it sends is refused

What you see: The applet triggers and logs that it ran, but the config change or show command never happens.
Why: With AAA command authorisation, the applet's commands go to TACACS+ with no username, and the server refuses them. Also, the applet starts in user EXEC, so it needs enable and configure terminal like a person would.
Prevent it: Give EEM a username that the TACACS+ server authorises, and start applet actions with enable and configure terminal.

event manager session cli username EEM-SVC
!
event manager applet EXAMPLE
 event syslog pattern "LINEPROTO-5-UPDOWN"
 action 1.0 cli command "enable"
 action 2.0 cli command "configure terminal"
!
debug event manager action cli

Causes an outageACI

A new ACI L3Out sticks in OSPF EXSTART

What you see: The OSPF neighbour between a border leaf and an outside router reaches EXSTART or EXCHANGE and goes no further.
Why: ACI L3Out interfaces default to an MTU of 9000. The outside router is at 1500, and OSPF won't finish exchanging databases with a neighbour whose MTU doesn't match.
Prevent it: Set the MTU on the L3Out interface profile to match the router, or raise the router's to match. Don't use mtu-ignore to hide the difference.

! on the outside router
show ip ospf neighbor
show interfaces GigabitEthernet0/1 | include MTU
! on the leaf
show ip ospf neighbors vrf PROD:VRF1

Quietly wrongACI

An APIC cluster of four or six

What you see: A four-controller ACI cluster loses two controllers, and some of the configuration becomes read-only although half the cluster is still up.
Why: APIC data is split into shards with replicas on three controllers. Even-sized clusters don't give a clean majority when half the cluster is lost, and Cisco doesn't recommend them.
Prevent it: Build three, five or seven APICs, and spread them across failure domains. Use a standby APIC if you need a spare.

Causes an outageNX-OS

With asymmetric IRB, routing between VLANs works only to some leaves

What you see: Hosts in different subnets can talk when they're on certain leaves but not others, and traffic in one direction disappears.
Why: Asymmetric IRB routes on the ingress leaf and bridges into the destination VNI, so every leaf needs every VNI it might send to, whether or not it has hosts in it. Symmetric IRB uses a layer 3 VNI and doesn't need that.
Prevent it: Use symmetric IRB with a layer 3 VNI per tenant VRF. If you must use asymmetric, configure every VNI on every leaf.

show nve vni
show bgp l2vpn evpn summary
show vrf VRF1 detail

Quietly wrongIOS / IOS XE

Devices added to the 9800 local MAC filter are still refused

What you see: You add a device to the controller's local MAC filter list and it still fails MAC authentication.
Why: The 9800's local database only matches MACs written as aaaabbbbcccc, with no dots, colons or dashes.
Prevent it: Enter MACs with no separators, and convert any list you import.

username aabbccddeeff mac
!
show wireless client mac-address aabb.ccdd.eeff detail

Quietly wrongIOS / IOS XE

An IPv6 summary on the wrong boundary

What you see: An IPv6 summary leaves some routes out, or covers addresses it shouldn't.
Why: Each hextet is hexadecimal. 2001 is 0x20 and 0x01, so 32 and 1, not 20 and 1. Working the boundary out as if it were decimal gives the wrong prefix length.
Prevent it: Write the hextet that changes in binary before choosing the prefix length. Use the subnet calculator's IPv6 mode to check.

Causes an outageIOS / IOS XE

Turning on TrustSec on an uplink drops it

What you see: You add cts manual to a switch uplink over SSH and lose the session. The link bounces while the two ends agree.
Why: Enabling TrustSec on an interface resets the link. If that's your path to the switch, you lose it until both ends match. On subinterfaces it has to be on the physical interface and the subinterface.
Prevent it: Configure it from a path that doesn't use that link, one link at a time, both ends in the same change window.

interface TenGigabitEthernet1/1/1
 cts manual
  policy static sgt 2 trusted
!
show cts interface TenGigabitEthernet1/1/1

Causes an outageNX-OS

Enabling NPV on a Nexus erases the configuration

What you see: Someone runs feature npv on a configured Nexus switch, and it reboots with a blank configuration.
Why: On the Nexus 5000 and 6000, and other platforms that work the same way, turning on NPV mode erases the configuration and reloads the switch. Check the configuration guide for yours.
Prevent it: Enable NPV first, before any other configuration. On a live switch, save and copy the config off first and plan an outage.

copy running-config bootflash:pre-npv.cfg
feature npv
!
show npv status