clifford.network / toolbox

free tool

Troubleshooting Flows

Step-by-step checks for the faults that turn up on every network: the swapped switch you cannot reach, the EtherChannel that will not bundle, HSRP that does not fail over, OSPF, EIGRP and BGP that will not come up, NAT that works only sometimes, a backup link that will not take over, and a vPC that will not forward. Each step gives the command, what healthy looks like, and the usual causes when it does not.

How to work any ticket

Ask what changed

Most faults follow a change. Check show logging | include CONFIG.I and show archive log config all.

Bottom up

Cable and port, then VLAN and trunk, then addressing and routing, then services. Do not debug BGP over a broken trunk.

Prove each layer

Run the check, compare it with what good looks like, and only then move up.

Half-split long paths

Test from the middle of the path first, then split the half that fails. How.

Move the problem

Swap the cable, the optic, the port or the device, one at a time. If the fault moves with it, you have found it.

Compare with a baseline

Paste a known-good capture and the current one into the Change Checker.

Fix one thing at a time

Change, test, note it. Two changes at once and you will not know which one fixed it.

Call the vendor early

Stuck? Open the case now, not when the MD asks what Cisco said. Capture the show tech first. Forget being the hero.

All 20 faults at a glance

Every flow written out in full: the symptom, each check in order, what healthy looks like and the usual causes. Pick a fault in the list above to work through it step by step with copyable commands.

A replaced switch cannot be managed, backed up or send syslog

Switching

Users are working again after a hardware swap, but you cannot SSH to the new switch, config backups fail and its logs never arrive.

  1. Uplink is up and talking to the right neighbour
    • show interfaces status | include connected
    • show cdp neighbors

    Healthy: The uplinks are connected and CDP shows the distribution switches on the expected ports.

    Usual causes:

    • Cabled to different ports from the old switch, so the uplink config is on the wrong interfaces.
    • Uplink in an EtherChannel on one side only.
  2. Trunk is up and carries the management VLAN
    • show interfaces trunk

    Healthy: The uplink is trunking, and the management VLAN is in "allowed and active" and "forwarding".

    Usual causes:

    • Port left as access, or DTP did not negotiate a trunk.
    • Management VLAN missing from the allowed list at either end.
    • Native VLAN mismatch.
  3. Management VLAN exists and its SVI is up
    • show vlan brief
    • show ip interface brief | include Vlan

    Healthy: The VLAN is active and the SVI is up/up with the right address.

    Usual causes:

    • VLAN never created, or wiped by VTP.
    • SVI still shutdown, or the address typed on the wrong VLAN.
  4. The switch has a way off its own subnet
    • show running-config | include default-gateway|ip route|ip routing

    Healthy: A default gateway (layer 2 switch) or a default route (ip routing on) pointing at the distribution SVI.

    Usual causes:

    • ip default-gateway set on a switch that has ip routing on, where it is ignored.
    • Gateway address from the old site template.
  5. VTP has not changed the VLAN database
    • show vtp status

    Healthy: Mode transparent or off, or the expected domain with a sensible revision.

    Usual causes:

    • A spare switch in server mode with a higher revision has overwritten the VLANs.
  6. Services use the right source
    • show running-config | include source-interface|logging host|tftp

    Healthy: Logging, TFTP and AAA source from the management SVI, so server ACLs recognise the switch.

    Usual causes:

    • No source-interface, so packets leave from an address the servers do not expect.
  7. VTY lines will accept you
    • show running-config | section line vty
    • show access-lists
    • show ip ssh

    Healthy: transport input ssh, an access-class that includes your subnet, and SSH enabled.

    Usual causes:

    • No RSA key, so SSH is off.
    • access-class not updated for the management subnet.
    • aaa new-model with no local user and no reachable TACACS+.

An EtherChannel will not bundle

Switching

A new or changed port-channel shows members as I, s or D, or one side is up and the other is not.

  1. Read the flags
    • show etherchannel summary
    • show port-channel summary (NX-OS)

    Healthy: The port-channel is (SU) and every member is (P).

    Usual causes:

    • (I) stand-alone: the far end is not running LACP or PAgP.
    • (s) suspended: the members do not match each other or the far end.
    • (D) down: a physical problem on that member.
  2. Both ends use the same negotiation
    • show running-config interface Gi1/0/49
    • show lacp neighbor
    • show running-config interface ethernet1/49 (NX-OS)
    • show lacp neighbor (NX-OS)

    Healthy: channel-group mode active at both ends, or active and passive, and LACP neighbours listed on every member.

    Usual causes:

    • mode on at one end and LACP at the other.
    • Two passive ends: neither starts negotiation.
    • Members on the far side are in a different channel-group number from each other.
  3. Every member is configured the same
    • show interfaces trunk
    • show interfaces status
    • show port-channel compatibility-parameters (NX-OS)

    Healthy: Same speed, duplex, switchport mode, allowed VLANs and native VLAN on every member and on the port-channel itself.

    Usual causes:

    • One member configured on its own after the bundle was built.
    • Allowed VLAN list changed on one member instead of on the port-channel interface. The log says CANNOT_BUNDLE2 ... (vlan mask is different).
  4. Nothing has error-disabled the bundle
    • show interfaces status err-disabled
    • show logging | include ETHCNTR|EC-|PM-4
    • show interface status err-disabled (NX-OS)
    • show logging logfile | include ETHPORT|ETH_PORT_CHANNEL (NX-OS)

    Healthy: No err-disabled members.

    Usual causes:

    • EtherChannel guard has disabled the "on" side of a misconfigured pair.

A client gets no address, or a 169.254 address

Switching

A PC on a working port says it has no network, or picks up a self-assigned 169.254 address.

  1. Port is up, in the right VLAN and has learnt the MAC
    • show interfaces status | include Gi1/0/10
    • show mac address-table interface Gi1/0/10

    Healthy: connected, the expected access VLAN, and the client MAC learnt in that VLAN.

    Usual causes:

    • Wrong access VLAN, or the VLAN does not exist on this switch.
    • Port err-disabled or administratively down.
  2. Spanning tree lets it forward straight away
    • show spanning-tree interface Gi1/0/10 portfast

    Healthy: PortFast enabled, port forwarding.

    Usual causes:

    • No PortFast, so the client gives up on DHCP during the 30 seconds of listening and learning.
  3. Port security or 802.1X is not holding it
    • show port-security interface Gi1/0/10
    • show access-session interface Gi1/0/10 details

    Healthy: No violations, and the session authorised in the right VLAN.

    Usual causes:

    • Port security maximum reached.
    • 802.1X failing and no MAB or fallback VLAN.
    • Authorised into the guest or auth-fail VLAN.
  4. The VLAN reaches the gateway
    • show interfaces trunk

    Healthy: The VLAN is allowed and forwarding on every trunk up to the switch that holds its SVI.

    Usual causes:

    • VLAN pruned from an uplink trunk.
  5. The gateway relays DHCP
    • show running-config interface Vlan10
    • ping 10.0.60.10 source Vlan10

    Healthy: ip helper-address points at the DHCP server, and the server answers a ping sourced from the SVI.

    Usual causes:

    • No helper address on the SVI.
    • Server has no route back to the client subnet.
    • Firewall between them blocks UDP 67 and 68.
  6. DHCP snooping is not dropping the offers
    • show ip dhcp snooping
    • show ip dhcp snooping statistics

    Healthy: The uplink towards the server is trusted.

    Usual causes:

    • Uplink not trusted, so offers are dropped.
    • Option 82 inserted by the access switch and rejected by the relay. Use ip dhcp relay information trust-all on the relay, or stop inserting it.
  7. The server has a pool for this subnet
    • show ip dhcp pool
    • debug ip dhcp server packet

    Healthy: A pool whose network contains the relay address (the SVI the client is on), and the debug shows DISCOVER then OFFER.

    Usual causes:

    • A typo in the pool network, so the request matches no pool and is dropped with no log message.
    • The relay SVI address sits outside every pool.
    • Exclusions (a global command) covering the whole range.
  8. The scope has addresses left
    • show ip dhcp pool
    • show ip dhcp conflict

    Healthy: Free addresses in the pool, or on the Windows or other DHCP server.

    Usual causes:

    • Scope exhausted.
    • Conflicts marking addresses unusable.

The network is slow or drops in and out: a loop or spanning-tree churn

Switching

Several VLANs go slow at once, MAC addresses flap between ports, or CPU spikes on the switches.

  1. Look for MAC flapping
    • show logging | include MACFLAP|LOOP|BPDU
    • show logging logfile | include MAC_MOVE|L2FM|BPDU (NX-OS)

    Healthy: No MACFLAP messages.

    Usual causes:

    • The same MAC moving between two ports: a loop through those ports or the devices behind them.
  2. Find where topology changes come from
    • show spanning-tree detail | include ieee|occurr|from|is exec
    • show spanning-tree detail | include ieee|occurr|from|is exec (NX-OS)

    Healthy: Few topology changes, the last one some time ago.

    Usual causes:

    • Changes every few seconds. The "from" interface leads towards the cause. Follow it switch by switch.
  3. The root is where you designed it
    • show spanning-tree root
    • show spanning-tree root (NX-OS)

    Healthy: The core or distribution switch you chose is root for each VLAN.

    Usual causes:

    • An access switch or a forgotten lab switch has become root.
  4. Edge ports are protected
    • show spanning-tree summary
    • show running-config | include bpduguard|portfast
    • show spanning-tree summary (NX-OS)
    • show running-config | include bpduguard|port type edge (NX-OS)

    Healthy: PortFast and BPDU Guard on user ports, root guard on downlinks.

    Usual causes:

    • Someone has connected a small switch or bridged two wall ports.
  5. No one-way links
    • show udld neighbors
    • show interfaces counters errors
    • show udld neighbors (NX-OS)
    • show interface counters errors (NX-OS)

    Healthy: UDLD bidirectional on fibre uplinks, no errors.

    Usual causes:

    • A fibre transmitting in one direction only, so a blocked port starts forwarding. Use UDLD aggressive and loop guard.

HSRP does not fail over, or both routers are active

Gateway

Clients lose their gateway when one switch reboots, or both switches claim to be the active router.

  1. Read the state on both switches
    • show standby brief
    • show hsrp brief (NX-OS)

    Healthy: One Active, one Standby, and each lists the other as its partner.

    Usual causes:

    • Both Active: they cannot hear each other's hellos.
    • Standby shown as "unknown".
  2. They can hear each other
    • show interfaces trunk
    • show standby Vlan10
    • show interface trunk (NX-OS)
    • show hsrp interface vlan 10 (NX-OS)

    Healthy: The VLAN is carried on the link between the two switches, and the group number, version and authentication match.

    Usual causes:

    • VLAN missing from the inter-switch trunk.
    • Group number or version different at each end.
    • Authentication key or mode mismatch.
    • An ACL dropping the hellos (224.0.0.2 for version 1, 224.0.0.102 for version 2).
  3. Preempt and tracking do what you expect
    • show standby Vlan10 | include Preempt|Priority|track
    • show track
    • show hsrp interface vlan 10 | include Preempt|Priority|Track (NX-OS)
    • show track (NX-OS)

    Healthy: preempt with a delay on the intended active router, and any tracked object Up.

    Usual causes:

    • Preempt missing, so the primary never takes back over.
    • Decrement too small to drop the priority below the standby.
  4. Spanning tree agrees with HSRP
    • show spanning-tree root
    • show spanning-tree root (NX-OS)

    Healthy: The STP root for the VLAN is the HSRP active switch.

    Usual causes:

    • Different switches, so traffic crosses the inter-switch link twice.
  5. Clients see the virtual MAC
    • show ip arp | include 0000.0c
    • show ip arp | include 0000.0c (NX-OS)

    Healthy: The gateway address maps to the HSRP virtual MAC (0000.0c07.acXX for v1, 0000.0c9f.fXXX for v2).

    Usual causes:

    • Hosts hold the physical MAC of one switch, often because the gateway was configured as the real interface address.

OSPF neighbours do not form, or get stuck

Routing

show ip ospf neighbor is empty, or neighbours sit in INIT, EXSTART or EXCHANGE.

  1. See which state it reaches
    • show ip ospf neighbor
    • show ip ospf neighbors (NX-OS)

    Healthy: FULL (or 2WAY to DROTHERs on a broadcast segment).

    Usual causes:

    • Nothing at all: hellos are not getting through or are being ignored.
    • INIT: hellos one way only.
    • EXSTART or EXCHANGE: almost always an MTU mismatch.
  2. OSPF is running on the interface and not passive
    • show ip ospf interface brief
    • show ip ospf interface brief (NX-OS)

    Healthy: The interface is listed in the right area and process.

    Usual causes:

    • Interface missed by the network statement.
    • passive-interface default without a matching no passive-interface.
  3. The parameters that must match, do
    • show ip ospf interface Gi0/0/1
    • show ip ospf interface ethernet1/1 (NX-OS)

    Healthy: Same area, subnet and mask, hello and dead timers, network type and authentication at both ends.

    Usual causes:

    • Different area IDs.
    • Hello 10 at one end and 30 at the other.
    • Stub flag set at one end only.
    • Authentication on one end, or different keys.
  4. MTU matches
    • show interfaces Gi0/0/1 | include MTU
    • show interface ethernet1/1 | include MTU (NX-OS)

    Healthy: Identical MTU at both ends.

    Usual causes:

    • Different MTU. Match it rather than using ip ospf mtu-ignore.
  5. Router IDs are unique
    • show ip ospf | include ID
    • show logging | include DUP_RTRID
    • show ip ospf | include Router ID (NX-OS)
    • show logging logfile | include DUP (NX-OS)

    Healthy: Every router has its own router ID.

    Usual causes:

    • A copied config with the same router ID. Change it and clear the process in a window.
  6. Nothing filters OSPF packets
    • show ip interface Gi0/0/1 | include access list
    • show running-config interface ethernet1/1 | include access-group (NX-OS)

    Healthy: No inbound ACL, or one that permits OSPF (protocol 89) and 224.0.0.5 and 224.0.0.6.

    Usual causes:

    • A new edge or CoPP ACL that forgot OSPF.

Neighbours are up but routes are missing

Routing

Adjacencies are fine, yet a subnet is not in the table, often straight after redistribution or a migration.

  1. Is the route in the database but not the table?
    • show ip ospf database | include 10.20.0.0
    • show ip route 10.20.0.0
    • show ip ospf database | include 10.20.0.0 (NX-OS)
    • show ip route 10.20.0.0 (NX-OS)

    Healthy: The LSA exists and the route is installed.

    Usual causes:

    • In the database, not the table: a distribute-list in, a better route from another protocol, or the forwarding address is unreachable.
    • Not in the database at all: the source router is not advertising it.
  2. Redistribution is complete
    • show running-config | section router
    • show ip protocols
    • show running-config | section router (NX-OS)
    • show ip ospf (NX-OS)

    Healthy: redistribute with a route-map you expect, a seed metric where needed, and subnets on older IOS.

    Usual causes:

    • Older IOS without the subnets keyword redistributes only classful networks into OSPF.
    • Redistribution into EIGRP or RIP with no metric, so nothing is sent.
    • A route-map with only deny entries, which denies everything.
  3. The area type allows it
    • show ip ospf | include Area|stub|nssa
    • show ip ospf | include Area|stub|nssa (NX-OS)

    Healthy: External routes go nowhere near a stub area, or the area is an NSSA.

    Usual causes:

    • External (type 5) routes are blocked by a stub area, by design.
  4. A summary is not hiding it
    • show running-config | include area .* range|summary-address
    • show running-config | include range|summary-address (NX-OS)

    Healthy: Summaries match the design.

    Usual causes:

    • An area range or summary covering the subnet, with a hole in it.
  5. Mutual redistribution is not looping
    • show ip route 10.20.0.0 longer-prefixes
    • show ip route | include EX|E2
    • show ip route 10.20.0.0/16 longer-prefixes (NX-OS)
    • show ip route | include ospf-1 type-2 (NX-OS)

    Healthy: Routes are learnt from their home protocol.

    Usual causes:

    • Routes redistributed back into their source protocol. Tag on the way out and deny the tag on the way back in.

EIGRP neighbours drop or routes are missing

Routing

EIGRP peers do not come up, flap, or come up without the routes you expect.

  1. Check the neighbour table
    • show ip eigrp neighbors

    Healthy: Every expected neighbour, with a stable uptime and a Q count of 0.

    Usual causes:

    • Missing neighbour.
    • Uptime keeps resetting: retransmissions, a one-way link or a mismatch.
  2. Basics that must match
    • show ip protocols
    • show ip eigrp interfaces

    Healthy: Same AS number, same K values, interfaces in the same subnet, not passive.

    Usual causes:

    • AS number typo.
    • Changed K values on one router.
    • Interface missing from the network statement or passive.
  3. Authentication matches
    • show key chain
    • show running-config | section key chain

    Healthy: Same key ID and key string, and key lifetimes valid now on both routers.

    Usual causes:

    • Key IDs different at each end.
    • A key lifetime that has expired or not yet started, often because NTP is not set.
  4. Stub and summary settings
    • show running-config | section router eigrp

    Healthy: Stub routers only at the edge, with the right advertise options.

    Usual causes:

    • eigrp stub receive-only on a router that needs to advertise its LAN.
    • Auto-summary on older IOS summarising discontiguous networks.
    • Split horizon on a hub interface hiding spokes from each other.

A BGP session will not establish

Routing

The neighbour sits in Idle, Connect or Active and never reaches Established.

  1. Read the state
    • show ip bgp summary
    • show bgp ipv4 unicast summary (NX-OS)

    Healthy: A prefix count in the State/PfxRcd column.

    Usual causes:

    • Active: trying to open TCP and failing. Usually reachability, source address or a filter.
    • Idle: not trying, often shut down, or reset by maximum-prefix.
    • Idle (Admin) or Idle (PfxCt): administratively shut or over the prefix limit.
  2. The peer address is reachable from the right source
    • ping 203.0.113.1 source Gi0/0/0
    • show running-config | include neighbor 203.0.113.1
    • ping 203.0.113.1 source 203.0.113.2 (NX-OS)
    • show running-config bgp (NX-OS)

    Healthy: Ping works from the address the peer expects, and update-source is set when peering between loopbacks.

    Usual causes:

    • Peering to a loopback without update-source.
    • eBGP to a loopback without ebgp-multihop or disable-connected-check.
  3. AS numbers and TCP 179 are right
    • show ip bgp neighbors 203.0.113.1 | include BGP state|remote AS|Last reset
    • show bgp ipv4 unicast neighbors 203.0.113.1 | include BGP state|remote AS|Last reset (NX-OS)

    Healthy: The remote AS matches what the provider gave you.

    Usual causes:

    • Wrong remote-as. The log shows a bad AS notification.
    • A firewall or ACL blocking TCP 179.
  4. Authentication matches
    • show logging | include BADAUTH|MD5
    • show logging logfile | include BGP|MD5 (NX-OS)

    Healthy: No authentication errors.

    Usual causes:

    • %TCP-6-BADAUTH: the password differs or is missing at one end.

BGP is up but a prefix is not sent or received

Routing

The session is Established, but the provider does not see your prefix, or you do not see theirs.

  1. What are you actually sending?
    • show ip bgp neighbors 203.0.113.1 advertised-routes
    • show bgp ipv4 unicast neighbors 203.0.113.1 advertised-routes (NX-OS)

    Healthy: Your prefix is listed.

    Usual causes:

    • Not listed: it is not in your BGP table, or an outbound filter removes it.
  2. The prefix is in your BGP table
    • show ip bgp 198.51.100.0
    • show ip route 198.51.100.0 255.255.255.0
    • show bgp ipv4 unicast 198.51.100.0/24 (NX-OS)
    • show ip route 198.51.100.0/24 (NX-OS)

    Healthy: The prefix is in BGP with a > (best) marker.

    Usual causes:

    • network statement with no exact match in the routing table. Add a static route to Null0.
    • No > marker: not the best path, often because the next hop is unreachable.
  3. Outbound policy allows it
    • show running-config | include neighbor 203.0.113.1
    • show ip prefix-list
    • show running-config bgp (NX-OS)
    • show ip prefix-list (NX-OS)

    Healthy: Prefix lists, filter lists and route-maps permit the prefix.

    Usual causes:

    • A prefix list missing the new prefix, or with the wrong length.
    • A route-map changed and no soft clear afterwards.
  4. iBGP specifics
    • show ip bgp 10.0.0.0 | include from|Next hop|best
    • show bgp ipv4 unicast 10.0.0.0/8 (NX-OS)

    Healthy: Next hops reachable, and every iBGP router either fully meshed or served by a route reflector.

    Usual causes:

    • External next hop unknown inside your AS. Use next-hop-self towards iBGP peers.
    • iBGP routes not passed between iBGP peers without a full mesh or route reflector.
  5. The provider is accepting it
    • show ip bgp neighbors 203.0.113.1 | include prefixes
    • show bgp ipv4 unicast neighbors 203.0.113.1 | include prefixes (NX-OS)

    Healthy: The provider confirms the prefix is in their table.

    Usual causes:

    • The provider's prefix list does not include it. They need the change on their side.

Inside hosts cannot reach the internet, or only sometimes

Services

Users cannot browse, or browsing works for some people some of the time.

  1. NAT is translating at all
    • show ip nat translations
    • show ip nat statistics

    Healthy: Translations for inside hosts, and hits climbing.

    Usual causes:

    • No translations: the traffic is not matching the NAT rule or the interfaces are not marked.
  2. Interfaces are marked the right way round
    • show running-config | include ip nat inside|ip nat outside|^interface

    Healthy: ip nat inside on LAN interfaces, ip nat outside on the internet interface.

    Usual causes:

    • inside and outside swapped, or a new interface not marked.
  3. The ACL matches the inside subnets
    • show access-lists

    Healthy: Every inside subnet is permitted, and hit counts are rising.

    Usual causes:

    • A new VLAN not added to the NAT ACL.
    • An extended ACL written back to front, with the inside subnet as the destination. It never matches.
    • An ACL that permits any, which can translate traffic you did not mean to.
  4. The pool is not running out
    • show ip nat statistics | include pool|misses|allocated

    Healthy: overload configured, so many hosts share one address.

    Usual causes:

    • A dynamic pool without overload: the first few hosts work, everyone after them fails until translations time out. That is the "sometimes" fault.
  5. There is a route out and back
    • show ip route 0.0.0.0
    • ping 8.8.8.8 source Gi0/0/1

    Healthy: A default route, and pings sourced from the inside interface succeed.

    Usual causes:

    • No default route, or traffic leaving by a path that is not marked ip nat outside.
  6. Replies can find their way back
    • show ip nat translations verbose
    • show ip nat statistics | include hits|misses

    Healthy: The translated address is the outside interface, or a pool the provider routes to you, and replies arrive on the same router.

    Usual causes:

    • A NAT pool from a range nobody upstream routes to you: translations appear, replies never come back.
    • Two edge routers, and replies return through the one with no translation.

Policy-based routing is not taking effect

Services

Traffic that should take the policy path still follows the routing table, or only some of it does.

  1. The policy is on the interface the traffic arrives on
    • show ip policy
    • show running-config interface GigabitEthernet0/0/1 | include policy

    Healthy: ip policy route-map on the ingress interface for the traffic you are steering.

    Usual causes:

    • Applied on the exit interface instead. PBR acts on packets as they arrive.
    • Testing with a ping from the router itself, which interface PBR never sees. Test from a host, or use ip local policy route-map.
  2. The traffic matches the route-map
    • show route-map PBR
    • show access-lists

    Healthy: Match counters on the route-map entry rise while a host tests.

    Usual causes:

    • ACL source and destination swapped.
    • A deny line in the ACL or the route-map, which sends that traffic to normal routing rather than dropping it.
    • A later sequence number catching the traffic first because the order is wrong.
  3. The next hop is usable
    • show ip route 10.20.0.1
    • show track

    Healthy: The next hop is in the routing table, and any tracked object is Up.

    Usual causes:

    • Next hop gone, so set ip next-hop quietly falls back to normal routing.
    • set ip default next-hop used where an explicit route exists, so it is ignored. A default route does not count as explicit.
  4. Prove the decision in a lab or a quiet window
    • debug ip policy
    • traceroute 10.50.0.10 (from a host)

    Healthy: The debug says the packet was policy routed, and the host traceroute takes the policy path.

    Usual causes:

    • "policy rejected -- normal forwarding": no match, or the next hop is unusable. Go back a step.
    • Do not run debug ip policy on a busy production router without an ACL to limit it.

It works one way but not the other

Routing

A can reach B, but B cannot reach A, or a session opens from one side only.

  1. Test from the right place
    • ping vrf BRANCH 10.9.9.100 source GigabitEthernet0/0/1
    • show vrf

    Healthy: The test runs in the same VRF and from the same source as the real traffic.

    Usual causes:

    • A plain ping from a router whose interface is in a VRF uses the global table and fails while the host side works.
    • Ping sourced from a loopback or interface the far end has no route back to.
  2. Both directions have a route
    • show ip route 10.9.9.100
    • traceroute 10.9.9.100

    Healthy: Each end has a route to the other, and the traceroutes each way cross the same devices.

    Usual causes:

    • Return route missing on one router.
    • Asymmetric paths through a stateful firewall, which drops the reply it never saw leave.
  3. Nothing on the path translates the source
    • show ip nat translations
    • show running-config | include ip nat

    Healthy: No translation on any hop, or translations you expected and can route back to.

    Usual causes:

    • NAT on an intermediate router changes the source, and replies go to an address with no route back.
  4. Policy and filters on the return path
    • show ip policy
    • show ip interface GigabitEthernet0/0/1 | include access list|verify

    Healthy: No PBR, inbound ACL or strict uRPF that treats the two directions differently.

    Usual causes:

    • PBR steering the reply somewhere else.
    • An inbound ACL on one side only.
    • Strict uRPF (ip verify unicast source reachable-via rx) dropping traffic that arrives on a different interface from the route back.
  5. Layer 2 is telling the truth
    • show ip arp 10.9.9.100
    • show mac address-table address 0050.56aa.bb01

    Healthy: The ARP entry and the MAC table both point at the current device and port.

    Usual causes:

    • A stale ARP entry after a box swap.
    • MAC entries aging out long before ARP, so the switch floods replies to the whole VLAN.

A router or switch is slow and the CPU is high

Platform

SSH is sluggish, routing protocols flap, or monitoring shows CPU near 100%.

  1. Process or interrupts?
    • show processes cpu sorted | exclude 0.00
    • show processes cpu history

    Healthy: Low overall, and the second figure in "80%/60%" (interrupts) small.

    Usual causes:

    • High interrupt figure: traffic being punted to the CPU.
    • One process at the top: start with that process.
  2. Traffic is not being process switched
    • show ip interface | include CEF|line protocol
    • show interfaces stats

    Healthy: CEF enabled on every interface, and almost all packets switched in hardware or by CEF.

    Usual causes:

    • CEF disabled by a feature or a command.
    • ACL log keywords or IP options sending traffic to the CPU.
    • TTL-expired packets from a routing loop.
  3. Nothing is debugging
    • show debugging

    Healthy: No debugs running.

    Usual causes:

    • A debug left on after troubleshooting. undebug all.
  4. Something is not hammering the box
    • show processes cpu sorted | include SNMP|ARP|IP Input
    • show interfaces | include broadcasts

    Healthy: Normal SNMP, ARP and IP Input load.

    Usual causes:

    • Monitoring polling the whole MIB every minute.
    • An ARP or broadcast storm, often from a loop.
  5. Control plane policing is not dropping what matters
    • show policy-map control-plane

    Healthy: Drops only in the classes you expect to police.

    Usual causes:

    • Routing protocol or management traffic in a heavily policed class.

You cannot SSH to a device that is clearly up

Management

The device answers pings and passes traffic, but SSH times out or is refused.

  1. SSH is actually enabled
    • show ip ssh
    • show crypto key mypubkey rsa

    Healthy: SSH enabled, version 2, and an RSA key present.

    Usual causes:

    • No RSA key, so the SSH server is off.
  2. VTY lines accept SSH
    • show running-config | section line vty

    Healthy: transport input ssh on every VTY line in use.

    Usual causes:

    • transport input telnet only, or a vty 5 15 block left with old settings.
  3. The access-class lets you in
    • show access-lists

    Healthy: Your source subnet is permitted and the hit count rises when you try.

    Usual causes:

    • Your address is not in the list.
    • Arriving through a VRF without vrf-also on the access-class.
  4. There are free lines and no login block
    • show users
    • show login

    Healthy: Free VTY lines, and login not in quiet mode.

    Usual causes:

    • All lines held by dead sessions. clear line from the console.
    • login block-for has put the device in quiet mode after failed attempts.
  5. Authentication works
    • test aaa group TACACS <user> <password> legacy
    • show tacacs

    Healthy: User successfully authenticated, and the TACACS+ server reachable.

    Usual causes:

    • Server unreachable from the source the device uses.
    • Server rejects the account. Local fallback only applies when the server does not answer.

Hosts reach their own VLAN but not the others

Switching

PCs ping hosts in their own VLAN, but not their gateway, or not anything in another VLAN, on a router-on-a-stick or layer 3 switch design.

  1. The host has the right address, mask and gateway
    • ipconfig /all (Windows) or ip addr; ip route (Linux)

    Healthy: An address in the VLAN's subnet, the right mask, and the gateway set to the VLAN's gateway address.

    Usual causes:

    • Gateway set to another VLAN's gateway, or left blank.
    • Wrong mask, so the gateway looks off-subnet.
  2. The access port is in the right VLAN
    • show interfaces FastEthernet0/1 switchport
    • show vlan brief

    Healthy: Access mode, in the VLAN the subnet belongs to, and the VLAN active.

    Usual causes:

    • Port left in VLAN 1.
    • VLAN never created on this switch, so the port is inactive.
  3. The trunk towards the gateway carries the VLAN
    • show interfaces trunk

    Healthy: The uplink is trunking and the VLAN is allowed, active and forwarding.

    Usual causes:

    • VLAN missing from the allowed list at either end.
    • Port negotiated access instead of trunk.
  4. Router-on-a-stick: the subinterfaces are up, on the cabled port
    • show ip interface brief
    • show cdp neighbors
    • show running-config | section interface GigabitEthernet0/0

    Healthy: The parent and every subinterface up/up, on the interface CDP shows facing the switch.

    Usual causes:

    • Parent interface left shut down, which holds every subinterface down.
    • Subinterfaces built on a different port from the one cabled, often from another model's template.
  5. Router-on-a-stick: each subinterface tags the right VLAN
    • show vlans (on the router)
    • show interfaces GigabitEthernet0/0.10 | include Encapsulation

    Healthy: encapsulation dot1Q matches the VLAN for every subinterface, and the native VLAN matches the switch trunk.

    Usual causes:

    • dot1Q number typed as the subinterface number instead of the VLAN.
    • Native VLAN set on the subinterface but not on the switch, or the other way round.
  6. Layer 3 switch: routing is on and the SVIs are up
    • show ip route
    • show ip interface brief | include Vlan

    Healthy: Every SVI up/up and its subnet in the routing table as connected.

    Usual causes:

    • ip routing never entered, so the switch does not route between its own SVIs.
    • SVI down because no port in that VLAN is up (autostate), or left shutdown.
  7. Nothing filters between the VLANs
    • show ip interface | include line protocol|access list
    • show access-lists

    Healthy: No ACL on the gateway interfaces, or ACLs whose hit counts match what you expect.

    Usual causes:

    • An inbound ACL on one VLAN that blocks the return traffic for another.
    • A host firewall dropping ping from other subnets, which looks like a routing fault.

A DMVPN spoke cannot reach the data centre

Routing

A branch on a DMVPN has lost the hubs, never joined, or reaches the data centre through one hub only. Branch-to-branch traffic may also be going the long way round.

  1. The spoke can reach each hub's public address
    • ping vrf INET 198.51.100.2 source GigabitEthernet0/0
    • show ip route vrf INET

    Healthy: Replies from every hub, and a default route in the front-door VRF pointing at the provider.

    Usual causes:

    • The internet default is in the global table but the tunnel uses tunnel vrf INET, or the other way round.
    • The provider, or a firewall on the path, is down or dropping traffic.
  2. IKE completes with every hub
    • show crypto isakmp sa
    • debug crypto isakmp (briefly, then undebug all)

    Healthy: QM_IDLE to each hub.

    Usual causes:

    • MM_NO_STATE and "no pre-shared key" on the hub: the keyring is not in the front-door VRF, or the key differs.
    • UDP 500 (or 4500 through NAT) blocked on the path.
    • The ISAKMP policies share no encryption, hash or DH group.
  3. IPsec carries traffic in both directions
    • show crypto ipsec sa | include peer|encaps|decaps
    • show crypto session

    Healthy: For each hub, the encrypt and decrypt counters both rise while traffic flows.

    Usual causes:

    • Encrypt rising, decrypt stuck: ESP (IP protocol 50) is blocked, or NAT is in the path without NAT-T.
    • No SA at all after IKE: the transform set or PFS group does not match.
  4. The spoke is registered with NHRP
    • show dmvpn
    • show ip nhrp nhs detail
    • show ip nhrp (on the hub)

    Healthy: Every hub shown UP on the spoke, and the hub lists the spoke's tunnel address against its current public address.

    Usual causes:

    • The tunnel key differs from the hub's, so the hub decrypts the packets and drops them.
    • Different NHRP authentication strings.
    • The hub still holds the spoke's old public address after a change, because the spoke has no ip nhrp registration no-unique.
  5. Routing neighbours form over the tunnel
    • show ip eigrp neighbors
    • show ip nhrp multicast (on the hub)

    Healthy: Every hub is a neighbour with a steady uptime, and the hub's multicast list includes every registered spoke.

    Usual causes:

    • The hub has no ip nhrp map multicast dynamic (needed on classic IOS, hidden default on IOS XE), so its hellos never reach the spokes. Look for "retry limit exceeded".
    • The spoke's nhs lines lack the multicast keyword, so its hellos never reach the hubs.
  6. The routes are the ones the design expects
    • show ip route eigrp
    • show ip route 0.0.0.0
    • show ip route vrf CUST-A 0.0.0.0 (on the hub)

    Healthy: The spoke has the hubs' default from both hubs and nothing it should not. Each hub's own default is the real one, not a summary.

    Usual causes:

    • One hub advertises more specific routes than the other and quietly takes all the traffic.
    • The hub's default summary installed a Null0 route with distance 5 over its real default.
    • A static default on the spoke beats the default learned over the tunnel.
  7. Branch-to-branch traffic goes direct
    • traceroute 10.1.2.10 source 10.1.1.1 (twice)
    • show ip route nhrp
    • show dmvpn detail

    Healthy: The second traceroute has no hub address in it, and show ip route nhrp lists the other branch's LAN.

    Usual causes:

    • No ip nhrp shortcut on this spoke, so it ignores the hubs' redirects. Test both directions; it can work one way only.
    • No ip nhrp redirect on the hubs.
    • Something between the branches blocks IKE or ESP, so the direct tunnel never builds and traffic keeps going through a hub.

A vPC is down, suspended, or a VLAN will not pass

Data centre

A server or access switch on a vPC has lost one leg or one VLAN, or show vpc reports something other than success.

  1. The peers are talking
    • show vpc brief
    • show vpc peer-keepalive

    Healthy: Peer status: peer adjacency formed ok. vPC keep-alive status: peer is alive. Configuration and type-2 consistency: success.

    Usual causes:

    • Keepalive down: wrong VRF, wrong source or destination, an ACL in the way, or mgmt0 cabled to a standby supervisor.
    • Peer-link down with the keepalive up: the secondary has suspended its vPCs and SVIs on purpose.
    • Different domain IDs on the two peers, so they never form one domain.
  2. The peer-link is up and carries the VLAN
    • show port-channel summary
    • show interface trunk
    • show spanning-tree inconsistentports

    Healthy: The peer-link is (SU) with every member (P), and the VLAN is allowed and forwarding on it.

    Usual causes:

    • The VLAN was added to the vPC but not to the peer-link, so it does not forward on the vPC.
    • Bridge Assurance has blocked the peer-link because BPDUs stopped arriving.
    • A peer-link member on a port type or speed that does not match the others.
  3. Global settings match on both peers
    • show vpc consistency-parameters global

    Healthy: Every type 1 line has the same local and peer value.

    Usual causes:

    • Spanning-tree mode, MST region, or the global Bridge Assurance, loop guard or port type defaults changed on one peer only. With graceful consistency check, the secondary suspends every vPC.
    • A VLAN created on one peer only, which is suspended until it exists on both.
  4. This vPC matches on both peers
    • show vpc consistency-parameters vpc 20
    • show running-config interface port-channel20

    Healthy: Consistency success, with the same mode, speed, MTU, native VLAN and LACP mode on both peers.

    Usual causes:

    • MTU changed on one peer only: a type 1 mismatch that suspends the vPC on the secondary.
    • vpc 20 configured on one switch only, or with a different number.
    • Allowed VLAN lists differ: only the VLANs on both sides forward.
  5. The attached device sees one partner
    • show port-channel summary
    • show lacp neighbor interface port-channel20

    Healthy: Members (P) on both peers, and the device reports the same LACP system ID on all its links: the vPC system MAC.

    Usual causes:

    • The device has two separate port-channels, one to each switch, so spanning tree blocks one.
    • A server not yet sending LACP (during PXE boot, say), so its ports are suspended.
    • Two vPC pairs facing each other with the same domain ID, so the bundle flaps.
    • mode on at one end and LACP at the other.
  6. Roles and recent events explain it
    • show vpc role
    • show logging logfile | include VPC

    Healthy: The operational primary is the switch you expect, and nothing has reloaded or flapped recently.

    Usual causes:

    • A reload swapped the roles, so the switch you think is primary is the one suspending ports.
    • A peer has just reloaded and its vPCs are waiting out delay restore. That is normal for the configured time.
    • Only one switch came back after a power cut and auto-recovery is not configured, so its vPCs stay down.
  7. Routing and the gateway are built for vPC
    • show hsrp brief
    • show ip ospf neighbors
    • show mac address-table address 0000.0c9f.f00a

    Healthy: HSRP active on one peer and standby on the other, with the virtual MAC flagged as a gateway (G) on both. A routed adjacency between the peers on its own link.

    Usual causes:

    • HSRP tracking has moved the gateway, and traffic bridged over the peer-link is dropped.
    • A router or firewall forms routing adjacencies over the vPC without layer3 peer-router.
    • No routed path between the peers, so the switch that lost its uplinks drops what it receives.

A port or link is slow, or keeps dropping

Switching

One user, server or uplink is slow, file transfers stall, or the link flaps, while everything else on the switch is fine.

  1. The link is up at the speed you expect
    • show interfaces GigabitEthernet1/0/10 status
    • show interface ethernet1/10 status (NX-OS)

    Healthy: Connected and full duplex, at the speed of the device (a-full, a-1000).

    Usual causes:

    • Half duplex, or 100 Mb on a gigabit device: a broken pair in the cable, or one end hard-set.
    • Speed and duplex hard-set at one end and left on auto at the other.
  2. Errors are not climbing
    • show interfaces GigabitEthernet1/0/10 | include CRC|collision|error|reset
    • show interfaces counters errors
    • show interface ethernet1/10 counters errors (NX-OS)

    Healthy: Error counters at zero, or not rising between two checks a minute apart.

    Usual causes:

    • CRC or FCS errors: cabling, a dirty fibre connector or a failing optic.
    • Late collisions on what should be full duplex: a duplex mismatch.
    • Runts and giants: a faulty NIC, or an MTU mismatch.
  3. The optic is in range
    • show interfaces TenGigabitEthernet1/1/1 transceiver detail
    • show interface ethernet1/49 transceiver details (NX-OS)

    Healthy: Receive power inside the thresholds, with a margin.

    Usual causes:

    • Receive power close to the low warning: a dirty connector, a bad patch lead, or a run too long for the optic.
    • The wrong optic for the fibre, such as a multimode optic on singlemode.
  4. It is not simply full
    • show interfaces GigabitEthernet1/0/10 | include rate|drops
    • show interface ethernet1/10 | include rate|discard (NX-OS)

    Healthy: Load well below line rate, and output drops not rising.

    Usual causes:

    • Output drops on a busy uplink: real congestion, or bursts that a five-minute average hides. Set load-interval 30 and look again.
    • Drops on a quiet port: small buffers meeting short bursts.
  5. Nothing is shutting the port down
    • show logging | include 1/0/10
    • show interfaces status err-disabled
    • show power inline GigabitEthernet1/0/10
    • show logging logfile | include ethernet1/10 (NX-OS)
    • show interface status err-disabled (NX-OS)

    Healthy: No flaps, err-disable events or power faults in the log.

    Usual causes:

    • Port security, BPDU guard or storm control acting on the port.
    • A PoE device asking for more power than the port can give, and cycling.
    • A cable or NIC fault making the link bounce.
  6. Move the problem
    • test cable-diagnostics tdr interface GigabitEthernet1/0/10
    • show cable-diagnostics tdr interface GigabitEthernet1/0/10
    • show interface ethernet1/10 status (NX-OS)

    Healthy: Swap the patch lead, the optic or the port, one at a time. The fault either moves with the part you swapped or stays where it was, and now you know which.

    Usual causes:

    • It stays with the switch port: a hardware fault. Open a case with the evidence.
    • It follows the device: its NIC, driver or power-saving settings.

The backup link did not take over, or will not hand back

Routing

The primary circuit failed and users lost service, or the backup took over but traffic never came back, or it keeps flipping between the two.

  1. Both routes exist, at the right distances
    • show running-config | include ip route 0.0.0.0
    • show ip route 0.0.0.0

    Healthy: A primary route, tracked or learned from a routing protocol, and a backup with a higher distance. The primary is in the table now.

    Usual causes:

    • The backup has the lower distance, so it wins all the time.
    • The backup was never added, or points out of an Ethernet interface with no next hop.
  2. The primary route really goes away
    • show track
    • show ip sla statistics

    Healthy: The track is Up with return code OK, and goes Down when the provider fails, not only when the cable is pulled.

    Usual causes:

    • No tracking at all: the route stays while the interface is up, however broken the provider is.
    • The probe was never scheduled, so the track is Down all the time.
    • The probe target is the provider's own router, which answers while their network is down.
  3. The probe cannot escape by the backup
    • show ip route 192.0.2.53
    • show track 10 | include change

    Healthy: The probe target has a host route via the primary next hop and a Null0 route behind it, and the track changes state once per event.

    Usual causes:

    • The probe follows the default route, so after failover it succeeds through the backup and the primary keeps coming back.
  4. Return traffic comes back the same way
    • show ip nat translations
    • show ip nat statistics

    Healthy: Translations use the address of the link the traffic is leaving by.

    Usual causes:

    • NAT tied to one interface only, so traffic leaves the backup with the primary's address and nothing returns.
    • Old translations still pointing at the dead link until they time out.
  5. A routing protocol withdraws the dead path in time
    • show ip bgp summary
    • show ip ospf neighbor
    • show ip route 0.0.0.0

    Healthy: When the primary fails, its neighbour drops within the hold time and the backup path takes over.

    Usual causes:

    • BGP's default hold time is 180 seconds, so a silent failure takes three minutes to notice.
    • default-information originate always on the failed edge router keeps attracting traffic.
  6. Hand-back is controlled
    • show track 10
    • show logging | include TRACK

    Healthy: The primary returns only after a delay, and every change is in the log.

    Usual causes:

    • No delay up, so a flapping circuit drags users back and forth.
    • Nobody told the users that sessions drop on failover because the NAT address changes.