Field Notes

The AI told me my fix wouldn't work. It was the fix.

An afternoon on a firewall change with an AI assistant. It saved me hours. It was also wrong, and completely sure of itself, at least six times.

I suggested a fix. The AI told me it wouldn't work, that it would just bring the original problem back, and that I should restart the whole routing process on the firewall instead. That would have been a pain to explain to the client, and a pain to go through.

I tried my fix anyway. It worked.

That was one afternoon on a firewall change for a client, with an AI assistant alongside me. I won't name the kit or the client, because it doesn't matter. What matters is how it went.

It was good. Properly good.

In a few minutes it went through a change document that would have taken me an hour to check properly. It found a copy-and-paste typo, a missing setting and a gap in the rollback plan. Any one of those would have bitten on the night.

It turned a dense document into a side-by-side checklist I could tick through. When I asked for a working example, it went to the vendor's own guides and came back with the answer. And over several hours of troubleshooting, it kept the whole thread in its head while I worked on one bit at a time.

It's quick and it knows a lot. I'd have it on any change.

It was also wrong. A lot.

It gave me commands the kit flatly rejected. Right idea, wrong words for that system.

It "corrected" a setting that wasn't broken, and its version used a character the system doesn't allow.

Twice it gave me a neat theory for the fault. Both fitted the evidence it had. Both were wrong.

And then there was my fix, the one it said wouldn't work.

Here's the thing. It sounded exactly as sure of itself when it was wrong as when it was right. It did put itself right every time, but only once I'd put the evidence in front of it. If I hadn't pushed back, it would have stuck with every one of them.

None of it got near the live network, because I didn't run anything I hadn't checked myself first.

What I do now

Nothing new here. It's what any decent engineer already does, applied to a new kind of colleague.

1. Agree what counts as the truth before you start. Early on, I told it the commands in the document outranked the diagram. That settled several arguments before they started.
2. The kit has the final say. If it rejects a command, the command's wrong, however sure the AI sounded. More than once, the system's own help text put the AI right.
3. Feed it facts. Paste the real output, and ask where its answer comes from. When it went to the vendor's documentation, its answers got noticeably better. When it filled the gaps with what usually happens, they got worse.
4. Trust your instinct enough to test it. The AI wanted to restart the whole routing process. My fix was the right one. Thirty years of doing this counts as evidence too.
5. Own the decision. My name goes on the change record, not the AI's. If I couldn't explain a step to the client, I don't run it.

The change went in, it works, and the AI saved me real time getting there. I'll use it again tomorrow.

But I'll keep doing the thinking. The day you stop is the day its mistakes become yours.

Related: The design an AI model got wrong, done properly · How I run a network implementation · The 1% packet loss that two vendor tickets missed

The 1% packet loss that two vendor tickets missed

Voice breaking up under load in a data centre, tickets open with Cisco and the firewall vendor, and weeks spent on the complicated parts. The fix was one line that should have been in the template from day one.

What happened

A data centre with two Catalyst 4900M cores, 4948 access switches, an active/standby firewall pair and two MPLS routers to the carrier, with separate VRFs for the WAN and for voice. Under load, about 1% of packets through the core went missing. Data carried on, because TCP quietly resends what it loses. Voice could not: calls broke up at busy times, and the people on those calls were the board.

We looked at everything. The firewalls had receive buffer drops because every interface was pinned to one CPU core, a real problem worth fixing, but not this one. The carrier went looking for VRF bugs in its router software. Spanning tree got a tidy-up. Tickets stayed open with both vendors.

The clue

From a core switch, I pinged two neighbouring addresses in the same WAN subnet. One was clean every time. The other lost packets. Traceroute showed the difference: the clean one went firewall, core 1, destination. The lossy one went firewall, core 1, then core 2, so core 1 was sending those packets back out of the same VLAN they had arrived on.

What fixed it

no ip redirects on the WAN VRF SVI on both cores. The loss stopped the moment it went in.

Why one line mattered

• When a router forwards a packet out of the interface it arrived on, IOS assumes the sender could have gone straight to the next hop, and sends it an ICMP redirect saying so.
• On the Catalyst 4500 and 4900 family, building that redirect means the packet is forwarded by the CPU instead of in hardware. Cisco's own high-CPU guide shows it in the L3 Fwd Low queue.
• Firewalls and routers ignore redirects, so the sender never changes anything. Every packet in that flow goes to the CPU, not just the first.
• At low volume, the CPU keeps up and nobody notices. Under load, the queue overflows and packets are dropped: about 1%, and only when busy, which is exactly what we saw.

Other platforms behave differently: the Catalyst 6500 rate-limits these punts, and on Nexus no ip redirects is expected on SVIs with HSRP and vPC. Check what yours does before assuming it is harmless.

How to spot it

traceroute vrf WAN 10.20.0.36
! a hop where the packet leaves by the VLAN it came in on is the suspect
show ip traffic | include redirects
show ip interface Vlan100 | include redirects
! Catalyst 4500/4900: software-forwarded packets climbing
show platform cpu packet statistics | include L3 Fwd
show platform health | include K2CpuMan

Or capture on the transit VLAN with the Wireshark filter icmp.type == 5. A steady stream of redirects from a core switch to a firewall is the giveaway.

What I took from it

• Check the boring config before the clever stuff. We spent weeks on firewall CPU cores and carrier bugs. The answer was a default nobody had turned off.
• Put it in the template. Every routed interface where traffic can come in and go out the same way gets no ip redirects: transit VLANs between firewalls and routers, HSRP pairs, router-on-a-stick.
• A test that splits the problem beats a ticket. Two pings to neighbouring addresses did more than weeks of vendor escalation.
• A real finding is not always the cause. The firewall interface drops were genuine and still needed fixing. They just were not why voice broke up.
• Measure what voice feels. Ping misses loss that only shows under load. An IP SLA UDP jitter probe with voice markings, running all day, shows it.

interface Vlan100
 description WAN VRF transit
 vrf forwarding WAN
 no ip redirects
 no ip proxy-arp
!
! voice-like loss and jitter, all day (needs ip sla responder at the far end)
ip sla 50
 udp-jitter 10.20.0.36 16384 codec g711alaw
 tos 184
ip sla schedule 50 life forever start-time now
show ip sla statistics 50

Related: The gotcha version · The circuit that was fine · Troubleshooting Flows · Wireshark Filter Builder

IOS XE is starting to refuse insecure commands. Clean up before you upgrade

Telnet, plain HTTP, TFTP, SNMP communities, MD5 keys and type 7 passwords are on their way out of IOS XE. The early releases only warn. 26.1 starts refusing some of them, and it catches people out in two different ways.

What is changing

Cisco is retiring insecure features from IOS XE in three stages.

• Warning, from 17.18.2. The command still works, but the CLI and syslog flag it with a message naming the command, the reason and the fix (%SYS-4-INSECURE_DYNAMIC_WARNING).
• Restriction, from 26.1.1. Commands on the restricted list are refused unless the device is in insecure mode (system mode insecure). Not everything that warns is restricted yet, and the list grows with each release.
• Removal. The feature goes. Cisco says insecure mode itself will be removed in a later release, and advises against moving to 26.2 or 27.1 without migrating first.

The two ways it catches you

• Upgrades hide it. If a switch has restricted config when it first boots 26.1, it puts itself into insecure mode and adds system mode insecure to the config, so nothing breaks. Everything looks fine. The work has only been moved to the release where the feature disappears.
• New switches don't. A switch out of the box has no startup config, so it starts in secure mode. Paste the template you have used for years and the restricted lines are rejected. If the template arrives by zero-touch provisioning, nobody sees the errors. You find out when the switch never appears in monitoring.

The upgrade also converts type 0 and type 7 passwords to type 6. Expect every config backup to show a difference the first night after an upgrade, and tell whoever reads the compliance reports before they raise it as an unauthorised change.

What to replace, and with what

Find out what you have now

On any release, this pulls out the usual suspects. It also catches OSPF, BGP and SNMP keys using MD5, which are worth a look while you are in there.

show running-config | include transport input|ip http server|snmp-server community|password 7|password 0|ip ftp|ip tftp|md5|ip bootp server

On 26.1 and later, the device tells you itself:

show system security mode
show system insecure configuration
show system insecure profile
! force a rescan instead of waiting up to 30 minutes
test system secure ?

show system insecure configuration lists the restricted config in use. It is rescanned about every 30 minutes, and the test command forces a rescan. Cisco's own pages give two spellings for that command, so check it with the question mark. show system insecure profile lists every pattern the release checks for, whether you use it or not.

The order I would do it in

1. Get SSH working from everywhere that manages the device: jump hosts, console servers, the automation account. Then remove Telnet, with a session left open while you do it.
2. Move backups and image copies to SCP or SFTP. The SCP server needs AAA exec authorisation.
3. Add SNMPv3 users to the monitoring platform, confirm every device still polls, then remove the communities.
4. Change NTP keys on the servers and the devices in the same window, or clocks stop syncing.
5. Reset local accounts as type 9 secrets and set a master key for type 6.
6. Fix the day-0 template and any zero-touch script, so new switches start clean.
7. Run show system insecure configuration until it is empty, then no system mode insecure. Add "system mode insecure is present" to your compliance checks.

A clean baseline

ip ssh version 2
crypto key generate rsa general-keys label SSH modulus 3072
ip ssh rsa keypair-name SSH
line vty 0 15
 transport input ssh
 transport output ssh
!
no ip http server
ip http secure-server
ip http tls-version TLSv1.2
!
aaa new-model
aaa authentication login default local
aaa authorization exec default local
ip scp server enable
!
username netadmin privilege 15 algorithm-type scrypt secret <password>
enable algorithm-type scrypt secret <password>
key config-key password-encrypt
password encryption aes
!
no snmp-server community <old-community>
snmp-server group NMS v3 priv
snmp-server user nms01 NMS v3 auth sha <auth-pass> priv aes 128 <priv-pass>
snmp-server host 10.0.0.50 version 3 priv nms01
!
ntp authenticate
ntp authentication-key 1 hmac-sha2-256 <key>
ntp trusted-key 1
ntp server 10.0.0.10 key 1
!
no ip bootp server

Adjust the AAA lines to match how you already authenticate. If you use TACACS+ or RADIUS, keep local as the fallback rather than the first method.

Related: The template that works on old switches and fails on new ones · The upgrade quietly put the switch in insecure mode · VTY 0 4 · SNMP read-write

How to read a design brief

A design brief is mostly noise with five or six sentences that decide everything. The skill is finding those sentences before you start drawing boxes.

Read it once for the story, then again with a pen

The first pass tells you what kind of job it is: a replacement, a merger, a scaling problem, a failure to fix, a new service. The second pass is for the lines that rule options in or out. I mark four kinds:

• Numbers. Budgets, dates, head counts, outage limits, data loss limits, site counts. "No outage longer than 15 minutes" kills more options than any technology debate.
• What the business can live without, and for how long. "The practices can work on paper for four hours" is a design requirement, even though nobody wrote it as one.
• The team. Five people working office hours cannot run what a thirty-person operations centre can.
• Things that cannot change. A vendor's support terms, a contract end date, devices that only speak one protocol.

Treat what you don't know as a black box

Every brief has a system you have never worked with: a smart meter, a SCADA outstation, a medical imaging archive. You don't need to know how it works inside. You need to know what it connects to, how much traffic it sends, what happens when it can't connect, and who supports it.

"Works" is not the same as "best"

Most options in a design review would work. The best one meets every stated requirement with the least cost, risk and effort to run. A design that meets a requirement nobody asked for, such as zero data loss when the business said fifteen minutes, is not better. It is more expensive.

When new information arrives, re-read what you decided

Projects change direction. An email says the vendor only supports the internet, or the carrier admits both circuits share a duct. Go back to the documents and check which earlier choices still stand. Don't defend a decision just because you made it.

The last three entries in Proven Designs are worked cases that start from a brief like this and end with config and sign-off tests.

Related: Start with why · Best practice is not a reason · Choosing between designs on paper

Your markings across a carrier MPLS network: uniform, pipe and short pipe

You mark voice as EF at the branch. Does the carrier honour it, change it, or ignore it on the way out to your other site? The answer depends on which of three QoS modes the carrier runs, and most people never ask.

When a packet enters an MPLS network, the provider edge router sets the 3-bit traffic class (EXP) field in the label it adds. Cisco IOS copies it from the IP precedence by default. What happens next is the carrier's choice.

Why it matters to you

• The last link, from the carrier's provider edge down to your site, is usually the slowest and most congested. Whichever marking the far provider edge queues on decides whether your voice survives a busy afternoon.
• In pipe mode, what counts is the class the carrier put your traffic in when it arrived. If your marking at the entry point doesn't map to their priority class, your EF traffic travels as best effort, and nothing at the far end can fix it.
• In uniform mode, a carrier policer that remarks out-of-contract traffic in the core also changes the DSCP your far site receives. Your own QoS policy at that site then treats it differently.

Questions to ask before you sign

1. Which mode do you run, and is it the same on every provider edge?
2. Which DSCP values map to which of your classes, and what happens to a value you don't recognise?
3. Do you remark traffic that exceeds a class's bandwidth, or drop it?
4. Is the egress queue towards my site based on my markings or yours?

Then test it: send marked traffic from one site, capture at the other, and compare the DSCP values.

Related: Do you actually need QoS? · Which routing protocol to run with your provider · Wireshark filter builder

Layer 2 between sites: VPLS or EVPN

If you genuinely need one VLAN across several sites, there are two ways a carrier or your own MPLS core will give it to you. They look the same from a switch port and behave very differently when something goes wrong.

VPLS

• The network learns MAC addresses the way a switch does: by flooding unknown traffic and watching where replies come from.
• Every site has a pseudowire to every other site. Fine for ten sites, heavy at a hundred.
• A site with two connections normally uses one at a time. Both active risks a loop.
• It is a mature carrier service and still widely sold.

EVPN

• MAC addresses (and often the IP behind them) are advertised in BGP, so the network knows where hosts are before traffic arrives. Far less flooding.
• A site can use both connections at once (all-active multihoming), with the network preventing loops and duplicate broadcasts.
• Routers can answer ARP for known hosts locally, and the network notices when a host moves.
• Routing and bridging can live in the same service, so traffic between VLANs does not have to go back to one site.

Whichever you use, you are still building one broadcast domain

A loop or a broadcast storm at one site reaches every site in the VLAN. Before you buy either:

• Ask whether the requirement really needs layer 2. Most "we need the same subnet" requests are about addresses not changing during disaster recovery, and routing solves that more safely.
• Put storm control on every port facing the service, and BPDU guard where your switches should never see the carrier's spanning tree.
• Agree with the carrier what they do with your BPDUs and how many MAC addresses you can send before they start dropping.

For a new build where you choose the technology, EVPN is the better answer. Buying from a carrier, take what they sell, but design your edge as if the far end could loop at any moment.

Related: Where does a failure stop? · Proven Designs · Design: disaster recovery without stretching a VLAN

Connecting your network to a public cloud

There are three ways to reach a cloud provider privately, and the choice is less about bandwidth than about routing: what you advertise, what comes back, and which path wins when both are up.

The options

• IPsec VPN over the internet to the provider's VPN gateway, usually with BGP over the tunnel. Quick to set up, cheap, and limited by the internet path and the gateway's throughput.
• A private interconnect (the providers' direct connection services), delivered through a colocation facility or a carrier. Predictable latency and higher capacity, with lead times measured in weeks.
• Your SD-WAN fabric extended into the cloud with virtual edge routers there. One policy for branches and cloud, at the cost of running more routers.

Design points that catch people

• Prefix limits. Providers cap how many routes you can advertise to them, and on some services exceeding the cap takes the BGP session down. Summarise before you connect, and check the limit for the service you bought.
• Two paths, one winner. With a VPN and a private interconnect both up, decide which one carries traffic in each direction, and make both sides agree. Otherwise traffic goes out on one and returns on the other, and the stateful firewall in the middle drops it.
• Redundancy that is real. Two connections in the same colocation facility share fate. Providers publish which locations are separate; use two.
• Overlapping addresses. Cloud teams often pick the same private ranges as the data centre. Agree an address plan for the cloud before the first network is built there.
• The cost runs one way. Most providers charge for traffic leaving the cloud, not arriving. A design that sends backups or replication out of the cloud every night can cost more than the circuit.

Related: Two WAN carriers · RPO and RTO · Routing Tricks

Zero trust, from the network engineer's side

Zero trust gets sold as a product. It is really one idea: being on the network should not, on its own, grant access to anything. Most of the work to get there is network work.

What changes

• Access follows identity, not location. A user or device proves who it is, and policy decides what it can reach. Being plugged into the office, or connected to the VPN, is no longer a pass.
• Segments get smaller. Instead of one large internal zone, systems sit in groups that can reach only what they need. Ransomware that lands on one laptop should find very little it can talk to.
• Remote access moves from network to application. Instead of putting a home laptop on the corporate network with a VPN, users connect through a broker to the specific applications they are allowed.
• Every decision is logged. Who reached what, from which device, and whether it was allowed.

Where the network does the work

• Network access control with 802.1X, so the switch port or wireless network knows who is connected and can place them in the right segment.
• Segmentation you can enforce: VRFs, group-based policy or firewalls between zones, rather than one flat VLAN per floor.
• Enforcement points in the paths traffic actually takes, including branch internet breakout and cloud.
• Flow telemetry, so you can see what talks to what before you write the rules that restrict it.

How to start

Pick the systems that would hurt most if reached by the wrong person: payments, control systems, backups, the management network itself. Record their flows for a few weeks. Put them in their own segment with rules built from what you recorded. Then move the next group. Trying to do the whole estate at once is how zero trust projects stall.

Related: IT and OT segmentation in plain English · Design: separating OT from IT with a DMZ · Build the firewall rules from what the logs tell you

Start a design with "why", not with boxes

Most bad designs are good answers to the wrong question. Before you pick a single technology, get the requirements out of people, and keep asking why until you reach something the business actually cares about.

Four kinds of requirement

• Business: what the organisation is trying to achieve. Open ten branches next year. Pass a PCI audit. Cut WAN costs by a third.
• Functional: what the network must do. Branches reach the payment platform. Guests get internet and nothing else.
• Technical: how well it must do it. Failover inside a second. Latency under 50 ms to the data centre.
• Application: what the applications need. Voice needs low jitter. A legacy app needs layer 2 between two sites. A replication job needs 2 Gbps every night.

Then the constraints

Budget, existing kit, skills in the team, contracts that can't be broken, the circuit that takes four months to order, a change freeze over Christmas. A constraint isn't a requirement, but it rules options out just as firmly.

Ask why until you hit the business

"We need a stretched VLAN between the data centres." Why? "So the servers can move." Why do they need to move? "So we can fail over if a data centre is lost." How quickly, and how much data can be lost? Now you're talking about recovery objectives, and a stretched VLAN is only one of several answers, and not the safest.

The requirements nobody states

Nobody writes "it must be secure", "it must be supportable by the team we have" or "it must not fall over when one box dies". They expect all three anyway. Write them down yourself and get them agreed, because they are the ones you will be judged on.

Know which kind of job it is

A greenfield build, a change to a running network, a merger, a migration off a technology, or a scaling problem each carry different risks. In anything but a greenfield, the existing network is a requirement of its own: whatever you design has to get from here to there without an outage nobody agreed to.

Related: Best practice is not a reason · Choosing between designs on paper · How I run an implementation

Best practice is not a reason

"It's best practice" ends a lot of design discussions. It shouldn't. A best practice is someone else's answer to someone else's requirements. The four traps below catch experienced engineers more often than new ones.

• Assumptions. "They'll obviously want dual links." Maybe the site is a warehouse that can run on 4G for a day. Ask. An assumption is a requirement nobody agreed to, and it will be the first thing questioned when the bill arrives.
• Overdesign. More redundancy, more features and more protocols feel safer. Each one is also something else to fail, to upgrade and to understand at 2am. Build what the requirements need, plus a sensible margin for growth, and stop.
• Best practices. Useful as a starting point and a checklist. Not a substitute for knowing why. If you can't explain how a best practice serves this network's requirements, you don't yet know whether it applies.
• Preconceived notions. The design you have built five times before is the one you'll reach for. Sometimes it's right. Check it against the requirements as if it were someone else's proposal.

A simple test for every element of a design: which requirement does this meet? If the answer is none, it's a candidate for removal.

Related: Start with why · Every abstraction leaks

Where does a failure stop?

Every design has an answer to "when this breaks, what else breaks with it?", whether anyone wrote it down or not. Draw the blast radius of each component before you build, and look hard at the things that quietly share a fate.

Failure domains

A failure domain is everything that is affected when one thing fails. A VLAN is a failure domain for a loop. An OSPF area is one for SPF churn. A stack is one for a bad software upgrade. Good designs keep them small and put clear edges between them.

• Hierarchy and modularity give you the edges: each building block (a campus, a branch type, a data centre pod) connects to the rest at a few known points.
• Summarise at those edges, so a flapping subnet inside a block doesn't make every router in the network recalculate.
• Route between blocks, don't bridge. A stretched VLAN joins two failure domains into one.

Shared fate, the hidden kind

Two of everything doesn't help if both depend on one thing. Look for:

• Two circuits from different carriers in the same duct, or through the same exchange.
• Both power supplies on the same distribution board.
• Both switches of a pair on the same software version, upgraded on the same night.
• A vPC or cluster pair whose keepalive runs over the link it is meant to protect.
• Primary and backup both reliant on one DNS server, one authentication server or one management VRF.

Do it on paper first

For each component in the design, write one line: what fails with it, and how users notice. If one line covers half the business, that's where the design needs work.

Related: Every abstraction leaks · Split brain · The keepalive path

RPO and RTO: the two numbers that should size the design

Disaster recovery conversations go round in circles until someone asks two questions: how much data can we afford to lose, and how long can we be down? The answers decide the links, the addressing and how much the whole thing costs.

• RPO, recovery point objective: how much data, measured in time, the business can lose. An RPO of 15 minutes means replicating at least every 15 minutes. An RPO of zero means synchronous replication.
• RTO, recovery time objective: how long the service can be unavailable before it's back, at the recovery site if need be.

The business sets these, per application, not the network team. Your job is to tell them what each answer costs.

What they mean for the network

• Bandwidth. Replication must move all changed data within the RPO. If 60 GB changes in a busy hour and the RPO is 15 minutes, you need to move 15 GB in 15 minutes, about 130 Mbps sustained, before overheads and growth.
• Distance. An RPO of zero means every write waits for the far site to confirm it, so latency goes straight into application performance. That usually limits synchronous replication to sites tens of kilometres apart.
• Addressing. A short RTO pushes people towards stretching subnets so servers keep their addresses. Re-addressing with DNS changes, or a global load balancer, keeps the sites as separate failure domains. Decide it here, not in the middle of an outage.
• The network's own recovery. Firewall policy, NAT, routing to the recovery site and the internet presence (public addresses, BGP announcements) all have to work there too. Test them in the DR exercise, not just the servers.

An RTO of four hours with a design that needs a day of manual reconfiguration isn't an RTO. It's a hope.

Related: Where does a failure stop? · Proven Designs

Choosing between designs on paper

Two or three options, each with fans in the room. A weighted decision matrix turns the argument into a table everyone can see, and makes the reasons for the choice easy to explain later.

1. List the criteria that come from the requirements: cost over five years, resilience, time to deploy, skills needed to run it, fit with the existing network.
2. Agree a weight for each with the people who own the requirements, before anyone scores an option. That stops people weighting the criteria to suit their favourite.
3. Score each option against each criterion, say 1 to 5, with a short reason for every score.
4. Multiply and add up.

The numbers matter less than the conversation. A close result tells you the choice is genuinely balanced, so pick the option that is easier to reverse. A low score on a heavily weighted criterion (team skills, above) tells you what has to be fixed whichever option wins: training, a partner, or a longer timeline.

Keep the table with the design. In two years, when someone asks why it was built that way, the answer is written down.

Related: Start with why · Two WAN carriers

Joining two networks after a merger

The deal is signed and the two networks need to talk by Monday. The first connection is easy. Doing it without creating a mess you live with for a decade takes a plan with stages.

The questions to answer first

• Overlapping addresses. Both companies use 10.0.0.0/8, and some ranges clash. Plan where translation happens, and which side renumbers in the long run.
• Routing. Different IGPs, possibly different BGP AS numbers. Decide whether one protocol absorbs the other, or BGP sits between them as the glue.
• Security policy. Until the security teams have agreed a common policy, the other company is an untrusted network. Put a firewall between them.
• Shared services. DNS, directory, email and time. These are usually what people actually need first.

A staged approach

1. Interconnect: one or two controlled links, firewalled, with NAT where addresses clash, and only the routes needed for the services that must be shared.
2. Join the routing with clear boundaries: eBGP between the two, or redistribution at two points with tags and filters. Keep the boundary where it can be tested and changed.
3. Migrate sites and services across in waves, renumbering as you go, with a rollback for each wave.
4. Remove the temporary NAT, the duplicate services and the old circuits. Otherwise the temporary becomes the design.

If AS numbers have to change, BGP can keep presenting the old number to existing neighbours while the routers move to the new one, so the neighbours don't all have to change on the same night.

Related: Changing AS number without touching the neighbours · Redistribution rules · NAT between VRFs

Route reflectors hide paths

Route reflectors fix the iBGP full-mesh problem. They also quietly change which exit your routers use, and slow down failover, because each reflector passes on only the one path it thinks is best.

What changes when you add a reflector

• Exit choice moves to the reflector. Normally each router picks the exit with the lowest IGP cost from where it sits: hot potato routing. With a reflector, every client gets the exit that is closest to the reflector. A branch in the north can end up using an exit in the south because the reflector sits in the south.
• Backup paths disappear. If two exits learn the same prefix, the reflector sends its clients only the best one. When that exit fails, clients have nothing to fall back on until the reflector recalculates and sends a new path.
• Reflectors don't change the next hop. Every client must still be able to reach every next hop through the IGP.

The fixes

• Place reflectors with the topology in mind: in each region, or close to where the exits are, so their idea of "closest" matches the clients'.
• Send more than one path. Add-path lets a reflector advertise a second path as well as the best one. Best-external lets an edge router advertise its own external path even when it prefers someone else's.
• In an MPLS VPN, a unique route distinguisher per PE makes each PE's copy of a prefix a different route, so the reflector passes them all on.
• Two reflectors per cluster, each client peering with both. And remember that a reflector out of the forwarding path still needs to resolve every next hop.

Hot potato or cold potato

Hot potato hands traffic to the next network as soon as possible: cheap for you, sometimes poor for the user. Cold potato carries it across your own network to the exit closest to the destination: better for the user, more of your bandwidth. Most enterprises want hot potato for internet traffic. Make it a decision rather than a side effect of where the reflector happens to sit.

Related: BGP that fails over in under a second · Learning BGP path selection · Steering with communities

BGP that fails over in under a second

With a full internet table, losing an edge router can mean minutes of reconvergence, prefix by prefix. Prefix Independent Convergence makes failover take the same time whether you carry ten routes or a million.

Why BGP is slow to fail over

When an exit dies, the router has to find a new best path for every prefix that used it, then rewrite the forwarding table entry for each. Hundreds of thousands of prefixes take time, and traffic to each is dropped until its turn comes.

How PIC fixes it

• PIC core: the forwarding table points prefixes at a shared next-hop entry instead of copying the path into every prefix. When the IGP path to that next hop changes, one entry is updated and every prefix follows. Most current platforms build it this way already.
• PIC edge: the router works out a backup path for each prefix in advance and installs it alongside the primary. When the primary exit fails, it switches to the backup straight away, then tidies up the BGP side afterwards.

router bgp 65000
 address-family ipv4
  bgp additional-paths install
  bgp advertise-best-external
!
show ip bgp 198.51.100.0
show ip cef 198.51.100.0 detail

PIC edge only works if a backup path is actually known. If a route reflector sends only its best path, or the other edge router keeps its own external path to itself, there is nothing to install. That's what best-external and add-path are for.

And it needs fast detection. Instant switchover after a three-minute hold timer is still a three-minute outage. Pair it with BFD or fast external fall-over.

Related: Route reflectors hide paths · Why your routing takes 40 seconds to notice

Which routing protocol to run with your MPLS provider

Static, eBGP, OSPF or EIGRP to the provider's PE. Each works, but they behave very differently when a site has two links or a backdoor link of its own. Here is how I would choose.

• Static: fine for a single-homed site with one or two subnets. Failover depends entirely on the link going down, which a provider fault often doesn't do.
• eBGP: the usual right answer. You control what you send and accept, you can prefer one link with local preference, and the provider can't accidentally flood your IGP.
• OSPF: keeps your internal routes looking internal, which is attractive when the WAN replaces a network that was all OSPF. It gets complicated as soon as there is a backdoor link.
• EIGRP: similar appeal, similar complications, and only where the provider supports it.

What catches people with eBGP

• Every site uses the same private AS, so a site rejects routes that contain its own AS number. The provider sets as-override, or you accept your own AS with allowas-in.
• A site with two links can advertise routes it learned from the WAN back into the WAN. Site of origin marks routes with their site, so a route never goes back to where it came from.

What catches people with OSPF

• Routes across the provider arrive as inter-area routes. A backdoor link between two sites in the same area carries intra-area routes, which always win, so the backdoor carries everything. A sham link across the provider fixes it.
• The provider's PEs need a matching domain ID, or your routes arrive as externals.

Whichever you choose, get the provider's routing design in writing: which protocol, which timers, what they filter and what they summarise. Two sites failing over differently because two PEs were configured differently is a common, slow fault.

Related: Two WAN carriers · Two WAN providers, one VRF · VPN, VRF and MPLS

Two WAN carriers: what you gain and what you take on

Two MPLS carriers sounds like the obvious way to stop one carrier's outage taking out every site. It does that. It also makes you responsible for joining two networks that were never designed to meet.

What you gain

• No single carrier fault, routing mistake or core upgrade takes everything down.
• Leverage at contract renewal.

What you take on

• Routing between the two. Each carrier only knows its own sites. Sites on both carriers become the meeting points, and without filtering, a dual-homed site can end up carrying traffic between the two carriers, over its own links.
• Different behaviour. Different QoS classes and markings, different PE-CE options, different failover timers, different support processes.
• Single-homed sites. A site on only one carrier still loses everything when that carrier fails. Decide which sites justify both.
• Physical diversity. Two carriers often share the same last mile, the same duct or the same exchange. Ask both for route diagrams, in writing, before you sign.

The design rules I'd use

• eBGP to both carriers, with a clear preference per region, set with local preference.
• Mark routes with the carrier they came from, using a community, and never re-advertise one carrier's routes to the other unless that's deliberate.
• Map your QoS classes to each carrier's classes explicitly, and test that markings survive.
• Test a full carrier outage at a quiet time, with traffic, before you need it.

An SD-WAN overlay across two carriers, or across a carrier and the internet, hides many of these differences. It doesn't remove the shared-duct problem.

Related: Routing with your MPLS provider · Choosing between designs · Becoming a transit network

OSPF over hub-and-spoke and full mesh

OSPF was built for links where every router can talk to every other. Hub-and-spoke WANs and big full meshes break that assumption, and the network type you choose decides what works.

Hub and spoke (DMVPN, Frame Relay before it)

• Broadcast network type: a DR is elected, and it must be the hub, because only the hub can reach everyone. Set ip ospf priority 0 on every spoke. Next hops are preserved, so spokes can build direct tunnels to each other.
• Point-to-multipoint: no DR to get wrong, but the hub becomes the next hop for everything. Fine when all traffic goes via the hub, or with DMVPN phase 3, where the hub redirects spokes to talk directly.
• Scaling: all spokes in one area means every spoke's tunnel flap is flooded to every other spoke. Put the spokes in a stub or totally stubby area, keep the area small, or use EIGRP or BGP for large DMVPN networks.

Full mesh

Every router floods every change to every neighbour, so a full mesh of n routers sends far more copies of each update than it needs. On large meshes, stop flooding on some links with ip ospf database-filter all out, leaving at least two paths for every router to hear updates. (IS-IS has mesh groups for the same job.)

interface Tunnel0
 ip ospf network broadcast
 ip ospf priority 0       ! on every spoke
!
! or, hub-only traffic or phase 3
interface Tunnel0
 ip ospf network point-to-multipoint
!
show ip ospf interface Tunnel0 | include Network Type|State|Designated

The one combination that never works: broadcast type with a spoke as DR. It forms adjacencies only with the routers it can reach, and routes appear and disappear depending on who won the election.

Related: Dual-hub DMVPN design · OSPF is only link-state inside an area · DMVPN troubleshooting

OSPF or IS-IS, and how to move between them

For most enterprises the IGP choice is OSPF by habit. IS-IS has real advantages in big networks. And if you ever need to move, the default administrative distances make it surprisingly gentle.

Where IS-IS is stronger

• It runs directly over layer 2, not IP, so it can't be attacked by IP packets from across the network.
• New features and address families are added as new fields (TLVs), so IPv6 and segment routing slot in without a second protocol.
• Area borders sit on links, not inside routers, and a flat level 2 backbone scales a long way. That's why it's common in service provider cores.

Where OSPF is stronger

• Most enterprise engineers know it, and so do most firewalls and appliances.
• Area types (stub, NSSA) give fine control over what each part of the network learns.

For a campus or an ordinary enterprise WAN, either works. Choose the one your team can troubleshoot at 2am.

Migrating without a big bang

OSPF's administrative distance is 110 and IS-IS's is 115, so you can run both side by side and OSPF keeps winning:

1. Configure IS-IS everywhere, alongside OSPF. Nothing changes in the routing table yet.
2. Check that IS-IS has the same neighbours and would produce the same routes: compare its database and topology with OSPF's.
3. Raise OSPF's distance above 115 (distance 120), router by router or all in one window. IS-IS routes take over.
4. Run like that for a while. Then remove OSPF.

router isis CORE
 net 49.0001.0100.0000.0001.00
 is-type level-2-only
 metric-style wide
interface TenGigabitEthernet0/0/0
 ip router isis CORE
 isis network point-to-point
!
show isis neighbors
show isis topology
!
router ospf 1
 distance 120

Rolling back at any step is one command: put OSPF's distance back.

Related: OSPF is only link-state inside an area · Redistribution rules

Choose the multicast model before the rendezvous point

Multicast designs usually start with "where do we put the RP?" The better first question is whether you need one at all. The application decides which PIM model fits.

• Source-specific multicast (SSM): receivers ask for a specific source and group. No rendezvous point, no shared tree, far less to go wrong. It needs IGMPv3 on the receivers and applications that know the source. Default range 232.0.0.0/8. The best choice for one-to-many feeds such as video distribution or market data from known sources.
• Any-source multicast (ASM, sparse mode): receivers ask for a group from any source, and a rendezvous point introduces sources to receivers. Use it when receivers can't name the source. Make the RP redundant with anycast RP: two or more routers sharing one address, keeping each other informed of active sources.
• Bidirectional PIM: one shared tree, with no per-source state. Built for many-to-many applications where every member is also a source, such as some trading and conferencing systems. The RP address doesn't even need to belong to a router.

ip multicast-routing
ip pim ssm default
interface Vlan10
 ip pim sparse-mode
 ip igmp version 3
!
show ip mroute
show ip igmp groups

Then the practical rules

• PIM on every interface in the path, including the transit links nobody thinks about.
• IGMP snooping on the switches, so multicast reaches only the ports that asked for it.
• The unicast routing table decides the multicast path (reverse path forwarding). Asymmetric routing makes multicast fail in ways unicast never shows.
• Ask what happens when the WAN is congested. A multicast video feed doesn't back off like TCP.

Related: Nexus EVPN fabric (multicast underlay) · mDNS stays in the VLAN

Do you actually need QoS?

QoS only does something when a queue builds. On a LAN with fast uplinks that is rare; on a WAN link it happens every day. Put the effort where the congestion is, and keep the policy simple enough that someone else can support it.

Where it matters

• Speed changes: a 1 Gbps LAN feeding a 100 Mbps WAN circuit. That's where queues build and packets drop.
• Sub-rate circuits: a gigabit port delivering a 200 Mbps service. Unless you shape to 200 Mbps, the carrier's policer drops the excess, ignoring your priorities completely.
• Real-time traffic: voice and video suffer from delay and jitter long before they show loss.

Rules that keep it supportable

• Mark at the edge, trust at a defined boundary. Trust the phone, or the access port the phone sits on, not every PC that has set its own markings.
• Few classes. Four to eight classes cover almost every enterprise. Each extra class is something else to map to the carrier and explain.
• Cap the priority queue. Priority traffic beyond about a third of the link starves everything else.
• Match the carrier. Find out which markings they honour, how many classes they offer, and what they do to traffic that arrives marked for a class you haven't bought.

If the link never congests, QoS does nothing. If it congests all the time, QoS only decides who suffers. Sometimes the right answer is more bandwidth.

Related: Output drops on a shaped port

Passed the CCNA. Now the job starts

An engineer who had just passed the CCNA posted about the week after: tunnels that are up but quietly dropping traffic, BGP changes they could follow but not explain, carriers who reply "normal parameters on our end", and a vocabulary nobody teaches. It's a fair list, and every one of us has stood there. Here is what I would tell them.

The exam teaches config. The job is mostly evidence.

The CCNA proves you know how things are meant to work. Most of the job is proving where they don't, to people who would rather it wasn't them. That takes three habits more than more theory:

• Measure two layers at once. When something degrades, test the layer you think is broken and the one underneath it at the same time. That single habit answers most "tunnel or circuit?" questions. How to do it for a tunnel.
• Predict before you change. Before every runbook change, write down what you expect to see afterwards: which path wins, which prefix moves, which counter rises. Then check. When you're right, you've learned it. When you're wrong, you've found the gap before it found you. Applied to BGP and PBR.
• Write tickets a carrier can act on. Times, circuit ID, evidence from both sides of their equipment, and one specific request. What to attach, and how to say it's on their side.

The words

Hard down, degraded, flapping, demarc, soak test, RFO. Most of it is everyday shop talk, some belongs in a formal email, and a few terms mean slightly different things to different carriers. The plain-English list, with which ones to use in writing.

What to pay attention to while the experience builds

• Keep a notebook of every fault: the symptom, what you checked, what it turned out to be. After a year, it's the most useful reference you own.
• Save the output before and after every change. The difference between them is where you learn what a command really does.
• Read the closure notes senior engineers write. Copy the shape, not the words.
• When a carrier fixes something, ask what it was. "Fault cleared" teaches you nothing. "Errored port on our aggregation switch, moved to a spare" teaches you what to look for next time.

And yes, much of it is experience. But experience is just evidence you've collected and kept. You can start collecting on purpose today.

Related: BGP failover ticket lab · Routing Tricks · Ticket and Email Writer · The circuit that was fine

A degraded tunnel, or the circuit underneath it?

The tunnel is up, the routing is up, and users say it's slow or dropping. Before you touch the crypto config, find out which layer is losing the packets. Test the overlay and the underlay at the same time, from the same box, and compare.

1. Test both layers, side by side

From the router at one end, run a long ping across the internet to the far end's public address (the underlay), and another through the tunnel to an inside address at the far end (the overlay). Same time, same count, same size.

! underlay: outside interface to the peer's public address (add vrf if you use a front-door VRF)
ping vrf INET 203.0.113.20 source GigabitEthernet0/0/1 repeat 1000 size 1400 timeout 1
!
! overlay: inside to inside, through the tunnel
ping 10.20.0.1 source Loopback0 repeat 1000 size 1300 timeout 1

• Underlay losing, overlay losing about the same: it's the circuit or the internet path. The tunnel is just the messenger. Go to the carrier with the underlay figures.
• Underlay clean, overlay losing: it's the tunnel itself. Keep reading.
• Both clean, users still unhappy: check the application, the server or the far LAN before blaming either.

2. If the tunnel is the problem

• Size. Repeat the overlay ping at 100, 1300 and 1400 bytes with df-bit. If small gets through and large fails, it's MTU or fragmentation, not loss. Set the tunnel MTU and clamp TCP MSS.
• Counters at both ends. Compare encrypts on one side with decrypts on the other. If one end encrypts and the other never decrypts as many, packets are being lost or filtered between them, often ESP or UDP 4500 being dropped by something in the path.
• Replay drops. QoS that reorders packets inside a tunnel can push them outside the anti-replay window, so they're discarded even though they arrived. Look for replay errors in the logs and the counters.
• Load. Does it get worse when the link is busy? That points at a policer on the carrier side, a shaper on yours, or an encryption engine running out of headroom.

show crypto ipsec sa peer 203.0.113.20 | include encaps|decaps|errors|replay
show crypto session remote 203.0.113.20 detail
show interfaces Tunnel1 | include drops|rate
show logging | include REPLAY|CRYPTO

3. Narrow down the underlay

• Your handoff first. Errors, CRCs and light levels on the interface facing the carrier. If they're clean, write that down: it's your evidence.
• Their first hop. Ping the carrier's next hop (their PE) as well as the far end. Loss that starts at their first hop and carries on is their access circuit. Loss that only starts further out is the wider path.
• The other end. Run the same test from the far router. A fault on the far site's access circuit shows up there and not at your end.
• Over time. Intermittent faults rarely show up in a two-minute test. An IP SLA probe on each layer gives you hours of history to put in front of the carrier.

ip sla 30
 icmp-echo 203.0.113.20 source-interface GigabitEthernet0/0/1
 vrf INET
 frequency 10
 history hours-of-statistics-kept 24
ip sla schedule 30 life forever start-time now
ip sla 31
 icmp-echo 10.20.0.1 source-interface Loopback0
 frequency 10
 history hours-of-statistics-kept 24
ip sla schedule 31 life forever start-time now
!
show ip sla statistics aggregated 30
show ip sla statistics aggregated 31

Two graphs, overlay and underlay, on the same timeline. When they dip together it's the circuit. When only the overlay dips, it's yours.

Related: MTU Calculator · DMVPN troubleshooting flow · Latency and Loss Monitor · Opening the carrier case

Opening a carrier case that gets worked, not parked

"Link is up, normal parameters on our end." Then three days of nothing. A ping result on its own invites exactly that reply. A case with the right evidence and one clear request is much harder to park.

What to put in the first message

• The circuit ID and the site, exactly as the carrier knows them. Wrong references cost a day.
• Hard down or degraded, in those words, because it sets the priority.
• When it started, with the time zone, and whether it's constant or comes and goes.
• The business impact in one line: "The Manila call centre cannot take calls."
• Evidence that your side is clean: no errors on the handoff interface, light levels in range, and when you cleared the counters.
• Evidence of where it breaks: a long ping or trace from your edge to their first hop and beyond, with timestamps, packet size and loss. Loss that starts at their first hop is the strongest thing you can send.
• What you have already done: rebooted the CPE, swapped the patch lead, tested from the far end.
• One specific request. "Please check your PE interface facing us for errors in the last 24 hours" gets an answer. "Please investigate" gets "normal parameters".
• Site access: contact name, phone and hours, so an engineer visit isn't delayed a day.

When they say it's fine

A single test passing doesn't mean much for an intermittent fault. Reply calmly, with what would prove it either way:

Thanks for checking. The fault comes and goes, so a single test can pass. Our monitoring shows 3 to 5% loss between 09:00 and 11:00 local time, starting at your first hop. Please could you check the error history on your side for that window, or run a monitored test over the same period?

Then ask for three things before you put the phone down: the ticket number, the time of the next update, and how to escalate if that update doesn't come.

Saying "it's on your side" without a fight

• Point at the evidence, not the people: "From our edge, the loss starts at your first hop" rather than "your network is broken".
• Show you've ruled out your own kit first. It makes the carrier's engineer an ally rather than a defendant.
• Ask, don't demand: "Please could you check..." works in every language and every culture.
• Keep the tone the same in the fourth email as in the first. Escalate with the paper trail, the dates and the impact, not with frustration.

Read their manual too

If the carrier's box at your site is the suspect, read its documentation. I once had a circuit that "tested fine" for weeks until I asked the provider to try one specific mode on their own device. It burst into life. A specific request based on their own kit is much harder to dismiss.

Related: The circuit that was fine · Latency and Loss Monitor · Ticket and Email Writer · The words carriers use

The words carriers use, in plain English

Hard down, degraded, flapping, demarc, soak test, RFO. Nobody teaches this vocabulary, and if English is your second language it's hard to tell shop talk from what belongs in a formal email. Here is the working list, with which ones to use in writing.

Three phrases that work in any email

• "Please could you confirm..." asks for a clear yes or no.
• "The evidence suggests..." points at the cause without blaming anyone.
• "Our next update to the business is at 14:00, so we need a reply before then." gives a real deadline and the reason for it.

When the meaning really matters, check the term in the carrier's own documentation or contract. Words like "demarc" and "soak test" are used slightly differently from one carrier to the next.

Related: Opening a carrier case · Ticket and Email Writer

Learning BGP path selection and PBR for real

Following the runbook, the change works, and you couldn't explain why. That's how most of us start with BGP. Three ideas and one habit turn it from pattern matching into understanding.

Idea 1: you control your way out, and only influence your way in

• Outbound traffic (how you leave) is decided by your routers. Local preference is the main tool: highest wins, and it's shared with every router in your AS.
• Inbound traffic (how others reach you) is decided by other people's routers. You can only influence it: AS-path prepending, MED to a neighbour with several links to you, more specific prefixes, or the provider's own communities.

Most confusion goes away once you sort every change into one of those two lists.

Idea 2: the first few steps decide almost everything

The best-path list is long, but in real networks most decisions are made by the first few: weight (Cisco only, this router only), local preference, locally originated, AS-path length, origin, MED. Learn those properly. Look up the rest when you need it.

show ip bgp 198.51.100.0
show ip bgp neighbors 192.0.2.1 received-routes
show ip bgp neighbors 192.0.2.1 advertised-routes

The first command lists every path for a prefix and marks the best one. Read it until you can say which step chose it.

Idea 3: PBR overrides routing, on the way in, for matching traffic only

• A route-map on an interface catches matching packets as they arrive on that interface and sends them somewhere else. Everything that doesn't match is routed normally.
• It doesn't affect traffic the router itself generates, unless you add a local policy.
• If every next hop in the policy is unavailable, the packet falls back to the routing table. A policy route is a preference, not a fence.

The habit: predict, then check

Before you apply any runbook change, write down what you expect: which path will become best, which neighbour will receive which prefix, which way the traffic will flow. Apply it. Check against your prediction. When you were wrong, work out why before you move on.

Do the same in a lab first, where wrong costs nothing. Break things on purpose: remove a filter, raise a local preference, prepend on the wrong neighbour, and watch what moves.

Related: BGP failover ticket lab · Routing Tricks · Dual-homed BGP edge design · BGP Regex Tester · How to read a route-map

Drain the traffic before you touch the router

Shutting a router down mid-change makes the network find out the hard way, and every flow through it drops while routing converges. Move the traffic away first, wait for it to go quiet, then do the work.

Most change plans say "reload the router" and let the routing protocols sort it out. They do, eventually. Meanwhile traffic is black-holed for the hold time, the dead interval or however long BFD takes to notice, and that is the outage people remember.

Every routing protocol has a polite way to say "stop using me". Use it, check the traffic has moved, and only then start.

OSPF

router ospf 1
 max-metric router-lsa
! afterwards
router ospf 1
 no max-metric router-lsa

The router advertises its links at the maximum cost, so everyone routes around it if they have any other path, but it still forwards traffic for anyone who doesn't. Add max-metric router-lsa on-startup 300 permanently so a router that reloads doesn't attract traffic before BGP and its tables are ready.

IS-IS

router isis
 set-overload-bit

BGP

router bgp 65010
 neighbor 192.0.2.1 shutdown graceful 120 community 65535:0

The graceful shutdown community (65535:0) asks the neighbour to drop the routes to the lowest preference, waits the time you give it, then closes the session. Check the neighbour actually honours the community. If they don't, set a low local preference or prepend on the routes you send them before shutting the session.

First hop redundancy

Lower the HSRP or VRRP priority so the other router becomes active before you start, rather than letting the hello timers decide.

Nexus

NX-OS wraps all of this up as maintenance mode: system mode maintenance isolates the switch from the routing protocols in one step, and no system mode maintenance brings it back.

Then check it moved

show interfaces | include rate
show ip ospf database router self-originate
show ip bgp neighbors 192.0.2.1 advertised-routes

Watch the interface rates fall towards zero before you go any further. If they don't, something still prefers this box, and you've found it before it became an outage rather than after.

Related: How I run an implementation · Change Window Kit

Why your routing takes 40 seconds to notice a dead neighbour

When there is a switch, a provider handoff or a media converter between two routers, the link stays up after the far end dies. The routing protocol only notices when its timers run out. BFD fixes that.

The problem

If a router's own port goes down, routing reacts in a moment. But most links aren't a cable between two routers. There's a switch, a provider's Ethernet service or a firewall in the middle. When the far router dies, your port stays up and your router keeps sending traffic into nothing until the protocol gives up:

• OSPF: the dead interval, 40 seconds by default on Ethernet.
• EIGRP: the hold time, 15 seconds.
• BGP: the hold time, 180 seconds by default.

BFD

Bidirectional Forwarding Detection sends small, fast hellos between the two routers and tells the routing protocol the moment they stop. Typical settings detect a failure in well under a second.

interface GigabitEthernet0/0/1
 bfd interval 300 min_rx 300 multiplier 3
!
router ospf 1
 bfd all-interfaces
!
router bgp 65010
 neighbor 192.0.2.1 fall-over bfd

That's 300 ms hellos and three missed means down, so about a second. Both ends need BFD configured, and the slower of the two settings wins.

show bfd neighbors details

Things to be careful with

• Don't go aggressive by default. 50 ms timers on a busy or software-forwarding router will flap the neighbour, which is worse than slow detection. Start at 300 ms and tighten only where it's needed and the platform runs BFD in hardware.
• BFD through a firewall needs UDP 3784 (and 3785 for echo) allowed, and some firewalls don't pass it well. Test it.
• For eBGP between loopbacks you need multihop BFD, which is configured differently from single-hop.
• If you can't run BFD, shorten the protocol timers instead, for example timers bgp 10 30. It's cruder, but much better than three minutes.

Related: Drain the traffic before you touch the router · Proven Designs

The flapping link that keeps the whole network busy

A circuit that bounces every few seconds makes every router recalculate every time. IP event dampening takes a flapping interface out of routing until it has been stable for a while.

A dirty fibre or a failing provider circuit rarely dies cleanly. It goes down and up, down and up. Each transition triggers SPF runs, route updates and CPU spikes on every router in the area, and traffic keeps getting pulled back onto a link that's about to fail again.

Dampening gives the interface a penalty each time it goes down. When the penalty passes a threshold, routing treats the interface as down, whatever its physical state, until the penalty has decayed.

interface GigabitEthernet0/0/2
 dampening

The defaults on IOS and IOS XE:

• Each flap adds 1000 to the penalty.
• The penalty halves every 5 seconds (the half-life).
• Above 2000 the interface is suppressed. Two quick flaps in a row will do it.
• Below 1000 it's released again.
• It's never held down for longer than 20 seconds by default.

Those defaults catch very fast flapping. For a circuit that bounces every minute or so, lengthen the half-life and the maximum suppress time so it stays out of routing for longer:

interface GigabitEthernet0/0/2
 dampening 30 1000 2000 120

show dampening interface
show interfaces dampening

Only use it on links that have somewhere else to send the traffic. Dampening the only uplink at a site turns a flaky connection into no connection.

And it's a sticking plaster. The real fix is the fibre, the optic or a conversation with the provider, and the dampening counters are good evidence for that conversation.

Related: Why your routing takes 40 seconds to notice a dead neighbour · Latency and Loss Monitor

OSPF is only link-state inside an area

Inside an area every router has the full map. Between areas it just gets a list of prefixes and costs from the ABRs, which is distance vector in all but name. That explains most of the odd OSPF rules.

What a router actually knows

Inside its own area, every OSPF router has the same database: every router, every link, every cost. It runs SPF over that and works out the best path itself.

For anything in another area, it doesn't get the topology. The ABR sends a type 3 summary that says "I can reach 10.20.0.0/16 at a cost of 30". The router adds its own cost to the ABR and trusts the number. That's how a distance-vector protocol works.

Why the backbone rule exists

Distance-vector protocols can loop. OSPF stops inter-area loops with one blunt rule: all traffic between areas goes through area 0, and ABRs only use summaries learned over the backbone. That gives you a hub-and-spoke shape, and a hub and spoke can't loop.

It's why a non-backbone area must touch area 0, and why a virtual link exists as the patch when it doesn't.

What it means in practice

• Traffic can take a longer path than you expect. Two sites in area 10 with a direct link between them still send area 20 traffic via whichever ABR advertises the lowest total cost, not via the shortest physical path.
• Convergence can briefly go wrong. When a prefix disappears, routers in another area may briefly route to it via an ABR that hasn't withdrawn its summary yet. It lasts a few seconds, but a floating static or backup path won't take over until those summaries go.
• Summarise at the ABR. Since other areas only see summaries anyway, make them proper ones with area 10 range. A flapping subnet then doesn't trigger SPF in every area.

router ospf 1
 area 10 range 10.10.0.0 255.255.0.0
!
show ip ospf database summary 10.10.0.0
show ip ospf border-routers

Related: Where the OSPF default route comes from · Cisco Gotchas

Where the OSPF default route comes from, area by area

Whether a router gets a default from OSPF depends on the area type, and one of them, NSSA, doesn't give you one unless you ask. One table and one rule save a lot of head-scratching.

The NSSA trap

People convert a stub area to NSSA so they can redistribute a static route at the branch, and the branch loses its default. Stub gave one for free and NSSA doesn't. Add default-information-originate to the NSSA statement on the ABR, or use no-summary if a default is all the area needs.

The always trap

default-information originate only advertises a default while the router has one in its routing table. Adding always advertises it regardless, so an edge router that has lost its internet link keeps attracting all the traffic and drops it. Use always only when something else guarantees the path.

router ospf 1
 default-information originate metric 10 metric-type 1
!
show ip ospf database external 0.0.0.0
show ip ospf database nssa-external
show ip route 0.0.0.0

With two edge routers, metric-type 1 makes each router add its internal cost, so traffic goes to the nearer exit. With the default E2, every router sees the same cost and the choice is a tie-break.

Related: OSPF is only link-state inside an area · Default routes in BGP

NTP: the boring service that breaks VPNs and investigations

Wrong time doesn't look like a fault until a certificate fails, a VPN won't come up or you try to line up logs from ten devices after an incident. Ten lines of config prevent all three.

Why it matters

• Certificates. They have start and end times. A router that boots with its clock in 2002 rejects every certificate it's shown, so certificate-based VPNs and secure management fail.
• Logs. After an incident you'll want to put events from the firewall, the core and the WAN router in order. Without the same time on every device, you can't.
• Small routers. Many branch routers have no battery-backed clock. After a power cut they don't know the time until NTP tells them.

The config

ntp server 10.0.0.11 prefer
ntp server 10.0.0.12
ntp server 10.0.0.13
ntp source Loopback0
!
clock timezone GMT 0
clock summer-time BST recurring last Sun Mar 1:00 last Sun Oct 2:00
service timestamps log datetime msec localtime show-timezone
service timestamps debug datetime msec localtime show-timezone

• At least three servers. With two that disagree, the router can't tell which is right. With three or more, the odd one out is outvoted.
• Source it from a loopback so the server's ACL only needs one address per device, whichever interface the packet leaves by.
• Put the timestamps in the logs. Accurate time does nothing for you unless it appears in the syslog messages.
• Don't use ntp master casually. It makes the router claim to be an authoritative clock even when it isn't synchronised. Use it only on a box you mean to be the time source.
• Authenticate it where someone could feed you a false time.

show ntp associations
show ntp status
show clock detail

In show ntp associations, the asterisk marks the server you're synchronised to. No asterisk means no sync, whatever the clock says.

Related: Cisco Gotchas: log timestamps you cannot line up · Troubleshooting

Ping fifty addresses from the router in one go

After a change you often need to check that a list of hosts still answers from the router's point of view. IOS has a scripting shell built in, and a three-line loop does it.

tclsh
foreach ip {
10.10.1.1
10.10.1.10
10.10.2.1
10.20.0.5
} { ping $ip repeat 2 timeout 1 }
tclquit

Paste the list in, and the router pings each one in turn. A line of exclamation marks is a pass, dots are a fail. Run it before the change, save the output, run it again afterwards and compare.

Variations

! from a particular source, or in a VRF
foreach ip {10.10.1.1 10.10.1.10} { ping vrf CORP $ip source Loopback0 repeat 2 }

! any exec command works, not just ping
foreach i {1 2 3 4} { exec "show interfaces GigabitEthernet1/0/$i" }

Things to know

• It needs privilege level 15.
• Type tclquit when you're done. Until then you're still in the Tcl shell, where some commands behave differently.
• It's on IOS and IOS XE. NX-OS has its own tclsh and Python as well, with slightly different syntax.
• Keep the lists in your change plan. The same list before and after is the evidence.

Related: CLI Filter Builder · Change Window Kit

Two equal paths, and one of them carries nothing

You built equal-cost paths at every layer, and the second link at the distribution layer sits idle. It's called CEF polarisation, and it comes from every router making the same hashing decision.

How CEF shares load

With equal-cost routes, a Cisco router doesn't alternate packet by packet. It hashes the source and destination addresses and sends each flow down one path. That keeps packets in order, which is what you want.

Why it polarises

The core router hashes flows into "left" and "right". All the "left" flows arrive at the left distribution router. If that router uses the same hash, every one of those flows lands on the same answer again, so it sends them all down one of its two uplinks. The other one carries nothing.

The fix

ip cef load-sharing algorithm universal

The universal algorithm mixes a value unique to each router into the hash, so each layer splits traffic differently. It's the default on most current IOS and IOS XE platforms, so polarisation usually means an old box, an old setting or a platform that does it in hardware differently. On those, adding the ports to the hash helps too:

ip cef load-sharing algorithm include-ports source destination

Checking it

show ip cef 10.20.0.0/16 detail
show ip cef exact-route 10.1.1.10 10.20.5.20

exact-route tells you which path a given flow takes, which is the quickest way to prove polarisation. Pick a handful of real source and destination pairs and see if they all land on the same link.

And a few large flows will always load links unevenly, however good the hash. Equal-cost paths share flows, not bandwidth.

Related: A 4 x 10G port-channel that carries 10G · BGP uses one link out of two

Default routes in BGP: three ways in, and the one that black-holes

BGP treats 0.0.0.0/0 differently from every other prefix. There are three ways to get one into BGP, and the most common one keeps advertising a default after the router has lost the internet.

1. Redistribution on its own will not do it

Put a static default on a router, add redistribute static under BGP, and every static route appears in the BGP table except the default. IOS leaves it out on purpose. You have to ask for it.

router bgp 65010
 address-family ipv4
  redistribute static
  default-information originate   ! now the static default comes in too

2. A network statement for 0.0.0.0

router bgp 65010
 address-family ipv4
  network 0.0.0.0

Like any network statement, it only works while a default route is in the routing table. Once it is in the BGP table, it is advertised like any other prefix, to iBGP and eBGP neighbours alike, unless you filter it.

That is fine in a private network running BGP as its core protocol, where you want the branches to have one route instead of hundreds. On an internet edge it is usually a mistake: you end up offering a default to neighbours who never asked for one, including your other ISP.

3. Per neighbour, with default-originate

Most of the time what you actually want is to send a default to one neighbour, such as a branch or a customer, without it sitting in your own BGP table.

router bgp 65010
 address-family ipv4
  neighbor 192.0.2.6 default-originate

The catch: plain default-originate sends the default whether or not this router can reach anything. If the uplink dies, the router keeps telling the branch "send everything to me", and drops it all. That is a black hole, and it is the failure that matters in a dual-homed site.

Make it conditional

Pick a route you only have while the uplink is healthy, for example the provider's loopback or a /32 learned from the core. Then send the default only while that route is in the routing table.

ip prefix-list UPLINK-HEALTHY seq 5 permit 198.51.100.1/32
!
route-map SEND-DEFAULT-IF-UP permit 10
 match ip address prefix-list UPLINK-HEALTHY
!
router bgp 65010
 address-family ipv4
  neighbor 192.0.2.6 default-originate route-map SEND-DEFAULT-IF-UP

Lose the uplink, lose the /32, and the default is withdrawn. The branch then uses its other path instead of sending traffic into a router that cannot forward it.

Checks

show ip bgp 0.0.0.0
show ip bgp neighbors 192.0.2.6 advertised-routes
show ip route 198.51.100.1

Test it properly: shut the uplink in the change window and watch the default disappear on the branch. A tracking route that is also learned some other way, for example through an IGP backup, will never go away and the condition will never trigger.

The same trap exists in OSPF with default-information originate always. It is on the Gotchas page.

Related: Proven Designs: BGP edge · Cisco Gotchas · The /20 in a prefix list is not a mask

Output drops on a shaped 3850 port: it was the buffers

A Catalyst 3850 WAN port shaped to 300 Mbps, a clean-looking policy, and millions of drops in class-default within hours. The policy was fine. The port simply did not have enough buffer to absorb the bursts.

The ticket

A 3850 handed off to a 300 Mbps WAN service over a gigabit port. The parent policy shaped everything to 300 Mbps and a child policy gave voice and video priority. Users complained, and the counters showed why: millions of output drops in class-default in about ten hours.

policy-map PM-WAN-CHILD
 class CM-VOICE
  priority level 1
  police rate percent 10
 class CM-VIDEO
  priority level 2
  police rate percent 10
 class class-default
  bandwidth remaining percent 100
!
policy-map PM-WAN-SHAPE
 class class-default
  shape average 300000000
  service-policy PM-WAN-CHILD
!
interface GigabitEthernet1/0/23
 service-policy output PM-WAN-SHAPE

Why it drops

Shaping a gigabit port down to 300 Mbps means traffic arrives faster than it can leave, so it has to wait somewhere. On a 3850 that somewhere is small. By default a 1 Gbps port gets 300 buffers and a 10 Gbps port 1,800, at 256 bytes each, with some room to borrow from a shared pool. Bursty TCP fills that in moments, and the overflow is dropped.

The fix

Let the queues borrow more from the shared pool, then give the class that is dropping a bigger share of the port's buffers.

conf t
 qos queue-softmax-multiplier 1200
end
!
policy-map PM-WAN-CHILD
 class CM-VOICE
  queue-buffers ratio 10
 class CM-VIDEO
  queue-buffers ratio 20
 class class-default
  queue-buffers ratio 70

The multiplier is global and goes from 100 (the default) to 1200. Check that it has taken by looking at the Softmax column for the port.

show platform qos queue config GigabitEthernet1/0/23
show platform qos queue stats GigabitEthernet1/0/23
show policy-map interface GigabitEthernet1/0/23 output class class-default

Things I would check on any 3850 or 9300 with drops

• Is it a burst problem or a bandwidth problem? If the average is well under the shaped rate and you still drop, it is buffers.
• If you edit a policy while it is applied and the counters start looking strange, remove the service-policy from the interface and put it back.
• Tune and watch. These values depend on your traffic, so change one thing, then compare the drop counters over the same busy period.

Related: Ping works, so it's not the network · Cisco Gotchas

Replacing a dead switch in a stack without taking the stack down

The guidelines I wrote for a colleague swapping a failed member out of a Catalyst 3750 stack. The same checks apply to 3850 and 9300 stacks today.

The risk with a stack replacement is not the failed switch. It is the new one joining with the wrong software, the wrong template or a higher priority, and upsetting the members that are still working. These are the checks, in order.

Before you touch the replacement

• Back up the stack. Save the running config and copy off vlan.dat.
• Note the number and priority of the member you are replacing. The master is the member with the highest priority (up to 15). The replacement needs a lower priority than the current master.
• Match the software. The replacement must run exactly the same IOS as the stack. Check with show version and upgrade it on the bench first if it differs.
• Match the SDM template. Check with show sdm prefer. A template mismatch keeps the member out of the stack.
• Wipe the replacement. Erase its startup config and its vlan.dat, so it brings nothing with it, including any old switch N provision lines.

show switch
show version
show sdm prefer
! on the replacement, before it joins
switch 2 priority 1
write erase
delete flash:vlan.dat

Putting it in

• Cable the replacement while it is powered off, into exactly the same stack ports the old one used. That way it picks up the old member number, and the stack applies that member's port config, which is still there under the switch N provision line.
• Power it on only when every cable is in.
• Watch it join. It should come up as a member, not take over.

show switch
show platform stack manager all

Newer stacks (3850, 9200, 9300) can auto-upgrade a member running different software if you have enabled it. I still check the version first, because a stack deciding on its own what to do at 2am is not something I want to be surprised by.

Related: Label before you pull · Change Window Kit

The /20 in a prefix list is not a mask

Someone exporting a /20 between VRFs could see the /21s and /24s inside it, but never the /20 itself. Three things turned up, and the first is the one that catches most people.

The problem

A static route to Null0 for a /20 was redistributed into BGP and exported from one VRF to another with an export map. On the far side, the routes inside the block appeared but the /20 did not.

ip route vrf VRF-A 10.16.0.0 255.255.240.0 Null0
!
ip prefix-list EXPORT-PL seq 5 permit 10.16.0.0/20
!
route-map EXPORT-MAP permit 10
 match ip address prefix-list EXPORT-PL
 set extcommunity rt 65000:100 additive
!
vrf definition VRF-A
 address-family ipv4
  export map EXPORT-MAP

1. Length is not a mask

In a prefix list, 10.16.0.0/20 means "the first 20 bits must be 10.16.0.0, and the route must be exactly a /20". Add ge and le to match a range of lengths.

permit 10.16.0.0/20            ! only the /20 itself
permit 10.16.0.0/20 le 24      ! the /20 and anything inside it down to /24
permit 10.16.0.0/16 ge 20 le 20 ! any /20 inside 10.16.0.0/16

Most mistakes go the other way: people write the /20 expecting it to match everything inside, and only the /20 gets through. Here the question was simply which routes the export needed, and writing that down settled the list.

2. Look for another list that denies it

Further down the config was a second prefix list denying the same block with le 32, applied elsewhere in the routing policy. Search the whole config for the prefix, not just the list you think is in play.

show running-config | include 10.16.0.0
show ip prefix-list detail
show ip bgp vpnv4 vrf VRF-A 10.16.0.0 255.255.240.0

3. Make sure the change has actually taken

After editing the route-map, the export behaved as if the old version was still in force. Removing the export map from the VRF, putting it back and refreshing BGP sorted it. If the counters and the output disagree with the config, do that before you start doubting the logic.

Overlapping statics are worth a look too. Here there was a /20 and a /21 to Null0 for the same space, which made the outputs harder to read than they needed to be.

Related: ACL, prefix list, route-map, distribute-list · VPN, VRF and MPLS

Before an NX-OS upgrade: the bug list, the EPLDs and the bootflash

The prep I did before upgrading a pair of Nexus 7000s with F2 line cards. Most of the work happened weeks before the change window.

1. Read the bugs for your hardware, not the release notes headline

I searched the Bug Search Tool for the exact line card and supervisor part numbers we had, on the release we were running and the one we were going to. Then I listed each bug with the release that fixed it. On F2 cards that list included link flaps on empty ports, CRC errors on a group of ports, a port-channel load-balancing problem with layer 2 traffic and an upgrade failure on F2E. Most were fixed in the target release, which became part of the case for the change.

2. Check the impact before the night

show install all impact kickstart bootflash:<kickstart> system bootflash:<system>
show install all impact epld bootflash:<epld-image>
show sprom module <slot>

The impact check tells you whether each module upgrade is hitless or disruptive. EPLD upgrades are a separate step, can be disruptive, and some depend on the module's hardware version, which is why show sprom is in the list.

3. Clean the bootflash on both supervisors

Delete the old images first. Running out of space mid-copy is a known way to fail an upgrade before it starts.

dir bootflash:
dir bootflash://sup-standby/
delete bootflash:<old-image>

4. Write the plan and the back-out plan

The detailed plan had every command in order, the expected output and the checks between steps: vPC consistency, port-channels, routing neighbours. The back-out plan said exactly which images to boot and how to get there. A short "stripped down commands" page for the night itself meant nobody was scrolling through the long version at 1am.

Related: When the revert fails · Change Window Kit

UCS: the commands I kept for a blade that will not behave

A short list from looking after Cisco UCS, for the blade that shows memory errors, the CIMC that stops answering and the IOM that might be the real problem.

Clear memory errors on a blade

scope server 4/7
reset-all-memory-errors
commit-buffer

scope server takes chassis and blade, so 4/7 is chassis 4, blade 7. If the errors come straight back, it is a DIMM, not a glitch.

Reset the CIMC when the blade stops reporting

scope server 4/7
scope cimc
reset
commit-buffer

Resetting the CIMC restarts the blade's management controller, not the server itself. The host keeps running.

Check the fabric interconnect cluster

connect local-mgmt a
show cluster extended-state

Both fabric interconnects should be up, one primary and one subordinate, with HA ready. Check this before and after any firmware work.

Look at the IOMs

show tech chassis 1 iom 1 brief | egrep 'fixup|lostar'
show tech chassis 1 iom 2 brief | egrep 'fixup|lostar'

Errors on the IOM, the module linking the chassis to the fabric interconnect, show up as problems on the blades behind it. Check both IOMs before you blame a blade.

Back up two ways, on a schedule

UCS Manager has two scheduled backups, and you want both. Full State is the one you restore after losing the fabric interconnects. All Configuration is the one you import to rebuild or compare config. Send them off the box with SFTP and check that the files are actually arriving.

Related: Split brain · When the revert fails

VPN, VRF and MPLS: three words people use for the WAN

Project managers, architects and the occasional blue-sky thinker call the company WAN a VPN, a VRF, MPLS, "VFRs" (really wrong) or "that network thing". Here is what each one actually is, at business level.

This is the high-level version. Each of these comes in many configurations, depending on what you need and what the provider is selling. The aim here is the real-world business meaning, not the protocol detail.

VPN

Say VPN and most people think of encryption, because that is what they use when they work from home. VPN stands for Virtual Private Network, and it comes in many forms, with or without encryption.

At its simplest, a VPN is a private environment that runs over another network you do not own.

Say company A has offices in New York, Paris and Peckham and needs them all to talk to each other privately. A service provider can deliver that over its existing infrastructure. Company B can buy the same product, and the two companies will never see each other.

Private does not automatically mean encrypted. It means separated.

VRF

VRF stands for Virtual Routing and Forwarding. It lets several isolated routing tables exist on the same physical kit: inside your own company, inside the provider's network, or both.

That means your company can keep data, voice, video and guest wireless apart so that they cannot see each other, even though they share the same switches and routers. Provider networks use exactly the same idea to keep customer X apart from customer Y.

For two VRFs to talk to each other they need something in the middle. Traditionally that is a firewall, so you control what crosses, though it can be a router.

vrf definition VOICE
 address-family ipv4
 exit-address-family
!
interface Vlan20
 vrf forwarding VOICE
 ip address 10.20.0.1 255.255.255.0
!
show ip route vrf VOICE
ping vrf VOICE 10.20.0.10

One trap: vrf forwarding removes the IP address from the interface, so put it on first and the address second.

MPLS

MPLS, Multiprotocol Label Switching, is one of the mechanisms that carries a VPN from point A to point B.

Routers keep a table of destinations and the next hop to reach each one, and they share those tables with each other. Multiply that by a million or so routes and you have the internet. In August 2014 the global table passed 512,000 routes, and older routers whose hardware was sized for 512,000 started dropping routes. That was "512k day".

MPLS puts a short label on each packet at the edge of the provider's network. Each router in the core reads the label, swaps it for the next one and passes the packet on, without looking at the customer's addresses at all. Combined with VRFs at the edges, that is what gives you an MPLS VPN.

VRFs on their own, often called VRF-lite, run on almost any router or layer 3 switch. MPLS VPNs need more: BGP to carry each customer's routes and label switching end to end. That is why you mostly meet them inside a service provider's network.

Related: Two WAN providers, one VRF · Proven Designs

The greatest network you've never seen

Almost every intercontinental byte you send travels through a cable on the sea bed. Most people have no idea the cables exist.

1858

On 29 July 1858, two steam ships met in the middle of the Atlantic and spliced together the two halves of a cable around 3,200 km long. For the first time, Europe and North America were linked by telegraph: a bit like email, with fewer cat pictures.

Queen Victoria's first message, 98 words, took almost 16 hours to get through. The cable failed within a few weeks, but a global revolution had begun.

Today

TeleGeography now tracks more than 600 active and planned submarine cables, with over 1.5 million kilometres in service. Over most of the ocean a cable is about as thick as a garden hose, carrying fibres as thin as a hair. Many of the newest are funded by the big cloud and content companies. Satellites carry only a tiny share of international traffic; the rest goes through these cables.

Yet in a world of wireless and mobile phones, people barely know they exist. The internet has become a mythical "cloud" that only gets noticed when it goes down. Please stop saying you broke the internet.

Why it matters to a network engineer

Light in fibre travels at about 200,000 km a second, roughly two thirds of its speed in a vacuum. London to New York is about 5,600 km in a straight line and the cables are longer than that, so a round trip can never be much under 60 ms, however good the network is. When a trace shows a big jump in delay at one hop that stays the same to the end, that is usually an ocean, not a fault.

The same internet is why branch offices no longer need a provider's private WAN for everything. With SD-WAN, a site can use local broadband, fibre or 5G to reach headquarters and cloud applications directly. A private WAN still has its place, but it is no longer the only option.

See every cable on the interactive map at submarinecablemap.com.

Related: Latency and Loss Monitor · The Networking Multiverse · Proven Designs

Split brain: when both halves think they are in charge

A pair built for resilience fails in the one way it was never meant to: both members go active at once. Mine was a Nexus 1000V after a UCS upgrade. The lesson applies to every redundant pair you own.

What happened

We had a Nexus 1000V running on a UCS platform: two virtual supervisors (VSMs) in an active and standby pair, and a VEM on every host doing the switching. After an upgrade of the UCS underneath, it started misbehaving.

The redundancy status told the story. Both VSMs had gone active, and the VEMs had attached themselves to one or the other.

show system redundancy status
show module

Two supervisors each believing they own the switch is worse than one supervisor failing. Each one is making decisions for its share of the hosts and neither knows about the other.

The fix

Set the roles explicitly, then reboot one VSM so it comes back as the standby and syncs from the active one.

system redundancy role primary
! on the other VSM
system redundancy role secondary
! then reload the VSM that should be standby

The Nexus 1000V for VMware has since reached end of life, but split brain has not. vPC peers, firewall clusters, wireless controller pairs and stacked switches can all end up there.

What I do now

• Check the redundancy status of every pair before and after any change underneath it: hypervisor, chassis, fabric interconnect or firmware. Upgrading the platform under an HA pair is a change to the pair.
• Know how each pair decides it is alone. A vPC uses the peer-keepalive, a cluster uses its sync and heartbeat links. Make sure that path cannot share a fate with the thing it is watching.
• Write the recovery step into the plan before you need it: which member you reboot and why. At 2am you do not want to be choosing.

Related: The one vPC rule · The keepalive path gotcha

Label before you pull: the night lightning hit the building

A call at about 2am, a building struck by lightning, and a core switch with half its slots dead. What got the servers back was a roll of labels and a new config for every port.

The call

Around 2am my manager rang: "I know you're not on call, so you know I wouldn't be calling if it wasn't urgent." I was in the car within ten minutes. The building had been struck by lightning and it had taken everything out.

Fix the core first

The first item on the list was the core switch, a Catalyst 6009 running CatOS. We thought only the layer 3 daughter card on its supervisor had blown. The maintenance contract was a good one, the part was on site within about an hour, and we swapped it.

The servers on that switch were still offline. That was not a surprise: when every minute costs money, you fix routing and the core building blocks first, because until the network can get from A to B, servers are expensive table tops. The core came first and the servers came next.

Half a backplane

The diagnosis took a while. The outage had left the chassis with only every other slot working. The servers were spread evenly across the line cards, so there was just enough room to move everything onto the slots that still worked. Easier said than done:

1. Label every cable with the port it was in.
2. Pull them out and move them to one side.
3. Move the switch modules around until they all came up.
4. Write a new config for every port, in CatOS.
5. Plug every cable into its new home, working from the labels.

That was one of many things to fix in what turned into a 16-hour day.

Why the labels mattered

Without them, you are guessing which server or uplink was in which port, at night, with nobody to ask. With them, the move is mechanical and you can check every port against its old configuration.

show interfaces status
show running-config interface GigabitEthernet1/0/1
show module

What I do now

I design around "what can I do to prevent this" rather than "if only I had done this".

• Keep a current copy of the configuration and the interface status off the box, so you know what every port was doing even if the switch will not boot.
• Label both ends of every cable when it goes in, not when it comes out, and label patching in groups so a whole line card can be moved as a block.
• Restore service on what works first. The proper fix can wait for daylight and a change window.
• Spread important uplinks and server pairs across line cards, so one failed card cannot take both legs.

Related: Half-splitting a fault · Surviving the major incident call

The circuit that was fine

A simple internet circuit running slowly, a provider adamant their kit was fine, and an afternoon with the manual for a box I did not own.

What happened

The complaint was internet speed on an ordinary provider circuit. I went through the whole network, point to point, every device the traffic crossed. Each one checked out. The only thing left was the provider's own equipment.

The provider was adamant it was fine. They had tested it from the beginning, more than once.

What fixed it

I read the manual for their device one afternoon, then asked them to try one change: put it into its layer 3 mode. It burst into life.

What I took from it

• Prove your own side first, with evidence. Nobody listens to "it is not us" without it. Interface counters, errors, speed and duplex at each hop, and a test from the hand-off point.
• Read the manual for the box you do not own. Provider equipment has modes and settings the first-line engineer may not know about. If you know what to ask for, you can ask for one specific change instead of another round of "it tests fine".
• "Tests fine" means it passed their test. Ask what the test was, from where to where, and whether it matches what your users are doing.

show interfaces GigabitEthernet0/0 | include rate|error|duplex|CRC
ping 8.8.8.8 source GigabitEthernet0/0 repeat 1000 size 1400
traceroute 8.8.8.8 source GigabitEthernet0/0

Related: Half-splitting a fault · Slow or dropping port flow

When the back-out plan fails too

Two firewall nights. In one, the upgrade failed, the revert failed and then the hardware failed. In the other, BGP did not come up because a step was missing from the plan.

The upgrade that kept failing

A software upgrade on a Check Point management server failed. The built-in revert, the thing the back-out plan relied on, failed as well. Then the hardware failed.

What saved it was a backup. We restored it onto new hardware and carried on.

A revert feature lives on the box you are trying to rescue. If the box is the problem, the revert goes down with it. The only back-out that survives a hardware failure is a backup kept somewhere else, and a restore you know works because you have done one.

The step nobody wrote down

On a different firewall install, BGP failed to come up. The cause was a step missing from the documentation: the people who wrote it knew to do it, so it was never written down. The plan was rewritten afterwards so the next person would not hit the same wall.

What I do now

• Take a full backup immediately before the change, copy it off the box, and check it is there and not empty.
• Treat the back-out plan as a set of steps to test, not a sentence. If it says "restore from backup", know how long a restore takes and onto what.
• Write the plan so someone who has never seen the network could follow it. Anything you do from memory belongs in it.
• After every change that did not go to plan, update the document the same week. That is the moment you know exactly what was missing.

Related: How I run an implementation · The revert timer needs an archive · Change Window Kit

Proving it is not the network: jumbo frames end to end

When the server team says it is the network, the answer is a path trace, every hop, with the MTU at each one. In a converged data centre that path is longer than it looks.

Where this comes from

I supported a converged data centre of UCS compute, Nexus switching, storage and VMware. The servers belonged to another team. When something went wrong with a virtual machine, the network was blamed first, and the only way to answer was to trace every virtual interface the traffic used and show where it was or was not broken.

One of the faults this turned up was jumbo frames on the VMware kernel interfaces, the ones that carry storage and vMotion traffic. Jumbo frames only work when every hop agrees, so the whole path has to be traced.

The path, and where the MTU lives

A packet only gets through at the smallest MTU on the path. One hop at 1500 and the rest at 9000 gives you the worst kind of fault: small packets work, so ping works, and large transfers stall.

Proving it

Test from the host with the don't-fragment bit set and a payload that only fits a 9000-byte frame. 8972 is 9000 less 20 for the IP header and 8 for ICMP.

vmkping -I vmk1 -d -s 8972 10.20.30.40
! fails, but this works:
vmkping -I vmk1 -d -s 1472 10.20.30.40

Then walk the path and read the MTU at each hop until you find the one that is smaller.

show queuing interface ethernet1/1
show interface ethernet1/1 | include MTU
show policy-map system type network-qos

What I do now

• Draw the full path, virtual hops included, before arguing about whose fault it is.
• Set jumbo frames on the whole path in one change, or not at all.
• Keep the vmkping test in the change plan, so it is proved working on the night.

Related: MTU calculator · The SVI MTU gotcha · Abstractions leak

Surviving the major incident call

Senior managers on the line, a service down, and results demanded now. What I learnt from years of being the network engineer on those calls.

For years I was a regular on global major incident calls with senior management. They are high pressure: the business is losing money and the people on the call want a result now. The technical work is only part of it. The rest is how you talk to the people waiting for you.

What works

• Learn who they are and what they expect. Some want detail. Some want one line and a time. Work it out early and give them that.
• Be honest and upfront. If you do not know yet, say so, and say what you are doing to find out. A guess that turns out wrong costs you the room for the rest of the call.
• Be descriptive. "The link to the second data centre is down and traffic has moved to the backup, which is congested" tells them what they need. "We are looking into it" does not.
• Keep them informed, before they ask. Say when the next update will be and give it on time, even if the update is "no change".
• Keep setting their expectations. What happens next, how long it will take, and what it will and will not fix. Then do it again when anything changes.

Things that help on the night

• One person talks for the network. Everyone else works.
• Keep a timeline as you go: what you saw, what you changed, when. It becomes the incident report.
• Open the vendor case early. If you are stuck, saying the case is already open with logs attached is a better update than another half hour of trying.

Related: Comms Writer for the update emails · Troubleshooting Flows

The VLAN I added to the wrong interface

An incident report from one of my own changes. The fix took a minute. The lesson took longer, and it is why every EtherChannel change I do now starts with a two-line check.

What happened

I was adding a VLAN for a new guest service across a client site. One of the trunks it needed was a two-member port-channel between the core switch and the server switch. I added the VLAN to one of the physical member interfaces instead of to the port-channel.

The switch did exactly what it is designed to do. The member no longer matched the bundle, so it was suspended, and the logs said so plainly:

%EC-5-CANNOT_BUNDLE2: Te1/1/1 is not compatible with Te2/1/1 and will be suspended (vlan mask is different)
%EC-5-CANNOT_BUNDLE2: Te1/1/1 is not compatible with Po100 and will be suspended (vlan mask is different)

The impact

Three application VLANs went down for fifteen minutes. Nothing else was touched, because every other VLAN on that site was trunked over more than one path. Those three were not: an earlier project had put them on that single port-channel as a temporary design, under time pressure, and the move to the new network had not happened yet.

The fix

Take the VLAN off the member, add it to the port-channel interface, and let IOS push it down to both members. The bundle came straight back.

interface TenGigabitEthernet1/1/1
 switchport trunk allowed vlan remove 104
interface Port-channel100
 switchport trunk allowed vlan add 104

What I do now

• Configure the port-channel interface, never the members. The only lines that belong on a member are its description and its channel-group.
• Before any EtherChannel change, compare the bundle with its members in one command, and again afterwards:

show etherchannel summary
show running-config | section interface (TenGigabitEthernet1/1/1|TenGigabitEthernet2/1/1|Port-channel100)$

• Do not let application VLANs depend on one trunk, even "for now". Temporary designs last for years and nobody remembers which links are the only path.
• Write the incident report the same day and say plainly whose mistake it was. Mine said "I take responsibility for the error" in the first paragraph. Clients forgive a mistake that is owned and explained. They do not forgive one they have to dig out.

Try it yourself: the second fault in the EtherChannel lab is this exact mistake. Related: An EtherChannel will not bundle · Gotcha: a VLAN on one member suspends the bundle

How I run a network implementation, start to handover

The process I have used on implementations for years, from the first design document to handing the network to the people who will support it. None of it is clever. All of it is there because something went wrong once without it.

1. High-level design

A short summary of the solution and the technologies in it, written so a manager can say yes or no to the approach.

2. Low-level design

The detail an engineer can build from: hardware and software with summary configuration, VLAN numbers, IP addressing, firewall rules, VRFs, diagrams and the kit list.

3. Plan the build on paper first

• List every switch port and patch panel port you will use.
• Draw the cabinets and connectivity, fibre and copper runs, patch panel use and switch port use.
• Read the existing configuration and work out exactly what has to change on it.
• Write the configuration scripts, and have another engineer review them.
• Dry-run the scripts. Problems found here cost minutes, not a night.

4. The implementation plan

Submit it with everything above for approval. Mine always has the same sections: objective, success criteria, when and where, who is involved and how to reach them, pre-implementation tasks, the implementation steps, a back-out plan with how long it takes, post-implementation tasks, testing and a risk table.

• Hold a go or no-go call before the window. If anything is missing, it is a no-go.
• Tell the NOC, and any provider involved, before you touch anything.

5. Build it, and know when to stop

Small differences from the plan get written down as you go. A significant difference is a different rule entirely:

1. Stop all work.
2. Fall back to where you started.
3. Amend the design, the plan and the scripts.
4. Resubmit the plan.

Redesigning in the window, tired, with the change board asleep, is how a two-hour change becomes a two-day outage.

6. Update the documents

Every small change made during the build goes back into the design, the plan and the configuration documents, and into the CMDB.

7. Acceptance into service, then handover

Pull the designs, plans, diagrams and configuration into one acceptance-into-service document, then hand the solution to the live support team, with a walk-through if it is new to them. Finish with a short post-implementation review, even when it went perfectly.

The Change Window Kit MOP Builder writes steps 4 and 5 for ten common changes, go or no-go call and review included.

ACL, prefix list, route-map, distribute-list: who does what

Four names that get used as if they were the same thing. They are not. Two of them match, one matches and changes, and one is just a place to plug the others in.

The short version

• Access lists, prefix lists, AS-path lists and community lists are matchers. Give them a packet or a route and they answer one of three things: matched a permit, matched a deny, or matched nothing. Nothing else.
• A route-map is policy. It calls the matchers, and for whatever matches it can permit, deny or change something: a metric, a next hop, local preference, a community, the path a packet takes.
• Distribute-list, access-group, filter-list and the route-map keyword on a neighbor are attachment points. They are where you apply a matcher or a policy. A distribute-list is to routing updates what ip access-group is to packets.
• Redistribution is the odd one out. On IOS and IOS XE it only takes a route-map. If you want an ACL or prefix list there, wrap it in a route-map.

Rules that save you later

1. Everything ends in an implicit deny. ACLs, prefix lists and route-maps alike. A route-map made only of deny entries denies everything, including what you meant to keep. End it with an empty permit.
2. Use prefix lists for routes, not ACLs. A standard ACL can only see the network address, so 10.0.0.0/8 and 10.0.0.0/24 look the same to it. A prefix list matches the length too, and le and ge give you ranges of lengths.
3. An OSPF distribute-list in only filters your own routing table. The LSAs are still in the database and still flooded to your neighbours, so they still learn the route. To stop a route spreading inside OSPF, filter or summarise at the ABR with area filter-list or area range.
4. BGP policy changes do not touch routes you have already exchanged. Change a route-map, then soft clear that one neighbour in the direction you changed. Never a hard clear of everything.

Worked example: do not become a transit AS

Two internet providers, one company. Without an outbound filter, your router can advertise ISP A's routes to ISP B, and some of the internet's traffic tries to cross your office link. The AS path of a route you originate yourself is empty, so ^$ matches exactly your own routes.

ip as-path access-list 1 permit ^$
ip prefix-list OUR-PREFIXES seq 5 permit 198.51.100.0/24
!
router bgp 64500
 neighbor 203.0.113.1 remote-as 64496
 address-family ipv4
  network 198.51.100.0 mask 255.255.255.0
  neighbor 203.0.113.1 filter-list 1 out
  neighbor 203.0.113.1 prefix-list OUR-PREFIXES out

The filter-list keeps out anything learned from another AS. The prefix list is the belt and braces: even if something odd ends up with an empty path, only your own block leaves. The same result can be had with a route-map that matches as-path 1, and that is the version to use if you also want to set something, such as prepending.

Check what you are actually sending before and after:

show ip bgp regexp ^$
show ip bgp neighbors 203.0.113.1 advertised-routes
clear ip bgp 203.0.113.1 soft out

Related: BGP AS-Path Regex Tester · Gotcha: clear ip bgp * · Gotcha: a network statement that advertises nothing

[100-199] is three characters, not a hundred numbers

A BGP regex that looks right, passes a quick read in a change review, and matches the wrong AS numbers.

Someone wants to match any private AS from 65100 to 65199 in a path and writes this:

ip as-path access-list 10 permit _65[100-199]_

It reads naturally. It is wrong. Square brackets match one character from a set. [100-199] is the characters 1, 0, the range 0 to 1, 9 and 9, which boils down to a single 0, 1 or 9. So the expression matches AS 650, 651 and 659, and none of 65100 to 65199.

Write the digits position by position instead:

ip as-path access-list 10 permit _651[0-9][0-9]_

For ranges that do not line up so neatly, such as 64512 to 65534, the expression gets long. That is what the BGP regex tester and the number-range match in the CLI Filter Builder are for: they write the alternatives for you, and the tester now warns when it sees a multi-digit range in brackets.

Always prove an expression against the real table before it goes into a policy:

show ip bgp regexp _651[0-9][0-9]_

Related: Gotcha: [^255] does not mean "not 255"

Two WAN providers, one VRF: moving sites across a prefix at a time

The design I use when a customer changes MPLS provider and the old and new networks have to run side by side for months. Each site moves one subnet at a time, and rolling back is one line.

The situation is common. The incumbent MPLS is handed off on one VLAN, the new provider's routers on another, and the new contract adds local internet breakout at each site, where before everything went to a central default route. The sites cannot all move on one night.

The design

• One WAN VRF per site on the core switch. Both providers live in it, each on its own SVI with its own OSPF process, so neither provider ever learns the other's routes.
• Nothing is advertised by accident. Internal subnets are static routes in the VRF towards the firewall, and each provider gets only the statics its prefix list allows.
• Inbound, a distribute-list on each process accepts only the remote site ranges you expect from that provider.
• A standard addressing block per site for the handoffs: a /24 split into /29s, with the switch SVI on .1 and the provider's two routers on .2 and .3.
• The internet VLAN is layer 2 only on the switch, from the provider router straight to the firewall's outside interface. The switch never routes it.

vrf definition SITE_WAN
 rd 10.100.11.1:10
 address-family ipv4
 exit-address-family
!
interface Vlan999
 description Old provider MPLS
 vrf forwarding SITE_WAN
 ip address 10.100.11.20 255.255.255.248
interface Vlan1000
 description New provider MPLS
 vrf forwarding SITE_WAN
 ip address 10.100.11.1 255.255.255.248
interface Vlan1002
 description Firewall inside this VRF
 vrf forwarding SITE_WAN
 ip address 10.100.11.25 255.255.255.248
!
ip route vrf SITE_WAN 10.65.90.0 255.255.255.0 10.100.11.28 name VOICE
ip route vrf SITE_WAN 10.66.71.0 255.255.255.0 10.100.11.28 name PRINTERS
!
ip prefix-list TO-OLD seq 10 permit 10.66.71.0/24
ip prefix-list TO-NEW seq 10 permit 10.65.90.0/24
route-map TO-OLD permit 10
 match ip address prefix-list TO-OLD
route-map TO-NEW permit 10
 match ip address prefix-list TO-NEW
!
router ospf 100 vrf SITE_WAN
 router-id 10.100.11.20
 passive-interface default
 no passive-interface Vlan999
 network 10.100.11.16 0.0.0.7 area 0
 redistribute static subnets route-map TO-OLD
router ospf 900 vrf SITE_WAN
 router-id 10.100.11.1
 passive-interface default
 no passive-interface Vlan1000
 network 10.100.11.0 0.0.0.7 area 0
 redistribute static subnets route-map TO-NEW

Migrating, and rolling back

Moving a subnet is moving one line from TO-OLD to TO-NEW. Rolling it back is moving it again. The change board sees a one-line change per subnet, and the provider sees a clean, expected advertisement.

Three things that catch people

• Two OSPF processes, one prefix. If both providers advertise the same remote subnet, both routes have a distance of 110 and their metrics are never compared, so whichever arrived first stays installed. Set distance on one process to make the choice deliberate during the overlap.
• The passive exception must name the real interface. With passive-interface default, the no passive-interface line has to match the interface OSPF actually runs on: the SVI, not the physical port behind it. Get it wrong and the adjacency never forms, with no error.
• mtu-ignore hides a problem. Provider handoffs often need ip ospf mtu-ignore to get past ExStart. It works, but large packets can still be dropped across the link, so ask the provider for their MTU and match it where you can.

Related: ACLs, route-maps and distribute-lists · Gotcha: two OSPF processes, same prefix · Proven design: firewall transit per VRF

IT and OT segmentation in plain English: zones, conduits and the two-firewall rule

How I lay out a power or industrial site network to follow IEC 62443 without drowning in the standard. Three zones, a firewall for each, and a register that says what crosses where.

Industrial networks are easy to attack for reasons that are not the engineers' fault: hard-coded passwords on controllers, software that cannot be patched without recertifying the plant, and historically flat /16 networks where every PLC can see every workstation. The standard's answer is segmentation into zones, joined only through controlled conduits.

The layout

• Three zones: CORP for internal IT, DMZ for anything higher risk or shared with outsiders, and OT for the control systems: SCADA, HMIs, PLCs and RTUs.
• Each zone has its own firewall cluster, always at least two nodes, with IPS and application control on. Separation is a VLAN at layer 2 and a VRF at layer 3, so a zone keeps its own routing table even on shared switches.
• The two-firewall rule: nothing goes from CORP to OT directly. Traffic passes through the DMZ and through the OT firewall, so a compromise in IT has two policies to beat, not one.
• Walled gardens inside OT. The flat OT network is then split further, VRF and VLAN per area, firewalled from each other, so a breach stays where it started.

The access policy, in one table

The zone register

The part most sites skip, and the part an assessor asks for first. For every zone and every conduit, write down:

• Its name, and who is accountable for it.
• Its logical boundary (subnets, VLANs, VRF) and its physical boundary (which cabinets and devices).
• Whether it is safety-related.
• Every logical and physical access point, and the data flows through each one.
• The zones and conduits it connects to.
• Its assets, with their criticality.
• Its target security level, the requirements and policies that apply, and the assumptions it relies on.

Keep it next to the firewall rule base. When a rule request arrives, the register tells you whether the conduit exists at all.

Related: Proven design: firewall transit per VRF · Two WAN providers, one VRF

Half-split the fault, and keep half-splitting at 2am

The fastest way to find a fault on a long path, and the method people abandon exactly when they need it most.

A user in one building cannot reach a server in the data centre. Between them are an access switch, a distribution pair, the core, a firewall and the data centre switches. You can start at one end and walk the whole path, or you can cut the problem in half.

The method

1. Define the path. Write down every hop the traffic should take, in order, both ways. If you cannot write it down, that is the first thing to fix.
2. Pick the middle. Not the box you suspect. The middle.
3. Measure there. Is the traffic arriving? Is it leaving? With the right source address and in the right VRF?
4. Compare with what good looks like. A measurement only means something if you know what it should be.
5. Move to the half that is wrong, and split it again.

Six hops take three measurements, not six. Twelve hops take four.

What it needs from you

• You know the path well enough to name the middle of it.
• You know what normal looks like at each point: the routes, the ARP entries, the counters.
• You know how to measure at each point without disturbing anything.

None of these is trivial on a large network, which is why diagrams, baselines and a saved pre-change capture matter so much. They are what let you half-split instead of guess.

Measuring in the middle

Test from the middle with the user's subnet as the source, so the reply has to come back the way the user's would:

ping 10.50.0.10 source Vlan10
traceroute 10.50.0.10 source Vlan10

When you need to see the packets themselves, an on-box capture on a Catalyst 9000 avoids walking to the rack with a laptop:

monitor capture CAP interface GigabitEthernet1/0/1 both
monitor capture CAP match ipv4 host 10.10.0.25 any
monitor capture CAP start
! reproduce the fault, then
monitor capture CAP stop
show monitor capture CAP buffer brief

The 2am trap

Hours into a fault, tired, it is tempting to jump outside the split: "it's probably the firewall", and off you go to a box the last measurement had already cleared. You measure something interesting, it seems to explain everything, and it is the wrong thing. Stay in the half your evidence points to. If the measurement does not match what you expected, trust the measurement and question your expectation, not the other way round.

Related: Troubleshooting Flows · Change Checker

Every abstraction leaks. Know where

Summaries, tunnels, VLANs stretched between sites and NAT all hide something. The fault turns up where you stopped looking.

Most of network design is hiding detail on purpose. A summary hides the routes inside it. A tunnel hides the path underneath. NAT hides the real address. That is what makes big networks manageable. The cost is that every one of them leaks, and the leak always appears somewhere you were not looking.

A habit that catches it

• For each abstraction in a design, write one line on what it hides and what that will cost when it goes wrong.
• Test the failure of the hidden thing, not just the abstraction. Fail a component route behind the summary. Fail the underlay under the tunnel.
• Prefer the abstraction that leaks least for the problem at hand. Aim for the design with the fewest moving parts that still does the whole job.

Related gotchas: summary discard route · tunnel MTU · recursive routing · port-channel hashing · NAT return path

Taking the old one out is part of the project

The new system is live and everyone is pleased. Then the old one stays, half-connected, for years.

Projects celebrate the go-live. Very few budget for the removal. So the old firewall keeps a cable plugged in "just in case", the old routes stay in the table, the old VLAN still trunks to every closet, and the next engineer has to work out which of two of everything is real.

Every leftover is something to patch, monitor, license and understand. And when it fails, it fails in a way nobody remembers designing for.

Put the removal in the plan

• List what the new thing replaces while you design it: devices, circuits, VLANs, routes, firewall rules, DNS entries, monitoring, support contracts.
• Give the removal its own change, a few weeks after go-live, with the same rollback discipline as any other.
• Make removal part of acceptance. The project is not closed until the old configuration is gone and the documents show the network as it now is.
• Cancel what costs money. Old circuits and support contracts often run for years after the kit is switched off.

Find what is left behind

show ip route static
show vlan brief | include active
show interfaces description | include OLD|TEMP|SPARE
show ip interface brief | include down

Every one of those lines either has an owner and a reason, or it is a candidate for the next removal change.

Related: How I run an implementation, start to handover

The one vPC rule behind most vPC outages

vPC makes two Nexus switches look like one, but only at layer 2. Learn the one rule that protects it, and most of the strange vPC faults stop being strange.

vPC lets a server or access switch build one port-channel with a link to each of two Nexus switches. Every link forwards, nothing is blocked by spanning tree, and either switch can fail. At layer 2 the pair behaves as one switch. At layer 3 it is still two routers, each with its own address, its own routing table and its own MAC.

The rule

Traffic that arrives over the peer-link is not allowed to leave by a vPC member port. The device on the far side of that vPC has a link to both switches, so the switch it came from should have delivered it directly. Sending it out again could loop it or duplicate it, so the hardware drops it. The only exception is when that device's links to the other switch are all down, so this switch is now the only way to reach it.

In normal running, almost nothing crosses the peer-link, because each switch delivers to its own vPC members. The rule only bites when something forces traffic over it.

What forces traffic over the peer-link

The other half: two brains that must agree

Each switch has its own control plane and its own configuration. They compare notes over the peer-link and decide which of them stands down when they cannot. That is why:

• The keepalive has to take a different path from the peer-link. It is how each switch tells a dead partner from a dead link.
• Some settings have to match exactly. Spanning-tree mode, MTU, port-channel mode and a few more are checked, and a mismatch suspends ports on the secondary.
• Every vPC VLAN must also be on the peer-link, or it does not forward on the vPC.
• The roles do not swap back. After a reload the switch you configured as primary may be secondary, and it is the secondary that suspends its ports when the peer-link fails.

Before and after every change on a vPC pair

show vpc brief
show vpc role
show vpc consistency-parameters global
show vpc orphan-ports
show port-channel summary

Change one peer, check the consistency output, then change the other straight away. Never leave the pair half done over a break.

Related: vPC pair · routed uplinks and firewalls · vPC troubleshooting flow · vPC lab

Build the firewall rules from what the logs tell you

Nobody can list every flow an old application needs. So let the traffic tell you, then lock it down.

There are two ways to write a firewall policy. Allow only what is needed and deny the rest, or deny the known dangerous things and allow the rest. The first is far safer, easier to audit and the one most compliance standards expect. It is also the one that breaks things on day one if you guess the flows wrong, which is why so many firewalls end up with "permit ip any any" quietly sitting at the bottom.

The way through is to learn the flows before you enforce them.

The approach

1. Put the servers behind the firewall first, with a policy that permits everything and logs every match. Nothing breaks, and now you can see.
2. Collect for a full business cycle. A week at least, and include a month end if the application does month-end work. Backups, batch jobs and quarterly reports all show up eventually.
3. Turn the logs into rules. Group by source, destination and port. Each group with a business owner becomes a specific permit. Ask about anything nobody owns.
4. Add the specific permits above the catch-all, then watch the catch-all's hit count fall towards zero.
5. Swap the catch-all for an explicit deny with logging, in a change window, with the owners on hand.

On an ASA

access-list INSIDE-IN extended permit ip any any log
show access-list INSIDE-IN
show conn address 10.30.20.10
packet-tracer input inside tcp 10.30.20.10 51000 10.30.10.20 443

An ACL entry with the log keyword produces syslog message 106100 for each new flow, with the source, destination and port. Send them to your syslog server, not the local buffer, or a week of data will not fit.

Two design points that help

• Protected subnets should have no other way in. If the core switch also has an interface in the protected VLAN, traffic can route round the firewall. Give the firewall the only gateway on that VLAN.
• Routed mode is safer than transparent mode for this. A transparent firewall is easier to add, but a server plugged into the wrong VLAN will still work and sit outside the policy without anyone noticing.

Related: vPC core with a firewall pair · Firewall transit /29 per VRF

Redistribution without the surprises

Passing routes from one protocol to another is easy to configure and easy to get subtly wrong. These are the rules that keep it boring.

Redistribution turns up in mergers, migrations and anywhere a provider, a firewall or an old WAN speaks a different protocol from the rest of the network. Configuring it takes a line. The trouble comes later: routes that never arrive, routes that take the long way round, and routes that flip between two paths every few seconds.

The rules

1. Redistribute at one point if you can. One router, one direction of policy, nothing to loop. If you need two points for resilience, the rest of these rules are not optional.
2. Always set the seed metric yourself. EIGRP will not advertise OSPF or BGP routes without one, and OSPF's default external metric tells the rest of the network nothing.
3. Filter with a route-map, every time. Redistribute what you mean, by prefix-list, and nothing else. NX-OS will not let you do it any other way.
4. Tag on the way out, deny the tag on the way back. A route that came from EIGRP should never be redistributed back into EIGRP by the other router.
5. Check the administrative distances at each redistribution router. An OSPF external route (110) beats an EIGRP external route (170), so a router can prefer the other protocol's copy of a route that started on its own side.
6. Summarise at the boundary where you can. A summary is easier to filter, and it hides flaps on the far side.
7. Test the failure of each redistribution router, and the recovery, before you sign it off.

The distances that decide it

A safe two-point pattern

route-map EIGRP-TO-OSPF deny 10
 match tag 110
route-map EIGRP-TO-OSPF permit 20
 match ip address prefix-list EIGRP-NETS
 set tag 90
route-map OSPF-TO-EIGRP deny 10
 match tag 90
route-map OSPF-TO-EIGRP permit 20
 match ip address prefix-list OSPF-NETS
 set tag 110
!
router ospf 1
 redistribute eigrp 100 subnets route-map EIGRP-TO-OSPF
 distance ospf external 171
router eigrp 100
 redistribute ospf 1 metric 100000 10 255 1 1500 route-map OSPF-TO-EIGRP

Configure the same on both redistribution routers. The distance only changes how those two routers choose; everyone else is unaffected.

Proving it

show ip route 10.40.40.0
show ip ospf database external 10.40.40.0
show ip eigrp topology 10.1.1.0/24
show route-map

On each redistribution router, every route should be learnt from the protocol it started in. The route-map counters tell you whether the deny entries are catching anything.

Related: Redistribution lab · Routes missing flow · The mutual redistribution gotcha

Read a route-map like a sentence

Route-maps look like ACLs and behave differently in ways that catch people out: OR where you expected AND, and an ending that means three different things.

A route-map is a numbered list of entries. Each entry says if these match, then permit or deny, and set these things. The router reads the entries in sequence order and stops at the first one that matches.

Inside one entry

• No match line means the entry matches everything. That is how you write "everything else".
• Different kinds of match on separate lines, such as a prefix-list and a tag, are combined with AND.
• Several lists on one match line are combined with OR. If you type two matches of the same kind on separate lines, IOS merges them into one line, so they become OR as well.
• Set lines only run when the entry permits.

route-map EXAMPLE permit 10
 match ip address prefix-list A
 match ip address prefix-list B
 match tag 90
! read as: (in A or in B) and tagged 90
show route-map EXAMPLE

What happens at the end

Every route-map ends with an invisible deny for anything that matched no entry. What deny means depends on where the route-map is used:

So a route-map for redistribution or BGP that changes some routes and should leave the rest alone needs a final permit entry with no match. Leave it off and you filter everything you did not mention.

route-map FROM-ISP permit 10
 match ip address prefix-list PREFER
 set local-preference 200
route-map FROM-ISP permit 20

Habits that help

• Number entries in tens, so there is room to insert one later.
• Name prefix-lists after what they hold, not after the route-map.
• Read show route-map after every change. It shows what IOS stored and how many matches each entry has had.
• After changing a BGP route-map, soft-clear the neighbour, or nothing changes for routes already received.

Related: ACLs, route-maps and distribute-lists · The AND that becomes OR · The ending that changes meaning