Coping Mechanisms For A Lying ARP Cache

I Swear To Tell The Truth

Prevarication. Untruths. Misdirection. Deceit. Call it what you will, lying is ugly. While we expect lying from people (there’s an American TV show built around this premise), we expect our network gear to tell only the truth. The command line is supposed to a bastion of honesty. And indeed, the command line will tell you exactly what its perception of reality is. The magic (and our job as engineers) is determining when the device in question is woefully misguided…when what it thinks it knows is no longer true.

Caches can be guilty of storing bad data. When they first learned their data, they had learned truth. But as a cache’s data ages, the possibility increases that the cached data becomes stale: out of sync with reality. When cache gives you stale data, it’s lying: a stiff penalty we sometimes pay for performance.

In Your Stack, Improving Your Performance

In networking, practically all network stacks have an address resolution protocol (ARP) cache as a standard feature. What is ARP used for? Simply stated, ARP discovers the link-layer (think Ethernet MAC) address when only the network layer (think IP address) is known. Keep in mind that ARP isn’t specific to IP or to Ethernet; various flavors of ARP are used with other network protocols and transports. For this discussion, we’ll assume Ethernet and IPv4, though.

Well okay, I’m with you so far…but WHY do we need to know what the link-layer address is? I mean, who cares? A host can know the IP address of another host he wishes to communicate with, but that’s not enough information to send a packet across the network wire. In addition to the remote IP address, the host must be able to build a link-layer frame of a type appropriate for the network media to which the host is connected. In the vast majority of today’s LANs, that media is Ethernet. So inside of an *Ethernet* frame will go the IP packet. ARP is used to discover the Ethernet MAC address of a remote IP address necessary for the destination MAC component of the Ethernet frame. Still with me? Good.

Next step then…what is ARP *cache* used for? Like any cache, an ARP cache stores MAC-IP mappings, so that the next time the host needs to send an IP packet to that destination IP address, the Ethernet destination MAC is already known. An network ARP request is not required before the creation of the frame wrapper if the requisite destination MAC can be pulled out of ARP cache. Naturally, this improves network performance as a whole.

• ARP requests are broadcasts. Broadcasts, by definition, are sent to all hosts on a network segment. The higher the volume of broadcasts on a segment, the lower the overall segment performance (think aggregate throughput). ARP cache helps reduce the number of broadcasts.
• ARP queries take longer to send and get a response than doing a lookup against a local cache. Again, a performance boost.

Show Me Your ARP Cache

Would you like to see the ARP cache on your network-attached device?

• On a Windows host, type “arp -a“.
• On a *NIX host, type “arp -an“.
• On a Cisco router or layer 3 switch, type “show ip arp” or “show ip arp vlan X” where X is your VLAN number of choice.

I’ll spare you the command output, because they all look similar: a list of IPs with associated MAC addresses that are currently stored in the device’s ARP cache.

Why Did The New Core Switch Break The Network?

Very early this past Saturday morning, I was remotely guiding a core switch upgrade for a small data center several hours distant from me. I’d built the switch and shipped it to the site, where one of the local engineers had racked and powered it. The maintenance window was to move the cables over from the scary 10 year old switch that made me nervous every time I did a “wr mem” to the shiny new one. The core switch being replaced acted as the router for all the VLANs at this site. This was not a “dual core” design, where the core switch was one of a pair of switches acting as backups of one another. Rather, this core switch was *it*; without this switch, intervlan traffic was impossible, as no other device on the network was providing routing services.

Being remote and having no backdoor, I was relying on the local engineer to tell me what was going on. Moving the cables was the easy bit. All the fiber uplinking remote access switches was moved over without incident; all the lights were green. The copper to move over the Internet border firewalls was similarly successful. Shortly thereafter, the IPSEC site-to-site tunnel I needed to access the site reconnected, and I was able to log into the shiny new core switch. After resolving some anticipated speed/duplex and trunk issues, we still found that a number of hosts were unable to talk outside of their local VLAN. Hmm…

Now, this was not my first rodeo. I’ve replaced a few routers and switches in my day. Stale ARP caches telling lies and lies and lies is an annoying issue that can really get your knickers in a twist if you don’t know what’s going on. However, I was expecting (and dreading) this very circumstance.

Let’s say you’ve got host 10.100.100.10 in Vlan 100 needing to talk to some host outside of Vlan 100. Now, before the core switch replacement, this worked fine. 10.100.100.10 knew that to talk to someone outside Vlan 100, he’d need to forward that intervlan traffic to his default gateway, let’s say 10.100.100.254, where it would be routed. 10.100.100.10 also had an ARP cache entry that told him what the MAC of 10.100.100.254 was, because sometime earlier, he’d put an ARP request on the wire, asking for the MAC of 10.100.100.254, to which the core switch had responded. With that information readily in his ARP cache, 10.100.100.10 would merely need to peek inside his cache, pull out the appropriate MAC, and then wrap his outbound IP packet in an Ethernet frame bound for the MAC of 10.100.100.254 to obtain routing services. Life was grand whenever he needed to communicate with far away subnets.

That is, right up until I replaced the ancient core switch with a new one. “Now I done it,” so to speak. While the default gateway IP was certainly the same — 10.100.100.254 — the MAC associated with that IP address was changed. Remember that Ethernet MAC addresses are assigned by manufacturers and are supposed to be unique to every Ethernet interface the world over. As a result, every host on Vlan 100 had a lying ARP cache. Hosts including my imaginary 10.100.100.10 would frame their traffic heading for the default gateway with a stale destination MAC stored in local ARP cache back when the old core switch was in production. They would then put that ill-formed frame on the wire where it would go…nowhere. Well technically, that frame went *everywhere*: the switches in the VLAN would try to deliver the frame, unicast-flooding it out every port, where the frame would be ignored by everyone. No one in the VLAN had that MAC address anymore.

The core switch was new and there was nothing wrong with it, but the network was broken.

Beat Your ARP Caches Into Submission

So how do you, the unicorn-taming, leprechaun-catching network engineer, resolve this issue? With broken ARP caches all over the network, you look a bit of an idiot. “Hey, network guy. Nice new switch that doesn’t work. You gonna fix it? Should we cut back to the old switch? I think I’m gonna call my boss because none of the servers are working. I need a hug!” You’d like to hug him with a good swift smack to the side of his Windows-loving head with the dirty side of your keyboard (you know, all that cheezle and dead skin that falls in between the keys), but you’re better than that. No, you’re going to provide a *solution*.

• Wait. As with any cache, the entries in an ARP cache have a time-to-live. Eventually, the TTL will expire, and hosts will start working again as they put fresh ARP requests on the wire and discover the new MAC. How long? That depends on the host OS, but generally speaking a period of a few hours. I’m too lazy to check at the moment, but 300 minutes comes to mind as pretty standard time.
• Clear ARP caches on critical devices by hand instead of waiting for TTL to expire. Most devices have a way to purge the ARP cache at the command line, forcing the device to put an ARP request on the network wire. This will force the ARP cache to be repopulated.
• The “swatting a fly with a Buick” approach is to reboot the broken system. A reboot is never a graceful answer, but depending on your circumstances and who you’re working with, it’s a solution easy to understand for all involved, albeit not always expedient.
• Use a MAC address that doesn’t change from one switch to the other. I’m referring to a first-hop redundancy protocol, like HSRP. HSRP computes a MAC address that floats between members in a HSRP group; that computed MAC is always the same, varying only by VLAN number. Therefore, if you replace an HSRP switch with another HSRP switch, your hosts using the new switch as a default gateway won’t know the difference. There’s nothing stopping you from running HSRP on your core switch, even if you’ve only got a single core.
• Note that some devices use gratuitous ARPs to notify remote hosts about their new MAC addresses. This is commonly found in hosts using high-availability protocols where there is no floating MAC address, but merely ownership of a MAC shifting from one host to another. For example, F5 Networks load-balancers rely on a gratuitous ARP during a high-availability active/standby failover event to notify hosts of the change in roles from one load-balancer to the other.

So let’s hear from the rest of you clever people. How to you beat the stale ARP cache problem?