Tony Bourke really stirred up a hornets’ nest with this one! Who would have thought that “storage NAT” would be so controversial?

NPIV and NPV are among the two most ill-named of acronyms I’ve come across in IT, especially since they sound very similar, yet do two fairly different things. NPIV is an industry-wide term and is short for N_Port ID Virtualization, and NPV is a Cisco-specific term, and is short for N_Port Virtualization. Huh? Yeah, not only do they sound similar, but the names give very little indication as to what they do.

He followed up with a second post on analogies for NPV/NPIV…

When comparing Fibre Channel to Ethernet/IP, it’s important to remember that they are different. In fact, significantly different. The only purpose for relating Fibre Channel to Ethernet/IP is for the purpose of relating those who are familiar with Ethernet/IP to the world of Fibre Channel.

© Stephen Foskett for Gestalt IT, 2012. | NPV and NPIV
Read more posts categorized as Networking, Storage

Cisco’s UCS B-Series is an exemplary modern blade system

Blade servers are easily recognized in the data centers, trade shows, and product catalogs of today: They’re the ones that nestle together in an enclosure, sharing some resources rather than standing on their own in a rack or on the floor. But what is the essential element that separates a blade from any other kind of server?

Let’s consider some defining characteristics of the blade server species:

1. Physically, a rack-mounted blade enclosure contains a number of server blades. Each blade is smaller than a 19″ rack, allowing more to be packed into the same space. This contrasts with larger, self-enclosed tower or rack-mount servers.
2. Server blades rely on the blade enclosure for critical supporting functions like power, cooling, and I/O ports, so they cannot stand alone. This further improves density and efficiency. Stand-alone servers, on the other hand, include these functions in their case.
3. Blade systems include some sort of management device that monitors the blades and can control some of their functions. Conventional servers do not always have this kind of consolidated management.
4. The blade chassis includes consolidated and shared I/O channels, ranging from keyboard, video, and mouse (KVM) to networking and storage (usually Ethernet and Fibre Channel). These add flexibility, since external ports can be shared by multiple blades and reconfigured without disruption.
5. Blade systems are optimized for high availability, with hot-swap components everywhere from power to fans to the blades themselves. Since these are shared, it is more efficient to purchase redundant parts for a blade system than for each server individually.

HP’s blade systems (like this loaded c3000) make up half the market today

To me, these five elements are key to a modern blade system. Without them, a blade solution cannot meet the expectations of buyers (or the promises of vendors!) And what are these benefits? I took a look at the marketing materials for the leading companies in the space (HP, Dell, IBM, and Cisco), and this is what they promise:

1. Efficiency – More processing power in a smaller footprint (physical size, power consumption, cooling, and weight)
2. Manageability – Simpler and cheaper systems management
3. Reliability – High availability thanks to redundant components
4. Performance – Improved I/O performance thanks to shared network and storage features
5. Flexibility – Simplified cabling that can be reconfigured in software

© Stephen Foskett for Gestalt IT, 2012. | What Is a Blade Server?
Read more posts categorized as Cloud Computing, Featured, Server Virtualization

It's not going to be this easy to bridge Fibre Channel and Ethernet!

Before the holidays, I posed a question on Google+ that generated quite a bit of interest and feedback. Now that it has settled down a bit I’d like to summarize the unresolved elements to make FCoE truly a world-class storage interconnect.

Setting the Stage

FCoE has been a controversial topic in both storage and networking, and for good reason. No one would deny that Ethernet is not an ideal transport mechanism for block storage I/O. “Porting” Fibre Channel to run on Ethernet networks has been a supreme technical challenge, and many companies and individuals have labored long and hard to make FCoE a reality.

Now that FCoE is specified in the standard and has been deployed in production environments, the question turns to its future. Will it take off and seize the mantle of dominance currently held by what I like retroactively to call “Fibre Channel over Fibre Channel?” Will they coexist for the next decade, with FCoE mainly deployed in “block” environments such as Cisco UCS? Or will FCoE ultimately fail to catch on, displaced by some other storage protocol like plain FC, iSCSI, NFS, or something entirely different?

The data center needs a flexible new protocol to meet the needs of virtual environments, and convergence of storage and data networking makes a great deal of sense in these environments. This was the root of my question, and I ask it in all earnestness.

My question: What elements remain unresolved to make FCoE truly world-class? What should the vendors be prioritizing? Here are the answers I received.

Technical Considerations

Link Aggregation on CNA’s

Converged network adapters (CNA’s) allow multiple protocols to access a single Ethernet connection, but some also include multiple ports that can be aggregated. In traditional Ethernet networks, link aggregation is a respectable approach for performance and availability. But storage networks have traditionally relied on host-based MPIO software, and these features are mutually exclusive. The zeitgeist seems to be a recommendation to avoid link aggregation on CNA’s that are used for storage networks.

How Do You Handle Virtual Machine Mobility?

As I described recently, virtual machine mobility is a major technical challenge for existing networks. The VMware proposal, the VXLAN, seems to be gaining traction right now. But this is only a solution for data networking. How will FCoE SANs handle virtual machine mobility? This remains unresolved as far as I can tell, though Ethernet switch vendors have come up with their own answers. Brocade demonstrated just such a solution at Networking Field Day 2, and I know that others have answers as well. But will there be an interoperable industry solution?

How Should FCoE Be Implemented Over Longer Distances?

Fibre Channel has traditionally relied on routers and other protocols (FCIP and iFCP) to span distances, but FCoE raises the possibility of native traversal. While it is certainly possible to span distances with FCoE, this is definitely not a recommended or supported idea. Without TCP/IP, or any routing mechanism, it’s just a bad idea. But I imagine that it won’t be long before vendors decide to give it a go anyway.

Implementation Considerations

Is TRILL Required for FCoE Networks?

This has been one of my own questions since the very beginning. Clearly, edge only FCoE works just fine without TRILL. But as networks become more complicated, and virtual machines move, it seems an awfully good idea to have some protocol to alleviate East-West routing concerns. I feel much better with TRILL (or some similar Ethernet fabric technology) in a complicated FCoE network.

Should All Switches Be Full FC Forwarders?

There are number of ways to implement FCoE on Ethernet network, and not all involve building a full Fibre Channel stack in each switch. While many (including myself) assumed that FCoE implied Fibre Channel forwarding in all switches, this is clearly not the direction taken by vendors, at least initially. Perhaps the current “Ethernet forwarding” approach is only a stepping stone, or perhaps it will emerge as the dominant FCoE standard.

How Will OpenFCoE and LoM Be Used?

OpenFCoE is a software solution allowing FCoE to be run without a CNA. If this became popular, it wouldn’t be long before data center architects began looking at LAN on Motherboard (LoM) and even 10GBASE-T as a potential SAN alternative. Will this be used in the long run? It could happen, but it’s certainly not something that’s here at the moment. But OpenFCoE is a real player, especially with Intel’s backing.

How Will Technologies like Zoning Interoperate?

Many networkers are just now beginning to see the true complexity of Fibre Channel SANs. Although interoperability of higher-level Fibre Channel functions between vendors has never been a priority in “FC over FC” SANs, Ethernet could change things. I would not be at all surprised to see a groundswell of customer support demanding greater levels of interoperability from FCoE than from FC, and zoning and VSAN is the likely first beachhead.

The Big Question: When Will We See the “Killer App” For FCoE

Just about everyone agreed that the real challenge for FCoE is market acceptance. Customers aren’t yet demanding FCoE, and vendors are finding it hard to articulate a compelling case to move from “tried-and-true” FC. Convergence, cost savings, and performance have all been put forth, but customers aren’t biting. Perhaps they just need a little time and a little more proof.

This post relies extensively on feedback from a number of people, including Ivan Pepelnjak, Tony Bourke, J Metz, Dmitri Kalintsev, Derick Winkworth, David Hardaker, Juan Lage, and Corey Hines.

© sfoskett for Stephen Foskett, Pack Rat, 2012. |
Eight Unresolved Questions About FCoE


This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2012. | Eight Unresolved Questions About FCoE
Read more posts categorized as Storage

It isn’t always easy to get where you need to go!

Consider the following situation: You go to lunch with your good friends, John and Mary. Halfway through a rousing discussion of the latest Hollywood movie, Mary starts talking about the fantastic action sequences while John criticizes the romantic angle. You realize something mine-bending has happened: John now has Mary’s personality, and vice versa. It’s like they have switched brains or something!

This truly weird situation isn’t likely to happen in person, but occurs all the time in the data center. Virtualization of server, network, and storage services illuminates the link between physical resources and functional applications. A running virtual machine can instantly move from one server, network adapter, HBA, or LUN to another. And when it happens, traditional components have no idea how to react.

The Challenges of Mobility

Mobility is perhaps the “killer app” of virtualization, but it is also the killer of traditional IT systems. Let’s consider the challenges of this “Twilight Zone” moment.

• The operating system expects a consistent hardware environment, which is exactly what the hypervisor creates
• The LAN must be prepared to redirect all network traffic instantly and seamlessly to one or more new physical interfaces
• The SAN similarly must be able to reroute all I/O to a new pair of HBA’s without missing a beat
• The storage array must be able to re-present capacity to a new physical device, and must maintain snapshots and other configurations
• The backup system must also be able to maintain consistency over time even as machines relocate to different server and storage locations

All of this must be done while maintaining quality of service (QoS), access control, reporting, and appropriate segmentation at all levels. This is an incredibly challenging task, and no conventional protocol (IP, Ethernet, NFS, SCSI, Fibre Channel, etc.) is anymore ready then you are when you’re good friends switch personalities.

Two Paths

So much of the development that is currently taking place in IT focuses on accommodating this “mobility issue”. Two key approaches have emerged to take on this challenge:

• “In a vacuum” technologies (like VXLAN) assume that no other changes will be made, so the focus is on maintaining complete compatibility in front and behind
• “Clean sheet” technologies (usually from startups) take a different approach, throwing out compatibility in favor of technical elegance

Both of these approaches have merit. Attempting to maintain compatibility only works so far (just ask a Windows API programmer), but it leverages the existing environment and recognizes that most people are not ready for wholesale change. Clean sheet designs always make more sense, but they rarely attain mass acceptance. Nearly every technology we rely on today is full of bolt-ons in the name of compatibility. Some, like Ethernet and x86, actually work pretty well, too.

The Stack of Lies

The difference between virtualization and cloud computing is exactly this same distinction. Hypervisors, NPV, NAT, thin provisioning, and so many other virtualization technologies exist mainly to maintain compatibility in a vacuum. In contrast, true cloud computing dispenses with the entire stack and creates a new platform for applications.

This is, perhaps, the reason that cloud computing is not taken off in the enterprise. Simply put, IT is not prepared to ditch everything they have ever used even in the face of a demonstrably superior alternative. Currently, the highest use of cloud is behind gateways and virtualization engines that bring it back down to earth.

This brings us to the stack of lies called server virtualization. Any “modern” virtualized data center is built on lie after lie, with each level telling the other what it wants to hear. The volume manager lies to the operating system, hypervisor lies to the volume manager, and the storage array lies to the hypervisor. The same sad state of affairs allows networking and even memory to function in a virtual world.

But these shaky stacks of lies have difficulty adapting to motion, since no level truly “knows” the reality of the world around. The depressing truth is that a bowl of spaghetti like VXLAN is perhaps the highest form of art we can expect in a virtual data center.

Stephen’s Stance

As a techie, I am always drawn to clean sheet designs that offer technical elegance along with functionality. But I know that, realistically, products that assume nothing about the world around them and bend over backward to maintain compatibility are more likely to succeed. Still, I maintain hope that the issues of virtual machine mobility will be solved in an elegant way, rather than adding to the “stack of lies”.

© Stephen Foskett for Gestalt IT, 2012. | The Terrifying True Story Of Virtual Machine Mobility
Read more posts categorized as Server Virtualization, Storage

The traditional datacenter is comprised of servers, network, and storage. We have all seen major changes in server architectures (including newer processors, new instruction sets, faster RAM, PCIe, and Blade architecture) and storage architecture (SAN/NAS functionality, SSDs/EFDs, caching improvements, replication, and storage tiering). These improvements have shown major benefits to the users of these systems and have kept capital investments in IT moving along as IT departments have been able to improve stability and performance due to them.

However, there has been a lack in mainstream improvement in network performance as well as a newcomer to the datacenter, virtualization, that stand to make a major change in how datacenters operate in the very near future.

The last major datacenter network change involved the adoption of 1Gbps networking from 10/100. I am sure there are some network guys that would love to debate this to no end (feel free to talk amongst yourselves, then). Server hardware began to include 1Gb NICs onboard and the switch manufacturers dropped the price of the 1Gb switching. At that point, almost anyone with some money and a datacenter could immediately realize the benefit of increasing the network performance 10-fold.

Since then, though, the network has been lacking in performance gains. Technologies and techniques, such as Infiniband, Port Channels, and NIC teaming all assist in boosting the performance in some fashion, but they are not commodity like being able to plug in an existing 1Gb NIC into a 1Gb switchport and everything work faster. However, fairly recently, the introduction of 10Gb networking has emerged. The same pattern will follow as with the adoption of 1Gb networking… NICs onboard, switching costs drop, adoption ensues. Suddenly, 10Gb networking will penetrate the market.

The new player to the datacenter is virtualization. Now, it is common to think of virtualization as server virtualization (aka – VMware vSphere, Microsoft Hyper-V, Citrix XenServer, etc…). However, virtualization is really just a layer of abstraction. In some situations, like server virtualization, this happens by the use of a hypervisor to abstract the hardware from the software layer. However, the same concept of virtualization as abstraction can be applied to other areas of the datacenter.

So, what can be abstracted and virtualized? Well… almost anything that is historically tied to being hardware specific. Categories like I/O Virtualization (Companies: Aprius and Xsigo), Storage Virtualization (Companies: Isilon/EMC, NetApp, Avere), and Network Virtualization (Cisco, Open vSwitch, VMware, and Citrix). Suddenly, what used to be specific to hardware has been lifted into a realm where it is not necessarily the case. Sure, PCIe cards are still tied to hardware for access… however, it is not necessarily on the physical server itself.

Higher bandwidth networking in the datacenter means that more and more data can be put on a network at any given time without impacting other operations. So, I/O functions like Fibre Channel storage and PCIe I/O are suddenly able to exist on the datacenter Ethernet network. Virtualization is becoming the mechanism to allow for the convergence of non-traditional data on the Ethernet network.

The addition of virtualization and high bandwidth networks is leading to a new shift in the functions of the datacenter components. Now, we are seeing more and more intelligence being placed on the service level and less and less functionality in the core datacenter components. Products are coming to market that remove functionality from the datacenter components and perform functions themselves. SAN caches are being placed in the data path that interrupt the flow to the primary SAN in order to increase performance. PCI cards are being removed from hosts and shared across multiple hosts, over Ethernet. Network decisions (VLANs, for example) are now being deployed to virtual servers, like ESXi and Hyper-V… virtual switches are even extending across multiple hosts (vDS, in vSphere environments). Some very powerful networking switches are being virtualized and deployed in virtual server environments (see Cisco Nexus 1000v and Open vSwitch) and away from the formerly closed hardware environments.

The change in the functions of core datacenter components is really a mixed bag. Partially a case of “what a great idea!” and “just because you can does not mean you should”. I feel that the virtualization of server hardware is leading to some pretty cool things… mostly, a time to re-question how a service is provided. Vendors are re-thinking about needing an expansion being a hardware device or a service being provided over the network. This leads to server hardware becoming smaller, more dense, and more cost efficient (lower cooling and power needs with high performance).

The Network virtualization is very much a middle ground for me. Removing network intelligence from network devices is not a good idea. The network devices are designed and built for efficiency and logic as to how to process the data. Removing that logic and placing it elsewhere is a step in the wrong direction. However, being able to extend network logic into areas once unavailable becomes a major benefit to everyone. Projects/products like Open vSwitch are really pushing the adoption of virtual network switches into the forefront of the server virtualization world as it provides major network functionality at the virtual switch level that puts the “devices” at the same level as traditional hardware switches.

Storage virtualization is another middle ground area for me. Abstracting the data on the storage device and allowing the device to logically place the data on tiers is amazing technology. This allows for higher performance on most frequently used data, block and file level access to data on the same storage groups, file access via metadata, and so much more. The issue becomes with the ever-so-precious data path. Storage admins are very hesitant to place a device in the middle of the storage path. Suddenly, there is another variable that can impact the quality and consistency of the Corporate data. So, while a product like Avere is so cool, it is inline and people are weary of that.

All of the virtualized components listed above require some level of Ethernet bandwidth for them to work properly. By increasing the bandwidth available on the network, these new technologies are making their way into the datacenters. We just need to ask ourselves “just because we can, does that mean we should?” 

© Bill for Gestalt IT, 2010. | Virtualization and High Bandwidth Datacenter–How the Datacenter Landscape Is Changing
Read more posts categorized as All, Server Virtualization

Some of you know I took on a new job earlier this year, where the challenge was (and is) to transform a globally distributed network for a growing company into an enterprise class operation. A major focus area has been eliminating single points of failure (SPOFs): single links, single routers, single firewalls, etc. If it can break and consequently interrupt traffic flow, part of my job is to design around the SPOF within the constraints of a finite budget.

The network documentation I inherited ranged from “mostly right but vague and outdated” to “a complete and utter fantasy requiring mind-altering substances to make sense of”. Ergo, untrustworthy to the point of being useless beyond perhaps slideware to show a particularly dim collection of simians. I have therefore been doing complete network explorations, building new documentation as I go.

To my horror, I one day discovered an egregious SPOF, where a single, fragile piece of CAT5 provided the sole physical path between two major concentrations of network activity. If that link ran into any trouble, an entire room containing hundreds of physical and virtual servers (and their storage) would have been cut off from the rest of the company.

To eliminate the physical path SPOF, the easy choice was to transform the single link into an etherchannel. This I did; the single 1Gbps became a 4x1Gbps etherchannel plumbed back to one core switch. For good measure, I added a second 4x1Gbps, plumbed to a second core switch. Spanning-tree roots had already been established such that even-numbered VLANs would traverse one of the 4x1Gbps etherchannels, and odds the other…which you can read more about here if interested.

All should now be sweetness and light, right? A 1Gbps SPOF (and probable bottleneck) was transformed into a load-distributed pair of 4x1Gbps etherchannels, and hey, if they weren’t complaining about the 1Gbps link before, they ought to be blissfully happy now!

Mad Maths

Enter the scaling problem: when it comes to etherchannel, 1+1 does not equal 2.

The reason adding more physical links does not proportionally grow your available bandwidth is that your friendly neighborhood Cisco switch does not load-balance across etherchannel members frame by frame. You might assume that the frame #1 gets sent down etherchannel member #1, frame #2 down etherchannel member #2, etc. in a round-robin fashion. Reality is rather different. What the switch actually does is math. The sort of math will vary depending on the capabilities of the switch, and on what you have configured.

Commonly available etherchannel load-balancing methods include source and destination MACs, source and destination IPs, and (my personal favorite) source and destination layer 4 port. To determine which etherchannel member will be used to forward a frame, the switch performs mad maths based on the load-balancing method you’ve selected. The practical upshot is that the same conversation is always going to be forwarded across the same etherchannel member, because the math always works out the same.

This behavior can impact the network. Imagine backup server BEAST with enough horsepower to fill a 1Gbps link who is runing a restore operation to server NEEDY. BEAST and NEEDY are uplinked to different switches interconnected by an etherchannel. As the restore runs, each frame is hashed by the switch to determine which etherchannel member to forward across; the math will work out the same for every frame, meaning the entire conversation between BEAST and NEEDY is going to be forwarded across the same etherchannel member. The result is kind of like the picture above – one member that’s crushed, while the other members lie comparatively idle.

Congestion Indignities

The switch is not sensitive to an etherchannel member getting crushed; the switch just keeps on doing mad maths. Therefore, some other conversations heading across the link will just happen to get hashed to the same link that the BEAST-NEEDY restore operation is using. Those other unfortunate conversations will therefore suffer the indignities that happen during link congestion: dropped frames and increased latency. The real-world experience is that certain applications act slow or throw errors. Storage could dismount. Monitoring applications get upset as thresholds are exceeded.

Yuck.

Of course, it’s now up to the network engineer (you) to discover why the alarms are going off, track down the offending traffic flow (you are modeling your interswitch links, right?), and figure out what is to be done about it. In my experience, you won’t have a lot of luck explaining what’s happening to non-network people. I’ve had a hard time explaining that 1+1 doesn’t equal 2, (or 1+1+1+1 doesn’t equal 4). You don’t really have a 2Gbps or 4Gbps link just because you’ve built a fancy etherchannel. You’ve really got multiple parallel 1Gbps links, any one of which can still get congested in BEAST-NEEDY scenarios.

So Fix It, Network Guy

There’s a few ways to tackle the challenge of 1+1 not equaling 2.

1. Learn your traffic patterns. See if you can group heavy hitters into the same switch. That’s a pretty old-school way to go after the problem, and it won’t scale to large data center deployments. But you can find wins in this approach from time to time.
2. Build a dedicated link. By this, I mean that you could build a link dedicated to just the traffic that’s causing the interswitch etherchannel all the heartburn. If your etherchannel is a trunk carrying a whole bunch of VLANs, you could build a parallel link that carries traffic for just a problem VLAN, while pruning that problem off of the etherchannel trunk. Might help, might not, depending on your situation…and of course, it’s a “one-off” fix, not a scalable solution necessarily. Some shops build networks dedicated to storage or to backup, and plumb specific interfaces on hosts to these specific networks for exactly this reason. There are increased in hardware, cabling, and complexity to make it happen, though.
3. Add even more 1Gbps links to the etherchannel. This is not terribly practical. At the end of the day, you still have a potential bottleneck, but at least you’ve decreased the number of conversations that are likely to get hashed to a congested link.
4. Replace the 1Gbps links with 10Gbps links. Increasing bandwidth is always an option. The jump to 10Gbps is a tough one, though: new switch hardware, higher power requirements, and likely new cabling will be required. And don’t forget to break out your checkbook.
5. Apply QoS. If you have known offenders or predictable traffic patterns, you can write a QoS scheme to help manage the congestion. I tend to pump traffic like this through a traffic shaper, but there are other approaches, such as guaranteeing minimum bandwidth to important traffic, while dumping the link beast into the scavenger class. I have found that latency still tends to suffer when using a guaranteeing minimum bandwidth (CBWFQ) scheme. I have had the best luck with shaping.
6. Tweak the beastly application. It’s not uncommon for certain applications to have a built-in throttle, so that you can cap network utilization right at the app. Talk to your system engineer and see…I’ve heard they’re people, too.

© ethan for Gestalt IT, 2010. | The Scaling Limitations of Etherchannel -Or- Why 1+1 Does Not Equal 2
Read more posts categorized as All, Networking

Image via Wikipedia

I Swear To Tell The Truth

Prevarication. Untruths. Misdirection. Deceit. Call it what you will, lying is ugly. While we expect lying from people (there’s an American TV show built around this premise), we expect our network gear to tell only the truth. The command line is supposed to a bastion of honesty. And indeed, the command line will tell you exactly what its perception of reality is. The magic (and our job as engineers) is determining when the device in question is woefully misguided…when what it thinks it knows is no longer true.

Caches can be guilty of storing bad data. When they first learned their data, they had learned truth. But as a cache’s data ages, the possibility increases that the cached data becomes stale: out of sync with reality. When cache gives you stale data, it’s lying: a stiff penalty we sometimes pay for performance.

In Your Stack, Improving Your Performance

In networking, practically all network stacks have an address resolution protocol (ARP) cache as a standard feature. What is ARP used for? Simply stated, ARP discovers the link-layer (think Ethernet MAC) address when only the network layer (think IP address) is known. Keep in mind that ARP isn’t specific to IP or to Ethernet; various flavors of ARP are used with other network protocols and transports. For this discussion, we’ll assume Ethernet and IPv4, though.

Well okay, I’m with you so far…but WHY do we need to know what the link-layer address is? I mean, who cares? A host can know the IP address of another host he wishes to communicate with, but that’s not enough information to send a packet across the network wire. In addition to the remote IP address, the host must be able to build a link-layer frame of a type appropriate for the network media to which the host is connected. In the vast majority of today’s LANs, that media is Ethernet. So inside of an *Ethernet* frame will go the IP packet. ARP is used to discover the Ethernet MAC address of a remote IP address necessary for the destination MAC component of the Ethernet frame. Still with me? Good.

Next step then…what is ARP *cache* used for? Like any cache, an ARP cache stores MAC-IP mappings, so that the next time the host needs to send an IP packet to that destination IP address, the Ethernet destination MAC is already known. An network ARP request is not required before the creation of the frame wrapper if the requisite destination MAC can be pulled out of ARP cache. Naturally, this improves network performance as a whole.

• ARP requests are broadcasts. Broadcasts, by definition, are sent to all hosts on a network segment. The higher the volume of broadcasts on a segment, the lower the overall segment performance (think aggregate throughput). ARP cache helps reduce the number of broadcasts.
• ARP queries take longer to send and get a response than doing a lookup against a local cache. Again, a performance boost.

Show Me Your ARP Cache

Would you like to see the ARP cache on your network-attached device?

• On a Windows host, type “arp -a“.
• On a *NIX host, type “arp -an“.
• On a Cisco router or layer 3 switch, type “show ip arp” or “show ip arp vlan X” where X is your VLAN number of choice.

I’ll spare you the command output, because they all look similar: a list of IPs with associated MAC addresses that are currently stored in the device’s ARP cache.

Why Did The New Core Switch Break The Network?

Very early this past Saturday morning, I was remotely guiding a core switch upgrade for a small data center several hours distant from me. I’d built the switch and shipped it to the site, where one of the local engineers had racked and powered it. The maintenance window was to move the cables over from the scary 10 year old switch that made me nervous every time I did a “wr mem” to the shiny new one. The core switch being replaced acted as the router for all the VLANs at this site. This was not a “dual core” design, where the core switch was one of a pair of switches acting as backups of one another. Rather, this core switch was *it*; without this switch, intervlan traffic was impossible, as no other device on the network was providing routing services.

Being remote and having no backdoor, I was relying on the local engineer to tell me what was going on. Moving the cables was the easy bit. All the fiber uplinking remote access switches was moved over without incident; all the lights were green. The copper to move over the Internet border firewalls was similarly successful. Shortly thereafter, the IPSEC site-to-site tunnel I needed to access the site reconnected, and I was able to log into the shiny new core switch. After resolving some anticipated speed/duplex and trunk issues, we still found that a number of hosts were unable to talk outside of their local VLAN. Hmm…

Now, this was not my first rodeo. I’ve replaced a few routers and switches in my day. Stale ARP caches telling lies and lies and lies is an annoying issue that can really get your knickers in a twist if you don’t know what’s going on. However, I was expecting (and dreading) this very circumstance.

Let’s say you’ve got host 10.100.100.10 in Vlan 100 needing to talk to some host outside of Vlan 100. Now, before the core switch replacement, this worked fine. 10.100.100.10 knew that to talk to someone outside Vlan 100, he’d need to forward that intervlan traffic to his default gateway, let’s say 10.100.100.254, where it would be routed. 10.100.100.10 also had an ARP cache entry that told him what the MAC of 10.100.100.254 was, because sometime earlier, he’d put an ARP request on the wire, asking for the MAC of 10.100.100.254, to which the core switch had responded. With that information readily in his ARP cache, 10.100.100.10 would merely need to peek inside his cache, pull out the appropriate MAC, and then wrap his outbound IP packet in an Ethernet frame bound for the MAC of 10.100.100.254 to obtain routing services. Life was grand whenever he needed to communicate with far away subnets.

That is, right up until I replaced the ancient core switch with a new one. “Now I done it,” so to speak. While the default gateway IP was certainly the same – 10.100.100.254 – the MAC associated with that IP address was changed. Remember that Ethernet MAC addresses are assigned by manufacturers and are supposed to be unique to every Ethernet interface the world over. As a result, every host on Vlan 100 had a lying ARP cache. Hosts including my imaginary 10.100.100.10 would frame their traffic heading for the default gateway with a stale destination MAC stored in local ARP cache back when the old core switch was in production. They would then put that ill-formed frame on the wire where it would go…nowhere. Well technically, that frame went *everywhere*: the switches in the VLAN would try to deliver the frame, unicast-flooding it out every port, where the frame would be ignored by everyone. No one in the VLAN had that MAC address anymore.

The core switch was new and there was nothing wrong with it, but the network was broken.

Beat Your ARP Caches Into Submission

So how do you, the unicorn-taming, leprechaun-catching network engineer, resolve this issue? With broken ARP caches all over the network, you look a bit of an idiot. “Hey, network guy. Nice new switch that doesn’t work. You gonna fix it? Should we cut back to the old switch? I think I’m gonna call my boss because none of the servers are working. I need a hug!” You’d like to hug him with a good swift smack to the side of his Windows-loving head with the dirty side of your keyboard (you know, all that cheezle and dead skin that falls in between the keys), but you’re better than that. No, you’re going to provide a *solution*.

• Wait. As with any cache, the entries in an ARP cache have a time-to-live. Eventually, the TTL will expire, and hosts will start working again as they put fresh ARP requests on the wire and discover the new MAC. How long? That depends on the host OS, but generally speaking a period of a few hours. I’m too lazy to check at the moment, but 300 minutes comes to mind as pretty standard time.
• Clear ARP caches on critical devices by hand instead of waiting for TTL to expire. Most devices have a way to purge the ARP cache at the command line, forcing the device to put an ARP request on the network wire. This will force the ARP cache to be repopulated.
• The “swatting a fly with a Buick” approach is to reboot the broken system. A reboot is never a graceful answer, but depending on your circumstances and who you’re working with, it’s a solution easy to understand for all involved, albeit not always expedient.
• Use a MAC address that doesn’t change from one switch to the other. I’m referring to a first-hop redundancy protocol, like HSRP. HSRP computes a MAC address that floats between members in a HSRP group; that computed MAC is always the same, varying only by VLAN number. Therefore, if you replace an HSRP switch with another HSRP switch, your hosts using the new switch as a default gateway won’t know the difference. There’s nothing stopping you from running HSRP on your core switch, even if you’ve only got a single core.
• Note that some devices use gratuitous ARPs to notify remote hosts about their new MAC addresses. This is commonly found in hosts using high-availability protocols where there is no floating MAC address, but merely ownership of a MAC shifting from one host to another. For example, F5 Networks load-balancers rely on a gratuitous ARP during a high-availability active/standby failover event to notify hosts of the change in roles from one load-balancer to the other.

So let’s hear from the rest of you clever people. How to you beat the stale ARP cache problem?

© ethan for Gestalt IT, 2010. | Coping Mechanisms For A Lying ARP Cache
Read more posts categorized as All, Networking

TRILL – TRansparent Interconnection of Lots of Links – is proposed with no technical implementation details in RFC5556.  TRILL’s proposal can be encapsulated thusly:  shove the logic of a layer 3 routing protocol down into layer 2.  Why?  So that switches can bridge traffic via the most efficient path while still avoiding topology loops.  TRILL is therefore an interoperable alternative to spanning-tree.  Spanning-tree creates a single-path for traffic to flow by computing the fastest way back to the root bridge, blocking alternate paths to avoid loops.  TRILL does away with the root-bridge concept, and instead proposes to compute the most efficient path to any destination within a VLAN, and then forward along that path, with no paths in a blocking state.

Why Now?  Do We Really Need TRILL?

You might ask the question “Why do we need this now?”, with the logic that spanning-tree has done very well for a number of years, so what technology is driving a radical new layer 2 path discovery mechanism?  In fairness, TRILL is not “radical” in that current TRILL drafts propose using IS-IS in switches that are being termed “rbridges“.  The radical bit is computing those paths to remote MAC addresses across a layer 2 topology instead of to a remote layer 3 IP subnet – you know, routing.  But is TRILL solving a problem we really have?

I want to answer that question by way of an example.  Consider the diagram below, showing 2 access switches and 2 core switches.  For our purposes, the topology displayed represents one VLAN.  The link colors are largely insignificant.

Assuming Core1 is the spanning-tree root-bridge of this topology and that all links are equal-cost, which ports will become spanning-tree blocked ports?  Core2 will block all but C1-C2.  Access1 will block all but A1-C1.  Access2 will block all but A2-C1.

Let’s say we have a host A attached to Access1, and a host B attached to Access2.  What path will the data flow when host A and B communicate (east-west travel)?  Access1 <=A1-C1=> Core1 <=A2-C1=> Access2.  The Core1 switch is an extra hop. The most efficient path would be Access1 <=A1-A2=> Access2, leaving Core1 out of the forwarding path.

Assuming TRILL was running on these 4 switches instead of spanning-tree, each switch would compute the most efficient path to each remote layer 2 destination (MAC address) and use it.  No more blocked ports; no more unused links.  In a world where my hypothetical hosts A and B represent VMware clusters doing large data volumes of vMotion, and link A1-A2 could represent expensive 10G or an etherchannel of multiple 10G ports, using that A1-A2 link becomes highly desirable.

In a well-architected data center design leveraging TRILL, traffic will be distributed more evenly across the available interswitch links, maximizing throughput, minimizing latency, and improving performance scale as new links are added (i.e. when you add a new expensive link, it will actually get used, so your overall data center traffic capacity scales up).

Highlights, Excerpts, and Interesting Bits from RFC5556

• From the abstract: “Routing tends not to take full advantage of alternate paths, or even non-overlapping pairwise paths (in the case of spanning trees). This document addresses these concerns and suggests applying modern network-layer routing protocols at the link layer.”
• From 1. Introduction: “With spanning trees, the bandwidth across the subnet is limited because traffic flows over a subset of links forming a single tree…It is thus useful to consider a new approach that combines the features of these two existing solutions, hopefully retaining the desirable properties of each. Such an approach would develop a new kind of bridge system that was capable of using network-style routing, while still providing Ethernet service. It allows reuse of well-understood network routing protocols to benefit the link layer.”
• From 2. The TRILL Problem: “The spanning tree often results in inefficient use of the link topology; traffic is concentrated on the spanning tree path, and all traffic follows that path even when other more direct paths are available. The addition in IEEE 802.1Q of support for multiple spanning trees helps a little, but the use of multiple spanning trees requires additional configuration, the number of trees is limited, and these defects apply within each tree regardless.”
• From 2.2 Multipath Forwarding: “Using spanning trees reduces aggregate bandwidth by forcing all such paths onto one tree, while modern routing causes such paths to be selected based on a cost metric. However, extensions to modern routing protocols enable even greater aggregate bandwidth by permitting traffic flowing from one endpoint to another to be sent over multiple, typically equal-cost, paths.”
• From 2.5 IEEE 802.1 Bridging Protocols: “There have been a variety of IEEE protocols beyond the initial shared-media Ethernet variant, including 802.1D, 802.1w, 802.1Q, 802.1v, and 802.1s. This document presumes the above variants are supported on the Ethernet subnet, i.e., that a TRILL solution would not interfere with (i.e., would not affect) any of the above.”
• From 3.3 Forwarding Loop Mitigation: “Solutions to TRILL are intended to use adapted network-layer routing protocols that may introduce transient loops during routing convergence. A TRILL solution thus needs to provide support for mitigating the effect of such routing loops…These types of mechanisms limit the impact of loops or detect them explicitly. Mechanisms with similar effect should be included in TRILL solutions.”
• From 3.4 Spanning Tree Management: “In order to address convergence under reconfiguration and robustness to link interruption (Section 2.2), participation in the spanning tree (STP) must be carefully managed. The goal is to provide the desired stability of the TRILL solution and of the entire Ethernet link subnet, which may include bridges using STP. This may involve a TRILL solution participating in the STP…”
• From 3.8 Optimizations: “There are a number of optimizations that may be applied to TRILL solutions. These must be applied in a way that does not affect functionality as a tradeoff for increased performance…For example, in many bridged LANs, there are topologies such that central (“core”) bridges which have both a greater volume of traffic flowing through them as well as traffic to and from a larger variety of end station than do non-core bridges. This means that such core bridges need to learn a large number of end station addresses and need to do lookups based on such addresses very rapidly. This might require large high speed content addressable memory making implementation of such core bridges difficult. Although a TRILL solution need not provide such optimizations, it may reduce the need for such large, high speed content addressable memories or provide other similar optimizations.”
• From 3.9 Internet Architecture Issues: “TRILL solutions are intended to have no impact on the Internet network layer architecture. In particular, the Internet and higher layer headers should remain intact when traversing a deployed TRILL solution, just as they do when traversing any other link subnet technologies.”
• From 4. Applicability: “TRILL solutions are not intended to span separate Ethernet link subnets interconnected by network-layer (e.g., router) devices, except via link-layer tunnels, where such tunnels render the distinct subnet undetectably equivalent from a single Ethernet link subnet.”
• From 5. Security Considerations: “TRILL solutions are not intended to be a solution to Ethernet link subnet vulnerabilities, including spoofing, flooding, snooping, and attacks on the link control plane (STP, flooding the learning cache) and link-network control plane (ARP). Although TRILL solutions are intended to provide more stable routing than STP, this stability is limited to performance, and the subsequent robustness is intended to address non-malicious events.”

© ethan for Gestalt IT, 2010. | Traveling East-West Might Get A Little Easier: Highlights from the TRILL RFC5556
Read more posts categorized as All, Networking

“Convergence” is a buzzword seen in the IT press constantly these days.  All convergence means is placing communications that used to ride on its own network onto one unified network; Ethernet’s cheapness, ubiquity, and ever-growing link speeds makes it the network everything is moving towards.  The first big convergence move was to combine voice networks with data networks, using IP telephony.  The challenges of a converged voice/data network include prioritizing voice traffic over pretty much anything else during times of link congestion, and keeping call quality high by delivering datagrams in a predictable time with a predictable gap in between those datagrams.

Frankly, these problems were and are a big pain in the collective backsides of network engineers everywhere.  Ethernet and IP are not transports intended to deliver traffic in a predictable, prioritized fashion.  Ethernet is a best-effort frame delivery system that’s only survived as long as it has by reducing collision domains down to one via switches.  IP uses higher-level protocols for reliability and underlying devices for prioritization.  Therefore, delivering VoIP frames and packets across a converged infrastructure means a carefully designed and deployed quality of service plan; the larger and more complex the network environment, the greater the potential pain.  Throw in congestion-prone wide-area links of varying transports with their own forwarding nuances (ATM is not frame-relay is not MPLS is not PPP), and the network engineer must know a lot about a lot to deliver an effective end-to-end QoS design.  Even worse, no one hears the engineer scream who is configuring QoS on multiple platforms, all of which might require unique configurations depending on hardware and software to net the same results, even within the same vendor product families.

Convergence Redux

Convergence has evolved with the increasing affordability and adoption of 10-gigabit Ethernet.  With the bandwidth and low-latency of 10G, the industry has pushed towards adding storage to the converged data center Ethernet, the idea being to eventually eliminate the unique Storage Area Network.  Fibre channel is the protocol of major concern here, in that other storage protocols like iSCSI are more tolerant of changing network throughput characteristics.  FC is not tolerant of a changing environment.  FC expects that frames will be delivered on-time, every time, and therein lies a significant challenge for the converged data center Ethernet.  While 10G is an awful lot of bandwidth, simply adding Even More Bandwidth (the tried-and-true method of capacity management for engineers who don’t want to think too hard) isn’t a safe answer.  No matter how much bandwidth is available, the Ethernet carrying Fibre Channel traffic (FCoE) must be able to guarantee a lossless path from host to disk and back.

Don’t Drop The Baby!

Enter Data Center Bridging, or DCB.  DCB comprises a set of proposed standards designed to extend Ethernet such that we can:

• Leverage flow control on up to 8 virtual links. (IEEE 802.1Qbb – Priority-based Flow Control) We can issue ethernet PAUSE frames on specific virtual links, and not interrupt forwarding for the entire physical link.  Practically speaking, we could tell everyone but the storage virtual link to shut up for a moment.
• Prioritize traffic classes within a virtual link. (IEEE 802.1Qaz – Enhanced Transmission Selection)  Here, we can use QoS techniques within a virtual link to prioritize certain kinds of traffic.  Voice – you rule!  Network engineer’s web surfing – go to the head of the line!  Bittorrent from Joe down the hall – tail drop.
• Exchange DCB information with other DCB devices. (also part of IEEE 802.1Qaz – DCBX, and expected to leverage LLDP)  Two devices can learn about each other’s DCB-related link characteristics.
• Optionally notify upstream senders of downstream congestion, allowing the sender to mitigate the congestion through rate-shaping.  (IEEE 802.1Qau – Congestion Notification)  In a backwards way, this reminds me of RSVP, where you can do an end-to-end bandwidth reservation across multiple network devices.  I’ll be interested to see how this is implemented.

All of this gives us the ability to guarantee that storage traffic has both the forwarding capacity and priority over other traffic flows when needed.  Done right, an FCoE frame should never hit the floor.  That said, I am interested to see how much hands-on engineering will be required to make this work as intended.  Legacy QoS is far from automatic in the real world, and it seems fair to assume that a well-designed DCB implementation will require rather a lot of whiteboarding, implementing, monitoring, and tweaking before it perfectly serves the environment it has been deployed in.

Hey, Pal – Ya Wanna Buy An Enhanced Ethernet?

DCB has a couple of implementations:  Data Center Ethernet (DCE) is Cisco’s flavor, and includes an implementation of TRILL.  Cisco markets DCE and related advanced data center technologies under the term “FabricPath“, which is baked into the Nexus 7000 product line.  In some Cisco documentation, they refer to DCE as a superset of DCB and another implementation of DCB, Converged Enhanced Ethernet (CEE).  CEE has a much broader group of networking companies behind it, including Broadcom, Brocade, Cisco, Emulex, HP, IBM, Juniper, and QLogic, and is helping to craft how the DCB standards will finally look when ratified.

The point of DCB then is to provide an interoperable set of standards whereby storage traffic can be guaranteed the bandwidth and flow characteristics it requires across an Ethernet transport, while co-existing with other traffic behaving quite differently.

The problem?  At this point, the jury is out on what form of converged storage the market will claim.  FCoE is seeing increased adoption, but uptake has been slow.  iSCSI has been around for a while, is generally well-understood by engineers who deploy it, and a popular choice.  Vendors are pushing various and usually proprietary fabric schemes.  Cisco, HP, Brocade, and IBM are all making acquisitions and creating product lines to sell you a single-vendor solution for storage, servers, and a converged network to run it on.

One thing’s for certain:  it’s a fun time to be a network engineer.

© ethan for Gestalt IT, 2010. | Don’t Drop The Baby: Data Center Bridging Wants Storage To Trust Ethernet
Read more posts categorized as All, Featured, Networking, Storage

Due to people commitments we recorded a double length show which will be released in two parts. This is Part 2 and Part 1 was released next weekend.

What You’ll Hear…

• On the increase in 10GbE in the Data Centre and what’s driving that. And looking at why we aren’t comfortable with the HP Flex-10 networking module for their blade servers.
• We are planning on using IP Storage, after consulting with the Server team. The cost of FC doesn’t work for us, and FCoE still isn’t here.
• A look at transient traffic loads and how the deployment of VMware DRS changes the way our backbone looks. The importance of dynamic traffic flows to VMware and what we need to do to support that.
• We take a detailed and critical look at what Cisco doesn’t tell you about the Nexus 7000 – the bad things, the missing features such as QoS, MPLS and lack of value. Not to mention the generally underwhelming performance of the product. Also, it’s big, it uses a lot of power and runs hot.

You can find Jeremy Filliben at http://jeremyfilliben.com and @jfilliben.

You can find Steve Rossen on @steve

You can find Ivan Pepelnjak at http://ioshints.info and on @ioshints.

IOS Hints Live – San Jose September 2010

You can book to join the event at ioshintsdatacenter.eventbrite.com/. There are only a limited number of seats at this unique event where Ivan Pepelnjak and Greg Ferro will both be available to discuss, review and develop your designs.

Feedback

Follow the Packet Pushers on Twitter (@packetpushers | Greg @etherealmind | Dan@rovingengineer | Ethan @ecbanks) and send your queries and comments about the show to packetpushers@gmail.com.  We want to hear from you!

© Etherealmind for Gestalt IT, 2010. | Show 17 – Big Hot and Heavy Switches – Part 2
Read more posts categorized as Favorites, Networking