Some of you know I took on a new job earlier this year, where the challenge was (and is) to transform a globally distributed network for a growing company into an enterprise class operation. A major focus area has been eliminating single points of failure (SPOFs): single links, single routers, single firewalls, etc. If it can break and consequently interrupt traffic flow, part of my job is to design around the SPOF within the constraints of a finite budget.

The network documentation I inherited ranged from “mostly right but vague and outdated” to “a complete and utter fantasy requiring mind-altering substances to make sense of”. Ergo, untrustworthy to the point of being useless beyond perhaps slideware to show a particularly dim collection of simians. I have therefore been doing complete network explorations, building new documentation as I go.

To my horror, I one day discovered an egregious SPOF, where a single, fragile piece of CAT5 provided the sole physical path between two major concentrations of network activity. If that link ran into any trouble, an entire room containing hundreds of physical and virtual servers (and their storage) would have been cut off from the rest of the company.

To eliminate the physical path SPOF, the easy choice was to transform the single link into an etherchannel. This I did; the single 1Gbps became a 4x1Gbps etherchannel plumbed back to one core switch. For good measure, I added a second 4x1Gbps, plumbed to a second core switch. Spanning-tree roots had already been established such that even-numbered VLANs would traverse one of the 4x1Gbps etherchannels, and odds the other…which you can read more about here if interested.

All should now be sweetness and light, right? A 1Gbps SPOF (and probable bottleneck) was transformed into a load-distributed pair of 4x1Gbps etherchannels, and hey, if they weren’t complaining about the 1Gbps link before, they ought to be blissfully happy now!

Mad Maths

Enter the scaling problem: when it comes to etherchannel, 1+1 does not equal 2.

The reason adding more physical links does not proportionally grow your available bandwidth is that your friendly neighborhood Cisco switch does not load-balance across etherchannel members frame by frame. You might assume that the frame #1 gets sent down etherchannel member #1, frame #2 down etherchannel member #2, etc. in a round-robin fashion. Reality is rather different. What the switch actually does is math. The sort of math will vary depending on the capabilities of the switch, and on what you have configured.

Commonly available etherchannel load-balancing methods include source and destination MACs, source and destination IPs, and (my personal favorite) source and destination layer 4 port. To determine which etherchannel member will be used to forward a frame, the switch performs mad maths based on the load-balancing method you’ve selected. The practical upshot is that the same conversation is always going to be forwarded across the same etherchannel member, because the math always works out the same.

This behavior can impact the network. Imagine backup server BEAST with enough horsepower to fill a 1Gbps link who is runing a restore operation to server NEEDY. BEAST and NEEDY are uplinked to different switches interconnected by an etherchannel. As the restore runs, each frame is hashed by the switch to determine which etherchannel member to forward across; the math will work out the same for every frame, meaning the entire conversation between BEAST and NEEDY is going to be forwarded across the same etherchannel member. The result is kind of like the picture above – one member that’s crushed, while the other members lie comparatively idle.

Congestion Indignities

The switch is not sensitive to an etherchannel member getting crushed; the switch just keeps on doing mad maths. Therefore, some other conversations heading across the link will just happen to get hashed to the same link that the BEAST-NEEDY restore operation is using. Those other unfortunate conversations will therefore suffer the indignities that happen during link congestion: dropped frames and increased latency. The real-world experience is that certain applications act slow or throw errors. Storage could dismount. Monitoring applications get upset as thresholds are exceeded.

Yuck.

Of course, it’s now up to the network engineer (you) to discover why the alarms are going off, track down the offending traffic flow (you are modeling your interswitch links, right?), and figure out what is to be done about it. In my experience, you won’t have a lot of luck explaining what’s happening to non-network people. I’ve had a hard time explaining that 1+1 doesn’t equal 2, (or 1+1+1+1 doesn’t equal 4). You don’t really have a 2Gbps or 4Gbps link just because you’ve built a fancy etherchannel. You’ve really got multiple parallel 1Gbps links, any one of which can still get congested in BEAST-NEEDY scenarios.

So Fix It, Network Guy

There’s a few ways to tackle the challenge of 1+1 not equaling 2.

1. Learn your traffic patterns. See if you can group heavy hitters into the same switch. That’s a pretty old-school way to go after the problem, and it won’t scale to large data center deployments. But you can find wins in this approach from time to time.
2. Build a dedicated link. By this, I mean that you could build a link dedicated to just the traffic that’s causing the interswitch etherchannel all the heartburn. If your etherchannel is a trunk carrying a whole bunch of VLANs, you could build a parallel link that carries traffic for just a problem VLAN, while pruning that problem off of the etherchannel trunk. Might help, might not, depending on your situation…and of course, it’s a “one-off” fix, not a scalable solution necessarily. Some shops build networks dedicated to storage or to backup, and plumb specific interfaces on hosts to these specific networks for exactly this reason. There are increased in hardware, cabling, and complexity to make it happen, though.
3. Add even more 1Gbps links to the etherchannel. This is not terribly practical. At the end of the day, you still have a potential bottleneck, but at least you’ve decreased the number of conversations that are likely to get hashed to a congested link.
4. Replace the 1Gbps links with 10Gbps links. Increasing bandwidth is always an option. The jump to 10Gbps is a tough one, though: new switch hardware, higher power requirements, and likely new cabling will be required. And don’t forget to break out your checkbook.
5. Apply QoS. If you have known offenders or predictable traffic patterns, you can write a QoS scheme to help manage the congestion. I tend to pump traffic like this through a traffic shaper, but there are other approaches, such as guaranteeing minimum bandwidth to important traffic, while dumping the link beast into the scavenger class. I have found that latency still tends to suffer when using a guaranteeing minimum bandwidth (CBWFQ) scheme. I have had the best luck with shaping.
6. Tweak the beastly application. It’s not uncommon for certain applications to have a built-in throttle, so that you can cap network utilization right at the app. Talk to your system engineer and see…I’ve heard they’re people, too.

© ethan for Gestalt IT, 2010. | The Scaling Limitations of Etherchannel -Or- Why 1+1 Does Not Equal 2
Read more posts categorized as All, Networking

Networking may be straightforward, but the world of networking terminology is not. I’ve been steeped in the strange lingo of Ethernet for many years, but I still get confused by some of the terms. What’s the difference between 1000BASE-CX, 1000BASE-SX, and 1000BASE-T? In this post, I’m going to tackle this Ethernet network naming convention.

Networking Basics

Let’s get the first two components of the network name out of the way:

The first part is the signaling rate in megabits per second. In layman’s terms, this is the speed of the network at hand. You are likely to come across one of the following:

• 10 megabits
• 100 megabits – Fast Ethernet
• 1000 megabits – Gigabit Ethernet, GbE, or 1000BASE-X
• 10,000 megabits – 10 Gigabit Ethernet, 10GbE, or 10GBASE-X
• 40,000 megabits – 40 Gigabit Ethernet, 40GbE, or 40GBASE-X
• 100,000 megabits – 100 Gigabit Ethernet, 100GbE, or 100GBASE-X

It may strike you as odd that the next part is always the word, “BASE”. But there is a reason for this, too.BASE refers “baseband”, meaning that this is an unfiltered line not requiring a digital modulation scheme. Back in the day, there was a 10PASS-TS version of Ethernet that used a signaling scheme similar to a modem, but baseband is dominant today.

So 100BASE refers to a Fast Ethernet connection that uses the unfiltered cable for transmission.

BASE-What?

The third part of an Ethernet network type refers to the cabling used to carry the signals. The earliest forms of Ethernet used coaxial cable, but thin twisted-pair cabling became popular in the mid-1990s. Faster versions of Ethernet also often use fiber optics rather than electrical signals.

There are a bewildering assortment of physical interconnects for Ethernet. But the naming system isn’t as complex as it might appear:

• The first letter tells us which kind of wire we are talking about:
  • “T” means twisted-pair cable (e.g. the common Cat5 in use today)
  • “K” means a copper backplane
  • “C” means balanced copper cable
  • “F” means optical cable
  • “B” uses two wavelengths over a single optical cable
  • “S” means short-range multi-mode optical cable (less than 100 m)
  • “L” means long-range single- or multi-mode optical cable (100 m to 10 km)
  • “E” means extended-range optical cable (10 km to 40 km)
  • “Z” means long-range single-mode cable at a higher wavelength
• Next is the coding scheme for data on the wire
  • “X” means 4B/5B block coding for Fast Ethernet or 8B/10B block coding for Gigabit Ethernet
  • “R” means 64B/66B block coding
• Finally, we have a number representing the number of parallel “lanes” for data
  • “1″ would mean serial (non-parallel) but is omitted instead
  • “4″ or “10″ are available for copper wire
  • Just about any other number could be used for optical lanes or wavelengths

Examples

Now let’s look at some examples:

• Back in the day, 10BASE-T became more common than coaxial 10BASE2. It was a simple 10 megabit baseband signal over common twisted-pair.
• When Fast Ethernet first rolled out, there was some concern that traditional (usually Cat3) cabling couldn’t handle 100 megabits. Some early implementations used four copper pairs (100BASE-T4) or fiber optics (100BASE-FX), but nearly every 100 megabit Ethernet connection today is 100BASE-TX, using plain two pairs on plain Cat5 cable.
• Gigabit Ethernet had a similar history. Many were concerned that two pairs on unshielded Cat5 wiring could not handle 1000 megabits per second, so optical (1000BASE-SX) and balanced shielded wiring (1000BASE-CX) were specified. Although an unshielded 2-pair standard was developed (1000BASE-TX), it never really caught on. Therefore, today’s predominant gigabit LAN connection, 1000BASE-T, uses all four pairs of unshielded twister-pair wiring on a Cat5 cable (see note 1).
• The 10 Gigabit Ethernet world has mostly shifted to the block coding scheme from Fibre Channel, 64B/66B, which is denoted by the letter “R”. This gives us a family of fiber optic cables (10GBASE-SR, LR, ER, etc), and a copper backplane interconnect (10GBASE-KR). The earlier copper wiring standard (10GBASE-CX4) used InfiniBand-like 4-lane cables and 8B/10B signaling, as did 10GBASE-KX4 on the backplane. A backwards-compatible twisted-pair 10GBASE-T has also been developed, but work continues to make it power-efficient enough to be practical (see note 2).
• Looking ahead, we see Higher-Speed Ethernet emerging: 40GBASE-KR4 for backplane use, multi-mode optical 40GBASE-SR4 and 100GBASE-SR10, and long-range single-mode optical 40GBASE-LR4 and 100GBASE-LR10.

As you can see, all this alphabet soup does have some consistency. Common unshielded twisted pair wiring is all “BASE-T”, optics are denoted according to their range (“S”, “L”, “E”), and backplanes use “K” copper. Clear as mud?

Note 1: Lots of people (and even equipment makers) incorrectly refer to common Gigabit Ethernet as “1000BASE-TX”, but this really should be called “1000BASE-T”.

Note 2: We will probably never see a 10GBASE-TX, which would use just 2 pairs of unshielded twisted pair copper wiring.

© Stephen Foskett for Gestalt IT, 2010. | 1000Base-What?
Read more posts categorized as Networking, Server Virtualization, Storage

Ever since Microsoft and Intel declared that the combination of Windows and Nehalem could deliver over a million iSCSI IOPS, I’ve been curious about just how they did it. What black magic could push that many I/Os over a single Ethernet connection? And what was on the other end? Now Intel has revealed all in a whitepaper, and the results are surprising!

What iSCSI Did

Let’s review the test for a moment. In March, Microsoft and Intel demonstrated that the combination of Windows Server 2008 R2 and the Xeon 5500 could saturate a 10 Gb Ethernet link, pushing iSCSI throughput to wire speed. That’s 1,174 MB/s, right around the theoretical maximum of a ten-gigabit link, given a tiny bit of overhead. The pair reunited in January to show that this same combination could deliver an astonishing million I/O operations per second, too.

Both of these results are astonishing. Sure, many high-end Fibre Channel SANs and storage systems blast out gigabytes of data and millions of I/O operations every second, but these tests are much more focused. Benchmarks are perilous, but the folks at Microsoft and Intel devised a fairly clever and focused set. Rather than a “mine’s bigger” contest, the pair only needs to prove that iSCSI can play with the pros.

The side effect is a demonstration of the capabilities of Microsoft and Intel components. Microsoft showed off the capabilities of Windows Server 2008 R2, Hyper-V, and their software iSCSI initiator, while Intel can brag about the Xeon 5500 server platform and X520-2 10 Gb Ethernet Server Adapter with their 82599EB controller. Your mileage may vary, but it is possible to construct a true storage monster on an average server budget.

Intel Inside

Let’s start by looking at the configuration of the local end of the tested configuration. I’m a storage guy so I think of it as the initiator, but you might say it’s the server, the client, or the host. Regardless, the system under test (SUT) is what was put under the microscope. The configuration was a common one: A high-end computer packing an Intel Xeon CPU and 82599-based 10 Gb Ethernet adapter. Most data centers have a machine or two just like this one.

Looking closely, we see that the test in question relied on the following key components:

• Intel’s “Shadycove” S5520SC workstation-class motherboard
• The Intel Xeon W5580 CPU (4 cores, 8 MB cache, 3.20 GHz)
• 24 GB of DDR3 RAM
• Intel “Niantic” 82599EB 10 Gb Ethernet controller
• Microsoft Windows Server 2008 R2 x64

This combination would set you back about $7,500 – $450 for the motherboard, $1,500 for the CPU, 6 2 GB DDR3 SDRAM modules at $80 each, $1,200 for the Intel X520 NIC, and $4,000 for an Enterprise copy of Windows Server 2008 R2. Not cheap, but not an exotic server either.

Initiate and Optimize

The secret to push the tested system to perform like it did is in the optimizations in the server platform, the NIC, and Windows Server itself.

• The Xeon 5500 processor series includes many enhancements:
  • An integrated memory controller allows for faster RAM access
  • QuickPath interconnect (QPI) replaces the old front-side bus and enhances I/O off the core
  • A new I/O subsystem with PCIe integrated into the CPU
  • MSI-X expands the number of interrupts a PCI device can use
  • New instructions for on-board CRC-32C decoding, speeding up iSCSI digest processing
• The 82599 Ethernet controller also includes enhanced capabilities:
  • VMDq maps I/O queues to multiple cores and virtual machines, reducing I/O bottlenecks
  • Offload of TCP segmentation and receive-side coalescing
  • Interestingly, it does not appear that VMDc/SR-IOV was employed in the test
• Microsoft Windows Server 2008 R2 and Hyper-V are ready to use all of these features and more:
  • R2 uses multi-core CPUs more effectively in general
  • Receive-side scaling (RSS) spreads the I/O workload across all four Xeon cores
  • The iSCSI initiator now allows CRC digest offload (using the new Xeon command set)
  • Numerous “NUMA I/O” optimizations in the initiator
  • TCP/IP Nagle can be disabled in the registry
  • Hyper-V VMQ allows the network packets to be copied directly into the guest virtual machine’s memory

Whew! Put all of these optimizations in a blender and Hyper-V virtual machine iSCSI access will be twice as fast as before. No kidding!

Stay On Target

But we knew all of this back in January. We also saw that a Cisco Nexus 5020 switch was used to fan out to 10 software iSCSI targets. But until now there was no mention of what targets were used exactly.

The final footnotes in Intel’s whitepaper reveals that the storage backing the million IOPS test was none other than StarWind Software’s iSCSI SAN! It is unclear what led Microsoft and Intel to use this particular iSCSI target (the earlier throughput tests ran on NetApp filers), but it does speak to the quality of this product.

It is not clear how many disk drives were used, but I would guess that SSDs or ramdisks might have been employed to pull a million IOPS. Network optimizations are also not mentioned, though jumbo frames would not be a benefit in an IOPS test.

StarWind’s software runs on Microsoft Windows and creates a full-featured iSCSI target, complete with data mirroring, automatic failover and failback, replication, snapshots, and thin provisioning. The company prices their iSCSI SAN at $6,000 for two nodes and competes with the likes of DataCore and Open-E. But the StarWind solution seems at a glance to be more full-featured than these other offerings.

Try It Yourself!

I imagine many folks like me might be tempted to try to reproduce these results. More valuable would be a set of best practice guidelines for the deployment of software iSCSI in Windows Server 2008 R2 and Hyper-V environments. Given the relatively modest hardware involved, there should be nothing stopping us!

These test results also prompted me to get in touch with StarWind to try their iSCSI target software. I was pleasantly surprised to learn that they are currently offering free non-production licenses to VMware vExperts, VCPs, and VCIs as well as Microsoft MVPs, MCPs, and MCT Professionals. Many of my readers fall into one (or more) of those buckets, and I applaud the company for this offer. If only more companies realized the value in giving away test licenses to influencers and thought leaders!

© Stephen Foskett for Gestalt IT, 2010. | How Did Microsoft and Intel Get 1 Million iSCSI IOPS?
Read more posts categorized as Server Virtualization, Storage

How fast can iSCSI get?

“Maximizing Hyper-V iSCSI Performance with Microsoft and Intel” might sound like another “blah blah” marketing piece, but a little birdy tells me that this webcast will drop a bombshell about iSCSI performance.

Lots of storage and networking folks don’t give iSCSI and Microsoft the credit they deserve. “iSCSI is cheap and easy,” they say, “but real performance requires Fibre Channel.” Those of us who have an open mind about such things know that this is simply not the case. The fastest SAN I ever saw was based on iSCSI, and Microsoft demonstrated wire-speed iSCSI over 10 Gb Ethernet in March. I never saw a Fibre Channel SAN (even an 8 Gb/s one) push over a gigabyte per second over a single link!

Still, ask the average sysadmin and they will tell you that iSCSI isn’t for high performance applications. That’s why folks should tune in to this webcast, as Microsoft and Intel knock down another iSCSI performance myth. Note that even though Hyper-V is called out in the title and description, this discussion is really about Windows Server 2008 R2 and applies equally regardless of whether or not you use Microsoft’s hypervisor.

Watch this space for a summary of the news immediately following the announcement.

• What: Maximizing Hyper-V iSCSI Performance with Microsoft and Intel webcast
• When: Thursday, January 14, 2010 8:00 AM Pacific Time
• Where: MSEvents.Microsoft.com
• Who: Anyone interested in high-performance storage and server I/O

© sfoskett for Stephen Foskett, Pack Rat, 2010. |
Microsoft and Intel Pushing iSCSI Performance Limits


This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2010. | Microsoft and Intel Pushing iSCSI Performance Limits
Read more posts categorized as Server Virtualization, Storage