Networking may be straightforward, but the world of networking terminology is not. I’ve been steeped in the strange lingo of Ethernet for many years, but I still get confused by some of the terms. What’s the difference between 1000BASE-CX, 1000BASE-SX, and 1000BASE-T? In this post, I’m going to tackle this Ethernet network naming convention.

Networking Basics

Let’s get the first two components of the network name out of the way:

The first part is the signaling rate in megabits per second. In layman’s terms, this is the speed of the network at hand. You are likely to come across one of the following:

• 10 megabits
• 100 megabits – Fast Ethernet
• 1000 megabits – Gigabit Ethernet, GbE, or 1000BASE-X
• 10,000 megabits – 10 Gigabit Ethernet, 10GbE, or 10GBASE-X
• 40,000 megabits – 40 Gigabit Ethernet, 40GbE, or 40GBASE-X
• 100,000 megabits – 100 Gigabit Ethernet, 100GbE, or 100GBASE-X

It may strike you as odd that the next part is always the word, “BASE”. But there is a reason for this, too.BASE refers “baseband”, meaning that this is an unfiltered line not requiring a digital modulation scheme. Back in the day, there was a 10PASS-TS version of Ethernet that used a signaling scheme similar to a modem, but baseband is dominant today.

So 100BASE refers to a Fast Ethernet connection that uses the unfiltered cable for transmission.

BASE-What?

The third part of an Ethernet network type refers to the cabling used to carry the signals. The earliest forms of Ethernet used coaxial cable, but thin twisted-pair cabling became popular in the mid-1990s. Faster versions of Ethernet also often use fiber optics rather than electrical signals.

There are a bewildering assortment of physical interconnects for Ethernet. But the naming system isn’t as complex as it might appear:

• The first letter tells us which kind of wire we are talking about:
  • “T” means twisted-pair cable (e.g. the common Cat5 in use today)
  • “K” means a copper backplane
  • “C” means balanced copper cable
  • “F” means optical cable
  • “B” uses two wavelengths over a single optical cable
  • “S” means short-range multi-mode optical cable (less than 100 m)
  • “L” means long-range single- or multi-mode optical cable (100 m to 10 km)
  • “E” means extended-range optical cable (10 km to 40 km)
  • “Z” means long-range single-mode cable at a higher wavelength
• Next is the coding scheme for data on the wire
  • “X” means 4B/5B block coding for Fast Ethernet or 8B/10B block coding for Gigabit Ethernet
  • “R” means 64B/66B block coding
• Finally, we have a number representing the number of parallel “lanes” for data
  • “1″ would mean serial (non-parallel) but is omitted instead
  • “4″ or “10″ are available for copper wire
  • Just about any other number could be used for optical lanes or wavelengths

Examples

Now let’s look at some examples:

• Back in the day, 10BASE-T became more common than coaxial 10BASE2. It was a simple 10 megabit baseband signal over common twisted-pair.
• When Fast Ethernet first rolled out, there was some concern that traditional (usually Cat3) cabling couldn’t handle 100 megabits. Some early implementations used four copper pairs (100BASE-T4) or fiber optics (100BASE-FX), but nearly every 100 megabit Ethernet connection today is 100BASE-TX, using plain two pairs on plain Cat5 cable.
• Gigabit Ethernet had a similar history. Many were concerned that two pairs on unshielded Cat5 wiring could not handle 1000 megabits per second, so optical (1000BASE-SX) and balanced shielded wiring (1000BASE-CX) were specified. Although an unshielded 2-pair standard was developed (1000BASE-TX), it never really caught on. Therefore, today’s predominant gigabit LAN connection, 1000BASE-T, uses all four pairs of unshielded twister-pair wiring on a Cat5 cable (see note 1).
• The 10 Gigabit Ethernet world has mostly shifted to the block coding scheme from Fibre Channel, 64B/66B, which is denoted by the letter “R”. This gives us a family of fiber optic cables (10GBASE-SR, LR, ER, etc), and a copper backplane interconnect (10GBASE-KR). The earlier copper wiring standard (10GBASE-CX4) used InfiniBand-like 4-lane cables and 8B/10B signaling, as did 10GBASE-KX4 on the backplane. A backwards-compatible twisted-pair 10GBASE-T has also been developed, but work continues to make it power-efficient enough to be practical (see note 2).
• Looking ahead, we see Higher-Speed Ethernet emerging: 40GBASE-KR4 for backplane use, multi-mode optical 40GBASE-SR4 and 100GBASE-SR10, and long-range single-mode optical 40GBASE-LR4 and 100GBASE-LR10.

As you can see, all this alphabet soup does have some consistency. Common unshielded twisted pair wiring is all “BASE-T”, optics are denoted according to their range (“S”, “L”, “E”), and backplanes use “K” copper. Clear as mud?

Note 1: Lots of people (and even equipment makers) incorrectly refer to common Gigabit Ethernet as “1000BASE-TX”, but this really should be called “1000BASE-T”.

Note 2: We will probably never see a 10GBASE-TX, which would use just 2 pairs of unshielded twisted pair copper wiring.

© Stephen Foskett for Gestalt IT, 2010. | 1000Base-What?
Read more posts categorized as Networking, Server Virtualization, Storage

Ever since Microsoft and Intel declared that the combination of Windows and Nehalem could deliver over a million iSCSI IOPS, I’ve been curious about just how they did it. What black magic could push that many I/Os over a single Ethernet connection? And what was on the other end? Now Intel has revealed all in a whitepaper, and the results are surprising!

What iSCSI Did

Let’s review the test for a moment. In March, Microsoft and Intel demonstrated that the combination of Windows Server 2008 R2 and the Xeon 5500 could saturate a 10 Gb Ethernet link, pushing iSCSI throughput to wire speed. That’s 1,174 MB/s, right around the theoretical maximum of a ten-gigabit link, given a tiny bit of overhead. The pair reunited in January to show that this same combination could deliver an astonishing million I/O operations per second, too.

Both of these results are astonishing. Sure, many high-end Fibre Channel SANs and storage systems blast out gigabytes of data and millions of I/O operations every second, but these tests are much more focused. Benchmarks are perilous, but the folks at Microsoft and Intel devised a fairly clever and focused set. Rather than a “mine’s bigger” contest, the pair only needs to prove that iSCSI can play with the pros.

The side effect is a demonstration of the capabilities of Microsoft and Intel components. Microsoft showed off the capabilities of Windows Server 2008 R2, Hyper-V, and their software iSCSI initiator, while Intel can brag about the Xeon 5500 server platform and X520-2 10 Gb Ethernet Server Adapter with their 82599EB controller. Your mileage may vary, but it is possible to construct a true storage monster on an average server budget.

Intel Inside

Let’s start by looking at the configuration of the local end of the tested configuration. I’m a storage guy so I think of it as the initiator, but you might say it’s the server, the client, or the host. Regardless, the system under test (SUT) is what was put under the microscope. The configuration was a common one: A high-end computer packing an Intel Xeon CPU and 82599-based 10 Gb Ethernet adapter. Most data centers have a machine or two just like this one.

Looking closely, we see that the test in question relied on the following key components:

• Intel’s “Shadycove” S5520SC workstation-class motherboard
• The Intel Xeon W5580 CPU (4 cores, 8 MB cache, 3.20 GHz)
• 24 GB of DDR3 RAM
• Intel “Niantic” 82599EB 10 Gb Ethernet controller
• Microsoft Windows Server 2008 R2 x64

This combination would set you back about $7,500 – $450 for the motherboard, $1,500 for the CPU, 6 2 GB DDR3 SDRAM modules at $80 each, $1,200 for the Intel X520 NIC, and $4,000 for an Enterprise copy of Windows Server 2008 R2. Not cheap, but not an exotic server either.

Initiate and Optimize

The secret to push the tested system to perform like it did is in the optimizations in the server platform, the NIC, and Windows Server itself.

• The Xeon 5500 processor series includes many enhancements:
  • An integrated memory controller allows for faster RAM access
  • QuickPath interconnect (QPI) replaces the old front-side bus and enhances I/O off the core
  • A new I/O subsystem with PCIe integrated into the CPU
  • MSI-X expands the number of interrupts a PCI device can use
  • New instructions for on-board CRC-32C decoding, speeding up iSCSI digest processing
• The 82599 Ethernet controller also includes enhanced capabilities:
  • VMDq maps I/O queues to multiple cores and virtual machines, reducing I/O bottlenecks
  • Offload of TCP segmentation and receive-side coalescing
  • Interestingly, it does not appear that VMDc/SR-IOV was employed in the test
• Microsoft Windows Server 2008 R2 and Hyper-V are ready to use all of these features and more:
  • R2 uses multi-core CPUs more effectively in general
  • Receive-side scaling (RSS) spreads the I/O workload across all four Xeon cores
  • The iSCSI initiator now allows CRC digest offload (using the new Xeon command set)
  • Numerous “NUMA I/O” optimizations in the initiator
  • TCP/IP Nagle can be disabled in the registry
  • Hyper-V VMQ allows the network packets to be copied directly into the guest virtual machine’s memory

Whew! Put all of these optimizations in a blender and Hyper-V virtual machine iSCSI access will be twice as fast as before. No kidding!

Stay On Target

But we knew all of this back in January. We also saw that a Cisco Nexus 5020 switch was used to fan out to 10 software iSCSI targets. But until now there was no mention of what targets were used exactly.

The final footnotes in Intel’s whitepaper reveals that the storage backing the million IOPS test was none other than StarWind Software’s iSCSI SAN! It is unclear what led Microsoft and Intel to use this particular iSCSI target (the earlier throughput tests ran on NetApp filers), but it does speak to the quality of this product.

It is not clear how many disk drives were used, but I would guess that SSDs or ramdisks might have been employed to pull a million IOPS. Network optimizations are also not mentioned, though jumbo frames would not be a benefit in an IOPS test.

StarWind’s software runs on Microsoft Windows and creates a full-featured iSCSI target, complete with data mirroring, automatic failover and failback, replication, snapshots, and thin provisioning. The company prices their iSCSI SAN at $6,000 for two nodes and competes with the likes of DataCore and Open-E. But the StarWind solution seems at a glance to be more full-featured than these other offerings.

Try It Yourself!

I imagine many folks like me might be tempted to try to reproduce these results. More valuable would be a set of best practice guidelines for the deployment of software iSCSI in Windows Server 2008 R2 and Hyper-V environments. Given the relatively modest hardware involved, there should be nothing stopping us!

These test results also prompted me to get in touch with StarWind to try their iSCSI target software. I was pleasantly surprised to learn that they are currently offering free non-production licenses to VMware vExperts, VCPs, and VCIs as well as Microsoft MVPs, MCPs, and MCT Professionals. Many of my readers fall into one (or more) of those buckets, and I applaud the company for this offer. If only more companies realized the value in giving away test licenses to influencers and thought leaders!

© Stephen Foskett for Gestalt IT, 2010. | How Did Microsoft and Intel Get 1 Million iSCSI IOPS?
Read more posts categorized as Server Virtualization, Storage

How fast can iSCSI get?

“Maximizing Hyper-V iSCSI Performance with Microsoft and Intel” might sound like another “blah blah” marketing piece, but a little birdy tells me that this webcast will drop a bombshell about iSCSI performance.

Lots of storage and networking folks don’t give iSCSI and Microsoft the credit they deserve. “iSCSI is cheap and easy,” they say, “but real performance requires Fibre Channel.” Those of us who have an open mind about such things know that this is simply not the case. The fastest SAN I ever saw was based on iSCSI, and Microsoft demonstrated wire-speed iSCSI over 10 Gb Ethernet in March. I never saw a Fibre Channel SAN (even an 8 Gb/s one) push over a gigabyte per second over a single link!

Still, ask the average sysadmin and they will tell you that iSCSI isn’t for high performance applications. That’s why folks should tune in to this webcast, as Microsoft and Intel knock down another iSCSI performance myth. Note that even though Hyper-V is called out in the title and description, this discussion is really about Windows Server 2008 R2 and applies equally regardless of whether or not you use Microsoft’s hypervisor.

Watch this space for a summary of the news immediately following the announcement.

• What: Maximizing Hyper-V iSCSI Performance with Microsoft and Intel webcast
• When: Thursday, January 14, 2010 8:00 AM Pacific Time
• Where: MSEvents.Microsoft.com
• Who: Anyone interested in high-performance storage and server I/O

© sfoskett for Stephen Foskett, Pack Rat, 2010. |
Microsoft and Intel Pushing iSCSI Performance Limits


This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2010. | Microsoft and Intel Pushing iSCSI Performance Limits
Read more posts categorized as Server Virtualization, Storage