“Bridging versus routing” brings us to the perennial networking debate: Are low-level protocols better?

Unless you’re “in the know”, terms like “layer 2″ can seem mysterious, making it all the more plausible when someone touts the benefits. It seems logical: “Bare-metal” communication must be better, faster, and cheaper than higher-level “everything over IP” approaches, right? But it’s not quite that simple.

This piece assumes you know something about the OSI model.

Ivan Pepelnjak posted a great overview of the “bridging versus routing” debate (Bridging and Routing: is there a difference?), and Marko Milivojevic posed the question in response: “I’m one of those who doesn’t understand the whole L2 obsession of the modern networking world, but…”

It really is an obsession: Data communications folks continually argue about the merits and trade-offs between high-level network protocols and low-level communications. We hear it in storage all the time: FCoE proponents assume performance benefits, and AoE fans add cost advantages to the mix. But many of these claims are unsubstantiated, and iSCSI and NAS protocols like SMB and NFS just keep rolling forward. If storage over IP is so bad, why does iSCSI work (and perform) so well?

One thing often missing in the “layer 2″ arguments is what’s missing when you skip the network layer. There’s a reason IP is so widespread: It may not be the best protocol ever, but it works really well in a huge variety of situations and there is a vast pool of associated technologies that can be drawn upon when using it.

IP can run over just about anything, from FireWire to SONET, so any protocol using IP can (theoretically) run there, too. I’ve run iSCSI over Wi-Fi and WAN links, and it works great out of the box with 10 Gb Ethernet. Protocols that are tightly linked to a layer-2 protocol face stiff challenges when moving to different data links. Witness the difficulty moving Fibre Channel to 10 Gb Ethernet, including all those data center bridging technologies. In fact, when faced with the challenge of long-distance Fibre Channel SAN communication, encapsulation over IP was a natural choice.

IP also has a myriad of wonderful technologies to choose from. The creators of iSCSI were able to pull authentication, encryption, lossless communication, and many other features straight from the existing toy box. Developers of new non-IP protocols have to invent their own solutions to these problems, often with disastrous results. Why reinvent the wheel? Just apply a little CHAP, some IPsec, and roll it in TCP and you’re done!

Photo credit: “Akashi Kaikyo Bridge 明石海峽大橋” by Shenghung Lin

© Stephen Foskett for Gestalt IT, 2010. | The Lure of Layer 2
Read more posts categorized as Networking, Storage

Power

If you are using copper patch leads for 10GbE, you are going to a need a lot of power. Using standard copper can use up to 45W per port (although 10GBaseCX-4 apparently uses 4.5W per port).

The IEEE is working on Energy Efficient Ethernet (802.3az) technology that will allow links to auto-negotiate down to lower speeds or go to “sleep’ during periods of inactivity which will further reduce power consumption.

Cabling

10GBaseT copper uses 650Mhz frequency spectrum and needs high quality cabling to work reliably. This means that you need to test, properly, your existing Cat5 or replace it with Cat6A or better. If you use Cat6A or even Cat6, the cable is physically much larger (and you may not have the space in your computer room). In this case, you will have up to 100 metre cable length. If you use Cat5 or Cat5e, the distance is much shorter depending on the quality of your cable, typically less than 40 metres and would probably need testing for assured reliability.

10GBaseSR uses multimode cabling but has different cable lengths depending on type of cabling and certain combinations will require mode conditioning patch leads.

The current list of 10GB XENPAK / X2 interfaces from Cisco show the confusion that the different types of cabling causes. For example consider the following table showing the LAN options (I’ve removed the WAN units) and the variation in cabling types:

The impact of cabling

In a recent project to plan a refit of an existing data centre, the 10GbE cabling needs was a major problem. Because of constraints in the change control and risk management, we eventually decided to use 1Gb ethernet because the time needed to get long change windows exceeded the length of the project.

And in other projects, the cost of recabling the fibre optic to meet the new requirement for 10GbE was prohibitive for smaller works. That is, we couldn’t just add a “patch of green” to an existing facility and extend the new switch as funds became available.

Which is weird, because it reminds me of the Token Ring / FDDI / Ethernet wars back in 1995 or so.

High Cost

If you take the time build budgetary pricing around a Cisco Nexus 7000 you will quickly realise that the cost of 10GbE capable switch is really expensive. I found that a typically configured Nexus 7018 with a good number of 10GbE and some 1GbE was around GBP£500K / USD$800K. Admittedly, this was a fully loaded model but forms the basis for a cost analysis against our existing Cat6500 choices. Frankly, I couldn’t convince anyone that this was a good idea.

Sure, the Nexus 7000 is good product (not a great product in my opinion) and offers some 10GbE capability but the lack of features and high cost means that 10GbE is still not a part of our short term strategy. I wonder how many other people have a similar problem ?

Which Year was that ?

It seems that every year is the year of 10 Gigabit Ethernet.

2009

In March 2009 The Register posted an article
Intel dubs 2009 ‘The Year of 10Gb Ethernet’.

2008

Silvano Gai, in his book Data Center Networks and Fibre Channel over Ethernet, Section 1.5.2 claimed “2008 is the year of 10GE”.

2004

Amd as far back as 2004, people have been claiming that “this is the year ….”

The market for 10-Gigabit Ethernet might start to show some signs of life in 2004. An economy gone cold, combined with its comparatively high price tag, kept down shipments of 10-GigE ports to a few thousand last year. But vendors predict that the 10-Gbps version of Ethernet will continue along a familiar trajectory. “This will be the year of 10-Gigabit Ethernet,” declared Richard Brand, director of network architecture for Nortel’s advanced technology group

Highbeam – Note this is behind a registration wall

And this article on CNET (really) 10-Gigabit Ethernet comes alive

The market for 10-gigabit-per-second Ethernet switching got off to a slow start, but now that corporate customers are looking for more speed on their networks, the technology seems to be hitting its stride.

And the fact ?

10GbE hasn’t really happened has it ? The standards took a long time to finish, and the prices have been very high for both the cabling and switching equipment. Server manufacturers aren’t putting the chips on their motherboards because of high power consumption. But most importantly, almost no-one need the bandwidth except for certain niche applications.

My Prediction

There is no question that 10Gigabit Ethernet is going to happen. Eventually. But there isn’t enough money or momentum to make 2010 a watershed year. There still isn’t enough demand for bandwidth in most parts of the networks to require the upgrade and CIO’s are investing in Virtualisation this year, not Networking.

That said, areas that require long investment cycles may buy 10Gb Ethernet for preemption against future upgrades (and thus downtime) will drive a surge in 10GbE purchases this year. For example, upgrades to data centres and storage networks (for those using iSCSI and FCoE) may purchase 10GbE switches and routers to build high performance backbones but continue to connect servers and edge switches at 1GbE. These are high visibility, high value purchases that will create a lot of marketing noise and management attention. The reality, however, is that 10GbE will be adopted small scale, and will not be used in the distribution switches, or the wiring closet, or the WAN. ((Note that it’s different for Service Providers whom I expect will have a lot more interest in 10GbE for their WAN backbones and may actually make investments in their backbones soon.))

I’m expecting 10GbE to get slow, progressive adoption over the next three years. It’s not an industry revolution, and not enough people need to increase bandwidth to drive rapid. The only use for 10GbE in the Enterprise is for Data Centres where Storage and Virtualization/Blade Servers are driving adoption.

No one else cares.

© Etherealmind for Gestalt IT, 2010. | Is this the year of 10 Gigabit Ethernet in the LAN?
Read more posts categorized as Featured, Networking

How fast can iSCSI get?

“Maximizing Hyper-V iSCSI Performance with Microsoft and Intel” might sound like another “blah blah” marketing piece, but a little birdy tells me that this webcast will drop a bombshell about iSCSI performance.

Lots of storage and networking folks don’t give iSCSI and Microsoft the credit they deserve. “iSCSI is cheap and easy,” they say, “but real performance requires Fibre Channel.” Those of us who have an open mind about such things know that this is simply not the case. The fastest SAN I ever saw was based on iSCSI, and Microsoft demonstrated wire-speed iSCSI over 10 Gb Ethernet in March. I never saw a Fibre Channel SAN (even an 8 Gb/s one) push over a gigabyte per second over a single link!

Still, ask the average sysadmin and they will tell you that iSCSI isn’t for high performance applications. That’s why folks should tune in to this webcast, as Microsoft and Intel knock down another iSCSI performance myth. Note that even though Hyper-V is called out in the title and description, this discussion is really about Windows Server 2008 R2 and applies equally regardless of whether or not you use Microsoft’s hypervisor.

Watch this space for a summary of the news immediately following the announcement.

• What: Maximizing Hyper-V iSCSI Performance with Microsoft and Intel webcast
• When: Thursday, January 14, 2010 8:00 AM Pacific Time
• Where: MSEvents.Microsoft.com
• Who: Anyone interested in high-performance storage and server I/O

© sfoskett for Stephen Foskett, Pack Rat, 2010. |
Microsoft and Intel Pushing iSCSI Performance Limits


This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2010. | Microsoft and Intel Pushing iSCSI Performance Limits
Read more posts categorized as Server Virtualization, Storage

NetApp and Microsoft saturated a 10 Gb Ethernet link - can everyone else?

Along with many tidbits about storage advances in Windows Server 2008 and 2008 R2, this WinHEC presentation by Microsoft’s Suzanne Morgan demonstrated that the combination of the Windows iSCSI Initiator and NetApp FAS 3070 filer could saturate a 10 Gb Ethernet link. How many other storage arrays can do that?

The configuration tested was an IBM x3550 with dual 2 GHz CPUs, 4 GB of RAM, and an Intel 82598 adapter. This is not a special server – in fact, it’s pretty low-end! The connection was tuned with RSS, NetDMA, LRO, LSO, and jumbo frames and maxed out over 4 MCS connections per second. I’m not sure what kind of access they were doing (I’ll ask Suzanne), but it’s pretty impressive that the NetApp Filer could push 1,174 megabytes per second!

So I have a simple question for all of the storage vendors in the house: Can your array saturate a 10 GbE link? I am not asking for a complex performance test or maximum performance claims, just a simple yes or no, along with at least a bit of supporting data about the configuration and performance. Maybe duplicate the setup from that Microsoft slide.

I am not trying to suggest that this kind of performance is impossible, that NetApp is unique or special, or to trap anyone into a blog war. I am just curious about the performance capabilities of the various iSCSI targets out there!

© Stephen Foskett for Gestalt IT, 2009. | Wire-Speed 10 Gb iSCSI, Anyone?
Read more posts categorized as All, Exclusive, Featured, Storage, Top Story