Do we really need parallel NFS?

Did some pNFS proponent slip a love potion into the coffee at EMC? Suddenly it’s pNFS time at the company known for its reluctance to embrace file sharing and filesystems in general. The purple prose is flying, with Chad Sakac declaring himself “a big fan of the application of NFS” and Chuck Hollis extolling the “inherent simplicity and ease-of-management of NFS.” The NetApp guys must be amused by the bear hug from Hopkinton, but many are seeing deja-vu all over again.

Chad’s Icky Bits

(Apologies for that heading, but those are Chad’s words, not mine)

Chad Sakac’s red rose for pNFS included a few thorns aimed at good old NFSv3. He calls these the “icky bits” and spills some ink over them:

1. “NFS Server failure behavior,” says Chad, leads to issues as serious as “a guest OS crash” and administrators “resorting to unnatural acts” to compensate. He talks about EMC’s DART OS being optimized to fail over in under a minute to avoid application issues and the difficulty in actually accomplishing this feat.
2. Chad also points out that “NFS client limitations” can lead to “unexpected bottlenecks.” Load balancing large workloads across multiple gigabit Ethernet NICs means hand-tuning, since NFS pins traffic to a single MAC address.

Certainly these limitations were known to many in the storage industry, but haven’t they also been addressed repeatedly? NetApp, EMC, and BlueArc do indeed suggest adjusting NFS heartbeat values to allow time for the cluster to recover, but this seems more a limitation of their clustered server architecture than of NFS itself. Scale-out NFS servers from Isilon and HP don’t seem to require these “unnatural acts.”

As for client limitations, manually balancing client loads is a reality in many large storage architectures, not just NFS. Perhaps the fact that NFS can handle so many more I/O requests in a given timeslice makes this more of an issue, but it tends to be transient.

Chad has repeatedly expressed his love for NFS, especially as a datastore for VMware. Clearly, he intended to point out these “icky bits” to highlight the possibilities for pNFS. But the method used (calling them “icky” for one) resembles mud slinging.

Chuck Wants pNFS

(Chuck’s titles also lend themselves to mis-reading)

Chuck Hollis is more careful in his wording, extolling the virtues of pNFS without calling anything “icky”. Indeed, there’s just one NetApp dig: He says their “emulated containers of LUNs” are “hardly optimized”, which is a welcome change of tone from previous debates.

But the underlying message is the same: pNFS is new and wonderful, encouraging proliferation of hand-holding, flower distribution, and rainbows. Again I ask, is this really true? Is pNFS ready for this kind of adulation when, as Chuck points out, “it’s going to take a while before the rest of the portfolio, industry and ecosystem catches up.  Maybe a year or so.”

Seriously? A year until pNFS is ready for mass enterprise adoption? Admittedly, EMC has been working on pNFS (as MPFS) for a long time, but predictions of “just another year” for a major protocol transition set off warning bells. This is doubly true when most clients (including VMware) don’t yet offer even basic support.

Stephen’s Stance

One wonders if airing this dirty laundry is an attempt to highlight EMC’s pNFS work or to discredit plain old NFS as a datacenter protocol. As I wrote about in Our New Thing Is Awesome (‘Cause Our Old Thing Sucked), the “parade of progress” sometimes degenerates into “out with the old,” and this is perilous for purveyors of durable goods like storage systems.

I am also very concerned with the proliferation of “layout types” within pNFS. It seems that every vendor has a hand in the protocol, and each is adding their own technology to the mix. We started with files and now have both objects and blocks. Will these be widely supported? Do we really need them? Or will pNFS start looking like Bluetooth: Bloated, incompletely-implemented, and ignored except for special use cases.

But my motivation behind this post is simpler than that. I would like to pose a question: Is NFS (v3) really that “icky”? Do we really need pNFS? Or have these problems been solved previously?

© Stephen Foskett for Gestalt IT, 2010. | Is NFS v3 Really That Bad?
Read more posts categorized as All, Storage

Those HP StorageWorks guys have a lot of sounds-good-but-sells-nothing crap in their locker. Remember RISS? That’s the role the ExDS9100 whatever fancy box is doing now. Then he’s got to somehow knock sense into the PolyServe takeover (PolyCurve? No, forget that one) and Ibrix. We really kicked everyone’s NAS butt with Celerra. But EVA, damn it, EVA 2 could be good.

We got a unified stack with Cisco and VMware, but Cisco and servers, I don’t know. How good is Chambers really, I mean really really? He’s spread himself awful thin over the thousand and one things Cisco is doing and, I know, Cisco is Cisco. But Chambers isn’t going to be there for ever, and Hurd is probably going to outlive him.

If the Cisco servers don’t really hack it (and, you know, they may not), it’s not like Dell, HP and IBM are wallflowers at the server dance. Then there’s that Acer guy that says we’re all dinosaurs. He’s nibbling at the foursome’s heels and we need a backstop. Okay, put that off to one side for a minute. Let’s think about growth.

We’re a $53.4 billion market capitalization company. Ok, $36.6 billion if you strip out VMware. How are we going to gow more? I says to my guys, “look, there’s no more real growth in the storage business. To do something big there we gotta buy HDS or NetApp and both would give us a world of hurt in integrating their products, especially NetApp’s, and there’s a real chance the sum of the parts could be less than the sum of the separate pieces. Let’s not even go there.”

The information infrastructure stuff is growing. Hollis keeps yacking at me (even more than normal) about this. It’s two or three times a week now! Doesn’t that guy ever sleep or eat or play golf? But I tell him, “that stuff is organic and that means slow and we need something quicker than twenty years from now.” I’ll be long gone then. I want something big on my crown. I don’t wanna slink off like Warmenhoven, always thinking I coulda been a contender.

This is getting kinda long, so I’ll just cut it off here. But I got an idea that’ll kill you. Seriously, it’ll blow you away. But first I gotta make some calls and set some stuff in motion. Maybe I’ll tell you about it on Monday or something.

© Don Joey for Gestalt IT, 2010. | What’s Keeping Me Up At Night
Read more posts categorized as All, Industry Confidential

Why all the interest? Data Domain has emerged as the leader in data deduplication, a key technology for controlling the rapid growth of duplicate data in enterprise IT. Although both EMC and NetApp offer deduplication products, Data Domain’s products were widely lauded and adopted by end users, especially in smaller organizations.

Although many questions were raised about the fit for Data Domain within NetApp’s product lineup, the fit in EMC makes more sense. Their Quantum-based VTLs are expensive and enterprise-focused, while a new Data Domain-powered line might have broader appeal. Although EMC appears to have cemented their Quantum relationship with a $100 million investment to keep the backup products vendor afloat, the fact that they did not buy the company outright might indicate that they had second thoughts. EMC’s Chuck Hollis claims that Data Domain makes sense at many levels throughout EMC’s product portfolio.

Others have suggested that the offer was purely defensive, designed to keep Data Domain out of NetApp’s hands or to drive up the eventual price the West coast team would eventually have to pay. This would be a very expensive gamble, however, since NetApp already has homegrown deduplication technology and could walk away entirely.

© Stephen Foskett for Gestalt IT, 2009. | EMC Takes On NetApp For Data Domain’s Affections
Read more posts categorized as Exclusive, Featured, Storage

Hiding in the shadow of the huge VMware vSphere 4 announcement was a very interesting introduction by EMC: PowerPath/VE. As I mentioned in my post on storage changes in vSphere 4, PowerPath/VE plugs into the new pluggable storage architecture (PSA) found in vSphere 4 versions of ESX and takes over the decision-making and heavy-lifting tasks related to communicating with storage systems.

Driving Massive I/O

Chuck Hollis treated us to a discussion of vSphere as an I/O Engine on his blog this morning with some background on multipath IO (MPIO for short), but I’m not sure he did the topic justice. In my opinion, server virtualization is the greatest I/O driver ever brought into the data center, and it messes with all of our preconceived notions about I/O at the same time.

What’s so special about server virtualization?

1. Hypervisors concentrate I/O, shifting loads that were formerly distributed to a large number of I/O channels into a far fewer channels. Picture 10 servers doing what they do. Now put all 10 in a single physical box. All of their storage access must now share a bus, a host adapter, a cable, and perhaps a LUN on the storage system. It’s the difference between lemonade and lemon juice!
2. Hypervisors randomize I/O, chunking everything up and mixing it together. Forget about the carefully-designed read-ahead algorithms and caching used in enterprise storage – VMware, Hyper-V and the rest throw those expectations out the window! Virtualization is a blender – it grinds up your lemons, skin, seeds, and all!
3. Hypervisors demand low I/O latency, forcing infrastructure to get quicker, not just faster. This is one reason that caching, solid state disks, and 10 GbE are going to be huge in virtual environments – all reduce latency by orders of magnitude! As any car guy will tell you, quick and fast are two very different things!

The upshot of all of this is that virtual servers are very very hard to satisfy when it comes to I/O. And the “back end” has always been a bit of a bottleneck for virtualization software. Now we have VMware claiming that vSphere 4 can push over 300,000 I/O operations per second (IOPS) without resorting to VMDirectPath and similar “cheater” measures. Of course not all IOPS are equal, and I doubt that that 300k number would hold up with a real-world workload, but it’s impressive nonetheless!

A Brief History of MPIO

Let’s turn back to multipath I/O. PowerPath/VE is just the latest in a long line of path managers, not all of which have been well-loved. Back in my HP-UX days I learned to make the most of PVlinks, the native path management on that operating system. It wasn’t always easy to get it to work well, but it sure was nice to have a path manager built into the operating system! Veritas also offered a multi-platform path manager, DMP, which worked with a variety of array types. Back in the day, both were limited to simple failover and lacked the “intelligence” to deal with the peculiarities of the weird storage arrays we learned to not hate.

Array-specific path managers from storage vendors were much more successful. CLARiiONs used ATF, Hitachi arrays used HDLM, IBM had SDD, and of course EMC had PowerPath. EMC introduced PowerPath in 1997, the software reportedly having been developed by Conley Corporation, which EMC acquired the next year and turned into its Cambridge (MA) development center. After acquiring Data General, EMC adapted PowerPath to support CLARiiON, pushing ATF off stage right. Then they kept right on developing the software, adding support for IBM, HDS, and HP arrays and data migration.

Meanwhile, Microsoft decided that HP and Veritas were on to something when they developed standard path management software, so they began working on a standard multi-path IO (MPIO) driver for Windows. But Microsoft learned a thing or two from the mediocre device support in those old solutions, so they decided to allow vendors to plug their own smarts into the standard Windows Server 2000/2003 MPIO framework. Microsoft provided basic failover capability and third parties, including EMC, wrote their own device-specific modules (DSMs). This MPIO support evolved and spread, standard on Microsoft’s iSCSI initiator and Hyper-V virtualization platform. PowerPath 5.2.1 for Windows already supported Hyper-V thanks to this.

PowerPath and VMware PSA

VMware also learned a thing or two from HP and Microsoft. Although basic path failover support has been included in ESX for years, vSphere 4 takes it to a new level with pluggable storage architecture (PSA). Every version of ESX 4 includes native multipathing (NMP), but Enterprise Plus licensees can use vendor-supplied plugins to enable more advanced path management. As I noted on Tuesday, there are two three different levels of path selection: Basic path-selection plugins (PSPs), more advanced storage array type plugins (SATPs), and complete multi-path plugins (MPPs).

This is what EMC has introduced: An MPP for vSphere 4 called PowerPath/VE. Like the DSM for Windows MPIO, PowerPath/VE for vSphere slots right into an existing MPIO framework and enables advanced path selection and load balancing without mucking with the internals of the hypervisor. PowerPath/VE has all sorts of smarts in it. It has eight different predictive load balancing policies, proactive disconnect, bus testing, and HBA monitoring.

Super VMware guy Chad Sakac described PowerPath/VE as part of the launch. He notes that EMC is first out of the gate with a multipathing plugin for vSphere, but I suspect that just about every vendor will release similar functionality pretty quickly. In particular I expect support to come from NetApp and 3PAR, since they’re so interested in VMware support.

Licensing Questions

One thing really stuck out in the vSphere launch: PSA is only included in the top-of-the-line Enterprise Plus license. Presumably, this means that, in addition to paying for a PowerPath/VE license, users will have to spring for maximum ESX, too. This is a dumb move, if you ask me. Microsoft made MPIO successful by giving it away with every copy of Windows. They even included it in the free iSCSI initiator download. VMware, in contrast, seems to be actively limiting PSA’s usefulness to the top tier of users. If it was up to me, I would set the VMware MPIO free!

I’m working with EMC and VMware to determine the extent of the NMP/PSA/PowerPath licensing mess. I’ll update this post as I find out the answers!

1. Does every edition of ESX 4 include the basic VMware native multipathing (NMP)?
2. Can one use a vendor-supplied PSA plugin like PowerPath/VE without an enterprise plus license?
3. Does it matter (to licensing) if the plugin is a PSP or an SATP?
4. If “no” to 2 or 3, can PSA be added separately without the plus license if someone wants to use something like PowerPath/VE?

Update: I received a nice email from an EMC engineer correcting me about the plugin types. This kind of open communication is why the web is so great! It turns out that PowerPath/VE is a sort of super plugin called an MPP, not “just” an SATP or PSP. I’ve updated the section above!

© sfoskett for Stephen Foskett, Pack Rat, 2009. |
PowerPath To The Virtual People


This post was categorized as Computer history, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2009. | PowerPath To The Virtual People
Read more posts categorized as Server Virtualization, Storage