In this blog entry I describe the current state of path management and outline some recommendations for the use/implementation of path management software in the data center.

Background

The history of path management is very wrapped up with the history of the storage array vendors. Prior to the advent of PowerPath from EMC, path management was very much a part of the operating system. OS’s such as IBM’s VM, DEC’s VMS, and other mainframe and mini-computer operating systems all had the ability to communicate with their storage via more than one path and to load balance the I/O across those paths. However, when EMC and other storage vendors started to move into the open systems world, where the OS’s had a “small system” background, they found that path management was not something offered by the operating system.

Early version of MS Windows, and various flavors of UNIX all had no way to address storage across more than one path, or if they did, it was simply a failover path without the ability to load-balance the I/Os. In response to this situation EMC developed PowerPath, and other storage vendors such as Hitachi soon followed suit. At that time, the path management software developed by the storage vendors only supported that storage vendor’s particular arrays. In other words EMC’s PowerPath only supported EMC arrays, Hitachi’s HDLM software only supported Hitachi arrays, and so forth.

While this allowed a customer to optimize the connections between their hosts and their storage, it also had the effect of locking the customer into a single vendors arrays simply because it became very difficult to support more than one vendor’s array on the same host, and switching vendors also became more difficult by adding a significant level of effort to the migration process since all of the path management software all had to be replaced as well.

This situation remained the same for a number of years, until EMC announced support for non-EMC vendors in their PowerPath product. This announcement was a part of EMC’s plans at the time to move from a pure hardware company to a more software driven business. Along with the announcement of PowerPath for other storage vendors, EMC also announced a set of APIs that would allow management of non-EMC arrays from their flagship Control Center product and several other smaller changes to help position EMC as a software company.

Unfortunately, these changes were never fully recognized in the EMC software, nor were the EMC Sales teams particularly enthusiastic about the move away from a focus on “big iron” hardware that they had made so much money with for such a long time. This left some of the EMC products, such as PowerPath, in a position where they had some support for other vendors arrays, but that support was not complete from either an array specific feature perspective, or in the case of PowerPath, from a array vendor perspective. For example, aside from support for their own arrays, EMC PowerPath provides support for HDS (and some of the OEMs of HDS products such as HP) and some IBM arrays as well. However, support for other significant array vendors in the market such as 3PAR, Compellant, NetApp, etc. is notably missing. As a matter of fact, EMC has not added support for any array vendor since their original announcement in 2003 of support for HDS, IBM, and HP.

Things began to change when Microsoft introduced MPIO as part of the Windows operating system starting with the Windows 2000/2003 versions of the software. Microsoft, having learned from those that went before them, decided to provide a standardized mechanism for path management, but at the same time they also allow the storage array vendors to provide a plug-in (call a DSM) if they wanted to add additional capabilities for path management to MPIO.

SUN developed it’s version of MPIO, called MPxIO in 2003, IBM introduced it’s version of MPIO in 2002, Linux announced Device Mapper in 2005, and the last vendor to announce MPIO software as part of the operating system was HP which announced their version of MPIO in 2007. Note that HP had a non-OS integrated version of path management called PVLinks available since the early 1990s. Therefore, by 2005 virtually every operating system in use in the data center had path management built in and the need for array vendor supported software simply no longer existed in order to provide path management.

At the same time as all of the above was happening, a company called VERITAS as part of their VERIAS Volume Manager was developing one true independent piece of path management software. VERITAS was positioning itself as an OS and array independent storage management company, and therefore it developed it’s own suite of tools for volume management, a file system, and path management software called DMP (Dynamic Multi Pathing). DMP has had its issues over the years, but was particularly popular with SUN customers, if for no other reason than it came with Foundation Suite which was very popular with SUN Solaris customers.

VMware Path Management

All of the above addresses path management purely from a host and storage array perspective. However, with the introduction of VMware another player entered the path management landscape. From the beginning, VMware required users to utilize the path management tat was built into VMware. The path management software built into VMware 3.5 and older provided basic path management feature, specifically path failover, but did not provide load balancing or any array specific features. This provided users of VMware some ability to tolerate path outages, but defiantly limited VMware’s ability to provide for high I/O applications. This limit wasn’t the only reason that early versions of VMware didn’t support high I/O applications, but it certainly needed to be addressed should VMware ever want to be able to support these high I/O applications.

With the introduction of vSphere, VMware has finally addressed this issue, and more. Much like with MPIO VMware has now introduced a mechanism for storage array vendors to provide a “plug-in” into the path management functions built into vSphere called a storage array type plug-in (SATP) for the new Native Multipathing Plugins (NMP) module. One of the first vendors to take advantage of this capability was EMC with their PowerPath/VE product which provides support for EMC arrays.

Recommendations for Current Approach

There currently is no panacea when it comes to picking an approach to path management in the data center. Once again it boils down to two philosophies. Do you want to try to utilize a single path management software, or do you want to utilize the path management software that is built into the operating system?

Option #1 – Utilize a single path management software

For this option, a single piece of software is chosen and then loaded on every host to provide path management. There are really only two options for this software. You can utilize Symantec (once VERITAS) DMP, or you can use EMC PowerPath. These are the only two products in wide distribution that provide multi-vendor array support and a wide variety of host support as well.

• PROs: The main advantage to utilizing a single piece of software is management of the path management software. You have a single product that is well understood provided by a single vendor to manage. Theoretically this should reduce your management costs, and provide for a more reliable and stable environment.
• CONs: The down side to this approach is the cost involved. Since the software is a purchased product, a license for every host in the datacenter must be purchased from the vendor, or some kind of enterprise license obtained. In either case, tracking of the software licenses, and general management of the software often offsets the cost benefits of having a single vendor for path management. The second major disadvantage of utilizing a single path management software product is that you are locked into the list of supported hosts and arrays that the vendor chooses to provide. In the case of PowerPath the list of non-EMC arrays is very short, and appears to show no signs of ever growing any larger. In the case of DMP, there is also a question of where that product is heading, and how much development in the form of additional array support Symantec intends to provide. This creates a situation where the path management policy can prevent the organization from purchasing a new array that might provide significant cost benefit advantages. Finally, the management of versions of the path management software, the switch firmware, the operating system, and the array firmware create a complex matrix and increase the cost of support significantly. It can also slow down the rollout of newer models of arrays, switches, and host operating systems delaying cost saving.

Option #2 – Utilize the path management software built into the operating system

For this option, the built in path management software supplied with the operating system is utilized, along with the DSM (where available) to provide path management for all of the arrays in the datacenter.

• PROs: With this approach dependence on third party software to provide path management is eliminated, and the support matrix for the host OS, switch and array is reduced making support much more straightforward. With the addition of a DSM to provide array specific features, all of the capabilities of the array are made available, without the need for third part software or vendor lock in. The operating system vendor, with whom a close support relationship already exists, provides support for the path management software reducing the possibility of three way finger pointing in the case of a problem. Finally, the ability to support any array which you feel provides you with a business advantage in terms of saving costs, providing additional features, and/or additional speed is once again on the table rather than the short list of arrays supported by the third party path management software vendors.
• CONs: Since each operating system provides it’s own path management software some additional learning and understanding of those different path management software products would need to be maintained by the appropriate support team (storage or OS). This could add some additional costs in terms of training and support.

Recommendations

At this point in time, it is my recommendation that path management software integrated with the operating system be utilized rather than third party software. I believe that the flexibility to utilize nearly any array on the market today to address any storage issues you might face without being locked into a particular vendor, or even a short list of vendors, far outweighs any minor added overhead involved with learning multiple path management software interfaces.

© Joerg Hallbauer for Gestalt IT, 2010. | Path Management Software Recommendations
Read more posts categorized as Featured, Server Virtualization, Storage

Hiding in the shadow of the huge VMware vSphere 4 announcement was a very interesting introduction by EMC: PowerPath/VE. As I mentioned in my post on storage changes in vSphere 4, PowerPath/VE plugs into the new pluggable storage architecture (PSA) found in vSphere 4 versions of ESX and takes over the decision-making and heavy-lifting tasks related to communicating with storage systems.

Driving Massive I/O

Chuck Hollis treated us to a discussion of vSphere as an I/O Engine on his blog this morning with some background on multipath IO (MPIO for short), but I’m not sure he did the topic justice. In my opinion, server virtualization is the greatest I/O driver ever brought into the data center, and it messes with all of our preconceived notions about I/O at the same time.

What’s so special about server virtualization?

1. Hypervisors concentrate I/O, shifting loads that were formerly distributed to a large number of I/O channels into a far fewer channels. Picture 10 servers doing what they do. Now put all 10 in a single physical box. All of their storage access must now share a bus, a host adapter, a cable, and perhaps a LUN on the storage system. It’s the difference between lemonade and lemon juice!
2. Hypervisors randomize I/O, chunking everything up and mixing it together. Forget about the carefully-designed read-ahead algorithms and caching used in enterprise storage – VMware, Hyper-V and the rest throw those expectations out the window! Virtualization is a blender – it grinds up your lemons, skin, seeds, and all!
3. Hypervisors demand low I/O latency, forcing infrastructure to get quicker, not just faster. This is one reason that caching, solid state disks, and 10 GbE are going to be huge in virtual environments – all reduce latency by orders of magnitude! As any car guy will tell you, quick and fast are two very different things!

The upshot of all of this is that virtual servers are very very hard to satisfy when it comes to I/O. And the “back end” has always been a bit of a bottleneck for virtualization software. Now we have VMware claiming that vSphere 4 can push over 300,000 I/O operations per second (IOPS) without resorting to VMDirectPath and similar “cheater” measures. Of course not all IOPS are equal, and I doubt that that 300k number would hold up with a real-world workload, but it’s impressive nonetheless!

A Brief History of MPIO

Let’s turn back to multipath I/O. PowerPath/VE is just the latest in a long line of path managers, not all of which have been well-loved. Back in my HP-UX days I learned to make the most of PVlinks, the native path management on that operating system. It wasn’t always easy to get it to work well, but it sure was nice to have a path manager built into the operating system! Veritas also offered a multi-platform path manager, DMP, which worked with a variety of array types. Back in the day, both were limited to simple failover and lacked the “intelligence” to deal with the peculiarities of the weird storage arrays we learned to not hate.

Array-specific path managers from storage vendors were much more successful. CLARiiONs used ATF, Hitachi arrays used HDLM, IBM had SDD, and of course EMC had PowerPath. EMC introduced PowerPath in 1997, the software reportedly having been developed by Conley Corporation, which EMC acquired the next year and turned into its Cambridge (MA) development center. After acquiring Data General, EMC adapted PowerPath to support CLARiiON, pushing ATF off stage right. Then they kept right on developing the software, adding support for IBM, HDS, and HP arrays and data migration.

Meanwhile, Microsoft decided that HP and Veritas were on to something when they developed standard path management software, so they began working on a standard multi-path IO (MPIO) driver for Windows. But Microsoft learned a thing or two from the mediocre device support in those old solutions, so they decided to allow vendors to plug their own smarts into the standard Windows Server 2000/2003 MPIO framework. Microsoft provided basic failover capability and third parties, including EMC, wrote their own device-specific modules (DSMs). This MPIO support evolved and spread, standard on Microsoft’s iSCSI initiator and Hyper-V virtualization platform. PowerPath 5.2.1 for Windows already supported Hyper-V thanks to this.

PowerPath and VMware PSA

VMware also learned a thing or two from HP and Microsoft. Although basic path failover support has been included in ESX for years, vSphere 4 takes it to a new level with pluggable storage architecture (PSA). Every version of ESX 4 includes native multipathing (NMP), but Enterprise Plus licensees can use vendor-supplied plugins to enable more advanced path management. As I noted on Tuesday, there are two three different levels of path selection: Basic path-selection plugins (PSPs), more advanced storage array type plugins (SATPs), and complete multi-path plugins (MPPs).

This is what EMC has introduced: An MPP for vSphere 4 called PowerPath/VE. Like the DSM for Windows MPIO, PowerPath/VE for vSphere slots right into an existing MPIO framework and enables advanced path selection and load balancing without mucking with the internals of the hypervisor. PowerPath/VE has all sorts of smarts in it. It has eight different predictive load balancing policies, proactive disconnect, bus testing, and HBA monitoring.

Super VMware guy Chad Sakac described PowerPath/VE as part of the launch. He notes that EMC is first out of the gate with a multipathing plugin for vSphere, but I suspect that just about every vendor will release similar functionality pretty quickly. In particular I expect support to come from NetApp and 3PAR, since they’re so interested in VMware support.

Licensing Questions

One thing really stuck out in the vSphere launch: PSA is only included in the top-of-the-line Enterprise Plus license. Presumably, this means that, in addition to paying for a PowerPath/VE license, users will have to spring for maximum ESX, too. This is a dumb move, if you ask me. Microsoft made MPIO successful by giving it away with every copy of Windows. They even included it in the free iSCSI initiator download. VMware, in contrast, seems to be actively limiting PSA’s usefulness to the top tier of users. If it was up to me, I would set the VMware MPIO free!

I’m working with EMC and VMware to determine the extent of the NMP/PSA/PowerPath licensing mess. I’ll update this post as I find out the answers!

1. Does every edition of ESX 4 include the basic VMware native multipathing (NMP)?
2. Can one use a vendor-supplied PSA plugin like PowerPath/VE without an enterprise plus license?
3. Does it matter (to licensing) if the plugin is a PSP or an SATP?
4. If “no” to 2 or 3, can PSA be added separately without the plus license if someone wants to use something like PowerPath/VE?

Update: I received a nice email from an EMC engineer correcting me about the plugin types. This kind of open communication is why the web is so great! It turns out that PowerPath/VE is a sort of super plugin called an MPP, not “just” an SATP or PSP. I’ve updated the section above!

© sfoskett for Stephen Foskett, Pack Rat, 2009. |
PowerPath To The Virtual People


This post was categorized as Computer history, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2009. | PowerPath To The Virtual People
Read more posts categorized as Server Virtualization, Storage

VMware officially launched their next-generation (version 4) enterprise family of products today under the “vSphere 4″ name. As I’ve been doing for the last few major ESX releases, I’m focusing this post on the storage changes present in vSphere 4.

For more information on earlier updates, see my articles:

• Storage Fixes in VMware ESX Server 3.5 Update 2
• Storage Changes in VMware ESX 3.5 Update 3
• Storage Changes in VMware ESX 3.5 Update 4
  

One first step is VMware’s whitepaper, “What’s New in VMware vSphere 4: Storage“.

Licensing and Availability of Features

The most important change at a glance is in the licensing model for VMware.

• A new low-end “standard” level includes all of the components a small business might need, including thin provisioning of storage, at a very attractive price.
• The “advanced” level includes advances in the area of data protection.
• At the “enterprise” level, live migration of storage is enabled.
• Top of the heap is “enterprise plus”, which enables the intriguing plug-in third-party multipathing support previously rumored.
• Enterprise customers with current support will get vStorage thin provisioning and data protection when they upgrade.

Each step up the licensing ladder enables important new storage features (source: VMware)

Thin Provisioning

One of the most widespread storage features is native thin provisioning. VMware ESX 4 will allocate storage in 1 MB chunks as capacity is used. This isn’t really completely new – similar support was enabled by default for virtual disks on NFS in VI 3, and thin provisioning could be enabled on the command line for block-based storage as well. It was also present in VMware’s desktop products, including my own copy of Fusion. And ESX allows thick-to-thin conversion during Storage VMotion.

The difference with vSphere 4 is that thin provisioning is fully supported and integrated into every version of ESX. Although many storage arrays now also offer thin storage, the addition of native, integrated thin provisioning right in ESX is huge. This alone will be a major capacity (and thus, cost) savings feature! VMware claims 50% storage savings in their lab tests.

Dynamic Expansion of VMFS Volumes

VMFS volumes can now grow (and, in some cases, shrink) online without resorting to spanning to a new LUN. Under vSphere 4, VMFS volumes can grow to take advantage of expanded LUNs (up to 2 TB per LUN). The old method still works as well, and multi-LUN spanned VMFS volumes can grow when any of their LUNs is expanded.

Enhanced Storage VMotion

Like thin provisioning, Storage VMotion has been elevated to first-class status, supported just about everywhere you’d want it. It’s in all the likely spots within vCenter.

Storage VMotion gives serious storage flexibility now, enabling (almost) any-to-any migration of VMFS volumes: Pick up a Fibre Channel, iSCSI, or NFS disk image and move it to another datastore running any of those protocols to convert live. And you can do thick-to-thin provisioning at the same time.

Under the hood, the whole infrastructure has been revised. Storage VMotion leverages VMware’s change block tracking instead of disk snapshots now, speeding up the migration process and reducing the (formerly excessive) memory and CPU requirements of Storage VMotion in 3.5. This is the same technology leveraged by vSphere’s High Availability features, by the way.

Pluggable Storage Architecture (PSA)

Only “Enterprise Plus” licensees will get to use it, but the vSphere family also sports a new pluggable storage architecture (PSA) which will initially be leveraged to deliver vendor-specific multipath support. Note that the native multipath support in vSphere continues to be round-robin fail-over only – it will not load balance I/O across multiple paths or make more intelligent decisions about which paths to use.

vSphere 4′s Pluggable Storage Architecture allows third-party developers to replace ESX’s storage I/O stack (source: VMware)

As you may gather from this VMware illustration (but would probably miss since it’s not all that comprehensible), there are two classes of third-party plug-ins:

1. Basic path-selection plugins (PSPs) will merely optimize the choice of which path to use, ideal for active/passive type arrays
2. Full storage array type plugins (SATPs) will allow load balancing across multiple paths in addition to path selection for active/active arrays

EMC also announced PowerPath/VE for vSphere, integrating their popular multi-platform path management software directly into ESX. It’s not clear at this point whether PowerPath will require an Enterprise Plus license (or if it will come with one) or if it will work with all editions, but I’m sure that will be clarified soon. My EMC contacts do tell me that PowerPath/VE is licensed on a per-socket basis (like VMware of yore) and that EMC sales reps have some room to get creative on licensing.

iSCSI Software Initiator Enhancements

It’s harder to claim it as a new feature, but the iSCSI software initiator has also been tweaked and tuned to use less CPU time and deliver better throughput. The iSCSI configuration process has also been smoothed out so one no longer needs to have a live Service Console connection in order to communicate with an iSCSI target. And changes made in the general tab are now global, so they’ll propagate down to each target. Bi-directional CHAP is also added, so the target can now be authenticated in addition to the initiator.

vSphere also includes a paravirtualized iSCSI driver (PVSCSI) which works like vmxnet to present a higher-performance iSCSI adapter within certain supported guest OSes.

More information on multipathing iSCSI can be found at Yellow Bricks and in Chad Sakac’s mega-post on the topic.

High-Performance I/O

vSphere really pours on the I/O power. They claim a 3x increase, to “over 300,000 I/O operations per second”, but mentioned in the launch that they’ve gotten to 400,000 in some workloads. This is really amazing, and if it’s true in real-world use means that VMware ESX can now host just about any application you can think of.

It’s funny, but just a few weeks before the announcement VMware’s Chief Data Center architect, Scott Davis, told me at Storage Networking World that vSphere would double I/O performance. I thought this was a bold statement at the time, but VMware went further in their official documents, claiming 3x improvement. Now we have an on-stage admission that it could be up to 4x! This is serious stuff, folks: Can VMware really do 400,000 IOPS?

One question is whether these IOPS improvements require the new VMDirectPath I/O for Storage, which binds a physical Fibre Channel HBA to a single guest OS, or if they’re generalized across all systems.

vCenter Improvements

vCenter now includes more storage information:

• Each VM and ESX in the inventory has a tab showing storage information and allowing users to set alarms on storage use – setting capacity alarms becomes extremely important when thin provisioning is used!
• vCenter inventory has a view showing datastore details
• There’s also a nice storage topology map

Data Protection APIs

VMware also enhanced the APIs for consolidated backup (VCB) into “vStorage Data Protection”. Available only with Advanced or Enterprise licenses, these APIs allow enterprise backup vendors to develop specific integration with VMware without VCB in the middle.

Conclusion

vSphere is loaded with storage improvements, though many seem like incremental updates rather than new features. Admins will welcome thin provisioning support, and the full integration of Storage VMotion is certainly welcome, but I think the changes to vCenter will be most noticeable. I’ll be looking into EMC’s PowerPath/VE and the amazing I/O improvements as we move forward and learn more about the next ESX!

Duncan Epping posted much more vSphere Linkage on his Yellow Bricks blog if you’re interested in non-storage features.

© Stephen Foskett for Gestalt IT, 2009. | Storage Changes in the VMware vSphere 4 Family
Read more posts categorized as All, Server Virtualization, Storage