Although it’s rare in the PC world, multipath I/O is not new in enterprise IT. I’ve been juggling paths to storage and networks as long as I’ve been a systems administrator, and that’s a bit longer than I care to admit. But the proliferation of technologies has made it difficult to understand path management. What’s the difference between “dual active” and “active/active”? Is “active/passive” really that bad?

What is Multipath? And Why?

The good old days: One device, one path

In the beginning, computers connected to peripherals and other computers through a single bus or channel and life was easy. Although one might mistake the names of the dominant printer connection (parallel) for some kind of multipath system when compared to the modem connection (serial), this was not the case. Only the bits traveled in parallel – the logical connection was a simple single path.

Early-90′s servers might confuse admins with two SCSI connections to a single device

Then things got complicated. The SCSI protocol allowed for multiple devices in a chain, and even for two different “initiators” (computers or controllers) to interact with these “targets”. Some folks even dual-attached devices to a single computer with multiple controllers.

Why would one device and one computer need more than one connection? It boils down to two factors:

1. Performance – I/O channels have typically been slower than the computer could handle, so multiple channels might be used to increase the amount of data that can flow in and out.
2. Reliability – If one connection failed, the other might still be usable, reducing the risk of an outage.

Late-90′s enterprise systems might have four or more paths to a single storage array

Pretty soon, enterprise computer architecture had gotten incredibly complex. I remember connecting a massive HP V-class server to an EMC Symmetrix with eight separate Fibre Channel cables. Each disk “LUN” showed up twice, and we had hundreds of them. We managed all of these virtual storage paths using HP’s PVLinks dynamic multipathing software. We used Veritas DMP and EMC PowerPath to do pretty much the same thing on Solaris and other UNIX systems.

Active/Passive to Active/Active

The earliest path management software provided two incredibly important functions: It figured out which of the SCSI targets it saw were actually different names for the same one, and it allowed the operating system to choose one and fail over to the other in case of an interruption. These were Active/Passive links – no matter how many paths were presented (and Fibre Channel switches sometimes presented eight or more), only one was active at any one time.

Modern systems have abstracted and virtual I/O channels, making path management much more important

But the EMC Symmetrix and similar high-end storage systems changed all this. Symmetrix storage was fully virtualized – the presentation of LUNs to servers was entirely disconnected from the actual disks and RAID sets in the array. This meant the Symmetrix could handle I/O requests across different paths and controllers for the same LUN. EMC and the rest responded with Active/Active path management software, allowing I/O to travel in parallel for the first time.

How is Dual Active Different?

Not everything called Active/Active is created equal. In fact, many supposed Active/Active setups really shouldn’t be called that since they don’t use both paths for all data. Instead, I like to call these Dual Active – both paths are active but with different data.

Consider the differences between the following two solutions:

A true active/active setup uses all paths for all data all the time

A “dual active” setup uses both paths, but each target is directed to one or the other

See the difference? Although the paths are active in both cases, they are not the same. Both approaches have merit, and neither is inherently superior, but they should have different names applied. Even active/passive has its place, since simplicity is often a virtue.

Dual Active Outside Storage

These same concepts apply outside the field of storage and I/O. Many server clustering systems use the same terminology, right down to the misapplication of “active/active” when “dual active” is more appropriate. It’s easy to miss the significance of this difference, but it can make more of an impact in clustering since CPU workloads are harder to balance.

Let me know what you think. If there is interest, I might dive into path management strategies like round robin!

© Stephen Foskett for Gestalt IT, 2010. | Multipath: Active/Passive, Dual Active, and Active/Active
Read more posts categorized as Storage

Like Ibrix, which was acquired by HP under better circumstances last year, and ONstor, similarly scooped up by LSI, Exanet was best known for clustered NAS software. Their ExaStore software, bundled with IBM and Xyratex hardware, put up impressive scalability and performance numbers. Dell will likely leverage this software with their own hardware as a NAS complement to their leading EqualLogic iSCSI line. Exanet’s Xyratex-sourced DX line of Fibre Channel storage devices is unlikely to be at all attractive to Dell.

Rumors of a white knight for Exanet were widespread last year. Fujitsu Siemens Computers was said to be the front-runner in May, and Exanet’s name came up mid-year as HP and LSI made their moves. Dell was apparently willing to put up some money to gain access to Exanet’s NAS technology later in the year but the company’s investors reportedly scuttled that deal. Plan B for the folks in Round Rock seems to have been to wait it out and secure the technology from the now-moribund company. If Dell keeps the doors open, Exanet’s R&D center will become their first such facility in Israel.

The acquisition gives Dell a retort when HP presents their invigorated Ibrix line to enterprise customers. It is likely that Dell will follow HP with a software/hardware NAS bundle possibly featuring their PowerEdge blades. Back-end storage could come from many sources: Dell’s own PowerVault MD line, their EqualLogic PS iSCSI gear, or the Dell/EMC CLARiiON AX and CX lines. Exanet is known as a higher-end scale-out offering (think Isilon or HP’s PolyServe) rather than a general-purpose NAS.

Two key questions arise from Dell-owned NAS software:

1. Would Dell reduce their reliance on Microsoft’s Windows Storage Server, as used in their PowerVault NX300 and NX3000 NAS devices? I suspect not, since Exanet is not a low-end product and Windows-powered NAS has typically sold into a separate market niche. Rival HP has certainly continued pushing Microsoft-powered gear since buying Ibrix.
2. What does this mean for Dell’s relationship with EMC? The company only started selling the EMC Celerra NX4 last year, and a homegrown Exanet/PowerVault solution is not a drop-in replacement. Dell’s relationship with EMC continues getting deeper, and a split is unlikely in the near term.

At this point, this looks like a vote for Exanet’s technology and a reaction to HP’s PolyServe and Ibrix moves, not a statement against EMC or Celerra. Indeed, considering that Dell was merely investigating an OEM relationship before this all fell out, it doesn’t look like a strategic move at all for the company. Exanet will likely become a new line item, but Dell’s storage roster will look largely the same.

© Stephen Foskett for Gestalt IT, 2010. | Dell Scoops Up Exanet After All
Read more posts categorized as Exclusive, Featured, Storage, Top Story

HAM combines conventional ingredients to create a whole new flavor

High Availability Manager consists of three main components:

1. A conventional multi-pathing agent like HDLM on each server. This enables the server to continue accessing the storage if one USP fails. It will “think” it’s talking to a single storage target, but will actually be talking to two USPs that can be metro distances (60-100 miles) away, given proper connectivity. Microsoft MPIO will probably be supported as well, and VMware native multi-pathing (NMP) should come shortly after release. Don’t hold your breath for PowerPath to be officially supported in the short term, but it ought to work fine without changes.
2. Existing TrueCopy synchronous storage replication technology will keep the data and quorum disks (see below) in lock-step. This gives the limitation on distance between systems, since latency is the enemy of storage protocol performance. Once the arrays move too far apart, write performance will suffer on the local array while it waits for data to be copied.
3. Conventional clustering technology with a heartbeat and shared quorum disk lets both arrays know what’s going on. The quorum “lives” on the remote side, with that secondary array watching to make sure the primary array is still running. If the heartbeat goes away, the secondary array marks the HAMmed LUNs read/write and starts handling I/O from the servers, which will just have failed over.

So HDS’ secret HAM sauce is ketchup. There’s no amazing new technology here, and maybe that’s for the best in the kind of huge, conservative enterprise environments that will use this product. The big change was in programming the USP controllers to monitor the quorum disk and orchestrate the entire failover. In fact, this might even be considered a feature of TrueCopy, not a standalone product.

HDS ought to reconsider one element of the HAM pitch, however. Although it will undoubtedly yield a very highly-available storage architecture, nothing provides 100% availability. There are many moving parts involved, and unplanned outages can still happen. Multipath driver bugs are not unheard of: Back in the day, one version of a certain three-letter company’s product just plain refused to fail over! An instantaneous outage on one of the host channels or a skipped heartbeat on the USP controllers could also cause a failed failover. Plus, there is no provision for automated fail-back. Once the failover occurs, the system would certainly be operating in a degraded-availability mode and would require a (planned) outage to re-establish operations.

All that being said, HAM remains an attractive offering for shops with multiple USPs visible to critical servers. They can turn on the HAM software with existing hardware and add peace of mind, knowing that everything is that much more available. One aspect that really impresses is the fact that each USP can be running a different firmware, reducing the risk of upgrade-induced outages. Once it ships to customers (in the fourth quarter of this year), we will have to consider the (as-yet unnamed) cost.

And what about the fact that HAM will also allow seamless upgrades to a new generation of USP hardware? HDS still isn’t talking about that possibility!

© Stephen Foskett for Gestalt IT, 2009. | HDS High Availability Manager: How It Works
Read more posts categorized as Exclusive, Featured, Storage, Top Story