EMC VFCache is a Simple Read Cache

I was not pre-briefed on this product, and I’m not all that thrilled at the prospect of attending a launch webinar, so what you read here is based on my own research and reading of the available information as of this morning.

When EMC announced Project Lightning last year, company insiders expressed surprise to me. It seems that many had never heard of the project, and those that had didn’t think it was far enough along to be announced. I didn’t even bother to write about the Project Lightning announcement at the time. But today EMC unveils the production product that came out of Project Lightning.

VFCache is a filter driver that caches writes

EMC VFCache appears to be a simple and straightforward offering:

1. A PCIe SSD from Micron or LSI sits in the server and acts as a read cache to accelerate performance
2. EMC software also runs on the server, snooping on I/O and filling the cache with relevant data

There’s not a lot more to the product than that. EMC will sell the PCIe SSD and bundled software as VFCache, and will no doubt market the heck out of this product. Perhaps the only novel twist is the so-called “split-card” mode, which allows the card to act as a write cache. But EMC only supports this for transient “throwaway” data with direct attached storage (DAS) as a backend. There’s no way a conservative, enterprise focused company like EMC would risk sanctioning a writeback cache with no redundancy or data protection features.

VFCache uses a filter driver installed in the VM guest

Perhaps the biggest limitation of the initial VFCache offering is its limited applicability to enterprise server virtualization environments. VFCache uses a filter driver installed in each VM guest, and includes no hypervisor drivers though there is a vCenter plug-in. This makes VMware vMotion very tricky, involving scripting to remove and re-add storage. This means VMware SRM will not easily work, and there is no support for clustering, either.

This is no surprise, since VFCache appears to the host as a local storage volume (AKA, a disk drive or LUN) which would disappear if a virtual machine is moved to another server. Virsto solved this problem by virtualizing storage presentation to the hypervisor, and Fusion-io’s ioTurbine software does not interfere with vMotion either. EMC will likely go in this direction in the future, but it’s a big hole in the product for now.

You might also like reading Micron Bursts Into the PCIe SSD Market to learn more about the card EMC is using

The News: EMC Is in the Host-Based Storage Business

The primary use case for this product is server I/O acceleration. This is desperately needed, as applications and servers are rapidly outrunning the capabilities of conventional storage arrays. EMC and other legacy array manufacturers initially tried to address this I/O imbalance with tiered storage and in array caching. Indeed, these technologies are fairly effective at accelerating the performance of conventional disk storage arrays.

But flash manufacturers like Fusion-io (not to mention Micron and LSI) absolutely demolished storage array performance with their in-server offerings. EMC faced the prospect of losing out on the high-performance storage market. EMC simply could not allow their bread-and-butter enterprise customers to look elsewhere for strategic, high-performance storage for high-profile applications.

VFCache gives EMC salespeople a silver bullet when customers demand maximum performance, but this launch may not spell doom for the flash startups. For one thing, it legitimizes host-based flash cards as a viable component of enterprise storage architectures. It also opens the door to comparison between SAN storage and non–SAN alternatives that go well beyond what EMC is currently offering.

Shared Flash Storage Is on Deck: Project Thunder

“Project Thunder” will externalize the PCIe flash cards over a high-performance “Server Area Network”

As part of the VFCache introduction, EMC is also talking about Project Thunder, a shared version of VFCache. At the very least, thunder will allow multiple servers to access a shared pool of flash cache. This should allow VMware vMotion and DRS to function, and could be much more than that.

EMC could build a high-availability, high-performance all-flash storage array that may even use InfiniBand as an interconnect. The new Nimbus Data E-Class storage array matches this description perfectly, and their CEO tells me that performance over InfiniBand is indeed comparable to in-server PCIe flash cards. It seems logical for EMC to enter this market, if only to disrupt the momentum of Fusion-io and the rest of the all-flash storage upstarts.

Read more about the Nimbus E-Class: The First Big, Redundant, All-Flash Enterprise Array

The only fly in the ointment here is the recent consolidation of the InfiniBand market. Mellanox bought Voltaire, and QLogic sold out to Intel, putting that protocol on tenuous grounds. Perhaps 40 or 100 Gb Ethernet will emerge as a viable alternative for high-performance connectivity, or perhaps these products will retrench on shared PCI Express instead. Micron recently purchased Virtensys for just such a product, and Xsigo has been making big waves in the area of converged I/O as well. The market clearly need something better than Fibre Channel for maximum performance storage, even if InfiniBand isn’t it.

Stephen’s Stance

EMC VFCache (née Project Lightning) is a fairly simple offering: A server-based PCIe flash card that acts as a read cache with no integration with storage arrays or hypervisors. But EMC’s entrance into the host-based flash storage market is a powerful demonstration of the wave of disruption caused by flash-based storage and high-performance computing. Although I am not all that impressed with the product itself, I would be distressed if EMC had not introduced it.

VFCache illustrations are copyright EMC Corporation and are used here with permission

© Stephen Foskett for Gestalt IT, 2012. | EMC VFCache (aka “Project Lightning”) Is One Small Step, But an Important One
Read more posts categorized as Server Virtualization, Storage

Every technician develops a set of tools and practices that serve them well. In this post, I present my best practices for enterprise IT.

Although many people are cynical about the whole idea of best practices, I’m a believer. I think that such beasts do exist, just that too many companies, analysts and especially consultants spend too much time applying the label to whatever works in their best interest at the time. To counteract this cesspool of non-best practices, I thought it best to put down a few ideas of my own. Following are four fundamental best practices I have distilled from almost 20 years in enterprise IT. I wonder if you agree with them.

Before proceeding, I suggest that you take a look at the three principle tests that I use to define a best practice.

Best practice 1: Minimize Complexity

IT has a tough position in business. We are ignored while everything is working well, and in trouble when things start falling apart. We are held accountable, but generally cannot control our own destiny. The only way we have a fighting chance is to minimize the complexity of the solutions we must support.

As I said in Sailing the Titanic (Why We Need ILM and Then Some!): “To make a timely (and tired) Harry Potter analogy, IT are the house-elves of the business – powerful but subservient, with little input into what happens above and around them.”

This is where the current trends of virtualization, consolidation, and standardization comes in. A single administrator can manage a far-larger environment if it’s standardized, and these systems tend to be far more reliable as well. Sadly, minimizing complexity is the antithesis of a true hacker’s heart. Who wouldn’t want to play with the latest and greatest toys? Why not build an intricate and interconnected system? But these solutions fail my “best practices” test: They’re risky, unusual and just plain imprudent.

Best Practice 2: Use the Right Tool For the Job

Any woodworker or mechanic will tell you that using the right tool makes any job easier. Even if it works, it’s just bad practice to masquerade network-attached storage (NAS) as a block device or a storage area network (SAN) as a collection of shared files. I am always skeptical of so many oddball innovations trumpeted by vendors large and small because they seem to fail the “right tool” practice.

I wrote more about this in Use Process Solutions For Process Problems, Technical Solutions For Technical Ones and Why Do I Ignore NAS?

But sometimes they work out just fine. You see, best practices can’t predict the future or what works in a certain situation. It’s better to minimize complexity and stick to what you know than to press for the perfect solution every time. It wouldn’t be right to bring SAN storage into an all-NAS environment just for a single database application, for example. Although we generally shouldn’t try to drive a nail with a screwdriver, it can be the best option when a hammer can’t be found!

Best Practice 3: Prepare For Failure

The hacker in me is tempted to stop tinkering as soon as a system is functional, but the realist in me knows that I have to make it twice as good. You can’t prepare for every eventuality, but you can seek out probable points of failure and harden them.

For example, see Chris Evans’ discussion of Choosing Between Monolithic and Modular Architectures

Many of the fundamental best practices in enterprise IT come down to this. Redundancy from RAID and N+1 components, the widespread use of multi-path technology, and high-availability design are all examples of preparing for the worst. We would not bother with all of this if we didn’t know that errors occur even in well-engineered systems, and that everything we build is sensitive to failure.

Best Practice 4: Align Expectations with Reality

We all face conflicting demands, and we are tempted to offer best-case responses to keep everyone happy. But excessive optimism leaves IT folks stressed and end users frustrated when things don’t go right. It’s time to adjust expectations.

See The Techie/Business Schism, where I talk about the looming disconnect even within IT

Concepts like service level agreements (SLAs) and business impact analyses come from an innate knowledge that the real world doesn’t always conform to the best-case assumption. Although we don’t have real control over user demands, we can try to set correct expectations by admitting what we can and can’t do realistically.  Don’t let those who rely on us labor under misconceptions of our capabilities.

Stephen’s Stance

We can’t let others lead us astray with talk of “best practices”, but we can define some of our own. My three-part best practices test, and these four general practices, have served me well over the years. What are your best practices? Image credits: Dave’s Bike Tools by bre pettis

© Stephen Foskett for Gestalt IT, 2010. | Four Fundamental Best Practices for Enterprise IT
Read more posts categorized as Featured, Storage, Tech

First of all, let’s take a look at the individual terms:

High Availability

According to Wikipedia, you can define availability in the following ways:

The degree to which a system, subsystem, or equipment is operable and in a committable state at the start of a mission, when the mission is called for at an unknown, i.e., a random, time. Simply put, availability is the proportion of time a system is in a functioning condition.

The ratio of (a) the total time a functional unit is capable of being used during a given interval to (b) the length of the interval.

And most online dictionaries seem to have a similar definition of availability. When we are talking about HA, we imply that we want the functioning condition of your system to be increased.

Going by the above you will also notice that there is no fixed definition of the availability. Simply put, it would mean that you need to put your own definition in place when talking about HA. You need to define what HA means in your environment. I’ve had customers that needed HA and defined this as the system having a certain amount of uptime, which is one way to measure it.

On the other hand you would be hard pressed if you were able to work with your system, but the data that you were working with was corrupted because one of your power users made an error during a copy job and wrote an older data set in the wrong spot. This would mean that your system is in itself available. You can log on to it, you can work with it, but the output you are going to get will be wrong.

To me, such a scenario would mean that your system isn’t available. After all, it’s not about everything being online. It’s about using a system in the way you would expect it to work. But when you ask most people in IT about availability, the first thing you will likely hear is something related to uptime or downtime. So, my tip to you is once again:

Define what “available” means to you and your organization/customer!

Disaster Recovery

Let’s do the same thing as before and let’s turn to some general definitions. Wikipedia defines disaster the following way:

disaster is a perceived tragedy, being either a natural calamity or man-made catastrophe. It is a hazard which has comes to fruition. A hazard, in turn, is a situation which poses a level of threat to life, health, property, or that may deleteriously affect society or an environment.

And recovery is defined the following way (when it comes to health):

Healing, or Cure, the process of recovering from an injury or illness.

So, in a nutshell this is about bouncing back to your feet once a disaster strikes. Now again, it’s important to define what you would call a disaster, but at least there seems to be some sort of common understanding that anything that would get you back up and running after an entire site goes down, usually falls under the label of a DR solution.

It all boils down to definitions!

When you talk to other companies or vendors about HA and/or DR, you will soon notice that most have a different understanding of what HA and DR are. Your main focus should be to have a clear definition for yourself. Try to find out the importance and value of your solution and base your requirements on that. Ask yourself simple questions like for example:

• What is the maximum downtime I can cope with before I need to start working again? 8 hours per year? 1 hour per year? 4 hours per month? What is my RPO and RTO
• How do I handle planned maintenance? Can I bring everything down or do I need to distribute my maintenance across independent entities?
• Can I afford the loss of any data at all? Can I afford the partial loss of data?
• What if we see a city-wide power outage? Do I need a failover site, or are all my users in the same spot and won’t be able to work anyway?

Questions like these will help you realize that not everything you have running has the same value. Your development system with 6000 people working on it worldwide might need better protection than your productive system that is only being used by 500 people spread through the Baltic region.

Or in short

Knowing what kind of protection you need is key. Fact is that both HA and DR solutions never come cheap. If you need the certainty that your solution is available and able to recover from a disaster, you will notice that the price tag will quickly skyrocket. Which is another reason to make sure that you know exactly what kind of protection you need, and creating that definition is the most important starting point. Once you have your own definition, make sure that you communicate those definitions and requirements so that all parties are on the same page. It should make your life a little easier in the end.

© Bas for Gestalt IT, 2010. | How do you define high availability and disaster recovery?
Read more posts categorized as Featured, Server Virtualization, Storage

HAM combines conventional ingredients to create a whole new flavor

High Availability Manager consists of three main components:

1. A conventional multi-pathing agent like HDLM on each server. This enables the server to continue accessing the storage if one USP fails. It will “think” it’s talking to a single storage target, but will actually be talking to two USPs that can be metro distances (60-100 miles) away, given proper connectivity. Microsoft MPIO will probably be supported as well, and VMware native multi-pathing (NMP) should come shortly after release. Don’t hold your breath for PowerPath to be officially supported in the short term, but it ought to work fine without changes.
2. Existing TrueCopy synchronous storage replication technology will keep the data and quorum disks (see below) in lock-step. This gives the limitation on distance between systems, since latency is the enemy of storage protocol performance. Once the arrays move too far apart, write performance will suffer on the local array while it waits for data to be copied.
3. Conventional clustering technology with a heartbeat and shared quorum disk lets both arrays know what’s going on. The quorum “lives” on the remote side, with that secondary array watching to make sure the primary array is still running. If the heartbeat goes away, the secondary array marks the HAMmed LUNs read/write and starts handling I/O from the servers, which will just have failed over.

So HDS’ secret HAM sauce is ketchup. There’s no amazing new technology here, and maybe that’s for the best in the kind of huge, conservative enterprise environments that will use this product. The big change was in programming the USP controllers to monitor the quorum disk and orchestrate the entire failover. In fact, this might even be considered a feature of TrueCopy, not a standalone product.

HDS ought to reconsider one element of the HAM pitch, however. Although it will undoubtedly yield a very highly-available storage architecture, nothing provides 100% availability. There are many moving parts involved, and unplanned outages can still happen. Multipath driver bugs are not unheard of: Back in the day, one version of a certain three-letter company’s product just plain refused to fail over! An instantaneous outage on one of the host channels or a skipped heartbeat on the USP controllers could also cause a failed failover. Plus, there is no provision for automated fail-back. Once the failover occurs, the system would certainly be operating in a degraded-availability mode and would require a (planned) outage to re-establish operations.

All that being said, HAM remains an attractive offering for shops with multiple USPs visible to critical servers. They can turn on the HAM software with existing hardware and add peace of mind, knowing that everything is that much more available. One aspect that really impresses is the fact that each USP can be running a different firmware, reducing the risk of upgrade-induced outages. Once it ships to customers (in the fourth quarter of this year), we will have to consider the (as-yet unnamed) cost.

And what about the fact that HAM will also allow seamless upgrades to a new generation of USP hardware? HDS still isn’t talking about that possibility!

© Stephen Foskett for Gestalt IT, 2009. | HDS High Availability Manager: How It Works
Read more posts categorized as Exclusive, Featured, Storage, Top Story