Cisco’s UCS B-Series is an exemplary modern blade system

Blade servers are easily recognized in the data centers, trade shows, and product catalogs of today: They’re the ones that nestle together in an enclosure, sharing some resources rather than standing on their own in a rack or on the floor. But what is the essential element that separates a blade from any other kind of server?

Let’s consider some defining characteristics of the blade server species:

1. Physically, a rack-mounted blade enclosure contains a number of server blades. Each blade is smaller than a 19″ rack, allowing more to be packed into the same space. This contrasts with larger, self-enclosed tower or rack-mount servers.
2. Server blades rely on the blade enclosure for critical supporting functions like power, cooling, and I/O ports, so they cannot stand alone. This further improves density and efficiency. Stand-alone servers, on the other hand, include these functions in their case.
3. Blade systems include some sort of management device that monitors the blades and can control some of their functions. Conventional servers do not always have this kind of consolidated management.
4. The blade chassis includes consolidated and shared I/O channels, ranging from keyboard, video, and mouse (KVM) to networking and storage (usually Ethernet and Fibre Channel). These add flexibility, since external ports can be shared by multiple blades and reconfigured without disruption.
5. Blade systems are optimized for high availability, with hot-swap components everywhere from power to fans to the blades themselves. Since these are shared, it is more efficient to purchase redundant parts for a blade system than for each server individually.

HP’s blade systems (like this loaded c3000) make up half the market today

To me, these five elements are key to a modern blade system. Without them, a blade solution cannot meet the expectations of buyers (or the promises of vendors!) And what are these benefits? I took a look at the marketing materials for the leading companies in the space (HP, Dell, IBM, and Cisco), and this is what they promise:

1. Efficiency – More processing power in a smaller footprint (physical size, power consumption, cooling, and weight)
2. Manageability – Simpler and cheaper systems management
3. Reliability – High availability thanks to redundant components
4. Performance – Improved I/O performance thanks to shared network and storage features
5. Flexibility – Simplified cabling that can be reconfigured in software

© Stephen Foskett for Gestalt IT, 2012. | What Is a Blade Server?
Read more posts categorized as Cloud Computing, Featured, Server Virtualization

A Spectrum of Compatibility

VMware vSphere will run on a wide variety of hardware, so it’s no surprise that vendors feel emboldened to claim compatibility when they shouldn’t. To me, the minimum acceptable level of compatibility is a listing on the official VMware hardware compatibility list (HCL). I would never recommend production use of a product and configuration that is not listed there. “Because I said it works” is not a valid assurance of compatibility.

For more, read I Don’t Care If You Claim It Works (The Support Matrix Blues)

Whenever a vendor claims VMware compatibility, you can bet I am looking it up in the HCL. Time and again, I’ll interrupt a vendor briefing questioning their claims of compatibility. It’s not enough to have product X in the list when you’re pitching product Y! I will accept “we’re working on it” as an answer, but you can bet I’ll check later to see if it ever made the list!

For products that are listed in the VMware HCL, my next question concerns availability of a vCenter plug-in. Although many devices don’t actually need a plug-in, this is typically the next step companies will take when it comes to VMware vSphere. I might actually look upon a product favorably if it doesn’t need a plug-in to function correctly, but it’s definitely best for administrators to be able to configure infrastructure products from within vCenter.

When it comes to block storage products, support for the three main VAAI primitives is a major positive factor. Now that vSphere 5 is here, basic VAAI support is becoming more common. This is because it supports the simpler T10 interfaces for block zeroing, atomic test and set, and extended copy. Once again, any vendor claiming VAAI compatibility gets a quick check in the VMware HCL to find out just what that means. Bonus points for supporting both vSphere 4 and 5.

Check out A Complete List of VMware VAAI Primitives for more info!

I’m really impressed when I come across a storage device that supports advanced VMware integration, NFS such NFS VAAI primitives or VASA. This is really cutting edge stuff at this point, but it won’t be so impressive 6 months from now. By then, I expect more and more companies will support all of VAAI and offer VASA providers. And maybe by then we’ll have more information about what comes next for VMware integration!

True, VMware VASA is Not Much (Yet)

Stephen’s Stance

I’m a storage guy, so my line of questioning revolves around storage integration features. But I encourage everyone to question vendors who claim compatibility with VMware or any other complicated technology. “Because I said so” is not an acceptable answer: We have the right to demand HCL listings, vCenter plug-ins, and advanced API support.

Note: This is the sort of stuff I’ll get into in my “Building Virtual Infrastructure” Seminar Series - Why not sign up and join us? It’s free!

© Stephen Foskett for Gestalt IT, 2012. | “Our Storage Array Is Compatible with VMware…” Says Who?
Read more posts categorized as Server Virtualization, Storage

EMC VFCache is a Simple Read Cache

I was not pre-briefed on this product, and I’m not all that thrilled at the prospect of attending a launch webinar, so what you read here is based on my own research and reading of the available information as of this morning.

When EMC announced Project Lightning last year, company insiders expressed surprise to me. It seems that many had never heard of the project, and those that had didn’t think it was far enough along to be announced. I didn’t even bother to write about the Project Lightning announcement at the time. But today EMC unveils the production product that came out of Project Lightning.

VFCache is a filter driver that caches writes

EMC VFCache appears to be a simple and straightforward offering:

1. A PCIe SSD from Micron or LSI sits in the server and acts as a read cache to accelerate performance
2. EMC software also runs on the server, snooping on I/O and filling the cache with relevant data

There’s not a lot more to the product than that. EMC will sell the PCIe SSD and bundled software as VFCache, and will no doubt market the heck out of this product. Perhaps the only novel twist is the so-called “split-card” mode, which allows the card to act as a write cache. But EMC only supports this for transient “throwaway” data with direct attached storage (DAS) as a backend. There’s no way a conservative, enterprise focused company like EMC would risk sanctioning a writeback cache with no redundancy or data protection features.

VFCache uses a filter driver installed in the VM guest

Perhaps the biggest limitation of the initial VFCache offering is its limited applicability to enterprise server virtualization environments. VFCache uses a filter driver installed in each VM guest, and includes no hypervisor drivers though there is a vCenter plug-in. This makes VMware vMotion very tricky, involving scripting to remove and re-add storage. This means VMware SRM will not easily work, and there is no support for clustering, either.

This is no surprise, since VFCache appears to the host as a local storage volume (AKA, a disk drive or LUN) which would disappear if a virtual machine is moved to another server. Virsto solved this problem by virtualizing storage presentation to the hypervisor, and Fusion-io’s ioTurbine software does not interfere with vMotion either. EMC will likely go in this direction in the future, but it’s a big hole in the product for now.

You might also like reading Micron Bursts Into the PCIe SSD Market to learn more about the card EMC is using

The News: EMC Is in the Host-Based Storage Business

The primary use case for this product is server I/O acceleration. This is desperately needed, as applications and servers are rapidly outrunning the capabilities of conventional storage arrays. EMC and other legacy array manufacturers initially tried to address this I/O imbalance with tiered storage and in array caching. Indeed, these technologies are fairly effective at accelerating the performance of conventional disk storage arrays.

But flash manufacturers like Fusion-io (not to mention Micron and LSI) absolutely demolished storage array performance with their in-server offerings. EMC faced the prospect of losing out on the high-performance storage market. EMC simply could not allow their bread-and-butter enterprise customers to look elsewhere for strategic, high-performance storage for high-profile applications.

VFCache gives EMC salespeople a silver bullet when customers demand maximum performance, but this launch may not spell doom for the flash startups. For one thing, it legitimizes host-based flash cards as a viable component of enterprise storage architectures. It also opens the door to comparison between SAN storage and non–SAN alternatives that go well beyond what EMC is currently offering.

Shared Flash Storage Is on Deck: Project Thunder

“Project Thunder” will externalize the PCIe flash cards over a high-performance “Server Area Network”

As part of the VFCache introduction, EMC is also talking about Project Thunder, a shared version of VFCache. At the very least, thunder will allow multiple servers to access a shared pool of flash cache. This should allow VMware vMotion and DRS to function, and could be much more than that.

EMC could build a high-availability, high-performance all-flash storage array that may even use InfiniBand as an interconnect. The new Nimbus Data E-Class storage array matches this description perfectly, and their CEO tells me that performance over InfiniBand is indeed comparable to in-server PCIe flash cards. It seems logical for EMC to enter this market, if only to disrupt the momentum of Fusion-io and the rest of the all-flash storage upstarts.

Read more about the Nimbus E-Class: The First Big, Redundant, All-Flash Enterprise Array

The only fly in the ointment here is the recent consolidation of the InfiniBand market. Mellanox bought Voltaire, and QLogic sold out to Intel, putting that protocol on tenuous grounds. Perhaps 40 or 100 Gb Ethernet will emerge as a viable alternative for high-performance connectivity, or perhaps these products will retrench on shared PCI Express instead. Micron recently purchased Virtensys for just such a product, and Xsigo has been making big waves in the area of converged I/O as well. The market clearly need something better than Fibre Channel for maximum performance storage, even if InfiniBand isn’t it.

Stephen’s Stance

EMC VFCache (née Project Lightning) is a fairly simple offering: A server-based PCIe flash card that acts as a read cache with no integration with storage arrays or hypervisors. But EMC’s entrance into the host-based flash storage market is a powerful demonstration of the wave of disruption caused by flash-based storage and high-performance computing. Although I am not all that impressed with the product itself, I would be distressed if EMC had not introduced it.

VFCache illustrations are copyright EMC Corporation and are used here with permission

© Stephen Foskett for Gestalt IT, 2012. | EMC VFCache (aka “Project Lightning”) Is One Small Step, But an Important One
Read more posts categorized as Server Virtualization, Storage

Today’s announcement of the E-Class storage array is an important milestone for Nimbus Data and solid-state storage in the enterprise. Until now, most solid-state storage arrays have been fairly small-scale, focused on point performance rather than enterprise-wide capacity. But the E-Class, which scales to 500 TB and sports a redundant, multi-protocol interface, is the first all-flash array to go toe to toe at the top of the market.

The State of Solid

No one would deny that solid-state storage is making a huge impact on the market. With mind-bending performance and reduced power requirements, flash memory matches up nicely with modern data center requirements. But the one missing element has always been capacity: NAND flash is more expensive than magnetic disks on a gigabyte by gigabyte basis.

Check out my 4 Horsemen series for more on the issues of disk!

The solid-state enterprise storage market started with point products targeted at specific needs within the data center. Companies like Texas Memory Systems and Violin have long supported the most challenging database applications with their external arrays, while Fusion-io, Virident, Micron and others stashed flash within the server. These companies were able to sell expensive storage into performance-hungry niches, but have found it difficult to address the capacity needs of the broader storage market.

Technology has the answer to this challenge, as demonstrated by Pure Storage, Nimbus, SolidFire, and others. Thin provisioning makes up much of the difference in cost, and deduplication or compression can even bring parity on a per-capacity cost basis. And even with these features turned on, solid-state storage arrays absolutely murder spinning disks in terms of random I/O performance.

Accumulating Nimbus

Unlike Whiptail, Pure Storage, Kaminario, SolidFire, and the rest of the startup crowd, Nimbus Data is not a new company. Founded by former TrueSAN wunderkind, Thomas Isakovich, Nimbus began as a disk storage startup before transitioning to an all-flash lineup two years ago. The company has steadily improved its product line over the years, adding NFS and SMB for a unified storage experience as well as InfiniBand for extreme performance.

Unlike most other companies in the space, Nimbus builds their own flash memory modules from raw NAND. This allows the company to avoid some of the tricky engineering required to qualify and adapt to the peculiarities of existing consumer or enterprise SSD modules. It also gives the company greater control and better flexibility to launch new capacity points when they are ready, rather than when their suppliers give the go-ahead.

Nimbus is rolling out a dual-PCB SSD module which doubles performance and capacity

The E-Class includes a new dual-PCB module which stripes data internally for better performance and capacity. This bumps each Nimbus E2000X enclosure to 20 TB, twice the capacity previously achieved, in just two rack units. And each of these enclosures draws as little as 100 W, allowing them to be stacked tall without exceeding the power capacity of typical datacenters.

E Is for Enterprise

The real innovation in the Nimbus E-Class is a brand-new “dual active” redundant controller architecture. Most previous solid-state arrays had a single controller, requiring users to mirror two entire arrays for high-availability. In contrast, most enterprise storage systems feature multiple active controllers with no single point of failure.

Read Multipath: Active/Passive, Dual Active, and Active/Active to understand what I’m on about here!

The E-Class introduces a new controller architecture for Nimbus. Each controller services all access to a LUN or volume until a failure is detected, in which case the alternate controller immediately comes online. But both controllers can have active storage at once, in what I call a “dual active” scenario. Although not truly “active/active”, the E-Class is in a different league from older single controller arrays.

Interestingly, the performance of Nimbus’ solid-state enclosures is high enough that the controllers do not need to mirror internal cache or hash tables. They simply write them out to SSD to be picked up in the event of a failure. This simplifies engineering for a dual controller system, and may lead to additional controllers added in the future.

Straightforward Pricing

Solid-state storage provides much more performance per dollar than spinning disk, but most customers still pay on a raw capacity basis. Rather than rocking the boat with alternative pricing models, Nimbus sticks to a straightforward method: $25,000 per controller, plus $100,000 per 10 TB enclosure. A minimum E-Class configuration includes two controllers and one enclosure for $150,000 with no extra cost for software licensing or features.

This seems fairly expensive for 10 TB of storage, but is actually quite competitive even with disk-based storage systems in the high-end, high-feature enterprise market. Thin provisioning increases the usable capacity of the E-Class, and the all SSD architecture means performance will not suffer. Unlike PCIe solutions, the E-Class is a shared, networked device and can be used by many servers at once.

Features and More Features

Speaking of features, Nimbus includes just about anything you could ask for in a storage array:

• In-line thin provisioning and deduplication
• Snapshots and synchronous or asynchronous replication
• 10 Gb NFS (2, 3, and 4) as well as SMB (CIFS/SMB1 and, SMB2)
• 10 Gb iSCSI
• 8 Gb Fibre Channel
• 40 Gb QDR InfiniBand

Although it has not yet been announced, Nimbus has added VMware VAAI support to the HALO operating system found in the S- and E-Class arrays. The company will support the major block storage components for vSphere 5: Block zeroing, atomic test and set, and full copy. Tom told me that Nimbus found the T10 interfaces fairly straightforward to implement but are still working on the NFS primitives. Although Nimbus does not yet have a vCenter plug-in, I expect that one is in the works.

Tom also tells me Nimbus is a “hypervisor hugger“, in that they intend to support features there rather than try to add them to the array. This is a smart choice for a smaller company, and I am glad to see Nimbus embracing the server virtualization market. I imagine an array like the E-Class would totally demolish any competing disk-based array in a virtual infrastructure deployment!

Stephen’s Stance

Nimbus has always been an interesting company, with a longer history in the storage world than most startups. Their switch to all-flash architecture was perfectly timed with the market shift, and the new E-Class comes just at the right moment. Boasting 500 TB of maximum capacity, a fully redundant “dual active” controller architecture, massive performance (even InfiniBand), and complete feature set (once VAAI is released), Nimbus may have hit on their hands.

© Stephen Foskett for Gestalt IT, 2012. | Nimbus E-Class: The First Big, Redundant, All-Flash Enterprise Array
Read more posts categorized as Storage

Stepping out of a subway car is an entirely different matter when it's moving!

Mobility of virtual machines is a sticky wicket. As open systems infrastructure becomes increasingly virtualized, administrators and managers wish to use the technology to balance workload, ease migration, and provide better availability. Although technology is improving, actually moving virtual machines is not always a piece of cake. Let’s lay down a baseline of information so we may begin a discussion on the true nature of virtual machine mobility.

Mobility of What?

Let us consider first the question of what exactly is being moved. Systems administrators often focus on “the machine”, which encompasses the operating system and configured state of the virtual machine itself. But the true “mass” of the system is its stored data. Hypervisor vendors have come up with different techniques of moving these two essential elements, reflecting the unique characteristics of each.

• The virtual machine is an instance of an operating system along with its state and configuration. Mobility of virtual machines requires all of this to be preserved, along with any I/O channels. Live migration of virtual machine requires that any active network sessions be maintained, along with RAM content, registers and buffers, and so many other elements.
• The virtual machine image (commonly referred to as “storage”) is the static content addressed by a virtual machine. Typically a VMDK or similar virtual disc image, it must be accessible to the virtual machine at all times. Live migration of a virtual machine image is tricky, but perhaps not quite as complex as live migration of a running operating system.

VMware, Microsoft, and others recognize these 2 distinct elements to be migrated, and have come up with a variety of complementary technologies for each:

• vMotion is VMware’s virtual machine migration solution, and has continually evolved with each iteration of the hypervisor. DRS leverages vMotion to automate mobility. VMware has also created Storage vMotion and Storage DRS as complements to handle mobility of virtual machine images.
• Microsoft Hyper-V Live Migration is conceptually similar to vMotion, though newer and less full-featured. With Hyper-V 3.0, Microsoft will introduce Storage Live Migration as a complementary technology akin to Storage vMotion. Most other virtual machine managers also support some form of live migration, though live migration of storage is less common.

Mobility in What State?

One of the key benefits of virtual machine technology is the ability to “run anywhere” on dissimilar hardware. From the very beginning, hypervisors have provided the ability to create a universal virtual machine image that would run on a variety of supported platforms.

This leads to one of the key values of server virtualization in the data center: Disaster recovery. The ability to take a virtual machine image and system state and bring it online after a disaster is a true revolution for open systems IT. The benefits of the single usage of server virtualization technology easily justify the investment to many businesses.

But this sort of “cold” migration seems passé when compared to the live or “hot” migration possible with technologies like VMware vMotion. Live migration is much more difficult, since active client sessions must be preserved in activity must not be greatly interrupted.

This is the second great question that must be asked when considering virtual machine mobility: In what state will the virtual machine be moved? Will it be a cold, powered down image of the system? A suspended or paused operating system image? Or a full, running machine?

Mobility to Where?

Once we have decided whether we are discussing virtual machine migration or movement of storage resources, we must consider the scope of the movement. The ability to move a virtual machine from one member of the cluster to another has now become fairly common. But what about systems that are not related in a cluster? Or that are spread over great distances?

• The nice thing about clusters is that they share resources before and after a virtual machine is moved. It is practical to move the running virtual machine, its storage, or both independently and to expect that performance will not dramatically suffer as a result. The cluster can also preserve network connections, and even I/O state, without much impact on clients or other external elements.
• It is a bit more difficult to move systems within a data center, since one must maintain the I/O connections that might be interrupted. It is fairly trivial to configure an IP network and storage array to allow multiple machines to access the same iSCSI or NFS storage resources. It is a little more difficult to configure Fibre Channel (and, by extension, FCoE) SAN’s to handle this sort of dynamic movement, but it is not impossible. Although moving a running machine from one network port to another could cause client access to be interrupted, technologies like VXLAN allow these sessions to continue, and improved network switching technology should reduce performance impact.
• Moving the machine to a different data center is another matter entirely. Stretching a layer-2 Ethernet LAN or Fibre Channel SAN across a metro or greater distance, while possible, will always be problematic. IP routing is flexible, but it takes time for changes to propagate when live machines are moved. And it is difficult to keep storage in sync over long distances due to the amount of time it takes for information to transit. Again, all of these challenges are being addressed in various ways, but they’re still hard!

Stephen’s Stance

Moving cold virtual machine images from system to system, or even across great distances, is one of the main selling points of server virtualization. But it becomes much more difficult to manage movement of virtual machines that are still running, especially outside cluster or across WAN links. When talking about virtual machine mobility, it is important to consider what is being moved, the state it is in, and where it is going.

Note: This discussion is part of “Building Virtual Infrastructure”, my new seminar series with Truth in IT.

© sfoskett for Stephen Foskett, Pack Rat, 2012. |
Virtual Machine Mobility: Of What, and to Where and in What State?


This post was categorized as Enterprise storage, Features, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2012. | Virtual Machine Mobility: Of What, and to Where and in What State?
Read more posts categorized as Storage

I Promise Not To Quote That Old Annoying Dell PC Slogan

I’ve been buying Dell computers for decades, but not really because I loved them. Sure, my XPS laptop was awesome, but it burned out its motherboard and I never really touched the RMA replacement, having bought a MacBook Pro in the meantime. Enterprise buyers seem to have the same ambivalence about Dell. They buy it, but I’m not sure they really “buy” the company as an IT partner.

I’m sure you’ve heard the same comments as me: “Dell just assembles off-the-shelf components and sells them in volume” or “Dell’s a follower, not an innovator.” There seems to be a great deal of respect for Dell’s ability to produce competitive products and sell them at reasonable cost. Truly, most of their competitors would love to have this kind of reputation. But most of their competitors also have a reputation for partnership, innovation, and solution selling.

Dell Is Making An Effort

It seems clear that Dell would like to change this attitude, and they are investing serious resources to make it happen. While acquisitions like Compellent and Force10 raised eyebrows in storage and networking, it is the activity I see behind the scenes that paints the clearest picture. Dell isn’t just buying into new markets, they’re investing to change the company.

When Dell acquired EqualLogic in 2008, many assumed it was a tactical investment to increase margins over the (resold) EMC storage equipment the company was then pushing. Pundits were similarly dismissive of the acquisition of Perot Systems in 2009, calling it a “me too” effort after HP acquired rival EDS. Regardless of the motivations, however, Dell was becoming more of a serious challenger to HP and IBM every day.

After failing to acquire 3PAR in 2010, then picking up Compllent shortly after, accusations that Dell was “mini me” to HP were rampant. But HP stumbled mightily in 2011, and many in IT quickly lost confidence in that company’s management. All the while, Dell moved forward, increasing in-house IP and expanding enterprise offerings.

What Is The Result?

Today, one sees a very different landscape than just last year. Dell’s acquisitions focused on some of the ripest spots in storage and networking, and no one would disagree that the company has the ability strongly to push these products. Compellent and Force10 went from interesting startups to serious contenders overnight.

Dell really pulled out all the stops to tell us they are "Fluid by Design"

More importantly, Dell has retained much of the innovation these companies offered, from employees to support programs. Last week, I attended the Dell Storage Forum in London, an event initiated by Compellent prior to the acquisition. At the event, I talked to many Dell employees who came to the company through acquisition but had now been given power to challenge the status quo in their respective areas.

If Dell really intended only to push product, why retain marketing personnel? Why invest in the Dell Storage Forum? Why continue Compellent’s beloved Co-Pilot support program?

Then there are the products. Dell leveraged its investment in Ocarina Networks to create a deduplicating backup appliance, the new DR4000. They salvaged file system startup ExaNet and are beginning to bring scale out technology to market. The latest revision of the Compellent software finally brings it to parity in terms of VMware support. And Dell is really working to sell their DX Object Store.

This is the sort of activity one would expect from a contender, not a “box pusher”.

Stephen’s Stance

In the words of Malcolm Reynolds, my days of not taking Dell seriously are certainly coming to a middle. Dell is investing in product IP, innovative marketing and PR events, customer support, and personnel. This does not mean that Dell is instantly a player in the enterprise storage and networking markets, or that all this work will pay off. But I don’t laugh when I hear Dell boast that they intend to be a “top three” enterprise storage company in a few years. It could happen.

Disclaimer: Dell sponsored two Tech Field Day events in 2011, paid me as a speaker at two DX events, and paid for my trip to Dell Storage Forum in London. But no one can buy a post on this site, and I did similar business with IBM, HP, Cisco, and many other companies. This is my opinion.

© sfoskett for Stephen Foskett, Pack Rat, 2012. |
Why Should Anyone Take Dell Seriously in Enterprise Storage?


This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2012. | Why Should Anyone Take Dell Seriously in Enterprise Storage?
Read more posts categorized as Storage

It's not going to be this easy to bridge Fibre Channel and Ethernet!

Before the holidays, I posed a question on Google+ that generated quite a bit of interest and feedback. Now that it has settled down a bit I’d like to summarize the unresolved elements to make FCoE truly a world-class storage interconnect.

Setting the Stage

FCoE has been a controversial topic in both storage and networking, and for good reason. No one would deny that Ethernet is not an ideal transport mechanism for block storage I/O. “Porting” Fibre Channel to run on Ethernet networks has been a supreme technical challenge, and many companies and individuals have labored long and hard to make FCoE a reality.

Now that FCoE is specified in the standard and has been deployed in production environments, the question turns to its future. Will it take off and seize the mantle of dominance currently held by what I like retroactively to call “Fibre Channel over Fibre Channel?” Will they coexist for the next decade, with FCoE mainly deployed in “block” environments such as Cisco UCS? Or will FCoE ultimately fail to catch on, displaced by some other storage protocol like plain FC, iSCSI, NFS, or something entirely different?

The data center needs a flexible new protocol to meet the needs of virtual environments, and convergence of storage and data networking makes a great deal of sense in these environments. This was the root of my question, and I ask it in all earnestness.

My question: What elements remain unresolved to make FCoE truly world-class? What should the vendors be prioritizing? Here are the answers I received.

Technical Considerations

Link Aggregation on CNA’s

Converged network adapters (CNA’s) allow multiple protocols to access a single Ethernet connection, but some also include multiple ports that can be aggregated. In traditional Ethernet networks, link aggregation is a respectable approach for performance and availability. But storage networks have traditionally relied on host-based MPIO software, and these features are mutually exclusive. The zeitgeist seems to be a recommendation to avoid link aggregation on CNA’s that are used for storage networks.

How Do You Handle Virtual Machine Mobility?

As I described recently, virtual machine mobility is a major technical challenge for existing networks. The VMware proposal, the VXLAN, seems to be gaining traction right now. But this is only a solution for data networking. How will FCoE SANs handle virtual machine mobility? This remains unresolved as far as I can tell, though Ethernet switch vendors have come up with their own answers. Brocade demonstrated just such a solution at Networking Field Day 2, and I know that others have answers as well. But will there be an interoperable industry solution?

How Should FCoE Be Implemented Over Longer Distances?

Fibre Channel has traditionally relied on routers and other protocols (FCIP and iFCP) to span distances, but FCoE raises the possibility of native traversal. While it is certainly possible to span distances with FCoE, this is definitely not a recommended or supported idea. Without TCP/IP, or any routing mechanism, it’s just a bad idea. But I imagine that it won’t be long before vendors decide to give it a go anyway.

Implementation Considerations

Is TRILL Required for FCoE Networks?

This has been one of my own questions since the very beginning. Clearly, edge only FCoE works just fine without TRILL. But as networks become more complicated, and virtual machines move, it seems an awfully good idea to have some protocol to alleviate East-West routing concerns. I feel much better with TRILL (or some similar Ethernet fabric technology) in a complicated FCoE network.

Should All Switches Be Full FC Forwarders?

There are number of ways to implement FCoE on Ethernet network, and not all involve building a full Fibre Channel stack in each switch. While many (including myself) assumed that FCoE implied Fibre Channel forwarding in all switches, this is clearly not the direction taken by vendors, at least initially. Perhaps the current “Ethernet forwarding” approach is only a stepping stone, or perhaps it will emerge as the dominant FCoE standard.

How Will OpenFCoE and LoM Be Used?

OpenFCoE is a software solution allowing FCoE to be run without a CNA. If this became popular, it wouldn’t be long before data center architects began looking at LAN on Motherboard (LoM) and even 10GBASE-T as a potential SAN alternative. Will this be used in the long run? It could happen, but it’s certainly not something that’s here at the moment. But OpenFCoE is a real player, especially with Intel’s backing.

How Will Technologies like Zoning Interoperate?

Many networkers are just now beginning to see the true complexity of Fibre Channel SANs. Although interoperability of higher-level Fibre Channel functions between vendors has never been a priority in “FC over FC” SANs, Ethernet could change things. I would not be at all surprised to see a groundswell of customer support demanding greater levels of interoperability from FCoE than from FC, and zoning and VSAN is the likely first beachhead.

The Big Question: When Will We See the “Killer App” For FCoE

Just about everyone agreed that the real challenge for FCoE is market acceptance. Customers aren’t yet demanding FCoE, and vendors are finding it hard to articulate a compelling case to move from “tried-and-true” FC. Convergence, cost savings, and performance have all been put forth, but customers aren’t biting. Perhaps they just need a little time and a little more proof.

This post relies extensively on feedback from a number of people, including Ivan Pepelnjak, Tony Bourke, J Metz, Dmitri Kalintsev, Derick Winkworth, David Hardaker, Juan Lage, and Corey Hines.

© sfoskett for Stephen Foskett, Pack Rat, 2012. |
Eight Unresolved Questions About FCoE


This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2012. | Eight Unresolved Questions About FCoE
Read more posts categorized as Storage

Windows 8 server editions will include a filter driver for NTFS for data deduplication

The next version of Microsoft Windows Server includes integrated data deduplication technology. Microsoft is positioning this as a boon for server virtualization and claims it has very little performance impact. But how exactly does Microsoft’s de-duplication technology work?

Introducing Windows 8 Deduplication

Let’s make one thing clear right from the start: Microsoft started from a clean sheet and invented their own deduplication technology. This is not a licensed, cloned, or copied feature as far as I can tell. There are some clever aspects to it, along with a few head scratchers for folks like me who’ve seen lots of different deduplication approaches.

Microsoft’s deduplication is layered onto NTFS in Windows 8, and will be a feature add-on for Server users. It is implemented as a filter driver on a per volume basis, with each volume a complete, self describing unit. It is cluster aware, and fully crash consistent on all operations. This is a pretty neat trick: As is typical for Microsoft, deduplication will be a simple, transparent feature.

Now let’s talk for a moment about what Windows 8 deduplication is not.

• It is a server-only feature, like so many of Microsoft’s storage developments. But perhaps we might see it deployed in low-end or home servers in the future.
• It is not supported on boot or system volumes.
• Although it should work just fine on removable drives, deduplication requires NTFS so you can forget about FAT or exFAT. And of course the connected system must be running a server edition of Windows 8.
• Although deduplication does not work with clustered shared volumes, it is supported in Hyper-V configurations that do not use CSV.
• Finally, deduplication does not function on encrypted files, files with extended attributes, tiny (less than 64 kB) files, or re-parse points.

Some Technical Details on Deduplication in Windows 8

Microsoft Research spent 2 years experimenting with algorithms to find the “cheapest” in terms of overhead. They select a chunk size  for each data set. This is typically between 32 KB and 128 KB, but smaller chunks can be created as well. Microsoft claims that most real-world use cases are about 80 KB. The system processes all the data looking for “fingerprints” of split points and selects the “best” on the fly for each file.

After data is de-duplicated, Microsoft compresses the chunks and stores them in a special “chunk store” within NTFS. This is actually  part of the System Volume store in the root of the volume, so dedupe is volume-level. The entire setup is self describing, so a deduplication NTFS volume can be read by another server without any external data.

There is some redundancy in the system as well. Any chunk that is referenced more than x times (100 by default) will be kept in a second location. All data in the filesystem is checksummed and will be proactively repaired. The same is done for the metadata. The deduplication service includes a scrubbing job as well as a file system optimization task to keep everything running smoothly.

Windows 8 deduplication cooperates with other elements of the operating system. The Windows caching layer is dedupe-aware, and this will greatly accelerate overall performance. Windows 8 also includes a new “express” library that makes compression “20 times faster”. Compressed files are not re-compressed based on filetype, so zip files, Office 2007+ files, etc will be skipped and just deduped.

New writes are not deduped – this is a post-process technology. The data deduplication service can be scheduled or can run in “background mode” and wait for idle time. Therefore, I/O impact is between “none and 2x” depending on type. Opening a file is less than 3% greater I/O and can be faster if it’s cached. Copying a large file can make some difference (e.g. 10 GB VHD) since it adds additional disk seeks, but multiple concurrent copies that share data can actually improve performance.

Stephen’s Stance

Although I am intrigued by Microsoft’s new deduplication technology in Windows 8 server, I still have many questions about its usefulness and impact on performance. Concentrating duplicate data in the system volume makes sense from a technical perspective, but could lead to an I/O hotspot on the disk. This is especially true for external caching storage systems, since there is no integration between Microsoft deduplication and storage array features. I am particularly concerned about the use of deduplication with VHD files in Hyper-V, since it could eat up valuable system RAM and impact I/O performance.

If you would like to try Microsoft deduplication for yourself, I am happy to report that it is included in the developer preview of Windows 8 that is available on Dev Center. Here are a few commands to get you started, and read Rick Vanover’s post too!

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication
Import-Module Deduplication
Enable-DedupVolume E:
get-dedupvolume

Note: I am a Microsoft MVP and Microsoft briefs me on upcoming technologies under NDA. This post is based on a Microsoft briefing from November which was said at the time not to be covered by any NDA. All of this information could be gleaned by experimenting with the Windows 8 developer preview, but it’s much easier to just go to the source.

© Stephen Foskett for Gestalt IT, 2012. | Microsoft Adds Data Deduplication to NTFS in Windows 8
Read more posts categorized as Server Virtualization, Storage

It isn’t always easy to get where you need to go!

Consider the following situation: You go to lunch with your good friends, John and Mary. Halfway through a rousing discussion of the latest Hollywood movie, Mary starts talking about the fantastic action sequences while John criticizes the romantic angle. You realize something mine-bending has happened: John now has Mary’s personality, and vice versa. It’s like they have switched brains or something!

This truly weird situation isn’t likely to happen in person, but occurs all the time in the data center. Virtualization of server, network, and storage services illuminates the link between physical resources and functional applications. A running virtual machine can instantly move from one server, network adapter, HBA, or LUN to another. And when it happens, traditional components have no idea how to react.

The Challenges of Mobility

Mobility is perhaps the “killer app” of virtualization, but it is also the killer of traditional IT systems. Let’s consider the challenges of this “Twilight Zone” moment.

• The operating system expects a consistent hardware environment, which is exactly what the hypervisor creates
• The LAN must be prepared to redirect all network traffic instantly and seamlessly to one or more new physical interfaces
• The SAN similarly must be able to reroute all I/O to a new pair of HBA’s without missing a beat
• The storage array must be able to re-present capacity to a new physical device, and must maintain snapshots and other configurations
• The backup system must also be able to maintain consistency over time even as machines relocate to different server and storage locations

All of this must be done while maintaining quality of service (QoS), access control, reporting, and appropriate segmentation at all levels. This is an incredibly challenging task, and no conventional protocol (IP, Ethernet, NFS, SCSI, Fibre Channel, etc.) is anymore ready then you are when you’re good friends switch personalities.

Two Paths

So much of the development that is currently taking place in IT focuses on accommodating this “mobility issue”. Two key approaches have emerged to take on this challenge:

• “In a vacuum” technologies (like VXLAN) assume that no other changes will be made, so the focus is on maintaining complete compatibility in front and behind
• “Clean sheet” technologies (usually from startups) take a different approach, throwing out compatibility in favor of technical elegance

Both of these approaches have merit. Attempting to maintain compatibility only works so far (just ask a Windows API programmer), but it leverages the existing environment and recognizes that most people are not ready for wholesale change. Clean sheet designs always make more sense, but they rarely attain mass acceptance. Nearly every technology we rely on today is full of bolt-ons in the name of compatibility. Some, like Ethernet and x86, actually work pretty well, too.

The Stack of Lies

The difference between virtualization and cloud computing is exactly this same distinction. Hypervisors, NPV, NAT, thin provisioning, and so many other virtualization technologies exist mainly to maintain compatibility in a vacuum. In contrast, true cloud computing dispenses with the entire stack and creates a new platform for applications.

This is, perhaps, the reason that cloud computing is not taken off in the enterprise. Simply put, IT is not prepared to ditch everything they have ever used even in the face of a demonstrably superior alternative. Currently, the highest use of cloud is behind gateways and virtualization engines that bring it back down to earth.

This brings us to the stack of lies called server virtualization. Any “modern” virtualized data center is built on lie after lie, with each level telling the other what it wants to hear. The volume manager lies to the operating system, hypervisor lies to the volume manager, and the storage array lies to the hypervisor. The same sad state of affairs allows networking and even memory to function in a virtual world.

But these shaky stacks of lies have difficulty adapting to motion, since no level truly “knows” the reality of the world around. The depressing truth is that a bowl of spaghetti like VXLAN is perhaps the highest form of art we can expect in a virtual data center.

Stephen’s Stance

As a techie, I am always drawn to clean sheet designs that offer technical elegance along with functionality. But I know that, realistically, products that assume nothing about the world around them and bend over backward to maintain compatibility are more likely to succeed. Still, I maintain hope that the issues of virtual machine mobility will be solved in an elegant way, rather than adding to the “stack of lies”.

© Stephen Foskett for Gestalt IT, 2012. | The Terrifying True Story Of Virtual Machine Mobility
Read more posts categorized as Server Virtualization, Storage

The great battle of enterprise storage is on!

The time has come to take sides on the core question of storage for virtual servers: Do you want storage intelligence to live in the hypervisor or the array? Most administrators are already lining up on one side or the other, unintentionally casting their vote while the rest flounder. But the storage industry must wake up and embrace the divide.

Hypervisor Huggers Unite!

Do you “heart” virtualization?

VMware’s vSphere dominates the world of enterprise server virtualization and has become the core element of the modern open systems datacenter. Microsoft recognizes this but has been unable to field a competitive hypervisor ecosystem for the virtual datacenter. Today, vSphere is the state of the art and nowhere is this more apparent than in storage.

In just a few years, VMware has delivered and updated a host of advanced storage functionality, from provisioning to migration and load balancing to backup and data protection. vSphere 5 includes an advanced and scalable storage virtualization layer, delivering everything a datacenter needs. VMFS sculpts basic block storage into a shared resource for virtual machines, with snapshots, policy-based layout and movement, and flexible allocation and thin provisioning.

Most VMware administrators are “server guys” and relish these features. They have never experienced an automated “storage service” like this, and the enterprise storage world has never been able to construct anything remotely as flexible, user-friendly, and functional. And Hypervisor Huggers don’t need complex enterprise storage arrays to do it: They can use basic iSCSI or Fibre Channel devices to provide performance and capacity and let VMware do the rest!

Storage DRS is exemplary of the new virtual datacenter world. Introduced in vSphere 5 (and restricted to the pricey Enterprise Plus license), Storage DRS uses the core technology of Storage vMotion to dynamically balance I/O and capacity across a diverse pool of storage. Storage DRS even uses Policy-Driven Storage and VASA to enforce tiered storage and data placement strategy. This kind of virtualization has been a “holy grail” quest for the enterprise storage industry, but they’ve never delivered on their promises.

Cheers for Storage Stalwarts!

Do you wish all this virtualization nonsense would just go away?

But not every IT environment wants be 100% vSphere focused, and many aren’t convinced that dumb storage is the smartest place for their data. These Storage Stalwarts want smarter and better-integrated storage arrays, and VMware is innovating here as well.

VMware’s Storage APIs for Array Integration (VAAI) technology is squarely aimed at this market. VAAI allows vSphere to hand off heavy storage operations to the high-end storage arrays from the major players. It works transparently, too, coordinating cloning without the kind of scripting and hair-pulling that used to require. VAAI in vSphere 4.1 also includes block zeroing support and something called “atomic test and set” which we’ll get to in a moment. Microsoft announced their own cloning integration, ODX, but it won’t ship until Windows Server 8 appears sometime next year.

But cloning is just the tip of the iceberg when it comes to Hypervisor-to-array integration. The rising army of NAS users have seen the glory of more-complete array integration for a while, and they’re not quiet about it. They love that VMware’s NFS protocol support makes storage “disappear” in vCenter, becoming just another resource with integrated thin provisioning and flexible allocation and movement.

VMware is moving aggressively to please their Storage Stalwarts, adding more VAAI support for block and file in vSphere 5. But, as the company laid out at VMworld 2011, neither access method is ideal for virtual servers. So VMware has been pushing the enterprise array vendors for ever-greater integration. They see a future where a VAAI-based protocol enables arrays to de-multiplex I/O streams from the hypervisor and intelligently handle per-VM data.

Stephen’s Stance

You can spot a Hypervisor Hugger by their big LUNs: They would rather treat storage as a bulk commodity, and array vendors should be lining up to get their business. Storage Stalwarts will jump on each new VMware innovation, finally making use of the capabilities they have spent over a decade paying for but not utilizing. The only untenable stance is trying to keep a foot in both worlds: It’s foolish to buy an enterprise array and use it as bulk storage!

© Stephen Foskett for Gestalt IT, 2011. | Are You a Hypervisor Hugger or a Storage Stalwart?
Read more posts categorized as Featured, Server Virtualization, Storage