Stepping out of a subway car is an entirely different matter when it's moving!

Mobility of virtual machines is a sticky wicket. As open systems infrastructure becomes increasingly virtualized, administrators and managers wish to use the technology to balance workload, ease migration, and provide better availability. Although technology is improving, actually moving virtual machines is not always a piece of cake. Let’s lay down a baseline of information so we may begin a discussion on the true nature of virtual machine mobility.

Mobility of What?

Let us consider first the question of what exactly is being moved. Systems administrators often focus on “the machine”, which encompasses the operating system and configured state of the virtual machine itself. But the true “mass” of the system is its stored data. Hypervisor vendors have come up with different techniques of moving these two essential elements, reflecting the unique characteristics of each.

• The virtual machine is an instance of an operating system along with its state and configuration. Mobility of virtual machines requires all of this to be preserved, along with any I/O channels. Live migration of virtual machine requires that any active network sessions be maintained, along with RAM content, registers and buffers, and so many other elements.
• The virtual machine image (commonly referred to as “storage”) is the static content addressed by a virtual machine. Typically a VMDK or similar virtual disc image, it must be accessible to the virtual machine at all times. Live migration of a virtual machine image is tricky, but perhaps not quite as complex as live migration of a running operating system.

VMware, Microsoft, and others recognize these 2 distinct elements to be migrated, and have come up with a variety of complementary technologies for each:

• vMotion is VMware’s virtual machine migration solution, and has continually evolved with each iteration of the hypervisor. DRS leverages vMotion to automate mobility. VMware has also created Storage vMotion and Storage DRS as complements to handle mobility of virtual machine images.
• Microsoft Hyper-V Live Migration is conceptually similar to vMotion, though newer and less full-featured. With Hyper-V 3.0, Microsoft will introduce Storage Live Migration as a complementary technology akin to Storage vMotion. Most other virtual machine managers also support some form of live migration, though live migration of storage is less common.

Mobility in What State?

One of the key benefits of virtual machine technology is the ability to “run anywhere” on dissimilar hardware. From the very beginning, hypervisors have provided the ability to create a universal virtual machine image that would run on a variety of supported platforms.

This leads to one of the key values of server virtualization in the data center: Disaster recovery. The ability to take a virtual machine image and system state and bring it online after a disaster is a true revolution for open systems IT. The benefits of the single usage of server virtualization technology easily justify the investment to many businesses.

But this sort of “cold” migration seems passé when compared to the live or “hot” migration possible with technologies like VMware vMotion. Live migration is much more difficult, since active client sessions must be preserved in activity must not be greatly interrupted.

This is the second great question that must be asked when considering virtual machine mobility: In what state will the virtual machine be moved? Will it be a cold, powered down image of the system? A suspended or paused operating system image? Or a full, running machine?

Mobility to Where?

Once we have decided whether we are discussing virtual machine migration or movement of storage resources, we must consider the scope of the movement. The ability to move a virtual machine from one member of the cluster to another has now become fairly common. But what about systems that are not related in a cluster? Or that are spread over great distances?

• The nice thing about clusters is that they share resources before and after a virtual machine is moved. It is practical to move the running virtual machine, its storage, or both independently and to expect that performance will not dramatically suffer as a result. The cluster can also preserve network connections, and even I/O state, without much impact on clients or other external elements.
• It is a bit more difficult to move systems within a data center, since one must maintain the I/O connections that might be interrupted. It is fairly trivial to configure an IP network and storage array to allow multiple machines to access the same iSCSI or NFS storage resources. It is a little more difficult to configure Fibre Channel (and, by extension, FCoE) SAN’s to handle this sort of dynamic movement, but it is not impossible. Although moving a running machine from one network port to another could cause client access to be interrupted, technologies like VXLAN allow these sessions to continue, and improved network switching technology should reduce performance impact.
• Moving the machine to a different data center is another matter entirely. Stretching a layer-2 Ethernet LAN or Fibre Channel SAN across a metro or greater distance, while possible, will always be problematic. IP routing is flexible, but it takes time for changes to propagate when live machines are moved. And it is difficult to keep storage in sync over long distances due to the amount of time it takes for information to transit. Again, all of these challenges are being addressed in various ways, but they’re still hard!

Stephen’s Stance

Moving cold virtual machine images from system to system, or even across great distances, is one of the main selling points of server virtualization. But it becomes much more difficult to manage movement of virtual machines that are still running, especially outside cluster or across WAN links. When talking about virtual machine mobility, it is important to consider what is being moved, the state it is in, and where it is going.

Note: This discussion is part of “Building Virtual Infrastructure”, my new seminar series with Truth in IT.

© sfoskett for Stephen Foskett, Pack Rat, 2012. |
Virtual Machine Mobility: Of What, and to Where and in What State?


This post was categorized as Enterprise storage, Features, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2012. | Virtual Machine Mobility: Of What, and to Where and in What State?
Read more posts categorized as Storage

Windows 8 server editions will include a filter driver for NTFS for data deduplication

The next version of Microsoft Windows Server includes integrated data deduplication technology. Microsoft is positioning this as a boon for server virtualization and claims it has very little performance impact. But how exactly does Microsoft’s de-duplication technology work?

Introducing Windows 8 Deduplication

Let’s make one thing clear right from the start: Microsoft started from a clean sheet and invented their own deduplication technology. This is not a licensed, cloned, or copied feature as far as I can tell. There are some clever aspects to it, along with a few head scratchers for folks like me who’ve seen lots of different deduplication approaches.

Microsoft’s deduplication is layered onto NTFS in Windows 8, and will be a feature add-on for Server users. It is implemented as a filter driver on a per volume basis, with each volume a complete, self describing unit. It is cluster aware, and fully crash consistent on all operations. This is a pretty neat trick: As is typical for Microsoft, deduplication will be a simple, transparent feature.

Now let’s talk for a moment about what Windows 8 deduplication is not.

• It is a server-only feature, like so many of Microsoft’s storage developments. But perhaps we might see it deployed in low-end or home servers in the future.
• It is not supported on boot or system volumes.
• Although it should work just fine on removable drives, deduplication requires NTFS so you can forget about FAT or exFAT. And of course the connected system must be running a server edition of Windows 8.
• Although deduplication does not work with clustered shared volumes, it is supported in Hyper-V configurations that do not use CSV.
• Finally, deduplication does not function on encrypted files, files with extended attributes, tiny (less than 64 kB) files, or re-parse points.

Some Technical Details on Deduplication in Windows 8

Microsoft Research spent 2 years experimenting with algorithms to find the “cheapest” in terms of overhead. They select a chunk size  for each data set. This is typically between 32 KB and 128 KB, but smaller chunks can be created as well. Microsoft claims that most real-world use cases are about 80 KB. The system processes all the data looking for “fingerprints” of split points and selects the “best” on the fly for each file.

After data is de-duplicated, Microsoft compresses the chunks and stores them in a special “chunk store” within NTFS. This is actually  part of the System Volume store in the root of the volume, so dedupe is volume-level. The entire setup is self describing, so a deduplication NTFS volume can be read by another server without any external data.

There is some redundancy in the system as well. Any chunk that is referenced more than x times (100 by default) will be kept in a second location. All data in the filesystem is checksummed and will be proactively repaired. The same is done for the metadata. The deduplication service includes a scrubbing job as well as a file system optimization task to keep everything running smoothly.

Windows 8 deduplication cooperates with other elements of the operating system. The Windows caching layer is dedupe-aware, and this will greatly accelerate overall performance. Windows 8 also includes a new “express” library that makes compression “20 times faster”. Compressed files are not re-compressed based on filetype, so zip files, Office 2007+ files, etc will be skipped and just deduped.

New writes are not deduped – this is a post-process technology. The data deduplication service can be scheduled or can run in “background mode” and wait for idle time. Therefore, I/O impact is between “none and 2x” depending on type. Opening a file is less than 3% greater I/O and can be faster if it’s cached. Copying a large file can make some difference (e.g. 10 GB VHD) since it adds additional disk seeks, but multiple concurrent copies that share data can actually improve performance.

Stephen’s Stance

Although I am intrigued by Microsoft’s new deduplication technology in Windows 8 server, I still have many questions about its usefulness and impact on performance. Concentrating duplicate data in the system volume makes sense from a technical perspective, but could lead to an I/O hotspot on the disk. This is especially true for external caching storage systems, since there is no integration between Microsoft deduplication and storage array features. I am particularly concerned about the use of deduplication with VHD files in Hyper-V, since it could eat up valuable system RAM and impact I/O performance.

If you would like to try Microsoft deduplication for yourself, I am happy to report that it is included in the developer preview of Windows 8 that is available on Dev Center. Here are a few commands to get you started, and read Rick Vanover’s post too!

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication
Import-Module Deduplication
Enable-DedupVolume E:
get-dedupvolume

Note: I am a Microsoft MVP and Microsoft briefs me on upcoming technologies under NDA. This post is based on a Microsoft briefing from November which was said at the time not to be covered by any NDA. All of this information could be gleaned by experimenting with the Windows 8 developer preview, but it’s much easier to just go to the source.

© Stephen Foskett for Gestalt IT, 2012. | Microsoft Adds Data Deduplication to NTFS in Windows 8
Read more posts categorized as Server Virtualization, Storage

Much has been said about virtualization platform interoperability. VMware is constantly pushing forward with trying to achieve ANSI standards for common virtualization components and functions. One of which is the most useful… the virtual hard disk format. In theory, I should be able to take a virtual disk from my Hyper-V environment and import into my vSphere environment. I have seen this work on my workstation when I imported my Windows XP Mode drive into my Workstation setup!

However, the non-VMware virtualization providers have been combining their forces into a new cloud computing platform, called OpenStack (slogan: “Open source software to build private and public clouds”). OpenStack is a software layer that supports the operation of multiple hypervisors… including Xen, KVM, QEMU, and UML (User Mode Linux).

Installation (per installation instructions) occurs on a Debian/Ubuntu based OS. So, out of the box, it appears as though we are going to be working with a potential Type 2 hypervisor…or multiple hypervisors on a single physical platform (?!). Resource management between the multiple hypervisors looks to be interesting, then.

Much of the effort has been focused on “avoiding vendor lock-in” in virtualization services. Apparently, people do not like being locked into a single vendor for services. Although, I would reckon that any of the virtualization companies involved with the OpenStack would love to be in the situation VMware finds themselves in… at the forefront of the virtualization market. I would like to go out on a limb and state that VMware loves the success it finds, but it is still working on interoperability between virtualization providers and products. Look at the ANSI work that VMware has gone through for virtualization standards. Additionally, the conscious decision to allow 3rd party providers to interact with the vSphere environment (ex: offloaded VM antivirus, backups, monitoring, etc…). Hyper-V and XenServer do not have this level of flexibility. The VMware ecosystem is such that many other vendor products (ex: System Center Virtual Machine Manager) can manage them. The same cannot be said for the other vendors.

Surprisingly, what is bringing this to the forefront of the virtualization industry is the inclusion of Microsoft into the mix. Suddenly, a known closed source company and one of the most prolific companies that distribute a vendor lock in set of products is trying to get in on the action. Prior to this, it could be said that OpenStack was primarily open source hypervisor products. However, with the inclusion of the Hyper-V hypervisor, that model has changed.

The inclusion of Hyper-V really opens an interesting avenue of discussion, though. What is Microsoft’s intention here and how is it going about it? Are they going to open Hyper-V to the open source community? Will this run natively in the Linux environment the other hypervisors are running in or will any hosting provider need to operate a Windows environment to support this?

Supporting multiple hypervisors can and will lead to nightmares as each hypervisor product is being developed independently of one another. So, it is possible that one hypervisor is being developed at 2x the rate as the others. How are updates going to be handled? What if one requires different libraries than another?

What makes the big three virtualization vendors so useful is the inclusion of sophisticated and intuitive GUI and APIs for management. However, the management of the environment appears to be restricted to the CLI of the host server. Right away, this will drive away any non-savvy customers. So, I doubt that customers will flock to the OpenStack environment en masse. Sure, some customers have the resources to handle this, but not enough to make a difference… Statement: yes. Impact: no.

As far as cloud infrastructure is concerned, what is wrong with vendor lock in, really? The virtualization providers know how to interact with their own products and drastically increase their functionality and power. Sticking with VMware ensures compatibility and functionality across products. What is the need to run virtual machines in a Hyper-V environment locally and UML in the cloud? While the servers could interact with each other, management functions will differ significantly, and the portability is drastically limited. Contrast this with using the VMware vCloud initiative in your local and public clouds. Management and APIs are identical regardless of the location of the virtual machines (either in Public or Private clouds). Portability is not an option because the virtual machines exist on a common platform.

My feeling is that this is very much about some lesser virtualization providers ganging up to try and beat VMware in the cloud infrastructure game. While the intent is great, the complexity is much greater as there are more hypervisors to support and potential for instability and abnormal product growth. Rather, this is more of a proof of concept project for what “the cloud should be like”. However, the OpenStack environment and the other virtualization providers should take note of the VMware vCloud initiative as it is showing how true cloud operation should work.

Links:

OpenStack:http://www.openstack.org/

Installation Instructions: http://wiki.openstack.org/NovaInstall

http://ostatic.com/blog/as-microsoft-and-vmware-square-off-keep-your-eyes-on-openstack

http://gigaom.com/cloud/microsoft-joins-openstack-to-add-hyper-v-support/

http://cloudcomputing.info/en/news/2010/07/vmware-answers-to-rackspace-on-openstack.html

© Bill for Gestalt IT, 2010. | Cloud Fight – OpenStack Steps Up
Read more posts categorized as Cloud Computing, Server Virtualization

The discussion was wide-ranging, and we’ve split it into two episodes. This second episode includes discussion of the following:

• HP’s unique position as a soup-to-nuts infrastructure supplier
• The enterprise readiness of 3COM, D-Link, Virtual Connect, Xsigo
• Cisco’s secret plan with UCS marketing
• The concept of integrating the VMware hypervisor as firmware in the stack
• The exact definition of stack – what does it really mean?
• Is there a difference between a “marketing stack” or reference architecture and a “real stack”?
• What is the compelling economic driver that makes people move to vertical integration?
• What is VMware’s long-term strategy for stack wars?
• Was vBlock designed for the cloud?

Joining the call are the following Gestalt IT authors and Field Day delegates:

• Jason Boche (@JasonBoche, Boche.net)
• Rich Brambley (@RBrambley, VM /ETC)
• Carlo Costanzo (@CCostan, VMware Info)
• Chris Evans (@ChrisMEvans, The Storage Architect)
• Greg Ferro (@EtherealMind, EtherealMind)
• Edward Haletky (@Texiwill, The Virtualization Practice)
• Joerg Hallbauer (, Joerg’s Storage Blog)
• Robin Harris (@StorageMojo, Storage Mojo)
• Rod Haywood (@Rodos, Musings of Rodos)
• John Hickson (@StudioSystems, Studio Sysadmins)
• Scott Lowe (@Scott_Lowe, ScottLowe.org)
• John Obeto (@JohnObeto, Absolutely Windows)
• Devang Panchigar (@StorageNerve, StorageNerve)
• Bas Raayman (@BasRaayman, Bas Raayman)
• Ed Saipetch (@EdSai, Breathing Data)

Download the stack wars roundtable podcast now, or subscribe in iTunes to tune in to the discussion!

© Stephen Foskett for Gestalt IT, 2010. | Podcast 7: Stack Wars Roundtable 2
Read more posts categorized as Cloud Computing, Exclusive, Featured, Server Virtualization, Tech Field Day Roundtable Podcast, Top Story

But what does this all mean to enterprise IT? Who benefits from these vertically-integrated stacks? Who stands to lose out? To paraphrase the wise Jedi master, Yoda, “cloudy, the future is.” So, the stack wars have begun!

What’s a Stack?

The largest IT infrastructure vendors are aligning their own products and joining together in alliances to create vertically-integrated “datacenter in a box” stacks that include server, storage, networking, and higher-level software for virtualization and management. Cisco and EMC are certainly at the forefront of this development with their Acadia VCE vBlock, as demonstrated and deconstructed at our Tech Field Day earlier this month. The reinvigorated HP has its own offering in this space, and is unique in its ability to use all in-house hardware, and IBM and Oracle/Sun are serious challengers. Today, Hitachi announced their own Unified Compute Platform as well.

The battle of the stacks extends upward into software as well. Although VMware and Microsoft (and don’t forget Citrix) are quick to point out that they can run on anyone’s hardware stack, they are being pulled into the war through alliances, ownership, and old feuds. VMware forms the “V” in the Cisco/EMC “VCE” stack, though they also linked up with Cisco and NetApp in February. HP and Microsoft signed an even bigger agreement in January, suggesting that Hyper-V is their preferred virtualization solution. Hitachi’s UCP digs deep into the Microsoft solution set and mixes in home-grown software to do everyone else one better. But perhaps it will be Oracle/Sun who will deliver a real software-focused stack, given Oracle’s historical focus.

All of these stacks have a few things in common:

1. Standard hardware configurations are specified for ease of purchasing and support
2. The hardware stack includes blade servers, integrated I/O technology, Ethernet networking for connectivity, and SAN or NAS storage
3. Unifying software is included to manage the hardware components in one interface
4. A joint services organization is available to help in selection, architecture, and deployment
5. Higher-level software, from the virtualization hypervisor through application platforms, will be included as well

Who’s Driving?

Although all of these stacks include software components, it is interesting to note that it is the hardware guys who are in the driver’s seat. Microsoft and VMware are surely wary of tying up too closely with any one server or storage vendor, lest they lose the ability to compete in other environments. One expects these two to announce partnerships across a few hardware stacks, though it is unlikely that they will both fully integrate with every offering.

When it comes to stack hardware, only HP has a complete solution, having recently acquired 3COM, LeftHand, and Ibrix. IBM, Dell, and Oracle are thin on networking technology, though their storage story is stronger than many give them credit for. EMC clearly lacks the server and networking components and so tied up with a complementary Cisco who is strong in both areas. Then there is Hitachi, with a solid reputation in storage and strong but relatively unknown blade server offering but no clarity on the networking side. Perhaps HDS is already on the phone with Xsigo, VirtenSys, or Arista?

Certainly, all of these hardware vendors would like to offer a complete solution. This leads one to assume that many of the smaller hardware companies will soon be acquired, since they cannot hope to offer their own stack. The networking and virtual I/O companies are especially juicy targets, but don’t count storage out either. One could certainly imagine Cisco picking up a storage company or two and leaving EMC at the altar! Compellent is undervalued right now, and 3PAR must look pretty good, too.

The Cloud Angle

What doe these stacks have to do with that other megatrend in IT, cloud computing? There are two angles here:

1. Integrated stacks are the perfect foundation for private and public cloud deployment, providing Infrastructure as a Service (IaaS) as a turnkey purchase. I called this the “home theater in a box” concept at Tech Field Day: A single line item gets you all the components you need to roll out a large-to-huge virtualization-ready infrastructure.
2. Integrated stacks are a lame attempt for the hardware guys to stay relevant in a public cloud world. Public cloud providers are taking a top-down approach to infrastructure with most relying on generic commodity servers, storage, and networking rather than buying from the big-iron providers in the stack wars. This consolidation mirrors the dying breaths of the British auto industry, which attempted to retain their hold on the market through scale rather than competitive products.

It’s not immediately clear whether one, both, or neither of these arguments truly provides the impetus for these hardware stacks, but the vendors are all pushing them as cloud-ready. Certainly, the advent of cloud computing is a major factor.

Only the Beginning

This is only the beginning of the stack wars. I have asked the Gestalt IT authors and Tech Field Day delegates to weigh in on this subject, and expect to see a number of insightful pieces written on the topic in the coming weeks. I will continue covering the topic as well. Please consider subscribing to the Gestalt IT RSS feed or following @GestaltIT on Twitter to stay up to date on the latest developments. Thoughtful comments and responses are always welcome as well.

© Stephen Foskett for Gestalt IT, 2010. | The Stack Wars Have Begun!
Read more posts categorized as Cloud Computing, Exclusive, Featured, Networking, Server Virtualization, Storage, Top Story

The first screenshot shows my DroboPro configuration.  It’s currently up to 7 drives and 9.4TB; although I’m only using a fraction of that space at the moment while I do some testing (the majority of disks in the unit were also purchased for testing).  I’ve created seven 2TB drives (more than the Drobo capacity) and connected them to my Hyper-V server.  From there, I’ve formatted four of the drives and you can see those as drive letters H: through to K: on the second screenshot.  Two of the volumes are hosting Hyper-V guests occupying around 19GB of storage in total.  This is directly reflected by the capacity of space used on the DroboPro and the two C:\ drives, one of which is shown in the third screenshot.

The DroboPro is ‘thinning’ the thin provisioned Hyper-V virtual disks, enabling me to in effect scale my Hyper-V storage requirements to the absolute minimum I need to, even if the virtual disks for the operating systems are bigger (in this instance 126GB per virtual drive).  The next step is to extend the storage usage on the virtual guests and see what happens.
My first attempt is to use an old favourite, fsutil and create a few large empty files.  Unfortunately this strategy doesn’t work; the NTFS file system on the Hyper-V guest seems to detect that this is a sparse file and even though the virtual file system shows the storage as being allocated, this isn’t passed back and allocated on the Hyper-V server.  This seems to indicate that Hyper-V itself is truly filesystem and thin provisioning aware.

My second attempt consisted of finding a large file and copying it repeatedly.  This process was more successful and the results can be seen in the 6th screenshot; I added around 9GB of additional files, pushing up the DroboPro capacity and the space stored in the VHD snapshot by equal amounts.

I then deleted my snapshots, effectively merging them into the main VHD file.  As the merge took place, the base VHD file increased, taking more overall capacity on the DroboPro.  After the merge had completed and the Hyper-V guest restarted, the capacity used returned to the total in actual use; the DroboPro reclaimed the released storage.

Summary

What have I learned?  Without any additional configuration, the DroboPro is thin provisioning aware with Hyper-V.  This is “thin on thin” as the Hypervisor and the DroboPro both implement thin provisioning at their own level.  Snapshots need some consideration, especially when merging into the main virtual disk (VHD).  Thought also needs to be given to the best way to lay out the Hyper-V storage; bear in mind that to get the thin provisioning benefits, each DroboPro drive must be configured with a single NTFS volume – spanned and striped volumes will not work.

I’d say the DroboPro was good to go with Hyper-V.  All we need now is vmfs support!

© Chris for Gestalt IT, 2010. | DroboPro – Hyper-V’s Best Friend!
Read more posts categorized as Featured, Server Virtualization, Storage

Ever since Microsoft and Intel declared that the combination of Windows and Nehalem could deliver over a million iSCSI IOPS, I’ve been curious about just how they did it. What black magic could push that many I/Os over a single Ethernet connection? And what was on the other end? Now Intel has revealed all in a whitepaper, and the results are surprising!

What iSCSI Did

Let’s review the test for a moment. In March, Microsoft and Intel demonstrated that the combination of Windows Server 2008 R2 and the Xeon 5500 could saturate a 10 Gb Ethernet link, pushing iSCSI throughput to wire speed. That’s 1,174 MB/s, right around the theoretical maximum of a ten-gigabit link, given a tiny bit of overhead. The pair reunited in January to show that this same combination could deliver an astonishing million I/O operations per second, too.

Both of these results are astonishing. Sure, many high-end Fibre Channel SANs and storage systems blast out gigabytes of data and millions of I/O operations every second, but these tests are much more focused. Benchmarks are perilous, but the folks at Microsoft and Intel devised a fairly clever and focused set. Rather than a “mine’s bigger” contest, the pair only needs to prove that iSCSI can play with the pros.

The side effect is a demonstration of the capabilities of Microsoft and Intel components. Microsoft showed off the capabilities of Windows Server 2008 R2, Hyper-V, and their software iSCSI initiator, while Intel can brag about the Xeon 5500 server platform and X520-2 10 Gb Ethernet Server Adapter with their 82599EB controller. Your mileage may vary, but it is possible to construct a true storage monster on an average server budget.

Intel Inside

Let’s start by looking at the configuration of the local end of the tested configuration. I’m a storage guy so I think of it as the initiator, but you might say it’s the server, the client, or the host. Regardless, the system under test (SUT) is what was put under the microscope. The configuration was a common one: A high-end computer packing an Intel Xeon CPU and 82599-based 10 Gb Ethernet adapter. Most data centers have a machine or two just like this one.

Looking closely, we see that the test in question relied on the following key components:

• Intel’s “Shadycove” S5520SC workstation-class motherboard
• The Intel Xeon W5580 CPU (4 cores, 8 MB cache, 3.20 GHz)
• 24 GB of DDR3 RAM
• Intel “Niantic” 82599EB 10 Gb Ethernet controller
• Microsoft Windows Server 2008 R2 x64

This combination would set you back about $7,500 – $450 for the motherboard, $1,500 for the CPU, 6 2 GB DDR3 SDRAM modules at $80 each, $1,200 for the Intel X520 NIC, and $4,000 for an Enterprise copy of Windows Server 2008 R2. Not cheap, but not an exotic server either.

Initiate and Optimize

The secret to push the tested system to perform like it did is in the optimizations in the server platform, the NIC, and Windows Server itself.

• The Xeon 5500 processor series includes many enhancements:
  • An integrated memory controller allows for faster RAM access
  • QuickPath interconnect (QPI) replaces the old front-side bus and enhances I/O off the core
  • A new I/O subsystem with PCIe integrated into the CPU
  • MSI-X expands the number of interrupts a PCI device can use
  • New instructions for on-board CRC-32C decoding, speeding up iSCSI digest processing
• The 82599 Ethernet controller also includes enhanced capabilities:
  • VMDq maps I/O queues to multiple cores and virtual machines, reducing I/O bottlenecks
  • Offload of TCP segmentation and receive-side coalescing
  • Interestingly, it does not appear that VMDc/SR-IOV was employed in the test
• Microsoft Windows Server 2008 R2 and Hyper-V are ready to use all of these features and more:
  • R2 uses multi-core CPUs more effectively in general
  • Receive-side scaling (RSS) spreads the I/O workload across all four Xeon cores
  • The iSCSI initiator now allows CRC digest offload (using the new Xeon command set)
  • Numerous “NUMA I/O” optimizations in the initiator
  • TCP/IP Nagle can be disabled in the registry
  • Hyper-V VMQ allows the network packets to be copied directly into the guest virtual machine’s memory

Whew! Put all of these optimizations in a blender and Hyper-V virtual machine iSCSI access will be twice as fast as before. No kidding!

Stay On Target

But we knew all of this back in January. We also saw that a Cisco Nexus 5020 switch was used to fan out to 10 software iSCSI targets. But until now there was no mention of what targets were used exactly.

The final footnotes in Intel’s whitepaper reveals that the storage backing the million IOPS test was none other than StarWind Software’s iSCSI SAN! It is unclear what led Microsoft and Intel to use this particular iSCSI target (the earlier throughput tests ran on NetApp filers), but it does speak to the quality of this product.

It is not clear how many disk drives were used, but I would guess that SSDs or ramdisks might have been employed to pull a million IOPS. Network optimizations are also not mentioned, though jumbo frames would not be a benefit in an IOPS test.

StarWind’s software runs on Microsoft Windows and creates a full-featured iSCSI target, complete with data mirroring, automatic failover and failback, replication, snapshots, and thin provisioning. The company prices their iSCSI SAN at $6,000 for two nodes and competes with the likes of DataCore and Open-E. But the StarWind solution seems at a glance to be more full-featured than these other offerings.

Try It Yourself!

I imagine many folks like me might be tempted to try to reproduce these results. More valuable would be a set of best practice guidelines for the deployment of software iSCSI in Windows Server 2008 R2 and Hyper-V environments. Given the relatively modest hardware involved, there should be nothing stopping us!

These test results also prompted me to get in touch with StarWind to try their iSCSI target software. I was pleasantly surprised to learn that they are currently offering free non-production licenses to VMware vExperts, VCPs, and VCIs as well as Microsoft MVPs, MCPs, and MCT Professionals. Many of my readers fall into one (or more) of those buckets, and I applaud the company for this offer. If only more companies realized the value in giving away test licenses to influencers and thought leaders!

© Stephen Foskett for Gestalt IT, 2010. | How Did Microsoft and Intel Get 1 Million iSCSI IOPS?
Read more posts categorized as Server Virtualization, Storage

How fast can iSCSI get?

“Maximizing Hyper-V iSCSI Performance with Microsoft and Intel” might sound like another “blah blah” marketing piece, but a little birdy tells me that this webcast will drop a bombshell about iSCSI performance.

Lots of storage and networking folks don’t give iSCSI and Microsoft the credit they deserve. “iSCSI is cheap and easy,” they say, “but real performance requires Fibre Channel.” Those of us who have an open mind about such things know that this is simply not the case. The fastest SAN I ever saw was based on iSCSI, and Microsoft demonstrated wire-speed iSCSI over 10 Gb Ethernet in March. I never saw a Fibre Channel SAN (even an 8 Gb/s one) push over a gigabyte per second over a single link!

Still, ask the average sysadmin and they will tell you that iSCSI isn’t for high performance applications. That’s why folks should tune in to this webcast, as Microsoft and Intel knock down another iSCSI performance myth. Note that even though Hyper-V is called out in the title and description, this discussion is really about Windows Server 2008 R2 and applies equally regardless of whether or not you use Microsoft’s hypervisor.

Watch this space for a summary of the news immediately following the announcement.

• What: Maximizing Hyper-V iSCSI Performance with Microsoft and Intel webcast
• When: Thursday, January 14, 2010 8:00 AM Pacific Time
• Where: MSEvents.Microsoft.com
• Who: Anyone interested in high-performance storage and server I/O

© sfoskett for Stephen Foskett, Pack Rat, 2010. |
Microsoft and Intel Pushing iSCSI Performance Limits


This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2010. | Microsoft and Intel Pushing iSCSI Performance Limits
Read more posts categorized as Server Virtualization, Storage

In fact, it’s difficult to tell exactly what High Availability Manager (we’re calling it HAM for now) really is. As noted by our own Chris Evans (USP-V – So Long And Thanks For All The Fish), HAM isn’t the next-generation USP-V with more-granular thin provisioning, automatic storage tiering, and expanded scalability. It isn’t HDS’ answer to the EMC Symmetrix V-Max and it’s forthcoming FAST technology.

So what is HDS up to here? For one thing, they’re not pre-announcing technology that won’t be delivered for some time. HAM appears to be in use at beta customers now, and will no doubt be generally available in short order.

As Storagebod tweeted, “it’s a teaser…the technology to move is part one but you need something to move to. New USP will be part two!”

This Can’t Be All!

Although I have no inside information, it is simple to speculate on what HDS will deliver soon. If I was the product manager for USP-V-II, here’s what I would demand:

1. The next-generation USP-V must be far more scalable than anything we’ve seen yet. This means it must surpass the 8 “engines” and 2,400 drives of the current V-Max from EMC.
2. It must also offer automatic tiering of storage from flash/SSD to Fibre Channel to SATA. In fact, HDS can one-up everyone else (except maybe IBM) by extending tiering to older, external, and even third-party storage arrays. Now that would be something!
3. It must include next-generation datacenter Ethernet support, including 10 Gb FCoE and iSCSI.
4. It must integrate with VMware vSphere and Microsoft Hyper-V, including native multipathing and full API support for snapshots and replication.
5. It must be an easy upgrade from current storage systems, especially USP-V but also third-party arrays.
6. Since it’ll be huge, it must have template-based management for hosts, LUNs, and HA features.

Ham It Up

My fear is that HAM will be used to provide some of these features in a not-exactly-integrated fashion. It will certainly be used for the upgrade path, but let’s hope it’s not the final word for the others. Consider that the “clustering” features of HAM would allow HDS to claim extreme scalability across geographies between multiple USP instances. Although this would not pass the sniff test of many who are hip-deep in enterprise storage, it would likely satisfy the sales-pitch one-upmanship needs when competing with EMC.

On reflection, all of the features mentioned sound exactly like what EMC announced last month! So HDS must have more up their sleeves. Even this USP-II can’t be all HDS has to offer!

© Stephen Foskett for Gestalt IT, 2009. | HDS’ HAM-Fisted Announcement Can’t Be All
Read more posts categorized as Exclusive, Featured, Storage, Top Story

Hiding in the shadow of the huge VMware vSphere 4 announcement was a very interesting introduction by EMC: PowerPath/VE. As I mentioned in my post on storage changes in vSphere 4, PowerPath/VE plugs into the new pluggable storage architecture (PSA) found in vSphere 4 versions of ESX and takes over the decision-making and heavy-lifting tasks related to communicating with storage systems.

Driving Massive I/O

Chuck Hollis treated us to a discussion of vSphere as an I/O Engine on his blog this morning with some background on multipath IO (MPIO for short), but I’m not sure he did the topic justice. In my opinion, server virtualization is the greatest I/O driver ever brought into the data center, and it messes with all of our preconceived notions about I/O at the same time.

What’s so special about server virtualization?

1. Hypervisors concentrate I/O, shifting loads that were formerly distributed to a large number of I/O channels into a far fewer channels. Picture 10 servers doing what they do. Now put all 10 in a single physical box. All of their storage access must now share a bus, a host adapter, a cable, and perhaps a LUN on the storage system. It’s the difference between lemonade and lemon juice!
2. Hypervisors randomize I/O, chunking everything up and mixing it together. Forget about the carefully-designed read-ahead algorithms and caching used in enterprise storage – VMware, Hyper-V and the rest throw those expectations out the window! Virtualization is a blender – it grinds up your lemons, skin, seeds, and all!
3. Hypervisors demand low I/O latency, forcing infrastructure to get quicker, not just faster. This is one reason that caching, solid state disks, and 10 GbE are going to be huge in virtual environments – all reduce latency by orders of magnitude! As any car guy will tell you, quick and fast are two very different things!

The upshot of all of this is that virtual servers are very very hard to satisfy when it comes to I/O. And the “back end” has always been a bit of a bottleneck for virtualization software. Now we have VMware claiming that vSphere 4 can push over 300,000 I/O operations per second (IOPS) without resorting to VMDirectPath and similar “cheater” measures. Of course not all IOPS are equal, and I doubt that that 300k number would hold up with a real-world workload, but it’s impressive nonetheless!

A Brief History of MPIO

Let’s turn back to multipath I/O. PowerPath/VE is just the latest in a long line of path managers, not all of which have been well-loved. Back in my HP-UX days I learned to make the most of PVlinks, the native path management on that operating system. It wasn’t always easy to get it to work well, but it sure was nice to have a path manager built into the operating system! Veritas also offered a multi-platform path manager, DMP, which worked with a variety of array types. Back in the day, both were limited to simple failover and lacked the “intelligence” to deal with the peculiarities of the weird storage arrays we learned to not hate.

Array-specific path managers from storage vendors were much more successful. CLARiiONs used ATF, Hitachi arrays used HDLM, IBM had SDD, and of course EMC had PowerPath. EMC introduced PowerPath in 1997, the software reportedly having been developed by Conley Corporation, which EMC acquired the next year and turned into its Cambridge (MA) development center. After acquiring Data General, EMC adapted PowerPath to support CLARiiON, pushing ATF off stage right. Then they kept right on developing the software, adding support for IBM, HDS, and HP arrays and data migration.

Meanwhile, Microsoft decided that HP and Veritas were on to something when they developed standard path management software, so they began working on a standard multi-path IO (MPIO) driver for Windows. But Microsoft learned a thing or two from the mediocre device support in those old solutions, so they decided to allow vendors to plug their own smarts into the standard Windows Server 2000/2003 MPIO framework. Microsoft provided basic failover capability and third parties, including EMC, wrote their own device-specific modules (DSMs). This MPIO support evolved and spread, standard on Microsoft’s iSCSI initiator and Hyper-V virtualization platform. PowerPath 5.2.1 for Windows already supported Hyper-V thanks to this.

PowerPath and VMware PSA

VMware also learned a thing or two from HP and Microsoft. Although basic path failover support has been included in ESX for years, vSphere 4 takes it to a new level with pluggable storage architecture (PSA). Every version of ESX 4 includes native multipathing (NMP), but Enterprise Plus licensees can use vendor-supplied plugins to enable more advanced path management. As I noted on Tuesday, there are two three different levels of path selection: Basic path-selection plugins (PSPs), more advanced storage array type plugins (SATPs), and complete multi-path plugins (MPPs).

This is what EMC has introduced: An MPP for vSphere 4 called PowerPath/VE. Like the DSM for Windows MPIO, PowerPath/VE for vSphere slots right into an existing MPIO framework and enables advanced path selection and load balancing without mucking with the internals of the hypervisor. PowerPath/VE has all sorts of smarts in it. It has eight different predictive load balancing policies, proactive disconnect, bus testing, and HBA monitoring.

Super VMware guy Chad Sakac described PowerPath/VE as part of the launch. He notes that EMC is first out of the gate with a multipathing plugin for vSphere, but I suspect that just about every vendor will release similar functionality pretty quickly. In particular I expect support to come from NetApp and 3PAR, since they’re so interested in VMware support.

Licensing Questions

One thing really stuck out in the vSphere launch: PSA is only included in the top-of-the-line Enterprise Plus license. Presumably, this means that, in addition to paying for a PowerPath/VE license, users will have to spring for maximum ESX, too. This is a dumb move, if you ask me. Microsoft made MPIO successful by giving it away with every copy of Windows. They even included it in the free iSCSI initiator download. VMware, in contrast, seems to be actively limiting PSA’s usefulness to the top tier of users. If it was up to me, I would set the VMware MPIO free!

I’m working with EMC and VMware to determine the extent of the NMP/PSA/PowerPath licensing mess. I’ll update this post as I find out the answers!

1. Does every edition of ESX 4 include the basic VMware native multipathing (NMP)?
2. Can one use a vendor-supplied PSA plugin like PowerPath/VE without an enterprise plus license?
3. Does it matter (to licensing) if the plugin is a PSP or an SATP?
4. If “no” to 2 or 3, can PSA be added separately without the plus license if someone wants to use something like PowerPath/VE?

Update: I received a nice email from an EMC engineer correcting me about the plugin types. This kind of open communication is why the web is so great! It turns out that PowerPath/VE is a sort of super plugin called an MPP, not “just” an SATP or PSP. I’ve updated the section above!

© sfoskett for Stephen Foskett, Pack Rat, 2009. |
PowerPath To The Virtual People


This post was categorized as Computer history, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2009. | PowerPath To The Virtual People
Read more posts categorized as Server Virtualization, Storage