Stepping out of a subway car is an entirely different matter when it's moving!

Mobility of virtual machines is a sticky wicket. As open systems infrastructure becomes increasingly virtualized, administrators and managers wish to use the technology to balance workload, ease migration, and provide better availability. Although technology is improving, actually moving virtual machines is not always a piece of cake. Let’s lay down a baseline of information so we may begin a discussion on the true nature of virtual machine mobility.

Mobility of What?

Let us consider first the question of what exactly is being moved. Systems administrators often focus on “the machine”, which encompasses the operating system and configured state of the virtual machine itself. But the true “mass” of the system is its stored data. Hypervisor vendors have come up with different techniques of moving these two essential elements, reflecting the unique characteristics of each.

• The virtual machine is an instance of an operating system along with its state and configuration. Mobility of virtual machines requires all of this to be preserved, along with any I/O channels. Live migration of virtual machine requires that any active network sessions be maintained, along with RAM content, registers and buffers, and so many other elements.
• The virtual machine image (commonly referred to as “storage”) is the static content addressed by a virtual machine. Typically a VMDK or similar virtual disc image, it must be accessible to the virtual machine at all times. Live migration of a virtual machine image is tricky, but perhaps not quite as complex as live migration of a running operating system.

VMware, Microsoft, and others recognize these 2 distinct elements to be migrated, and have come up with a variety of complementary technologies for each:

• vMotion is VMware’s virtual machine migration solution, and has continually evolved with each iteration of the hypervisor. DRS leverages vMotion to automate mobility. VMware has also created Storage vMotion and Storage DRS as complements to handle mobility of virtual machine images.
• Microsoft Hyper-V Live Migration is conceptually similar to vMotion, though newer and less full-featured. With Hyper-V 3.0, Microsoft will introduce Storage Live Migration as a complementary technology akin to Storage vMotion. Most other virtual machine managers also support some form of live migration, though live migration of storage is less common.

Mobility in What State?

One of the key benefits of virtual machine technology is the ability to “run anywhere” on dissimilar hardware. From the very beginning, hypervisors have provided the ability to create a universal virtual machine image that would run on a variety of supported platforms.

This leads to one of the key values of server virtualization in the data center: Disaster recovery. The ability to take a virtual machine image and system state and bring it online after a disaster is a true revolution for open systems IT. The benefits of the single usage of server virtualization technology easily justify the investment to many businesses.

But this sort of “cold” migration seems passé when compared to the live or “hot” migration possible with technologies like VMware vMotion. Live migration is much more difficult, since active client sessions must be preserved in activity must not be greatly interrupted.

This is the second great question that must be asked when considering virtual machine mobility: In what state will the virtual machine be moved? Will it be a cold, powered down image of the system? A suspended or paused operating system image? Or a full, running machine?

Mobility to Where?

Once we have decided whether we are discussing virtual machine migration or movement of storage resources, we must consider the scope of the movement. The ability to move a virtual machine from one member of the cluster to another has now become fairly common. But what about systems that are not related in a cluster? Or that are spread over great distances?

• The nice thing about clusters is that they share resources before and after a virtual machine is moved. It is practical to move the running virtual machine, its storage, or both independently and to expect that performance will not dramatically suffer as a result. The cluster can also preserve network connections, and even I/O state, without much impact on clients or other external elements.
• It is a bit more difficult to move systems within a data center, since one must maintain the I/O connections that might be interrupted. It is fairly trivial to configure an IP network and storage array to allow multiple machines to access the same iSCSI or NFS storage resources. It is a little more difficult to configure Fibre Channel (and, by extension, FCoE) SAN’s to handle this sort of dynamic movement, but it is not impossible. Although moving a running machine from one network port to another could cause client access to be interrupted, technologies like VXLAN allow these sessions to continue, and improved network switching technology should reduce performance impact.
• Moving the machine to a different data center is another matter entirely. Stretching a layer-2 Ethernet LAN or Fibre Channel SAN across a metro or greater distance, while possible, will always be problematic. IP routing is flexible, but it takes time for changes to propagate when live machines are moved. And it is difficult to keep storage in sync over long distances due to the amount of time it takes for information to transit. Again, all of these challenges are being addressed in various ways, but they’re still hard!

Stephen’s Stance

Moving cold virtual machine images from system to system, or even across great distances, is one of the main selling points of server virtualization. But it becomes much more difficult to manage movement of virtual machines that are still running, especially outside cluster or across WAN links. When talking about virtual machine mobility, it is important to consider what is being moved, the state it is in, and where it is going.

Note: This discussion is part of “Building Virtual Infrastructure”, my new seminar series with Truth in IT.

© sfoskett for Stephen Foskett, Pack Rat, 2012. |
Virtual Machine Mobility: Of What, and to Where and in What State?


This post was categorized as Enterprise storage, Features, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2012. | Virtual Machine Mobility: Of What, and to Where and in What State?
Read more posts categorized as Storage

I am curious about this idea but have another idea. Sure syncing between two Drobo enclosures would be great, but wouldn’t a file replication application, either on the Drobo or on the desktop do the job just as well? Perhaps what Drobo should consider is some technology and a subscription service to replicate data stored on my Drobo with Amazon S3.

Set a price point based on the size of the customers Drobo at the time of sale or create several models of pricing so that small medium and large Drobo storage can be replicated there.

Like a dropbox situation for your Drobo. Just a thought at this point, but syncing to the cloud might be an additional solution especially if the customer only has one site.

Perhaps per GB pricing won’t work here because there isn’t a flat fee structure, but allowing Drobo to provide x TB of disk at S3 for data replication would help with off site storage/backup and allow the data to be backed up by Amazon to reduce failure points.

For the smaller shops considering or even using a Drobo, this might be a great solution for off site backup. Sure multiple Drobo units syncing across the network or a VPN is a great solution, it just seems that a way to push this straight to the cloud, using something similar to DroboSync might be an additional offering.

Now if only I were a developer with the time to create such an application or if it existed or was being planned already… Thoughts anyone?

Would be great to heard from the Tech Field Day delegates attending in February, perhaps we can get some ideas from Drobo on the topic.

© derek for Gestalt IT, 2011. | Synchronization from Drobo to the cloud
Read more posts categorized as All, Desktop, Storage

How many of you run clusters of servers? HA/CMP? Veritas Cluster? Microsoft Cluster? VMWare Clustering? I suspect lots of you do? How many of you cluster NAS heads? Yet again, I suspect lots of you do? How many of you cluster arrays? Not so many I guess? Certainly in my experience, it is uncommon to cluster an array. And when I talk about clustering an array, I don’t mean the implementation of replication.

So, if you don’t cluster your arrays; how do you protect against the failure of a RAID rank? Statistically unlikely but it is it more or less unlikely than a loss of data-centre? I’m not sure and the failure of a RAID rank for many people could well mean the invocation of the disaster recovery plan. Why?

The loss of a RAID rank might well lead to the loss of an application/service and if it is an absolutely business critical service, can you bring it up at the remote replication site in isolation? As a discrete component? If you can, can you cope with increased transaction times due to latency? Many applications now have complex interactions with partner applications; these might not be well understood. So the failure of RAID rank could lead to the invocation of the Disaster Recovery Plan. Actually in my experience, this is very nearly always the case unless the service has been designed with recovery in mind; this requires infrastructure and application teams to work together, something which we are not exactly good at.

But you now take the challenge and make sure that every application can be failed over as a discrete component. Excellent, a winner is you! You know the impact of loosing a RAID rank, you know what applications it impacts, you’ve done your service mappings etc, etc. And you have been very careful to lay things out to minimise a single RAID failure’s impact.

And then you implement automated storage tiering. Firstly, you now have no idea in advance what impact a RAID rank failure may have; you have no idea what applications may be impacted. And actually, the failure of a single RAID rank may well have huge impact. We could be looking at restoring many terabytes of data to cope with the failure of a couple of terabytes and many applications failing.

It will depend on the implementation of the automated storage tiering and I am concerned that at present we do not know enough about the various implementations which will hitting our arrays over the next eighteen months. So despite automation making things day-to-day a lot easier, we cannot treat it as Automagic Storage Tiering; we need to know how this works and how we plan to manage this.

And perhaps for key applications, we will need to cluster storage arrays locally; that in itself will bring challenges.

I’m still a big fan of automated storage tiering but over the next few months, I would like to see the various vendors start talking about how they mitigate some of this risk. Barry Burke has made a big thing about the impact of a double disk failure on an XIV array in the past; in a FAST v2 environment, I would like to see how EMC mitigate against very similar problems.

I would also like to know what impact of a PAM card failure from NetApp is; does the array degrade to the extent where it is not useable? What kind of tools can NetApp give me to assess potential impact. As Preston points out here; failure of individual components within an array could have significant impacts.

We are heading to a situation where technology gets every more complex and arguably ever more reliable. But we rely on it to ever more greater extents; so we must understand risks and mitigations to a much greater amount than we have in the past.

© Martin for Gestalt IT, 2010. | Disastrous Thinking
Read more posts categorized as Storage

This is the final post in a series on Enterprise Data Migration Strategies.  Previous posts:

Enterprise Computing: Data Migration Strategies – Part I

Enterprise Computing: Data Migration Strategies – Part II

Enterprise Computing: Data Migration Strategies Part III

Enterprise Computing: Data Migration Strategies – Part IV

Previously we’ve discussed how to plan, structure and organise migrations.  In this post, I’ll touch on some of the tools which may be used to perform migration work.

One Size Does Not Fit All

First of all, its worth pointing out that no single solution fits all needs; migration methods are varied and the specific configuration in place demands the best solution at the time.  Therefore it pays to have an arsenal of tools at your disposal and know how you’d use each one.

Array-Based Migration

Storage arrays already have many tools for performing migrations.  These exist today for business purposes; remote replication to another array; local replication within an array using clones and snapshots.  The benefit of using in-array technology is the migration work is taken away from the host and potentially can be executed within minimal customer interaction.  On the negative side, most replication technologies which move data between arrays are product specific – i.e. SRDF on EMC DMX arrays isn’t compatible with HDS’ TrueCopy.  This shouldn’t be a surprise because they are proprietary technologies which gain their advantage by being specifically coded and optimised to the storage platform itself.  There are however tools like EMC’s Open Replicator which can move data between vendor/family technology.  Open Replicator does have restrictions though – depending on the type/direction of replication, incremental copying isn’t available and requires a full copy sync to complete, potentially removing the benefit of using the tool altogether.

Virtualised Migration

Sitting slightly higher up the “storage stack”, it is possible to do migrations using a virtualisation technology sitting above (or integrated with) the storage array.  For example, IBM’s SVC can be used to manage data migrations and sits above all storage arrays; HDS’ USP (equivalent to HP XP models) has a facility called Universal Volume Manager (UVM) which can perform the same work and is built into the array.  Incipient have a solution called INSP (Incipient Network Storage Platform).  If not already deployed, these tools will need an outage to be installed in the data path.  Both impact the World Wide Name (WWN) the host sees, so host changes may also be necessary, depending on operating system.  The benefit of these technologies, once installed, is that they allow data to be moved dynamically “under the covers” without involvement of the customer or work on the host server.  As with all technologies, there are restrictions under certain circumstances and you should check with the product vendor for those.  It may well be that you want to move the virtualisation tool at the end of the migration so another outage may also be required.

Fabric Migration

Moving slighly higher, we have the ability to perform data migrations in the storage fabric (SAN) itself.  Example products include Brocade’s Data Migration Manager (DMM) and EMC’s InVista.  Storage migration in-fabric requires the deployment of hardware in a SAN switch that intercepts I/O and redirects a second copy to another device.  Potentially these devices can be installed in the data path without distruption but will require an outage to cut over to the new target volumes.

Host Migration

Finally at the top of the stack we have host-based migrations.  Even at this level there are still a number of choices.  If a Logical Volume Manager is installed (e.g. Veritas Foundation Suite/Volume Manager), then migrations can be performed using this software without host disruption.  This is often a good choice of tool if the target devices are in a different array, if outages can’t be taken or if the LUNs are being re-organised or restructured.  Unfortunately this also means having either host-access given to the storage teams (plus O/S knowledge to complete the work) or requiring the platform teams to perform the migration work.  Both of these options may be a problem in certain organisational structures.  One word of warning using LVMs – if LUNs are being replaced by using “evacuate” functionality (where a LUN at a time is swapped with another) then a potential data integrity problem exists, especially if the LUNs are also replicated.  The risk occurs because data spans two arrays and if remotely replicated, then writes at the DR site might not be in integrity timestamp order.  Failure in either array can result in an outage.

For mainframe customers, there’s the fantastic TDMF (also available in an Open Systems version)

If LVMs are not available, then good old-fashioned data copying is the order of the day.  There are many tools to do this, too numerous to mention here, but be aware, that this method is likely to mean protracted downtime as storage shouldn’t be active and be accessed whilst it is being copied.  It is also possible to migrate data within an application, again, there are too many options to mention here.

Hopefully this article provides a flavour of the migration tools out there.  Please add comments or ping me if you’ve any specific tools you would like me to mention and I’ll add them on as a separate page.

© Chris for Gestalt IT, 2009. | Enterprise Computing: Data Migration Strategies – Part V
Read more posts categorized as All, Storage

With EMC’s Mirrorview/S (sync) there is a distance limitation of between 50km and 200km depending on what fibre optics you are using (short/long wave vs. dwdm).  Mirrorview/A (async) is more widely used over an IP WAN connection but can also be used over fibre as well.

Mirrorview/S

Pros:

• Synchronous – Exact copy of data on production
• Little to no data lost

Cons:

• Distance limited (60km using short wave gbics, long wave gbics or optical extenders, 200km using dense wave division multiplexors)
• WAN link more expensive (fibre vs. copper/ip) unless Fibre Channel over IP converters are used and those are still a little expensive

Mirrorvew/A

Pros:

• Cheaper WAN link between sites (IP usually)
• Writes to prod don’t have to wait on mirror site to write
• Not distance limited like sync replication

Cons:

• Data can be lost depending on write intervals from prod to DR site

What you need to know

Array-based mirroring is a great way to protect multiple hosts in an environment instead of buying per-server or per-application replication.  As I’ve discussed before, the biggest drawback is that it provides a restartable copy which isn’t the same as an active-active cluster application transaction-level replication (i.e. Oracle Dataguard, Exchange CCR, MySQL Master/Slave replication).  Be careful of adequate LAN/WAN line quality, poor comm lines can cause insanely painful headaches (troubleshooting, added latency, etc).  Get line tests done to determine available bandwidth, line quality and latency.

© edsai for Gestalt IT, 2009. | Sync or Async Replication?
Read more posts categorized as All, Storage

HAM combines conventional ingredients to create a whole new flavor

High Availability Manager consists of three main components:

1. A conventional multi-pathing agent like HDLM on each server. This enables the server to continue accessing the storage if one USP fails. It will “think” it’s talking to a single storage target, but will actually be talking to two USPs that can be metro distances (60-100 miles) away, given proper connectivity. Microsoft MPIO will probably be supported as well, and VMware native multi-pathing (NMP) should come shortly after release. Don’t hold your breath for PowerPath to be officially supported in the short term, but it ought to work fine without changes.
2. Existing TrueCopy synchronous storage replication technology will keep the data and quorum disks (see below) in lock-step. This gives the limitation on distance between systems, since latency is the enemy of storage protocol performance. Once the arrays move too far apart, write performance will suffer on the local array while it waits for data to be copied.
3. Conventional clustering technology with a heartbeat and shared quorum disk lets both arrays know what’s going on. The quorum “lives” on the remote side, with that secondary array watching to make sure the primary array is still running. If the heartbeat goes away, the secondary array marks the HAMmed LUNs read/write and starts handling I/O from the servers, which will just have failed over.

So HDS’ secret HAM sauce is ketchup. There’s no amazing new technology here, and maybe that’s for the best in the kind of huge, conservative enterprise environments that will use this product. The big change was in programming the USP controllers to monitor the quorum disk and orchestrate the entire failover. In fact, this might even be considered a feature of TrueCopy, not a standalone product.

HDS ought to reconsider one element of the HAM pitch, however. Although it will undoubtedly yield a very highly-available storage architecture, nothing provides 100% availability. There are many moving parts involved, and unplanned outages can still happen. Multipath driver bugs are not unheard of: Back in the day, one version of a certain three-letter company’s product just plain refused to fail over! An instantaneous outage on one of the host channels or a skipped heartbeat on the USP controllers could also cause a failed failover. Plus, there is no provision for automated fail-back. Once the failover occurs, the system would certainly be operating in a degraded-availability mode and would require a (planned) outage to re-establish operations.

All that being said, HAM remains an attractive offering for shops with multiple USPs visible to critical servers. They can turn on the HAM software with existing hardware and add peace of mind, knowing that everything is that much more available. One aspect that really impresses is the fact that each USP can be running a different firmware, reducing the risk of upgrade-induced outages. Once it ships to customers (in the fourth quarter of this year), we will have to consider the (as-yet unnamed) cost.

And what about the fact that HAM will also allow seamless upgrades to a new generation of USP hardware? HDS still isn’t talking about that possibility!

© Stephen Foskett for Gestalt IT, 2009. | HDS High Availability Manager: How It Works
Read more posts categorized as Exclusive, Featured, Storage, Top Story

Dim the lights, chill the ham...

HDS: Hello, I am HDS man/Would you like some HDS HAM?

Bloggers: I’d like to know ’bout HDS HAM/What is it, oh HDS man?

HDS: HAM moves bits from here to there/Available anywhere!

Bloggers: My data has replication/Your brain must be on vacation!

HDS: HAM automates operation/Don’t you want that long vacation?

Bloggers: This sounds like what we had before/Amuse me now before I snore!

HDS: HAM reduces TCO/You will be the staff hero!

Bloggers: TCO is just shell games/Show me! Prove it! Name some names!

HDS: Thompson Reuters tried it first/Just go talk to Chris Crowhurst

Bloggers: I’m too lazy for that call/Tell me more! Is that all?

HDS: HAM is good for enterprises/HAM is for large data sizes!

Bloggers: Some don’t have those data sizes/Some are not large enterprises!

HDS: HAM requires two USPs/That a single server sees

Bloggers: My server does not see that far/My bandwidth costs less than a car!

HDS: HAM requires MPIO/So the host knows where to go

Bloggers: Good thing Redmond made that free/Not priced it like EMC!

EMC: They’re just following our tracks/You’ll get the same stuff from V-Max!

Bloggers: Quiet, boys from Hopkinton/V-Max is still under-done!

HDS: Why not look some more at HAM/You may like it blogger-man

Bloggers: You know, this HAM is not half bad/Price it, ship it, we’ll be glad!

© Stephen Foskett for Gestalt IT, 2009. | A Taste Of HAM (Apologies To The Doctor)
Read more posts categorized as Exclusive, Featured, Storage, Top Story

In fact, it’s difficult to tell exactly what High Availability Manager (we’re calling it HAM for now) really is. As noted by our own Chris Evans (USP-V – So Long And Thanks For All The Fish), HAM isn’t the next-generation USP-V with more-granular thin provisioning, automatic storage tiering, and expanded scalability. It isn’t HDS’ answer to the EMC Symmetrix V-Max and it’s forthcoming FAST technology.

So what is HDS up to here? For one thing, they’re not pre-announcing technology that won’t be delivered for some time. HAM appears to be in use at beta customers now, and will no doubt be generally available in short order.

As Storagebod tweeted, “it’s a teaser…the technology to move is part one but you need something to move to. New USP will be part two!”

This Can’t Be All!

Although I have no inside information, it is simple to speculate on what HDS will deliver soon. If I was the product manager for USP-V-II, here’s what I would demand:

1. The next-generation USP-V must be far more scalable than anything we’ve seen yet. This means it must surpass the 8 “engines” and 2,400 drives of the current V-Max from EMC.
2. It must also offer automatic tiering of storage from flash/SSD to Fibre Channel to SATA. In fact, HDS can one-up everyone else (except maybe IBM) by extending tiering to older, external, and even third-party storage arrays. Now that would be something!
3. It must include next-generation datacenter Ethernet support, including 10 Gb FCoE and iSCSI.
4. It must integrate with VMware vSphere and Microsoft Hyper-V, including native multipathing and full API support for snapshots and replication.
5. It must be an easy upgrade from current storage systems, especially USP-V but also third-party arrays.
6. Since it’ll be huge, it must have template-based management for hosts, LUNs, and HA features.

Ham It Up

My fear is that HAM will be used to provide some of these features in a not-exactly-integrated fashion. It will certainly be used for the upgrade path, but let’s hope it’s not the final word for the others. Consider that the “clustering” features of HAM would allow HDS to claim extreme scalability across geographies between multiple USP instances. Although this would not pass the sniff test of many who are hip-deep in enterprise storage, it would likely satisfy the sales-pitch one-upmanship needs when competing with EMC.

On reflection, all of the features mentioned sound exactly like what EMC announced last month! So HDS must have more up their sleeves. Even this USP-II can’t be all HDS has to offer!

© Stephen Foskett for Gestalt IT, 2009. | HDS’ HAM-Fisted Announcement Can’t Be All
Read more posts categorized as Exclusive, Featured, Storage, Top Story

Continuing the series blog post on Storage Resource Analysis (SRA), this post focuses on the “IT – Storage World of 2009” and the requirements / importance of it around today’s overall Storage Strategy.

Read the entire series of posts on Storage Resource Analysis (SRA):

Part 1: Storage Resource Analysis and Storage Economics

Part 2: The IT – Storage World of 2009

Part 3: The IT – Storage Budgets of 2009

Part 4: Some Fundamental Questions

Part 5: Facts about your Data

Part 6: Inconsistencies in Storage Environments

Part 7: The Technical Case

Part 8: The Business Case

Part 9: The End Result

The following are the important facets of Today’s IT – Storage Environments. 

• Complexity
• Infrastructure
• Cost
• Tiering
• Replication & Archiving
• Virtualization
• Content Management
• Deduplication
• Client Usage Billing
• Database Growth
• File System Growth

The Categories are now divided into Sub Categories: These categories and sub categories are primarily focused around the Storage Domain.

Complexity around Environments

• Business Processes
• Automation
• Faster response times
• SLA’s
• Applications requirements
• OS’s requirements
• Networking Layers interface
• Fabric Layers technology
• Virtualization Layers
• Technology Evolution
• Uptime Requirements
• Multi Vendor Support
• Mergers and Acquisitions
• Storage Consolidation
• Storage Migration

Infrastructure Requirements

• Datacenter Requirements
• Application availability
• Application Support
• Storage backend Support
• Outsourcing
• Insourcing
• Cost Management
• Power Requirements
• Utilization of Storage (x% from CIO)

Cost

• CapEx Reduction
• OpEx Reduction
• Cost of Support
• Penalties

Tiering Requirements

• Application Tiering
• Business Process Tiering
• User Data Tiering
• Cost Savings

Replication & Archiving

• Sarbanes Oxley
• HIPPA
• SEC
• Other Compliance
• Critical Business Requirements
• Critical Application Requirements

Virtualization

• Utilization
• Efficiency
• Uptime
• Floor Space
• Green Data Center

Content Management

• Business Processes
• User Data
• Meta Data

Deduplication

• Backup Windows
• File Transfers times
• Bandwidth

Client Usage Billing

• Centralized Shared Storage
• Centralized Management
• Charge Backs and Bill Backs
• Complex processes for Usage Billing

Database Growth

• Ever growing databases
• Database requirements in terms of storage
• Database Log files
• Log Shipping
• Critical Applications around Databases

File System Growth

• Ever Growing file systems
• User Data
• Business Process Data

While we talk about the IT – Storage Industry and its surrounding requirements and impact, we need to know the Storage Economics behind it.

While the File Systems are consistently growing, Databases are exponentially growing, Compliance is a must and business processes are becoming critical every day, how do we manage and leverage what we have plus help reduce CapEx / OpEx and still manage to keep uptime/ efficiency / utilization.

Example:

I recently had a chance to visit a MNC (Multi National Company) to talk about Storage. We had long 6 hour meetings with various Application Owners, Business Group Owners, Strategic Planning folks, member’s from the CIO’s office, Storage Administrator and Host Administrator. During these sessions, we asked them a few basic questions about their Storage environments related to usage, outages, charge backs, new acquisition, storage management, process management, etc.

It was very surprising to see disconnect of priorities that various different teams had based on the role they played in that organization. Certain folks were talking about budget reductions, while some had priorities to play around with the latest technologies, while some had priorities about the application uptime, while some were focused on growth and some thought about ease of use.

But during the entire time we did not hear any common messages from these various teams about efficiency and optimization of their Storage Environments…

Well, let’s move on…

What do you do to keep up with the IT – Storage requirements?

Is someone helping you purchase new Storage?

© Devang for Gestalt IT, 2009. | Storage Resource Analysis (SRA): Part 2
Read more posts categorized as Storage

CC-by-SA 2.5 image "Golf Bunker" by Ken123

Today, many companies are considering a compete migration from a physical to a virtual infrastructure. Based on the promises of cost savings, administrative efficiencies, and improved resource utilization, virtualization is seen as the technology that makes it possible to do more with less. However, there are many pitfalls to consider when virtualizing server infrastructure. This article suggests planning decisions to be considered by the CIO, IT Director, design architect, and IT Manager that, if overlooked, could ruin the total cost of ownership (TCO) and the return on on investment (ROI) expected from this virtual infrastructure.
�

Migration pitfalls

Implementing virtualization is not a simple process of building a server. Too often, management makes the mistake of expecting that the administrators and architects of the server team will be solely responsible for virtualization decisions. In reality, the impact of a move to virtualization should be viewed exactly as if a company were moving their physical servers to a new data center. Storage, networking, security, Active Directory, messaging, web presence, and all of the systems and services necessary to the organization must be considered together. Some of these groups should be more involved than others in the actual planning and ultimate design decisions, but virtualization strategy affects everyone. Key members from all of these focus areas must be trained and given panning responsibilities to achieve a collaborative plan and design.

A potentially larger “gotcha” once everyone is involved in the planning process is the temptation to include all of the upgrade and service redesign projects that have been “in the queue” for months (or longer). Although the migration may seem like a great opportunity to upgrade to the latest operating system, separate instances from poorly performing database servers, or build the new CRM system, these added activities bring complexity and could pose road blocks to the core goal of the virtualization initiative: consolidating hardware. These projects should be tabled and taken on later, after the new infrastructure is in place. In fact, the flexibility of a virtual infrastructure will actually make these projects simpler.

Performance pitfalls

Condensation of numerous physical servers into a virtual environment is intended to increase the effective utilization of processing power and other system resources, but it is easy to overshoot the target and overload the new system. Although the average load of a number of servers might be within the capabilities of a new virtual infrastructure, resource spikes can be a serious problem. Consider that periods of high demand tend to cluster across servers through the day and week, as employees begin the workday, the backup process begins, or a software build is kicked off. If performance is not measured and correlated carefully before migrating to a virtual infrastructure, these resource spikes can make the new infrastructure unacceptably slow.

One often overlooked performance challenge is the randomization of demands introduced by virtualization. Traditional computer design assumed that processes would proceed through their steps sequentially, with the system devoting resources from start to end. Multitasking operating systems challenged this assumption, but the allocation of memory, I/O, and storage remained clustered. But highly-utilized virtual servers upset this predictable flow, rapidly switching from one task to another entirely different one. In short, virtualization randomizes accesses that were once sequential and transforms large requests into many tiny ones. CPU manufacturers have responded with clever techniques to save and restore registers and handle memory mapping, but this blender effect is far from solved. I/O and storage systems are particularly affected, as they are optimized to stream large sequential operations rather than small random ones.

Storage pitfalls

Although one might think that virtualization reduces storage requirements as systems are consolidated, the opposite is often true. Many of the functions that help drive down CAPEX and OPEX costs in virtualized environments require the use of shared networked storage like SAN or NAS. Data that had once been scattered around the data center on internal drives is consolidated on these networked storage devices. Thousands of redundant copies of operating system files, for example, end up sitting on the storage array. The ease of creating virtual machines from templates tends to lead to virtual “server sprawl” far worse than in the physical world. Storage array capacity is also needed for swap space or paging files as well as popular enhancements like snapshots, DR copies, and backup images.

The result is a flood of redundant data that must be taken into account when planning capacity needs. Although storage is typically not the top item in an IT budget, virtualization can cause it to rise and chew up some or all of the savings from the project. It is difficult to avoid these issues, but some techniques can help mitigate the problem. For example, deduplication of primary storage and thin provisioning can help reduce the storage footprint of the virtualized infrastructure.

Administration pitfalls

Virtual sprawl can cripple infrastructure support teams if it is not managed. The challenge of administering this new virtual environment is often ignored in the implementation rush. Rather than managing 3,000 servers, many wake up and find that they are managing 30,000 virtual machines. This is partly because server build times are reduced from days to minutes, and partly because server consolidation on virtual infrastructure is so effective that idle guests are barely noticed. If left unmanaged, the “gasoline fire” spread of VMs could create a burden in the form of permissions, backups, upgrades, patches, and monitoring than what existed in the physical infrastructure.

To combat these scenarios, IT departments need to start with process and policies. Change control becomes paramount. Second in importance is a new server request and approval process. Build standards need to be created and adhered to. Finally, virtual servers must to be audited for activity and then powered off or removed if idle. The good news is that there are numerous life cycle management and automation tools now available for helping an organization provision, maintain, report, and decommission virtual machines.

Backup/Restore pitfalls

Backup is another area where TCO and ROI, as well as performance, can be negatively impacted by virtualization without appropriate planning. As mentioned above, virtual server hardware is often sized based on the average server load. For example, consider the impact of combining ten servers with average CPU utilization of just 10% onto a single physical device. The ROI would appear to be excellent, but the large spike in utilization that happens during backup can cause unacceptable performance issues or even application failures. The performance hit during backup can be mitigated through off-machine backup using like VMware VCB or storage snapshots, but the cost and effort of implementing these features must be taken into account when planning the new system. The conventional approach to backup, loading backup agents on each system and backing them up on a schedule, does not translate in the virtual world.

Disaster recovery pitfalls

One of the primary driving factors for moving to a virtualized environment is its potential positive impact on disaster recovery (DR). Although the DR benefits are real, the requirements for the storage system can become a pitfall. Extra DR data is pushed onto storage arrays and then often replicated to another location. All of this extra data requires increased storage capacity, array features, and WAN bandwidth. The capacity issues can potentially be addressed using primary storage deduplication and thin provisioning, and WAN optimization appliances can reduce bandwidth requirements.

While virtualization does make servers more portable, thus making DR easier, the disaster recovery copy process is a major challenge. Some vendors now have tools like VMware Site Recovery Manager (SRM) to assist in this orchestration, but these tools are somewhat immature, and care should be taken when planning to use them. For example, most do not address failback, so administrators may find themselves doing a lot of work to return to operation at their primary data center after a failover.

© Gestalt for Gestalt IT, 2009. | Planning for Virtual Infrastructure: Avoid the Pitfalls
Read more posts categorized as Exclusive, Featured, Server Virtualization, Top Story