I am in Seattle, WA for the third GestaltIT Tech Field Day (TFD). This time I am part of the presenting team for Veeam Software, and we are first up on the agenda. As with other recent events, I am using Cover It Live (CiL) to provide real time coverage and of our presentation and the reaction of the delegates (bloggers) attending. The best part about using Cover It Live is the event can be played back in the future, so check it out even if you missed it live.

I’m automatically pulling in the tweets of all involved, so I should have a good stream of both bloggers and sponsors represented here.

I’m using the following twitter lists:

• @TechFieldDay/tfd3-sponsors
• @TechFieldDay/tfd3-delegates

Look for my occasional commentary as well, but I’m not sure how much time I’ll actually have to participate. I’ll end the coverage when our time is up, which should be roughly 3 to 4 hours.

Click through the page break for the CiL widget and tune into the fun!

By the way, we are providing the first ever public demo of Veeam Backup and Replication 5.0 SureBackup!

© Rich for Gestalt IT, 2010. | Live Coverage – GestaltIT Tech Field Day Veeam Presentation
Read more posts categorized as Events, Server Virtualization

As follow-up to my blog here; I’d like to share yet more thoughts on availability and the potential negative impacts on some of the new technologies out there.

How many of you run clusters of servers? HA/CMP? Veritas Cluster? Microsoft Cluster? VMWare Clustering? I suspect lots of you do? How many of you cluster NAS heads? Yet again, I suspect lots of you do? How many of you cluster arrays? Not so many I guess? Certainly in my experience, it is uncommon to cluster an array. And when I talk about clustering an array, I don’t mean the implementation of replication.

So, if you don’t cluster your arrays; how do you protect against the failure of a RAID rank? Statistically unlikely but it is it more or less unlikely than a loss of data-centre? I’m not sure and the failure of a RAID rank for many people could well mean the invocation of the disaster recovery plan. Why?

The loss of a RAID rank might well lead to the loss of an application/service and if it is an absolutely business critical service, can you bring it up at the remote replication site in isolation? As a discrete component? If you can, can you cope with increased transaction times due to latency? Many applications now have complex interactions with partner applications; these might not be well understood. So the failure of RAID rank could lead to the invocation of the Disaster Recovery Plan. Actually in my experience, this is very nearly always the case unless the service has been designed with recovery in mind; this requires infrastructure and application teams to work together, something which we are not exactly good at.

But you now take the challenge and make sure that every application can be failed over as a discrete component. Excellent, a winner is you! You know the impact of loosing a RAID rank, you know what applications it impacts, you’ve done your service mappings etc, etc. And you have been very careful to lay things out to minimise a single RAID failure’s impact.

And then you implement automated storage tiering. Firstly, you now have no idea in advance what impact a RAID rank failure may have; you have no idea what applications may be impacted. And actually, the failure of a single RAID rank may well have huge impact. We could be looking at restoring many terabytes of data to cope with the failure of a couple of terabytes and many applications failing.

It will depend on the implementation of the automated storage tiering and I am concerned that at present we do not know enough about the various implementations which will hitting our arrays over the next eighteen months. So despite automation making things day-to-day a lot easier, we cannot treat it as Automagic Storage Tiering; we need to know how this works and how we plan to manage this.

And perhaps for key applications, we will need to cluster storage arrays locally; that in itself will bring challenges.

I’m still a big fan of automated storage tiering but over the next few months, I would like to see the various vendors start talking about how they mitigate some of this risk. Barry Burke has made a big thing about the impact of a double disk failure on an XIV array in the past; in a FAST v2 environment, I would like to see how EMC mitigate against very similar problems.

I would also like to know what impact of a PAM card failure from NetApp is; does the array degrade to the extent where it is not useable? What kind of tools can NetApp give me to assess potential impact. As Preston points out here; failure of individual components within an array could have significant impacts.

We are heading to a situation where technology gets every more complex and arguably ever more reliable. But we rely on it to ever more greater extents; so we must understand risks and mitigations to a much greater amount than we have in the past.

© Martin for Gestalt IT, 2010. | Disastrous Thinking
Read more posts categorized as Storage

This is no joke.  Right now southern Indiana is under water.  Some organizations I have worked with are fine but Columbus Regional Hospital and many others aren’t.  Their data center was in a basement (which isn’t rare) and took on a surge of water.  Within what may have been a few hours, it was completely submerged.  They had to evacuate all patients to other area hospitals.  They’ve got a lot to overcome in the next couple of weeks but with the proper infrastructure, the pain can be lessened.

It’s true that most causes of data unavailability are human (planned and unplanned) and that natural disasters rank low on the list.  However, this is no reason to sit idle.

Things to think about:

1. How long can I survive without my information infrastructure
2. Do I have a good copy of data offsite that meets my recovery point objectives?
3. What will a recovery look like (local, geo-dispersed, etc.)
4. Do I have a REAL plan I can act on?
5. What could make recovery easier (VMware SRM, async replication, etc.)?

The point isn’t to ask you a comprehensive list of questions.  The point is to get YOU to think about the fact that it’s only a matter of time before something happens.  Columbus Regional Hospital was unlucky.  I’m not sure of the level of DR plans they have but I do know that many of the folks there have been working hard to improve this prior to the flooding that just occurred.  We don’t have enough hours/days/minutes to be completely prepared but it would serve everyone well to sit down and think about the details of their business continuity plans.

I’ve been with enough organizations to know when people are confident about their DR/BCP plans and when they’re just closing their eyes and filling out binders of material with the thought, “This is the best we’re going to do, there is so much other stuff that has to get done first that this BCP plan won’t even matter.”  There are many local vendors around that can help you plan.  Just think how easy it would be to recover with a few servers running VMware and virtual machines ready to turn on the instant something happens.  It may not be the whole piece to your DR strategy but it would get you up and running a lot quicker.

© edsai for Gestalt IT, 2009. | Why you need to be serious about Business Continuity
Read more posts categorized as All, Featured, Storage

CC-by-SA 2.5 image "Golf Bunker" by Ken123

Today, many companies are considering a compete migration from a physical to a virtual infrastructure. Based on the promises of cost savings, administrative efficiencies, and improved resource utilization, virtualization is seen as the technology that makes it possible to do more with less. However, there are many pitfalls to consider when virtualizing server infrastructure. This article suggests planning decisions to be considered by the CIO, IT Director, design architect, and IT Manager that, if overlooked, could ruin the total cost of ownership (TCO) and the return on on investment (ROI) expected from this virtual infrastructure.
�

Migration pitfalls

Implementing virtualization is not a simple process of building a server. Too often, management makes the mistake of expecting that the administrators and architects of the server team will be solely responsible for virtualization decisions. In reality, the impact of a move to virtualization should be viewed exactly as if a company were moving their physical servers to a new data center. Storage, networking, security, Active Directory, messaging, web presence, and all of the systems and services necessary to the organization must be considered together. Some of these groups should be more involved than others in the actual planning and ultimate design decisions, but virtualization strategy affects everyone. Key members from all of these focus areas must be trained and given panning responsibilities to achieve a collaborative plan and design.

A potentially larger “gotcha” once everyone is involved in the planning process is the temptation to include all of the upgrade and service redesign projects that have been “in the queue” for months (or longer). Although the migration may seem like a great opportunity to upgrade to the latest operating system, separate instances from poorly performing database servers, or build the new CRM system, these added activities bring complexity and could pose road blocks to the core goal of the virtualization initiative: consolidating hardware. These projects should be tabled and taken on later, after the new infrastructure is in place. In fact, the flexibility of a virtual infrastructure will actually make these projects simpler.

Performance pitfalls

Condensation of numerous physical servers into a virtual environment is intended to increase the effective utilization of processing power and other system resources, but it is easy to overshoot the target and overload the new system. Although the average load of a number of servers might be within the capabilities of a new virtual infrastructure, resource spikes can be a serious problem. Consider that periods of high demand tend to cluster across servers through the day and week, as employees begin the workday, the backup process begins, or a software build is kicked off. If performance is not measured and correlated carefully before migrating to a virtual infrastructure, these resource spikes can make the new infrastructure unacceptably slow.

One often overlooked performance challenge is the randomization of demands introduced by virtualization. Traditional computer design assumed that processes would proceed through their steps sequentially, with the system devoting resources from start to end. Multitasking operating systems challenged this assumption, but the allocation of memory, I/O, and storage remained clustered. But highly-utilized virtual servers upset this predictable flow, rapidly switching from one task to another entirely different one. In short, virtualization randomizes accesses that were once sequential and transforms large requests into many tiny ones. CPU manufacturers have responded with clever techniques to save and restore registers and handle memory mapping, but this blender effect is far from solved. I/O and storage systems are particularly affected, as they are optimized to stream large sequential operations rather than small random ones.

Storage pitfalls

Although one might think that virtualization reduces storage requirements as systems are consolidated, the opposite is often true. Many of the functions that help drive down CAPEX and OPEX costs in virtualized environments require the use of shared networked storage like SAN or NAS. Data that had once been scattered around the data center on internal drives is consolidated on these networked storage devices. Thousands of redundant copies of operating system files, for example, end up sitting on the storage array. The ease of creating virtual machines from templates tends to lead to virtual “server sprawl” far worse than in the physical world. Storage array capacity is also needed for swap space or paging files as well as popular enhancements like snapshots, DR copies, and backup images.

The result is a flood of redundant data that must be taken into account when planning capacity needs. Although storage is typically not the top item in an IT budget, virtualization can cause it to rise and chew up some or all of the savings from the project. It is difficult to avoid these issues, but some techniques can help mitigate the problem. For example, deduplication of primary storage and thin provisioning can help reduce the storage footprint of the virtualized infrastructure.

Administration pitfalls

Virtual sprawl can cripple infrastructure support teams if it is not managed. The challenge of administering this new virtual environment is often ignored in the implementation rush. Rather than managing 3,000 servers, many wake up and find that they are managing 30,000 virtual machines. This is partly because server build times are reduced from days to minutes, and partly because server consolidation on virtual infrastructure is so effective that idle guests are barely noticed. If left unmanaged, the “gasoline fire” spread of VMs could create a burden in the form of permissions, backups, upgrades, patches, and monitoring than what existed in the physical infrastructure.

To combat these scenarios, IT departments need to start with process and policies. Change control becomes paramount. Second in importance is a new server request and approval process. Build standards need to be created and adhered to. Finally, virtual servers must to be audited for activity and then powered off or removed if idle. The good news is that there are numerous life cycle management and automation tools now available for helping an organization provision, maintain, report, and decommission virtual machines.

Backup/Restore pitfalls

Backup is another area where TCO and ROI, as well as performance, can be negatively impacted by virtualization without appropriate planning. As mentioned above, virtual server hardware is often sized based on the average server load. For example, consider the impact of combining ten servers with average CPU utilization of just 10% onto a single physical device. The ROI would appear to be excellent, but the large spike in utilization that happens during backup can cause unacceptable performance issues or even application failures. The performance hit during backup can be mitigated through off-machine backup using like VMware VCB or storage snapshots, but the cost and effort of implementing these features must be taken into account when planning the new system. The conventional approach to backup, loading backup agents on each system and backing them up on a schedule, does not translate in the virtual world.

Disaster recovery pitfalls

One of the primary driving factors for moving to a virtualized environment is its potential positive impact on disaster recovery (DR). Although the DR benefits are real, the requirements for the storage system can become a pitfall. Extra DR data is pushed onto storage arrays and then often replicated to another location. All of this extra data requires increased storage capacity, array features, and WAN bandwidth. The capacity issues can potentially be addressed using primary storage deduplication and thin provisioning, and WAN optimization appliances can reduce bandwidth requirements.

While virtualization does make servers more portable, thus making DR easier, the disaster recovery copy process is a major challenge. Some vendors now have tools like VMware Site Recovery Manager (SRM) to assist in this orchestration, but these tools are somewhat immature, and care should be taken when planning to use them. For example, most do not address failback, so administrators may find themselves doing a lot of work to return to operation at their primary data center after a failover.

© Gestalt for Gestalt IT, 2009. | Planning for Virtual Infrastructure: Avoid the Pitfalls
Read more posts categorized as Exclusive, Featured, Server Virtualization, Top Story

Now that I’ve sat the VMware Site Recovery Manager (SRM) class, done the labs, and had some design and implementation time with the product I am reminded of a scene from the movie The Wizard of Oz. “Pay no attention to that man behind the curtain” is a famous line from the movie which comes from the scene when Dorothy and gang discover that the mighty and powerful Wizard they fear is really just an elaborate machine controlled by an ordinary man.

I am not suggesting that SRM is a sham. In fact, it provides automation of virtual infrastructure fail over between sites that is truly wizard-like. Understand however, VMware SRM software is just the last piece of the total data center recovery “machine”. Many organizations may be seeking the semblance of automated site fail over, but have they really considered in detail what it takes to start up their business critical systems at a secondary location?

A simple determination of readiness for SRM’s wizardry is answering this question: “Can you create (or have you already created) a document listing the complete shut down and start up of your business infrastructure?” Many call this a disaster recovery playbook or runbook. Better yet, have you provisioned and tested the physical resources you need to actually fail over to another location based on the runbook that was created? If and when that’s done, there are numerous business continuity technologies considered other than SRM. Those that have already realize that SRM along with consolidation to virtual infrastructure will replace several sections of their runbook and several pieces of hardware. The point, however, is that SRM does not replace the contents of the entire runbook.

For those that are considering SRM, take the time to at least put on paper every possible step you would need to restart your business in another data center. This includes the mundane like providing power and cooling, racks, internet access, and telephones as well as the mission critical like employee access, email, and business processing. Analyze how much time it would take to rebuild and restart systems, and what recovery point objectives (RPO) and recovery time objectives (RTO) are acceptable for each system or service. When it’s time to map your disaster site fail over solution to the runbook you will clearly see the efficiencies and the speed that VI 3.5 and SRM allow.

Therefore, unlike the line from The Wizard of Oz, you better pay close attention to the man and machine behind the curtain in order to achieve the prodigious expectation and pyrotechnic results of a SRM implementation.

© Rich for Gestalt IT, 2009. | Implementing VMware SRM: Pay Attention to that Man Behind the Curtain
Read more posts categorized as All, Server Virtualization, Top Story