For a great overview of the entire U1 release check out Rick Vanover’s post vSphere 4.1 update 1 released from his Rickatron Blog and via his Servers and Storage Column/Blog at TechRepublic.

The rest of this post contains cut and pastes from the Release Notes and some commentary about them from me. I want to stress again that these are issues that have now been fixed!

Finally, I’ll point out the one huge VM backup issue (that I can think of right now) that still does not appear to be resolved.

• Cannot take quiesced snapshots of Microsoft Windows Server 2008 R2 virtual machine running vCenter Server 4.1 When creating a snapshot of a Microsoft Windows Server 2008 R2 virtual machine that has vCenter Server 4.1 installed, the snapshot operation might fail to complete. This issue occurs on Microsoft Windows Server 2008 R2 virtual machines when the ADAM database is installed. The issue is resolved in this release.

The change from 2008 to 2008 R2 has caused quite a few applications to stumble, so it is good to see that VMware users can now install vCenter on the latest and greatest OS from Microsoft running as a virtual machine without having to worry about various hassles, not to mention coming up with a separate backup or business continuity solution for arguably one of the most important VMs in the environment.

• Creation of large .vmdk files on NFS might fail. When you create a virtual disk (.vmdk file) with a large size, for example, more than 1TB, on NFS storage, the creation process might fail with an error: A general system error occurred: Failed to create disk: Error creating disk. This issue occurs when the NFS client does not wait for sufficient time for the NFS storage array to initialize the virtual disk after the RPC parameter of the NFS client times out. By default the timeout value is 10 seconds. This fix provides the configuration option to tune the RPC timeout parameter using the esxcfg-advcfg -s <Timeout> /NFS/SetAttrRPCTimeout command.

Although specific to users of NFS storage, this one was definitely a head scratcher. Possible scenarios where this problem could rear it’s ugly head include full VM restores and even VM replication. Add the possibility of dedicating a large .vmdk as the backup repository for your backup server VM.

Let me know if I missed any other resolved issues specific to VM backups.

The issue still unresolved

Unfortunately, nothing in vSphere 4.1 U1 appears to address the loss of connectivity when committing snapshots for VMs using Change Block Tracking (CBT) while running on NFS storage.  This leaves VMware admins stuck between a rock and hard place – disable CBT or move VMs off of NFS. More about the problem can be found in VMware’s KB Article:

Virtual machine freezes temporarily during snapshot removal on an NFS datastore in a ESX/ESXi 4.1 host

© Rich for Gestalt IT, 2011. | vSphere 4.1 U1 Released. Fixes Specific For VM Backups
Read more posts categorized as All, Server Virtualization

The following screenshot shows the mysterious snapshot as I found it on my lab domain controller

In my case, the scheduled job that needs to create and then normally commit the snapshot is a Veeam Backup and Replication job. At some point in the life of my lab I did run out of space, and although it’s not an issue anymore, the snapshot still exists even when the jobs aren’t running! I was a bit surprised that backup jobs could even complete with this snapshot in place.

VMware KB article 1003302 explains that the Consolidate Helper snapshot is actually created when trying to commit (delete all) a snapshot.

From Delete All snapshot operation results in a Consolidate Helper snapshot when a datastore has insufficient disk space:

“If you try to initiate a Delete All snapshot for a virtual machine using Snapshot Manager, and if that virtual machine is on a datastore that does not have sufficient space for the snapshot, the following message displays in VMware Infrastructure (VI) Client:

msg.hbacommon.outofspace: there is no more space for the redo log of <VMname>-0000xx.vmdk.

You are given the option to abort or retry.

• If you choose Abort, the virtual machine is powered off, the snapshot is aborted, and a Consolidate Helper snapshot is created. The Snapshot Manager UI displays that Consolidate Helper snapshot. You can delete the Consolidate Helper snapshot after you have made space available.
• If you click Retry, the Snapshot Manager returns to Consolidate Helper snapshot mode unless you have made more disk space available.”

Personally, I never saw the VI/vSphere Client message.

The KB Article suggests deleting the snapshot when your datastore has enough free space and even offers instructions for extending your datastore.

Luckily, my snapshot was only consuming 29.14 KB of space so it went under the radar for about 8 days. I’ve blogged before about how unnoticed ESX snapshots can be dangerous.

Clicking the Delete All button from the Snapshot Manager Window (previous screenshot) after making sure there were no active backup jobs removed my snapshot painlessly this time.

This apparently is one of a few new Changes to the ESX Snapshot mechanism.

© Rich for Gestalt IT, 2010. | Consolidate Helper Snapshot Appears On vSphere VM
Read more posts categorized as Server Virtualization

I’m automatically pulling in the tweets of all involved, so I should have a good stream of both bloggers and sponsors represented here.

I’m using the following twitter lists:

• @TechFieldDay/tfd3-sponsors
• @TechFieldDay/tfd3-delegates

Look for my occasional commentary as well, but I’m not sure how much time I’ll actually have to participate. I’ll end the coverage when our time is up, which should be roughly 3 to 4 hours.

Click through the page break for the CiL widget and tune into the fun!

By the way, we are providing the first ever public demo of Veeam Backup and Replication 5.0 SureBackup!

© Rich for Gestalt IT, 2010. | Live Coverage – GestaltIT Tech Field Day Veeam Presentation
Read more posts categorized as Events, Server Virtualization

Side note – I hate saying “V A A I”. IMO it’s quite a tongue twister. I have adopted saying “V double A I” because it is much easier to get out.

VAAI Introduced 2 years ago

Although it does not exist in vSphere 4.0 today, believe it or not VAAI was first introduced at VMworld 2008 when the vStorage API was announced.

VMware Unveils New vStorage Technologies to Deliver Leading Efficiency and Manageability for the Virtual Datacenter Operating System

New Capabilities Deliver Deeper Integration with Storage Partner Functionality
vStorage enables intelligent integration of storage products within the Virtual Datacenter OS through the new vStorage APIs.

vStorage APIs deliver tight integration of advanced capabilities from storage partners with the Virtual Datacenter OS from VMware.  vStorage APIs for array integration will enable customers to leverage array-based capabilities, such as snapshots, provisioning, replication and restore, directly with individual virtual machines in conjunction with the clustering and pooling capabilities of VMware Infrastructure.  vStorage APIs for multi-pathing will provide customers the ability to integrate advanced load balancing capabilities provided by leading partners’ multi-pathing software with their virtual environment.

If you are like me you were probably so overwhelmed by the news of the VCD-OS (That’s what VMware was calling the hybrid Cloud at the time … I guess?) that you missed (or were not capable or ready to comprehend) how complete of a strategy already existed for building Infrastructure as a service (IaaS) with the vStorage API. For me, looking back 2 years later it is amazing to me it was all there – at least in concept.

Don’t confuse VADP and VAAI

VAAI is only one part of the vStorage API. VADP (vStorage API for Data Protection) is another, but the these two seem to have been easily confused lately. VADP is enabling access to files on VMFS without VCB, but VAAI has nothing to do with that today. Will it in the future? Maybe if the process of creating and committing snap shots can be handled by the storage device processor, but that’s the key point and difference. VAAI will offload functionality to the storage processor and thus reduce compute load on the hypervisor. VADP allows third party technology partners to perform full image virtual machine backups.

What vSphere functions can be offloaded with VAAI?

Once again I’ll link to a post from VMworld 2008. Chad Sakac (@sakacc) wrote about what VAAI back then when it was first announced. In So, what does vStorage really mean? Sakac lists several functions that a VAAI aware storage device could do. I’ve pasted and edited for length Sakac’s original list. Quoting Chad:

• Acceleration via storage offloads using SCSI driver primitives [omitted] - think “I/O” dedupe (not to be confused with Storage Dedupe).  This can have a HUGE impact on storage network utilization – much more efficient, and very important for lower bandwidth storage networks like 1Gbps iSCSI.
• Thin Provisioning Integration
• The ESX server tells the array what BLOCKS makeup a VM, so svmotion can be offloaded to the array as an alternative to the file-copy method now, snapshots can potentially be done on VMs even for block devices.  Again, this uses SCSI driver primitives

How will VAAI do this?

Much of this still remains to be seen, but according to Sakac and the others I’ve spoken with about VAAI recently, you’ll be able to buy a storage device with processors that are VAAI capable. That means a hardware list that will probably start out small and grow quickly. Sakac likens VAAI integration in the future to buying servers with Intel VT-x or AMD-V technology today. You just don’t buy servers for vSphere without these on chip technologies.

In my own terms, a VAAI  aware storage device will be able to understand commands from vSphere and perform storage functions with it’s own compute resources.

When will we see this new hardware? Honestly I’m not sure, but I imagine we are getting close.

So, what does VAAI mean for vSphere virtualization?

Simply put, VAAI will allow more hypervisor compute power to remain available for running virtual machines. This means greater VM to host ratios without more physical resourcesand even larger workloads in a single VM. This also means faster VMotions, Storage VMotions, cloning, VM deployment and more. I’m guessing that this intelligent “pillow talk” between vSphere and storage will ultimately lead to safer disk over allocation and dynamic adjustments to storage workloads never before possible. Those are just a few possibilities off the top of my head.

© Rich for Gestalt IT, 2010. | What Is VAAI And What Does It Mean For Virtualization?
Read more posts categorized as Featured, Server Virtualization, Storage

“I wanted to remind everyone, of what I have already seen floating around the internet, but still important enough to remind.  Our next release of SRM is going to require a 64 bit OS.  This is the same as our next release of VC as it too will require a 64 bit host OS.  This change is required to support the increased capabilities of our products.  As we scale our products to match our customers needs, generally 1 – 2 years in advance of where they will need all the capabilities of a given product we have had to use a 64 bit OS.  This will show itself in increased numbers in things like more simultaneous vSphere client connections.”

To me these new operating system (OS) requirements mean we will see even more instances of vCenter as a VM (virtual machine). It only seems logical that a least path of resistance is to virtualize the management server in order to upgrade, especially considering all have already invested in 64 bit hardware for their hypervisors if they decided to upgrade to vSphere 4 in the first place. To go a step further, I’m willing to argue that it will be more common for an IT Department to justify the cost of additional ESX hosts, even if only dedicated for management, then it will to deploy new servers for physical instances of vCenter.

The looming transition to a console-less ESXi eventually means more management virtual appliances in the future too. Solutions which will continue to need a ESX console or similar will have to substitute their own appliance to operate with ESXi. This means even more justification for additional ESX/ESXi hosts and thus greases the decision to virtualize vCenter as well. I expect to see management clusters of ESX hosts become more common in the future than even the use of management networks today. ESX hosts have bigger and badder hardware now than ever before allowing for higher consolidation ratios and larger applications to easily run in virtual machines, but it will be interesting to see if the vCenter as a VM best practices change over time. I personally feel that continuing to separate the database from the virtualized vCenter will continue to be a smart choice. Running a separate, and even virtualized, SQL instance ensures not only better performance of vCenter as a VM but enhances DR scenarios. In fact, those that already have the vCenter database on a remote instance will likely have a safer upgrade to the 64 bit vCenter. The new 64 bit requirements will no doubt make for an interesting migration scenario, and I’m sure we will see some positive and negative opinions. Let me know your thoughts on a 64 bit vCenter as a VM in the future!

© Rich for Gestalt IT, 2010. | Future vCenter And SRM Requirement For 64 bit OS Means More vCenter VMs
Read more posts categorized as Server Virtualization

Here’s the VMware Communities: Survey for SRM customers – please … original post from May 27:

Hello SRM users, The VMware vCenter Site Recovery Manager (SRM) product team is looking for product feedback on SRM deployments. If you have purchased SRM, we would like to hear from you. Your participation will be very valuable to us and the information you provide will be used to improve the SRM product going forward. You can provide your feedback by completing the survey The survey should take no longer than 15 minutes and will expire on June 10, 2010. Please note that this survey is for SRM customers only. Upon completion of the survey, if you are among the 1st 1000 respondents, VMware will donate $10 per response to charity. You will also receive a link to download the electronic copy of Mike Laverick’s book “Administering VMware Site Recovery Manager 4.0” upon completion of the survey. We appreciate you taking the time to provide us with your valuable feedback. Thank you, The VMware SRM Team

Check out Laverick’s post on this survey as well: SRM Customer? Complete this survey, and VMware will donate $10 to UNICEF

© Rich for Gestalt IT, 2010. | Take VMware SRM Survey, Get a Free Copy of Laverick’s Book, and Donate $10 to UNICEF
Read more posts categorized as Featured, Server Virtualization, Top Story

As I understand Local Mode, VMware View will be able to coordinate the offload of intensive desktops operations such as graphics to the hardware of a user’s notebook, thin client, or PC. Since the VMware View Client is installed as an application on the operating system of the end user’s system, this seems to be a better technical fit with VMware’s expertise in type 2 hypervisor products, or hosted virtualization, such as VMware Workstation, Server, Player, and Fusion. This focus in development also seems more aligned with the Teradici partnership enabling the software based PCoIP protocol.

Combined with the promise of offline synchronization, where changes to the local copy of the VM will be replicated to the master copy in the data center, VMware View Local Mode would definitely make VMware View instantly compatible with the widest selection of hardware possible. This in turn creates the least path of resistance for the VMware VDI solution to provide remote and mobile access to a corporate virtual machine even when users are abroad.

Developing a bare metal client hypervisor has obviously been a bigger than expected challenge for both Citrix and VMware judging by the delays in release experienced by both companies so far. As more and more corporations consider the concept of BYOC (allowing employees to bring your own computer), VMware now seems to be backing out of the hardware platform support arena and falling back to what they do best. Although I believe a smart decision has been made here, there will no doubt be some public “crow to eat” in Palo Alto.

Be sure to read the entire VMware View Blog post linked above, but the rest of this post is the VMware Desktop Team’s own words about View Client Local Mode:

Enter View Client with Local Mode.

Unlike what others are pushing with much hyped “Type-1″ client hypervisors, View Client with Local mode installs cleanly and non-destructively on an existing Windows Laptop or PC.  You don’t need to start with brand-new hardware from an OEM.  You don’t need to wipe out or destructively modify an existing OS installation and void your service/support contracts.   It’s “just software” that solves real problems.

Once View Client with Local Mode is in place, you’ll be able to do things like…

As an end user:

• Checkout a desktop from the datacenter to run on the client device regardless of network connectivity.
• Enjoy a full fidelity desktop experience with 3D DirectX9c support, Aeroglass effects on Win7, support for devices like web-cams and VOIP.
• Immediately get access to a replica desktop if the client device (e.g. laptop) fails or is lost.
• Get a real choice of client devices.

As an administrator:

• Manage and monitor Local Mode desktops centrally from the same View administrative console and VM images as the rest of a VMware View deployment.
• Maintain control over end user desktops – revoking privileges and pulling data back to the datacenter on demand.
• Know corporate data is secure from outside attack with AES 128 or 256-bit encryption.
• Reduce overall CAPEX costs, both on newly purchased client devices, and lower upfront datacenter investment.
• Leverage an existing hardware fleet to deploy Local Mode desktops.
• Cleanly separate the Windows update/refresh cycle from hardware update/refresh cycles.

To visualize all this, below you see a managed Windows XP desktop running Google Earth with 3D buildings enabled (using DirectX) on top of an unmanaged Windows 7 laptop:

© Rich for Gestalt IT, 2010. | VMware Answers XenClient Release With View Client Local Mode
Read more posts categorized as Server Virtualization

Almost a week later and after a flurry of comments from most of the vendors leveraging CBT for virtual machine backups, VMware has published a KB article on the subject.

VMware KB: Reverting to a pre-existing snapshot under specific conditions can cause incremental backups based on CBT (Changed Block Tracking) to become inconsistent

The KB Article describes the exact scenario that causes the problem:

Four things need to occur in the following sequence before there is a possibility of this issue occuring:

1. A VM with hardware version 7 needs to have a snapshot present AND has been backed up previously by a backup product leveraging CBT
2. A backup product performs an incremental backup of VM and leverages CBT to determine changed blocks since last backup
3. After incremental backup is complete, user manually reverts snapshot on the VM
4. A backup product performs an incremental backup of VM and leverages CBT to determine changed blocks since last backup

This issue is caused by arguably a unique set of circumstances, but it is important for VMware administrators to be aware of none-the-less. I’ve blogged about (other bloggers blogging about) ESX snapshots being like a loaded gun before, and here is another example of why.

VMware offers the following resolution for now:

“The workaround for this issue is to do a full VM backup after a snapshot revert operation. If the backup application does not allow this as an option, you will need to remove the CTK files for that VM. The CTK files mentioned are stored with the virtual machine on the datastore and can be removed via the Datastore Browser. This delete operation can be safely done while the VM is running.”

Disclaimer: I work for Veeam Software, the creators of Veeam Backup and Replication.

Veeam Software has confirmed that Backup and Replication v4.1 successfully handles this issue without corruption in all but one specific scenario of manually caused events as described in the Veeam Forums here: http://www.veeam.com/forums/viewtopic.php?f=2&t=3699&p=15139#p15139.

The hotfix has been created for this one remaining scenario and is in “testing for validation” as of this writing.

In the same linked thread Veeam also recommends that until the patch is available workaround this last scenario by “disabling the use of changed block tracking in the Advanced job settings for all jobs which process VMs where manual snapshot reversal may happen, and triggering Full Backup on these jobs to heal the backup file (in case you believe you may have this scenario happened before for some VMs).”

© Rich for Gestalt IT, 2010. | Pre-existing Snapshot Could Cause Inconsistent Incrementals Using vSphere CBT
Read more posts categorized as Server Virtualization

For those who do not know, the vExpert Award was introduced by VMware in 2009. Best described as an award similar to Microsoft’s MVP, I was lucky enough to be named one of the 300 vExperts awarded last year. Check out my acceptance speech post from last February. All joking aside, the award distinguishes those “who have significantly contributed to the community of VMware users over the past year.”

The application process is not a contest or an election campaign, so if you have a candidate that may not be as well known in the vEcosphere already but has significantly contributed to the VMware and virtualization community be sure to fill out an application for him/her. All it takes is one application to recognize someone for their efforts in 2009!

© Rich for Gestalt IT, 2010. | Nominate 2010 vExperts Now
Read more posts categorized as Server Virtualization

So, this post is about taking a deeper look at where the master template VM resides versus where the cloned template is destined. My math my be a little off or may not account for every factor involved, but my point is to be close enough to demonstrate that the disk/array/LUN design is can be the culprit more times than not.

When I started this post I emailed for some help. I asked for a sanity check from some storage experts. I’ve been reasonably happy with my own answer until now, but I figured I do some research before adding the content to VM /ETC. I got back a single reply that I am paraphrasing: “Sounds about right. Let me think about it some more and if I can stump you with anything else I’ll let you know.” He never did so I’ll take that as a positive confirmation meaning “yes VM moron, it is that simple.” Good enough for me! If anyone can point out any other factors I am not properly accounting for please leave a comment.

The following is part of my email for help. It not only explains my test scenario but it illustrates the problem and resolution as well. At the end of this post I make some suggestions for bettering the time it takes to clone a VM.

The email for help

Oh wise and all powerful masters of the disk,

I humbly submit the following concept for your review. Guide me to a greater disk performance understanding when cloning a VM in VMware ESX environment.

Here’s the scenario:

• Cloning a VM takes a long time – 10 GB VM using only 3.5 GB of space takes roughly 45 min to an hour to clone.
• The master template and the clone reside on the same disk and NFS mount.
• Yeah, it’s a single SATA disk in a lab. I know, it should suck.

I’m trying to explain the expected speed of read and writes using the IOPs calculator here: http://www.wmarow.com/strcalc/

See the attached screen shot for the values I put in the calculator, but the results I’m interested in are:

• with 50% reads and 50% writes (master and clone on same disk) average throughput (MB/s) is 1.2
• I used 50% reads and 50% writes for the cache.

which means to me that

• 3548 MB / 1.2 (MB/s) = 2957 secs or 0.82 hrs.

Here’s the screen shot of the IOPS Calculator I linked in the email for help:

Suggestions for improvement

Obviously, the type/performance of the disks, the number of disks, and the type of array makes a huge difference. I should also point out that I am using 8 ms as the value for the seek latency. I’m not as focused on technical accuracy because my point is served without it, but changing this value makes a significant difference as well. If you want technical accuracy and more explanation about some of the numbers to use in the calculator check out these posts on the topic of IOPS and the impact on a virtual environment:

• http://www.yellow-bricks.com/2009/12/23/iops/
• http://vmtoday.com/2009/12/storage-basics-part-ii-iops/
• http://virtualgeek.typepad.com/virtual_geek/2010/02/solving-a-weird-slow-performance-cloning-issue.html
• http://vpivot.com/2009/09/18/storage-is-the-problem/

In my case, moving the VM template to another disk/array or increasing the number of disks used on my NFS server would help because the reads and writes would be separated when the cloned VM resides on a different disk/array and the number of IOPs possible would be increased with more disks. Yes, this post uses a single SATA disk as a simple example, but the point is hopefully clear. Use the same logic and math for shared storage scenarios, all storage protocols, any vendor’s storage device, and all RAID types. Plug those values in the IOPs calculator to calculate your own results.

My ultimate point is to make everyone think about how the disk/array/LUN design decisions impact the behaviors of the virtual infrastructure.

As an example, if my lab NFS server was using 6 SATA disks configured as a RAID 5 array the calculation for expected time to clone changes as follows:

• 3548 MB / 2.99 (MB/s) = 1187 secs or 0.33 hrs.

Better, right? Hey, it’s a basement lab. It’s supposed to suck!

© Rich for Gestalt IT, 2010. | Why Does Cloning A VM From Template Take A Long Time?
Read more posts categorized as Featured, Server Virtualization, Storage