Stepping out of a subway car is an entirely different matter when it's moving!

Mobility of virtual machines is a sticky wicket. As open systems infrastructure becomes increasingly virtualized, administrators and managers wish to use the technology to balance workload, ease migration, and provide better availability. Although technology is improving, actually moving virtual machines is not always a piece of cake. Let’s lay down a baseline of information so we may begin a discussion on the true nature of virtual machine mobility.

Mobility of What?

Let us consider first the question of what exactly is being moved. Systems administrators often focus on “the machine”, which encompasses the operating system and configured state of the virtual machine itself. But the true “mass” of the system is its stored data. Hypervisor vendors have come up with different techniques of moving these two essential elements, reflecting the unique characteristics of each.

• The virtual machine is an instance of an operating system along with its state and configuration. Mobility of virtual machines requires all of this to be preserved, along with any I/O channels. Live migration of virtual machine requires that any active network sessions be maintained, along with RAM content, registers and buffers, and so many other elements.
• The virtual machine image (commonly referred to as “storage”) is the static content addressed by a virtual machine. Typically a VMDK or similar virtual disc image, it must be accessible to the virtual machine at all times. Live migration of a virtual machine image is tricky, but perhaps not quite as complex as live migration of a running operating system.

VMware, Microsoft, and others recognize these 2 distinct elements to be migrated, and have come up with a variety of complementary technologies for each:

• vMotion is VMware’s virtual machine migration solution, and has continually evolved with each iteration of the hypervisor. DRS leverages vMotion to automate mobility. VMware has also created Storage vMotion and Storage DRS as complements to handle mobility of virtual machine images.
• Microsoft Hyper-V Live Migration is conceptually similar to vMotion, though newer and less full-featured. With Hyper-V 3.0, Microsoft will introduce Storage Live Migration as a complementary technology akin to Storage vMotion. Most other virtual machine managers also support some form of live migration, though live migration of storage is less common.

Mobility in What State?

One of the key benefits of virtual machine technology is the ability to “run anywhere” on dissimilar hardware. From the very beginning, hypervisors have provided the ability to create a universal virtual machine image that would run on a variety of supported platforms.

This leads to one of the key values of server virtualization in the data center: Disaster recovery. The ability to take a virtual machine image and system state and bring it online after a disaster is a true revolution for open systems IT. The benefits of the single usage of server virtualization technology easily justify the investment to many businesses.

But this sort of “cold” migration seems passé when compared to the live or “hot” migration possible with technologies like VMware vMotion. Live migration is much more difficult, since active client sessions must be preserved in activity must not be greatly interrupted.

This is the second great question that must be asked when considering virtual machine mobility: In what state will the virtual machine be moved? Will it be a cold, powered down image of the system? A suspended or paused operating system image? Or a full, running machine?

Mobility to Where?

Once we have decided whether we are discussing virtual machine migration or movement of storage resources, we must consider the scope of the movement. The ability to move a virtual machine from one member of the cluster to another has now become fairly common. But what about systems that are not related in a cluster? Or that are spread over great distances?

• The nice thing about clusters is that they share resources before and after a virtual machine is moved. It is practical to move the running virtual machine, its storage, or both independently and to expect that performance will not dramatically suffer as a result. The cluster can also preserve network connections, and even I/O state, without much impact on clients or other external elements.
• It is a bit more difficult to move systems within a data center, since one must maintain the I/O connections that might be interrupted. It is fairly trivial to configure an IP network and storage array to allow multiple machines to access the same iSCSI or NFS storage resources. It is a little more difficult to configure Fibre Channel (and, by extension, FCoE) SAN’s to handle this sort of dynamic movement, but it is not impossible. Although moving a running machine from one network port to another could cause client access to be interrupted, technologies like VXLAN allow these sessions to continue, and improved network switching technology should reduce performance impact.
• Moving the machine to a different data center is another matter entirely. Stretching a layer-2 Ethernet LAN or Fibre Channel SAN across a metro or greater distance, while possible, will always be problematic. IP routing is flexible, but it takes time for changes to propagate when live machines are moved. And it is difficult to keep storage in sync over long distances due to the amount of time it takes for information to transit. Again, all of these challenges are being addressed in various ways, but they’re still hard!

Stephen’s Stance

Moving cold virtual machine images from system to system, or even across great distances, is one of the main selling points of server virtualization. But it becomes much more difficult to manage movement of virtual machines that are still running, especially outside cluster or across WAN links. When talking about virtual machine mobility, it is important to consider what is being moved, the state it is in, and where it is going.

Note: This discussion is part of “Building Virtual Infrastructure”, my new seminar series with Truth in IT.

© sfoskett for Stephen Foskett, Pack Rat, 2012. |
Virtual Machine Mobility: Of What, and to Where and in What State?


This post was categorized as Enterprise storage, Features, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2012. | Virtual Machine Mobility: Of What, and to Where and in What State?
Read more posts categorized as Storage

When designing VDI environments, a couple of things that you should consider are the .vswp file and the GuestOS paging file. In this article I am going to focus on the Paging file and hopefully in the not so distant future I will write a post about the .vswp file in a VDI environment.

What is point of the paging file (also known as the pagefile.sys)?

RAM is a limited resource. Virtual memory was introduced to help remove that limit.

Most modern operating system now use Virtual Memory. Virtual memory is a memory management technique. Applications running on a GuestOS reference memory using virtual memory addresses which are then automatically translated into RAM addresses by the hardware. These virtual memory address spaces are divided in pages or block, usually of 4KB.

If RAM resource is exhusted, the operating system will move 4KB pages of the virtual memory onto the computers hard disk to free up the physical memory (RAM) for other processes. In Windows operating systems, these pages are stored in the pagefile.sys.

A good way to think of this is;

Imagine a restaurant that has just open for the evening. When customers (Processes) arrive they get allocated a table (RAM) to sit and eat at. As the night goes on the restaurant get busier and free tables (RAM) begin to run out for the new customers (Processes) coming through the door. To free up spare tables (RAM) the waiter asks customers (Processes) who have finished eating if they wouldn’t mind moving to the bar (Virtual Memory) where they can continue drink.

Without the paging file, if the physical memory becomes full, applications including the operating system will have to waiting until physical memory becomes available before it can be stored in RAM ready for the CPU to process. As you can imagine this causes massive performance problems.

In summary, you NEED to have a paging file.
Read the rest of this entry »

© Simon for Gestalt IT, 2012. | VDI Paging Files – Big? Small? Or Non At All?
Read more posts categorized as Desktop, Server Virtualization

The great battle of enterprise storage is on!

The time has come to take sides on the core question of storage for virtual servers: Do you want storage intelligence to live in the hypervisor or the array? Most administrators are already lining up on one side or the other, unintentionally casting their vote while the rest flounder. But the storage industry must wake up and embrace the divide.

Hypervisor Huggers Unite!

Do you “heart” virtualization?

VMware’s vSphere dominates the world of enterprise server virtualization and has become the core element of the modern open systems datacenter. Microsoft recognizes this but has been unable to field a competitive hypervisor ecosystem for the virtual datacenter. Today, vSphere is the state of the art and nowhere is this more apparent than in storage.

In just a few years, VMware has delivered and updated a host of advanced storage functionality, from provisioning to migration and load balancing to backup and data protection. vSphere 5 includes an advanced and scalable storage virtualization layer, delivering everything a datacenter needs. VMFS sculpts basic block storage into a shared resource for virtual machines, with snapshots, policy-based layout and movement, and flexible allocation and thin provisioning.

Most VMware administrators are “server guys” and relish these features. They have never experienced an automated “storage service” like this, and the enterprise storage world has never been able to construct anything remotely as flexible, user-friendly, and functional. And Hypervisor Huggers don’t need complex enterprise storage arrays to do it: They can use basic iSCSI or Fibre Channel devices to provide performance and capacity and let VMware do the rest!

Storage DRS is exemplary of the new virtual datacenter world. Introduced in vSphere 5 (and restricted to the pricey Enterprise Plus license), Storage DRS uses the core technology of Storage vMotion to dynamically balance I/O and capacity across a diverse pool of storage. Storage DRS even uses Policy-Driven Storage and VASA to enforce tiered storage and data placement strategy. This kind of virtualization has been a “holy grail” quest for the enterprise storage industry, but they’ve never delivered on their promises.

Cheers for Storage Stalwarts!

Do you wish all this virtualization nonsense would just go away?

But not every IT environment wants be 100% vSphere focused, and many aren’t convinced that dumb storage is the smartest place for their data. These Storage Stalwarts want smarter and better-integrated storage arrays, and VMware is innovating here as well.

VMware’s Storage APIs for Array Integration (VAAI) technology is squarely aimed at this market. VAAI allows vSphere to hand off heavy storage operations to the high-end storage arrays from the major players. It works transparently, too, coordinating cloning without the kind of scripting and hair-pulling that used to require. VAAI in vSphere 4.1 also includes block zeroing support and something called “atomic test and set” which we’ll get to in a moment. Microsoft announced their own cloning integration, ODX, but it won’t ship until Windows Server 8 appears sometime next year.

But cloning is just the tip of the iceberg when it comes to Hypervisor-to-array integration. The rising army of NAS users have seen the glory of more-complete array integration for a while, and they’re not quiet about it. They love that VMware’s NFS protocol support makes storage “disappear” in vCenter, becoming just another resource with integrated thin provisioning and flexible allocation and movement.

VMware is moving aggressively to please their Storage Stalwarts, adding more VAAI support for block and file in vSphere 5. But, as the company laid out at VMworld 2011, neither access method is ideal for virtual servers. So VMware has been pushing the enterprise array vendors for ever-greater integration. They see a future where a VAAI-based protocol enables arrays to de-multiplex I/O streams from the hypervisor and intelligently handle per-VM data.

Stephen’s Stance

You can spot a Hypervisor Hugger by their big LUNs: They would rather treat storage as a bulk commodity, and array vendors should be lining up to get their business. Storage Stalwarts will jump on each new VMware innovation, finally making use of the capabilities they have spent over a decade paying for but not utilizing. The only untenable stance is trying to keep a foot in both worlds: It’s foolish to buy an enterprise array and use it as bulk storage!

© Stephen Foskett for Gestalt IT, 2011. | Are You a Hypervisor Hugger or a Storage Stalwart?
Read more posts categorized as Featured, Server Virtualization, Storage

All of us “Virtualization Admins” are always looking for more information about the performance of our environment. As VMware ESX products have progressed, more and more performance metrics have been gathered and presented to us… and we appreciate it oh so much.

However, one of the largest issues we need to contend with is tracking of this information longer term and being able to correlate the collected information to new issues and/or diagnose future capacity issues in the future.

Previously, we needed to invest in 3rd party company products to provide this functionality. In some cases, the information was difficult to gather correctly, provided very static alerting thresholds, and was not able to integrate well into existing virtual infrastructure.

Back on August 31, 2010, VMware announced the acquisition of Integrien… an up and coming product providing real time performance analytics for all kinds of environments… including virtual environments. Acquisitions are always so curious because we, the consumers, are always trying to figure out how it is going to be used to further the product line of the purchasing company. So, while the product from Integrien was interesting, seeing how it would fit into the vSphere realm as a VMware product was up in the air.

All of those questions are answered today, as VMware has announced the availability of the vCenter Operations products. vCenter Operations is being provided in three flavors:

• Standard
  • Handles vSphere environments.
  • Deployed as a virtual appliance that hooks into vCenter and is visible as a vSphere Client Plugin
  • Handles up to 500 VMs
• Advanced
  • Same as Standard edition
  • Includes additional Capacity Planning (aka – Standard Edition bundled with CapacityIQ)
• Enterprise
  • This is a whole new beast and includes the ability to monitor much more infrastructure than just virtual hosts and servers.

The product information from VMware will list off all sorts of neat features, functions, and purpose for their product. However, after being able to use the product during the pre-release period, I find that the following are the main value points for my environment:

1) 10,000’ view of Virtual Environment

All kinds of monitoring solutions exist that claim to be the single pane of glass that should be able to solve all of your problems. However, it appears as though this is the first to actually accomplish that.

Check out what I see:

Just from a glance, I can see that my environment is running well. The overview page uses the typical Green/Yellow/Red color scheme to indicate state. From this view, I can immediately see that my vCenter, Datacenters, Clusters, and ESX hosts are running within established parameters. I see there may be some problems with a handful of VMs. Instantly, I can see what is going on.

Without going into a complete demo, I can click on any object in the page and get relationship information (aka – which vCenter, which Datacenter, which Cluster, which ESX, and which VMs).

2) Defining what is normal for your environment

Determining what is normal is one of the most important aspects of what the vCenter Operations Standard product offers. What is normal to me is not normal for everyone else… we are all special in our own way and vCenter Operations Standard understands that.

When vCenter Operations Standard is installed and configured, you do not need to do much of anything. The installation guide is dead simple. Why is this?! Well, because vCenter Operations Standard learns about the behavior of your environment.

Rather than rely upon static definitions that would cause warning and critical alerts (example: Warn when RAM = 80%+ and Critical when RAM = 95%+), vCenter Operations watches what happens in your environment from day 1 and starts to determine what is normal for you. If you configure your applications to utilize almost all of the RAM assigned, the other solutions may alert you to a critical state. However, this is normal and expected. You would never expect to see an alert for when the server RAM utilization drops to 10%. However, by using dynamic thresholds in the learning and monitoring algorithms, vCenter Operations Standard is able to determine that high RAM usage is normal and alert you to when the situation is NOT normal… so, when the server drops to 10% RAM usage, you will get alerted because it is abnormal.

vCenter Operations relies on some crazy algorithms and analysis that any PhD in Rocket Science would love. Various algorithms exist in the environment that chew on the data as it is received. The results of the algorithms are selected based on most likely to be correct and used to represent the data in some fashion. So, there is a higher probability of the data and situation being statistically correct.

Check out what I see:

This is a view of a specific ESX host in my environment. You can see that normal is somewhere between 1-16 and is typically defined by the Memory usage. Additionally, you can see various statistics regarding the current workload, CPU, Memory, and ESX resources. Again, all from a single screen.

Now, compare that to a different ESX host that is hit a little harder:

Normal for this server is somewhere between 61-100 and is defined by the Memory usage.

3) Resource statistic aggregation to provide a more holistic view of what my environment is like on a historical basis.

This little nugget of joy made my day the first time I saw it.

vCenter Operations Standard is able to aggregate many statistics into a single value for you to see that represents your resource in the environment. So, rather than use resxtop to find NIC statistics in my environment, I can get those statistics from vCenter Operations Standard in a easy to read way.

For example: There have been many times where I wanted to get a good idea of how much data is passing through my NICs. Previously, I would need to get some batch data from resxtop, throw it into a spreadsheet, and process it. Or rely on some historical data from vCenter. Now, I can dig into the ESX host in vCenter Operations Standard and select the “ESX USED NETWORK INTERFACES” section at the bottom of the page.

Clicking on one of the number shows me even more information. This is the graph for Received Rate (KBps):

How cool is that?!

Similar statistics can be gathered for CPU, Memory, and Storage as well!

4) Analytics and Fires

Analytics are presented as hot zones and sized based on relationship to others. So, just looking at the graphic, you can get a sense of the relationship between other objects. I know, that is a little obscure. But, check this out:

From here, you can see that there is a fair amount of contention for the Exchange partition compared to that of the other datastores in the environment.

Having access to this data is still new to me and I am continually finding more and more ways to interpret it.

The vCenter Operations Standard product provides amazing insight into the your virtual environment. Whomever at VMware decided that Integrien was a suitable acquisition should get a high-five for this one.

No doubt, this is an insanely useful product and is definitely raising the bar in what is possible in the analytical world. I look forward to seeing this product grow and mature.

© Bill for Gestalt IT, 2011. | VMware vCenter Operations–Standard Edition
Read more posts categorized as Server Virtualization

The more and more I think about it, VMware is a farmer. No doubt they have created something very special. The ecosystem surrounding their foundation product, the hypervisor, is rich, fertile, and ready for harvesting.

Land = Hypervisor

There is no question that VMware has provided the most market-dominant virtualization layer out there. Continued development has yielded a smaller, more efficient, and more feature rich solution that we all see benefits from. Collaboration between VMware and core hardware vendors (network, storage, processing, and memory) have resulted in a product that provides functionality we all take for granted and rely upon heavily.

Seeds = APIs

Starting with v3.5, VMware really started a push for integrations with 3rd party vendors. This effort shot through the roof with the vSphere platform (v4.0). The theory being that if VMware opens up various APIs to the community, the community will come up with some wicked cool add-ons that provide functionality above and beyond what VMware could provide. So, rather than keep a closed and controlled environment, VMware enhanced their market presence by allowing other people to play with their toys.

This effort has turned small companies into virtualization powerhouses and larger companies areas to expand their expertise and existing technologies. Think about it:

• Veeam
• Xangati
• Vizioncore (now a part of Quest Software)
• Hyper9 (now a part of Solarwinds)
• TrendMicro
• Symantec
• Pano
• Thinstall
• SpringSource
• Zimbra
• Integrien
• TriCypher
• SlideRocket
• Shavlik

And the list goes on and on. The API availability has fostered so much new business opportunities for the companies on the list above and the countless others that are not listed.

Water/Care = Vibrant Community

VMware has fostered an incredibly active community surrounding their company.

• Active local VMware User Groups
• Social Media – The one and only, Mr. John Troyer (@jtroyer for you Twitter folk out there) is the ring leader behind this. He has become the face of VMware for so many of us out there. Just listen to the VMware Communities Roundtable podcast, sit at the social media area at VMworld, or follow his Twitter account and you will see what I mean.
• vExperts – Those individuals that go above and beyond to promote the VMware technnologies and brand.
• VMworld conferences

Ultimately, there is a great customer/vendor community that loves VMware and what they do.

How is VMware taking advantage of their farm

The obvious answer is that they are taking all of this to the bank. Sure enough, they are.

Their stock is doing well. Q1 results show great growth, especially when compared to Q1 of 2010:

• Revenues: $844 Million [+33% from Q1 2010]
• Net Income: $126 Million [ $78 million in Q1 2010]
• License Revenues: $245 million [+$34% from Q1 2010]
• Professional Services: $425 million [+32% from Q1 2010]
• [Source: http://virtualization.com/2011/05/06/vmware-reports-q1-2011-earnings-revenue-up-33-to-844-million-yoy/]

But, aside from the financials, VMware is able to harvest some amazing technologies and people from the farm. Remember the list of companies waaaayyyy up at the top of the blog post? VMware has identified some level of value and innovation in some of the companies and decided to purchase them and bring the technologies in-house. Of the entries in the list, the following were acquired by VMware and integrated into their product offering in one way or another:

• Thinstall  – Now known as ThinApp
• SpringSource
• Zimbra
• Integrien – vCenter Operations Standard/Advanced/Enterprise
• TriCypher
• SlideRocket
• Shavlik – VMware Go!

By bringing these companies in house, VMware is able to expand their offering AND force the other companies to innovate more. Acquisitions are a way to level-set the 3rd party community and provide a new avenue for growth.

Result

Take a hypervisor, sprinkle some APIs, apply and maintain a vibrant community, and you get what VMware is today…

References:

• virtualization.info – A look at VMware’s past acquisitions – Feb 1, 2010
• virtualization.com – VMware Reports Q1 2011 Earnings: Revenue Up 33% To $844 Million Yoy
• Google Finance – Google Finance – VMware Stock – 5yr Range

© Bill for Gestalt IT, 2011. | Who Knew VMware Would Be Such A Good Farmer?!
Read more posts categorized as All, Server Virtualization

After checking if I was able to ping the interface, I logged on to the machine to see if there were any obvious errors. The vCloud Director daemon was still running and so was the database, but a netstat did not show any listeners on the vCloud Director IP. So, after going over the vCloud Director log files, there was a pretty obvious error in the vcloud-container-info.log:

ORA-28001: the password has expired

So, you now stop your vCloud Director daemon and switch to your Oracle user to see what was going on inside of the DB:
sqlplus "/ as sysdba"

Now, list all the users to see if they have an expired password:
select username,ACCOUNT_STATUS,EXPIRY_DATE from dba_users;

Or display just the specific user:
select username,ACCOUNT_STATUS,EXPIRY_DATE from dba_users where username='VCLOUD';

And guess what came up:
USERNAME ACCOUNT_STATUS EXPIRY_DA
-------- -------------- ---------
VCLOUD EXPIRED 17-MAR-11

Expired is something that you don’t want to see for a user that is being used actively. So, let’s set the password again and unlock the user:
alter user VCLOUD identified by replace_this_with_your_password;
alter user VCLOUD account unlock;

So, once that is done, let’s check one more time:
SQL> select username,ACCOUNT_STATUS,EXPIRY_DATE from dba_users where username='VCLOUD';
USERNAME ACCOUNT_STATUS EXPIRY_DA
-------- -------------- ---------
VCLOUD OPEN 26-SEP-11

Now, start your vCloud Director daemon again, and in the log file you should no longer see the error, and the web interface should be working normally again.

© Bas for Gestalt IT, 2011. | Shorts: VMware vCloud Director not displaying the web portal
Read more posts categorized as Server Virtualization

When testing this, customers can often see unexpected results. This is often due to their misconception of how this function of View works. 

Why? View 4.5 looks for the datastore/s that have the highest "weighted available space". The formula that View 4.5 uses to calculate this is:

weighted_available_space = datastore_capacity * overcommit_factor – virtual_usage

datastore_capacity = As you would expect, the size of the datastore.

overcommit_factor = The Storage Overcommit setting applied to each datastore when selecting which datastores to use for your desktop pools.

• None - Storage is not overcommitted.
• Conservative - x4 the size of the datastore. (This is the default level.)
• Moderate - x7 the size of the datastore
• Aggressive - x15 the size of the datastore.

virtual_usage = The maximum possible usage of all existing VMs on this datastore, which is calculated by adding up current sizes of all disks and configured sizes of memories of all existing VMs on the datastore.

Examples

• DS1 - 1000GB (Datastore Size) * 4 (Conservative Overcommitment) – 0 (No VM's deployed) = 4000
• DS2 - 1000GB (Datastore Size) * 4 (Conservative Overcommitment) – ((20GB + 130MB)x5) (5 VM's already deployed) = 3865
• DS3 - 1000GB (Datastore Size) * 7 (Moderate Overcommitment) – ((20GB + 130MB)x5) (5 VM's already deployed) = 6865

Common Scenario

You have a desktop pool with DS1 as it's only datastore. You then decided to add a second datastore, DS2. You perform a Rebalance on the pool to spread the VM's across the two datastores, but nothing happens… Why? Because DS2's weighted_available_space is less than that of DS1's.

• If you increased the Overcommit level on DS2 to Moderate (x7) and ran Rebalance, the VM's would all migrate completely over to DS2 because it's weighted_available_space would be more than that of DS1 even though there are already 5 VM's on DS2. (See DS3 example)
• If you Storage vMotioned the 5 VM's off of DS2 and ran Rebalance, the VM's in the pool should evenly distribute themselves across both DS1 and DS2.

Summary

As you can see, by default you may not get your desired effect. But with a little bit of maths you can get the reaction you want from the Rebalance feature. I must stress, this is only for VMware View 4.5 and older. 

© Simon for Gestalt IT, 2011. | VMware View 4.5: Rebalance
Read more posts categorized as All, Desktop, Server Virtualization

The following was penned by Scott Boss and is a special feature for today

Sally Carmack, representative of a well-known California-based NAS storage company, today announced, “we have been working with VMware and the FCoTR Working Group and are pleased to announce that we have full support for VATRI in the next release of code for our storage platform.”  Scott Badger an engineer from the same company says that it has huge improvements over speed and stability over classic Fibre Channel access.

IBN, one of the oldest computer companies in the world, is announcing support both for FCoTR on the vOS mainframe as well as VMware 5 with VATRI on the mainframe platform.  William Farley from the mainframe engineering department claims, “we here at IBN have created a new engine for the mainframe that is optimized for vSphere 5 especially when using VATRI/FCoTR based storage. We haven’t been this happy since they ended work on FICON.”

Data Domani, a division of ETC, added, “we are implementing FCoTR and VATRI support into our line of storage arrays.  This will enhance the ability for vSphere 5 to do backups, restores and replication of the backups.  The FlexIO modules for FCoTR and the VATRI plugin will be released in May at our ETCWorld conference, with parts shipping in June of 2012.”  PR representatives at ETC had only one thing to say, “we don’t talk about things coming out in the future, but what you have asked us about is way cool technology. See us in May.”

Scott’s Stance

I have been working in the storage world over for ten years now and nothing has gotten me so eager to play with.  Once this technology (VATRI and FCoTR) is available, I am buying it for my home lab.  Who needs Fibre Channel when you have FCoTR?

© guest for Gestalt IT, 2011. | Storage Vendors Announce FCoTR and VATRI Support
Read more posts categorized as Server Virtualization, Storage

For a great overview of the entire U1 release check out Rick Vanover’s post vSphere 4.1 update 1 released from his Rickatron Blog and via his Servers and Storage Column/Blog at TechRepublic.

The rest of this post contains cut and pastes from the Release Notes and some commentary about them from me. I want to stress again that these are issues that have now been fixed!

Finally, I’ll point out the one huge VM backup issue (that I can think of right now) that still does not appear to be resolved.

• Cannot take quiesced snapshots of Microsoft Windows Server 2008 R2 virtual machine running vCenter Server 4.1 When creating a snapshot of a Microsoft Windows Server 2008 R2 virtual machine that has vCenter Server 4.1 installed, the snapshot operation might fail to complete. This issue occurs on Microsoft Windows Server 2008 R2 virtual machines when the ADAM database is installed. The issue is resolved in this release.

The change from 2008 to 2008 R2 has caused quite a few applications to stumble, so it is good to see that VMware users can now install vCenter on the latest and greatest OS from Microsoft running as a virtual machine without having to worry about various hassles, not to mention coming up with a separate backup or business continuity solution for arguably one of the most important VMs in the environment.

• Creation of large .vmdk files on NFS might fail. When you create a virtual disk (.vmdk file) with a large size, for example, more than 1TB, on NFS storage, the creation process might fail with an error: A general system error occurred: Failed to create disk: Error creating disk. This issue occurs when the NFS client does not wait for sufficient time for the NFS storage array to initialize the virtual disk after the RPC parameter of the NFS client times out. By default the timeout value is 10 seconds. This fix provides the configuration option to tune the RPC timeout parameter using the esxcfg-advcfg -s <Timeout> /NFS/SetAttrRPCTimeout command.

Although specific to users of NFS storage, this one was definitely a head scratcher. Possible scenarios where this problem could rear it’s ugly head include full VM restores and even VM replication. Add the possibility of dedicating a large .vmdk as the backup repository for your backup server VM.

Let me know if I missed any other resolved issues specific to VM backups.

The issue still unresolved

Unfortunately, nothing in vSphere 4.1 U1 appears to address the loss of connectivity when committing snapshots for VMs using Change Block Tracking (CBT) while running on NFS storage.  This leaves VMware admins stuck between a rock and hard place – disable CBT or move VMs off of NFS. More about the problem can be found in VMware’s KB Article:

Virtual machine freezes temporarily during snapshot removal on an NFS datastore in a ESX/ESXi 4.1 host

© Rich for Gestalt IT, 2011. | vSphere 4.1 U1 Released. Fixes Specific For VM Backups
Read more posts categorized as All, Server Virtualization

From the information below, I would personally recommend using the following:

• Windows XP: LSI Logic Parallel or SAS
• Windows 7: LSI Logic SAS

The PVSCSI controller will be the controller of the future so you may also want to keep this in mind. Of course you can make your own decisions. Most of the points made are desktop focused, however most will apply outside of a View environment.

IDE vs SCSI

Most people’s main argument around reasons to not use IDE are;

• The IDE adapter completes one command at a time – SCSI can queue commands.
• The SCSI adapter is better optimized for parallel performance.
• If you ever need to increase the size of an IDE Disk you would have to use VMware Converter to enlarge it. – Using SCSI it’s easier
• Maximum of 4 IDE Devices per VM. (This includes CDROM) – SCSI allows 60.

BusLogic vs LSI Logic

If you have decided to use SCSI, you now have to decide on whether to use the BusLogic or LSI Logic SCSI controller.

• BusLogic is not supported in a lot of the newer Operating Systems.
• BusLogic is not actively being improved by VMware or Guest OS’s (so bugs come along as things change in the OS and noone fixes them).
• XP doesn’t contain the LSI Logic driver so you will have to load it into the OS manually.
• “Out of the box” LSI out performs BusLogic. With a few additional configuration changes to the Queue Depth, BusLogic can perform on a par with LSI in VM’s that have less than 4GB of RAM – BusLogic is a 32 bit PCI device and is limited to DMA (Direct Memory Access) in the first 4GB of the guest memory, so if the guest uses more than 4GB memory the OS will need to use bounce buffering (doing IO to lower memory and copy up/down with the CPU), which adds overhead

*VMware’s recommendation is to use the LSI Logic adapter. “The LSI Logic adapter has improved performance and works better with generic SCSI devices.”

LSI Logic SAS

The default controller for a Windows 7 virtual machine is the LSI Logic SAS. There is no performance between this controller over the regular LSI Logic Parallel controller, so I therefore see no reason to change it.

It is worth noting, in Linux guests SCSI disk hotplug works better with SAS than with original parallel LSI. (You do not have to rescan the SCSI bus yourself to find a new device; on Windows you do not have to do rescan ever with ESX4.x, neither with LSI nor LSI-SAS)

Why not PVSCSI?

Virtual Desktops don’t tend to have a really high I/O. Using the PVSCSI driver with low throughput workloads may introduce a small increase in latency.

Pre ESX 4.1 the PVSCSI adapter had only been recommended for VMDK’s backed by fast (2,000+ IOPS) storage. Why? The PVSCSI driver only coalesces OIOs (Outstanding IOs) and not throughput (IOPS). What this means is; when the VM is requesting a lot of IO and the storage cannot deliver it, the PVSCSI driver is coalescing interrupts. Without the steady steam of IOs from the storage, there are no interrupts to coalesce. This can cause extra latency on low throughput environments.

The LSI Logic driver increases coalesce as OIO and IOPS increase. So if there is a low OIO and IOPS load no coalescing will occur, reducing latency.

Typically the case was:

• PVSCSI is best for workloads that drive more than 2000 IOPS and 8 outstanding I/Os.
• LSI Logic is best for workloads that drive lower I/O rates and fewer outstanding I/Os.

However! In ESX 4.1 this issue explained above has been fixed. This means that the PVSCSI could perform as well as the LSI Logic driver with low throughput. I have yet to see any official performance test results. If you have already used PVSCSI adapters and are not seeing any performance problems, I wouldn’t worry too much about making the change to LSI.

Summary

As you can see, there is a lot of Pro’s and Con’s to digest. I chose the SCSI LSI Logic controller for the following reasons:

• SCSI offers better performance than IDE
• LSI Logic is continually supported with newer OS’s and VMware releases, where BusLogic is being fazed out. So to help future proof your design, it would be better to use LSI.
• LSI offers better “Out of the box” performance over BusLogic
• LSI SAS is the default for Windows 7, great for Hot-Add disks and a must for MSCS.
• PVSCSI with ESX 4.1 is rumored to have equal performance with LSI Logic, but I have yet to see the test results.

What do you use on your site and why? Have I missed anything? let me know. I am interested to hear other peoples take on this, so please comment if you feel you have anything to add.

Sources

• http://www.vmware.com/pdf/ESX2_Storage_Performance.pdf
• http://communities.vmware.com/docs/DOC-10971.pdf
• http://www.vmware.com/pdf/vsphere4/r41/vsp_41_config_max.pdf
• http://www.vmware.com/pdf/vsp_4_san_design_deploy.pdf
• http://www.vmware.com/pdf/GuestOS_guide.pdf
• http://www.vmware.com/pdf/vsp_4_pvscsi_perf.pdf
• PVSCI and low IO workloads
• SCSI vs IDE
• http://myvirtualcloud.net/

© Simon for Gestalt IT, 2011. | VMware View Desktops: IDE or SCSI? BusLogic, LSI Logic or PVSCSI?
Read more posts categorized as All, Server Virtualization, Storage