For a great overview of the entire U1 release check out Rick Vanover’s post vSphere 4.1 update 1 released from his Rickatron Blog and via his Servers and Storage Column/Blog at TechRepublic.

The rest of this post contains cut and pastes from the Release Notes and some commentary about them from me. I want to stress again that these are issues that have now been fixed!

Finally, I’ll point out the one huge VM backup issue (that I can think of right now) that still does not appear to be resolved.

• Cannot take quiesced snapshots of Microsoft Windows Server 2008 R2 virtual machine running vCenter Server 4.1 When creating a snapshot of a Microsoft Windows Server 2008 R2 virtual machine that has vCenter Server 4.1 installed, the snapshot operation might fail to complete. This issue occurs on Microsoft Windows Server 2008 R2 virtual machines when the ADAM database is installed. The issue is resolved in this release.

The change from 2008 to 2008 R2 has caused quite a few applications to stumble, so it is good to see that VMware users can now install vCenter on the latest and greatest OS from Microsoft running as a virtual machine without having to worry about various hassles, not to mention coming up with a separate backup or business continuity solution for arguably one of the most important VMs in the environment.

• Creation of large .vmdk files on NFS might fail. When you create a virtual disk (.vmdk file) with a large size, for example, more than 1TB, on NFS storage, the creation process might fail with an error: A general system error occurred: Failed to create disk: Error creating disk. This issue occurs when the NFS client does not wait for sufficient time for the NFS storage array to initialize the virtual disk after the RPC parameter of the NFS client times out. By default the timeout value is 10 seconds. This fix provides the configuration option to tune the RPC timeout parameter using the esxcfg-advcfg -s <Timeout> /NFS/SetAttrRPCTimeout command.

Although specific to users of NFS storage, this one was definitely a head scratcher. Possible scenarios where this problem could rear it’s ugly head include full VM restores and even VM replication. Add the possibility of dedicating a large .vmdk as the backup repository for your backup server VM.

Let me know if I missed any other resolved issues specific to VM backups.

The issue still unresolved

Unfortunately, nothing in vSphere 4.1 U1 appears to address the loss of connectivity when committing snapshots for VMs using Change Block Tracking (CBT) while running on NFS storage.  This leaves VMware admins stuck between a rock and hard place – disable CBT or move VMs off of NFS. More about the problem can be found in VMware’s KB Article:

Virtual machine freezes temporarily during snapshot removal on an NFS datastore in a ESX/ESXi 4.1 host

© Rich for Gestalt IT, 2011. | vSphere 4.1 U1 Released. Fixes Specific For VM Backups
Read more posts categorized as All, Server Virtualization

Update: @SSauer points out that this is vShield, and posted some information here: A New Generation of vShield Security Products

vSwitch, vNetwork, vShield?

Most hypervisor products include an internal virtual network switch, but VMware’s ESX has multiple choices. The original “dumb” virtual Ethernet switch was augmented by vSwitch back in the ESX 3 days, bringing more-advanced configuration options.VMware improved and renamed the vSwitch in vSphere 4, creating the vNetwork Standard Switch (vSS).

But it was the introduction of vNetwork Distributed Switch (vDS) in vSphere 4 that really set VMware’s network capabilities apart. The champion of this field is Cisco, whose Nexus 1000V virtual switch extends their NX-OS datacenter networking OS right into the ESX world.

The Cisco Nexus 1000V runs both the supervisor and Ethernet modules inside the virtual ESX environment

As illustrated above, the Nexus 1000v consists of two key components:

1. The Virtual Supervisor Module is an implementation of NX-OS on an ESX virtual machine, and provides the interface and configuration of the virtual network
2. The Virtual Ethernet Module runs at a lower level in ESX, replacing the vSwitch for networking between VMs

One can think of the Cisco Nexus 1000v as a specialized replacement for the more-generic vNetwork Distributed Switch. Both include plug-in vSwitch replacements and centralized management, and both implement more-advanced network protocols like private VLANS and receive-rate limiting as well as supporting vMotion. But Cisco’s 1000v goes much further, adding PortChannel, LACP, security and QoS, and advanced management features.

See this comparison of vSwitch, vSS, vDS, and 1000v as well as Rich Brambley’s vSS/vDS post

From a technology standpoint, the key to both vDS and 1000v is the ability to replace the core ESX vSwitch with a more-capable alternative. Now let’s turn to what VMware might be introducing next.

What Do We Know?

Howie Xu, VMware Director of R&D, released a video discussing his sessions at VMworld, entitled ”The Future Direction of Networking Virtualization” (TA8361). This video begins with a quick pan past Xu’s whiteboard (pictured below) and includes a discussion of the state of the art, vision, and product and technology roadmap for VMware’s networking-related efforts.

The part that really piqued my interest was later in the video, when Xu talks about creating a “networking virtual chassis or hypervisor” to allow third-parties to develop and roll-out advanced networking devices within vSphere. VMware has already steamrolled through the heart of server-based applications, making VMware-based virtual appliances as common an installation format as the DVD. Now the company is turning its attention to the network. This is huge!

Xu speaks of both a platform and a service to support this ”open extensible networking virtual chassis platform,” and goes on to suggest that it could be used by “networking security, load balance, application acceleration, IP address management, and performance management” products. The virtual appliance marketplace is already populated by the familiar names in networking, from F5 to Bluecoat to Checkpoint. Therefore, VMware must be talking about something much deeper and more advanced than merely encouraging the creation of more virtual appliances!

The core question is whether VMware is opening up the “green box” in my diagram above to run third-party applications and what level of system access they will get.

Consider this a vFabric pre-intro!

Then there is the name. The whiteboard prominently includes the words, “vFabric Intro” in the corner. Judging by the rest of the readable content, this indicates that this new technology will indeed be called “vFabric” as Stu speculated.

Update: It appears that vFabric is not the name of this virtual chassis (thanks, Stu and Howie!). Good thing, too, since the name, “vFabric” is a registered trademark of QLogic Corporation, for “computer software for managing computer hardware, namely switches used in networks.”

What’s Coming?

I’m more of a storage guy, so I rang up my friend Greg “Etherealmind” Ferro and ran some ideas past him. Greg and I talked about the needs of network-based devices, and how they differ from traditional server-based applications.

Greg suggests that vFabric will really assist vMotion

Networkers have been conditioned with the belief that custom silicon is the best way to achieve low latency and high performance for network devices. The same could be said of the storage world, where companies like HDS, 3PAR, and BlueArc pride themselves on their custom ASICs. But EMC, HP, and others are proving that Intel’s server-class CPUs and peripheral busses now have the guts to go head-to-head with custom silicon. The networking world is no different, with many newer companies basing their products around industry-standard hardware.

But deploying these systems in a virtual environment is more challenging. Can a virtual machine hypervisor prioritize threads for network devices? Can it handle the overhead related to networking operations in real-time? What happens in the event of a DDoS or network flood? Most network devices run real-time operating systems like VxWorks or QNX to ensure packet throughput, but virtual environments are notorious for “overflow” of I/O or CPU load between guest machines.

The whiteboard provides some hints as to how VMware will tackle these issues. First, we spot the term, “latency-aware queueing,” which suggests that a mechanism will monitor the hypervisor and alter the queues for virtual network devices as the load changes. As latency rises, the hypervisor can move workloads to different processor cores or even alternate hardware using vMotion. We also spot a reference to “non-blocking”, suggesting an asynchronous I/O mechanism will reduce the likelihood that one of these virtual network devices will have to wait for data.

Both of these technologies are hallmarks of real-time operating systems (RTOS), and are critical to the design of scalable hypervisors like VMware’s ESX. It is likely that the company is developing an advanced hypervisor environment for these specialized devices, an evolution of vDS and the API that allows the Cisco Nexus 1000v to run its Virtual Ethernet Module.

Stephen’s Stance

If our assumptions are true, then this is an exciting development indeed. If VMware exposes the “green box” in our diagram above to third-party developers, we could see an entirely new and more-powerful ecosystem evolve around VMware vSphere. Running virtual network devices in a quasi-real-time environment will enable even-greater integration and flexibility.

The Nexus 1000v has not eliminated purchasing of Cisco hardware, and vFabric will not destroy the larger network device market. But we expect wide vendor support for the concept, especially those involved in lower-end and remote-office applications. We would love to see Palo Alto Networks, Infoblox, SolarWinds, and Vyatta, to name a few, developing next-generation applications for vFabric. Virtualization-aware integrated networking shouldn’t be the sole domain of Cisco.

© Stephen Foskett for Gestalt IT, 2010. | A VMware Hypervisor For Networkers?
Read more posts categorized as Networking, Server Virtualization

Static Configurations

Think back to the time before server virtualisation (yes, there was one).  Physically static servers failed over to other physically static servers located in remote data centres.  Once deployed, servers very rarely moved unless there were major physical data centre issues or an upgrade was being performed.  In fact, even when server upgrades occurred, it was typical to acquire a new server and rebuild the application and data on that new hardware to remove any issues with new server drivers, hardware firmware and so on.

Mobility Rules

Server Virtualisation changed all those restrictions.  By abstracting the hardware to generic devices it was possible to place a VMware host on shared storage and have any connected VMware server run that guest.  Very quickly the toolset of features improved to make that host movement transparent and as simple as clicking a button.  This meant servers could be accommodated on any hardware and scaled within the hypervisor.

Storage Restrictions

Unfortunately storage wasn’t so quick to keep up.  The static model of a single physical server in two locations worked well with storage replication technologies that required only one copy of the data to be active at any one time.  Moving an application to another data centre was typically a disaster recovery process and consequently a small outage was acceptable as the storage arrays “failed over” their LUNs to the remote location.  Once the DR issue was solved, the data could be “failed back’ to the original location.  It wasn’t usual to move servers between data centres as a standard operational process.

Virtual Machine environments weren’t well catered for.  Failover of replicated storage wasn’t a transparent process; there was a tradeoff between LUN size and the maximum number of LUNs a hypervisor could support; this made it more complex to architect a Virtual Machine environment with enterprise storage.  Even with vMotion on VMware, where a VM host could be moved transparently across physical hypervisor servers, the storage couldn’t be moved easily.  In fact, the storage restrictions were solved by implementing storage vMotion, rather than have the array achieve the data migration itself.

Step in Federation

This is where storage federation comes in.  It enables any and all copies of a LUN to be updated from multiple locations at the same time.  This means that both the hypervisor and the storage can be load balanced across multiple locations and physical hardware without having to bulk copy the data all the time.  Here’s a simple example to demonstrate the process.

Imagine a 1TB LUN on a storage array acting as a single datastore and supporting 20 production virtual machines.  In the “old” model, moving that LUN to another location would require impacting all 20 virtual machines and making another location the primary target of I/O operations.  There was no ability to go “sub-LUN” and to move individual machines to another location.  Storage vMotion could be used, but that would require replicating the entire VM to another LUN/datastore.  Any attempt to load balance the VM guests would be constrained by the time required to continually move data around the infrastructure.  In addition, moving a replicated VM from one LUN/datastore to another would mean compromsing DR until that LUN had been fully replicated to another location.

Now under a federated environment, that 1TB LUN would be updatable from any location, meaning individual VM hosts on the datastore could be updated from multiple locations at the same time.  This means there is a massive increase in the flexibility of managing workloads across physical locations, offering the ability to workload balance for business and operational benefits.

Summary

Ultimately, federated storage environments will be the future. Products like VPLEX are only the start.  The ultimate goal will mean workloads can be run anywhere, any time.  That will be cool.

© Chris for Gestalt IT, 2010. | Why Storage Federation Is What We Need
Read more posts categorized as Featured, Storage

So, this post is about taking a deeper look at where the master template VM resides versus where the cloned template is destined. My math my be a little off or may not account for every factor involved, but my point is to be close enough to demonstrate that the disk/array/LUN design is can be the culprit more times than not.

When I started this post I emailed for some help. I asked for a sanity check from some storage experts. I’ve been reasonably happy with my own answer until now, but I figured I do some research before adding the content to VM /ETC. I got back a single reply that I am paraphrasing: “Sounds about right. Let me think about it some more and if I can stump you with anything else I’ll let you know.” He never did so I’ll take that as a positive confirmation meaning “yes VM moron, it is that simple.” Good enough for me! If anyone can point out any other factors I am not properly accounting for please leave a comment.

The following is part of my email for help. It not only explains my test scenario but it illustrates the problem and resolution as well. At the end of this post I make some suggestions for bettering the time it takes to clone a VM.

The email for help

Oh wise and all powerful masters of the disk,

I humbly submit the following concept for your review. Guide me to a greater disk performance understanding when cloning a VM in VMware ESX environment.

Here’s the scenario:

• Cloning a VM takes a long time – 10 GB VM using only 3.5 GB of space takes roughly 45 min to an hour to clone.
• The master template and the clone reside on the same disk and NFS mount.
• Yeah, it’s a single SATA disk in a lab. I know, it should suck.

I’m trying to explain the expected speed of read and writes using the IOPs calculator here: http://www.wmarow.com/strcalc/

See the attached screen shot for the values I put in the calculator, but the results I’m interested in are:

• with 50% reads and 50% writes (master and clone on same disk) average throughput (MB/s) is 1.2
• I used 50% reads and 50% writes for the cache.

which means to me that

• 3548 MB / 1.2 (MB/s) = 2957 secs or 0.82 hrs.

Here’s the screen shot of the IOPS Calculator I linked in the email for help:

Suggestions for improvement

Obviously, the type/performance of the disks, the number of disks, and the type of array makes a huge difference. I should also point out that I am using 8 ms as the value for the seek latency. I’m not as focused on technical accuracy because my point is served without it, but changing this value makes a significant difference as well. If you want technical accuracy and more explanation about some of the numbers to use in the calculator check out these posts on the topic of IOPS and the impact on a virtual environment:

• http://www.yellow-bricks.com/2009/12/23/iops/
• http://vmtoday.com/2009/12/storage-basics-part-ii-iops/
• http://virtualgeek.typepad.com/virtual_geek/2010/02/solving-a-weird-slow-performance-cloning-issue.html
• http://vpivot.com/2009/09/18/storage-is-the-problem/

In my case, moving the VM template to another disk/array or increasing the number of disks used on my NFS server would help because the reads and writes would be separated when the cloned VM resides on a different disk/array and the number of IOPs possible would be increased with more disks. Yes, this post uses a single SATA disk as a simple example, but the point is hopefully clear. Use the same logic and math for shared storage scenarios, all storage protocols, any vendor’s storage device, and all RAID types. Plug those values in the IOPs calculator to calculate your own results.

My ultimate point is to make everyone think about how the disk/array/LUN design decisions impact the behaviors of the virtual infrastructure.

As an example, if my lab NFS server was using 6 SATA disks configured as a RAID 5 array the calculation for expected time to clone changes as follows:

• 3548 MB / 2.99 (MB/s) = 1187 secs or 0.33 hrs.

Better, right? Hey, it’s a basement lab. It’s supposed to suck!

© Rich for Gestalt IT, 2010. | Why Does Cloning A VM From Template Take A Long Time?
Read more posts categorized as Featured, Server Virtualization, Storage

“The integration will enable VMware vSphere users to manage privileged user access with Microsoft Active Directory, providing large enterprises with a scalable means to improve authentication and access control in virtualized environments to help meet IT security audit requirements. Likewise is a member of the VMware Technology Alliance Partner (TAP) program.”

I recently ran across this announcement via virtualization.info’s article VMware to embed Likewise authentication in next vSphere, but originally read about it in the vreference.com post ESX 4.1 to include likewise AD authentication?

Likewise Open is an “open source application that joins Linux, Unix, and Mac machines to Microsoft Active Directory and securely authenticates users with their domain credentials.” The Open edition is free to download and use. Likewise also offers an Enterprise version with premium features such as group policy management, auditing and compliance, directory migration, and a Operations Dashboard. Several other Enterprise features make administrators able to manage Active Directory users and computers from Unix, Linux, or Apple systems.

I also noticed on the Likewise Features page that the company advertises it’s services as “pre-bundled with various industry-leading server virtualization solution offerings from Citrix and VMware…”

My Thoughts

Outside of the added administrative convenience, it’s an interesting partnership. Will VMware and Likewise develop a stand alone directory services offering for the cloud? This could mean VMware now has the Email, file sharing, and directory services pieces all accounted for for a self sustaining virtual business infrastructure. Likewise also already provides a CIFS services product for sharing files too.

Likewise already allows Active Directory management via a GUI from non windows systems. Surely new versions of vCenter and the vSphere client will contain new tabs for this future configuration.

The Likewise announcement seems to be another of the various clues starting to surface about the next version of VMware vSphere.

© Rich for Gestalt IT, 2010. | Likewise Agreement Means Active Directory Integration In Future vSphere Versions
Read more posts categorized as Security, Server Virtualization

There are a few possible problems to consider. In this post I’ll first cover (with the help of a several others) general VM and vCenter vDS networking issues, but along the way I’ll explore thoughts about designing around a vDS for keeping vCenter as a VM.

The vDS Rock and A Hard Place

I’m not the first to recognize there are problems with using a vDS if vCenter goes down. Here’s a few posts that have already addressed the pitfalls to avoid.

VMware vSphere – VMware vNetwork Distributed Switch bug or limitation

“… if you lose virtual Center you will have no way in moving virtual machines between different port groups on the vNetwork Distribute Switch. In addition, you will not be able to get a virtual machine from the traditional virtual switch to a port group on the vNetwork Distributed Switch. Extra to that, you can’t move a VM to another VMware vNetwork Distribute Switch. So that means if you are using VMware vSphere vNetwork Distributed Switches & you lose virtual center you are almost disabled on the networking part. If you lose connectivity on the classic virtual switch & your adapter on the distributed switch are OK you still can’t move your virtual machines to that distributed switch till Virtual Center is back.”

See the entire blog post for screen shot examples of vDS and vSS portgroups available to VMs with and without vCenter.

Virtualizing vCenter With vDS Catch-22 is another post that explores what happened after taking down the vCenter VM for a routine migration.

“vCenter was shut down and unavailable, therefore, I had connected my vSphere client directly to the ESX4 host in which I transferred the VM to. When trying to configure the vCenter VM to use the vNetwork Distributed Switch (vDS) port group I had set up for all VM traffic, it was unavailable in the dropdown list of networks. The vCenter server was powered down and thus the vDS Control Plane was unavailable, eliminating my view of vDS networks.

This is a dilemma. Without a network connection, the vCenter server will not be able to communicate with the back end SQL database on a different box running SQL. This will cause the vCenter server services to not start and thus I’ll never have visibility to the vDS.”

Virtualizing vCenter with vDS: Another Catch-22 is from another blogger exploring the same problem that was inspired by the last post.

VMware has a KB Article explaining Configuring vSwitch or vNetwork Distributed Switch uplinks from the command line in ESX 4 which helps manually migrate dvPortgroups back to a vSS from the Service Console when in trouble. Interestingly enough, I couldn’t find a similar article for ESXi. I’ll assume the same process is available via “unsupported mode” in ESXi, but the potential for having to perform these actions under fire must be considered.

Use a Hybrid Mix of vSS and VDS

Does this mean a virtual infrastructure design should keep a vSS around? I would say “yes!”. Perhaps it’s now more important to dedicate 2 of the ESX host’s pNICs for the ESX Service Console / ESXi Management VMKernel isolated as a vSS. The 2 pNICs are not only for redundancy anymore, but also to support one or more standby VM portgroups in case they’re needed as a recovery network for VMs normally using the vDS. Of course, that means creating the appropriate trunking, and VLANs ahead of time. Have everything ready for a quick and easy change of critical VMs when needed.

Therefore, a hybrid design using both a vSS and a vDS is a smart “safety net” to have. Especially when an admin has to point the vSphere client directly at an ESX/ESXi host. The “safety net” vSS portgroups will be available from each host and the VMs can be easily switched via the vSphere Client GUI.

BUT, Does VMware Support vCenter As A VM Using A vDS?

Even though VMware now fully supports running vCenter virtualized, the question is not whether to run vCenter as a VM, but instead it’s whether VMware even supports a vCenter VM using a vDS!

VMware Communities: Virtual vCenter and vNetwork ….

“I called support about running vCenter within a distributed switch and they said point blank, “it is not supported”. They said because vCenter governs the distributed switch environment, you can’t have vCenter within the distributed switch.”

OK, so VMware support has not always told customers the correct support policy based on actual technical capabilities in the past, but it’s something serious to consider. In fact, based on what has been already explained it makes sense they wouldn’t support it. Besides, it’s more like “putting all your eggs in one basket” then ever before when a vCenter VM is placed on vDS.

I’ve brought up the logistical argument about the brains of the virtual infrastructure running in the environment it is managing before. Don’t misunderstand – I am an advocate for and I virtualize vCenter Server all the time, but I make sure to adhere to the best practices. On that note, the VMware KB Article Running VirtualCenter in a Virtual Machine (updated as of Aug 09) serves as a pointer to the VMware tech note http://www.vmware.com/vmtn/resources/798 which then points to the old VI3 tech note on this topic: http://www.vmware.com/pdf/vi3_vc_in_vm.pdf. VMware definitely needs to update the tech notes to include best practice for vCenter 4 as a VM in an environment containing a vDS!

After all of that why do I want to use a vDS again?

Finally, here is a quick reminder of why the extra design considerations are worth the trouble.

Comparing the vDS to a vSS:

vNetwork Distributed Switch on ESX 4.x – Concepts Overview

Comparing vNetwork Standard Switch with vNetwork Distributed Switch

The following features are available on both types of virtual switches:

• Can forward L2 frames
• Can segment traffic into VLANs
• Can use and understand 802.1q VLAN encapsulation
• Can have more than one uplink (NIC Teaming)
• Can have traffic shaping for the outbound (TX) traffic

The following features are available on Only Distributed Switch

• Can shape inbound (RX) traffic
• Has a central unified management interface through vCenter
• Supports Private VLANs (PVLANs)
• provides potential customisation of Data and Control Planes

© Rich for Gestalt IT, 2010. | Design Challenges Of Virtualized vCenter With A vNetwork Distributed Switch
Read more posts categorized as Networking, Server Virtualization

I've had a look around and I can find two KB articles from VMware: Here, and Here which talk about using Hot-Add on Linux OS's. Another VMware document focused on Microsoft: VMware Guest/Host OS Compatibility Guide There is Jason Boche's great article which he researched and tested himself, And Daniel Eason's summary on Hot-Add.

From these different articles I have taken all of the information and produced the following table which in theory should be all of the OS's which support VMware Hot-Add Memory/CPU.

Do you know different? If you are aware of other OS's that can make use of these features, please leave a comment below.

© Simon for Gestalt IT, 2009. | VMware Hot-Add Memory/CPU Support
Read more posts categorized as Server Virtualization

My new DroboPro arrived this week and so far I’m less than impressed.  As you know, I already have had a generation 1 Drobo for some time.  It has been a great device, doing exactly what I wanted.  After winning a second standard Drobo at Tech Field Day, I paid for the upgrade to the DroboPro in anticipation of a bigger and better machine.  Here’s what I’ve managed to achieve and determine so far.

The device itself is very different.  First of all, the DroboPro accomodates up to 8 drives and so is orientated horizontally (otherwise it would fall over).  At the back, the cabling supports Ethernet, USB and Firewire.  Unfortunately the plugs are recessed and consequently removing the Ethernet cable is a pain as you can’t easily squeeze the retaining lug that releases the cable.  The power supply is now integrated into the unit and there’s no separate PSU brick.  This could present a problem if the power supply fails, as it isn’t obvious how (or even if) the integrated power supply could be replaced.  I expect that power is now integrated to make the DroboPro more datacentre friendly.  One last comment; the device is considerably more more noisy than its smaller brother, presumably because it has to cool twice as many hard drives.

• USB and iSCSI Configuration – it isn’t clear whether the Drobo Dashboard can configure the DroboPro over either the USB connection or the iSCSI connection or whether USB connectivity needs to be retained for future configuration tasks.  Be aware – unplugging the USB cable seems to cause the Drobo to immediately reboot.
• Reduced Management – if the DroboPro is to be connected to an ESX server, then the documentation indicates the USB cable should be removed.  What’s then not clear is how the  ongoing status of the device can be managed.  For example, at this point, where should the Drobo Dashboard be run from?  The lack of a permanent management console means downtime to check the status of the device, to see the current occupancy levels and the effect of adding in additional capacity.
• Lack of detailed Information – As yet, I haven’t managed to work out where or whether there are details provided on the utilisation of each configured LUN.  The DroboPro allows for up to 16x 16TB LUNs, which is a huge volume of data (although ESX only allows iSCSI LUNs up to 2TB).  I’d like to be able to work out which of those 16 volumes consumes the most physical compared to logical space, especially with ESX where the Drobo’s intelligent file system aware technology currently doesn’t apply.
• Multi-Drobo Management.  I have more than one Drobo device now (DroboPro, DroboShare & Drobo).  The dashboard doesn’t appear to cater for multiple devices managed from the same machine.

© Chris for Gestalt IT, 2009. | Review: DroboPro – Part I
Read more posts categorized as Server Virtualization, Storage

Introducing EvoStor

Coming out of stealth at VMworld 2009 in San Francisco, startup EvoStor exemplifies this new virtualization-optimized storage concept. Taking existing storage concepts like scale-out grid and automated storage layout, EvoStor’s offering is designed exclusively to support the VMware vSphere platform. This level of focus is unprecedented in the industry, and demonstrates the incipient power of storage virtualization in general and VMware in particular.

So what exactly is EvoStor? Although their coming-out party at the Museum of Modern Art was well-attended and included an appearance by VMware CEO, Paul Maritz, many were left wanting details. The company set up a demo lab next door at the W hotel, and I was able to examine their product in more detail.

EvoStor is a scale-out clustering storage product running on commodity-based (though custom-built) storage server hardware. The company eschews many long-standing storage traditions, including RAID, and instead relies on intelligent data placement for data protection, performance, and availability. Each node contains a complete x86 server and has 12 slots full of SATA, SAS, or flash drives. The minimum configuration is three nodes, which the company promises will sell for under US$50,000.

What Do VMware Admins Need?

The product probably doesn’t sound all that novel at this point, but the innovation of the EvoStor platform lies in its integration with VMware. Conventional storage arrays use iSCSI or Fibre Channel connectivity to present LUNs (virtual disk drives) and rely on VMware’s VMFS and storage APIs for feature integration. Although the storage appears to be a conventional iSCSI LUN, EvoStor uses a proprietary back-end protocol over Ethernet. They couple their back-end storage intelligence with the VMware hypervisor, linking data placement with the demands of virtual machines.

See those iSCSI LUNs? They're not really LUNs, and they don't really use iSCSI!

Although the product is still in development, this concept holds great promise. The company could enable per-virtual machine performance or availability SLAs, backing these up with automated block-level storage tuning. A high-priority VM could be provisioned on flash with extra redundant copies, while a test machine could live on bulk SATA. Both would be thin-provisioned and could later be migrated on a granular basis to alternate tiers of storage as requirements changed.

Like Drobo, EvoStor is designed to automate powerful storage features and bring them to non-storage audiences. Interaction with the system is through vCenter and, although EvoStor-specific tabs are added, very little configuration and tuning is required. Storage is allocated and data placed automatically as virtual machines are added and the system promises to keep everything running without end-user input. Also like Drobo (and EqualLogic, LeftHand, and many others), EvoStor can be expanded simply by adding additional storage nodes, with each integrated into the existing environment.

EvoStor administration is dead-simple, integrated into vCenter

Achieving this kind of lights-out automation with conventional SAN storage is challenging. As VMware’s own Jon Bock told me at VMworld, the vStorage APIs are still appearing, evolving, and maturing. Virtual machine awareness currently requires information that just isn’t available to storage arrays. Instead, they take a shot in the dark, using what little information they have to divine performance requirements. Active blocks of any sort are moved to high-performance storage, while less-active data is migrated down. But they have no ability to logically group capacity below the LUN level in order to offer more advanced availability or reliability services.

Conclusion

One must approach the claims of any startup with skepticism, and this is especially true of one which has not yet shipped a product. But EvoStor’s concept is sound, and it lays out an exciting direction for storage to evolve. The key is in the company’s focus on what their target market needs, not just what their technology is (theoretically) capable of. EvoStor promises that their product will ship early next year.

This is the bottom line for me: Many VMware shops, especially smaller ones, have no history of specialized storage know-how and want something that just works. But VMware ESX has a nasty habit of pounding low-end storage systems with random I/O. VMware users in the midsize market need a storage solution that is integrated into the VMware experience, expands effortlessly, and self-tunes, and this is precisely what EvoStor is promising. I look forward to putting the product through its paces once it is out of the garage and generally available!

© Stephen Foskett for Gestalt IT, 2009. | EvoStor: VMware Storage Evolved!
Read more posts categorized as Exclusive, Featured, Server Virtualization, Storage, Top Story

VMFS Upgrade

Once of the good pieces of news to come out is that the VMFS changes in vSphere are minimal.  vSphere 4.0 introduces a minor point release (3.3.0 to 3.3.1) with some subtle changes,  so much so that it’s not really been documented anywhere.  Most of the changes with VMFS are actually delivered within the VMFS driver at the VMKernel level,  this is where most of the I/O improvements and features such as thin provisioning have been delivered as part of vSphere.

Upgrading VMFS was a major step in the upgrade from VMFS 2 to VMFS 3,  good to hear that there are no major drivers to upgrade VMFS as part of your vSphere upgrade.  Any new VMFS datastores created with the new vSphere hosts will of course be VMFS 3.3.1 however this is backwardly compatible with earlier versions of ESX 3.x.  If you really want to move onto the new version of VNFS, format some new datastores and use Storage vMotion to move your VM’s onto the new VMFS 3.3.1 datastores.

Thin Provisioning

Thin provisioning is one of the areas that excites me most about the new vSphere release.  I conducted a very quick survey of my employers development and system test ESX environments recently and found that currently we were only utilising 48% of virtual storage that had been provisioned.  It’s easy to see where immediate savings can be made simply by implementing vSphere and thin provisioning.  I’ll be using that in the cost benefits case for sure!

Thin provisioning is nothing new,  it has been available at the array level for a while now, so one of the big questions is where should I thin provision?  Well that really depends what kind of environment you have I suppose.  Smaller customers will benefit greatly from VMware thin provisioning as they probably don’t own arrays capable of TP.  Bigger companies on the other hand might well benefit from carrying out both as they have both the skill sets and the equipment to full utilise it at both levels.

Chad Sakac has written a superb article entitled “thin on thin where should you do thin provisioning vsphere 4.0 or array level” which goes deep into the new thin provisioning features and the discussions around what’s the best approach. I strongly suggest people give it a read,  it explains pretty much all you need to know.

Storage vMotion

The Storage vMotion in ESX 3.5 had a few limitations which vSphere addresses.  It’s now fully integrated with vCenter as opposed to being command line based in the previous version,  it allows for the moving of a VM between different storage types, i.e. FC, ISCSI or NFS.  One excellent usage of Storage vMotion is the ability to migrate your thick vm’s and change them to thin VM’s.  Perfect for reclaiming disk space and increasing utilisation without downtime, brilliant!

Storage vMotion has also been enhanced from an operational perspective. Previously storage vmotion involved taking a snapshot of a disk,  copying the parent disk to it’s new location and then taking the child snapshot and re-parenting the child disk with the parent.  This process required the 2 x the CPU and memory of the VM being migrated in order to ensure zero downtime.  In vSphere 4.0 Storage vMotion uses change block tracking and a process very similar to how vMotion deals with moving active memory between hosts.  The new Storage vMotion conducts an iterative process scanning what blocks have been changed, each iterative scan should result in smaller and smaller increments and when it gets down to a small enough size it conducts a very quick suspend / resume operation as opposed to using the doubling up resources method that it previously needed to.  Making it faster and more efficient than it was in it’s previous incarnation.

Para Virtualised SCSI

Para Virtualised SCSI (PVSCSI) is a new driver for I/O intensive virtual machines. VMware compare this to the vmxnet adapter,  which is an enhanced and optimised network driver providing higher performance.  PVSCSI is similar, it’s a specific driver that offers higher I/O throughput, lower latency and lower CPU utilisation within virtual machines. Figures discussed by Paul Manning on the recent Vmware community podcast included 92% increase in IOPS throughput and 40% decrease in latency when compared to the standard LSI / BUSLogic virtual driver.

A caveat of this technology is that the guest OS still has to boot from a non PVSCSI adapter (LSI / Buslogic),  you would look to add your PVSCSI adapter for your additional data virtual disks.  Currently only Windows 2003, Windows 2008 and RH Linux 5 have the software drivers to take adavantage of this new adapter.

Update  - Chad Sakac has posted a new EMCWorld I/O Performance comparison of the vSphere PVSCSI adpater vs the LSI SCSI adapter, check out the link for more details.

VMware Storage Book

Paul Manning mentioned on the recent podcast that VMware are planning a book dedicated to Virtualisation and storage in an attempt to consolidate the amount of documentation out there on Storage configuration and best practice.  Currently users need to look through 600 pages of the SAN Config guide and vendor guidelines. VMware would hope to try boil this down to a much more manageable 100 – 150 pages.

If you can’t wait that long, Chad Sakac has written the storage chapter in Scott Lowe’s new vSphere book which I believe is available for pre-order on Amazon

vSphere Storage WhitePaper

Paul Manning who I’ve mentioned in this blog post has written a great 10 page white paper explaining all of these features in more detail along with some of the more experimental features I haven’t mentioned.

http://www.vmware.com/files/pdf/VMW_09Q1_WP_vSphereStorage_P10_R1.pdf

© Craig for Gestalt IT, 2009. | vSphere 4.0 – What’s new in vSphere Storage
Read more posts categorized as Featured, Server Virtualization, Storage