Stepping out of a subway car is an entirely different matter when it's moving!

Mobility of virtual machines is a sticky wicket. As open systems infrastructure becomes increasingly virtualized, administrators and managers wish to use the technology to balance workload, ease migration, and provide better availability. Although technology is improving, actually moving virtual machines is not always a piece of cake. Let’s lay down a baseline of information so we may begin a discussion on the true nature of virtual machine mobility.

Mobility of What?

Let us consider first the question of what exactly is being moved. Systems administrators often focus on “the machine”, which encompasses the operating system and configured state of the virtual machine itself. But the true “mass” of the system is its stored data. Hypervisor vendors have come up with different techniques of moving these two essential elements, reflecting the unique characteristics of each.

• The virtual machine is an instance of an operating system along with its state and configuration. Mobility of virtual machines requires all of this to be preserved, along with any I/O channels. Live migration of virtual machine requires that any active network sessions be maintained, along with RAM content, registers and buffers, and so many other elements.
• The virtual machine image (commonly referred to as “storage”) is the static content addressed by a virtual machine. Typically a VMDK or similar virtual disc image, it must be accessible to the virtual machine at all times. Live migration of a virtual machine image is tricky, but perhaps not quite as complex as live migration of a running operating system.

VMware, Microsoft, and others recognize these 2 distinct elements to be migrated, and have come up with a variety of complementary technologies for each:

• vMotion is VMware’s virtual machine migration solution, and has continually evolved with each iteration of the hypervisor. DRS leverages vMotion to automate mobility. VMware has also created Storage vMotion and Storage DRS as complements to handle mobility of virtual machine images.
• Microsoft Hyper-V Live Migration is conceptually similar to vMotion, though newer and less full-featured. With Hyper-V 3.0, Microsoft will introduce Storage Live Migration as a complementary technology akin to Storage vMotion. Most other virtual machine managers also support some form of live migration, though live migration of storage is less common.

Mobility in What State?

One of the key benefits of virtual machine technology is the ability to “run anywhere” on dissimilar hardware. From the very beginning, hypervisors have provided the ability to create a universal virtual machine image that would run on a variety of supported platforms.

This leads to one of the key values of server virtualization in the data center: Disaster recovery. The ability to take a virtual machine image and system state and bring it online after a disaster is a true revolution for open systems IT. The benefits of the single usage of server virtualization technology easily justify the investment to many businesses.

But this sort of “cold” migration seems passé when compared to the live or “hot” migration possible with technologies like VMware vMotion. Live migration is much more difficult, since active client sessions must be preserved in activity must not be greatly interrupted.

This is the second great question that must be asked when considering virtual machine mobility: In what state will the virtual machine be moved? Will it be a cold, powered down image of the system? A suspended or paused operating system image? Or a full, running machine?

Mobility to Where?

Once we have decided whether we are discussing virtual machine migration or movement of storage resources, we must consider the scope of the movement. The ability to move a virtual machine from one member of the cluster to another has now become fairly common. But what about systems that are not related in a cluster? Or that are spread over great distances?

• The nice thing about clusters is that they share resources before and after a virtual machine is moved. It is practical to move the running virtual machine, its storage, or both independently and to expect that performance will not dramatically suffer as a result. The cluster can also preserve network connections, and even I/O state, without much impact on clients or other external elements.
• It is a bit more difficult to move systems within a data center, since one must maintain the I/O connections that might be interrupted. It is fairly trivial to configure an IP network and storage array to allow multiple machines to access the same iSCSI or NFS storage resources. It is a little more difficult to configure Fibre Channel (and, by extension, FCoE) SAN’s to handle this sort of dynamic movement, but it is not impossible. Although moving a running machine from one network port to another could cause client access to be interrupted, technologies like VXLAN allow these sessions to continue, and improved network switching technology should reduce performance impact.
• Moving the machine to a different data center is another matter entirely. Stretching a layer-2 Ethernet LAN or Fibre Channel SAN across a metro or greater distance, while possible, will always be problematic. IP routing is flexible, but it takes time for changes to propagate when live machines are moved. And it is difficult to keep storage in sync over long distances due to the amount of time it takes for information to transit. Again, all of these challenges are being addressed in various ways, but they’re still hard!

Stephen’s Stance

Moving cold virtual machine images from system to system, or even across great distances, is one of the main selling points of server virtualization. But it becomes much more difficult to manage movement of virtual machines that are still running, especially outside cluster or across WAN links. When talking about virtual machine mobility, it is important to consider what is being moved, the state it is in, and where it is going.

Note: This discussion is part of “Building Virtual Infrastructure”, my new seminar series with Truth in IT.

© sfoskett for Stephen Foskett, Pack Rat, 2012. |
Virtual Machine Mobility: Of What, and to Where and in What State?


This post was categorized as Enterprise storage, Features, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2012. | Virtual Machine Mobility: Of What, and to Where and in What State?
Read more posts categorized as Storage

How could a company as mighty and diverse as HP have had so many issues with executive management?

HP stumbled mightily in 2011, and it had nothing to do with product or people. Even sales remained strong, though the PC business is changing. HP’s mighty stumble was a crisis of confidence due to a chain of shenanigans at the very top. This culminated with the short reign of Léo Apotheker, leaving HP to reassure the market of its strategy.

HP And the Enterprise IT Industry

Sometimes, it’s hard to get a sense of scale when talking about very large things. How big is the solar system? How far away is the nearest star? The same is true of earthly things, exemplified by popular misconceptions about the global financial crisis. It’s difficult for people to understand just how much money a trillion dollars is.

In my little world of enterprise storage, it’s difficult to reconcile “big storage” players like EMC and NetApp with “big storage and everything else” players like HP, Dell, Oracle and IBM. Sure, EMC and NetApp lead the pack in terms of market share, but they’re nowhere near as large as the integrated players. HP has more than 7 times the revenue of EMC, which makes 3 times more than NetApp.

This old slide might need updating, but you get the picture…

HP is an incredibly diverse company, dominant in the PC, printing, and blade server market and top 5 just about everywhere else, including networking, services, and enterprise storage. And HP has nearly 325,000 employees, all working to move the company forward in one direction or another.

NetApp is a motorcycle, with one drive wheel pushing it forward at high speed; HP is more like NASA’s shuttle crawler-transporter, a 16 motor mammoth. Single-purpose companies can be agile, but they can also be derailed by market downturn or technological shift. Storage specialists like NetApp continually try to innovate and acquire to keep themselves vital, while larger companies like Cisco and EMC try to diversify while maintaining market leadership. HP doesn’t need to try; it is diverse.

HP Is a Very Large Thing

I watched HP's rise in the server market of the 1990's

HP has long been synonymous with innovation, high-technology, and silicon Valley. I have been an HP customer as long as I have been in IT, and watched as they integrated technology from Apollo and Convex in the 1990’ s. The server products that resulted became the dominant UNIX platform, but HP’ s innovative storage concepts didn’ t take the market by storm.

After HP merged with Compaq (which brought Tandem and Digital Equipment Corporation), the company vaulted ahead in the Wintel market and also gained valuable storage expertise. Throughout the last decade, HP was firing on all cylinders and dominant in nearly every arena. The company owns half the blade server market, is tied for first in servers, leads in PCs and printers, and is a contender in networking and storage. It’s simply impossible to say what HP is in a single sentence.

HP storage has an extremely broad product range, which management is working to reconcile. Acquisitions of LeftHand, Ibrix, and 3PAR gave HP storage shot in the arm to be sure. An injection of startup mojo has energized the marketing and product groups within HP just when the company needed it. HP’ s market share has grown somewhat as a result, though not as much as the hyper-focused NetApp. HP networking similarly took on 3Com, bedeviling Cisco in the Ethernet switch market.

The Executive Soap Opera

It takes a lot to bring a mammoth to its knees, but a shot between the eyes usually does the trick. While the many engines of HP push it forward, the company’s upper management has seemed, at times, suicidal. Business schools could design an entire curriculum around the folly of Patricia “I spy” Dunn and Mark “penny-pinching” Hurd. Who would think that HP management could top this?

From August 2010 through September 2011, HP dominated IT headlines in completely the wrong way. The board wanted a change, and selected Léo Apotheker to transform HP. But it was a soap opera from the very start, with Oracle hiring Mark Hurd and sending Apotheker into hiding among accusations of corporate espionage while at SAP.

The new CEO didn’t seem to understand HP at all, though he was intent on steering it in a new direction. Apotheker set about dismantling HP’s consumer and end-user businesses, killing Palm/WebOS and threatening to sell off the PC business. The company was to focus instead on enterprise computing, but these drastic moves spooked the entire industry.

It wasn’t long before HP’s board struck again, with a shake up at the hands of Ray Lane and former eBay CEO Meg Whitman brought in to replace Apotheker. The first order of business for the new HP executive team appears to be reassuring the entire world that management has not gone completely insane.

Stephen’s Stance

To an outsider like me, the most disappointing thing about HP’s mighty stumble is that it has nothing to do with the people who really make the company what it is. I have met many creative, hard-working individuals in HP’s storage, server, networking, and printer groups, and they could not be more different from the executive soap opera. I only hope that this new board and CEO will bring some stability and let HP cruise forward once again.

Disclaimer: HP has sponsored the Tech Field Day events which I organize, and has on occasion invited me to attend events at their expense.  But I do similar work with nearly every company in the IT industry, and this piece is my own opinion.

© sfoskett for Stephen Foskett, Pack Rat, 2012. |
HP’s Mighty Stumble


This post was categorized as Computer History, Enterprise storage, Gestalt IT. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2012. | HP’s Mighty Stumble
Read more posts categorized as Storage

I Promise Not To Quote That Old Annoying Dell PC Slogan

I’ve been buying Dell computers for decades, but not really because I loved them. Sure, my XPS laptop was awesome, but it burned out its motherboard and I never really touched the RMA replacement, having bought a MacBook Pro in the meantime. Enterprise buyers seem to have the same ambivalence about Dell. They buy it, but I’m not sure they really “buy” the company as an IT partner.

I’m sure you’ve heard the same comments as me: “Dell just assembles off-the-shelf components and sells them in volume” or “Dell’s a follower, not an innovator.” There seems to be a great deal of respect for Dell’s ability to produce competitive products and sell them at reasonable cost. Truly, most of their competitors would love to have this kind of reputation. But most of their competitors also have a reputation for partnership, innovation, and solution selling.

Dell Is Making An Effort

It seems clear that Dell would like to change this attitude, and they are investing serious resources to make it happen. While acquisitions like Compellent and Force10 raised eyebrows in storage and networking, it is the activity I see behind the scenes that paints the clearest picture. Dell isn’t just buying into new markets, they’re investing to change the company.

When Dell acquired EqualLogic in 2008, many assumed it was a tactical investment to increase margins over the (resold) EMC storage equipment the company was then pushing. Pundits were similarly dismissive of the acquisition of Perot Systems in 2009, calling it a “me too” effort after HP acquired rival EDS. Regardless of the motivations, however, Dell was becoming more of a serious challenger to HP and IBM every day.

After failing to acquire 3PAR in 2010, then picking up Compllent shortly after, accusations that Dell was “mini me” to HP were rampant. But HP stumbled mightily in 2011, and many in IT quickly lost confidence in that company’s management. All the while, Dell moved forward, increasing in-house IP and expanding enterprise offerings.

What Is The Result?

Today, one sees a very different landscape than just last year. Dell’s acquisitions focused on some of the ripest spots in storage and networking, and no one would disagree that the company has the ability strongly to push these products. Compellent and Force10 went from interesting startups to serious contenders overnight.

Dell really pulled out all the stops to tell us they are "Fluid by Design"

More importantly, Dell has retained much of the innovation these companies offered, from employees to support programs. Last week, I attended the Dell Storage Forum in London, an event initiated by Compellent prior to the acquisition. At the event, I talked to many Dell employees who came to the company through acquisition but had now been given power to challenge the status quo in their respective areas.

If Dell really intended only to push product, why retain marketing personnel? Why invest in the Dell Storage Forum? Why continue Compellent’s beloved Co-Pilot support program?

Then there are the products. Dell leveraged its investment in Ocarina Networks to create a deduplicating backup appliance, the new DR4000. They salvaged file system startup ExaNet and are beginning to bring scale out technology to market. The latest revision of the Compellent software finally brings it to parity in terms of VMware support. And Dell is really working to sell their DX Object Store.

This is the sort of activity one would expect from a contender, not a “box pusher”.

Stephen’s Stance

In the words of Malcolm Reynolds, my days of not taking Dell seriously are certainly coming to a middle. Dell is investing in product IP, innovative marketing and PR events, customer support, and personnel. This does not mean that Dell is instantly a player in the enterprise storage and networking markets, or that all this work will pay off. But I don’t laugh when I hear Dell boast that they intend to be a “top three” enterprise storage company in a few years. It could happen.

Disclaimer: Dell sponsored two Tech Field Day events in 2011, paid me as a speaker at two DX events, and paid for my trip to Dell Storage Forum in London. But no one can buy a post on this site, and I did similar business with IBM, HP, Cisco, and many other companies. This is my opinion.

© sfoskett for Stephen Foskett, Pack Rat, 2012. |
Why Should Anyone Take Dell Seriously in Enterprise Storage?


This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2012. | Why Should Anyone Take Dell Seriously in Enterprise Storage?
Read more posts categorized as Storage

When designing VDI environments, a couple of things that you should consider are the .vswp file and the GuestOS paging file. In this article I am going to focus on the Paging file and hopefully in the not so distant future I will write a post about the .vswp file in a VDI environment.

What is point of the paging file (also known as the pagefile.sys)?

RAM is a limited resource. Virtual memory was introduced to help remove that limit.

Most modern operating system now use Virtual Memory. Virtual memory is a memory management technique. Applications running on a GuestOS reference memory using virtual memory addresses which are then automatically translated into RAM addresses by the hardware. These virtual memory address spaces are divided in pages or block, usually of 4KB.

If RAM resource is exhusted, the operating system will move 4KB pages of the virtual memory onto the computers hard disk to free up the physical memory (RAM) for other processes. In Windows operating systems, these pages are stored in the pagefile.sys.

A good way to think of this is;

Imagine a restaurant that has just open for the evening. When customers (Processes) arrive they get allocated a table (RAM) to sit and eat at. As the night goes on the restaurant get busier and free tables (RAM) begin to run out for the new customers (Processes) coming through the door. To free up spare tables (RAM) the waiter asks customers (Processes) who have finished eating if they wouldn’t mind moving to the bar (Virtual Memory) where they can continue drink.

Without the paging file, if the physical memory becomes full, applications including the operating system will have to waiting until physical memory becomes available before it can be stored in RAM ready for the CPU to process. As you can imagine this causes massive performance problems.

In summary, you NEED to have a paging file.
Read the rest of this entry »

© Simon for Gestalt IT, 2012. | VDI Paging Files – Big? Small? Or Non At All?
Read more posts categorized as Desktop, Server Virtualization

It's not going to be this easy to bridge Fibre Channel and Ethernet!

Before the holidays, I posed a question on Google+ that generated quite a bit of interest and feedback. Now that it has settled down a bit I’d like to summarize the unresolved elements to make FCoE truly a world-class storage interconnect.

Setting the Stage

FCoE has been a controversial topic in both storage and networking, and for good reason. No one would deny that Ethernet is not an ideal transport mechanism for block storage I/O. “Porting” Fibre Channel to run on Ethernet networks has been a supreme technical challenge, and many companies and individuals have labored long and hard to make FCoE a reality.

Now that FCoE is specified in the standard and has been deployed in production environments, the question turns to its future. Will it take off and seize the mantle of dominance currently held by what I like retroactively to call “Fibre Channel over Fibre Channel?” Will they coexist for the next decade, with FCoE mainly deployed in “block” environments such as Cisco UCS? Or will FCoE ultimately fail to catch on, displaced by some other storage protocol like plain FC, iSCSI, NFS, or something entirely different?

The data center needs a flexible new protocol to meet the needs of virtual environments, and convergence of storage and data networking makes a great deal of sense in these environments. This was the root of my question, and I ask it in all earnestness.

My question: What elements remain unresolved to make FCoE truly world-class? What should the vendors be prioritizing? Here are the answers I received.

Technical Considerations

Link Aggregation on CNA’s

Converged network adapters (CNA’s) allow multiple protocols to access a single Ethernet connection, but some also include multiple ports that can be aggregated. In traditional Ethernet networks, link aggregation is a respectable approach for performance and availability. But storage networks have traditionally relied on host-based MPIO software, and these features are mutually exclusive. The zeitgeist seems to be a recommendation to avoid link aggregation on CNA’s that are used for storage networks.

How Do You Handle Virtual Machine Mobility?

As I described recently, virtual machine mobility is a major technical challenge for existing networks. The VMware proposal, the VXLAN, seems to be gaining traction right now. But this is only a solution for data networking. How will FCoE SANs handle virtual machine mobility? This remains unresolved as far as I can tell, though Ethernet switch vendors have come up with their own answers. Brocade demonstrated just such a solution at Networking Field Day 2, and I know that others have answers as well. But will there be an interoperable industry solution?

How Should FCoE Be Implemented Over Longer Distances?

Fibre Channel has traditionally relied on routers and other protocols (FCIP and iFCP) to span distances, but FCoE raises the possibility of native traversal. While it is certainly possible to span distances with FCoE, this is definitely not a recommended or supported idea. Without TCP/IP, or any routing mechanism, it’s just a bad idea. But I imagine that it won’t be long before vendors decide to give it a go anyway.

Implementation Considerations

Is TRILL Required for FCoE Networks?

This has been one of my own questions since the very beginning. Clearly, edge only FCoE works just fine without TRILL. But as networks become more complicated, and virtual machines move, it seems an awfully good idea to have some protocol to alleviate East-West routing concerns. I feel much better with TRILL (or some similar Ethernet fabric technology) in a complicated FCoE network.

Should All Switches Be Full FC Forwarders?

There are number of ways to implement FCoE on Ethernet network, and not all involve building a full Fibre Channel stack in each switch. While many (including myself) assumed that FCoE implied Fibre Channel forwarding in all switches, this is clearly not the direction taken by vendors, at least initially. Perhaps the current “Ethernet forwarding” approach is only a stepping stone, or perhaps it will emerge as the dominant FCoE standard.

How Will OpenFCoE and LoM Be Used?

OpenFCoE is a software solution allowing FCoE to be run without a CNA. If this became popular, it wouldn’t be long before data center architects began looking at LAN on Motherboard (LoM) and even 10GBASE-T as a potential SAN alternative. Will this be used in the long run? It could happen, but it’s certainly not something that’s here at the moment. But OpenFCoE is a real player, especially with Intel’s backing.

How Will Technologies like Zoning Interoperate?

Many networkers are just now beginning to see the true complexity of Fibre Channel SANs. Although interoperability of higher-level Fibre Channel functions between vendors has never been a priority in “FC over FC” SANs, Ethernet could change things. I would not be at all surprised to see a groundswell of customer support demanding greater levels of interoperability from FCoE than from FC, and zoning and VSAN is the likely first beachhead.

The Big Question: When Will We See the “Killer App” For FCoE

Just about everyone agreed that the real challenge for FCoE is market acceptance. Customers aren’t yet demanding FCoE, and vendors are finding it hard to articulate a compelling case to move from “tried-and-true” FC. Convergence, cost savings, and performance have all been put forth, but customers aren’t biting. Perhaps they just need a little time and a little more proof.

This post relies extensively on feedback from a number of people, including Ivan Pepelnjak, Tony Bourke, J Metz, Dmitri Kalintsev, Derick Winkworth, David Hardaker, Juan Lage, and Corey Hines.

© sfoskett for Stephen Foskett, Pack Rat, 2012. |
Eight Unresolved Questions About FCoE


This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you’d like to filter out or focus on posts like this.

© Stephen Foskett for Gestalt IT, 2012. | Eight Unresolved Questions About FCoE
Read more posts categorized as Storage

Note: I realize that they are doing this already, but until recently I hadn’t been close enough to a product that used it to worry much about it.

Then I started getting into Exchange 2010.  PowerShell for managing E-mail from almost all aspects of the process is a damn fine idea.  Now I have a reason to learn more than a few simple commands, because I might actually put them to use.

Where I have been

I signed up for an account at PowerShell.com to get my feet wet and hopefully participate in a community. I have found this to be somewhat useful in the past and am hoping that it allows me an outlet to go dig around in other peoples scripts to see if I can comprehend just what they are doing.

I am on the fence about ISE… I know at some point notepad will become a burden to use, but just starting out, I don’t think I need to worry about that just yet.  If anyone has any suggestions for things to look at in terms of ISE or just good resources for learning PowerShell, please post them in the comments.  I am anxious to get a jump on this thing in the hopes that it will be worth the effort to understand.

Tons to learn

I have tried a few things that I found while binging my way around the web and it has been interesting to see what is out there, surely I haven’t even scratched the surface yet.  I would think a PowerShell magazine or some newsletter type offering would be a huge benefit to the PowerShell community.  Maybe the guys at Redmond Magazine would consider getting something wild like that off the ground??

For now, I will begin re-perusing the books I have on PowerShell and dig into the Internet on the subject further.  Maybe there will be an event near me in the future that will help me learn… I will keep my eyes peeled for that for sure.

© derek for Gestalt IT, 2012. | PowerShell… What an awesome tool
Read more posts categorized as Desktop, Server Virtualization

Windows 8 server editions will include a filter driver for NTFS for data deduplication

The next version of Microsoft Windows Server includes integrated data deduplication technology. Microsoft is positioning this as a boon for server virtualization and claims it has very little performance impact. But how exactly does Microsoft’s de-duplication technology work?

Introducing Windows 8 Deduplication

Let’s make one thing clear right from the start: Microsoft started from a clean sheet and invented their own deduplication technology. This is not a licensed, cloned, or copied feature as far as I can tell. There are some clever aspects to it, along with a few head scratchers for folks like me who’ve seen lots of different deduplication approaches.

Microsoft’s deduplication is layered onto NTFS in Windows 8, and will be a feature add-on for Server users. It is implemented as a filter driver on a per volume basis, with each volume a complete, self describing unit. It is cluster aware, and fully crash consistent on all operations. This is a pretty neat trick: As is typical for Microsoft, deduplication will be a simple, transparent feature.

Now let’s talk for a moment about what Windows 8 deduplication is not.

• It is a server-only feature, like so many of Microsoft’s storage developments. But perhaps we might see it deployed in low-end or home servers in the future.
• It is not supported on boot or system volumes.
• Although it should work just fine on removable drives, deduplication requires NTFS so you can forget about FAT or exFAT. And of course the connected system must be running a server edition of Windows 8.
• Although deduplication does not work with clustered shared volumes, it is supported in Hyper-V configurations that do not use CSV.
• Finally, deduplication does not function on encrypted files, files with extended attributes, tiny (less than 64 kB) files, or re-parse points.

Some Technical Details on Deduplication in Windows 8

Microsoft Research spent 2 years experimenting with algorithms to find the “cheapest” in terms of overhead. They select a chunk size  for each data set. This is typically between 32 KB and 128 KB, but smaller chunks can be created as well. Microsoft claims that most real-world use cases are about 80 KB. The system processes all the data looking for “fingerprints” of split points and selects the “best” on the fly for each file.

After data is de-duplicated, Microsoft compresses the chunks and stores them in a special “chunk store” within NTFS. This is actually  part of the System Volume store in the root of the volume, so dedupe is volume-level. The entire setup is self describing, so a deduplication NTFS volume can be read by another server without any external data.

There is some redundancy in the system as well. Any chunk that is referenced more than x times (100 by default) will be kept in a second location. All data in the filesystem is checksummed and will be proactively repaired. The same is done for the metadata. The deduplication service includes a scrubbing job as well as a file system optimization task to keep everything running smoothly.

Windows 8 deduplication cooperates with other elements of the operating system. The Windows caching layer is dedupe-aware, and this will greatly accelerate overall performance. Windows 8 also includes a new “express” library that makes compression “20 times faster”. Compressed files are not re-compressed based on filetype, so zip files, Office 2007+ files, etc will be skipped and just deduped.

New writes are not deduped – this is a post-process technology. The data deduplication service can be scheduled or can run in “background mode” and wait for idle time. Therefore, I/O impact is between “none and 2x” depending on type. Opening a file is less than 3% greater I/O and can be faster if it’s cached. Copying a large file can make some difference (e.g. 10 GB VHD) since it adds additional disk seeks, but multiple concurrent copies that share data can actually improve performance.

Stephen’s Stance

Although I am intrigued by Microsoft’s new deduplication technology in Windows 8 server, I still have many questions about its usefulness and impact on performance. Concentrating duplicate data in the system volume makes sense from a technical perspective, but could lead to an I/O hotspot on the disk. This is especially true for external caching storage systems, since there is no integration between Microsoft deduplication and storage array features. I am particularly concerned about the use of deduplication with VHD files in Hyper-V, since it could eat up valuable system RAM and impact I/O performance.

If you would like to try Microsoft deduplication for yourself, I am happy to report that it is included in the developer preview of Windows 8 that is available on Dev Center. Here are a few commands to get you started, and read Rick Vanover’s post too!

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication
Import-Module Deduplication
Enable-DedupVolume E:
get-dedupvolume

Note: I am a Microsoft MVP and Microsoft briefs me on upcoming technologies under NDA. This post is based on a Microsoft briefing from November which was said at the time not to be covered by any NDA. All of this information could be gleaned by experimenting with the Windows 8 developer preview, but it’s much easier to just go to the source.

© Stephen Foskett for Gestalt IT, 2012. | Microsoft Adds Data Deduplication to NTFS in Windows 8
Read more posts categorized as Server Virtualization, Storage

It isn’t always easy to get where you need to go!

Consider the following situation: You go to lunch with your good friends, John and Mary. Halfway through a rousing discussion of the latest Hollywood movie, Mary starts talking about the fantastic action sequences while John criticizes the romantic angle. You realize something mine-bending has happened: John now has Mary’s personality, and vice versa. It’s like they have switched brains or something!

This truly weird situation isn’t likely to happen in person, but occurs all the time in the data center. Virtualization of server, network, and storage services illuminates the link between physical resources and functional applications. A running virtual machine can instantly move from one server, network adapter, HBA, or LUN to another. And when it happens, traditional components have no idea how to react.

The Challenges of Mobility

Mobility is perhaps the “killer app” of virtualization, but it is also the killer of traditional IT systems. Let’s consider the challenges of this “Twilight Zone” moment.

• The operating system expects a consistent hardware environment, which is exactly what the hypervisor creates
• The LAN must be prepared to redirect all network traffic instantly and seamlessly to one or more new physical interfaces
• The SAN similarly must be able to reroute all I/O to a new pair of HBA’s without missing a beat
• The storage array must be able to re-present capacity to a new physical device, and must maintain snapshots and other configurations
• The backup system must also be able to maintain consistency over time even as machines relocate to different server and storage locations

All of this must be done while maintaining quality of service (QoS), access control, reporting, and appropriate segmentation at all levels. This is an incredibly challenging task, and no conventional protocol (IP, Ethernet, NFS, SCSI, Fibre Channel, etc.) is anymore ready then you are when you’re good friends switch personalities.

Two Paths

So much of the development that is currently taking place in IT focuses on accommodating this “mobility issue”. Two key approaches have emerged to take on this challenge:

• “In a vacuum” technologies (like VXLAN) assume that no other changes will be made, so the focus is on maintaining complete compatibility in front and behind
• “Clean sheet” technologies (usually from startups) take a different approach, throwing out compatibility in favor of technical elegance

Both of these approaches have merit. Attempting to maintain compatibility only works so far (just ask a Windows API programmer), but it leverages the existing environment and recognizes that most people are not ready for wholesale change. Clean sheet designs always make more sense, but they rarely attain mass acceptance. Nearly every technology we rely on today is full of bolt-ons in the name of compatibility. Some, like Ethernet and x86, actually work pretty well, too.

The Stack of Lies

The difference between virtualization and cloud computing is exactly this same distinction. Hypervisors, NPV, NAT, thin provisioning, and so many other virtualization technologies exist mainly to maintain compatibility in a vacuum. In contrast, true cloud computing dispenses with the entire stack and creates a new platform for applications.

This is, perhaps, the reason that cloud computing is not taken off in the enterprise. Simply put, IT is not prepared to ditch everything they have ever used even in the face of a demonstrably superior alternative. Currently, the highest use of cloud is behind gateways and virtualization engines that bring it back down to earth.

This brings us to the stack of lies called server virtualization. Any “modern” virtualized data center is built on lie after lie, with each level telling the other what it wants to hear. The volume manager lies to the operating system, hypervisor lies to the volume manager, and the storage array lies to the hypervisor. The same sad state of affairs allows networking and even memory to function in a virtual world.

But these shaky stacks of lies have difficulty adapting to motion, since no level truly “knows” the reality of the world around. The depressing truth is that a bowl of spaghetti like VXLAN is perhaps the highest form of art we can expect in a virtual data center.

Stephen’s Stance

As a techie, I am always drawn to clean sheet designs that offer technical elegance along with functionality. But I know that, realistically, products that assume nothing about the world around them and bend over backward to maintain compatibility are more likely to succeed. Still, I maintain hope that the issues of virtual machine mobility will be solved in an elegant way, rather than adding to the “stack of lies”.

© Stephen Foskett for Gestalt IT, 2012. | The Terrifying True Story Of Virtual Machine Mobility
Read more posts categorized as Server Virtualization, Storage

This is Tom. Ask him about NAT!

After reading the post, I had some thoughts come to mind that I just had to throw into a reaction post.

New Culture

As new generations of individuals grow up and mature, it is expected that cultural shifts will take place. What I do not understand is how a culture of technological availability has morphed into an expectation that an individual can bring anything into the corporate environment and expect to use it for their job.

Too many times, I am approached by users bringing their personal laptop into the office and wanting to know how to connect it to the internal network. Or, users that want to connect their iPhones to the network so they can use Spotify or YouTube without using their cellular data plans… as though the corporate infrastructure and services are there to do their bidding.

This new culture developing assumes that everything in the outside world must be the same as in the corporate world. Their iPad can connect to GMail, so why not just connect it to the Exchange server?

What the user sees.                            What IT sees!

Unknown/Unowned Devices

The IT ecosystem is a carefully designed and tightly guarded world.

None shall pass!

Systems are selected carefully to ensure a proper balance between functionality, supportability, and stability. The discovery of an unknown device is enough to throw an IT professional into a fit of rage. The environment has been compromised in some fashion and there is potential to throw off the carefully designed balancing act.

The presence of an unknown device opens up a venerable Pandora’s Box and raises a huge red flag. Suddenly, the corporate environment is now vulnerable to a machine or device infested with trojans, a honeypot of virus infections, access to corporate resources, and not managed by IT.

IT has been assigned a critical role in modern businesses… provide tools that enable the business to function. Traditionally, this included the workstation, network, monitors, servers, etc… With more people feeling as though it is acceptable to provide their own devices, who is responsible for supporting them? What happens when the “S” key breaks off or the monitor is too blue for their liking. When IT owns and manages a device, IT is responsible. When the users owns the device, but is using it in a corporate environment, the answer is much foggier. An IT persons says the user is responsible. However, the true answer lies somewhere in the depths of politics and policy.

Unknown devices also introduce the loss of data control. The moment a user is allowed to bring in a USB drive, iPod, access GMail, or Dropbox, the data is no longer under any control of the company.

Corporate IT Adaptation

First and foremost, IT has a responsibility to the company to ensure the protection and function of corporate technological resources and systems.

However, with that said, IT needs to acknowledge the changing ways of technology. Anyone who has been in IT longer than 1 month knows that times have a way of changing and the minute you buy your phone, it is obsolete. That is the way of the world and 42 is the ultimate answer to the ultimate question of live, the universe, and everything.

Is Google “Deep Thought”?

IT departments need to be cognoscente of what exists in the marketplace, impacts (both positive and negative) to overall productivity/security, and the long term viability of those entities. A tablet, for example, may seem like a large phone (cough iPad cough). However, for an executive that spends more time meeting customers and reading email, it is a perfect tool to enable them to get their job done without needing a laptop… but how is it secured?

Security becomes one of the most important concerns for IT in a time where users have expectation of providing their own devices. NAC/NAP/Port Security ensures authorized devices are allowed on the network. Remote technologies (Application Presentation (XenApp/RemoteApp) and VDI (View, XenDesktop)) allow users to interact with applications running on protected and trusted infrastructure from unknown endpoints. Proper backups, snapshotting, and antivirus on the server and storage side ensure the data consistency is proper and recoverable in the event of a break in security.

Finally, IT needs to engage with the business to keep them abreast of concerns. Open dialogue with the business will help ensure technological expectations meet some sort of equilibrium between what IT feels is appropriate and what the business feels is necessary.

What do you really think, Bill?!

I whole heartedly do not like the idea of users bringing in their own devices for business use. Maybe I am cruisin for a bruisin (politically speaking), but I see my environment as known and trusted. The introduction of a new device takes some planning and testing because I have a responsibility to the company to provide a stable and operational environment. The introduction of a Mac laptop into my environment is not smooth. Exchange and SharePoint support is so horrible that Mac users need to use a Fusion VM running Windows 7 to fully function.

However, while it is possible to be completely restrictive and be more like “The Man”, I feel that the best way to manage the user owned devices converging on my environment is more political.

- I encourage the business to adapt corporate policies addressing the need to not bring personal devices into the environment.

- I encourage the business to develop a stricter definition of who needs email outside of the office, partial compensation for use of personal devices OR providing a company owned and managed phone, and which devices are supported.

- Have an open and friendly dialogue with those users that approach IT for assistance with personal devices. Being honest and frank about not supporting devices, needing management approval, and being unsure as to the functionality/operation of the device goes a long way.

I love the idea of new devices and new technology in the workplace. But, I want the introduction to be more structured and tested.

Tom – Thanks for the awesome post. Definitely food for thought and got my wheels spinning!

© Bill for Gestalt IT, 2011. | BYO(a)D Reaction
Read more posts categorized as Desktop, Featured

I have been waiting for some time to get some ESXi 5.0 awesomeness going on in my environment. vCenter has been sitting on v5 for some time and I have been deploying ESXi 5 in a couple stand-alone situations without any issues. So, now that I have more compute capacity in the data center, it is time to start rolling the remaining hosts to ESXi 5… or so I thought!

I downloaded ESXi 5.0.0 Kernel 469512 a while back and have been using that on my deployments. So far, so good… until today. Update Manager configured with a baseline –> Attach –> Scan –> Remediate –> back to business. Surely, Update Manager processes should take more time than the actual upgrade. About 30 minutes after starting the process, vCenter was showing that the remediation progress was a mere 22% complete and the host was unavailable. I used my RSA (IBM’s version of HP ILO or Dell DRAC) to connect to the console. Sure enough, it was stuck at loading some kernel modules. About 20 minutes later IT WAS STILL THERE!

Restarting the host did not resolve the issue. During the ESXi 5 load screen, pressing Alt + F12 loads the kernel messages. It turns out that iSCSI was having issues loading the datastores in an acceptable amount of time. I was seeing messages similar to:

A little research turned me onto the following knowledgebase article in VMware’s KB: ESXi 5.x boot delays when configured for Software iSCSI (KB2007108)

To quote:

This issue occurs because ESXi 5.0 attempts to connect to all configured or known targets from all configured software iSCSI portals. If a connection fails, ESXi 5.0 retries the connection 9 times. This can lead to a lengthy iSCSI discovery process, which increases the amount of time it takes to boot an ESXi 5.0 host.

So, I have 13 iSCSI stores on that specific host and multiple iSCSI VMkernel Ports (5). So, calling the iSCSI lengthy is quite the understatement.

The knowledgebase states that the resolution is applying ESXi 5.0 Express Patch 01. Fine. I can do that. And… there is a work around described in the article that states you can reduce the number of targets and network portals. I guess that is a workaround… after you have already dealt with the issue and the ridiculously long boot.

Finally, to help mitigate the issue going forward, VMware has released a new .ISO to download that includes the patch. However, this is currently available in parallel with the buggy .ISO ON THE SAME PAGE! Seriously. Get this… the only way to determine which one to download is:

As a virtualization admin, I know that I am using the Software iSCSI initiator in ESXi. But, why should that even matter at all?! There is a serious flaw in the boot process in version 469512  and that should be taken offline. Just because someone is not using Software iSCSI at the current time does not mean they are not going to in the future. So, if they download the faulty .ISO, they are hosed in the future. Sounds pretty crummy to me!

My Reaction

I am quite shocked that this made it out of the Q/A process at VMware in the first place. My environment is far from complex and I expect that my usage of the ESXi 5.0 hypervisor would be within any standard testing procedure. I try to keep my environment as vanilla as possible and as close to best practices as possible. 1.5 hours for a boot definitely should have been caught before release to the general public.

Additionally, providing the option to download the faulty ISO and the fixed ISO is a complete FAIL! As mentioned on the download page, this is a special circumstance due to the nature of the issue. I would expect that if this issue is as serious as the download page makes it out to be, the faulty ISO should no longer be available. There has to be a better way!

Conclusion

I have since patched the faulty ESXi 5.0 host to the latest/safest version, 504890, and boot times are back to acceptable. I will proceed with the remainder of the upgrades using the new .ISO and have deleted all references to the old version from my environment.

I have never run into an issue like this with a VMware product in my environment and I still have all the confidence in the world that VMware products are excellent. In the scheme of things, this is a bump in the road.

© Bill for Gestalt IT, 2011. | ESXi 5.0–1.5 Hour Boot Time During Upgrade
Read more posts categorized as Server Virtualization