When designing VDI environments, a couple of things that you should consider are the .vswp file and the GuestOS paging file. In this article I am going to focus on the Paging file and hopefully in the not so distant future I will write a post about the .vswp file in a VDI environment.

What is point of the paging file (also known as the pagefile.sys)?

RAM is a limited resource. Virtual memory was introduced to help remove that limit.

Most modern operating system now use Virtual Memory. Virtual memory is a memory management technique. Applications running on a GuestOS reference memory using virtual memory addresses which are then automatically translated into RAM addresses by the hardware. These virtual memory address spaces are divided in pages or block, usually of 4KB.

If RAM resource is exhusted, the operating system will move 4KB pages of the virtual memory onto the computers hard disk to free up the physical memory (RAM) for other processes. In Windows operating systems, these pages are stored in the pagefile.sys.

A good way to think of this is;

Imagine a restaurant that has just open for the evening. When customers (Processes) arrive they get allocated a table (RAM) to sit and eat at. As the night goes on the restaurant get busier and free tables (RAM) begin to run out for the new customers (Processes) coming through the door. To free up spare tables (RAM) the waiter asks customers (Processes) who have finished eating if they wouldn’t mind moving to the bar (Virtual Memory) where they can continue drink.

Without the paging file, if the physical memory becomes full, applications including the operating system will have to waiting until physical memory becomes available before it can be stored in RAM ready for the CPU to process. As you can imagine this causes massive performance problems.

In summary, you NEED to have a paging file.
Read the rest of this entry »

© Simon for Gestalt IT, 2012. | VDI Paging Files – Big? Small? Or Non At All?
Read more posts categorized as Desktop, Server Virtualization

Note: I realize that they are doing this already, but until recently I hadn’t been close enough to a product that used it to worry much about it.

Then I started getting into Exchange 2010.  PowerShell for managing E-mail from almost all aspects of the process is a damn fine idea.  Now I have a reason to learn more than a few simple commands, because I might actually put them to use.

Where I have been

I signed up for an account at PowerShell.com to get my feet wet and hopefully participate in a community. I have found this to be somewhat useful in the past and am hoping that it allows me an outlet to go dig around in other peoples scripts to see if I can comprehend just what they are doing.

I am on the fence about ISE… I know at some point notepad will become a burden to use, but just starting out, I don’t think I need to worry about that just yet.  If anyone has any suggestions for things to look at in terms of ISE or just good resources for learning PowerShell, please post them in the comments.  I am anxious to get a jump on this thing in the hopes that it will be worth the effort to understand.

Tons to learn

I have tried a few things that I found while binging my way around the web and it has been interesting to see what is out there, surely I haven’t even scratched the surface yet.  I would think a PowerShell magazine or some newsletter type offering would be a huge benefit to the PowerShell community.  Maybe the guys at Redmond Magazine would consider getting something wild like that off the ground??

For now, I will begin re-perusing the books I have on PowerShell and dig into the Internet on the subject further.  Maybe there will be an event near me in the future that will help me learn… I will keep my eyes peeled for that for sure.

© derek for Gestalt IT, 2012. | PowerShell… What an awesome tool
Read more posts categorized as Desktop, Server Virtualization

Windows 8 server editions will include a filter driver for NTFS for data deduplication

The next version of Microsoft Windows Server includes integrated data deduplication technology. Microsoft is positioning this as a boon for server virtualization and claims it has very little performance impact. But how exactly does Microsoft’s de-duplication technology work?

Introducing Windows 8 Deduplication

Let’s make one thing clear right from the start: Microsoft started from a clean sheet and invented their own deduplication technology. This is not a licensed, cloned, or copied feature as far as I can tell. There are some clever aspects to it, along with a few head scratchers for folks like me who’ve seen lots of different deduplication approaches.

Microsoft’s deduplication is layered onto NTFS in Windows 8, and will be a feature add-on for Server users. It is implemented as a filter driver on a per volume basis, with each volume a complete, self describing unit. It is cluster aware, and fully crash consistent on all operations. This is a pretty neat trick: As is typical for Microsoft, deduplication will be a simple, transparent feature.

Now let’s talk for a moment about what Windows 8 deduplication is not.

• It is a server-only feature, like so many of Microsoft’s storage developments. But perhaps we might see it deployed in low-end or home servers in the future.
• It is not supported on boot or system volumes.
• Although it should work just fine on removable drives, deduplication requires NTFS so you can forget about FAT or exFAT. And of course the connected system must be running a server edition of Windows 8.
• Although deduplication does not work with clustered shared volumes, it is supported in Hyper-V configurations that do not use CSV.
• Finally, deduplication does not function on encrypted files, files with extended attributes, tiny (less than 64 kB) files, or re-parse points.

Some Technical Details on Deduplication in Windows 8

Microsoft Research spent 2 years experimenting with algorithms to find the “cheapest” in terms of overhead. They select a chunk size  for each data set. This is typically between 32 KB and 128 KB, but smaller chunks can be created as well. Microsoft claims that most real-world use cases are about 80 KB. The system processes all the data looking for “fingerprints” of split points and selects the “best” on the fly for each file.

After data is de-duplicated, Microsoft compresses the chunks and stores them in a special “chunk store” within NTFS. This is actually  part of the System Volume store in the root of the volume, so dedupe is volume-level. The entire setup is self describing, so a deduplication NTFS volume can be read by another server without any external data.

There is some redundancy in the system as well. Any chunk that is referenced more than x times (100 by default) will be kept in a second location. All data in the filesystem is checksummed and will be proactively repaired. The same is done for the metadata. The deduplication service includes a scrubbing job as well as a file system optimization task to keep everything running smoothly.

Windows 8 deduplication cooperates with other elements of the operating system. The Windows caching layer is dedupe-aware, and this will greatly accelerate overall performance. Windows 8 also includes a new “express” library that makes compression “20 times faster”. Compressed files are not re-compressed based on filetype, so zip files, Office 2007+ files, etc will be skipped and just deduped.

New writes are not deduped – this is a post-process technology. The data deduplication service can be scheduled or can run in “background mode” and wait for idle time. Therefore, I/O impact is between “none and 2x” depending on type. Opening a file is less than 3% greater I/O and can be faster if it’s cached. Copying a large file can make some difference (e.g. 10 GB VHD) since it adds additional disk seeks, but multiple concurrent copies that share data can actually improve performance.

Stephen’s Stance

Although I am intrigued by Microsoft’s new deduplication technology in Windows 8 server, I still have many questions about its usefulness and impact on performance. Concentrating duplicate data in the system volume makes sense from a technical perspective, but could lead to an I/O hotspot on the disk. This is especially true for external caching storage systems, since there is no integration between Microsoft deduplication and storage array features. I am particularly concerned about the use of deduplication with VHD files in Hyper-V, since it could eat up valuable system RAM and impact I/O performance.

If you would like to try Microsoft deduplication for yourself, I am happy to report that it is included in the developer preview of Windows 8 that is available on Dev Center. Here are a few commands to get you started, and read Rick Vanover’s post too!

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication
Import-Module Deduplication
Enable-DedupVolume E:
get-dedupvolume

Note: I am a Microsoft MVP and Microsoft briefs me on upcoming technologies under NDA. This post is based on a Microsoft briefing from November which was said at the time not to be covered by any NDA. All of this information could be gleaned by experimenting with the Windows 8 developer preview, but it’s much easier to just go to the source.

© Stephen Foskett for Gestalt IT, 2012. | Microsoft Adds Data Deduplication to NTFS in Windows 8
Read more posts categorized as Server Virtualization, Storage

It isn’t always easy to get where you need to go!

Consider the following situation: You go to lunch with your good friends, John and Mary. Halfway through a rousing discussion of the latest Hollywood movie, Mary starts talking about the fantastic action sequences while John criticizes the romantic angle. You realize something mine-bending has happened: John now has Mary’s personality, and vice versa. It’s like they have switched brains or something!

This truly weird situation isn’t likely to happen in person, but occurs all the time in the data center. Virtualization of server, network, and storage services illuminates the link between physical resources and functional applications. A running virtual machine can instantly move from one server, network adapter, HBA, or LUN to another. And when it happens, traditional components have no idea how to react.

The Challenges of Mobility

Mobility is perhaps the “killer app” of virtualization, but it is also the killer of traditional IT systems. Let’s consider the challenges of this “Twilight Zone” moment.

• The operating system expects a consistent hardware environment, which is exactly what the hypervisor creates
• The LAN must be prepared to redirect all network traffic instantly and seamlessly to one or more new physical interfaces
• The SAN similarly must be able to reroute all I/O to a new pair of HBA’s without missing a beat
• The storage array must be able to re-present capacity to a new physical device, and must maintain snapshots and other configurations
• The backup system must also be able to maintain consistency over time even as machines relocate to different server and storage locations

All of this must be done while maintaining quality of service (QoS), access control, reporting, and appropriate segmentation at all levels. This is an incredibly challenging task, and no conventional protocol (IP, Ethernet, NFS, SCSI, Fibre Channel, etc.) is anymore ready then you are when you’re good friends switch personalities.

Two Paths

So much of the development that is currently taking place in IT focuses on accommodating this “mobility issue”. Two key approaches have emerged to take on this challenge:

• “In a vacuum” technologies (like VXLAN) assume that no other changes will be made, so the focus is on maintaining complete compatibility in front and behind
• “Clean sheet” technologies (usually from startups) take a different approach, throwing out compatibility in favor of technical elegance

Both of these approaches have merit. Attempting to maintain compatibility only works so far (just ask a Windows API programmer), but it leverages the existing environment and recognizes that most people are not ready for wholesale change. Clean sheet designs always make more sense, but they rarely attain mass acceptance. Nearly every technology we rely on today is full of bolt-ons in the name of compatibility. Some, like Ethernet and x86, actually work pretty well, too.

The Stack of Lies

The difference between virtualization and cloud computing is exactly this same distinction. Hypervisors, NPV, NAT, thin provisioning, and so many other virtualization technologies exist mainly to maintain compatibility in a vacuum. In contrast, true cloud computing dispenses with the entire stack and creates a new platform for applications.

This is, perhaps, the reason that cloud computing is not taken off in the enterprise. Simply put, IT is not prepared to ditch everything they have ever used even in the face of a demonstrably superior alternative. Currently, the highest use of cloud is behind gateways and virtualization engines that bring it back down to earth.

This brings us to the stack of lies called server virtualization. Any “modern” virtualized data center is built on lie after lie, with each level telling the other what it wants to hear. The volume manager lies to the operating system, hypervisor lies to the volume manager, and the storage array lies to the hypervisor. The same sad state of affairs allows networking and even memory to function in a virtual world.

But these shaky stacks of lies have difficulty adapting to motion, since no level truly “knows” the reality of the world around. The depressing truth is that a bowl of spaghetti like VXLAN is perhaps the highest form of art we can expect in a virtual data center.

Stephen’s Stance

As a techie, I am always drawn to clean sheet designs that offer technical elegance along with functionality. But I know that, realistically, products that assume nothing about the world around them and bend over backward to maintain compatibility are more likely to succeed. Still, I maintain hope that the issues of virtual machine mobility will be solved in an elegant way, rather than adding to the “stack of lies”.

© Stephen Foskett for Gestalt IT, 2012. | The Terrifying True Story Of Virtual Machine Mobility
Read more posts categorized as Server Virtualization, Storage

I have been waiting for some time to get some ESXi 5.0 awesomeness going on in my environment. vCenter has been sitting on v5 for some time and I have been deploying ESXi 5 in a couple stand-alone situations without any issues. So, now that I have more compute capacity in the data center, it is time to start rolling the remaining hosts to ESXi 5… or so I thought!

I downloaded ESXi 5.0.0 Kernel 469512 a while back and have been using that on my deployments. So far, so good… until today. Update Manager configured with a baseline –> Attach –> Scan –> Remediate –> back to business. Surely, Update Manager processes should take more time than the actual upgrade. About 30 minutes after starting the process, vCenter was showing that the remediation progress was a mere 22% complete and the host was unavailable. I used my RSA (IBM’s version of HP ILO or Dell DRAC) to connect to the console. Sure enough, it was stuck at loading some kernel modules. About 20 minutes later IT WAS STILL THERE!

Restarting the host did not resolve the issue. During the ESXi 5 load screen, pressing Alt + F12 loads the kernel messages. It turns out that iSCSI was having issues loading the datastores in an acceptable amount of time. I was seeing messages similar to:

A little research turned me onto the following knowledgebase article in VMware’s KB: ESXi 5.x boot delays when configured for Software iSCSI (KB2007108)

To quote:

This issue occurs because ESXi 5.0 attempts to connect to all configured or known targets from all configured software iSCSI portals. If a connection fails, ESXi 5.0 retries the connection 9 times. This can lead to a lengthy iSCSI discovery process, which increases the amount of time it takes to boot an ESXi 5.0 host.

So, I have 13 iSCSI stores on that specific host and multiple iSCSI VMkernel Ports (5). So, calling the iSCSI lengthy is quite the understatement.

The knowledgebase states that the resolution is applying ESXi 5.0 Express Patch 01. Fine. I can do that. And… there is a work around described in the article that states you can reduce the number of targets and network portals. I guess that is a workaround… after you have already dealt with the issue and the ridiculously long boot.

Finally, to help mitigate the issue going forward, VMware has released a new .ISO to download that includes the patch. However, this is currently available in parallel with the buggy .ISO ON THE SAME PAGE! Seriously. Get this… the only way to determine which one to download is:

As a virtualization admin, I know that I am using the Software iSCSI initiator in ESXi. But, why should that even matter at all?! There is a serious flaw in the boot process in version 469512  and that should be taken offline. Just because someone is not using Software iSCSI at the current time does not mean they are not going to in the future. So, if they download the faulty .ISO, they are hosed in the future. Sounds pretty crummy to me!

My Reaction

I am quite shocked that this made it out of the Q/A process at VMware in the first place. My environment is far from complex and I expect that my usage of the ESXi 5.0 hypervisor would be within any standard testing procedure. I try to keep my environment as vanilla as possible and as close to best practices as possible. 1.5 hours for a boot definitely should have been caught before release to the general public.

Additionally, providing the option to download the faulty ISO and the fixed ISO is a complete FAIL! As mentioned on the download page, this is a special circumstance due to the nature of the issue. I would expect that if this issue is as serious as the download page makes it out to be, the faulty ISO should no longer be available. There has to be a better way!

Conclusion

I have since patched the faulty ESXi 5.0 host to the latest/safest version, 504890, and boot times are back to acceptable. I will proceed with the remainder of the upgrades using the new .ISO and have deleted all references to the old version from my environment.

I have never run into an issue like this with a VMware product in my environment and I still have all the confidence in the world that VMware products are excellent. In the scheme of things, this is a bump in the road.

© Bill for Gestalt IT, 2011. | ESXi 5.0–1.5 Hour Boot Time During Upgrade
Read more posts categorized as Server Virtualization

The great battle of enterprise storage is on!

The time has come to take sides on the core question of storage for virtual servers: Do you want storage intelligence to live in the hypervisor or the array? Most administrators are already lining up on one side or the other, unintentionally casting their vote while the rest flounder. But the storage industry must wake up and embrace the divide.

Hypervisor Huggers Unite!

Do you “heart” virtualization?

VMware’s vSphere dominates the world of enterprise server virtualization and has become the core element of the modern open systems datacenter. Microsoft recognizes this but has been unable to field a competitive hypervisor ecosystem for the virtual datacenter. Today, vSphere is the state of the art and nowhere is this more apparent than in storage.

In just a few years, VMware has delivered and updated a host of advanced storage functionality, from provisioning to migration and load balancing to backup and data protection. vSphere 5 includes an advanced and scalable storage virtualization layer, delivering everything a datacenter needs. VMFS sculpts basic block storage into a shared resource for virtual machines, with snapshots, policy-based layout and movement, and flexible allocation and thin provisioning.

Most VMware administrators are “server guys” and relish these features. They have never experienced an automated “storage service” like this, and the enterprise storage world has never been able to construct anything remotely as flexible, user-friendly, and functional. And Hypervisor Huggers don’t need complex enterprise storage arrays to do it: They can use basic iSCSI or Fibre Channel devices to provide performance and capacity and let VMware do the rest!

Storage DRS is exemplary of the new virtual datacenter world. Introduced in vSphere 5 (and restricted to the pricey Enterprise Plus license), Storage DRS uses the core technology of Storage vMotion to dynamically balance I/O and capacity across a diverse pool of storage. Storage DRS even uses Policy-Driven Storage and VASA to enforce tiered storage and data placement strategy. This kind of virtualization has been a “holy grail” quest for the enterprise storage industry, but they’ve never delivered on their promises.

Cheers for Storage Stalwarts!

Do you wish all this virtualization nonsense would just go away?

But not every IT environment wants be 100% vSphere focused, and many aren’t convinced that dumb storage is the smartest place for their data. These Storage Stalwarts want smarter and better-integrated storage arrays, and VMware is innovating here as well.

VMware’s Storage APIs for Array Integration (VAAI) technology is squarely aimed at this market. VAAI allows vSphere to hand off heavy storage operations to the high-end storage arrays from the major players. It works transparently, too, coordinating cloning without the kind of scripting and hair-pulling that used to require. VAAI in vSphere 4.1 also includes block zeroing support and something called “atomic test and set” which we’ll get to in a moment. Microsoft announced their own cloning integration, ODX, but it won’t ship until Windows Server 8 appears sometime next year.

But cloning is just the tip of the iceberg when it comes to Hypervisor-to-array integration. The rising army of NAS users have seen the glory of more-complete array integration for a while, and they’re not quiet about it. They love that VMware’s NFS protocol support makes storage “disappear” in vCenter, becoming just another resource with integrated thin provisioning and flexible allocation and movement.

VMware is moving aggressively to please their Storage Stalwarts, adding more VAAI support for block and file in vSphere 5. But, as the company laid out at VMworld 2011, neither access method is ideal for virtual servers. So VMware has been pushing the enterprise array vendors for ever-greater integration. They see a future where a VAAI-based protocol enables arrays to de-multiplex I/O streams from the hypervisor and intelligently handle per-VM data.

Stephen’s Stance

You can spot a Hypervisor Hugger by their big LUNs: They would rather treat storage as a bulk commodity, and array vendors should be lining up to get their business. Storage Stalwarts will jump on each new VMware innovation, finally making use of the capabilities they have spent over a decade paying for but not utilizing. The only untenable stance is trying to keep a foot in both worlds: It’s foolish to buy an enterprise array and use it as bulk storage!

© Stephen Foskett for Gestalt IT, 2011. | Are You a Hypervisor Hugger or a Storage Stalwart?
Read more posts categorized as Featured, Server Virtualization, Storage

The PXE environment components included with the VCVA are not configured and turned off by default. So, if you’re ready to configure your VCVA for PXE, time to roll up your sleeves, crack those knuckles, and get ready to get your hands dirty.

Before we get started, though, and little caution (and disclaimer so I can sleep better at night):

I know nothing about your environment. You are following these instructions at your own risk. This setup will impact DHCP functionality on your network. Follow these instructions at your own risk and make the appropriate adjustments to work in your environment.
Additionally, I do not know everything about everything. So, you are going to need to rely upon your sleuthing abilities to help resolve issues that may arise.

These instructions assume some knowledge of CLI-based file editing (vi). So, please research how to use it if you are unsure.

Overview

A PXE environment via the VCVA requires the following components in your environment

• DHCP server
• TFTP server
• Web Server (for kickstart scripts)
• SYSLINUX (for pxeboot.0)
• Access to an ESXi 5.0 installation CD (perhaps you created on using my Image Builder tutorial)
• vCenter Virtual Appliance deployed
• Blank server to PXE boot and install ESXi 5.0 on (aka – the client)
• ESXi 5.0 installation .ISO
• HTTP server on the network (for hosting kickstart files – customization during installation)

For this exercise:

• Network: 192.168.226.0/24
• VCVA: 192.168.226.21
• DHCP Range: 192.168.226.200 – 254
• Default Gateway: 192.168.226.1

Configuration

0 – Log into the appliance as ‘root’

1 – Configure DHCP

‘dhcpd‘ will listen to IP address requests, provide an IP to use, direct the client to the “next-server” to continue PXE booting, and which file (filename) to download from the server.

• cd /var/lib/dhcp/etc
• cp -a dhcpd.conf dhcpd.conf.orig
• vi dhcpd.conf

Once inside of the file, ensure the following exists (highlighted for your ease of identification)

ddns-update-style ad-hoc;
allow booting;
allow bootp;

#gPXE options
option space gpxe;
option gpxe-encap-opts code 175 = encapsulate gpxe;
option gpxe.bus-id code 177 = string;
class “pxeclients”{
match if substring(option vendor-class-identifier, 0, 9) = “PXEClient”;
next-server 192.168.226.21;
filename “pxelinux.0″;
}
subnet 192.168.226.0 netmask 255.255.255.0 {
range 192.168.226.200 192.168.226.254;
}

Save the file and exit (hint: :wq)

2 – Configure TFTP

TFTP services are provided by the ‘atftpd’ daemon

• cd /etc/sysconfig
• cp –a atftpd atftpd.orig
• vi atftpd

Once inside the file adjust the “ATFTP_OPTIONS” line to read: “–daemon –user root”. Typically, the atftpd daemon runs as ‘nobody’. However, the TFTP root (/tftpboot/) is configured as owned by the ‘root’ user.

Save and exit the file.

3 – Get the SYSLINUX packages on the server

There is one package missing to make the PXE installation process work: ‘pxelinux.0′. ‘pxelinux.0‘ is an executable that is downloaded by the client in order to properly continue the PXE process (aka – download the files, execute the installer, etc…). ‘pxelinux.0‘ is provided by the SYSLINUX package. In order for PXE to work properly with the ESXi 5.0 installation, SYSLINUX version 3.86 (or higher) is needed.

• cd /tmp
• wget http://www.kernel.org/pub/linux/utils/boot/syslinux/3.xx/syslinux-3.86.tar.gz
• gunzip syslinux-3.86.tar.gz
• tar xvf syslinux-3.86.tar

Note: you can use YUM or copy the files to the server another way if you’d like. Regardless, get the files there. This example will continue to use the /tmp file as the landing area for the SYSLINUX files.

Copy the pxelinux.0 file to your TFTP root

• cp /tmp/syslinux-3.86/core/pxelinux.0 /tftpboot

4 – Prep the TFTP root for PXE

The TFTP root configured on the VCVA is located at /tftpboot. We are going to need to get the directory structure built out to support PXE.

• cd /tftpboot
• mkdir esxi50

By adding a directory, we are able to organize the TFTP server and support additional versions of ESXi going forward.

5 – Get the ESXi 5.0 CD contents onto the server

Seeing as the VCVA is a virtual appliance, it is easy to get the contents of the installation media onto the server.

• Mount the installation CD to the VCVA as a CD-ROM drive using the vSphere Client.
• mount /dev/cdrom /media
• cp –a /media* /tftpboot/esxi50/
• umount /dev/cdrom

6 – Configure PXELINUX

pxelinux is the utility that enables the PXE functionality. As mentioned before, pxelinux.0 is an executable that the server downloads. The executable provides functionality to parse a menu system, load kernels, options, customizations, modules, etc…, and boot the server. Since PXE can be used by multiple physical servers for multiple images, we need to configure pxelinux for this specific image.

• cd /tftpboot
• mkdir pxelinux.cfg
• cd pxelinux.cfg

pxelinux.0 looks for configuration files in the TFTP:/pxelinux.cfg directory.

pxelinux looks for a large number of configuration files… specific to a default/generic value. This allows server administrators to define a file based on a complete MAC address, partial MAC address, or none at all to determine which image to boot from. Since this is the first configuration on the VCVA, we are going to configure a default. Do your research if you want to adjust this from the default value.

The installation media contains a file called isolinux.cfg. We can use this as the basis for our file called ‘default’. Copy it from the installation media and start customizations:

• cp –a /tftpboot/esxi50/isolinux.cfg default
• chmod a+w default
• vi default

Ensure the appropriate lines match the following lines:

DEFAULT /esxi50/menu.c32
KERNEL /esxi50/mboot.cfg
APPEND -c /esxi50/boot.cfg

Save and Exit

7 – Configure the Kickstart file

Using a kickstart file, we can configure ESXi 5.0 automatically during installation. This requires that a file be placed on a server that is available to the client.  Sadly, the HTTP areas on the VCVA are not readily available… and, they may be erased during future upgrades. So, we need to use an external HTTP server somewhere on your network. (Note: NFS and FTP are options as well).

Add the following contents:

# Accept the EULA
vmaccepteula

#Set root password
rootpw supersecretpassword

#Install on first local disk
install –firstdisk –overwritevmfs

#Config initial network settings
network –bootproto=dhcp –device=vmnic0

 

In this example, we are saving the file to:

• HTTP Server: 192.168.226.1
• Directory: ks
• File: esxi50.txt
• URL: http://192.168.226.1/ks/esxi50.txt

8 – Configure the installation files

The CD installation media for ESXi 5.0 assumes a single installation point. Thus, all the files are placed at the root of the image. However, since we want to actually organize our installation root, we added the ‘/tftpboot/esxi50‘ directory and copied the files into it. We need to adjust the installation files in /tftpboot/esxi50 to reflect the change.

• cd /tftpboot/esxi50
• cp -a boot.cfg boot.cfg.orig
• vi boot.cfg
• Using the following picture as reference, add “/esxi50” to the paths for ‘kernel’ and ‘modules’

Save and quit

9 – Restart services to load the service configurations and configure to start with server

• /etc/init.d/dhcpd restart
• /etc/init.d/atftpd restart
• chkconfig –add dhcpd
• chkconfig –add atftpd

10 – Take a break

You made it this far… great job. At this time, we have configured DHCP, TFTP, pxelinux, copied installation media to the TFTP root, and configured the installation for our organizational purposes.

11 – Start your host and install away

[BELOW] Reading the Kickstart Script. No need to enter customization info anymore.

[BELOW] Checking contents of Kickstart file. You will see errors here if errors in file.

© Bill for Gestalt IT, 2011. | vSphere 5–PXE Installation Using vCenter Virtual Appliance
Read more posts categorized as All, Featured, Server Virtualization, Top Story

Sure they seem like small things, but over time, these small things like change in the couch cushions can add up to a big ticket problem.  For me, the issue wasn’t all that bad, but it did take some head scratching (outside the scope of the actual issue) and a brief conversation with someone wiser than I about the symptoms of my issue.

We don’t trust you anymore, go away

Windows 7 is a rather finicky OS (moreso that Windows XP, and probably a bit less so than the OS between XP and 7).  Because computers are still objects within Active Directory that access other secured resources within the directory, they too authenticate.  In reality, this means that computers have accounts equivalent to User objects within the AD environment. These accounts allow computers to tell Active Directory that they belong within the environment and should be allowed to access resources.  Just like when I logon to the domain and request access to resources by providing credentials, computers in the environment do the same.

If for some reason, the Domain Controller cannot match the credentials presented by the computer to what is stored in its database, the Domain Controller refuses authentication and presents a message about trust relationships.

I didn’t create credentials for the computer, what the heck do I do now?

When a computer is added to an Active Directory domain its account is established and the password set.  Then the password is managed by the computer and AD and changed automatically about every 30 days or so.  If the computer is no longer trusted by the domain, it is likely that the password is incorrect or has gotten lost in translation causing authentication to fail.

My issue was a replication issue which caused the computer accounts of a few workstations to fail authentication.  Because it is not the best idea to maintain only one domain controller in any Active Directory environment, and because of the way that AD manages information about objects, replication happens.

Perhaps an example will work here.  Suppose I create a user object for John Smith using Active Directory Users and Computers (ADUC) on a Domain Controller named creatively DC1 at my office.  John will be starting his new career as a data entry specialist in my company’s Houston office in a week or so.  Adding the user account for John to a DC in my office works just as well as if I had flown to Houston (or remoted into the DC there) and added the account.  Because replication sends all objects created, maintained, or deleted to all other replication partners within the domain, a user account created in my office on DC1 can be replicated to Houston on DC2 and when John gets to work, he can logon and all is well.

Replication happens in the background and is pretty much out of site when things are going smoothly, but from experience I can tell you that you should check in on your friend replication regularly.  Maybe not daily, but weekly for sure.  Just to make sure that objects in the directory are being moved around without errors.

What might cause replication problems?

There are any number of settings and configurations that can cause problems with replication.  Surely more than I have seen or have time to list here, but some of the basic things are:

• Improperly configured links
• Unmanaged Replication configurations
• Misconfigured Firewalls
• Equipment failure

Improperly configured links

When you establish replication between two (or more) Active Directory domain controllers, you create links between them that allow these DCs to exchange information.  The links are one way which means that each domain controller has two links to each replication partner.  The links can be configured to handle high speed links (fast connections, like you might see between domain controllers in the same site) and slow links (which may be used to link two remote locations).  When the links are configured correctly things work really well, but if you neglect to consider the speed of your Internet connection (on both ends) replication may suffer as a result.

Replicating information across a slow link that is configured to behave like a fast one might be a little less dire to watch than downloading a blu-ray quality video over a dial up connection, but missing information can have rather large repercussions in your environment which may be seen as inability to login, latent access or no access to resources and other things.

Unmanaged replication configurations

By this I am not suggesting that you check on replication statuses every day (depending of course on the size of your environment) but you should be looking at it regularly enough to know what is going on and that replications in all directions are happening as you need them to.

Because Active Directory is a multi-master beast, meaning that any machine configured as a domain controller carries just as much weight as any other machine configured as a domain controller, information for an object that has not yet replicated throughout the environment could be a problem.  As in my earlier example, if I created the user object for John Smith, and it failed to replicate to the domain controller in Houston by the time he needed to log in, we might have a problem.

The login would likely happen, but would take a significant amount of time because the most local domain controller didn’t have the information needed to handle the request.

Misconfigured Firewalls (and other Network issues)

Windows includes a firewall to help keep things out of your environment that shouldnt be there.  I would recommend disabling the firewall on all your Windows computers and servers because it will likely be a bigger headache than you are ready for.  Also because all organizations should use dedicated firewalls to protect their corporate assets from the outside world.

My issue with replication came at the hands of a misconfigured firewall.  The firewall was enabled for a good period of time which caused hiccups in the replication of information throughout my Active Directory environment. The symptoms displayed were the previously mentioned domain trust errors that popped up when logging on or trying to unlock a PC.

In my research and previous experience the best fix for the trust problem is to disjoin the affected system from the domain and delete the computer account from Active Directory.  Then rejoin the system to AD.  Normally this will take care of the symptom.  Not necessarily the problem.

Outages and Equipment Failures

There is the obvious replication issue with failures and downed equipment.  If the replication is scheduled to occur between two systems and one of those systems is down, obviously replication cannot happen.

Working on these issues is an interesting scenario as well.  For the sake of troubleshooting, the usual steps must be followed and checked out even if the steps do not solve the problem, they will likely help you down the path to correcting the problem.

The moral?

Do not be afraid to check out the functionality of your Active Directory environment, being proactive and working to pay attention to things like replication and group policy settings.  Keeping up with those tasks before the problem strikes and requires many late nights to correct.  You will still have some long nights working with Active Directory, but they can be worth it, without all the fires.

© derek for Gestalt IT, 2011. | Trust. It is an interesting thing in Active Directory.
Read more posts categorized as All, Desktop, Server Virtualization

All of us “Virtualization Admins” are always looking for more information about the performance of our environment. As VMware ESX products have progressed, more and more performance metrics have been gathered and presented to us… and we appreciate it oh so much.

However, one of the largest issues we need to contend with is tracking of this information longer term and being able to correlate the collected information to new issues and/or diagnose future capacity issues in the future.

Previously, we needed to invest in 3rd party company products to provide this functionality. In some cases, the information was difficult to gather correctly, provided very static alerting thresholds, and was not able to integrate well into existing virtual infrastructure.

Back on August 31, 2010, VMware announced the acquisition of Integrien… an up and coming product providing real time performance analytics for all kinds of environments… including virtual environments. Acquisitions are always so curious because we, the consumers, are always trying to figure out how it is going to be used to further the product line of the purchasing company. So, while the product from Integrien was interesting, seeing how it would fit into the vSphere realm as a VMware product was up in the air.

All of those questions are answered today, as VMware has announced the availability of the vCenter Operations products. vCenter Operations is being provided in three flavors:

• Standard
  • Handles vSphere environments.
  • Deployed as a virtual appliance that hooks into vCenter and is visible as a vSphere Client Plugin
  • Handles up to 500 VMs
• Advanced
  • Same as Standard edition
  • Includes additional Capacity Planning (aka – Standard Edition bundled with CapacityIQ)
• Enterprise
  • This is a whole new beast and includes the ability to monitor much more infrastructure than just virtual hosts and servers.

The product information from VMware will list off all sorts of neat features, functions, and purpose for their product. However, after being able to use the product during the pre-release period, I find that the following are the main value points for my environment:

1) 10,000’ view of Virtual Environment

All kinds of monitoring solutions exist that claim to be the single pane of glass that should be able to solve all of your problems. However, it appears as though this is the first to actually accomplish that.

Check out what I see:

Just from a glance, I can see that my environment is running well. The overview page uses the typical Green/Yellow/Red color scheme to indicate state. From this view, I can immediately see that my vCenter, Datacenters, Clusters, and ESX hosts are running within established parameters. I see there may be some problems with a handful of VMs. Instantly, I can see what is going on.

Without going into a complete demo, I can click on any object in the page and get relationship information (aka – which vCenter, which Datacenter, which Cluster, which ESX, and which VMs).

2) Defining what is normal for your environment

Determining what is normal is one of the most important aspects of what the vCenter Operations Standard product offers. What is normal to me is not normal for everyone else… we are all special in our own way and vCenter Operations Standard understands that.

When vCenter Operations Standard is installed and configured, you do not need to do much of anything. The installation guide is dead simple. Why is this?! Well, because vCenter Operations Standard learns about the behavior of your environment.

Rather than rely upon static definitions that would cause warning and critical alerts (example: Warn when RAM = 80%+ and Critical when RAM = 95%+), vCenter Operations watches what happens in your environment from day 1 and starts to determine what is normal for you. If you configure your applications to utilize almost all of the RAM assigned, the other solutions may alert you to a critical state. However, this is normal and expected. You would never expect to see an alert for when the server RAM utilization drops to 10%. However, by using dynamic thresholds in the learning and monitoring algorithms, vCenter Operations Standard is able to determine that high RAM usage is normal and alert you to when the situation is NOT normal… so, when the server drops to 10% RAM usage, you will get alerted because it is abnormal.

vCenter Operations relies on some crazy algorithms and analysis that any PhD in Rocket Science would love. Various algorithms exist in the environment that chew on the data as it is received. The results of the algorithms are selected based on most likely to be correct and used to represent the data in some fashion. So, there is a higher probability of the data and situation being statistically correct.

Check out what I see:

This is a view of a specific ESX host in my environment. You can see that normal is somewhere between 1-16 and is typically defined by the Memory usage. Additionally, you can see various statistics regarding the current workload, CPU, Memory, and ESX resources. Again, all from a single screen.

Now, compare that to a different ESX host that is hit a little harder:

Normal for this server is somewhere between 61-100 and is defined by the Memory usage.

3) Resource statistic aggregation to provide a more holistic view of what my environment is like on a historical basis.

This little nugget of joy made my day the first time I saw it.

vCenter Operations Standard is able to aggregate many statistics into a single value for you to see that represents your resource in the environment. So, rather than use resxtop to find NIC statistics in my environment, I can get those statistics from vCenter Operations Standard in a easy to read way.

For example: There have been many times where I wanted to get a good idea of how much data is passing through my NICs. Previously, I would need to get some batch data from resxtop, throw it into a spreadsheet, and process it. Or rely on some historical data from vCenter. Now, I can dig into the ESX host in vCenter Operations Standard and select the “ESX USED NETWORK INTERFACES” section at the bottom of the page.

Clicking on one of the number shows me even more information. This is the graph for Received Rate (KBps):

How cool is that?!

Similar statistics can be gathered for CPU, Memory, and Storage as well!

4) Analytics and Fires

Analytics are presented as hot zones and sized based on relationship to others. So, just looking at the graphic, you can get a sense of the relationship between other objects. I know, that is a little obscure. But, check this out:

From here, you can see that there is a fair amount of contention for the Exchange partition compared to that of the other datastores in the environment.

Having access to this data is still new to me and I am continually finding more and more ways to interpret it.

The vCenter Operations Standard product provides amazing insight into the your virtual environment. Whomever at VMware decided that Integrien was a suitable acquisition should get a high-five for this one.

No doubt, this is an insanely useful product and is definitely raising the bar in what is possible in the analytical world. I look forward to seeing this product grow and mature.

© Bill for Gestalt IT, 2011. | VMware vCenter Operations–Standard Edition
Read more posts categorized as Server Virtualization

Without reciting everything in the post, the single sentence that has encapsulated my feeling about their technology and approach was:

The biggest problem with the technology and the company direction is that there is no clear use case for this.

Until recently, I stood behind this statement. And, honestly, as it pertains to the Aprius approach I was presented, I still do. I am sure there are Aprius, Xsigo, Virtensys, and any other I/O Virtualization vendors/customers that dispute the statement. (note: I welcome comments!).

However, I believe I have found a great use case for I/O virtualization thanks to a presentation from Virtensys during the most recent Portland VMUG meeting.

The use case that was presented was huge and I see it being beneficial to all sort of environments (SMB, SME, Enterprise, Healthcare, Education, Research, etc…).

The Virtensys product utilizes PCIe extension cards in the PCIe slots on servers. Those cards connect to the the Virtensys product. For this example, we will assume the Virtensys solution is configured to share 10Gb NICs with the servers. If your Virtualization servers are sharing the 10Gb NIC through the Virtensys product, all network traffic is routed through the Virtensys solution. However, if the virtual machine on one server is trying to communicate to another virtual machine on another server and those servers are sharing the same NICs in Virtensys’ solution, the communications happen at PCIe speed, not NIC speed! Additionally, the traffic never hits the standard physical network layer.

The PCIe 2.0 standard allows for 500MB/s over a single lane (minus some overhead). So, a single lane can handle roughly 4Gb/s. A full 32-lane connection can handle 16GB/s (or 128Gb/s. Now, these are all technical values and some level of overhead and contention may need to be accounted for. But, the value from here is that PCIe bus is quicker than the 10Gb NIC that is being shared.

Use Case!!!: By utilizing the Virtensys solution, your network traffic is no longer hitting the physical network and can be transmitted at PCIe speeds!

I will be the first to admit that I am not entirely keen on the other offering that exist (again, I like comments!). I would like to think that this same use-case exists for the other I/O virtualization vendors out there. Assuming they do, I can see the I/O virtualization products being adopted by companies that can benefit from higher network throughput that is allowed. Assuming this is specific to Vrirtensys and you have needs for higher network throughput, you may want to check these guys out.

Disclaimer:

Virtensys was a presenter at the Portland VMUG meeting in May 2011 which I was the principle organizer. I am under no obligation to include them in any personal blogging I undertake (which this qualifies as). Virtensys provided the presentation that opened my eyes to the use case supplied above.

Resources:

• PCIe Spec Speeds – Wikipedia – PCI Express Architecture

© Bill for Gestalt IT, 2011. | I/O Virtualization Redux (c/o Virtensys)
Read more posts categorized as Server Virtualization