I’ve had my DroboPro for some time now and initially I had mixed results with it.  Although I was aware of the single connection to an iSCSI target restriction, I was slightly disappointed with the lack of vmfs support within the Drobo operating system, especially as the product sells itself on understanding the filesystem format.  My next move has been to test the DroboPro with Hyper-V and guess what?  Thin Provisioning is back on the agenda; Hyper-V uses NTFS to store hypervisor guests, making it perfect for use with DroboPro.

The first screenshot shows my DroboPro configuration.  It’s currently up to 7 drives and 9.4TB; although I’m only using a fraction of that space at the moment while I do some testing (the majority of disks in the unit were also purchased for testing).  I’ve created seven 2TB drives (more than the Drobo capacity) and connected them to my Hyper-V server.  From there, I’ve formatted four of the drives and you can see those as drive letters H: through to K: on the second screenshot.  Two of the volumes are hosting Hyper-V guests occupying around 19GB of storage in total.  This is directly reflected by the capacity of space used on the DroboPro and the two C:\ drives, one of which is shown in the third screenshot.

The DroboPro is ‘thinning’ the thin provisioned Hyper-V virtual disks, enabling me to in effect scale my Hyper-V storage requirements to the absolute minimum I need to, even if the virtual disks for the operating systems are bigger (in this instance 126GB per virtual drive).  The next step is to extend the storage usage on the virtual guests and see what happens.
My first attempt is to use an old favourite, fsutil and create a few large empty files.  Unfortunately this strategy doesn’t work; the NTFS file system on the Hyper-V guest seems to detect that this is a sparse file and even though the virtual file system shows the storage as being allocated, this isn’t passed back and allocated on the Hyper-V server.  This seems to indicate that Hyper-V itself is truly filesystem and thin provisioning aware.

My second attempt consisted of finding a large file and copying it repeatedly.  This process was more successful and the results can be seen in the 6th screenshot; I added around 9GB of additional files, pushing up the DroboPro capacity and the space stored in the VHD snapshot by equal amounts.

I then deleted my snapshots, effectively merging them into the main VHD file.  As the merge took place, the base VHD file increased, taking more overall capacity on the DroboPro.  After the merge had completed and the Hyper-V guest restarted, the capacity used returned to the total in actual use; the DroboPro reclaimed the released storage.

Summary

What have I learned?  Without any additional configuration, the DroboPro is thin provisioning aware with Hyper-V.  This is “thin on thin” as the Hypervisor and the DroboPro both implement thin provisioning at their own level.  Snapshots need some consideration, especially when merging into the main virtual disk (VHD).  Thought also needs to be given to the best way to lay out the Hyper-V storage; bear in mind that to get the thin provisioning benefits, each DroboPro drive must be configured with a single NTFS volume – spanned and striped volumes will not work.

I’d say the DroboPro was good to go with Hyper-V.  All we need now is vmfs support!

© Chris for Gestalt IT, 2010. | DroboPro – Hyper-V’s Best Friend!
Read more posts categorized as Featured, Server Virtualization, Storage

The Gestalt IT Field Day was a great success in bringing together a mixture of delegates from varying discplines. Following the presentations from 3Par and Symantec, there was heated debate about the implementation of Thin Provisioning and the ability to reclaim released storage resources. This post covers the basic concepts of Thin Provisioning and more importantly how deleted resources can be recovered over time.

Thin Provisioning Primer

The underlying concept of thin provisioning is pretty simple; provide storage resources to those requesting it only as they need it.

Think of a standard ‘thick provisioned’ environment.  As thick LUNs are created, the storage is assigned and mapped to that LUN to the full extent of the size requested.  See, the first graphic, which shows a RAID group of four 5GB drives.  I’ve assumed “RAID-0″ here for simplification, i.e. no RAID overhead.  Each LUN (coloured separately) is made up from a 1GB slice of the available disks.  Thick provisioning is great if the LUNs are all 100% allocated.  In that instance, 100% of the available physical space is used up.  However, it is never the case that 100% of a LUN is used and so wastage exists. 

Look at the second graphic.  This shows how thin provisioned LUNs work.  As storage is requested by the LUN, the space is mapped to physical blocks of storage.  In this example, none of the logical LUNs are fully utilised and so don’t consume their full theoretical capacity.  This means that the pool of space can be over-subscribed and a sixth new LUN created.  Obviously there’s no such thing as a free lunch or infinite storage resources and in this example if a further five blocks are requested then physical space would be exhausted.  The next request for a new storage block would result in an error situation and this represents the main concern with over-subscribing thin provisioned volumes.

Now we get the concept of thin provisioning, there are a further two aspects to consider.  Firstly, when we say a LUN isn’t 100% utilised, what to we mean?  Second, how can deleted blocks be returned to our free physical pool?

As LUNs are presented to hosts, they are formatted with a file system, for example on Windows it’s NTFS; a VMware environment would use VMFS.  The file system will have a standard layout which determines where the file index sits and the method in which files are allocated onto the disk.  Have a look at the third graphic.  This is a map of the C:\ drive for one of my servers.  Each block represents approximately 22MB.  You can clearly see the MFT (NTFS index) in the centre of the volume.  A large percentage of the disk is unused.  In a thin provisioned environment, storage would have been requested only for the blocks with valid data and in this way, a LUN can be less than 100% allocated. 

OK, so what happens if I create some files then delete them on the file system?  Most file systems remove a file by deleting the entry in the index rather than physically overwriting the file contents with binary zeros.  This is quick and efficient (if not slightly unsafe security wise).  The actual data isn’t overwritten and it is this ‘logical’ deletion that enables undelete utilities to work.  The trouble is, most disk arrays are not file system aware and so can’t detect the logical deletion of a file.  Those arrays that offer thin provisioning typically detect unwanted space by looking for blocks containing only ‘binary zeros’.  This means simply deleting files will not release unused space back to the free block pool (except for one storage device I’ll discuss in more detail another time, that’s Drobo). Arrays which are capable of recovering unused space need to see data overwritten in order to recover it.

This (finally) brings us to the cookie analogy.  Imagine cookies are my free pool blocks.  There are a number of ways in which storage arrays operate in handling thin provisioning – different cookie monster personalities:

• The Greedy Cookie Monster; grabs all the cookies he thinks he might eat, but never eats all of them and never returns any – this is the thick provisioning model.
• The Selfish Cookie Monster; only grabs cookies as he gets hungry but if he doesn’t eat them immediately, doesn’t give them back – this is thin provisioning with no zero block reclaim.  Eventually thin provisioning will become thick as all logical blocks in a LUN become mapped to physical storage.
• The Nice Cookie Monster; takes the cookies as he gets hungry but only returns uneaten cookies if asked – this is thin provisioning with manual zero block reclaim.  A manual process is required to zero out the unused space and to return it to the free pool.
• The Saintly Cookie Monster; takes the cookes as he gest hungry and offers them back immediately he realises he can’t eat them  – this is thin provisioning with automatic zero block or free space reclaim. 

So, of the storage arrays out there offering thin provisioning, which fit the various Cookie Monster personality types?  I’ll leave that for you to guess.

© Chris for Gestalt IT, 2009. | Enterprise Computing: Thin Provisioning and The Cookie Monster!
Read more posts categorized as Server Virtualization, Storage

Windows Storage Server is one of the most interesting products from Redmond. It is a specialized version of Windows Server 2003 R2 with integrated storage target capabilities, including iSCSI, NFS, and of course SMB file services. It also includes single-instance storage (file-level deduplication), distributed file system (DFS), integrated SAN management, file server resource management (FSRM), multi-path I/O (MPIO), and solid software and hardware RAID support. Want to get a copy for yourself? You can’t! Wondering why you haven’t heard much about it? Windows Storage Server is sold only as part of an integrated hardware/software combination available from major OEMs like HP and Dell.

Although Windows Server 2008, with its many storage feature updates, was released last year, the updated version of Windows Storage Server was still under construction until last month. But Windows Storage Server 2008 is available to manufacturers today! Expect to see some new Intel-based storage array announcements in the coming weeks!

For my personal experiences with Windows Storage Server 2008, see my blog post, I Can Finally Talk About Windows Storage Server 2008!

What’s new in Storage Server 2008? Plenty! Most of the features are inherited from Windows Server 2008:

1. Server Message Block (SMB) 2.0 is a re-working of the traditional Windows NAS protocol. Also present in Vista and Server 2008, SMB 2.0 reduces the notorious chattiness of SMB, combining multiple commands into a single packet, as well as allowing more simultaneous open connections and larger buffers as well as durable file handles.  Performance gains using both SMB 2.0 clients and servers has been phenominal with Tom’s Hardware showing a 5x gain in throughput over the Internet!
2. The NFS server (also present in Windows Server 2008) has been updated.
3. A new MMC snap-in called Storage Explorer lets you manage WMI-compliant SAN devices.
4. File Server Resource Manager (FSRM) is a full-featured storage resource management (SRM) application and has been improved greatly for both the Server and Storage Server versions of 2008 with quotas, file screening, and advanced reporting.
5. DFS-R and DFS-N are tweaked – Jose Barreto gets into this on his blog.
6. Smaller or branch offices (and low-end storage array vendors) will be interested in using BitLocker full-volume drive encryption to protect their data.
7. Storage Manager for SANs (SMfS) allows you to perform basic storage array administration tasks within Windows.
8. The Volume Shadow Copy Service (VSS) and DFS interfaces are improved and are now scriptable with PowerShell (through WMI).
9. The new but not yet complete Windows Server Backup system replaces the outdated and limited NTBackup system. Although it includes bare-metal recovery, Server Backup isn’t ready for prime time in my opinion. I’m looking for major improvements in R2!
10. Search 4.0 and 2008 SP2 adds full-text search for files stored on a Storage Server.
11. Server 2008 now automatically aligns filesystem boundaries with storage, which was one of those dark and secret skills us storage guys used to share amongst ourselves.  This can increase performance in high-I/O environments.
12. NTFS (in both Server 2008 and Vista) now has symbolic link support, just like UNIX and Mac OS X, and SMB 2.0 supports symlinks as well.
13. NTFS was also tuned and tweaked a bit for better stability and crash recovery.
14. The updated virtual disk service (VDS) supports LUN shrinking, online/offline and read-only/read-write settings, and SAN policies so new LUNs can be treated as either online, offline, or shared.
15. Failover clustering is simplified, requiring just a few clicks to set up.

Although all of these are also present in the basic Windows Server 2008 install, Windows Storage Server 2008 includes some unique features:

1. The included iSCSI target software is unique
   1. It’s been updated to support failover clustering
   2. It now includes VDS and VSS providers
   3. It’s been ported to StorPort
   4. Installation is now done in the standard Windows manner, with an MSI not an EXE
   5. You can now set up an IPv6-only iSCSI SAN and specify initiators directly using their IPv6 addresses
   6. CHAP secret security is enhanced
2. Single-Instance Storage (SIS) is the second major Storage Server differentiator, providing file-level deduplication
   1. SIS used to be limited to 6 volumes per node, but now scales to 128
   2. SIS now supports clustering, though SIS doesn’t span nodes
   3. A new command allows one to undo single-instancing instead of copying all of a volume’s content to another drive
   4. Enabling SIS is simpler – it’s now just a checkbox per volume in the Share and Storage Management UI
3. Many performance tuning tweaks are standard out of the box, though readers of the Windows Server 2008 Performance Tuning White Paper might be able to perform these on their own. One major standard tweak was removing 8.3 naming and disabling aliasing on filesystem – this led to an an 8% performance gain right out of the box.
4. Remote administration through HTTP is very cool, and ought to be standard on every version of Windows Server! Just point a web browser to the server and you will have an ActiveX or Java-based RDP client without installing any software.
5. Licensing is one more unique aspect. Storage Server does not require client-access licenses (CALs) so any number of clients can access the system without worrying about license management.

All in all, Storage Server 2008 is a solid move forward. I expect that the ability to do single-instance storage at full native speed will be very useful for corporate file servers and similar applications, and the enhancements overall are welcome as well. But this will be the last release of Windows Storage Server as a separate product. From now on, Microsoft simply release OEM storage server software on top of their standard Windows Server versions for OEM’s to use. There will not even be a special Service Pack 2 version of Storage Server 2008. Instead, expect OEMs to provide the regular Server 2008 SP2 as a suggested or required update to Storage Server users.

Microsoft is detailing the new version of Windows Storage Server 2008 in a webcast Thursday at 8 AM Pacific. You should also check out the official Microsoft site, and the Microsoft Storage Server blog, especially their post, A Brief History of Windows Storage Server Releases.

© Stephen Foskett for Gestalt IT, 2009. | Windows Storage Server-Based Systems Step Into 2008
Read more posts categorized as All, Exclusive, Featured, Storage, Top Story

Most physical to virtual migrations (P2V) of servers end up as virtual machines with the partitions encapsulated in virtual disk (.vmdk or .vhd) files. But what if the physical server already has a partition that’s configured through an iSCSI connection to the SAN, and what if that’s the same SAN that the new VM will run on? Of course, the new VM will have to be on a different LUN (formatted for use by the virtualization host), but should you encapsulate the current NTFS iSCSI partition or should you maintain the iSCSI initiator within the resulting VM? The former option depends on how much available SAN space you have to work with, the latter requires some extra thinking before you begin.

When you decide to maintain a server’s existing iSCSI partitions as a VM, there are several configuration considerations to plan for.

Multipathing Support for iSCSI is no longer needed in the VM

When you were configuring the iSCSI initiator chances are you used two physical network interface cards (NICs) for a redundant connection in the server operating system to the storage. You then used the NIC manufacturer’s drivers/management software to create a team and a virtual ip address. Your SAN was configured to allow an iSCSI initiator to connect via that NIC team virtual ip address.

As a VM that same team ip address will probably still be maintained as the initiator, but the need for two NICs and the former manufacturer’s drvers and software will be removed. The VM only needs a single vNIC to the iSCSI storage. The virtual host should be configured with a vSwitch mapped to two pNICs. Therefore the virtual host provides the redundant connection to the storage.

Be sure to remove the team configuration and the old NIC drivers and software.

Dedicate a vSwitch with it’s own pNICs for the VM iSCSI traffic

Separate the VM’s iSCSI traffic from the virtualization host’s iSCSI traffic. You could add an extra portgroup to your iSCSI vSwitch in VMware ESX for example, but ideally, you want 2 NICs dedicated to the host, and 2 other NICs dedicated to the VM(s). This requires separate vSwitches. This will maximize performance and provide redundancy.  

Consider the cables needed to the SAN switches

Before P2V each server needed 2 cables to the storage switches for redundancy. After P2V, each virtualization host will need 4 cables. Two of the cables will be replaced by the connection to the host’s dedicated LUNs where the VM’s operating systems and and other partitions are encapsulated. The second set of two cables will be for the VM’s initiator to access it’s own iSCSI partitions. 

Disconnect the iSCSI initiator before P2V

This is not a must do, but rather a safety net for the P2V migration process. Disconnecting the server’s iSCSI initiator ensures the LUNs you need to maintain will not be selectable as disks to be converted during the migration.

Be prepared to recreate any file shares and permissions

If you disconnect the iSCSI initiator as previously mentioned then be prepared to recreate any file permissions and shares that were configured. To be honest, I am not sure of the best way to prepare for this or if it’s even necessary, but in my experience I have had to recreate shares. Thank goodness it was never a complex user or department hierarchy as you can imagine the impact and administrator time needed that overlooking this would cause.

Check out this VMTN Communities thread on this topic too: VMware Communities: P2V when server has a LUN through iSCSI? …

© Rich for Gestalt IT, 2009. | P2V strategy for a Physical Server with an iSCSI Partition
Read more posts categorized as Server Virtualization