Windows 8 server editions will include a filter driver for NTFS for data deduplication

The next version of Microsoft Windows Server includes integrated data deduplication technology. Microsoft is positioning this as a boon for server virtualization and claims it has very little performance impact. But how exactly does Microsoft’s de-duplication technology work?

Introducing Windows 8 Deduplication

Let’s make one thing clear right from the start: Microsoft started from a clean sheet and invented their own deduplication technology. This is not a licensed, cloned, or copied feature as far as I can tell. There are some clever aspects to it, along with a few head scratchers for folks like me who’ve seen lots of different deduplication approaches.

Microsoft’s deduplication is layered onto NTFS in Windows 8, and will be a feature add-on for Server users. It is implemented as a filter driver on a per volume basis, with each volume a complete, self describing unit. It is cluster aware, and fully crash consistent on all operations. This is a pretty neat trick: As is typical for Microsoft, deduplication will be a simple, transparent feature.

Now let’s talk for a moment about what Windows 8 deduplication is not.

• It is a server-only feature, like so many of Microsoft’s storage developments. But perhaps we might see it deployed in low-end or home servers in the future.
• It is not supported on boot or system volumes.
• Although it should work just fine on removable drives, deduplication requires NTFS so you can forget about FAT or exFAT. And of course the connected system must be running a server edition of Windows 8.
• Although deduplication does not work with clustered shared volumes, it is supported in Hyper-V configurations that do not use CSV.
• Finally, deduplication does not function on encrypted files, files with extended attributes, tiny (less than 64 kB) files, or re-parse points.

Some Technical Details on Deduplication in Windows 8

Microsoft Research spent 2 years experimenting with algorithms to find the “cheapest” in terms of overhead. They select a chunk size  for each data set. This is typically between 32 KB and 128 KB, but smaller chunks can be created as well. Microsoft claims that most real-world use cases are about 80 KB. The system processes all the data looking for “fingerprints” of split points and selects the “best” on the fly for each file.

After data is de-duplicated, Microsoft compresses the chunks and stores them in a special “chunk store” within NTFS. This is actually  part of the System Volume store in the root of the volume, so dedupe is volume-level. The entire setup is self describing, so a deduplication NTFS volume can be read by another server without any external data.

There is some redundancy in the system as well. Any chunk that is referenced more than x times (100 by default) will be kept in a second location. All data in the filesystem is checksummed and will be proactively repaired. The same is done for the metadata. The deduplication service includes a scrubbing job as well as a file system optimization task to keep everything running smoothly.

Windows 8 deduplication cooperates with other elements of the operating system. The Windows caching layer is dedupe-aware, and this will greatly accelerate overall performance. Windows 8 also includes a new “express” library that makes compression “20 times faster”. Compressed files are not re-compressed based on filetype, so zip files, Office 2007+ files, etc will be skipped and just deduped.

New writes are not deduped – this is a post-process technology. The data deduplication service can be scheduled or can run in “background mode” and wait for idle time. Therefore, I/O impact is between “none and 2x” depending on type. Opening a file is less than 3% greater I/O and can be faster if it’s cached. Copying a large file can make some difference (e.g. 10 GB VHD) since it adds additional disk seeks, but multiple concurrent copies that share data can actually improve performance.

Stephen’s Stance

Although I am intrigued by Microsoft’s new deduplication technology in Windows 8 server, I still have many questions about its usefulness and impact on performance. Concentrating duplicate data in the system volume makes sense from a technical perspective, but could lead to an I/O hotspot on the disk. This is especially true for external caching storage systems, since there is no integration between Microsoft deduplication and storage array features. I am particularly concerned about the use of deduplication with VHD files in Hyper-V, since it could eat up valuable system RAM and impact I/O performance.

If you would like to try Microsoft deduplication for yourself, I am happy to report that it is included in the developer preview of Windows 8 that is available on Dev Center. Here are a few commands to get you started, and read Rick Vanover’s post too!

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication
Import-Module Deduplication
Enable-DedupVolume E:
get-dedupvolume

Note: I am a Microsoft MVP and Microsoft briefs me on upcoming technologies under NDA. This post is based on a Microsoft briefing from November which was said at the time not to be covered by any NDA. All of this information could be gleaned by experimenting with the Windows 8 developer preview, but it’s much easier to just go to the source.

© Stephen Foskett for Gestalt IT, 2012. | Microsoft Adds Data Deduplication to NTFS in Windows 8
Read more posts categorized as Server Virtualization, Storage

However, deduplication is slowly sneaking into new areas of IT… and we are seeing more and more benefit elsewhere. Backup clients, backup servers, primary storage, and who-knows-where in the future.

As deduplication is being deployed across the IT world, the technology continues to advance and become quicker and more efficient. So, in order to try and stay on top of your game, knowing a little about the techniques for deduplication may add another tool in your tool belt and allow you to make a better decision for your company/clients.

Deduplication is accomplished by sharing common blocks of data on storage environments and only storing the changes to the data versus storing a copy of the data AGAIN! This allows for some significant storage savings… especially when you consider that many of file changes are minor adjustments versus major data loads (at least as far as corporate IT user data).

So, how is this magic accomplished? – Great question, I am glad you asked! Enter Fixed Block deduplication and Variable Block deduplication…

Fixed Block deduplication involves determining a block size and segmenting files/data into those block sizes. Then, those blocks are what are stored in the storage subsystem.

Variable Block deduplication involves using algorithms to determine a variable block size. The data is split based on the algorithm’s determination. Then, those blocks are stored in the subsystem.

Check out the following example based on the following sentence: “deduplication technologies are becoming more an more important now.”

Notice how the variable block deduplication has some funky block sizes. While this does not look too efficient compared to fixed block, check out what happens when I make a correction to the sentence. Oops… it looks like I used ‘an’ when it should have been ‘and’. Time to change the file: “deduplication technologies are becoming more and more important now.”  File –> Save

After the file was changed and deduplicated, this is what the storage subsystem saw:

The red sections represent the changed blocks that have changed. By adding a single character in the sentence, a ‘d’, the sentence length shifted and more blocks suddenly changed. The Fixed Block solution saw 4 out of 9 blocks changed. The Variable Block solution saw 1 out of 9 blocks changed. Variable block deduplication ends up providing a higher storage density.

Now, if you determine you have something doing fixed block deduplication, don’t go and return it right now. It probably rocks and you are definitely seeing value in what you have. However, if you are in the market for something that deduplicates data, it is not going to hurt to ask the vendor if they use fixed block or variable block deduplication. You should find that you get better density and maximize your storage purchase even more.

Happy storing!

© Bill for Gestalt IT, 2011. | Fixed Block vs Variable Block Deduplication – A Quick Primer
Read more posts categorized as Featured, Storage

W. Curtis Preston launched his own series of Backup Central Live! seminars for 2011

Last week, after the Exec Event in Palo Alto, I joined my friend W. Curtis Preston for his first Backup Central Live! event. Curtis has spent years educating IT pros about data protection, this was the first week of a new series of self-produced events. And let me tell you, although I’ve seen him present dozens of times, Curtis was really in his element here. He held the packed room enthralled, and the vendor sponsors I talked to were very pleased about the event!

Introducing Backup Central Live!

The Backup Central Live! series are day-long seminars across the USA in 2011. Each event includes over 3 hours of content from “Mr. Backup”, W. Curtis Preston, as well as presentations from Jacob Farmer and the sponsoring vendors. The seminars are free for qualified end-users, which includes most of the readers of this blog!

Curtis and company will cover the challenges of backing up and recovering data in a variety of settings:

• Virtualized servers (e.g. VMware, Hyper-V, Xen)
• Very large servers and data centers
• Remote offices and laptops
• Data retained for multiple years

The session also includes technical detail about key products and technologies:

• Cloud Backup Services
• Deduplication
• Continuous data protection (CDP) and near-CDP
• Archive software
• Tape and its proper role

Attendees even get free breakfast and lunch, which was of a good hotel caterer quality in my opinion.

Stephen’s Stance

The Backup Central Live! crew does a great job putting together a professional event

I knew Curtis could put together quality backup content, but the crew deserves credit for such a professional and successful event. They attracted some great sponsors, too, including AppAsure, Aptare, FalconStor, NEC, Quantum, Spectra Logic, and Cirtas. And Jacob Farmer’s involvement was a pleasant surprise, too: I’ve always enjoyed the deep technical conversations I’ve had with him!

If you enjoyed my own backup, archiving, and storage seminars in the past, I know you’ll love this event. The next Backup Central Live! cities are as follows. If you’ll be around, you really ought to attend!

• Orlando, FL Feb 1 Register here
• Houston, TX Feb 8 Register here
• Chicago, IL Feb 22 Register here

My only suggestion for the crew is that they get a bigger room next time!

© Stephen Foskett for Gestalt IT, 2011. | See W. Curtis Preston’s Backup Central Live!
Read more posts categorized as Desktop, Events, Featured, Server Virtualization, Storage

CommVault introduces an updated version of their all-encompassing Simpana suite

CommVault is one of those enterprise IT companies that likes to go their own way. A spin-out of AT&T’s famed Bell Labs, CommVault’s Simpana software integrates many aspects of data management, from backup to e-discovery, under one umbrella. Last year, the company impressed me by adding cloud storage as a backup target equal in status to disk and traditional tape. Now the company is doing the same for storage-based snapshots, accelerating data protection for virtual machines.

Simpana 9: A Broad Update

Nearly every aspect of CommVault’s Simpana 9 is refreshed, though development and unveiling has been an ongoing process throughout the year. Following Simpana 8 by two years, the new software integrates work done integrating cloud storage and virtual machine backup over the last year or so. The “data management” portion of the product (including backup, archiving, and replication) was detailed this week, and one expects more information about the other half (“information management”) to be forthcoming.

Simpana 9 is available as of October 5 from OEMs, resellers, also direct from CommVault. Pricing is simplified this time around with CommVault using a capacity licensing model. Cost is based on the largest backup or archiving job, though the old licensing model still still available. A “circuit breaker reset” allows the system to go over the licensed level temporarily in a pinch.

Pick Your Target: Tape, Disk, Cloud, or Snapshot

There has been a long-standing back-and-forth discussion in the enterprise backup community regarding snapshots as backups. My stance has been that, while snapshots are certainly a more-effective mechanism for copying data than dump-to-tape, it takes more than data to have a useful backup. Any system worth using must schedule, manage, organize, and recover data. Therefore, although snapshot-based backup is a good idea, it wasn’t ready for prime time until it was integrated with a solid backup management system.

Elevating snapshots as a first-class backup target is exactly what CommVault is doing. Just as their earlier releases added cloud as a backup target, Simpana 9 makes snapshots just another option. Commvault recognizes that needs change over the lifecycle of a backup, and the location of data should change, too. A snapshot makes an excellent “operational recovery” target, allowing frequent copying and quick restore, but is sub-optimal for long-term archiving. Conversely, tape is a poor frequent target due to its linear nature but has proven itself for long-term offline archive.

Simpana 8 supported HDS and Dell for snapshots, now Simpana 9 adds volume leaders EMC and NetApp, along with HP, IBM, LSI, and Sun. Data is indexed and cataloged, and these are stored with the data on disk, tape, or cloud for later access. Speaking of cloud, Simpana 9 supports Amazon, Azure, Nirvanix, Rackspace, Iron Mountain, EMC Atmos, Dell DX, and HDS HCP.

You might also want to read CommVault Gives Cloud Storage A Seat At The Adult Table

VM Backup Acceleration

CommVault was demonstrating their abilities to back up machines using VMware VADP at VMworld. There, they took just 17 minutes to back up 500 machines using snapshots on an IBM XIV storage array. I imagine most of the supported platforms could do just as well.

Simpana 9 automatically discovers running guests using VMware vCenter. This allows for daily movement of machines since they are located each time and the backups are coordinated for universal, consistent coverage. No machines will be missed, either. In fact, given that many snapshots work on a block level, it would be impossible to not backup up clients sharing the same LUN.

On the back side, Simpana uses an ESXi proxy to grab snap content and catalog the virtual machines contained there. It then moves the data off to disk, tape, or cloud just like any other backup content.

CommVault is also working on Hyper-V support, using VSS rather than VADP.

Universal Dedupe

The third element of Simpana 9 is integrated source and target deduplication. Simpana’s client software deduplicates local data on its own before sending it to the server. These deduped blocks are then checked against the universal set on the server before being stored, potentially bringing a “best of both worlds” solution to the source-versus-target argument.

Migration to Simpana 9 From NetBackup and TSM

One of the long-standing criticisms of Simpana is that it was just too far-reaching for easy testing, migration, and uptake. It has been seen as an all-or-nothing proposition, requiring a commitment that is hard to win and harder to back away from in the event of trouble.

Simpana 9 addresses part of this concern with a so-called “Fast Pass” migration system that allows configuration to be pulled in from more-traditional backup products for a seamless transition. The Simpana data connector talks to NBU 6.0, 6.5, 7.0 and TSM 6.1 master server, discovers backup policies, and imports these (clients, schedule, history, etc) into the new configuration. Then automatic install software pushes CommVault agents into place and takes over operations, leaving the old system running for historical recoveries only.

Stephen’s Stance

I love the idea of promoting data from operational snapshots to tape or cloud for archiving. And I love the idea that end-users can choose the combination that fits their needs within a unified package. I’ve long been a fan of CommVault’s products, and this upgrade makes them that much more compelling. I’ll leave it to wiser minds to debate their implementation and effectiveness, but it sure looks good for me. The simplified licensing and purchasing model is welcome, too – backup is a thorny combination of pricing models.

The idea of migrating seamlessly from old to new is interesting, but I wonder how well it will work in practice. My time as a backup consultant taught me that these environments tend to be complicated, finicky, and outdated. It’s nice that CommVault can read in a TSM 6.1 configuration, but what about older versions? The same goes for NetBackup. And will it correctly handle the weird rule sets found there? But a greater issue is whether it should import the old settings at all: Perhaps migrating to a new backup system is a good time to revisit the assumptions behind your configuration!

All in all, Simpana 9 looks like a compelling update for CommVault lovers or those disaffected by their current platform. Organizations having difficulty meeting backup windows should also give it a look, since the snapshot technology should be a big help there. And anyone looking at cloud storage as a backup and archiving target should start with CommVault. Others will certainly be intrigued, but I am not sure that they will be compelled to upgrade by these features alone.

© Stephen Foskett for Gestalt IT, 2010. | CommVault Simpana 9 Takes Backup To A New Level
Read more posts categorized as Server Virtualization, Storage

Wandering the EMC World show floor was very revealing. Along with EMC’s new VPLEX, Chris Mellor and I spotted the unannounced SDRAM-based ZeusRAM SSD from STEC. While getting some hands-on time with Iomega’s new 12-drive storage array, I spotted an exciting but unannounced feature: The ix12-300r includes native Avamar backup client!

Native Avamar support in the ix12-300r? That’s new!

Chocolate and Peanut Butter

EMC’s Avamar backup product deserves much more press than it gets. Global deduplication is the secret sauce, allowing high efficiency for backups, both in terms of capacity and bandwidth usage. This means that Avamar is very well-suited for small business and remote office use.

Iomega’s new ix12 is similarly well-suited in this environment. It includes flexible storage allocation, both iSCSI SAN and NAS, and is bursting with features, from replication to webcam support. The 12-drive array fits perfectly into the small-business and remote-office markets with a low $5k-$10k price tag.

Avamar’s backup deduplication technology and Iomega’s “just right” small-business storage is a real “chocolate-and-peanut butter” combination. Both target the same markets and mixing them together is even better than selling them apart. Integrated Avamar client support makes the ix12 even more compelling, and will likely drive Avamar sales as well.

I talked to the Iomega execs at EMC World and they confirmed that this support will be included in the shipping ix12 software but was not yet supported by the company. They expect Avamar qualification to come later in the year, but I suspect that customer interest could accelerate this timetable. If you are interested in the combination, let Iomega and Avamar know!

ix12 Impressions

My release-day review of the ix12-300r was based on press materials and a briefing, but getting a hands-on look at the device revealed details beyond the Avamar support. I was quite pleased by the build quality of the array: It is solidly constructed and the connectors, fans, and power supplies looked to be of high quality. The designers seem to have paid attention to cooling and airflow, important in a box containing 12 spinning hard disk drives. The dual power supplies are hot-swappable and seemed fairly robust as well.

I also spotted two empty PCI Express slots at the rear of the ix12. One wonders what use Iomega will put these to. One can imagine adding additional gigabit or 10 gigabit Ethernet controllers, but solid-state storage (SSD) would be much more interesting. The ix12’s LifeLink software has been enhanced fore more-flexible RAID configuration; I wonder if it also includes the ability to tier storage to SSD. A mini array with EMC’s FAST technology would be compelling, but the use of SSD as a cache (FAST Cache?) might be simpler to implement.

Clearly, the ix12-300r is much more than a SATA-expanded ix4-200r. Iomega is adding real valuable business features to the system and moving it ever closer to the EMC and Dell CLARiiON products. Yet the price is still well below those systems. Chocolate and peanut butter, anyone?

© Stephen Foskett for Gestalt IT, 2010. | EMC Mixes Avamar Into Iomega ix12-300r
Read more posts categorized as Storage

I don’t want to mislead anyone – almost all of my fellow delegates where deeply impressed with the block by block algorithm lesson on Ocarina’s technology. The storage gurus where so into it I had to wait for the hands on labs to start before I could get my questions answered. In fact, I never did the labs because I spent the whole time at the white board understanding the deployment options.

My post is therefore about implementation options of an Ocarina solution. I’m using several quotes from posts by Carter George, co-founder of Ocarina Networks, taken from the Online Storage Optimization Blog to help me explain the deployment options. Then I’ll cover some ideas about how could Ocarina function in a virtual environment.

For those that want to know more about the deep dive on the Ocarina technology, follow the links at the end of this write up to my fellow attendee’s posts.

Ocarina needs an ethernet connection

First of all, Ocarina’s technology works with NFS storage and CIFS. That means ip connectivity must exist between the data shares and the Ocarina server / appliance for 2 of the 3 possible deployment options.  Basically, Ocarina will remotely monitor the storage device for select activity it can optimize via the normal management interface. I was told that there was no need for a dedicated or private network.

3 Deployment Methods

As the post Going Native CIFS explains, Ocarina can be deployed in one of three ways:

“Ocarina Inside”: Ocarina is embedded in a NAS vendor’s solution.

Examples of “Ocarina Inside” are EMC Celerra, HP Enterprise NAS, BlueArc, and HDS HNAS.  Additional “Ocarina Inside” partners will be announced soon. This is the best form of integration, because it makes deduplication and compression completely transparent to users and applications, and lets each storage vendor deliver all their full value-add, including in the CIFS protocol stack.

Ocarina Appliance: A split-band appliance

In the Ocarina Appliance case, Ocarina’s optimization happens out of the customer data path, but in order to expand files to their original state upon user access, the Ocarina intercepts read requests in-band. If an I/O (over CIFS or NFS) is to an Ocarina-optimized file, we step in, rehydrate the file, and pass it on to the user. This involves being a proxy for NFS and CIFS (and other protocols including WebDAV and http)

Ocarina Native Format Optimization (NFO): files are optimized in their native format

This is a special use of Ocarina where we take certain rich media file types – photos, images and video – and compress them in a special way. What we do is compress them, but have the output be a new, smaller file but in the same native format it started out in. We’ll take a JPEG photo, compress it, and produce as output another perfectly formed JPEG photo….just smaller. The same is true for example for Flash videos. Now in this case, there is no need for a decompressor or for Ocarina to be in the read path or on the protocol at all. We can read files from your NetApp, shrink them, write them back on to your NetApp and Ocarina need not be involved at all when users or applications go to access those files.”

Virtual Machines and Ocarina

I got the following information from the post titled Dedupe Misconceptions. The post specifically references VMware virtual machines, but the scenario can be easily imagined for any virtualization solution.

In a virtual machine environment, a storage array may be storing thousands of VMDK’s, the VMware files that store a given virtual machine. Inside each VMDK file is a complete virtual machine image, including the operating system, application files and user data. If you have 1,000 VMDK’s that holds virtual Windows machine, you’ll have tens of thousands of “files” inside that VMDK file, including a copy of Microsoft Windows, the application you are running the in the virtual machine, and often the data for that application as well. How much of the Windows operating system do you suppose is duplicated across the 1,000 VMDK’s in this example? Well, almost all of it. What’s more, the thousands of files that make up Windows do not change – are not changeable, in fact, unless you do an OS upgrade.

Large parts of the VMDK file are duplicate with others, and they stay the same, day after day. Perfect candidates for dedupe. Sure, the user data in a VMDK may change, but any competent dedupe solution is not deduplicating whole files – the dedupe solution is deduplicating something at sub-file granularity: blocks, objects, chunks, etc.

VI Design Ideas for Ocarina

Unfortunately, Ocarina cannot optimize primary storage for live VMs. This is not to say there is not a place in a virtual environment for the technology.

When I asked Ocarina how they felt they fit best with virtualization, It was suggested that VMs that remain powered off could be moved to a secondary NFS LUN / datastore that was compressed. Before these VMs could be powered on they would be migrated back to the primary datastores not optimized by Ocarina. At a high level, this seems like a great way to maintain backup clones of production VMs without provisioning twice as much storage. To that point, we also listened to a Nirvanix Enterprise Cloud Storage presentation while at Ocarina’s office.

Then thinking about other virtualization scenarios for Ocarnia, I considered how significant storage savings could be achieved for file servers, document management, interior design drawings, photography and film repositories, or any other applications that manages a lot of files. In order to consolidate hardware the server operating system could be encapsulated as a virtual disk, but the application data could be separated to a dedicated CIFS or NFS share. I’ve even posted before about leveraging the built in capability of NetApp filers to act as a network CIFS share. I mention NetApp specifically in my post, but remember, Ocarina works with any storage device. Ocarina would be optimizing the application data in these scenarios and not touching the virtualized pieces.

For those that want to know the deep Ocarina storage details here are several links to other Tech Field Day blog posts on Ocarina:

• www.techhead.co.uk -videos of CTO Goutham Rao
• rodos.haywood.org- more video of Goutham and a lot of detail about the Ocarina compression and de-dupe
• storagemonkees.com – a great summary of the Ocarina session with a ton of storage insight
• renegade.tweakblogs.net – Technology details and then more details in the comments
• blogs.techrepublic.com.com/datacenter – good overview of the Ocarina compression and dedupe

© Rich for Gestalt IT, 2009. | Considering Ocarina Networks Optimized Data For Virtual Environments
Read more posts categorized as Events, Server Virtualization, Storage

What’s interesting about this is now dedupe will be available for *free* unless Oracle does something stupid.  Sun’s implementation is complimentary to the already-existing filesystem compression.  I’m not sure how much of an issue this is yet but the current iteration can not take advantage of SHA256 acceleration in the SPARC Niagara2 CPUs but eventually we should see hardware acceleration implemented.

When will it be available? It should be available in the Opensolaris dev branches in the next couple of weeks as code was just committed to be part of snv_128.  General available in Solaris 10 will take a bit longer until the next update happens.

For OpenSolaris, you change your repository and switch to the development branches – should be available to public in about 3-3.5 weeks time.  Plenty of instructions on how to do this on the net and in this list.  — James Lever on the zfs-discuss mailing list

How do I use it? If you haven’t built an Opensolaris box before, you should start looking at this great blog post here.  I wouldn’t get things rolling until dedupe is in the public release tree.

Ah, finally, the part you’ve really been waiting for.

If you have a storage pool named ‘tank’ and you want to use dedup, just type this:

zfs set dedup=on tank

That’s it.

Like all zfs properties, the ‘dedup’ property follows the usual rules for ZFS dataset property inheritance. Thus, even though deduplication has pool-wide scope, you can opt in or opt out on a per-dataset basis.

– Jeff Bonwick http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup#comments

What does this mean to me? Depends.  For people who like to tinker, you can build your own NAS or iSCSI server with dedupe *and* compression turned on.  Modern CPUs keep increasing in speed and can handle this.  This is huge.  Now, should you abandon considering commercial dedupe appliances that are shipping today?  Not if you want a solution for production as this won’t be officially supported until it’s rolled into the next Solaris update.  For commercial dedupe technology vendors, this is another mark on the scorecard for the commoditization of dedupe.

What things do I need to be aware of? The bugs need to be worked out of this early on so apply standard caution.  READ JEFF’s BLOG POST FIRST!!! There is a verification feature, use it if you’re either worried about your data or using fletcher-4 as a hashing algorithm to speed up dedupe performance (zfs set dedup=verify tank or zfs set dedup=fletcher4,verify tank).

How do I stay up to date on ZFS in general? Subscribe to the zfs-discuss mailing list (also in forum format).  It can be high volume but it is worth it if you want to stay on top of all things zfs.

© edsai for Gestalt IT, 2009. | Data Dedupe comes to ZFS
Read more posts categorized as Server Virtualization, Storage

And frighteningly enough, we are in the process of making the whole thing worse! The two darling storage technologies of the moment; thin provisioning and de-dupe scare me witless. Both of these technologies in the wrong hands have the capability to bring a server estate to its knees. By wrong hands, I mean just about anybody.

Both of them allow you to virtualise capacity and allocate storage which isn’t actually there and hope that you never need it.

Thin provisioning is predicated on the inefficient practises that we have come to know and love; we all know that when a user asks for storage, they nearly always ask for too much! Thin provisioning allows us to logically allocate the disk and only when it gets written to, does it actually get consumed.

The problem is, what happens in the event of a perfect storm and every application wants its capacity at the same time? How much do you over commit your physical capacity? Or maybe not a perfect storm, you just realise that you’re going to add physical capacity above and beyond that which is supported by the array simply to cater for rate the thinnly provisioned storage is being consumed.  A rapid application migration ensues.

And then there is the scarey proposition of de-duped primary storage. You could be many times over-subscribed with de-duped storage; certainly in a virtualised server environment or a development environment where you have many copies of the same data. And then someone does something; a user decides to turn on encryption and what was many deduped copies of the same data actually becomes many full copies of the same data and you have run out of storage space in a spectacular fashion.

Also migrating deduped primary storage between arrays is going to be a lot of fun as well and is going to need a lot of planning. Deduping primary storage may well be one of the ultimate vendor lock-ins if we are not careful.

Both thin-provisioning and primary storage dedupe take a lot of control away from the storage team; this is not necessarily a bad thing but the storage team now need to understand a lot more about what their users are doing.

It will no longer be enough to just to think about supplying spinning rust which can deliver a certain amount of capacity and performance. We are going to have to understand what the users are doing day-to-day and we are going to have to communicate with each other.

And yes, we’ll need better tools which allow us to see what is going on in the environment but also to model the complex interactions and impacts of various events. We are going to need to know if a user is intending to do something like enabling encryption, a big data-refresh, operating system patching; events which in the past were not hugely significant could now have serious ramifications to our storage capacity.

I still think that thin-provisioning and de-dupe are good ideas but like all good ideas; they come with challenges and a certain amount of risk…

© Martin for Gestalt IT, 2009. | Living on a prayer
Read more posts categorized as All, Featured, Storage

Why all the interest? Data Domain has emerged as the leader in data deduplication, a key technology for controlling the rapid growth of duplicate data in enterprise IT. Although both EMC and NetApp offer deduplication products, Data Domain’s products were widely lauded and adopted by end users, especially in smaller organizations.

Although many questions were raised about the fit for Data Domain within NetApp’s product lineup, the fit in EMC makes more sense. Their Quantum-based VTLs are expensive and enterprise-focused, while a new Data Domain-powered line might have broader appeal. Although EMC appears to have cemented their Quantum relationship with a $100 million investment to keep the backup products vendor afloat, the fact that they did not buy the company outright might indicate that they had second thoughts. EMC’s Chuck Hollis claims that Data Domain makes sense at many levels throughout EMC’s product portfolio.

Others have suggested that the offer was purely defensive, designed to keep Data Domain out of NetApp’s hands or to drive up the eventual price the West coast team would eventually have to pay. This would be a very expensive gamble, however, since NetApp already has homegrown deduplication technology and could walk away entirely.

© Stephen Foskett for Gestalt IT, 2009. | EMC Takes On NetApp For Data Domain’s Affections
Read more posts categorized as Exclusive, Featured, Storage

Although Windows Server 2008, with its many storage feature updates, was released last year, the updated version of Windows Storage Server was still under construction until last month. But Windows Storage Server 2008 is available to manufacturers today! Expect to see some new Intel-based storage array announcements in the coming weeks!

For my personal experiences with Windows Storage Server 2008, see my blog post, I Can Finally Talk About Windows Storage Server 2008!

What’s new in Storage Server 2008? Plenty! Most of the features are inherited from Windows Server 2008:

1. Server Message Block (SMB) 2.0 is a re-working of the traditional Windows NAS protocol. Also present in Vista and Server 2008, SMB 2.0 reduces the notorious chattiness of SMB, combining multiple commands into a single packet, as well as allowing more simultaneous open connections and larger buffers as well as durable file handles.  Performance gains using both SMB 2.0 clients and servers has been phenominal with Tom’s Hardware showing a 5x gain in throughput over the Internet!
2. The NFS server (also present in Windows Server 2008) has been updated.
3. A new MMC snap-in called Storage Explorer lets you manage WMI-compliant SAN devices.
4. File Server Resource Manager (FSRM) is a full-featured storage resource management (SRM) application and has been improved greatly for both the Server and Storage Server versions of 2008 with quotas, file screening, and advanced reporting.
5. DFS-R and DFS-N are tweaked – Jose Barreto gets into this on his blog.
6. Smaller or branch offices (and low-end storage array vendors) will be interested in using BitLocker full-volume drive encryption to protect their data.
7. Storage Manager for SANs (SMfS) allows you to perform basic storage array administration tasks within Windows.
8. The Volume Shadow Copy Service (VSS) and DFS interfaces are improved and are now scriptable with PowerShell (through WMI).
9. The new but not yet complete Windows Server Backup system replaces the outdated and limited NTBackup system. Although it includes bare-metal recovery, Server Backup isn’t ready for prime time in my opinion. I’m looking for major improvements in R2!
10. Search 4.0 and 2008 SP2 adds full-text search for files stored on a Storage Server.
11. Server 2008 now automatically aligns filesystem boundaries with storage, which was one of those dark and secret skills us storage guys used to share amongst ourselves.  This can increase performance in high-I/O environments.
12. NTFS (in both Server 2008 and Vista) now has symbolic link support, just like UNIX and Mac OS X, and SMB 2.0 supports symlinks as well.
13. NTFS was also tuned and tweaked a bit for better stability and crash recovery.
14. The updated virtual disk service (VDS) supports LUN shrinking, online/offline and read-only/read-write settings, and SAN policies so new LUNs can be treated as either online, offline, or shared.
15. Failover clustering is simplified, requiring just a few clicks to set up.

Although all of these are also present in the basic Windows Server 2008 install, Windows Storage Server 2008 includes some unique features:

1. The included iSCSI target software is unique
   1. It’s been updated to support failover clustering
   2. It now includes VDS and VSS providers
   3. It’s been ported to StorPort
   4. Installation is now done in the standard Windows manner, with an MSI not an EXE
   5. You can now set up an IPv6-only iSCSI SAN and specify initiators directly using their IPv6 addresses
   6. CHAP secret security is enhanced
2. Single-Instance Storage (SIS) is the second major Storage Server differentiator, providing file-level deduplication
   1. SIS used to be limited to 6 volumes per node, but now scales to 128
   2. SIS now supports clustering, though SIS doesn’t span nodes
   3. A new command allows one to undo single-instancing instead of copying all of a volume’s content to another drive
   4. Enabling SIS is simpler – it’s now just a checkbox per volume in the Share and Storage Management UI
3. Many performance tuning tweaks are standard out of the box, though readers of the Windows Server 2008 Performance Tuning White Paper might be able to perform these on their own. One major standard tweak was removing 8.3 naming and disabling aliasing on filesystem – this led to an an 8% performance gain right out of the box.
4. Remote administration through HTTP is very cool, and ought to be standard on every version of Windows Server! Just point a web browser to the server and you will have an ActiveX or Java-based RDP client without installing any software.
5. Licensing is one more unique aspect. Storage Server does not require client-access licenses (CALs) so any number of clients can access the system without worrying about license management.

All in all, Storage Server 2008 is a solid move forward. I expect that the ability to do single-instance storage at full native speed will be very useful for corporate file servers and similar applications, and the enhancements overall are welcome as well. But this will be the last release of Windows Storage Server as a separate product. From now on, Microsoft simply release OEM storage server software on top of their standard Windows Server versions for OEM’s to use. There will not even be a special Service Pack 2 version of Storage Server 2008. Instead, expect OEMs to provide the regular Server 2008 SP2 as a suggested or required update to Storage Server users.

Microsoft is detailing the new version of Windows Storage Server 2008 in a webcast Thursday at 8 AM Pacific. You should also check out the official Microsoft site, and the Microsoft Storage Server blog, especially their post, A Brief History of Windows Storage Server Releases.

© Stephen Foskett for Gestalt IT, 2009. | Windows Storage Server-Based Systems Step Into 2008
Read more posts categorized as All, Exclusive, Featured, Storage, Top Story