Continuing the blog posts on Storage Resource Analysis (SRA), this post focuses on some facts about data that sits in Storage Environments.

Read the entire series of posts on Storage Resource Analysis (SRA):

Part 1: Storage Resource Analysis and Storage Economics

Part 2: The IT – Storage World of 2009

Part 3: The IT – Storage Budgets of 2009

Part 4: Some Fundamental Questions

Part 5: Facts about your Data

Part 6: Inconsistencies in Storage Environments

Part 7: The Technical Case

Part 8: The Business Case

Part 9: The End Result

Going to a very basic question about a Storage Environment, What percent of your storage is allocated and what percent of it is actually being used?

The more utilization you gain, the further you can push your new storage acquisition timeframes increasing your ROI and reducing your CapEx. The better efficiency and optimization you gain in your storage environments, the further you can push to reduce your OpEx.

So based on some latest data we have gathered from various different customer environments, believe it or not, on an average, this is what we saw…..

Allocated Storage

• Average Customer Production data: Between 20 to 24%
• Average Copy / Replication data: 20% more than Production data: Between 24 to 28%
• Raid Overhead associated with production data: Between 15 to 19%
• Storage Allocated but no owners: Between 18 to 22%

Unallocated Storage

• Storage unallocated meant for future use: 13 to 17%

To shed some light on production and copy / replication data, there are a lot of data inconsistencies being reported in terms of broken links, replication failures, source and bcv lun size mismatches, etc.

The above scenarios with allocated / unallocated storage will reduce your ROI (Return on Investment) and ROA (Return on Asset), further causing budgeting issues in an organization.  

As you go to various different teams and ask them about the allocated / unallocated storage, you will hear different answers throughout the entire organization. From everyone’s view these numbers are debatable.  

Experience:

This time around, a personal experience.

In addition to my Technology Solutions responsibilities for our customers, I also have responsibilities around managing internal IT infrastructure that includes the apps, databases , storage and other IT assets. We have a lot of new projects happening every month, quarter, etc and have to allocate storage for those needs.

Projects that are hot today, might not be hot in the next 3 months, or may be in next 6 months. Projects get scrapped, host systems get scrapped, but storage still sits there, churning and spinning. Believe it or not, as the economic times change, lots of things are enforced and as the IT managers internally started looking at all the storage, we were able to reclaim quite a bit that was in real terms stranded storage, which is now all allocated to new ongoing projects or ready to be allocated at a short notice.

Do you know how much of your storage today is allocated, unallocated, stranded and importantly how much of it is Production data in your environment? 

© Devang for Gestalt IT, 2009. | Storage Resource Analysis (SRA): Part 5
Read more posts categorized as Storage

3Par took wide striping and made it useable; EMC’s historic implementation using metas and hypers was painful and with the large arrays of today it becomes a full time job to performance manage an array. 3Par made it easy and much kudos to them for doing so. I think 3Par’s legacy will be the ease of management that they have brought to the Enterprise array (and thin provisioning).

I think it is worth pointing out to Barry, that you can simply use Wide-striping without Thin-provisioning with a 3Par box as well. LUNs do not need to be thin-provisioned and can be entirely pre-allocated.

Automated wide-striping simply makes the storage admin’s job easier; it de-skills it somewhat and hopefully it will bring an end to the endless poring over of spreadsheets trying to balance workloads.

SSDs will become just another feature with time as well; Barry wants this to be the case. It validates the decision to put SSDs into the DMX. Any good idea will eventually just become another feature if it is good enough and even if it is patented; people will find their way round eventually.

As Barry points out SSDs deliver massively increased IOPs and massively decreased response times; we need this, we desperately need this for some applications. Even if the magnetic disk manufacturers could get their disks to spin faster, the increase in power and cooling required would boil the oceans and hasten our demise as a race.

But until SSDs achieve parity per gigabyte with the cost of spinning disk, we need to find ways to efficiently use what is still a relatively expensive resource and SSDs probably are not the best fit for your file-serving and bulk storage requirements. The venerable Anandtech actually demonstrates with their benchmarks that using SSDs for log-files may not gain you much. It’s actually an interesting if slightly flawed look at SSDs, SATA and SAS; it would have been more interesting if they’d done more work and gone more granular into the data-base tables.

They need to be used for appropriate workloads and ideally we need something like this from Compellent. Unfortunately for Compellent, I have horrid suspicion that this level of automated tiered storage will simply become another feature; you can’t keep a good idea down! Once we have block-level migration both automatic and rules-based we can work out quickly and easily how much SSD we need.

SSDs with automated tiering will save you money; probably both in terms of TCA and TCO. SSDs without automated tiering will save you money in terms of TCA for appropriate workloads but may end-up costing you in terms of TCO because the work needed to identify, balance and move data around.

Of course, it all comes down to the cost of the software needed to manage and automate the process. If the software is too expensive and the vendors simply try to milk it as a new cash-cow; we’ll not realise the savings of this brave new world.

© Martin for Gestalt IT, 2009. | Just another feature…
Read more posts categorized as All, Storage

CC-by-SA 2.5 image "Golf Bunker" by Ken123

Today, many companies are considering a compete migration from a physical to a virtual infrastructure. Based on the promises of cost savings, administrative efficiencies, and improved resource utilization, virtualization is seen as the technology that makes it possible to do more with less. However, there are many pitfalls to consider when virtualizing server infrastructure. This article suggests planning decisions to be considered by the CIO, IT Director, design architect, and IT Manager that, if overlooked, could ruin the total cost of ownership (TCO) and the return on on investment (ROI) expected from this virtual infrastructure.
�

Migration pitfalls

Implementing virtualization is not a simple process of building a server. Too often, management makes the mistake of expecting that the administrators and architects of the server team will be solely responsible for virtualization decisions. In reality, the impact of a move to virtualization should be viewed exactly as if a company were moving their physical servers to a new data center. Storage, networking, security, Active Directory, messaging, web presence, and all of the systems and services necessary to the organization must be considered together. Some of these groups should be more involved than others in the actual planning and ultimate design decisions, but virtualization strategy affects everyone. Key members from all of these focus areas must be trained and given panning responsibilities to achieve a collaborative plan and design.

A potentially larger “gotcha” once everyone is involved in the planning process is the temptation to include all of the upgrade and service redesign projects that have been “in the queue” for months (or longer). Although the migration may seem like a great opportunity to upgrade to the latest operating system, separate instances from poorly performing database servers, or build the new CRM system, these added activities bring complexity and could pose road blocks to the core goal of the virtualization initiative: consolidating hardware. These projects should be tabled and taken on later, after the new infrastructure is in place. In fact, the flexibility of a virtual infrastructure will actually make these projects simpler.

Performance pitfalls

Condensation of numerous physical servers into a virtual environment is intended to increase the effective utilization of processing power and other system resources, but it is easy to overshoot the target and overload the new system. Although the average load of a number of servers might be within the capabilities of a new virtual infrastructure, resource spikes can be a serious problem. Consider that periods of high demand tend to cluster across servers through the day and week, as employees begin the workday, the backup process begins, or a software build is kicked off. If performance is not measured and correlated carefully before migrating to a virtual infrastructure, these resource spikes can make the new infrastructure unacceptably slow.

One often overlooked performance challenge is the randomization of demands introduced by virtualization. Traditional computer design assumed that processes would proceed through their steps sequentially, with the system devoting resources from start to end. Multitasking operating systems challenged this assumption, but the allocation of memory, I/O, and storage remained clustered. But highly-utilized virtual servers upset this predictable flow, rapidly switching from one task to another entirely different one. In short, virtualization randomizes accesses that were once sequential and transforms large requests into many tiny ones. CPU manufacturers have responded with clever techniques to save and restore registers and handle memory mapping, but this blender effect is far from solved. I/O and storage systems are particularly affected, as they are optimized to stream large sequential operations rather than small random ones.

Storage pitfalls

Although one might think that virtualization reduces storage requirements as systems are consolidated, the opposite is often true. Many of the functions that help drive down CAPEX and OPEX costs in virtualized environments require the use of shared networked storage like SAN or NAS. Data that had once been scattered around the data center on internal drives is consolidated on these networked storage devices. Thousands of redundant copies of operating system files, for example, end up sitting on the storage array. The ease of creating virtual machines from templates tends to lead to virtual “server sprawl” far worse than in the physical world. Storage array capacity is also needed for swap space or paging files as well as popular enhancements like snapshots, DR copies, and backup images.

The result is a flood of redundant data that must be taken into account when planning capacity needs. Although storage is typically not the top item in an IT budget, virtualization can cause it to rise and chew up some or all of the savings from the project. It is difficult to avoid these issues, but some techniques can help mitigate the problem. For example, deduplication of primary storage and thin provisioning can help reduce the storage footprint of the virtualized infrastructure.

Administration pitfalls

Virtual sprawl can cripple infrastructure support teams if it is not managed. The challenge of administering this new virtual environment is often ignored in the implementation rush. Rather than managing 3,000 servers, many wake up and find that they are managing 30,000 virtual machines. This is partly because server build times are reduced from days to minutes, and partly because server consolidation on virtual infrastructure is so effective that idle guests are barely noticed. If left unmanaged, the “gasoline fire” spread of VMs could create a burden in the form of permissions, backups, upgrades, patches, and monitoring than what existed in the physical infrastructure.

To combat these scenarios, IT departments need to start with process and policies. Change control becomes paramount. Second in importance is a new server request and approval process. Build standards need to be created and adhered to. Finally, virtual servers must to be audited for activity and then powered off or removed if idle. The good news is that there are numerous life cycle management and automation tools now available for helping an organization provision, maintain, report, and decommission virtual machines.

Backup/Restore pitfalls

Backup is another area where TCO and ROI, as well as performance, can be negatively impacted by virtualization without appropriate planning. As mentioned above, virtual server hardware is often sized based on the average server load. For example, consider the impact of combining ten servers with average CPU utilization of just 10% onto a single physical device. The ROI would appear to be excellent, but the large spike in utilization that happens during backup can cause unacceptable performance issues or even application failures. The performance hit during backup can be mitigated through off-machine backup using like VMware VCB or storage snapshots, but the cost and effort of implementing these features must be taken into account when planning the new system. The conventional approach to backup, loading backup agents on each system and backing them up on a schedule, does not translate in the virtual world.

Disaster recovery pitfalls

One of the primary driving factors for moving to a virtualized environment is its potential positive impact on disaster recovery (DR). Although the DR benefits are real, the requirements for the storage system can become a pitfall. Extra DR data is pushed onto storage arrays and then often replicated to another location. All of this extra data requires increased storage capacity, array features, and WAN bandwidth. The capacity issues can potentially be addressed using primary storage deduplication and thin provisioning, and WAN optimization appliances can reduce bandwidth requirements.

While virtualization does make servers more portable, thus making DR easier, the disaster recovery copy process is a major challenge. Some vendors now have tools like VMware Site Recovery Manager (SRM) to assist in this orchestration, but these tools are somewhat immature, and care should be taken when planning to use them. For example, most do not address failback, so administrators may find themselves doing a lot of work to return to operation at their primary data center after a failover.

© Gestalt for Gestalt IT, 2009. | Planning for Virtual Infrastructure: Avoid the Pitfalls
Read more posts categorized as Exclusive, Featured, Server Virtualization, Top Story

A quick discussion last night on Twitter got me thinking about the way technology is perceived in the industry at large.  In particular, the discussion related to NetApp and in my usual facetious way I said I wouldn’t quote NetApp and efficiency in the same sentence (whoa, I just did!).  

I may be being a little unfair.  From experience, NetApp is good and bad in the efficiency stakes; we all know about the 10% WAFL overhead and the default 20% snapshot reserve, but do we know that NetApp rounds down drives to a consistent size?  Do we know how many un-needed snapshots we have lying around or how many flexclones we created for testing, but don’t need any more?

On the positive side, snapshots are efficient; LUNs and shares can be thin provisioned; I can clone volumes from one another using Flexclone at an initial minimal or no cost in terms of extra storage.  Backups with SnapVault are efficient as they’re effectively an incremental forever design.

Unfortunately perception doesn’t always reflect reality.  Whilst the NetApp product range does undoubtedly offer many space reduction features, these may be poorly implemented by customers, leading to a view that NetApp technology is inefficient. 

So does this mean that vendors are to blame for not communicating the message about the efficiency of their products?  Well, I think there’s blame on both sides.  Going back to NetApp, I’m sometimes bewildered by the number of software features available in Data ONTAP, most of them beginning with Snap or Flex.  What complicates things further is the interaction (or not) of these features with each other.  I like to think I’m reasonably intelligent, however even as a consultant I struggle to understand the options and I’ve had discussions with NetApp previously where even their top people couldn’t definitively confirm whether certain products could be used in conjunction with each other.  This creates customer confusion and helps to perpetuate the myth of Netapp inefficiency.  

On the other hand, customers have an obligation to keep their own house in order; badly thought out designs and poorly implemented management practices also add fuel to the fire on the efficiency myth and you can’t blame the vendor for that.

Is there a solution?  For NetApp, perhaps some unbiased feature reviews would be good – I may choose to do a series on that in the near future.  Here’s another idea – let’s start a Storage Myths site – then we can confirm or knock down these myths one by one just like watching Myth Busters!  Send me your Storage Myths and we’ll compile a list….

© Chris for Gestalt IT, 2009. | Netapp and Efficiency – Managing Perception
Read more posts categorized as All, Storage

With latest announcements from NetApp, IBM and EMC (Next Generation Celerra) in early Feb 2009, EMC’s DMX announcement might come right around EMC World 2009, plus or minus a month.

With the absence of Barry Burke from the Storage Blogosphere community over the past 3 months, it seems like he is busy working on strategy for the new generation of DMX machines. 

Here is my wish list or expectations on the new DMX Platform.

Strategy, Cost, Marketing, Support…….

Do less with more!!!! This will have to be the reality of the new generation of DMX’s. In this tough economy and financial distress, if a new product is pitched with same efficiencies and overall similar ROI and TCO models, it will be hard to sell.

Some important CIO, CFO pitches would include, less foot print with more data, higher efficiency, delivered at 2/3^rd the cost of previous generation of machines, energy savings, etc. Key Differentiators would be the cost per TB of data storage, cost per TB of management (Storage Administrators, OpEx), a savings of 20 to 30% in this equation might come into play.

The cost of warranty of each DMX is pretty high, during the warranty phase EMC Support Labs in Hopkinton, Sydney, Cork and Bangalore are supporting these boxes on a 24 x 7 basis. If EMC can manage to bring down the cost of in warranty repair including labor, parts, labs, engineering support, the savings from all these can be passed on to the customer. In this market, EMC might offer an extension of a 3 year warranty to a 4 year warranty that might help with ROI and TCO models.

Partners……

Support for new generation DMX’s installs extended to ASN Partners.

Some portions of Enterprise channels will be designed to work like Commercial channels promoting premier partners and ASN partners to perform some work on the enterprise machines. 

Model Numbers……

The million dollar question, will it be called Symmetrix DMX-5?

Or will it be called
DMX-5-XP (Extra Performance),
DMX-5-EF (EFD optimized machine),
DMX-5-V (extended support for Virtualization),
DMX-5-950 (same naming convention as before),
DMX-5-8 (8GB I/O),
DMX-V (You can think its V for virtualization or V for roman letter 5)

Let’s not get hung up on the model numbers though.

Names…….

Will EMC for the first time drop the name Symmetrix from this generation of machines, this name comes from the Moshe days.

Technology…..

EMC is known to make a big bang with technology with all its new product releases and has been a leader in bringing new technology to the market. Let’s talk about a few technological aspects to look forward to in the new generation of DMX.

The underlying DMX-3 design has been different than the DMX and DMX2 generation of machines. The DMX-4 design has been pretty similar to the DMX-3’s.

The point I am trying to make, EMC had a time frame of 5 years since the DMX2’s to come up with a radically changed DMX-5 design. Will a completely new design come to fruition with this generation of machines?

Enginuity Code……

A new Family Code is possibly due with this generation of DMX. May be an Initial release level of 5874.xx.xx.

Continued NDU (Non Disruptive Upgrade) Everything concept.

Introduction of PaPS (Plug and Play Support) with disk.

Size…….

2 Cabinet: where one Cabinet is for Controllers, and the other for 2.5 inch Flash Disk. This model will be optimized for Flash Drives only and will be lighting fast.

2 or more Cabinets where the 1^st Cabinet is for Controllers and the other ones for drives, the additional drive cabinets can be used for 3.5 inch drives or for 2.5 inch drives depending on the cabinet type you purchase. Also supported with Flash Drives.

Total Raw Storage……..

2048 TB (Double the capacity from DMX-4)

Cache…..

1024 GB Cache (First Enterprise Storage Array to hit 1TB of Cache)

Maximum usable memory: 512GB

Controllers…….

Each DAF, GbE, FICON, ESCON controllers might be subdivided into 8 slices (ports, processors) creating further condensation of controllers, I/O, footprint, drives per DAF.

Additional backend ports will be added with this.

Each processor might be 2.4 Ghz PowerPC Chip.

Introduction of Clariion Type Concepts in Enterprise Storage…….

Plug and Play for disk replacements, where presence of an EMC CE onsite might not be required. This is pretty common with Clariion and NetApp Systems today.

Conceptual change of Global Memory to Local Memory, where memory is part of the controllers and not a global memory pool and Flash drives are used for certain memory operations as a vault.

Microcode upgrades being performed by the customer like its done for the Flarecode today.

EFD’s……..

After support for 73GB and 146GB Flash, will might see 200GB , 400GB  and 500GB disk on this new generation of machines?

With the use of EFD’s in Enterprise storage would the concept of IOPS with Storage change to GHz & MIPS.

Introduction of 2.5 inch drives…….

With some OEMs introducing support for 2.5 inch drives, we might see EMC moving in the same direction.

The next generation Clariion’s might have similar drives in them too.

Symapi…….

Today the Symapi database resides on the Service Processor. Service Processors die; crash or get interrupted in middle of a change (provisioning, allocation, and configuration) and all of sudden the customer finds themselves in middle of crisis. All the change windows scheduled will have to be rescheduled, PSE’s dialing into the boxes to troubleshoot and fix issues, etc.

Introduction of an IP based (ethernet) connection to the DMX (talking about the DMX and not the SP) with multiple paths of communication. The SYMAPI databases will be locally stored on the DMX rather than the SP. This is similar to the VCMDB and the SFS volumes which already reside on the DMX.

Introduction of VMware ESXi into the Service Processor Environment to run multiple instances of SP Software and Windows for diagnostics, remote call home, etc. May be One VMware install can call home on the highest priority errors to location 1 and the second calling home with low to medium errors at location 2 and create two different queues for support priority.

Ethernet…..

As mentioned earlier, an introduction of IP based Ethernet management port, allowing SMC (Symmetrix Management Console) to interface, ECC and other Components can communicate through the same infrastructure.

Hardware……

Channel support which would include FICON, ESCON, GbE, FC, iSCSI, RF and some initial support for FCoE.

2048TB of storage in 5 cabinets can only be achieved with 2.5 inch drives.

Introduction of LP SATA Drives…..

Introduction of Low Power SATA Drives to conserve energy.

Plug and Play Support……

It sounds unreasonable, but if this can be incorporated into the DMX Platform, it will really take the overall platform to new levels with configuration, provisioning, customer interface, management, etc. Imagine if you want to add new drives to your current DMX, no BIN file change, just plug the drives in and configure through SMC.

High Efficiency….

At least 30% increase in efficiency, usage, savings, power and reduction in administration, management, support.

Added interface friendliness for SMC usage.

I/O Improvements……

8GB I/O per second Backend?

Improved Cache Partitioning , Mirroring and Priority Controls……

Further enhancements related to cache partitioning and cache mirroring, allowing customers to prioritize cache based on applications, times of the day, etc to certain set of drives or interfaces.

New Conceptual Design with BCV’s, DRV’s, Snaps and Clones……

As I say conceptual, I am not sure if history can change with this new generation of DMX machines or the new code. The mirrors, bcv’s, drv’s, snaps, clones are all treated as mirrored positions, configuration like RAID-5, RAID-6 is hard to manage.

If the code has been completely rewritten using new technologies this might be a reality, working more at a lun level rather than drive levels.

EFD’s and Optimizer…….

With EFD’s the use of Symm Optimizer is not deemed necessary, will optimizer become history?

Policy based support for Atmos…….

This will be one of the best features to look forward with the release of new DMX’s. Will DMX have native support for Atmos or will it be through a policy based engine as additional physical hardware.

Enhanced Support for VMware……..

This is a given, limitations with DMX and VMware with usage of LUN #’s, with the new DMX we will see additional native support for VMware integration features.

Enhanced RSA Integration…..

A lot was seen with DMX-3 and DMX-4 with access controls, etc. Further enhancements to security aspects of the storage.

Enhanced Support for RAID 5 / RAID 6, possible modification of one of the RAID designs to make the product faster………

Introduction of a new RAID type with this generation of DMX’s to compete with NetApp’s RAID-DP.

Storage Virtualization…….

This is really questionable, not sure where EMC wants to take Storage Virtualization.

Virtual Provisioning licensing…….

Virtual Provisioning included as part of the microcode and at no additional cost.

Native support for Deduplication?

Could this happen with the latest DMX’s

Built in SRM tools?

Some support for build in SRM tools into SMC, will help customers identify issues with the DMX.

Advancements with Green Infrastructure……..

This is a given, big marketing pitch, energy savings of 30% at least.

Advancements in Workflow and Automation…….

Further advancements with Workflow, Automation in new versions of ECC and SMC.

Initial Support for FCoE…..

Initial FCoE support has been released on the Clariions. We will see some initial support for FCoE on the new DMX generation. 

EMC has discussed this topic so many different ways in the blogosphere, I am pretty sure we will see some early adaption of it in this generation of DMX’s.

Hope I did cover a lot of ground in terms of new technology that we can look forward from EMC. 

As usual comments always welcome. 

© Devang for Gestalt IT, 2009. | Expectations with new Generation of DMX Technology
Read more posts categorized as All, Storage

SVC and USP-V will go in with no data migration and you can simply use them to virtualise your existing physical LUNs; that is the beauty of keeping things relatively simple and not relying on an additional abstraction layer such as WAFL. You can then migrate into the native SVC and USP-V formats at your leisure if you wish but you are not forced into making changes which potentially lock you in to the virtualisation vendor.

This is extremely important in today’s environment; firstly it minimises the amount of swing space that you require to do the conversion into a virtualised environment, secondly it minimises the amount of potential outage and thirdly it’s not a one way trip.

If all I want to do is reclaim unused disk and I am prepared to take the level of disruption that putting a V-Series into my environment; I suspect I might be better spending the money on analysing my existing environment and coming up with a better, more efficient lay-out and working at doing things better in my current environment.

For example, I would consider looking at Virtual Provisioning in my EMC environment; sure the licenses cost but it’s not going to be as expensive as going to a V-Series. I could convert from BCVs to Clones or Snaps. There’s a multitude of things which could be done before going virtualised. And once you’ve got your environment fixed, then take up NetApp and their space guarantee; don’t make things easy for them!

Don’t fix with new technology that which could be fixed by being a little smarter with your existing technology. You should always ask the question when being sold something new, can I do that already? It is amazing how often the answer is yes!

WAFL is a great technology with some great features but putting it in as a virtualisation technology would be an expensive mistake at the moment.  Put NetApp in because you want to use NetApp but don’t put it in to virtualise your existing environment unless you are prepared for a whole lot of work. Take it from me, I’ve looked at it.

If you simply want to virtualise and build a consolidated pool of disk, you might well be better looking at SVC or USP-V. If you are looking at re-engineering your environment, NetApp is one of many companies you should look at but you know that anyway.

© Martin for Gestalt IT, 2009. | V is for value??
Read more posts categorized as All, Storage