Every technician develops a set of tools and practices that serve them well. In this post, I present my best practices for enterprise IT.

Although many people are cynical about the whole idea of best practices, I’m a believer. I think that such beasts do exist, just that too many companies, analysts and especially consultants spend too much time applying the label to whatever works in their best interest at the time. To counteract this cesspool of non-best practices, I thought it best to put down a few ideas of my own. Following are four fundamental best practices I have distilled from almost 20 years in enterprise IT. I wonder if you agree with them.

Before proceeding, I suggest that you take a look at the three principle tests that I use to define a best practice.

Best practice 1: Minimize Complexity

IT has a tough position in business. We are ignored while everything is working well, and in trouble when things start falling apart. We are held accountable, but generally cannot control our own destiny. The only way we have a fighting chance is to minimize the complexity of the solutions we must support.

As I said in Sailing the Titanic (Why We Need ILM and Then Some!): “To make a timely (and tired) Harry Potter analogy, IT are the house-elves of the business – powerful but subservient, with little input into what happens above and around them.”

This is where the current trends of virtualization, consolidation, and standardization comes in. A single administrator can manage a far-larger environment if it’s standardized, and these systems tend to be far more reliable as well. Sadly, minimizing complexity is the antithesis of a true hacker’s heart. Who wouldn’t want to play with the latest and greatest toys? Why not build an intricate and interconnected system? But these solutions fail my “best practices” test: They’re risky, unusual and just plain imprudent.

Best Practice 2: Use the Right Tool For the Job

Any woodworker or mechanic will tell you that using the right tool makes any job easier. Even if it works, it’s just bad practice to masquerade network-attached storage (NAS) as a block device or a storage area network (SAN) as a collection of shared files. I am always skeptical of so many oddball innovations trumpeted by vendors large and small because they seem to fail the “right tool” practice.

I wrote more about this in Use Process Solutions For Process Problems, Technical Solutions For Technical Ones and Why Do I Ignore NAS?

But sometimes they work out just fine. You see, best practices can’t predict the future or what works in a certain situation. It’s better to minimize complexity and stick to what you know than to press for the perfect solution every time. It wouldn’t be right to bring SAN storage into an all-NAS environment just for a single database application, for example. Although we generally shouldn’t try to drive a nail with a screwdriver, it can be the best option when a hammer can’t be found!

Best Practice 3: Prepare For Failure

The hacker in me is tempted to stop tinkering as soon as a system is functional, but the realist in me knows that I have to make it twice as good. You can’t prepare for every eventuality, but you can seek out probable points of failure and harden them.

For example, see Chris Evans’ discussion of Choosing Between Monolithic and Modular Architectures

Many of the fundamental best practices in enterprise IT come down to this. Redundancy from RAID and N+1 components, the widespread use of multi-path technology, and high-availability design are all examples of preparing for the worst. We would not bother with all of this if we didn’t know that errors occur even in well-engineered systems, and that everything we build is sensitive to failure.

Best Practice 4: Align Expectations with Reality

We all face conflicting demands, and we are tempted to offer best-case responses to keep everyone happy. But excessive optimism leaves IT folks stressed and end users frustrated when things don’t go right. It’s time to adjust expectations.

See The Techie/Business Schism, where I talk about the looming disconnect even within IT

Concepts like service level agreements (SLAs) and business impact analyses come from an innate knowledge that the real world doesn’t always conform to the best-case assumption. Although we don’t have real control over user demands, we can try to set correct expectations by admitting what we can and can’t do realistically.  Don’t let those who rely on us labor under misconceptions of our capabilities.

Stephen’s Stance

We can’t let others lead us astray with talk of “best practices”, but we can define some of our own. My three-part best practices test, and these four general practices, have served me well over the years. What are your best practices? Image credits: Dave’s Bike Tools by bre pettis

© Stephen Foskett for Gestalt IT, 2010. | Four Fundamental Best Practices for Enterprise IT
Read more posts categorized as Featured, Storage, Tech

First of all, let’s take a look at the individual terms:

High Availability

According to Wikipedia, you can define availability in the following ways:

The degree to which a system, subsystem, or equipment is operable and in a committable state at the start of a mission, when the mission is called for at an unknown, i.e., a random, time. Simply put, availability is the proportion of time a system is in a functioning condition.

The ratio of (a) the total time a functional unit is capable of being used during a given interval to (b) the length of the interval.

And most online dictionaries seem to have a similar definition of availability. When we are talking about HA, we imply that we want the functioning condition of your system to be increased.

Going by the above you will also notice that there is no fixed definition of the availability. Simply put, it would mean that you need to put your own definition in place when talking about HA. You need to define what HA means in your environment. I’ve had customers that needed HA and defined this as the system having a certain amount of uptime, which is one way to measure it.

On the other hand you would be hard pressed if you were able to work with your system, but the data that you were working with was corrupted because one of your power users made an error during a copy job and wrote an older data set in the wrong spot. This would mean that your system is in itself available. You can log on to it, you can work with it, but the output you are going to get will be wrong.

To me, such a scenario would mean that your system isn’t available. After all, it’s not about everything being online. It’s about using a system in the way you would expect it to work. But when you ask most people in IT about availability, the first thing you will likely hear is something related to uptime or downtime. So, my tip to you is once again:

Define what “available” means to you and your organization/customer!

Disaster Recovery

Let’s do the same thing as before and let’s turn to some general definitions. Wikipedia defines disaster the following way:

disaster is a perceived tragedy, being either a natural calamity or man-made catastrophe. It is a hazard which has comes to fruition. A hazard, in turn, is a situation which poses a level of threat to life, health, property, or that may deleteriously affect society or an environment.

And recovery is defined the following way (when it comes to health):

Healing, or Cure, the process of recovering from an injury or illness.

So, in a nutshell this is about bouncing back to your feet once a disaster strikes. Now again, it’s important to define what you would call a disaster, but at least there seems to be some sort of common understanding that anything that would get you back up and running after an entire site goes down, usually falls under the label of a DR solution.

It all boils down to definitions!

When you talk to other companies or vendors about HA and/or DR, you will soon notice that most have a different understanding of what HA and DR are. Your main focus should be to have a clear definition for yourself. Try to find out the importance and value of your solution and base your requirements on that. Ask yourself simple questions like for example:

• What is the maximum downtime I can cope with before I need to start working again? 8 hours per year? 1 hour per year? 4 hours per month? What is my RPO and RTO
• How do I handle planned maintenance? Can I bring everything down or do I need to distribute my maintenance across independent entities?
• Can I afford the loss of any data at all? Can I afford the partial loss of data?
• What if we see a city-wide power outage? Do I need a failover site, or are all my users in the same spot and won’t be able to work anyway?

Questions like these will help you realize that not everything you have running has the same value. Your development system with 6000 people working on it worldwide might need better protection than your productive system that is only being used by 500 people spread through the Baltic region.

Or in short

Knowing what kind of protection you need is key. Fact is that both HA and DR solutions never come cheap. If you need the certainty that your solution is available and able to recover from a disaster, you will notice that the price tag will quickly skyrocket. Which is another reason to make sure that you know exactly what kind of protection you need, and creating that definition is the most important starting point. Once you have your own definition, make sure that you communicate those definitions and requirements so that all parties are on the same page. It should make your life a little easier in the end.

© Bas for Gestalt IT, 2010. | How do you define high availability and disaster recovery?
Read more posts categorized as Featured, Server Virtualization, Storage

HAM combines conventional ingredients to create a whole new flavor

High Availability Manager consists of three main components:

1. A conventional multi-pathing agent like HDLM on each server. This enables the server to continue accessing the storage if one USP fails. It will “think” it’s talking to a single storage target, but will actually be talking to two USPs that can be metro distances (60-100 miles) away, given proper connectivity. Microsoft MPIO will probably be supported as well, and VMware native multi-pathing (NMP) should come shortly after release. Don’t hold your breath for PowerPath to be officially supported in the short term, but it ought to work fine without changes.
2. Existing TrueCopy synchronous storage replication technology will keep the data and quorum disks (see below) in lock-step. This gives the limitation on distance between systems, since latency is the enemy of storage protocol performance. Once the arrays move too far apart, write performance will suffer on the local array while it waits for data to be copied.
3. Conventional clustering technology with a heartbeat and shared quorum disk lets both arrays know what’s going on. The quorum “lives” on the remote side, with that secondary array watching to make sure the primary array is still running. If the heartbeat goes away, the secondary array marks the HAMmed LUNs read/write and starts handling I/O from the servers, which will just have failed over.

So HDS’ secret HAM sauce is ketchup. There’s no amazing new technology here, and maybe that’s for the best in the kind of huge, conservative enterprise environments that will use this product. The big change was in programming the USP controllers to monitor the quorum disk and orchestrate the entire failover. In fact, this might even be considered a feature of TrueCopy, not a standalone product.

HDS ought to reconsider one element of the HAM pitch, however. Although it will undoubtedly yield a very highly-available storage architecture, nothing provides 100% availability. There are many moving parts involved, and unplanned outages can still happen. Multipath driver bugs are not unheard of: Back in the day, one version of a certain three-letter company’s product just plain refused to fail over! An instantaneous outage on one of the host channels or a skipped heartbeat on the USP controllers could also cause a failed failover. Plus, there is no provision for automated fail-back. Once the failover occurs, the system would certainly be operating in a degraded-availability mode and would require a (planned) outage to re-establish operations.

All that being said, HAM remains an attractive offering for shops with multiple USPs visible to critical servers. They can turn on the HAM software with existing hardware and add peace of mind, knowing that everything is that much more available. One aspect that really impresses is the fact that each USP can be running a different firmware, reducing the risk of upgrade-induced outages. Once it ships to customers (in the fourth quarter of this year), we will have to consider the (as-yet unnamed) cost.

And what about the fact that HAM will also allow seamless upgrades to a new generation of USP hardware? HDS still isn’t talking about that possibility!

© Stephen Foskett for Gestalt IT, 2009. | HDS High Availability Manager: How It Works
Read more posts categorized as Exclusive, Featured, Storage, Top Story