Chargeback and Standards

Implementing efficient standards is a given in any IT organisation.  However inevitably what should be a single standard always turns out to be partial implementation of multiple standards as time progresses.  This occurs because standards change; companies acquire other companies and their technology; companies expand and so on.  Sometimes it is simply too cost prohibitive to retro-fit new standards especially when things like server names are intrinsic to the installation of a software product (like databases).

Irrespective of the issues, it is still necessary to adhere to standards in order to implement effective chargeback and billing.  After all, resources have to be attributed to owners at some stage.  This means maintaining an efficient CMDB (Configuration Management Database) relating IT resources to business owners/customers within the organisation.  Whether that is via a spreadsheet or custom DB doesn’t really matter; it’s the content and its accuracy that counts.

As storage resources are provisioned, then so these standards should be maintained.  I always strive to ensure any resources can be mapped back to a server or business unit.  For example, zoning names should contain the server name as a minimum.  On storage arrays, storage groups should be related to the host/owner as should WWN visible names.  The format of fields should be consistent too, in order to enable scripts to process the values; choose consistent separators between fields (like the ‘_’ symbol) for example.

Chargeback Measuring Tools

If the standards are right, then extracting the data for billing should be simple.  The process may be as basic as using scripts to collect data; it may mean using the reporting features of your provisioning software, as most tools have some kind of reporting mechanism.  It may mean using a bespoke reporting tool, like Storage Fusion’s SRA product.  However it is achieved, the requirements are simple; map configuration information to the service you are offering and assign costs to customers/owners on that basis.  For example, if your charging is based on tiers of storage, the reporting/measuring process needs to be able to identify the tier types.  Making the Service Catalogue too complex at the outset can be self defeating if billing can’t easily be implemented.

Chargeback Measuring Interval

How frequently should you measure and/or charge?  I always find this subject interesting as there are ways for customers to avoid paying if they are clever.  Imagine the following scenario; billing is run monthly on the last day of the month and charges are accrued on assigned storage on that date.  However the storage team are able to turn around requests for provisioning/decomissioning within 48 hours.  So, for your project, you request space on the 2nd day of the month, run all the analysis you need, back up your data and return the storage 3 days before the end of month, at which time the resources are returned to the free pool and when billing occurs, the customer pays nothing.

Whilst this example is an extreme case, and I doubt whether most users would be in a position to take advantage, it demonstrates that a badly designed service catalogue, delivery structure and billing mechanism can lose money.   Alternatively, measuring utilisation on a daily basis could be time consuming and expensive to operate, therefore a compromise has to be found and this will depend on your specific circumstances.  Some options could be:

• Bill monthly on a fixed date; assume customer inertia will mean very little “cheating” of the system will occur. Build that into the overall cost.
• Bill monthly and include any provisioning requests made that month – a minimum charge of one month regardless.
• Sample data daily or weekly and bill monthly based on average consumption.

There are no specific rules to abide by here.  What’s important is the way in which billing, product catalogue and operational management occur are all lined up and consistent.

© Chris for Gestalt IT, 2010. | Four Pillars – Service: More On Chargeback
Read more posts categorized as Featured, Storage

Definition

It’s worth pausing for a moment and discussing the terms Chargeback and Billing.  When computing was first made available as timesharing, customers were billed for their usage of the shared system.  The billing unit may have been time, CPU resources or some combination of metrics that represented utilisation.  Mainframe resources were so expensive that there had to be an efficient charging mechanism.   The concept of billing is something that was intrisically built into the mainframe design and even to this day, resources can be tracked using records produced by SMF (System Management Facility) and reported on through RMF (Resource Measurement Facility).  So billing represented a method of charging for usage that wasn’t directly related to the underlying hardware.

Chargeback implies a different methodology where the direct cost of delivering the service is charged back to the customer.  This can include people costs, but typically hasn’t, only covering the hardware provided itself.  Chargeback has its place, but when looking to develop a service, isn’t as flexible as billing.  All too often, chargeback is tied to a poorly implemented service catalog (or non-existent one).  Whilst the customer may pay for their equipment, there isn’t any flexibility when it comes to hardware replacement as the customer is aware of the technlogy used to deliver their service (and may be unwilling to move to new, untried hardware).  Here are a few additional chargeback/billing combinations that could be implemented:

• No chargeback – IT has a budget and they provide the resources to the business.  When resources are exhausted, the business have to justify or provide additional funds.
• Consumption-based – customers are charged directly for their usage.
• Shared-usage – customers are charged a share of the costs, not directly related to their usage, but perhaps size of business unit.
• Dedicated – customers are charged the whole cost of acquiring the technology.  Ths doesn’t work well for shared environments.
• Service-based – customers are charged for a service provided; this isn’t directly related to the specific technology in use.

Rationale

Whether you are implementing chargeback or billing, there needs to be a good reason for implementing.  Here are a few for consideration.

• To Reduce Costs – If resources appear to be free they will be consumed inefficiently; charging for usage helps controls this.
• Improved Utilisation – Being charged in proportion to your usage makes customers validate whether they really need the storage they are using.
• Improved Efficiency – this goes hand in hand with utilisation, however charging customers for storage can enable tiering to be implemented more efficiently.
• Charging Fairly – there will always be sensible customers and abusers (the broadband market shows us that).
• Manage Demand – It is possible to make charges both time and planning dependent (more on that later).
• Manage Tech Refresh – Abstracting cost and service catalogue from the hardware means new/cheaper/efficient technology can be introduced more easily.

What’s clear from the above points is that chargeback/billing can be used to change customer behaviour; users can be incentivised to be more efficient or to use cheaper technology.  Structured correctly, the overall cost of delivery of storage can include refresh funding, so as old devices are decommissioned, the cost of data migration is part of the overall charge.  I see this as one of the major issues with the way customers pay for their technology; the overall costs in the lifecycle of deployment, operation and refresh simply aren’t considered.

Metrics

What’s the best way to charge?  Here are a few typical metrics:

• Per GB of storage used.
• Per port on the SAN fabric.
• By Tier of storage.
• By contention ratio of storage port (higher cost for fewer hosts on a shared port)
• Charge for replication (both local and remote)
• Charge for deduplication (which may be a lower cost)
• Charge for thin versus thick provisioned LUNs
• Charge for SAN network bandwidth
• Charge for multi-path software
• charge for online backup copies
• charge for offline backup copies

Whatever metrics are used, the key intent is to charge for customer use of a service.  This needs to be abstract enough to be disconnected from technology, so charging for fibre channel ports may be too prescriptive; the cost may be described as “to be connected to the SAN” in general, providing a blended charge that would cover iSCSI, Fibre Channel or FCoE connectivity.

Implementing a Chargeback Process

As part of the implementation process, it’s worth considering having billing/chargeback principles established.  These can be provided to the customer.  Here are some examples:

• The charging model will be based on resource consumption of each user independently (e.g. user changing their utilisation doesn’t affect another user)
• Charging costs will be reviewed and changed on an annual/bi-annual/quarterly basis from 1 Jan 200x
• Charging will be based on storage in use on 28th day of each month
• Charging will/will not be based on utilisation (rather than allocation)
• Charging will be attributed at the host/server/LUN/file level
• A target of 100% cost recovery is the target goal
• Charging may result in an IT surplus/deficit from year to year, but will be a non-profit business
• Billing charges will be based on the published “Storage Catalogue”
• Users of equipment classed as legacy will be notified 6 months in advance of technology acquiring legacy status
• IT/Storage Team will strive to deliver price stability and/or reductions year-on-year
• Chargeback will be implemented as evolution rather than revolution

The internal cost of delivery of storage will include:

• Hardware and software costs
• Additional feature licences
• Power/cooling/space (environmental costs)
• People costs
• Training
• Network costs
• DR costs

There may be more, depending on how your technology is delivered (for instance managed data centres), but what’s essential is to baseline what it takes to deliver the service.  Quite simply the process would be:

1. Identify service cost components (as above)
2. Identify consumption metrics (service charging units)
3. Measure use
4. Model costs based on consumption metrics
5. Bill customers.

Other considerations, which I’ll save for future posts are:

• Standards – how they are important to chargeback
• Measuring tools
• Measurement interval
• Incentivising customer behaviour in favour of technology refresh
• Outsourcing some components
• Determining the customer
• Forecasting/Capacity Planning

There’s lots more to come, feedback on the article so far is very welcome.

© Chris for Gestalt IT, 2010. | Four Pillars – Service: Chargeback
Read more posts categorized as Featured, Storage

What’s in a Name?

Some people call it Service Catalogue or Product Catalogue, even storage tiers.  The name isn’t too important, however what’s essential is that the Service Catalogue should exhibit a number of key components:

• It should be abstracted from technology.
• It should provide service metrics – explaining what you get for your money.
• It should be able to scale up or down to meet customers requirements.
• It should provide charging information based on simple billable units.

Technology Abstraction

For those familiar with delivering IT, the concept of offering a service that is abstracted from the technology itself may seem a little confusing.  However, by continually referring to technology, organisations tie themselves into being able to offer only that technology itself as part of their service catalogue.  For example, if your “tier 1″ storage offering consists of 15K 300GB FC drives in an EMC DMX-4 array, then customers will expect you to deliver tier 1 storage from that kind of box.  What happens if EMC can offer a lower cost alternative through their latest technology like V-Max?  What happens if 300GB drives reach EOL?  What happens if another vendor (e.g. HP or Hitachi) can offer the equivalent technology at 20% reduction in cost?

By retaining a connection between the service offering and storage, IT providers restrict their ability to introduce new technology that can reduce cost and improve service.  If however, service tiers or product offerings are suitably abstracted from technology, then costs can be reduced by taking opportunities to (a) implement new, cheaper technology or features (b) implement a competitive environment with multiple vendors.

Of course, it’s easy to talk about bringing in cheaper technology, either by using newer, cheaper hardware or by bringing in a new vendor who is prepared to take a beating on price just to get into a new account.  The implications of this kind of disruption may negate the potential savings because existing processes or architectural design can’t cope with the change.  In many organisations the process for storage deployment is based around the hardware product, as is reporting and billing.  The clear message here is that it’s essential to ensure both the operational and engineering functions will work across multiple varied technologies.  The subject of new technology adopotion and integration will be the subject of a future post.

Comparison with Other Industries

As mentioned in previous posts, the concept of delivering Storage Services is typically compared to utility companies.  Consider our functional requirements for service in comparison to say, an Electricity or Phone company.

• It should be abstract from Technology. This rings true for the delivery of say, electricity to a domestic dwelling.  In the UK, everyone expects to receive 240V at 50Hz as a standard.  Changing to another supplier doesn’t require a change in electricity standard; the process of electricity generation doesn’t affect the end delivery, whether it is generated from wind, wave, nuclear or coal.
• It should provide service metrics. Whether explicitly written into a contract or not, I think most electricity users would expect their service to be available on a 24×7 basis.  In addition, the supply should be guaranteed to remain within certain tolerances – e.g. 240V±10% and 50Hz±1% for example.  Deviation outside of these standards would have a seriously adverse effect on equipment using the supply.
• It should scale. Unless you’re a commercial user, most electricity companies would expect to be able to increase and decrease their electricity consumption without consideration of how that supply was provided and without contacting the provider.  In fact in the UK the National Grid are responsible for monitoring demand and bringing supply in and out of use to match requirements.  This is all part of the service and therefore included in the cost of supply.
• It should consist of Billable Units. Check your electricity bill; you’ll find your charge is based on a price per kWh, a standard unit of measurement of power.  Your charges are directly related to your consumption and not to how the power was generated.  It’s worth noting that some providers will offer multiple tariffs, for example providing cheaper electricity at night or tying consumption into green issues.  These tariffs are analagous to storage tiers.

Building a Service Catalogue

How do you go about designing and implementing a storage catalogue?  Here are a few pointers:

• Look to include functional requirements (availability, response times).
• Look for cost & service differentiation.  Some organisations implement many different tiers of storage or product offerings.   Implementing multiple tiers can become expensive and if they’re not all utilsed efficiently, then the potential cost benefit is lost.  Implementing multiple tiers offers the ability to reduce costs by placing less active or less demanding data on cheaper resources.
• Look to drive behaviour; implement services that penalise lack of adherence to standards and so on, but reward forward planning and utilising forecasted resources.

A Basic Template

Here’s a generic example of delivering a service catalogue for Enterprise SAN Storage.  It keeps things simple by offering only three tiers;.  each can include extra features such as local or remote replication.

• Gold - Availability – 99.999%; Response time: <10ms on 99.9% of I/Os sampled over a 5 minute period. Connectivity – FC – $10/GB/Month
• Silver - Availability – 99.999%: reponse time <15ms on 99.9% of I/Os sampled over a 5 minute period – conncetiivty – FC – $5/GB/month
• Bronze – Availability 99.9%; reponse time < 20ms – connectivity = iSCSI – $2/GB/month

Additional “bolt-ons” for each tier include:

• Local replication. Add a local replica for an additional tier price per month (gold replica for $10/month)
• Remote replication. Add a remote replica for the cost of the replica plus 10% (e.g total cost per GB for gold is 22$/GB/month

Additional comments: Billing is based on average configured storage over the course of a calendar month, measured at 00:01 each day.  Totals are based on the storage used per server, rounded to the nearest GB.  Costs assume no more than 1 allocation/deallocation per server per month; additional requests will incur an additional configuration charge.  Costs do not include and host software or hardware required to connect to the storage.

Every organisation will have their own specific requirements that determine the best way to structure a storage catalogue.  In future posts, I’ll discuss more about the process of developing the catalogue, how it integrates with product selection and operational management.

© Chris for Gestalt IT, 2010. | Four Pillars – Service: The Service Catalogue
Read more posts categorized as Featured, Storage

My Four Pillars of Storage Management are defined in this initial post. To recap, I define Service as:

“Offering of services to business customers via a service catalog and measuring the ability to deliver to the business through KPIs and Service Level Agreements.”

If you look back to the early days of computing, the computer department was almost a law unto itself.  Access to the mainframe and other resources was at the whim of the Systems Programmer, a bearded, sandal-wearing, social skill-lacking individual who might just do what you ask him when he was ready.  Ricky Gervais parodied this character perfectly in The Office; the IT expert within the office was also expert at everything else including karting (”Oi, no professionals….”).  By the way, before everyone starts flaming me – I was a Systems Programmer on mainframe for a few years at the start of my career, so I speak from personal experience.

Fortunately IT has matured and we’ve mostly moved on to a service-orientated method for delivering computing (and in this case storage) resources.  To effectively deliver storage resources within an organisation, it is essential to move to a service-based model where storage is provided as a service offering that is planned and managed, rather than simply on-demand.

The typical example that is always quoted is to compare computing resources to utility companies providing electricity, gas and water.  Personally, I prefer in some respects to look at the telecoms market for better examples, in particular mobile ‘phone companies.  What all of these companies have in common is that they offer you a service and then charge you for your consumption.  On a periodic basis you are provided with statements and payment is taken.  Most critically, you have no real knowledge of how the service is delivered, what underlying technology was used e.g. Gas, Nuclear, Green, Coal forms of electricity generation.  To a certain degree you have no need to care unless of course the company can see some market advantage from promoting their use of a certain technology (for example, energy companies offering “Green” tariffs).

So it should be with storage.  The delivery of storage resources should be thought of as providing a service.  Moving to that model requires thinking of the following components.

In subsequent posts, I’ll discuss more on the Service details and explore how to achieve setting up service-based storage delivery.

© Chris for Gestalt IT, 2010. | Four Pillars – Service
Read more posts categorized as Featured, Storage

1. SERVICE – Offering of services to business customers via a service catalog and measuring the ability to deliver to the business through KPIs and Service Level Agreements.
2. OPERATIONAL – Implementation of best in class operational processes for delivering storage to customers, including provisioning, monitoring and capacity planning.
3. COMMERCIAL – creating a commercial framework which shares risk with the storage vendor and ensures purchasing can be made as efficiently as possible.
4. TECHNOLOGY – Selection and implementation of the best of breed technology solutions, designed and implemented to meet business requirements.

In reality, each of the Pillars could be treated as a subject in their own right, however my reason for combining them together is to emphasise that one pillar alone doesn’t support the Storage Infrastructure.  They need to act together co-operatively to deliver more than the components do individually and to stop them clashing with each other.

Take a simple example.  Many organisations configure their host-based software to be platform dependent.  Think of the firmware, driver, HBA and MPIO stack implemented today.  For EMC products, you may choose PowerPath;  For Hitachi, HDLM; for IBM perhaps SDD.  Using any of these software products immediately creates restrictions around the storage a host can be connected to.  This makes moving between platforms more complex; it removes the ability to easily bring in another vendor and/or move data around a complex environment.  By standardising on firmware/drivers/MPIO software that works with all storage in a multi-vendor environment, there’s no issue with moving data around or having a single host connected to multiple storage arrays.  This then makes the purchasing question easier by removing the barriers restricting storage & server platform combinations.  Vendors know that next big purchase will work across all platforms and that their technical advantage is removed.

Negotiation on price becomes more interesting and useful at that point.

I described this as a simple example; in the real world we’re moving to infrastructure towers, with vendors looking to increase their lock-in, which makes this discussion all that more relevant.

Look out for more 4 Pillars posts over the coming months.

© Chris for Gestalt IT, 2010. | The Four Pillars of Storage Management
Read more posts categorized as Featured, Storage