A while back, I discussed speculation from EMC around Emulex’s proposed cloud-block storage appliance, E3s (Enterprise Elastic Storage).  With my current focus on Cloud Storage, I thought it would be good to delve a bit deeper into some of the aspects of why block-based cloud computing could prove tricky and why without an appliance it may be impossible.

Block Storage Legacy

Today, block-based I/O still uses the SCSI protocol to communicate between a host/server and a disk storage device.  SCSI has been around since 1981 when devices were physically connected to the server itself using a controller board and old-style ribbon cables.  Obviously we’ve progressed somewhat since then and seen the virtualisation of the physical SCSI interface into Fibre Channel and IP SCSI implementations (and in the future FCoE).  Both FC and iSCSI have removed the need for a dedicated SCSI controller card and replaced it with Host Bus Adaptors (HBAs) and Network Interface Cards (NICs).  Irrespective of this, the underlying communication protocol remains the same and the concept of the “Initiator” (host/server) and “Target” (storage device) persist today.  The Initiator starts (or initiates) an I/O request; the target services that request, reading or writing data.

Writing In Blocks

We need to focus for a moment on the concept of block-based I/O versus file-based I/O.  Block-based I/O has no concept of the format of the data being written to the block device (let’s call it a LUN).  This is in contrast to file-based I/O where the storage device understands the data format and manages the content accordingly, ensuring data access is serialised correctly and that files are held in a logical structure (a file system).  Unfortunately block-based I/O is just “dumb” storage and the host itself is responsible for overlaying a file system onto block-based devices.   These JBBDs (”Just a  Block-Based Device”) can be used singly or combined in complex ways to create the file system the host sees.  This combination can be achieved using native Logical Volume Managers (LVMs) on the operating system, or add-ons such as Veritas Volume Manager.  Consequently, an individual LUN could contain an entire file system or only a small component of one.  Either way, the host operating system depends on one feature of the storage device to ensure data integrity and that’s Write Order Consistency.

Write Order Consistency

Preserving the order in which data is written to disk is a fundamental requirement for modern Journalling file systems like NTFS.  Retaining write consistency ensures the file system can be recovered in the event of a server failure or failure of the link between server and disk.  Ordered writes are also essential where data is replicated from one storage array to another usually at a remote location.  In the event that the primary array is lost, the file system can be recovered in a consistent fashion on the remote device, even if it isn’t 100% up to date.

Remote replication can occur either synchronously or asynchronously.  In synchronous mode, I/Os are acknowledged from the remote array before the I/O is confirmed as completed to the host.  This ensures both the primary and remote copies are write-order consistent because the host doesn’t receive acknowledgement of I/O completion until both the local and remote copies are written.  Write-order consistency is implicitly guaranteed.  However, the penalty for this guarantee is the increase in latency (or host I/O response time) which results and increases with the distance between the two devices.  Asynchronous replication is slightly different.  Write I/Os to the primary array are acknowledged immediately, then queued up for writing to the remote device.  The delay in writing data to the remote location is dependent on the bandwidth available and the latency (effectively the distance) between the devices. In any event, as long as write-order is maintained, the remote copy can be recovered even if it doesn’t represent the absolute latest copy of data.  One last consideration.  We touched earlier on how LVMs can combine single LUNs to create complex file systems.  When asychronous replication is involved, write I/O for a single entity like a file system need to be treated together for write-order consistency.  Therefore Consistency Groups allow multiple LUNs to be grouped together for write ordering, ensuring that all I/O for the file system is ordered correctly.  This requirement doesn’t exist for synchronous replication as the consistency is guaranteed by ensuring the write I/O has completed on both the source and target array before acknowledgement to the host.

The Need for an Appliance

Knowing how write-ordering affects consistency is important in understanding how a block-based device would be replicated into “the cloud”.  Due to latency issues, it is unlikely that synchronous replication would be offered as a method of replicating data from a host server into Amazon S3 or EMC Atmos.  Instead, the most likely process will be asynchronous processing and that means installation of a dedicated appliance.  The question is, where in the data path should it sit?

The Splitter

There’s no requirement for a host/server to be connected into a storage array in order to utilise cloud storage.  Instead, at some point in the data path between host and local disk, a copy of the write I/O needs to be taken.  This could occur at the file system level using a host agent or in a SAN environment could happen in the fabric or from the array itself.  Wherever data is taken from, some form of “I/O splitter” is needed to capture write I/O as it is being transferred to disk.    This technology already exists today in products like EMC’s RecoverPoint and Brocade’s Data Mobility Manager.

So here are our requirements for a block-based storage protocol:

• Write Order Consistency
• Consistency groups
• An I/O splitter or replicator

A Theoretical Implementation

Here’s how I think block-based storage could be implemented.  I’ll use the Atmos and Amazon S3 protocols to demonstrate the process.  Firstly, data will be stored in blocks.  Both S3 and Atmos store data as objects and so each object will need to represent a block.  The file system structure can be used to store individual LUNs, with a directory representing a LUN.  For example LUN 12, block 12343 could have the object name:

\S3\Array1\LUN12\12343

It’s worth noting that there’s a distinct difference between the way Atmos and S3 implement updates to objects.  S3 replaces the entire object, whereas Atmos allows part of an object to be updated.  So, Atmos could store an entire LUN as an object whereas S3 can’t, unless the entire LUN is replaced on each write.  Clearly that’s impractical but does indicate that each API implementation will have certain benefits and disadvantages.

So, how big are these blocks going to be?  Ideally, they’d be as small as a typical hard drive block at 512 bytes, however blocks of this size will seriously hamper throughput if write consistency is to be maintained; imagine 5ms latency into the cloud; that’s only 200 IOPS and consequently a throughput of 100KB/s. What’s more practical is a block size matching the O/S  file system and/or database, say 8KB.  Even at this size, with 5ms latency and a single thread of I/O, that’s only 1.6MB/s throughput.

Obviously this level of throughput is not going to be acceptable and there’s a real sticking point here.  The cloud isn’t intelligent.  It will write data as it’s received.  There’s no locking control and the delivery mechanism could be unpredictable.  If writes are issued in parallel, there’s no way to guarantee the I/Os are written to the cloud in the right order.  So perhaps a different approach is required.  Data writes to the target LUN need to be written in a log format, with the name of the object comprising both the block number and a sequence number.  This could be something simple as follows:

\Atmos\Array1\LUN12\SequenceNumber-BlockNumber

e.g.   \Atmos\Array1\LUN12\123445343434366-12343

As the LUN is written to, the sequence number (unique to the LUN or consistency group) is incremented for each write.  The I/Os can then be written in parallel as the sequence numbers track what has and has not been received.  At this point there are two choices – retain all the block updates (unlikely due to the growth in storage usage) or post process the writes, deleting all the written blocks where another later copy exists and where there are no gaps in the sequence numbers.  If there is a gap, then the LUN writes are only guaranteed back to the point where the sequence number gap occurred.  Restoring the LUN for access means processing the LUN block data before it can be read again by the host.

Summary

OK, so this post presents an idea of some of the issues involved in writing block-level data into the cloud.  Data needs to retain integrity and consistency, but performance and throughput are an issue.  Cloud storage has no intelligence, so writing and managing data needs to be handled somewhere, probably using an appliance.  The appliance guarantees the data integrity which can’t be achieved with the cloud alone.  Each Cloud Storage API implementation will have similar features, so using generic CRUD (Create/Read/Update/Delete) commands on objects representing blocks means any service could be used to store data.  It also enables data to be replicated between services so vendor lock-in can be avoided.

I’d be interested in receiving feedback to see if anyone else has thought how block-based cloud storage could be achieved.

© Chris for Gestalt IT, 2009. | Cloud Computing: Block-Based Storage
Read more posts categorized as Cloud Computing, Featured, Storage

Dave Graham posted an interesting article on his blog yesterday, relating to a new product from Emulex.  Called E3S or Emulex Enterprise Elastic Storage, the appliance (as it appears to be being positioned) allows block-level data to be migrated into the cloud for later access.

Now there are a few interesting points here.  First, the discussion related to block-level data, that is data written to and from LUNs rather than files.  Second, data is maintained consistently (“consistent copies” to quote Dave).  Third, data is encapsulated by the E3S device and returned back to the host as required.

I’m a little confused as to what the offering actually is; is it a backup solution; is it a replication solution?  Let’s think about this in more detail.

Block-Level Integrity

Anyone who knows how large enterprise arrays work will know data consistency and integrity is king.  The array itself has no concept of the format of the file system written onto the LUNs and consequently storage arrays must maintain write sequence integrity in order to maintain file system consistency.  This isn’t a problem with a single array and synchronous replication as the I/Os are written to cache in timestamp order and replicated to remote storage arrays in the same format.  Asynchronous replication operates in a similar manner, except that LUNs are grouped together based on their consistency requirements – usually all the LUNs presented to a single host or application.  If write order is not maintained or data fragments are lost, then the target copies can be rendered completely useless.

Data Integrity With Unreliable Transportation

LUN replication requires consistency and data integrity. Replication into the cloud occurs across an IP connection which doesn’t have the same performance guarantees and reliability as a dedicated point-to-point network like Fibre Channel.  The problem therefore, is to move data reliably to the cloud and provide integrity checking, packet resends and manage an unpredictable performance profile.

Unique Technology

IP-based replication technology which deals with the performance and reliability issues already exists in the market today.  In fact, EMC have a product doing just this – RecoverPoint.  They also have Open Replicator, which writes data to unlike devices across fibre channel networks.  There’s also Axxana’s Phoenix appliance which can move data across mobile networks.  So what exactly are Emulex offering that makes it unique compared to these technologies in the marketplace today?  We can only wait to find out.

Questions, Questions

So what questions should be asked of this kind of replication technology?  Here’s a few:

• How is data integrity maintained in replicating LUNs to “the cloud”?
• Does the E3S appliance cache data as it is forwarded to “the cloud”?
• What redundancy and integrity is built into the E3S appliance to ensure no data loss?
• What level of throughput can the E3S appliance maintain?
• How is multi-LUN replication integrity managed?
• How is multi-array replication integrity managed?
• Can data be accessed directly from “the cloud”?

There are a couple of other posts out there from Chris Mellor and Beth Pariseau.  You can find them here:

• Emulex Cloud Storage Gateway Revealed – Chris Mellor
• Emulex Plans Cloud HBA – Beth Pariseau

© Chris for Gestalt IT, 2009. | Cloud Computing: Emulex Enterprise Elastic Storage (E3S)
Read more posts categorized as All, Cloud Computing, Featured, Storage

The recent surge by storage networking vendors (Cisco and Brocade et al) to develop and roll out Ethernet based storage networks means that Storage and Networking teams are going to need to work together. The most common reaction from Storage professionals has been “I wouldn’t trust my storage data on the Data Network, I can’t rely on it!”.

Which is ridiculous and, ultimately, futile. As a Data Networking designer, I can assure you that is what the Voice Networking people said as the IP Telephony/VoIP technologies dismantled their careers. This isn’t any different.

FibreChannel is a single function network. Data Networks are multifunctional. And because of that, they are cheaper.

Why is Brocade walking away from FibreChannel ?

Brocade basically invented FibreChannel so that they could create a market that suited themselves very nicely. Which is fine because a number of other vendors joined in which validated the market that Brocade domainated.

But since Cisco bought the MDS its been clear that they were not going to settle for being number two and have been aggressively attacking the storage networking marketplace. Cisco knows that its strength is its existing customers and data networking, so the obvious progression is to move storage to the Data Network.

Enter FCoE………

Cisco picked a team of engineers and created a startup, Nuova Systems, with a brief to develop Ethernet Storage networking. They developed Fibre Channel over Ethernet and series of hardware products that had the silicon to support the requirements.

And they are serious requirements. The largest and most critical storage networks need low latency, low jitter and lossless network to ensure good operation. (Lets not talk about the fact that the storage marketplace has been so completely oversold on the story of lossless and low latency networks with FibreChannel that they now believe it is actually true).

So Nouva Systems built an Ethernet switch that has the necessary capabilities. It doesn’t seem to have been hugely difficult since it only took them a couple of years, and their are competitive products from companies like Woven Systems, that already have most of these features before they started.

They also started the process for a number of new Ethernet standards that allowed the storage technologies to exist.

So Cisco then bought Nuova back in in a buyout and positively threw the product into the marketplace with a blaze of hyperbole and whizzbang marketing. Three months later the products began to ship to selected companies. Yep, its the Nexus 5000 which has FibreChannel AND 10 Gigabit Ethernet in a single chassis. Hosts can now use FCoE to connect to existing FC networks and storage.

Storage Networks are actually quite small

One thing that Storage People seem to miss is that Storage Networks are really small. A few dozen FC switches is a big storage network. For Data Networks, several hundred switches is a big network. Its the importance of the data that “makes appear bigger in the mirror” and the lack of tolerance to interruption.

New Technology, New Designs

There are three missing pieces of the puzzle for Storage over Ethernet. They are:

• Design & Architecture
• Operational Skills and Management
• Technology Standards

Design & Architecture

Network Architects need time to learn the new technologies and design methods. Historically, Data Networks have not been sensitive to loss and new designs and thinking are need to handle this. Fortunately, the storage network is strictly limited to the Data Centre so this is relatively straight forward. Architects, generally, are able to adopt new technologies once the need is clear. (Enter the Account Manager……)

Operation and Support

This will require new procedures and some training, but it won’t be much different from what we do today. Some of the monitoring and analysis tools need upgrades to enhance their performance analysis, and the reporting tools that we use for monitoring bandwidth will also need upgrades. Today, this work is performed by the Storage team, and this work will move the Network Operations team.

Data Networking will need to improve it performance to match the service levels that Storage needs. But that is not a technology problem, it’s a business problem.

Standards

This is the current weakness in the Storage over Ethernet story. The key standards that control how the HOST interacts with the Network in QoS signalling and negotiation are not near complete. FCoE itself is nearing completion, and a number of new 802.1 standards that allow Ethernet networks to have new features that support much faster convergence (sub-second) and higher available bandwidth’s (SPBB) will be ready in the next year or so.

But I can’t trust it ? Can I ?

As it stands today, I can only agree that it would be a very brave decision to invest in FCoE. Lack of standards, limited experience and insufficient resources for knowledge and expertise mean that you will be doing a lot of hard work.

I personally think that Storage over Ethernet is a stupid idea and believe that Storage over IP will be far more successful for more than 90% of the market. However, I also recognise that:

• Cisco has spent more than a billion dollars bringing FCoE to market
• Brocade bought Foundry to get their own Ethernet products at a cost of $3 billion
• Companies like NetAPP and EMC have announced and are shipping FCoE enabled devices
• Emulex and QLogic have chipsets and adapters in the market. I believe Intel will have something shortly.

and the final push from Cisco is their Unified Computing Strategy for which FCoE is absolutely fundamental. And that is getting a lot of mindshare since it seems to move into markets dominated by HP & IBM.

So its time to accept the inevitable….

If there is one thing I have learned in twenty odd years of technology, it is this: Cheap Always Wins.

In the end, FibreChannel will never be big enough to be cheap like Ethernet and there is no chance that FibreChannel will continue for long now that a cheaper option exists. You can argue about technical superiority all you like, whine about feature this and feature that. but it won’t matter, in the end, cheap wins, and the Data Network is cheaper to buy, cheaper to run, and cheaper than Fibrechannel.

It’s time to start getting friendly with your Network Architect and get working on the new, cheaper, future. See if you can build a Storage Data Network that you can live with, not one that gets forced on you.

© Etherealmind for Gestalt IT, 2009. | Monofunctional or Multifunctional – Cheap always WINS
Read more posts categorized as Featured, Networking, Storage

Like clockwork, VMware has cranked out another update to their flagship enterprise product, ESX 3.5. The last update came out in early November, 2008, and included some major new functionality. What’s in store this time to intrigue storage folks? Not much.

For more information on earlier updates, see my articles:

• Storage Fixes in VMware ESX Server 3.5 Update 2
• Storage Changes in VMware ESX 3.5 Update 3

Expanded Support for Enhanced vmxnet Adapter

Not specifically a storage change, but the enhanced vmxnet adapter introduced back in the original release of ESX 3.5 now works with most versions of Windows Server 2003 and XP Pro. Look for improved performance when using guest-side SMB and NFS as well as the guest iSCSI initiator. Note that you cannot select this driver when configuring non-Enterprise Edition machines; you have to select Windows Server 2003 Enterprise Edition (64-bit) regardless of which version of Server 2003 you are using.

Expanded SAS and SATA Controller Support

If you’d like to install ESX on a server equipped with a PMC 8011, Intel ICH9 or ICH10, CERC 6/I SATA/SAS Integrated RAID Controller, or HP Smart Array P700m Controller, you’ll find happiness in Update 4.

The Intel controllers are especially important, as we’re seeing them used more and more and this driver is more full-featured than the earlier Broadcom HT 1000 and Intel ICH7 drivers. The Intel ICH9/ICH10 is a dual-mode (IDE/ATA and AHCI/SATA) driver, supports SATA hard drives, SSDs, and optical drives, and now enables VMFS support when in AHCI/SATA mode. It’s not clear whether VMware actually supports VMFS datastores on ICH9/10 SATA, but it says it works. Anyone want to try it out? One thing is certain: You can’t use SATA drives in a shared/clustered environment because SATA does not include reservations. See this tech note and especially this question:

Earlier, it was mentioned that we can create VMFS if we use AHCI/SATA mode. If so, why did VMware not claim VMFS support when using SATA controller running in AHCI/SATA mode?

VMware might decide to add support in the near future. There is no strong need to have VMFS support on a SATA drive, because native SATA protocol does not support reserve/release. Reserve/release is needed if VMFS is used as clustered file system in a shared disk environment.

PXE Boot Support

Rich at VM/ETC points out that Update 4 includes experimental PXE boot support for ESX and ESXi. As he notes, this has major implications for cloud computing platforms, since it means that ESX servers can boot guests without local storage at all. Very interesting! Let’s bet that Update 5 (expected in June or July) will include this as a supported option.

Updated QLogic, Emulex, and LSI Drivers

Like most ESX updates, this one included updated Fibre Channel drivers.

• The QLogic Fibre Channel Adapter driver and firmware (versions 7.08-vm66 and 4.04.06, respectively) include bug fixes and enhanced NPIV support.
• On the Emulex side, driver version 7.4.0.40 supports the company’s HBAnyware 4.0 management software.
• Users of SAS and SCSI LSI MegaRAIDs will find driver version 3.19vmw (megaraid_sas) and 2.6.48.18 vmw (mptscsi) which improves performance and enhances event handling capabilities.

Expanded Sun Storage Array Support

All you StorageTek loyalists out there will be happy to see support for Sun’s low-end StorageTek 2530 SAS array as well as the modular 6580 and 6780 Fibre Channel arrays. It looks like just about every model in Sun’s current storage lineup is now supported in ESX.

Expanded Network Card Support

Support for Gigabit cards is greatly expanded, including HP’s quad-port NC375i and dual-port NC362i and NC360m, Intel’s Gigabit CT and 82574L, and NetXtreme’s BCM5722, BCM5755, BCM5755M, and BCM5756. Intel’s widely-used 10-gig 82598EB cards are now supported as well.

Tweaks and Fixes

Looking through the release notes, a few storage-related tweaks and fixes stand out:

1. WMware can optionally automatically throttle back the queue depth when congestion is encountered. See Controlling LUN queue depth throttling in VMware ESX for 3PAR Storage Arrays for more information.
2. VMklinux module heap size can now be adjusted as LUN queue-depth values are increased. Since tuning LUN queue depths is one common trick of the storage trade to improve performance, especially in queue-stingy systems like ESX, this is welcome news. But call VMware support before you monkey with it!
3. An RDM-related issue where SCSI inquiry data over 36 bytes was truncated or corrupted (for example when using Microsoft VSS and NetApp SnapDrive) has been resolved.

Well, that’s all folks. I suggest you all read the release notes for yourself, and please leave a comment if you see an error in what I wrote here or have something to add!

© Stephen Foskett for Gestalt IT, 2009. | Storage Changes in VMware ESX 3.5 Update 4
Read more posts categorized as All, Server Virtualization, Storage