“Bridging versus routing” brings us to the perennial networking debate: Are low-level protocols better?

Unless you’re “in the know”, terms like “layer 2″ can seem mysterious, making it all the more plausible when someone touts the benefits. It seems logical: “Bare-metal” communication must be better, faster, and cheaper than higher-level “everything over IP” approaches, right? But it’s not quite that simple.

This piece assumes you know something about the OSI model.

Ivan Pepelnjak posted a great overview of the “bridging versus routing” debate (Bridging and Routing: is there a difference?), and Marko Milivojevic posed the question in response: “I’m one of those who doesn’t understand the whole L2 obsession of the modern networking world, but…”

It really is an obsession: Data communications folks continually argue about the merits and trade-offs between high-level network protocols and low-level communications. We hear it in storage all the time: FCoE proponents assume performance benefits, and AoE fans add cost advantages to the mix. But many of these claims are unsubstantiated, and iSCSI and NAS protocols like SMB and NFS just keep rolling forward. If storage over IP is so bad, why does iSCSI work (and perform) so well?

One thing often missing in the “layer 2″ arguments is what’s missing when you skip the network layer. There’s a reason IP is so widespread: It may not be the best protocol ever, but it works really well in a huge variety of situations and there is a vast pool of associated technologies that can be drawn upon when using it.

IP can run over just about anything, from FireWire to SONET, so any protocol using IP can (theoretically) run there, too. I’ve run iSCSI over Wi-Fi and WAN links, and it works great out of the box with 10 Gb Ethernet. Protocols that are tightly linked to a layer-2 protocol face stiff challenges when moving to different data links. Witness the difficulty moving Fibre Channel to 10 Gb Ethernet, including all those data center bridging technologies. In fact, when faced with the challenge of long-distance Fibre Channel SAN communication, encapsulation over IP was a natural choice.

IP also has a myriad of wonderful technologies to choose from. The creators of iSCSI were able to pull authentication, encryption, lossless communication, and many other features straight from the existing toy box. Developers of new non-IP protocols have to invent their own solutions to these problems, often with disastrous results. Why reinvent the wheel? Just apply a little CHAP, some IPsec, and roll it in TCP and you’re done!

Photo credit: “Akashi Kaikyo Bridge 明石海峽大橋” by Shenghung Lin

© Stephen Foskett for Gestalt IT, 2010. | The Lure of Layer 2
Read more posts categorized as Networking, Storage

This post is the first in a series covering the DroboElite SMB storage device from Data Robotics Inc.

The DroboElite is the most fully featured storage array in the Data Robotics range of devices.  Regular readers will know I’ve posted frequently on these devices, from the first Drobo “Classic” to the DroboPro, both of which I own and have purchased myself.  Data Robotics have kindly loaned me a DroboElite for this series of posts.  As you will see, the top of the range model offers all the features the other models provide, plus the benefits of multi-user support.

Background

The original Data Robotics storage device was what is now referred to as the Drobo “Classic”.  This was a 4-bay storage device, connecting to a single host PC/server using USB or Firewire and later eSATA.  This model evolved into the Drobo S and was complemented with the 8-bay DroboPro.  Finally, the DroboElite was released in November 2009, using the same form factor as the ‘Pro but with additional connectivity and features.  Although the ‘Pro and ‘Elite models shared the same look and feel, they are actually different products and so there’s no upgrade process to remove drives from the ‘Pro and install into an ‘Elite shell.

Physical

The DroboElite is an 8-bay device, accommodating standard 3.5″ SATA form factor drives.  The drives are inserted vertically (rather than horizontally in the 4-bay models), with drive’s top side pointing to the right.  No caddy or additional support is required.  From the front, there’s nothing to separate the ‘Pro and the ‘Elite as they look identical.  In fact, the removable magnetic front bezel only says “Drobo” and has no other distinguishing markings.  One thought worth considering for the future is providing a method of identifying multiple Drobo’s that have been installed into a rack or cabinet.  The drive lights can be flashed on and off but this only lasts for a few seconds and wouldn’t help an engineer who’s alone in a data centre or machine room.  This is a particularly important point as the ‘Elite is a multi-user device and so aimed at SMBs with machine rooms and rack-based equipment (a rack mount  kit is available).

Turning to the back of the ‘Elite, we start to see the differences with previous models.  The major one here is the presence of two Ethernet ports, providing network redundancy and multi-pathing support for iSCSI.  The power supply is still built in (and not removable), plus there’s a power switch and a USB connector – that it.  I’m undecided as to whether an integrated PSU is a good or a bad thing and I flip back and forth between internal/external and multiple supplies each time I think about it.  On reflection, the consideration here should be that the Drobo is not a “high availability” storage array in the modular or enterprise class.  It doesn’t support mulitple controllers or other features like mirrored cache, so in reality, multiple power supplies is probably over-configuration.  Obviously dual Ethernet connections is essential in a device that will support multiple hosts; this feature provides redundancy from failures in network equipment or more likely, enables maintenance to be performed on the network infrastructure without impacting the storage.

The addition of a power switch is a positive move (and essential as the power supply is now integrated).  Personally I’d have preferred the power switch to be mounted behind the magnetic bezel on the front of the unit.  This would mean once plumbed in, the ‘Elite (and ‘Pro for that matter) would be managed purely from the front, for disk insertions/removals.

I’ve one final comment on the physical aspects of the ‘Elite and that covers the insertion and removal of drives.  I never like removing spinning active drives and unfortunately there’s still no way to either mark a drive for removal or spin it down.  The unique BeyondRAID feature of the Drobo series means that any drive can be removed from a unit and the device ’self heals’ to compensate for the loss of capacity, rebalancing blocks of data redundantly across the remaining drives.  Of course drive removal doesn’t have to be done after a failure; the whole benefit of the Drobos is the capacity increase functionality (I frequently swap drives in/out as part of testing).  I’d therefore like to be able to spin down or at least highlight a drive I intended to pull, via software, before I physically pull it.  It gives me more confidence that I’ve pulled the correct drive when managing multiple units.

Additional specifications can be found here: http://www.drobo.com/pdf/droboelite-datasheet.pdf

In the next post, I’ll look at connectivity protocol support.

© Chris for Gestalt IT, 2010. | Hardware Review: Drobo Elite – Part I
Read more posts categorized as Featured, Storage

I’m not a fan of making press releases on behalf of other companies however once in a while, a news item catches my interest.  So it is with the announcement of the Violin Memory Inc. 3200 series of all-memory storage arrays.  Why are these interesting?  Because I think they are moving and potentially blurring the boundaries between spinning drives and memory-based permanent data storage.

Background

Building arrays from pure memory isn’t new; Texas Memory Systems have had the RamSan series of products on the market for some time now (and there are others out there).  Of course, the problem for many large organisations is how to make use of such an expensive and relatively small device.  There are plenty of use cases where flash/SSD may be useful, however (a) it is difficult to target exactly which applications and (b) for those applications that can be identified, potentially only part of the data will benefit from acceleration.

One solution has been to follow the route of the traditional vendors and add SSD as an extra device within the same hardware chassis.  This isn’t a solution to using SSD but rather a sticking plaster over the problem; the SSD may give better read performance but it is unlikely that writes will be accelerated to the level justified by the additional costs of the SSD device itself.  In addition, the SSD is sitting behind a traditional storage array.  Vendors such as EMC, IBM and Hitachi have spent millions of man-hours and hundreds of millions of dollars on software developments to help smooth the impact and manage the unpredictable performance of hard drives.  Remember that when an I/O request is received, the storage array has no idea where a mechanical device like a hard drive is positioned and so cache, algorithms and that other clever intellectual property have been used to mask these physical inadequacies.

However, despite vendors’ best efforts, spikes and unpredictable response times do occur and there’s no way to remove them and guarantee completely consistent I/O responses.

The Violin Approach

So what happens if you can remove the cost issues and buy an SSD-based array for the same price as tier 1 storage?  This is the route Violin Memory are taking to market – make the SSD storage array as closely priced to tier 1 arrays as possible.  Remove the thought process and complications of determining what to place on SSD by making the price argument irrelevant.

In reality, Violin haven’t reached that price parity yet; prices are quoted around the $20/GB mark, which is around double what I’d expect to see for tier 1 storage (depending on volume).  However it is in the order of magnitude where organisations can look at those troublesome applications that decide that the cost of additional servers, disk spindles or re-writing the application is outweighed by simply moving the application to a Violin SSD device.

I think this is the ultimate tipping point for SSD use; where the cost of improving application performance is exceeded by the cost of moving to SSD, then SSD will win.  Where improving application performance is justified by increased business advantage, the business case is written.

Tech Specs

OK, let’s have a look at the technical specifications for the techies amongst you.  Firstly, today’s device capacity sits at 10TB in 3U and is expected to grow to 20TB in Q3.  I’ve also been told that this capacity is expected to be close to 5x greater by the end of 2010, which means 100TB of memory-based storage in a 3U unit.

The 3200 supports PCIe (x4 & x8)  as well as 4/8Gb Fibre Channel and 10Gb iSCSI and FCoE.  Latency is less than 100 microseconds.

Violin array use VIMMs (Violin’s name for their flash memory cards.  These are grouped together into 1TB units, using RAID-5 technology to manage failures.  Maintenance can be performed online periodically to replace failed VIMM devices.

There’s one major issue with Flash/memory-based arrays that Violin claim to have addressed.  That is the issue of degraded performance over time.  Have a look at the following graphic, showing saturated workload on the Crucial C300 versus X25M from Intel.  This graph and the associated review can be found on Anandtech’s website here.  Very quickly with heavy use, the performance for these devices drops off.  Violin claim their array doesn’t suffer similar issues and can deliver sustained performance.  Of course, we can believe that statement once we’ve seen a review of the product delivering the performance as promised.

Futures

A 10/20TB capacity in 3U isn’t huge by today’s standards.  If Violin Memory can deliver on their promises and bring a 3 to 5-fold increase in performance by year end (with a continual reduction in price) then things start to look interesting.  I’d like to see the results of some long-term stress tests on the 3200 series devices.  I have some more material to post in the coming days, once I can validate what’s open and not under NDA/embarbgo.  In the meantime, here are some questions to ponder:

• Do I have any I/O bound applications?
• Can I measure/determine my I/O bound applications?
• Is there direct businss advantage from increasing I/O throughput?

If you can start answering yes to the above questions, then perhaps SSD-based arrays are for you.

© Chris for Gestalt IT, 2010. | Violin Memory Inc Release New All-SSD Array
Read more posts categorized as Featured, Storage

Why does network-attached storage (NAS) have such a poor reputation? This isn’t what the vendors want to be talking about, but some recent product announcements and discussions led to this thought. IT folks as a whole don’t trust NAS for real work, and 20 years of effort from big names like Sun, Microsoft, NetApp, IBM, and the rest hasn’t changed that.

Fear

Back when I used to teach the “Storage 101″ session at Storage Decisions, I was consistently amazed to find little awareness of enterprise NAS systems. People complained about LUNs and Fibre Channel but when I suggested using NFS or SMB their heads almost exploded. “We would never use that for application storage,” they shouted. “File servers are for home directories, not data center stuff!” Clearly, NAS faces an uphill battle.

In a recent piece I wrote, I referred to what I consider to be the prime best practice: Use the right tool for the job. It’s a simple statement, and one that resonates beyond IT and the technology world. But it can be devilishly difficult to see what the right tool is sometimes. Why not use NAS for virtual machine storage? NetApp has been beating that drum for years, yet NAS has a very small footprint in VMware. How about databases on NFS? Exchange over SMB? Block storage has a massive lead over NAS in all of these areas.

Rear this “best practices” piece, Use Process Solutions For Process Problems, Technical Solutions For Technical Ones

IT folks seem downright fearful of file-level storage protocols. Has NFS really burned them that badly over the decades? Can SMB/CIFS really be as bad as they think?

Loathing

I wonder if this terror has more to do with the products people have used than the fundamental concept of file services. Many NAS servers (and clients) are barely functional. Sadly, NFS and SMB are easy to get 80% right, but the 20% corner case interaction takes decades to overcome. My daily storage consulting work exposes me to a myriad of NAS configurations, and few of the multi-platform combinations end well.

Note: Although it has long been known by a variety of names, the Windows NAS protocol is currently called Server Message Block or SMB. Common Internet File System (CIFS) was a failed mid-1990’s attempt by Microsoft to make this protocol standard on the Internet.

Consider the Mac. Apple added an SMB client to OS X in 2001 but, despite many updates, it is far from reliable. Mac users in general loathe connecting to Windows file servers, and business users have located numerous bugs in the handling of Mac-specific file types. It’s bad enough that one company, GroupLogic, created an entire AFP server for Windows just to solve these tricky issues.

This situation often happens in reverse, too. Windows admins are justifiably cautious when deploying non-Windows SMB servers, whether software (Samba, Novell, etc) or system (NetApp, Celerra, BlueArc, etc). As a very early NetApp user, I watched their CIFS/SMB server evolve over a decade and a half into a fairly robust solution, but the early years were downright painful.

Lest you throw rocks at Redmond, know that SMB is not alone with functionality problems. The interoperability of NFS servers and clients is a bit better thanks to open(ish) standards and open source implementations, but its reputation is just as bad. And Apple’s proprietary AFP protocol is downright notorious.

I’ve been there myself many times. I tried to set up a home server based on open source software (Linux, FreeBSD, Samba, Netatalk, etc) but rejected it outright after many frustrating years. Today I use a Mac Mini for file sharing in OS X and serving iTunes music and movies (goodbye, Firefly!) And years of fighting with Samba in enterprise environments taught me two things: It’s possible to get it running well with Windows clients but it’s astonishingly easy to get it wrong.

Enterprise NAS?

We all know that interoperability is devilishly difficult. I don’t envy the NetApp and EMC engineers that have to tweak and tune their server for every possible client, bugs and all. And I am impressed that, after probably millions of man-hours of work, they were able to come up with something stable for a subset of use cases. But this just makes me even more cautious about third-party NAS servers.

I talk to storage vendors all the time, and many of their new products support NFS and SMB. But my internal alarms start going off when I hear about these products. There are two simple reasons for this:

1. As mentioned above, NAS is rare in primary data center applications. It may be common for user files (euphemistically called “unstructured data”) and certain distributed applications (simulation, rendering, etc), but most use cases still call for block SCSI (FC/iSCSI) storage.
2. As further mentioned, getting NAS right takes a massive amount of effort. New and small vendors tend to slap Samba on their (Linux-based) box and call it a day. This is very, very far from being sufficient for enterprise use.

This is why I usually ignore NAS functionality in storage systems except for long-tenured and deep-pocketed vendors. Although the world is turning to “Unified Storage” and multi-protocol support, I’m focusing primarily on block (SCSI) and cloud (REST) capability because the former has proven somewhat easier than NAS to get working and the latter is both simple and “green field” with no legacy concerns.

© Stephen Foskett for Gestalt IT, 2010. | Why Do I Ignore NAS?
Read more posts categorized as Storage

EMC’s Iomega unit today released the 12-drive rack-mount storage product we have all been waiting for. It was never a question of whether Iomega could produce such a beast: The EMC LifeLine software and Iomega hardware were definitely up for it. The question was always if EMC would direct Iomega to fill the gaping hole in their storage lineup between the 4-drive ix4-200r and the entry-business AX4 arrays. The new ix12-300r packs 12 drive bays, scaling from 4 TB all the way to 24 TB, and backs it with quad gigabit iSCSI, redundant power, and everything else the small data center needs.

Stepping Up

The ix12 is a big step up. Although they sold multi-drive RAID systems even before the EMC acquisition, this new device is unlike anything we’ve seen before from Iomega. This slim (2U) chassis is all drives up front, with 12 hot-swap slots full of 3.5″ SATA storage. Under the hood is a dual-core 3 GHz Intel Core2duo E8400 “Wolfdale” CPU, a major horsepower upgrade from the single-core Celeron in the ix4-200r. It also has double the memory (2 GB) compared to that 4-drive product.

If you’re concerned about performance after trying out a desktop ix4-200d, you needn’t worry. The desktop unit has a lowly 1.2 GHz Marvell 6281 and just 512 MB of RAM. While this might be enough for a desktop user, it could never handle the pounding of servers in a shared networked configuration. The ix4-200r, with its 3.2 GHz Intel Celeron D 352 “Cedar Mill“, offered much better performance even though its name was just one letter different. And the ix12’s CPU is three times faster still, though it remains a single-controller system.

More significant changes lurk around the back of the ix12. Dual redundant power supplies, a frequent request in this class, and variable-speed fans, surround four Ethernet ports. Each sports gigabit speed and the set supports Microsoft Windows MPIO, can be aggregated with 802.3ad, or used in VLAN configurations with up to 4 VLAN tags per port. The ix12 speaks just about every language, from NFS and SMB to AFP and iSCSI, and now adds WebDAV and DFS support, too.

A few limitations separate this new ix12 device from its enterprise-grade brothers, however. As noted, a single controller manages all access, so redundancy and parallel processing are limited. Although the ix12 sports 12 drive bays, it only has four SATA channels internally; each bay shares a channel with two others using SATA expanders. Don’t expect to push wire speed over all four Ethernet ports at once, even with all this newfound CPU power.

A Wall of Drives

Base ix12s ship with 4 drives installed, but we were disappointed to learn that additional drives must be purchased in 4-disk packs from Iomega. Although this decision is understandable, the ix series remains a holdout amid growing legions of bring-your-own-drive competitors. At least the company supports mixing and matching drive sizes, including 1 TB and 2 TB at present. We suspect that the unit uses the same reliable 5900 rpm Seagate Barracuda LP drives as the ix4-200d.

Iomega added a few tricks to the LifeLine software to take advantage of a possible 12 drives installed. First up is the addition of dual-parity RAID-6 for improved data protection. The company (and this reviewer) suggest this over RAID-5 once more than 5 drives are combined in one set. Don’t worry, though, because RAID configuration can be changed online and any unused drive can be used regardless of its location in the array. The ix12 also adds drive spindown, saving power when the RAID set isn’t in use.

Like the ix4, any portion of a RAID set can be carved out into an iSCSI LUN for Ethernet-connected hosts. Iomega claims that LUN provisioning times have improved with the added horsepower and software tweaks, and we hope this is true. A maximum of 256 LUNs can be configured in this way, though even 12 drives are unlikely to drive much performance to that many storage users.

Where to Use It?

Although not listed yet, Iomega promises that the ix12 will have a place on the Exchange ESRP, Windows Server and Hyper-V logo list, and VMware Compatibility Guide this month. It’s already the first Iomega product to be “EMC E-Lab Tested”, meaning it is on the EMC Support Matrix; this fact alone speaks volumes of EMC’s expectations for the unit. My own experience shows that Iomega iSCSI is fine for smaller VMware and Hyper-V deployments.

Clearly, the ix12 is a new kind of Iomega array. If the 200d and 200r were a sign that the company wanted to move out of the house, the ix12 is a demonstration that they have graduated. Starting at US $5,000, the ix12 is all business and its resume ought to impress in interviews. It can’t quite boast the scalability and redundancy of established arrays (including its brothers from EMC), but it ought to be an easy acquisition for companies looking for a little more storage here or there.

One is left with questions, though: How big will EMC let Iomega get? If 12 drives are acceptable, what about 24? Is SAS off limits? What about 10 Gigabit Ethernet and even Fibre Channel over Ethernet (FCoE) eventually? Can we dream of dual controllers? At some point, the Iomega lineup could even threaten the CLARiiON!

Then there is the competitive landscape. Iomega leapfrogged the 8-drive Data Robotics lineup and landed squarely in competition with the likes of the revitalized Overland Storage but at a much lower price. We also have Netgear, HP, Dell, and Promise, and there is an attractive D-Link box packing 15 drives and 10 GbE. Iomega also has to worry about its own big brother, the Dell/EMC AX4, starting around $12k. It’s a competitive market, and Iomega is in for a fight as even more vendors wake up to the possibilities in this market.

© Stephen Foskett for Gestalt IT, 2010. | Iomega Graduates and Goes to Work with the ix12-300r
Read more posts categorized as Storage

This week’s virtual Tech Field Day roundtable discussion focuses on Nimbus Data Systems. Tom Isakovich, CEO of Nimbus, gave the Tech Field Day Delegates a sneak peek at their new S-class solid state iSCSI storage array.

Nimbus claims that the S-class will deliver new levels of performance in a compact, reliable, and affordable package. The array starts at around $25,000 for 2.5 TB of thin-provisioned, deduplicated flash storage and scales to 100 TB and 500,000 IOPS. The array also boasts high-performance snapshots and integrated replication.

Joining Nimbus for this roundtable discussion were ten Tech Field Day delegates:

• Jason Boche (@JasonBoche, Boche.net)
• Carlo Costanzo (@CCostan, VMware Info)
• Chris Evans (@ChrisMEvans, The Storage Architect)
• Greg Ferro (@EtherealMind, EtherealMind)
• Scott D. Lowe (@ScottDLowe, Tech Republic)
• Scott Lowe (@Scott_Lowe, ScottLowe.org)
• Devang Panchigar (@StorageNerve, StorageNerve)
• Bas Raayman (@BasRaayman, Bas Raayman)
• Matt Simmons (@StandaloneSA, Standalone Sysadmin)
• Rick Vanover (@RickVanover, Virtualization Review)

Download the Nimbus roundtable podcast now, or subscribe in iTunes to tune in to the discussion!

© Stephen Foskett for Gestalt IT, 2010. | Podcast 4: Virtual Field Day Nimbus Roundtable
Read more posts categorized as All, Events, Exclusive, Featured, Storage, Tech Field Day Roundtable Podcast, Top Story

FCoE is taking enterprise storage by storm, but where is Microsoft?

Microsoft is a controversial company, generating a mix of criticism and praise everywhere they are involved. Although I’m definitely a Microsoft skeptic, I’ve been pleased by much of the work they have done in my field of enterprise storage. Their client-side (“initiator” to us storage folks) work has been especially notable, making me wonder why no other operating system vendor has as diverse and full-featured storage support as Microsoft. I would go so far as saying that the iSCSI SAN revolution would not have happened if not for the folks up in Redmond.

Enterprise storage is in the midst of another shift these days. iSCSI moved low-end and midrange servers to SANs by leveraging Ethernet for connectivity. Now, the storage and networking industry is pushing to do the same for high-end Fibre Channel SANs, bringing Fibre Channel over Ethernet (FCoE) to life before our eyes. Although users are not yet on board, the dominance of FCoE seems almost certain to many industry watchers.

Where Is Microsoft?

But where is Microsoft when it comes to FCoE? I see no public statements of support for the protocol. I see no participation in FCoE standards-setting (though the company did recently join the FCIA as an “Observer”). Most importantly, I see no software features developing solid enterprise support for FCoE within the Windows operating system. Put simply, Microsoft does not seem to be participating in FCoE development, either on the initiator or target side.

Some may say Microsoft doesn’t need to get involved. After all, most FCoE integration will rely on third-party hardware and drivers that appear to be standard Fibre Channel HBAs to Windows. But this argument is spurious: Microsoft never got very involved in Fibre Channel either, and the confusing state of third-party FC SAN connectivity in Windows is a stark contrast to the solid and reliable iSCSI initiator. The fact that Microsoft embraced iSCSI while allowing third parties to “own” FC is an indictment of their stance towards FCoE.

Then there is Microsoft’s FCoE  logo program. In March, I was pleased to note that Microsoft would at least add FCoE to their Windows Logo program. Yet there has been no sign of the Logo Kit, promised for December, and just one mention of this program in the last year: Microsoft delayed enforcement of FCoE Logos until December of 2010. As far as an outside observer would know, Microsoft hasn’t moved forward with even the most basic FCoE support, a Logo program, in a year.

I am not suggesting that Microsoft must develop software FCoE targets. In fact, a Microsoft FCoE target might prove to be counter-productive, discouraging storage vendors from competing. A software initiator would be welcome, but the high-end target audience makes this less critical than for iSCSI. Instead, Microsoft should begin to develop a converged I/O framework around the DCB standards, including a detailed FC and FCoE integration architecture for third-party hardware and software.

Unacceptable

The current state of affairs is simply unacceptable. Consider the systems using FCoE: Many criticize Microsoft’s presence in the data center at all, but no one can deny their power there. Microsoft Windows Server powers most enterprise compute tasks today, running on almost three quarters of data center servers. Windows Server will almost certainly be the leading user of Fibre Channel over Ethernet connectivity.

On the other side of the equation, storage and networking vendors from EMC and NetApp to Cisco and Emulex are pushing hard for the FCoE transition. Regardless of end user indifference, FCoE will dominate SAN deployments in the next few years. It’s a question of “when” not “if”  - DCB and FCoE will take the data center by storm.

Certainly, Microsoft can simply allow the FCoE ship to sail without them on board. They can allow third-party HBA/CNA, server, and networking vendors to bring FCoE support to Windows. But this represents a massive missed opportunity for the company and a capitulation in the high end of the enterprise market. Microsoft must immediately move to take leadership in FCoE: Add corporate resources, step up standards efforts, publish the Logo Kit, and develop a framework of support.

Disclaimer: I am a Microsoft MVP in the area of storage. Their kind treatment of me obviously didn’t color my perception of them with regards to FCoE. I also have an NDA with Microsoft, and they have briefed me on future Windows Server and storage developments. Nothing in this post should be construed to reveal the company’s future plans.

© Stephen Foskett for Gestalt IT, 2010. | Where Is Microsoft’s FCoE Support?
Read more posts categorized as Storage

VCMDB: Volume Control Manager Database

ACLX: Access Control Logix

VCM: Volume Control Manager device (where the database resides)

VCM Gatekeeper: Volume Control Manager Gatekeeper (database doesn’t reside on these devices)

SFS Volumes: Symmetrix File System Volumes

If you work with EMC Symmetrix systems, you know the importance of VCMDB. Introduced with Symmetrix 4.0 and used in every generation after that, VCMDB stands for Volume Control Manager Database). Also in the latest generation of systems the VCM device is at times also referenced as VCM Gatekeeper.

VCMDB is a relatively small device that is created on the Symmetrix system that allows for hosts access to various devices on the Symmetrix. VCMDB keeps an inventory of which devices have access to which host (HBA’s). Without a VCMDB in place, host systems will not be able to access the Symmetrix. The VCMDB should be backed up on regular intervals and would be helpful in a rainy day.

The VCMDB device size grew along with new generations of Symmetrix systems that got introduced, primarily a means to keep a track of more supported devices (hypers / splits) on these platforms. With the introduction of Symmetrix V-Max, the VCMDB concept is now a bit changed to ACLX (Access Control Logix). Access Logix is being used on the Clariion systems for years now.

Here are a few things to consider with VCMDB

• On the older Symmetrix systems (4.0, 4.8, 5.0 and 5.5), the VCMDB (device) is mapped to all the channels, host
• In these systems the VCMDB access is typically restricted by Volume Logix or ACL (access control lists)
• With the Symmetrix DMX, DMX2 Systems – Enginuity Code 5670, 5671 the VCM device only requires to be mapped to the Management stations
• Management stations include SYMCLI Server / Ionix Control Center Server / Symmetrix Management Console
• At all given times on the DMX, DMX2 platforms, the VCMDB would need to be mapped to at least one station to perform online SDDR changes. Alternatively this problem of not having device mapped to at least one host can also be fixed by the PSE lab
• Mapping VCMDB to multiple hosts, channels may make the device venerable to crashes, potential tampering, device attributes and data change
• You can write disable VCMDB to avoid the potential of the above
• With these systems, the host can communicate to the VCMDB via Syscalls
• The VCM Edit Director Flag (fibrepath) needs to be enabled for management stations to see VCM device
• The database (device masking database) of the VCMDB resides on the SFS volumes. This feature was introduced with DMX-3 / DMX-4 (5772 version of microcode). A 6 cylinder VCM Gatekeeper device is okay to use with these versions of microcode.
• Starting Symmetrix V-Max systems, the concept of ACLX was introducted for Auto Provisioning etc.
• VCM volumes are required to be mirrored devices like SFS volumes

Various different types of VCMDB

Type 0, Type 1, Type 2, Type 3, Type 4, Type 5, Type 6

• Type 0: Symmetrix 4.0, 32 Director System, 16 cylinder device size, Volume Logix 2.x
• Type 1: Symmetrix 4.8, 64 Director System, 16 cylinder device size, ESN Manager 1.x
• Type 2: Symmetrix 5.0/5.5, 64 Director System, 16 cylinder device size, ESN Manager 2.x
• Type 3: Symmetrix DMX, supports 32 fibre/ 32 iSCSI initiator records per port, 24 cylinder device in size. Enginuity 5569, Solutions Enabler 5.2, Support 8000 devices
• Type 4: Symmetrix DMX/DMX-2, supports 64 fibre/ 128 iSCSI initiator records per port, 48 cylinder device in size. Enginuity 5670, Solutions Enabler 5.3, Supports 8000 devices
• Type 5: Symmetrix DMX/DMX-2, supports 64 fibre / 128 iSCSI initiator records per port, 96 cylinder device in size, Enginuity 5671, Solutions Enabler 6.0, Supports 16000 devices
• Type 6: Symmetrix DMX-3, DMX-4, supports 256 fibre / 512 iSCSI initiator records per port, 96 cylinder device in size, Enginuity 5771, 5772 Solutions Enabler 6.0, Supports 64000 devices

Notes about various Types of VCMDB

• Type 3 of VCMDB can be converted to Type 4 VCMDB (code upgrade from 5669 to 5670 to 5671)
• Solutions enabler 5.2 and Solutions Enabler 5.3 can read/write Type 3 VCMDB
• Solutions enabler 5.3 can read/write Type 4 VCMDB
• VCMDB device is recommended to be a certain size, but it is okay to use a larger size device if no choices are available.

Converting various types of VCMDB using SymCLI

• If the device cylinder size is equal with a conversion you are attempting, the following will help you convert your VCMDB from type x to type y.
  • Backup the device
  • symmaskdb –sid <symmid> backup –file backup
  • Check the VCMDB type using
  • symmaskdb – sid <symmid> list database
  • Convert from type 4 to type 5
  • Symmaskdb – sid <symmid> convert –vcmdb_type 5 –file Covertfilename

To initialize VCMDB for the first time on a Symmetrix System

Within Ionix Control Center

• Click on the Symmetrix array you are trying to initialize the VCMDB
• Select Masking then VCMDB Management and then initialize
• Select a new backup and create a file name
• Create a file name with .sdm extenstion
• Click on Activate the VCMDB
• VCMDB backups are stored at \home\ecc_inf\data\hostname\data\backup\symmserial\
• Also it will be viewable within Ionix Control Center at Systems/Symmetrix/VCMDB Backups/

With SymCLI

• To query the VCMDB database
  • symmaskdb –sid <symmid> list database
  • To backup and init an existing VCMDB database
    • symmaskdb – sid <symmid> init –file backup

More technical deep dive coming soon on various other topics…including ACLX.

Cheers
@storagenerve

© Devang for Gestalt IT, 2010. | EMC Symmetrix: VCMDB and ACLX
Read more posts categorized as Storage

I’ve had my DroboPro for some time now and initially I had mixed results with it.  Although I was aware of the single connection to an iSCSI target restriction, I was slightly disappointed with the lack of vmfs support within the Drobo operating system, especially as the product sells itself on understanding the filesystem format.  My next move has been to test the DroboPro with Hyper-V and guess what?  Thin Provisioning is back on the agenda; Hyper-V uses NTFS to store hypervisor guests, making it perfect for use with DroboPro.

The first screenshot shows my DroboPro configuration.  It’s currently up to 7 drives and 9.4TB; although I’m only using a fraction of that space at the moment while I do some testing (the majority of disks in the unit were also purchased for testing).  I’ve created seven 2TB drives (more than the Drobo capacity) and connected them to my Hyper-V server.  From there, I’ve formatted four of the drives and you can see those as drive letters H: through to K: on the second screenshot.  Two of the volumes are hosting Hyper-V guests occupying around 19GB of storage in total.  This is directly reflected by the capacity of space used on the DroboPro and the two C:\ drives, one of which is shown in the third screenshot.

The DroboPro is ‘thinning’ the thin provisioned Hyper-V virtual disks, enabling me to in effect scale my Hyper-V storage requirements to the absolute minimum I need to, even if the virtual disks for the operating systems are bigger (in this instance 126GB per virtual drive).  The next step is to extend the storage usage on the virtual guests and see what happens.
My first attempt is to use an old favourite, fsutil and create a few large empty files.  Unfortunately this strategy doesn’t work; the NTFS file system on the Hyper-V guest seems to detect that this is a sparse file and even though the virtual file system shows the storage as being allocated, this isn’t passed back and allocated on the Hyper-V server.  This seems to indicate that Hyper-V itself is truly filesystem and thin provisioning aware.

My second attempt consisted of finding a large file and copying it repeatedly.  This process was more successful and the results can be seen in the 6th screenshot; I added around 9GB of additional files, pushing up the DroboPro capacity and the space stored in the VHD snapshot by equal amounts.

I then deleted my snapshots, effectively merging them into the main VHD file.  As the merge took place, the base VHD file increased, taking more overall capacity on the DroboPro.  After the merge had completed and the Hyper-V guest restarted, the capacity used returned to the total in actual use; the DroboPro reclaimed the released storage.

Summary

What have I learned?  Without any additional configuration, the DroboPro is thin provisioning aware with Hyper-V.  This is “thin on thin” as the Hypervisor and the DroboPro both implement thin provisioning at their own level.  Snapshots need some consideration, especially when merging into the main virtual disk (VHD).  Thought also needs to be given to the best way to lay out the Hyper-V storage; bear in mind that to get the thin provisioning benefits, each DroboPro drive must be configured with a single NTFS volume – spanned and striped volumes will not work.

I’d say the DroboPro was good to go with Hyper-V.  All we need now is vmfs support!

© Chris for Gestalt IT, 2010. | DroboPro – Hyper-V’s Best Friend!
Read more posts categorized as Featured, Server Virtualization, Storage

Ever since Microsoft and Intel declared that the combination of Windows and Nehalem could deliver over a million iSCSI IOPS, I’ve been curious about just how they did it. What black magic could push that many I/Os over a single Ethernet connection? And what was on the other end? Now Intel has revealed all in a whitepaper, and the results are surprising!

What iSCSI Did

Let’s review the test for a moment. In March, Microsoft and Intel demonstrated that the combination of Windows Server 2008 R2 and the Xeon 5500 could saturate a 10 Gb Ethernet link, pushing iSCSI throughput to wire speed. That’s 1,174 MB/s, right around the theoretical maximum of a ten-gigabit link, given a tiny bit of overhead. The pair reunited in January to show that this same combination could deliver an astonishing million I/O operations per second, too.

Both of these results are astonishing. Sure, many high-end Fibre Channel SANs and storage systems blast out gigabytes of data and millions of I/O operations every second, but these tests are much more focused. Benchmarks are perilous, but the folks at Microsoft and Intel devised a fairly clever and focused set. Rather than a “mine’s bigger” contest, the pair only needs to prove that iSCSI can play with the pros.

The side effect is a demonstration of the capabilities of Microsoft and Intel components. Microsoft showed off the capabilities of Windows Server 2008 R2, Hyper-V, and their software iSCSI initiator, while Intel can brag about the Xeon 5500 server platform and X520-2 10 Gb Ethernet Server Adapter with their 82599EB controller. Your mileage may vary, but it is possible to construct a true storage monster on an average server budget.

Intel Inside

Let’s start by looking at the configuration of the local end of the tested configuration. I’m a storage guy so I think of it as the initiator, but you might say it’s the server, the client, or the host. Regardless, the system under test (SUT) is what was put under the microscope. The configuration was a common one: A high-end computer packing an Intel Xeon CPU and 82599-based 10 Gb Ethernet adapter. Most data centers have a machine or two just like this one.

Looking closely, we see that the test in question relied on the following key components:

• Intel’s “Shadycove” S5520SC workstation-class motherboard
• The Intel Xeon W5580 CPU (4 cores, 8 MB cache, 3.20 GHz)
• 24 GB of DDR3 RAM
• Intel “Niantic” 82599EB 10 Gb Ethernet controller
• Microsoft Windows Server 2008 R2 x64

This combination would set you back about $7,500 – $450 for the motherboard, $1,500 for the CPU, 6 2 GB DDR3 SDRAM modules at $80 each, $1,200 for the Intel X520 NIC, and $4,000 for an Enterprise copy of Windows Server 2008 R2. Not cheap, but not an exotic server either.

Initiate and Optimize

The secret to push the tested system to perform like it did is in the optimizations in the server platform, the NIC, and Windows Server itself.

• The Xeon 5500 processor series includes many enhancements:
  • An integrated memory controller allows for faster RAM access
  • QuickPath interconnect (QPI) replaces the old front-side bus and enhances I/O off the core
  • A new I/O subsystem with PCIe integrated into the CPU
  • MSI-X expands the number of interrupts a PCI device can use
  • New instructions for on-board CRC-32C decoding, speeding up iSCSI digest processing
• The 82599 Ethernet controller also includes enhanced capabilities:
  • VMDq maps I/O queues to multiple cores and virtual machines, reducing I/O bottlenecks
  • Offload of TCP segmentation and receive-side coalescing
  • Interestingly, it does not appear that VMDc/SR-IOV was employed in the test
• Microsoft Windows Server 2008 R2 and Hyper-V are ready to use all of these features and more:
  • R2 uses multi-core CPUs more effectively in general
  • Receive-side scaling (RSS) spreads the I/O workload across all four Xeon cores
  • The iSCSI initiator now allows CRC digest offload (using the new Xeon command set)
  • Numerous “NUMA I/O” optimizations in the initiator
  • TCP/IP Nagle can be disabled in the registry
  • Hyper-V VMQ allows the network packets to be copied directly into the guest virtual machine’s memory

Whew! Put all of these optimizations in a blender and Hyper-V virtual machine iSCSI access will be twice as fast as before. No kidding!

Stay On Target

But we knew all of this back in January. We also saw that a Cisco Nexus 5020 switch was used to fan out to 10 software iSCSI targets. But until now there was no mention of what targets were used exactly.

The final footnotes in Intel’s whitepaper reveals that the storage backing the million IOPS test was none other than StarWind Software’s iSCSI SAN! It is unclear what led Microsoft and Intel to use this particular iSCSI target (the earlier throughput tests ran on NetApp filers), but it does speak to the quality of this product.

It is not clear how many disk drives were used, but I would guess that SSDs or ramdisks might have been employed to pull a million IOPS. Network optimizations are also not mentioned, though jumbo frames would not be a benefit in an IOPS test.

StarWind’s software runs on Microsoft Windows and creates a full-featured iSCSI target, complete with data mirroring, automatic failover and failback, replication, snapshots, and thin provisioning. The company prices their iSCSI SAN at $6,000 for two nodes and competes with the likes of DataCore and Open-E. But the StarWind solution seems at a glance to be more full-featured than these other offerings.

Try It Yourself!

I imagine many folks like me might be tempted to try to reproduce these results. More valuable would be a set of best practice guidelines for the deployment of software iSCSI in Windows Server 2008 R2 and Hyper-V environments. Given the relatively modest hardware involved, there should be nothing stopping us!

These test results also prompted me to get in touch with StarWind to try their iSCSI target software. I was pleasantly surprised to learn that they are currently offering free non-production licenses to VMware vExperts, VCPs, and VCIs as well as Microsoft MVPs, MCPs, and MCT Professionals. Many of my readers fall into one (or more) of those buckets, and I applaud the company for this offer. If only more companies realized the value in giving away test licenses to influencers and thought leaders!

© Stephen Foskett for Gestalt IT, 2010. | How Did Microsoft and Intel Get 1 Million iSCSI IOPS?
Read more posts categorized as Server Virtualization, Storage