I have been waiting for some time to get some ESXi 5.0 awesomeness going on in my environment. vCenter has been sitting on v5 for some time and I have been deploying ESXi 5 in a couple stand-alone situations without any issues. So, now that I have more compute capacity in the data center, it is time to start rolling the remaining hosts to ESXi 5… or so I thought!

I downloaded ESXi 5.0.0 Kernel 469512 a while back and have been using that on my deployments. So far, so good… until today. Update Manager configured with a baseline –> Attach –> Scan –> Remediate –> back to business. Surely, Update Manager processes should take more time than the actual upgrade. About 30 minutes after starting the process, vCenter was showing that the remediation progress was a mere 22% complete and the host was unavailable. I used my RSA (IBM’s version of HP ILO or Dell DRAC) to connect to the console. Sure enough, it was stuck at loading some kernel modules. About 20 minutes later IT WAS STILL THERE!

Restarting the host did not resolve the issue. During the ESXi 5 load screen, pressing Alt + F12 loads the kernel messages. It turns out that iSCSI was having issues loading the datastores in an acceptable amount of time. I was seeing messages similar to:

A little research turned me onto the following knowledgebase article in VMware’s KB: ESXi 5.x boot delays when configured for Software iSCSI (KB2007108)

To quote:

This issue occurs because ESXi 5.0 attempts to connect to all configured or known targets from all configured software iSCSI portals. If a connection fails, ESXi 5.0 retries the connection 9 times. This can lead to a lengthy iSCSI discovery process, which increases the amount of time it takes to boot an ESXi 5.0 host.

So, I have 13 iSCSI stores on that specific host and multiple iSCSI VMkernel Ports (5). So, calling the iSCSI lengthy is quite the understatement.

The knowledgebase states that the resolution is applying ESXi 5.0 Express Patch 01. Fine. I can do that. And… there is a work around described in the article that states you can reduce the number of targets and network portals. I guess that is a workaround… after you have already dealt with the issue and the ridiculously long boot.

Finally, to help mitigate the issue going forward, VMware has released a new .ISO to download that includes the patch. However, this is currently available in parallel with the buggy .ISO ON THE SAME PAGE! Seriously. Get this… the only way to determine which one to download is:

As a virtualization admin, I know that I am using the Software iSCSI initiator in ESXi. But, why should that even matter at all?! There is a serious flaw in the boot process in version 469512  and that should be taken offline. Just because someone is not using Software iSCSI at the current time does not mean they are not going to in the future. So, if they download the faulty .ISO, they are hosed in the future. Sounds pretty crummy to me!

My Reaction

I am quite shocked that this made it out of the Q/A process at VMware in the first place. My environment is far from complex and I expect that my usage of the ESXi 5.0 hypervisor would be within any standard testing procedure. I try to keep my environment as vanilla as possible and as close to best practices as possible. 1.5 hours for a boot definitely should have been caught before release to the general public.

Additionally, providing the option to download the faulty ISO and the fixed ISO is a complete FAIL! As mentioned on the download page, this is a special circumstance due to the nature of the issue. I would expect that if this issue is as serious as the download page makes it out to be, the faulty ISO should no longer be available. There has to be a better way!

Conclusion

I have since patched the faulty ESXi 5.0 host to the latest/safest version, 504890, and boot times are back to acceptable. I will proceed with the remainder of the upgrades using the new .ISO and have deleted all references to the old version from my environment.

I have never run into an issue like this with a VMware product in my environment and I still have all the confidence in the world that VMware products are excellent. In the scheme of things, this is a bump in the road.

© Bill for Gestalt IT, 2011. | ESXi 5.0–1.5 Hour Boot Time During Upgrade
Read more posts categorized as Server Virtualization

I had been planning on configuring a couple different CIFS servers (one for general file sharing, departmental files and things and one for home directories). Because these seem to be the fastest to implement and certainly to get out to the users, this might get things up and running sooner than other options.

However, I have also been considering configuring a Windows Storage Server to attach some iSCSI LUNs to and share out to the users that way. I realize that the Storage Server will have some overhead with it, but I will be able to continue using some storage monitoring tools we already own, and have an opportunity to get my hands on the latest iteration of Storage Server. Since a previous release of the product launched me into new and exciting experiences, it might be a good experience to check out the new version.

Waiting for licensing

Because the licensing for the applications to test this at work might be costly and take time to bring the appropriate approvals, my home lab might get me the first taste of things.

My current setup consists of a Windows Server 2008 R2 VM, A Windows Storage Server 2008 VM and a Windows 7 Enterprise client VM.

The storage will be VHDs inside the host server for now (maybe eventually additional storage will make it into the home lab).

Depending on testing

If the initial tests in my lab work out alright I might look into the additional licensing costs for Storage Server. Either way I am hoping to review the product once I get my feet wet.

If any of you have had a chance to take a crack at Windows Storage Server 2008 R2, what are your initial thoughts?  I would love to hear them in the comments.

© derek for Gestalt IT, 2011. | Windows Storage Server 2008 R2 and kicking tires
Read more posts categorized as All, Server Virtualization, Storage

A Compelling Product

At its core, Compellent is a storage software company. Their Storage Center software creates a “virtual storage array” from standard building blocks. The resulting storage product is one of the most flexible and (excuse the pun) compelling midrange storage systems on the market. Consider the following roundtable discussion from our recent Tech Field Day event in Seattle, which includes a discussion of Compellent’s products and technologies by independent voices in the industry:

Podcast: Play in new window | Download (Duration: 8:59 — 102.7MB)

Compellent’s Storage Center scales up, adding additional storage capacity to existing arrays without disruption and integrating this added bandwidth and capacity seamlessly. Compellent also has an impressive tiered storage and thin provisioning story, since data blocks are virtualized and can be moved between tiers dynamically. Full-featured integrated snapshot technology completes the picture, adding data protection credentials.

Compellent’s scale-out story is impressive as well, but is more about replication and virtualization than the clustering technology used by others, including Dell’s EqualLogic. Compellent’s newly-introduced Live Volume can be thought of as a volume manager for virtual servers, allowing their storage to exist simultaneously on two different storage arrays. Supporting VMware vSphere, Microsoft Hyper-V, and Citrix XenServer, Compellent claims that Live Volume enables a “grid of Compellent arrays”, but it’s not the sort of scale-out grid one would get from EMC’s recently purchased Isilon, HP’s LeftHand, Dell’s Compellent, or NAS specialist Panasas. It’s more of an active-active mirror of storage than a true cluster. Learn more about Live Volume in the following video by Rodos:

Let’s also take a moment to consider the hardware Compellent currently uses: They deploy and sell this software on SuperMicro servers with Xyratex disk enclosures, and Seagate hard disk drives, and connect everything with QLogic, Emulex, and Brocade storage networking gear. This is all pretty much standard-issue for storage vendors: Most no longer produce their own controllers or disk enclosures, and the standard components Compellent uses can be swapped from other sources if needed.

Compellent’s Mid-Market Strategy

Compellent has not traditionally competed in the stratosphere of enterprise storage dominated by EMC’s Symmetrix V-Max, the HDS/HP USP/VSP, and (to a lesser extend) the HP 3PAR InServ. Instead, the Minneapolis-based company has been happy to sell into the fat middle of the market, and has gained a loyal following in smaller-but-still-impressive enterprises. Nowhere is this more evident than at Compellent’s annual “C-Drive” customer conference: It’s amazing, really, for a smallish storage company to have such a devoted user base!

Although they offer two NAS front-ends, Compellent is a block storage player through and through. Their “Storage Center with NAS” offering is decidedly low-end, employing Windows Storage Server 2008 on a 1U server for SMB and NFS connectivity. The newer and more-impressive zNAS is based on the OpenSolaris operating system, including ZFS a friendly interface “co-developed” with Nexenta. Neither product is an in-house Compellent design, nor are they competitive in the high-end “big data” market touted by NAS leaders EMC, NetApp, and BlueArc.

Compellent’s Fit Within Dell

Assuming the acquisition goes through, attention will turn to the fit for Compellent’s Storage Center technology and marketing within Dell. What will Dell get for three quarters of a billion dollars? What will it mean to the enterprise storage market as a whole?

Dell’s EqualLogic acquisition, though expensive, is now widely seen as a success. The Round Rock management team allowed EqualLogic the freedom to develop and compete, while bolstering their hardware and manufacturing story with commodity components. The combination has become a dominant plater in the mid-market iSCSI storage space, and Dell is likely to attempt a repeat of that strategy here. It was widely rumored that 3PAR management preferred Dell’s to HP’s offering based on a promise of similar autonomy.

We assume Dell would replace the EMC-sourced CLARiiON with Compellent in their midrange storage arsenal, replacing the SuperMicro servers. Although Compellent would likely be allowed to develop and grow within the Dell umbrella, there is substantial overlap with the existing EqualLogic line. One would expect Compellent to focus more on Fibre Channel and FCoE (which they recently delivered), though iSCSI support is unlikely to be dropped. A longer-term strategy would see EqualLogic and Compellent combining their midrange storage IP and management team within Dell.

Considering their success selling CLARiiON block storage, Dell’s acquisition would strap a rocket to Compellent’s sales. Once the Dell hardware is grafted in, profits will grow substantially as well. This is a major win for Dell, which needs the kind of margins that only in-house IP can drive. The price is right, the market is ready, and the fit is solid.

What Comes Next

Some are suggesting that this is the end of the storage acquisition parade, but many holes remain. Dell still lacks enterprise NAS, data deduplication, and cloud storage success, and companies like Oracle have only just begun to bulk up their storage strategies. Dell’s acquisitions of Ocarina and Exanet have yet to pay dividends, likely driven by the immaturity of those products. One would not be at all surprised if they continued shopping in the storage space: Panasas, BlueArc, Sepaton, FalconStor, and Overland all look ripe for purchase. Although Compellent is a win for Dell, it is not likely to be their last acquisition.

Disclosure: Compellent sponsored and presented at Gestalt IT’s Tech Field Day 3.

© Stephen Foskett for Gestalt IT, 2010. | Thoughts On A Dell Acquisition Of Compellent
Read more posts categorized as Exclusive, Featured, Storage, Top Story

Note: These instructions are being made using an EMC Celerra and Unisphere 1.0. Your SAN hardware experience may vary. So, as always, take this as an example and test in your test environment FIRST.

Stage 0 – Determine how much space you need

The mistake I made in calculating some of the space was that I missed a couple vswap files. Oops… So, now I need to expand the datastore by about 3 GB in order to Storage vMotion the VM into place and keep some room for snaps (and whatnot).

Now… it is worth noting that Celerra SANs work with file systems as the base storage level (the base place you can keep data). If you are hip to Unix/Linux environments, you know that when you format a file system, a portion of the storage is consumed by inodes (and other structures depending on the system). So, a 20GB partition is not necessarily 20GB usable. By default, the Celerra is taking 2% of the disk space for whatever it wants.

Note – I am not an EMC guy, so there may be ways to tweak this. That’s cool.

Recall that I need 3GB extra (which would make it a 23GB file system). Extending by 3GB would leave me with 22.6GB usable (again, recall 2% overhead). Extending by 4GB, though, leaves me with 23.6GB usable (377.0MB overhead). So, 4GB it is!

Stage 1 – Extend the file system

Open the Unisphere Client and browse to the file systems. Locate the file system you are looking for.

Right-click and select “Extend”

So… now that we know we need to add 4GB, we just need to do some quick math because the Extend function wants sizes in MB. 1024MB/GB * 3GB = 4096MB. Plug “4096” into “Extend Size by (MB):” field and click OK.

Now Unisphere is showing that the file system has been extended. Pretty cool, huh!?

Stage 2 – Extend the iSCSI Share

Part of setting up the iSCSI access to a file system requires you to define how much of the file system will be used by the iSCSI access. Initially, I wanted all of the space to be used by iSCSI. So, I told the Celerra to use all 20GB (or 19.7GB after the 2% overhead). Now, though, we are still using the 19.7GB even though we have the additional 3.9GB available.

To resolve the issue, we need to extend the iSCSI share to consume the rest of the space.

In the Unisphere client, browse over to Sharing –> iSCSI

Locate the LUN ID of the share you want to extend. For us, this is LUN #19. Right-click on the share and select “Extend”.

Notice that the size is showing 19.684GB. We’re going to change this now to include the new space we added.

This step contains a little inconsistency in the Unisphere UI that drives me crazy. Specifically, the rounding and display of significant digits. In the graphic above, you see that the iSCSI share is 19.684GB in size. However, notice in the next graphic that we can extend by 3.9GB. Hmmm… What do you want to bet that it is not quite 3.9GB!?

Instead of trying to figure out the exact size, in MB, that we want to extend by, there is a little trick we can employ. The UI catches incorrect sizes, throws and error, and then corrects the error by replacing the value with the largest value possible. So, I am going to enter 4096 into the field knowing that it will fail. However, it will correct my work for me (ah… so nice).

Click OK and extend away.

Now, the Unisphere UI will show that the iSCSI share has been extended.

Stage 3 – Extend the Datastore in vSphere

Even though we have done some pretty cool data storage extension, the vSphere environment is not showing that. Grrrr… Luckily, there is an easy fix for that.

Locate the proper datastore by selecting an ESX host, Configuration tab –> Storage. Right-click on the datastore and select the “Properties” option.

Next, select the “Increase” button

In the following window, you will see a selection of the available devices that we can select to become an extent to the file system. Low and behold, the same LUN #19 has appeared (recall that the iSCSI LUN for the file system was #19!

Select that LUN and click Next.

Magically, vSphere knows exactly what to do with it.

vSphere sees that there is an additional 3.94GB of space available and that the ‘Free space’ will be used to expand the VMFS volume. Click Next!

Click Next to “Maximize capacity”.

Click Finish to begin the magic.

In a mere 6 seconds (for me), the VMFS datastore has been expanded and is available to use.

As a nice aside, this feature will also initiate a Storage Rescan of all ESX hosts connected to the datastore to ensure the change has been reflected everywhere.

Stage 4 – vPat-Yourself-On-The-Back

This procedure to extend a VMFS datastore is super easy. Again, different SAN vendors may have different utilities and procedures for enlarging an iSCSI share on the fly. So, make sure to do you research on the procedure. However, this should go to show that it can be super easy and any VMs running on the store will have little/no impact.

© Bill for Gestalt IT, 2010. | vSphere – Extending VMFS Datastore–Live (With Unisphere)
Read more posts categorized as Server Virtualization

“Convergence” is a buzzword seen in the IT press constantly these days.  All convergence means is placing communications that used to ride on its own network onto one unified network; Ethernet’s cheapness, ubiquity, and ever-growing link speeds makes it the network everything is moving towards.  The first big convergence move was to combine voice networks with data networks, using IP telephony.  The challenges of a converged voice/data network include prioritizing voice traffic over pretty much anything else during times of link congestion, and keeping call quality high by delivering datagrams in a predictable time with a predictable gap in between those datagrams.

Frankly, these problems were and are a big pain in the collective backsides of network engineers everywhere.  Ethernet and IP are not transports intended to deliver traffic in a predictable, prioritized fashion.  Ethernet is a best-effort frame delivery system that’s only survived as long as it has by reducing collision domains down to one via switches.  IP uses higher-level protocols for reliability and underlying devices for prioritization.  Therefore, delivering VoIP frames and packets across a converged infrastructure means a carefully designed and deployed quality of service plan; the larger and more complex the network environment, the greater the potential pain.  Throw in congestion-prone wide-area links of varying transports with their own forwarding nuances (ATM is not frame-relay is not MPLS is not PPP), and the network engineer must know a lot about a lot to deliver an effective end-to-end QoS design.  Even worse, no one hears the engineer scream who is configuring QoS on multiple platforms, all of which might require unique configurations depending on hardware and software to net the same results, even within the same vendor product families.

Convergence Redux

Convergence has evolved with the increasing affordability and adoption of 10-gigabit Ethernet.  With the bandwidth and low-latency of 10G, the industry has pushed towards adding storage to the converged data center Ethernet, the idea being to eventually eliminate the unique Storage Area Network.  Fibre channel is the protocol of major concern here, in that other storage protocols like iSCSI are more tolerant of changing network throughput characteristics.  FC is not tolerant of a changing environment.  FC expects that frames will be delivered on-time, every time, and therein lies a significant challenge for the converged data center Ethernet.  While 10G is an awful lot of bandwidth, simply adding Even More Bandwidth (the tried-and-true method of capacity management for engineers who don’t want to think too hard) isn’t a safe answer.  No matter how much bandwidth is available, the Ethernet carrying Fibre Channel traffic (FCoE) must be able to guarantee a lossless path from host to disk and back.

Don’t Drop The Baby!

Enter Data Center Bridging, or DCB.  DCB comprises a set of proposed standards designed to extend Ethernet such that we can:

• Leverage flow control on up to 8 virtual links. (IEEE 802.1Qbb – Priority-based Flow Control) We can issue ethernet PAUSE frames on specific virtual links, and not interrupt forwarding for the entire physical link.  Practically speaking, we could tell everyone but the storage virtual link to shut up for a moment.
• Prioritize traffic classes within a virtual link. (IEEE 802.1Qaz – Enhanced Transmission Selection)  Here, we can use QoS techniques within a virtual link to prioritize certain kinds of traffic.  Voice – you rule!  Network engineer’s web surfing – go to the head of the line!  Bittorrent from Joe down the hall – tail drop.
• Exchange DCB information with other DCB devices. (also part of IEEE 802.1Qaz – DCBX, and expected to leverage LLDP)  Two devices can learn about each other’s DCB-related link characteristics.
• Optionally notify upstream senders of downstream congestion, allowing the sender to mitigate the congestion through rate-shaping.  (IEEE 802.1Qau – Congestion Notification)  In a backwards way, this reminds me of RSVP, where you can do an end-to-end bandwidth reservation across multiple network devices.  I’ll be interested to see how this is implemented.

All of this gives us the ability to guarantee that storage traffic has both the forwarding capacity and priority over other traffic flows when needed.  Done right, an FCoE frame should never hit the floor.  That said, I am interested to see how much hands-on engineering will be required to make this work as intended.  Legacy QoS is far from automatic in the real world, and it seems fair to assume that a well-designed DCB implementation will require rather a lot of whiteboarding, implementing, monitoring, and tweaking before it perfectly serves the environment it has been deployed in.

Hey, Pal – Ya Wanna Buy An Enhanced Ethernet?

DCB has a couple of implementations:  Data Center Ethernet (DCE) is Cisco’s flavor, and includes an implementation of TRILL.  Cisco markets DCE and related advanced data center technologies under the term “FabricPath“, which is baked into the Nexus 7000 product line.  In some Cisco documentation, they refer to DCE as a superset of DCB and another implementation of DCB, Converged Enhanced Ethernet (CEE).  CEE has a much broader group of networking companies behind it, including Broadcom, Brocade, Cisco, Emulex, HP, IBM, Juniper, and QLogic, and is helping to craft how the DCB standards will finally look when ratified.

The point of DCB then is to provide an interoperable set of standards whereby storage traffic can be guaranteed the bandwidth and flow characteristics it requires across an Ethernet transport, while co-existing with other traffic behaving quite differently.

The problem?  At this point, the jury is out on what form of converged storage the market will claim.  FCoE is seeing increased adoption, but uptake has been slow.  iSCSI has been around for a while, is generally well-understood by engineers who deploy it, and a popular choice.  Vendors are pushing various and usually proprietary fabric schemes.  Cisco, HP, Brocade, and IBM are all making acquisitions and creating product lines to sell you a single-vendor solution for storage, servers, and a converged network to run it on.

One thing’s for certain:  it’s a fun time to be a network engineer.

© ethan for Gestalt IT, 2010. | Don’t Drop The Baby: Data Center Bridging Wants Storage To Trust Ethernet
Read more posts categorized as All, Featured, Networking, Storage

“Bridging versus routing” brings us to the perennial networking debate: Are low-level protocols better?

Unless you’re “in the know”, terms like “layer 2″ can seem mysterious, making it all the more plausible when someone touts the benefits. It seems logical: “Bare-metal” communication must be better, faster, and cheaper than higher-level “everything over IP” approaches, right? But it’s not quite that simple.

This piece assumes you know something about the OSI model.

Ivan Pepelnjak posted a great overview of the “bridging versus routing” debate (Bridging and Routing: is there a difference?), and Marko Milivojevic posed the question in response: “I’m one of those who doesn’t understand the whole L2 obsession of the modern networking world, but…”

It really is an obsession: Data communications folks continually argue about the merits and trade-offs between high-level network protocols and low-level communications. We hear it in storage all the time: FCoE proponents assume performance benefits, and AoE fans add cost advantages to the mix. But many of these claims are unsubstantiated, and iSCSI and NAS protocols like SMB and NFS just keep rolling forward. If storage over IP is so bad, why does iSCSI work (and perform) so well?

One thing often missing in the “layer 2″ arguments is what’s missing when you skip the network layer. There’s a reason IP is so widespread: It may not be the best protocol ever, but it works really well in a huge variety of situations and there is a vast pool of associated technologies that can be drawn upon when using it.

IP can run over just about anything, from FireWire to SONET, so any protocol using IP can (theoretically) run there, too. I’ve run iSCSI over Wi-Fi and WAN links, and it works great out of the box with 10 Gb Ethernet. Protocols that are tightly linked to a layer-2 protocol face stiff challenges when moving to different data links. Witness the difficulty moving Fibre Channel to 10 Gb Ethernet, including all those data center bridging technologies. In fact, when faced with the challenge of long-distance Fibre Channel SAN communication, encapsulation over IP was a natural choice.

IP also has a myriad of wonderful technologies to choose from. The creators of iSCSI were able to pull authentication, encryption, lossless communication, and many other features straight from the existing toy box. Developers of new non-IP protocols have to invent their own solutions to these problems, often with disastrous results. Why reinvent the wheel? Just apply a little CHAP, some IPsec, and roll it in TCP and you’re done!

Photo credit: “Akashi Kaikyo Bridge 明石海峽大橋” by Shenghung Lin

© Stephen Foskett for Gestalt IT, 2010. | The Lure of Layer 2
Read more posts categorized as Networking, Storage

Drobo Elite – Rear View

This post is the first in a series covering the DroboElite SMB storage device from Data Robotics Inc.

The DroboElite is the most fully featured storage array in the Data Robotics range of devices.  Regular readers will know I’ve posted frequently on these devices, from the first Drobo “Classic” to the DroboPro, both of which I own and have purchased myself.  Data Robotics have kindly loaned me a DroboElite for this series of posts.  As you will see, the top of the range model offers all the features the other models provide, plus the benefits of multi-user support.

Background

The original Data Robotics storage device was what is now referred to as the Drobo “Classic”.  This was a 4-bay storage device, connecting to a single host PC/server using USB or Firewire and later eSATA.  This model evolved into the Drobo S and was complemented with the 8-bay DroboPro.  Finally, the DroboElite was released in November 2009, using the same form factor as the ‘Pro but with additional connectivity and features.  Although the ‘Pro and ‘Elite models shared the same look and feel, they are actually different products and so there’s no upgrade process to remove drives from the ‘Pro and install into an ‘Elite shell.

Physical

The DroboElite is an 8-bay device, accommodating standard 3.5″ SATA form factor drives.  The drives are inserted vertically (rather than horizontally in the 4-bay models), with drive’s top side pointing to the right.  No caddy or additional support is required.  From the front, there’s nothing to separate the ‘Pro and the ‘Elite as they look identical.  In fact, the removable magnetic front bezel only says “Drobo” and has no other distinguishing markings.  One thought worth considering for the future is providing a method of identifying multiple Drobo’s that have been installed into a rack or cabinet.  The drive lights can be flashed on and off but this only lasts for a few seconds and wouldn’t help an engineer who’s alone in a data centre or machine room.  This is a particularly important point as the ‘Elite is a multi-user device and so aimed at SMBs with machine rooms and rack-based equipment (a rack mount  kit is available).

Turning to the back of the ‘Elite, we start to see the differences with previous models.  The major one here is the presence of two Ethernet ports, providing network redundancy and multi-pathing support for iSCSI.  The power supply is still built in (and not removable), plus there’s a power switch and a USB connector – that it.  I’m undecided as to whether an integrated PSU is a good or a bad thing and I flip back and forth between internal/external and multiple supplies each time I think about it.  On reflection, the consideration here should be that the Drobo is not a “high availability” storage array in the modular or enterprise class.  It doesn’t support mulitple controllers or other features like mirrored cache, so in reality, multiple power supplies is probably over-configuration.  Obviously dual Ethernet connections is essential in a device that will support multiple hosts; this feature provides redundancy from failures in network equipment or more likely, enables maintenance to be performed on the network infrastructure without impacting the storage.

The addition of a power switch is a positive move (and essential as the power supply is now integrated).  Personally I’d have preferred the power switch to be mounted behind the magnetic bezel on the front of the unit.  This would mean once plumbed in, the ‘Elite (and ‘Pro for that matter) would be managed purely from the front, for disk insertions/removals.

I’ve one final comment on the physical aspects of the ‘Elite and that covers the insertion and removal of drives.  I never like removing spinning active drives and unfortunately there’s still no way to either mark a drive for removal or spin it down.  The unique BeyondRAID feature of the Drobo series means that any drive can be removed from a unit and the device ’self heals’ to compensate for the loss of capacity, rebalancing blocks of data redundantly across the remaining drives.  Of course drive removal doesn’t have to be done after a failure; the whole benefit of the Drobos is the capacity increase functionality (I frequently swap drives in/out as part of testing).  I’d therefore like to be able to spin down or at least highlight a drive I intended to pull, via software, before I physically pull it.  It gives me more confidence that I’ve pulled the correct drive when managing multiple units.

Additional specifications can be found here: http://www.drobo.com/pdf/droboelite-datasheet.pdf

In the next post, I’ll look at connectivity protocol support.

© Chris for Gestalt IT, 2010. | Hardware Review: Drobo Elite – Part I
Read more posts categorized as Featured, Storage

Background

Building arrays from pure memory isn’t new; Texas Memory Systems have had the RamSan series of products on the market for some time now (and there are others out there).  Of course, the problem for many large organisations is how to make use of such an expensive and relatively small device.  There are plenty of use cases where flash/SSD may be useful, however (a) it is difficult to target exactly which applications and (b) for those applications that can be identified, potentially only part of the data will benefit from acceleration.

One solution has been to follow the route of the traditional vendors and add SSD as an extra device within the same hardware chassis.  This isn’t a solution to using SSD but rather a sticking plaster over the problem; the SSD may give better read performance but it is unlikely that writes will be accelerated to the level justified by the additional costs of the SSD device itself.  In addition, the SSD is sitting behind a traditional storage array.  Vendors such as EMC, IBM and Hitachi have spent millions of man-hours and hundreds of millions of dollars on software developments to help smooth the impact and manage the unpredictable performance of hard drives.  Remember that when an I/O request is received, the storage array has no idea where a mechanical device like a hard drive is positioned and so cache, algorithms and that other clever intellectual property have been used to mask these physical inadequacies.

However, despite vendors’ best efforts, spikes and unpredictable response times do occur and there’s no way to remove them and guarantee completely consistent I/O responses.

The Violin Approach

So what happens if you can remove the cost issues and buy an SSD-based array for the same price as tier 1 storage?  This is the route Violin Memory are taking to market – make the SSD storage array as closely priced to tier 1 arrays as possible.  Remove the thought process and complications of determining what to place on SSD by making the price argument irrelevant.

In reality, Violin haven’t reached that price parity yet; prices are quoted around the $20/GB mark, which is around double what I’d expect to see for tier 1 storage (depending on volume).  However it is in the order of magnitude where organisations can look at those troublesome applications that decide that the cost of additional servers, disk spindles or re-writing the application is outweighed by simply moving the application to a Violin SSD device.

I think this is the ultimate tipping point for SSD use; where the cost of improving application performance is exceeded by the cost of moving to SSD, then SSD will win.  Where improving application performance is justified by increased business advantage, the business case is written.

Tech Specs

OK, let’s have a look at the technical specifications for the techies amongst you.  Firstly, today’s device capacity sits at 10TB in 3U and is expected to grow to 20TB in Q3.  I’ve also been told that this capacity is expected to be close to 5x greater by the end of 2010, which means 100TB of memory-based storage in a 3U unit.

The 3200 supports PCIe (x4 & x8)  as well as 4/8Gb Fibre Channel and 10Gb iSCSI and FCoE.  Latency is less than 100 microseconds.

Violin array use VIMMs (Violin’s name for their flash memory cards.  These are grouped together into 1TB units, using RAID-5 technology to manage failures.  Maintenance can be performed online periodically to replace failed VIMM devices.

There’s one major issue with Flash/memory-based arrays that Violin claim to have addressed.  That is the issue of degraded performance over time.  Have a look at the following graphic, showing saturated workload on the Crucial C300 versus X25M from Intel.  This graph and the associated review can be found on Anandtech’s website here.  Very quickly with heavy use, the performance for these devices drops off.  Violin claim their array doesn’t suffer similar issues and can deliver sustained performance.  Of course, we can believe that statement once we’ve seen a review of the product delivering the performance as promised.

Futures

A 10/20TB capacity in 3U isn’t huge by today’s standards.  If Violin Memory can deliver on their promises and bring a 3 to 5-fold increase in performance by year end (with a continual reduction in price) then things start to look interesting.  I’d like to see the results of some long-term stress tests on the 3200 series devices.  I have some more material to post in the coming days, once I can validate what’s open and not under NDA/embarbgo.  In the meantime, here are some questions to ponder:

• Do I have any I/O bound applications?
• Can I measure/determine my I/O bound applications?
• Is there direct businss advantage from increasing I/O throughput?

If you can start answering yes to the above questions, then perhaps SSD-based arrays are for you.

© Chris for Gestalt IT, 2010. | Violin Memory Inc Release New All-SSD Array
Read more posts categorized as Featured, Storage

Why does network-attached storage (NAS) have such a poor reputation? This isn’t what the vendors want to be talking about, but some recent product announcements and discussions led to this thought. IT folks as a whole don’t trust NAS for real work, and 20 years of effort from big names like Sun, Microsoft, NetApp, IBM, and the rest hasn’t changed that.

Fear

Back when I used to teach the “Storage 101″ session at Storage Decisions, I was consistently amazed to find little awareness of enterprise NAS systems. People complained about LUNs and Fibre Channel but when I suggested using NFS or SMB their heads almost exploded. “We would never use that for application storage,” they shouted. “File servers are for home directories, not data center stuff!” Clearly, NAS faces an uphill battle.

In a recent piece I wrote, I referred to what I consider to be the prime best practice: Use the right tool for the job. It’s a simple statement, and one that resonates beyond IT and the technology world. But it can be devilishly difficult to see what the right tool is sometimes. Why not use NAS for virtual machine storage? NetApp has been beating that drum for years, yet NAS has a very small footprint in VMware. How about databases on NFS? Exchange over SMB? Block storage has a massive lead over NAS in all of these areas.

Rear this “best practices” piece, Use Process Solutions For Process Problems, Technical Solutions For Technical Ones

IT folks seem downright fearful of file-level storage protocols. Has NFS really burned them that badly over the decades? Can SMB/CIFS really be as bad as they think?

Loathing

I wonder if this terror has more to do with the products people have used than the fundamental concept of file services. Many NAS servers (and clients) are barely functional. Sadly, NFS and SMB are easy to get 80% right, but the 20% corner case interaction takes decades to overcome. My daily storage consulting work exposes me to a myriad of NAS configurations, and few of the multi-platform combinations end well.

Note: Although it has long been known by a variety of names, the Windows NAS protocol is currently called Server Message Block or SMB. Common Internet File System (CIFS) was a failed mid-1990’s attempt by Microsoft to make this protocol standard on the Internet.

Consider the Mac. Apple added an SMB client to OS X in 2001 but, despite many updates, it is far from reliable. Mac users in general loathe connecting to Windows file servers, and business users have located numerous bugs in the handling of Mac-specific file types. It’s bad enough that one company, GroupLogic, created an entire AFP server for Windows just to solve these tricky issues.

This situation often happens in reverse, too. Windows admins are justifiably cautious when deploying non-Windows SMB servers, whether software (Samba, Novell, etc) or system (NetApp, Celerra, BlueArc, etc). As a very early NetApp user, I watched their CIFS/SMB server evolve over a decade and a half into a fairly robust solution, but the early years were downright painful.

Lest you throw rocks at Redmond, know that SMB is not alone with functionality problems. The interoperability of NFS servers and clients is a bit better thanks to open(ish) standards and open source implementations, but its reputation is just as bad. And Apple’s proprietary AFP protocol is downright notorious.

I’ve been there myself many times. I tried to set up a home server based on open source software (Linux, FreeBSD, Samba, Netatalk, etc) but rejected it outright after many frustrating years. Today I use a Mac Mini for file sharing in OS X and serving iTunes music and movies (goodbye, Firefly!) And years of fighting with Samba in enterprise environments taught me two things: It’s possible to get it running well with Windows clients but it’s astonishingly easy to get it wrong.

Enterprise NAS?

We all know that interoperability is devilishly difficult. I don’t envy the NetApp and EMC engineers that have to tweak and tune their server for every possible client, bugs and all. And I am impressed that, after probably millions of man-hours of work, they were able to come up with something stable for a subset of use cases. But this just makes me even more cautious about third-party NAS servers.

I talk to storage vendors all the time, and many of their new products support NFS and SMB. But my internal alarms start going off when I hear about these products. There are two simple reasons for this:

1. As mentioned above, NAS is rare in primary data center applications. It may be common for user files (euphemistically called “unstructured data”) and certain distributed applications (simulation, rendering, etc), but most use cases still call for block SCSI (FC/iSCSI) storage.
2. As further mentioned, getting NAS right takes a massive amount of effort. New and small vendors tend to slap Samba on their (Linux-based) box and call it a day. This is very, very far from being sufficient for enterprise use.

This is why I usually ignore NAS functionality in storage systems except for long-tenured and deep-pocketed vendors. Although the world is turning to “Unified Storage” and multi-protocol support, I’m focusing primarily on block (SCSI) and cloud (REST) capability because the former has proven somewhat easier than NAS to get working and the latter is both simple and “green field” with no legacy concerns.

© Stephen Foskett for Gestalt IT, 2010. | Why Do I Ignore NAS?
Read more posts categorized as Storage

Iomega’s new ix12-300d brings EMC’s SOHO company into the data center

EMC’s Iomega unit today released the 12-drive rack-mount storage product we have all been waiting for. It was never a question of whether Iomega could produce such a beast: The EMC LifeLine software and Iomega hardware were definitely up for it. The question was always if EMC would direct Iomega to fill the gaping hole in their storage lineup between the 4-drive ix4-200r and the entry-business AX4 arrays. The new ix12-300r packs 12 drive bays, scaling from 4 TB all the way to 24 TB, and backs it with quad gigabit iSCSI, redundant power, and everything else the small data center needs.

Stepping Up

The ix12 is a big step up. Although they sold multi-drive RAID systems even before the EMC acquisition, this new device is unlike anything we’ve seen before from Iomega. This slim (2U) chassis is all drives up front, with 12 hot-swap slots full of 3.5″ SATA storage. Under the hood is a dual-core 3 GHz Intel Core2duo E8400 “Wolfdale” CPU, a major horsepower upgrade from the single-core Celeron in the ix4-200r. It also has double the memory (2 GB) compared to that 4-drive product.

If you’re concerned about performance after trying out a desktop ix4-200d, you needn’t worry. The desktop unit has a lowly 1.2 GHz Marvell 6281 and just 512 MB of RAM. While this might be enough for a desktop user, it could never handle the pounding of servers in a shared networked configuration. The ix4-200r, with its 3.2 GHz Intel Celeron D 352 “Cedar Mill“, offered much better performance even though its name was just one letter different. And the ix12’s CPU is three times faster still, though it remains a single-controller system.

More significant changes lurk around the back of the ix12. Dual redundant power supplies, a frequent request in this class, and variable-speed fans, surround four Ethernet ports. Each sports gigabit speed and the set supports Microsoft Windows MPIO, can be aggregated with 802.3ad, or used in VLAN configurations with up to 4 VLAN tags per port. The ix12 speaks just about every language, from NFS and SMB to AFP and iSCSI, and now adds WebDAV and DFS support, too.

A few limitations separate this new ix12 device from its enterprise-grade brothers, however. As noted, a single controller manages all access, so redundancy and parallel processing are limited. Although the ix12 sports 12 drive bays, it only has four SATA channels internally; each bay shares a channel with two others using SATA expanders. Don’t expect to push wire speed over all four Ethernet ports at once, even with all this newfound CPU power.

A Wall of Drives

Base ix12s ship with 4 drives installed, but we were disappointed to learn that additional drives must be purchased in 4-disk packs from Iomega. Although this decision is understandable, the ix series remains a holdout amid growing legions of bring-your-own-drive competitors. At least the company supports mixing and matching drive sizes, including 1 TB and 2 TB at present. We suspect that the unit uses the same reliable 5900 rpm Seagate Barracuda LP drives as the ix4-200d.

Iomega added a few tricks to the LifeLine software to take advantage of a possible 12 drives installed. First up is the addition of dual-parity RAID-6 for improved data protection. The company (and this reviewer) suggest this over RAID-5 once more than 5 drives are combined in one set. Don’t worry, though, because RAID configuration can be changed online and any unused drive can be used regardless of its location in the array. The ix12 also adds drive spindown, saving power when the RAID set isn’t in use.

Like the ix4, any portion of a RAID set can be carved out into an iSCSI LUN for Ethernet-connected hosts. Iomega claims that LUN provisioning times have improved with the added horsepower and software tweaks, and we hope this is true. A maximum of 256 LUNs can be configured in this way, though even 12 drives are unlikely to drive much performance to that many storage users.

Where to Use It?

Although not listed yet, Iomega promises that the ix12 will have a place on the Exchange ESRP, Windows Server and Hyper-V logo list, and VMware Compatibility Guide this month. It’s already the first Iomega product to be “EMC E-Lab Tested”, meaning it is on the EMC Support Matrix; this fact alone speaks volumes of EMC’s expectations for the unit. My own experience shows that Iomega iSCSI is fine for smaller VMware and Hyper-V deployments.

Clearly, the ix12 is a new kind of Iomega array. If the 200d and 200r were a sign that the company wanted to move out of the house, the ix12 is a demonstration that they have graduated. Starting at US $5,000, the ix12 is all business and its resume ought to impress in interviews. It can’t quite boast the scalability and redundancy of established arrays (including its brothers from EMC), but it ought to be an easy acquisition for companies looking for a little more storage here or there.

One is left with questions, though: How big will EMC let Iomega get? If 12 drives are acceptable, what about 24? Is SAS off limits? What about 10 Gigabit Ethernet and even Fibre Channel over Ethernet (FCoE) eventually? Can we dream of dual controllers? At some point, the Iomega lineup could even threaten the CLARiiON!

Then there is the competitive landscape. Iomega leapfrogged the 8-drive Data Robotics lineup and landed squarely in competition with the likes of the revitalized Overland Storage but at a much lower price. We also have Netgear, HP, Dell, and Promise, and there is an attractive D-Link box packing 15 drives and 10 GbE. Iomega also has to worry about its own big brother, the Dell/EMC AX4, starting around $12k. It’s a competitive market, and Iomega is in for a fight as even more vendors wake up to the possibilities in this market.

© Stephen Foskett for Gestalt IT, 2010. | Iomega Graduates and Goes to Work with the ix12-300r
Read more posts categorized as Storage