EMC Symmetrix BIN file, largely an unknown topic in the storage industry and practically there is no available information related to it. This post is just an attempt to shed some light as to what a BIN file is, how it works, what’s in it and why is it essential with the Enginuity code. Some EMC folks have capitalized on the BIN file as to the personality it brings to the Symmetrix, while the EMC competition always uses it against them as it introduces complexities in the storage environment with management and change control.

Personally I feel a Symmetrix wouldn’t be a Symmetrix if the BIN file weren’t there. The personality, characteristics, robustness, compatibility, flexibility, integration with OS’s, etc wouldn’t be there if the BIN file didn’t exist.

With the total number of OS’s, device types, channel interfaces and flags it supports today, sort of making it one of the most compatible storage arrays in the market. The configuration and compatibility on the Symmetrix can be verified using the E-Lab navigator available on Powerlink.

So here are some facts about the BIN file

• Only used with Symmetrix systems (Enginuity Code)
• BIN file stands for BINARY file
• BIN file holds all information about the Symmetrix configuration
• One BIN file per system serial number is required
• BIN file was used with Symmetrix Gen 1 in 1990 and is still used in 2010 with Symmetrix V-Max systems
• BIN file holds information on SRDF configurations, total memory, memory in slots, serial number of the unit, number of directors, type of directors, director flags, engines, engine ports, front end ports, back end ports, drives on the loop, drives on the SCSI bus, number of drives per loop, drive types in the slots, drive speeds, volume addresses, volume types, meta’s, device flags and many more settings
• The setup for host connection if the OS is Open Systems or Mainframe environments using FICON, ESCON, GbE, FC, RF, etc is all defined in the BIN file. Also director emulations, drive formats if OSD or CKD, format types, drive speeds, etc is all defined in the BIN file
• BIN file is required to make a system active. It is created based on customer specifications and installed by EMC during the initial setup
• Any ongoing changes in the environment related to hardware upgrades, defining devices, changing flags, etc is all accomplished using BIN file changes
• BIN file changes can be accomplished 3 ways.
  • BIN file change for hardware upgrades is typically performed by EMC only
  • BIN file change for other changes that are device, director, flags, meta’s, SRDF configurations etc is either performed through the SYMAPI infrastructure using SymCLI or ECC (Now Ionix) or SMC (Symmetrix Management Console) by the customer. (Edited based on the comments: Only some changes now require traditional BIN file change, typically others are performed using sys calls in enginuity environment)
• Solutions enabler is required on the Symcli, ECC, SMC management stations to enable SYMAPI infrastructure to operate
• VCMDB needs to be setup on the Symmetrix for SymCLI, ECC, SMC related changes to work
• Gatekeeper devices need to be setup on the Symmetrix front end ports for SymCLI, ECC, SMC changes to work
• For Symmetrix Optimizer to work in your environment, you need DRV devices setup on your Symmetrix.(Edited based on comments: Only required until DMX platform. Going forward with DMX3/4 & V-Max platforms it uses sys calls to perform Optimizer changes).

Back in the day

All and any BIN file changes on the Symmetrix 3.0, Symmetrix 4.0 used to be performed by EMC from the Service Processor. Over the years with introduction of SYMAPI and other layered software products, now seldom is EMC involved in the upgrade process.

Hardware upgrades

BIN File changes typically have to be initiated and performed by EMC, again these are the hardware upgrades. If the customer is looking at adding 32GB’s of Cache to the existing DMX-4 system or adding new Front End connectivity or upgrading 1200 drive system to 1920 drives, all these require BIN file changes initiated and performed by EMC. To my understanding the turn around time is just a few days with these changes, as it requires change control and other processes within EMC.

Customer initiated changes

Configuration changes around front end ports, creating volumes, creating meta’s, volume flags, host connectivity, configuration flags, SRDF volume configurations, SRDF replication configurations, etc can all be accomplished through the customer end using the SYMAPI infrastructure (with SymCLI or ECC or SMC). These are performed through Sys calls and not necessarily using traditional BIN changes DMX-3 systems onwards.)

Enginuity upgrade

Upgrading the microcode (Enginuity) on a DMX or a V-Max is not a BIN file change, but rather is a code upgrade. Back in the days, many upgrades were performed offline, but in this day and age, all changes are online and accomplished with minimum pains.

Today

So EMC has moved quite ahead with the Symmetrix architecture over the past 20 years, but the underlying BIN file change requirements haven’t changed over these 8 generations of Symmetrix.

Any and all BIN file changes are recommended to be done during quite times (less IOPS), at schedule change control times. Again these would include the ones that EMC is performing from a hardware perspective or the customer is performing for device/flag changes.

The process

During the process of a BIN file change, the configuration file typically ending with the name *.BIN is loaded to all the frontend directors, backend directors, including the global cache. After the upload, the system is refreshed with this new file in the global cache and the process makes the new configuration changes active. This process of refresh is called IML (Initial Memory Load) and the BIN file is typically called IMPL (Initial Memory Program Load) file.

A customer initiated BIN file works in a similar way, where the SYMAPI infrastructure that resides on the service processor allows the customer to interface with the Symmetrix to perform these changes. During this process, the scripts verify that the customer configurations are valid and then perform the changes and make the new configuration active.

To query the Symmetrix system for configuration details, reference the SymCLI guide. Some standard commands to query your system would include symcfg, symcli, symdev, symdisk, symdrv, symevent, symhost, symgate, syminq, symstat commands and will help you navigate and find all the necessary details related to your Symmetrix. Also similar information in a GUI can be obtained using ECC and SMC. Both will allow the customer to initiate SYMAPI changes.

Unless something has changed with the V-Max, typically to get an excel based representation of your BIN file, ask your EMC CE.

Issues

You cannot run two BIN files in a single system, though at times the system can end up in a state where you can have multiple BIN files on various directors. This phenomenon typically doesn’t happen to often, but an automated script when not finished properly can put the system in this state. At this point the Symmetrix will initiate a call home immediately and the PSE labs should typically be able to resolve these issues.

Additional software like Symmetrix Optimizer also uses the underlying BIN file infrastructure to make changes to the storage array to move hot and cold devices based on the required defined criteria. There have been quite a few known cases of Symmetrix Optimizer causing the above phenomenon of multiple BIN files. , Though many critics will disagree with that statement. (Edited based on comments: Only required until DMX platform. Going forward with DMX3/4 & V-Max platforms it uses sys calls to perform these Optimizer changes).

NOTE: One piece of advice, never run SYMCLI or ECC scripts for BIN file changes through a VPN connected desktop or laptop. Always run all necessary SymCLI / SMC / ECC scripts for changes from a server in your local environment. Very highly recommend, never attempt to administer your Symmetrix system with an iPhone or a Blackberry.

Hope in your quest to get more information on BIN files, this serves as the starting point..

Cheers
@storagenerve

NOTE: Read additional comments and clarifications on this topic at the Storagenerve Blog

© Devang for Gestalt IT, 2010. | EMC Symmetrix: Bin File
Read more posts categorized as Storage

A while back, I discussed speculation from EMC around Emulex’s proposed cloud-block storage appliance, E3s (Enterprise Elastic Storage).  With my current focus on Cloud Storage, I thought it would be good to delve a bit deeper into some of the aspects of why block-based cloud computing could prove tricky and why without an appliance it may be impossible.

Block Storage Legacy

Today, block-based I/O still uses the SCSI protocol to communicate between a host/server and a disk storage device.  SCSI has been around since 1981 when devices were physically connected to the server itself using a controller board and old-style ribbon cables.  Obviously we’ve progressed somewhat since then and seen the virtualisation of the physical SCSI interface into Fibre Channel and IP SCSI implementations (and in the future FCoE).  Both FC and iSCSI have removed the need for a dedicated SCSI controller card and replaced it with Host Bus Adaptors (HBAs) and Network Interface Cards (NICs).  Irrespective of this, the underlying communication protocol remains the same and the concept of the “Initiator” (host/server) and “Target” (storage device) persist today.  The Initiator starts (or initiates) an I/O request; the target services that request, reading or writing data.

Writing In Blocks

We need to focus for a moment on the concept of block-based I/O versus file-based I/O.  Block-based I/O has no concept of the format of the data being written to the block device (let’s call it a LUN).  This is in contrast to file-based I/O where the storage device understands the data format and manages the content accordingly, ensuring data access is serialised correctly and that files are held in a logical structure (a file system).  Unfortunately block-based I/O is just “dumb” storage and the host itself is responsible for overlaying a file system onto block-based devices.   These JBBDs (”Just a  Block-Based Device”) can be used singly or combined in complex ways to create the file system the host sees.  This combination can be achieved using native Logical Volume Managers (LVMs) on the operating system, or add-ons such as Veritas Volume Manager.  Consequently, an individual LUN could contain an entire file system or only a small component of one.  Either way, the host operating system depends on one feature of the storage device to ensure data integrity and that’s Write Order Consistency.

Write Order Consistency

Preserving the order in which data is written to disk is a fundamental requirement for modern Journalling file systems like NTFS.  Retaining write consistency ensures the file system can be recovered in the event of a server failure or failure of the link between server and disk.  Ordered writes are also essential where data is replicated from one storage array to another usually at a remote location.  In the event that the primary array is lost, the file system can be recovered in a consistent fashion on the remote device, even if it isn’t 100% up to date.

Remote replication can occur either synchronously or asynchronously.  In synchronous mode, I/Os are acknowledged from the remote array before the I/O is confirmed as completed to the host.  This ensures both the primary and remote copies are write-order consistent because the host doesn’t receive acknowledgement of I/O completion until both the local and remote copies are written.  Write-order consistency is implicitly guaranteed.  However, the penalty for this guarantee is the increase in latency (or host I/O response time) which results and increases with the distance between the two devices.  Asynchronous replication is slightly different.  Write I/Os to the primary array are acknowledged immediately, then queued up for writing to the remote device.  The delay in writing data to the remote location is dependent on the bandwidth available and the latency (effectively the distance) between the devices. In any event, as long as write-order is maintained, the remote copy can be recovered even if it doesn’t represent the absolute latest copy of data.  One last consideration.  We touched earlier on how LVMs can combine single LUNs to create complex file systems.  When asychronous replication is involved, write I/O for a single entity like a file system need to be treated together for write-order consistency.  Therefore Consistency Groups allow multiple LUNs to be grouped together for write ordering, ensuring that all I/O for the file system is ordered correctly.  This requirement doesn’t exist for synchronous replication as the consistency is guaranteed by ensuring the write I/O has completed on both the source and target array before acknowledgement to the host.

The Need for an Appliance

Knowing how write-ordering affects consistency is important in understanding how a block-based device would be replicated into “the cloud”.  Due to latency issues, it is unlikely that synchronous replication would be offered as a method of replicating data from a host server into Amazon S3 or EMC Atmos.  Instead, the most likely process will be asynchronous processing and that means installation of a dedicated appliance.  The question is, where in the data path should it sit?

The Splitter

There’s no requirement for a host/server to be connected into a storage array in order to utilise cloud storage.  Instead, at some point in the data path between host and local disk, a copy of the write I/O needs to be taken.  This could occur at the file system level using a host agent or in a SAN environment could happen in the fabric or from the array itself.  Wherever data is taken from, some form of “I/O splitter” is needed to capture write I/O as it is being transferred to disk.    This technology already exists today in products like EMC’s RecoverPoint and Brocade’s Data Mobility Manager.

So here are our requirements for a block-based storage protocol:

• Write Order Consistency
• Consistency groups
• An I/O splitter or replicator

A Theoretical Implementation

Here’s how I think block-based storage could be implemented.  I’ll use the Atmos and Amazon S3 protocols to demonstrate the process.  Firstly, data will be stored in blocks.  Both S3 and Atmos store data as objects and so each object will need to represent a block.  The file system structure can be used to store individual LUNs, with a directory representing a LUN.  For example LUN 12, block 12343 could have the object name:

\S3\Array1\LUN12\12343

It’s worth noting that there’s a distinct difference between the way Atmos and S3 implement updates to objects.  S3 replaces the entire object, whereas Atmos allows part of an object to be updated.  So, Atmos could store an entire LUN as an object whereas S3 can’t, unless the entire LUN is replaced on each write.  Clearly that’s impractical but does indicate that each API implementation will have certain benefits and disadvantages.

So, how big are these blocks going to be?  Ideally, they’d be as small as a typical hard drive block at 512 bytes, however blocks of this size will seriously hamper throughput if write consistency is to be maintained; imagine 5ms latency into the cloud; that’s only 200 IOPS and consequently a throughput of 100KB/s. What’s more practical is a block size matching the O/S  file system and/or database, say 8KB.  Even at this size, with 5ms latency and a single thread of I/O, that’s only 1.6MB/s throughput.

Obviously this level of throughput is not going to be acceptable and there’s a real sticking point here.  The cloud isn’t intelligent.  It will write data as it’s received.  There’s no locking control and the delivery mechanism could be unpredictable.  If writes are issued in parallel, there’s no way to guarantee the I/Os are written to the cloud in the right order.  So perhaps a different approach is required.  Data writes to the target LUN need to be written in a log format, with the name of the object comprising both the block number and a sequence number.  This could be something simple as follows:

\Atmos\Array1\LUN12\SequenceNumber-BlockNumber

e.g.   \Atmos\Array1\LUN12\123445343434366-12343

As the LUN is written to, the sequence number (unique to the LUN or consistency group) is incremented for each write.  The I/Os can then be written in parallel as the sequence numbers track what has and has not been received.  At this point there are two choices – retain all the block updates (unlikely due to the growth in storage usage) or post process the writes, deleting all the written blocks where another later copy exists and where there are no gaps in the sequence numbers.  If there is a gap, then the LUN writes are only guaranteed back to the point where the sequence number gap occurred.  Restoring the LUN for access means processing the LUN block data before it can be read again by the host.

Summary

OK, so this post presents an idea of some of the issues involved in writing block-level data into the cloud.  Data needs to retain integrity and consistency, but performance and throughput are an issue.  Cloud storage has no intelligence, so writing and managing data needs to be handled somewhere, probably using an appliance.  The appliance guarantees the data integrity which can’t be achieved with the cloud alone.  Each Cloud Storage API implementation will have similar features, so using generic CRUD (Create/Read/Update/Delete) commands on objects representing blocks means any service could be used to store data.  It also enables data to be replicated between services so vendor lock-in can be avoided.

I’d be interested in receiving feedback to see if anyone else has thought how block-based cloud storage could be achieved.

© Chris for Gestalt IT, 2009. | Cloud Computing: Block-Based Storage
Read more posts categorized as Cloud Computing, Featured, Storage

Typically look at a CX4-960, which can be scaled up to 960TB of raw storage and can have as many as 960 disk drives in it. With certain failure rates guaranteed, large number of drives can create a higher probability of failure. Every storage manufacturer these days includes some sort of hot sparing technology in the storage subsystems. EMC started offering this technology to its customers as Global Hot Spares. Then came an era where some value add offerings were brought in for proactive failures to minimize the chance of data loss. This brought to the table a technology that is termed as Proactive Hot Spares, where proactively failing drive is determined and global hot spare is kicked in.

I believe flare release 24 started offering Proactive hot spares. With this Flare release customers can proactively initiate a kickoff of hot spares through Navisphere or Naviseccli against a suspect drive.

Depending on the RAID type implemented, the RAID Groups can withstand drive failures and can run in degraded state without data loss or data unavailability. With RAID 6 implemented, a machine can have as many as 2 drive failures in the same RAID group, with RAID 5, a machine can have as many as 1 drive failure in the same RAID group, with RAID 1/0, RAID 1 a machine can have as many as 1 drive failure in the RAID group without data loss.

Drives supported on Clariion CX, CX3, CX4, AX and AX4 systems typically are FC (Fiber Channel), SATA II and ATA drives.

A Global Hot Spare has to be configured in an EMC Clariion system as a single RAID Group (with one drive). Once the RAID Group is created, a LUN should be bound as a Global Hot Spare before it could be activated.

The following is the sequence of steps that take place on a Clariion Subsystem related to Global Hot Spares (Supported on CX, CX3, CX4 systems)

1. Disk Drive failure: A disk drive failure in the system, Flare Code marks it bad.
2. Hot spare invoked: A preconfigured Global Hot Spare is invoked based on the Global Hot Spare selection criteria.
3. Rebuild: The Global Hot Spare is rebuilt from surviving raid group members.
4. Failed drive replaced: Failed disk drive is replaced with a good drive by a Customer Engineer
5. Copy Back: The Global Hot Spare copy has to finish before the new drive starts rebuilding. The rebuild or equalize happens in a sequential order of LBA (Logical Block Address) and not the LUNs bound no it.
6. Return Hot Spare: Once the sync of new drive is finished, the hot spare is invalidated (zero’ed) and put back in the Global Hot Spare pool.

The following is the sequence of steps that take place on a Clariion Subsystem related to Proactive Hot Spares (Supported on CX300, CX500, CX700, CX3, CX4). Proactive Hot Spares essentially use the same drives that are configured as Global Hot Spares.

1. Threshold of errors on Disk Drive: A drive gets hit with errors, it surpasses the number and type of those errors, and the flare code marks it as a potential candidate for failure.
2. Proactive Hot Spare invoked: Based on the potential candidate’s (drive) type, drive size and bus location a Global Hot Spare is indentified and the process is kicked off for data rebuild.
3. Potential candidate fails: Once the Proactive Hot Spare is synced, the flare code fails the indentified potential candidate.
4. Failed drive replacement: The failed drive is replaced by a Customer Engineer
5. Copy Back: From the proactive hot spare, the data is copied back to the newly inserted drive. The rebuilt or equalize happens in a sequential order of LBA (Logical Block Address).
6. Return Proactive Hot Spare: Once the sync of new drive is finished, the hot spare is invalidated (zero’ed) and put back to the Global Hot Spares pool.

The Global Hot Spares Selection Criteria:

The following are the criteria’s that are followed with selection (invoke) of a Global Hot Spare when a potential proactive candidate is identified or disk drive is failed. In the sequence listed below, Drive type is the first selection, Size of the drive is the second selection and location of the Global Hot Spare is the third selection. Speed of the drive (RPM) is not a selection criterion.

1. Type of Global Hot Spare Drive: As discussed above, Clariion Systems use three primary drive types. For FC and SATA II type drives, either or can be invoked against each other type. ATA drives can be invoked against an ATA drive failure.
2. Size of Global Hot Spare: Upon a disk failure, the drive size (Global Hot Spare) is examined by Flare Code. The size of failed drive is not the key in invoking the hot spare, but the total space of all LUNs (bound) on the drive is used as a determination criteria.
3. Location of Global Hot Spare: Based on the above two criteria, the location of the Global Hot Spare is considered as the third criteria. If the Global Hot Spare is on the same bus as the failed drive, it will be considered as the primary selection if the above two criteria’s are met. If the above two criteria’s are met and the drive is not on the same bus, then the Global Hot Spare is selected from other buses.

Other Considerations:

1. RAID Types: For the copy of data, with RAID 3 and RAID 5 data on the hot spare is built using the parity drive. With RAID 6 raid types, data on the hot spare is built using the RP (row parity) and / or DP (Diagonal Parity) depending on the number of failures in the RAID Groups. For the RAID 1/0 and RAID 1, data on the hot spare is built using the surviving mirrors.
2. Copy Times: The time required to copy or rebuilt a hot spare really depends on how large the drive is, the speed of the drive, the cache available on the drive, the cache available on the array, the type of the array, raid type and the current job processing on the array. Typical rebuilt times vary from 30 minutes to 90 minutes again depending upon how busy the storage subsystem is.
3. Global hot Spare types: For every 30 drives (2 DAE’s of drives), consider having 1 drive as a Global hot spare. Also verify, for every drive type (size, speed) in the machine, you have at least one configured global hot spare. Good idea to have global hot spares on various different buses and spread across multiple Service Processors.
4. Vault Drives: Vault Drives cannot be used for Global Hot Spares. The Vault drives are considered as the first 5 drives [ 0_0_0, 0_0_1, 0_0_2, 0_0_3, 0_0_4 ] on the Clariion System. If a vault drive fails, a Global Hot Spare takes over its position.
5. Rotational Speed: Rotational Speed of the Global Hot Spare is not considered before invoking it. It might be a good idea to have Global Hot Spares running 15K RPM’s potentially with large size drives.
6. Mixed Loop Speed: With certain Clariion Systems like CX3’s, available loop options are 4GB and / or 2GB and you can have a mixed loop speed in your machine, for hot spare selection the loop speed is not considered, in those cases it might be wise to have similar hot spares on both the 2GB and 4GB loops.

© Devang for Gestalt IT, 2009. | EMC Clariion Systems: Global Hot Spares & Proactive Hot Spares
Read more posts categorized as Storage

So the journey of Symmetrix systems started with Moshe Yanai (along with his team) joining EMC in late 80’s. A floating story says, the idea of a cache based disk array was initially pitched to both IBM and HP and was shot down.  EMC was predominately a mainframe memory selling company back in the late 1980’s. The Symmetrix products completely changed the direction of EMC in a decade.

Joe Tucci comes in at the end of 90’s from Unisys with a big vision. Wanted to radically change EMC. Through new acquisitions, new technologies, vision and foremost the integration of all the technologies created today’s EMC.

Symmetrix has always been the jewel of EMC. Back in the Moshe days, the engineers were treated so royally (Have heard stories about helicopter rides and lavish parties with a satellite bus waiting outside for a support call). Then comes the Data General acquisition in late 90’s that completely changed the game.

Some people within EMC were against the DG acquisition and didn’t see much value in it. While the Clariion DG backplane is what changed the Symmetrix to a Symmetrix DMX – Fiber Based Drives. Over this past decade, EMC radically changes its position and focuses on acquisitions, support, products, quality, efficiency, usability and foremost changing itself from a hardware company to an Information Solutions company focusing on software as its integral growth factor.  New acquisitions like Legato, Documentum, RSA, kept on changing the culture and the growth focus within EMC.

Then came VMware and it changed the rules of the game, EMC’s strategic move to invest into VMware paid off big time.  Then happens the 3-way partnership between VMware – EMC – Cisco, to integrate next generation products, V-Max (Symmetrix), V-Sphere and UCS are born.

Here we are in 2009, almost at the end of 20 years since the inception of the Symmetrix, the name, the product, the Enginuity code, the robust characteristics, the investment from EMC all stays committed with changing market demands.

Jumping back into the Symmetrix, here are a few articles you might find interesting, overall talking about various models, serial numbers of the machines and importantly a post on Enginuity Operating Environment.

• To read about EMC Symmetrix Enginuity Operating Environment
• To read about EMC Symmetrix Serial Number naming convention,
• To read about EMC Symmetrix Models in a previous blog post
• To read about various EMC models based on different Platforms
• To read about all EMC Clariion models since the Data General Acquisition

Symmetrix Family 1.0

• ICDA – Integrated Cache Disk Array
• Released 1990 and sold through 1993
• A 24GB total disk space introduced

Wow, I was in elementary school or may be middle school when this first generation Symmetrix was released….

• Symmetrix 4200

Symmetrix Family 2.0

• ICDA – Integrated Cache Disk Array
• Released 1991 and sold through 1994
• A 36GB total disk space
• Mirroring introduced

• Symmetrix 4400

Symmetrix Family 2.5

• ICDA – Integrated Cache Disk Array
• Released 1992 and sold through 1995
• RSF capabilities added

(I actually met a guy about 2 years ago, he was one of the engineers that had worked on developing the first RSF capabilities at EMC and was very instrumental in developing the Hopkinton PSE lab)

• Symmetrix 4800:

Symmetrix Family 3.0 also called Symmetrix 3000 and 5000 Series

• Released 1994 and sold through 1997
• ICDA: Integrated Cache Disk Array
• Includes Mainframe Support (Bus & Tag)
• Global Cache introduced
• 1GB total Cache
• NDU – Microcode
• SRDF introduced
• Supports Mainframe and open systems both
• Enginuity microcode 50xx, 51xx

• Symmetrix 3100: Open systems support, half height cabinet, 5.25 inch drives
• Symmetrix 5100: Mainframe support, half height cabinet, 5.25 inch drives
• Symmetrix 3200: Open Systems support, single cabinet, 5.25 inch drives
• Symmetrix 5200: Mainframe support, single cabinet, 5.25 inch drives
• Symmetrix 3500: Open Systems support, triple cabinet, 5.25 inch drives
• Symmetrix 5500: Mainframe support, triple cabinet, 5.25 inch drives

Symmetrix Family 4.0 also called Symmetrix 3000 and 5000 Series

• Released 1997 and sold through 2000
• RAID XP introduced
• 3.5 Inch drive size introduced
• On triple cabinet systems 5.25 inch drives used
• Supports Mainframe and Open Systems both
• Timefinder, Powerpath, Ultra SCSI support
• Enginuity microcode 5265.xx.xx, 5266.xx.xx

• Symmetrix 3330: Open Systems Support, half height cabinet, 32 drives, 3.5 inch drives
• Symmetrix 5330: Mainframe Support, half height cabinet, 32 drives, 3.5 inch drives
• Symmetrix 3430: Open Systems Support, single frame, 96 drives, 3.5 inch drives
• Symmetrix 5430: Mainframe Support, single frame, 96 drives, 3.5 inch drives
• Symmetrix 3700: Open Systems Support, triple cabinet, 128 drives, 5.25 inch drives
• Symmetrix 5700: Mainframe Support, triple cabinet, 128 drives, 5.25 inch drives

To read about EMC Symmetrix Hardware Components

Symmetrix Family 4.8 also called Symmetrix 3000 and 5000 Series

• Released 1998 and sold through 2001
• Symmetrix Optimizer Introduced
• Best hardware so far: least outages, least problems and least failures (not sure if EMC will agree to it, most customers do)
• 3.5 inch drives used with all models
• Enginuity microcode 5265.xx.xx, 5266.xx.xx, 5267.xx.xx

• Symmetrix 3630: Open Systems support, half height cabinet, 32 drives
• Symmetrix 5630: Mainframe support, half height cabinet, 32 drives
• Symmetrix 3830: Open Systems support, single cabinet, 96 drives
• Symmetrix 5830: Mainframe support, single cabinet, 96 drives
• Symmetrix 3930: Open Systems support, triple cabinet, 256 drives
• Symmetrix 5930: Mainframe support, triple cabinet, 256 drives

Models sold as 3630-18, 3630-36, 3630-50, 5630-18, 5630-36, 5630-50,3830-36, 3830-50, 3830-73, 5830-36, 5830-50, 5830-73, 3930-36, 3930-50, 3930-73, 5930-36, 5930-50, 5930-73 (the last two digits indicate the drives installed in the frame)

To read about EMC Symmetrix Hardware Components

Symmetrix Family 5.0 also called Symmetrix 8000 Series

[ 3000 (open sytems) + 5000 (mainframe) = 8000 (support for both) ]

• Supports Open Systems and Mainframe without BUS and TAG through ESCON
• Released 2000 and sold through 2003
• 181GB Disk introduced
• Enginuity microcode 5567.xx.xx, 5568.xx.xx

• Symmetrix 8130: Slim cabinet, 48 drives
• Symmetrix 8430: Single cabinet, 96 drives
• Symmetrix 8730: Triple cabinet, 384 drives

Some models sold as 8430-36, 8430-73, 8430-181 or 8730-36, 8730-73, 8730-181 (the last two digits indicate the drives installed in the frame)

To read about EMC Symmetrix Hardware Components

Symmetrix Family 5.5 LVD also called Symmetrix 8000 Series

• Released 2001 and sold through 2004
• LVD: Low Voltage Disk Introduced
• 146GB LVD drive introduced
• Ultra SCSI drives cannot be used with the LVD frame
• Mainframe optimized machines introduced
• 4 Slice directors introduced with ESCON and FICON
• FICON introduced
• Enginuity microcode 5567.xx.xx, 5568.xx.xx

• Symmetrix 8230: Slim cabinet, 48 drives, (rebranded 8130, non lvd frame)
• Symmetrix 8530: Single cabinet, 96 drives, (rebranded 8430, non lvd frame)
• Symmetrix 8830: Triple cabinet, 384 drives, (rebranded 8730, non lvd frame)
• Symmetrix 8230 LVD: LVD frame, slim cabinet, 48 LVD drives
• Symmetrix 8530 LVD: LVD frame, single cabinet, 96 LVD drives
• Symmetrix 8830 LVD: LVD frame, triple cabinet, 384 LVD drives
• Symmetrix z-8530: LVD frame, Single cabinet, 96 drives, optimized for mainframes
• Symmetrix z-8830: LVD frame, Triple cabinet, 384 drives, optimized for mainframe

Some models sold as 8530-36, 8530-73, 8530-146, 8530-181 or 8830-36, 8830-73, 8830-146, 8830-181 (the last two digits indicate the drives installed in the frame)

To read about EMC Symmetrix Hardware Components

Symmetrix DMX or also called Symmetrix Family 6.0

• Released Feb 2003 and sold through 2006
• Direct Matrix Architecture (Data General Backplane) introduced
• DMX800 was the first DMX system introduced
• 4 Slice directors introduced
• RAID 5 introduced after being introduced on DMX-3
• First generation with common DA / FA hardware
• Introduction of modular power
• Enginuity Microcode 5669.xx.xx, 5670.xx.xx, 5671.xx.xx

• Symmetrix DMX800: Single cabinet, DAE based concept for drives, 96 drives (I swear, a customer told me, they have ghost like issues with their DMX800)
• Symmetrix DMX1000: Single cabinet, 18 drives per loop, 144 drives total
• Symmetrix DMX1000-P: Single cabinet, 9 drives per loop, 144 drives total, P= Performance System
• Symmetrix DMX2000: Dual cabinet, modular power, 18 drives per loop, 288 drives
• Symmetrix DMX2000-P: Dual cabinet, modular power, 9 drives per loop, 288 drives, P=Performance System
• Symmetrix DMX3000-3: Triple cabinet, modular power, 18 drives per loop, 3 phase power, 576 drives

To read about EMC Symmetrix DMX Hardware components

To read about EMC Symmetrix DMX models and major differences

Symmetrix DMX2 or also called Symmetrix Family 6.5

• Released Feb 2004 and sold through 2007
• Double the processing using DMX2
• DMX and DMX2 frames are same, only directors from DMX must be changed to upgrade to DMX2, reboot of entire systems required with this upgrade
• RAID 5 introduced after being introduced on DMX-3
• 64GB memory introduced
• 4 Slice Directors
• Enginuity Microcode 5669.xx.xx, 5670.xx.xx, 5671.xx.xx

• Symmetrix DMX801: 2^nd generation DMX, Single cabinet, DAE based concept for drives, 96 drives, FC SPE 2 (I swear, a customer told me, they have ghost like issues with their DMX800)
• Symmetrix DMX1000-M2: 2^nd generation DMX, Single cabinet, 18 drives per loop, 144 drives
• Symmetrix DMX1000-P2: 2^nd generation DMX, Single cabinet, 9 drives per loop, 144 drives, P=Performance System
• Symmetrix DMX2000-M2: 2^nd generation DMX, Dual cabinet, 18 drives per loop, 288 drives
• Symmetrix DMX2000-P2: 2^nd generation DMX, Dual cabinet, 9 drives per loop, 288 drives, P=Performance System
• Symmetrix DMX2000-M2-3: 2^nd generation DMX, Dual cabinet, 18 drives per loop, 288 drives, 3 Phase power
• Symmetrix DMX2000-P2-3: 2^nd generation DMX, Dual cabinet, 9 drives per loop, 288 drives, P=Performance System, 3 Phase power
• Symmetrix DMX3000-M2-3: 2^nd generation DMX, Triple cabinet, 18 drives per loop, 576 drives, 3 Phase power

To read about EMC DMX Symmetrix Hardware components

To read about EMC Symmetrix DMX models and major differences

Symmetrix DMX-3 or also called Symmetrix 7.0

• Released July 2005 and still being sold
• 8 Slice directors
• 1920 disk (RPQ ‘ed to 2400 drives)
• DAE based concept introduced
• Symmetrix Priority Controls
• RAID 5 introduced and then implemented on older DMX, DMX-2
• Virtual LUN technology
• SRDF enhancements
• Concept of vaulting introduced
• Enginuity microcode 5771.xx.xx, 5772.xx.xx

• Symmetrix DMX-3 950: System Cabinet, Storage Bay x 2, 360 drives max, Modular Power, 3 Phase power
• Symmetrix DMX-3: System Cabinet, Storage Bay x 8 (Expandable), 1920 drives max, RPQ’ed to 2400 drives, 3 Phase power

To read about differences between EMC Symmetrix DMX3 and DMX4 platforms

Symmetrix DMX-4 or also called Symmetrix 7.0

• Released July 2007 and still being sold
• Virtual provisioning
• Flash Drives
• FC / SATA drives
• RAID 6 introduced
• SRDF enhancements
• Total Cache: 512 GB
• Total Storage: 1 PB
• Largest drive supported 1TB SATA drive
• Flash drives 73GB, 146GB later now support for 200GB and 400GB released
• 1920 drives max (RPQ’ed to 2400 drives)
• Enginuity microcode 5772.xx.xx, 5773.xx.xx

• Symmetrix DMX-4 950: System Cabinet, Storage Bay x 2, 360 drives max, Modular Power, 3 Phase power
• Symmetrix DMX-4: System Cabinet, Storage Bay x 8 (Expandable), 1920 drives max, RPQ’ed to 2400 drives, Modular power, 3 Phase Power

Some models sold as DMX-4 1500, DMX-4 2500, DMX-4 3500 and DMX-4 4500

• To read about a blog post on EMC Symmetrix: DMX4 Components
• To read about differences between EMC Symmetrix DMX3 and DMX4 platforms
• To read about different drives types supported on EMC Symmetrix DMX4 Platform
• To read about differences between EMC Symmetrix DMX4 and V-Max Systems

Symmetrix V-Max

• (Released April 2009)
• Enginuity Microcode 5874.xxx.xxx
• Total number of drives supported: 2400
• Total Cache: 1 TB mirrored (512GB usable)
• Total Storage: 2 PB

All features on the V-Max have been discussed earlier on the blog post linked below

• Symmetrix V-Max SE: Single System Bay, SE=Single Engine, Storage Bay x 2, 360 drives max, cannot be expanded to a full blown 8 engine system if purchased as a SE, 3 Phase power, Modular Power
• Symmetrix V-Max: System Cabinet, Storage Bay x 10, 2400 drives max, modular power, 3 phase power

• To read about differences between EMC Symmetrix DMX4 and V-Max Systems
• To read about different drives types supported on EMC Symmetrix V-Max Platforms
• To read all about the EMC Symmetrix V-Max Platform

Conclusion

I could have easily added total memory capacity per frame, total number of dedicated DA/DAF slots, total slots, total universal slots, total memory slots, but then I didn’t know information on some of the old systems and didn’t want to be incorrect on them.

Hope you have enjoyed reading this post, with a bit of history related to the Symmetrix platform. I am pretty positive, as of today you will not find this consolidated information on any blog or the manufacturers website.

I really wish, EMC decided to open blogging to some Symmetrix, Clariion, Celerra, Centera specialist that support these systems on a day to day basis, the information that could come out from those guys could be phenomenal. Barry Burke writes a lot of stuff, but again a lot of FUD from him against IBM and HDS, its great reading him, but only a controlled amount of technical information comes from him.

© Devang for Gestalt IT, 2009. | EMC Symmetrix, 20 Years in the making
Read more posts categorized as Featured, Storage

Here are some requirements and limitations related to using the RAID-6 technology on the EMC Clariion platforms.

• RAID-6 is only supported with Flare Release 26 and above on Clariion systems.
• Flare 26 only works on the EMC Clariion CX300, CX500, CX700, all CX3-xx platforms and all CX4-xxx platforms.|
• Any systems running below Flare Release 26 (example Release 13, 16, 19, 24) are not compatible to run RAID-6 (Clariion Systems like CX200, CX400 and CX600).

• Minimum disk required to support RAID-6 with Clariion systems is 2 or 4 or 6 or 8 or 14 data disks with 2 Parity disks (Your typical configuration would look like 2D+2P or 4D+2P or 6D+2P or 8D+2P or 14D+2P, where D = Data Disk and P = Parity Disk)
• To configure RAID-6, you will need even number of disk drives in the RAID Group that you are trying to configure.
• RAID-6 is supported on either EFD (Enterprise Flask Disk) or Fiber (FC) or ATA or SATA drives on EMC Clariion Systems.
• RAID-6 Raid group (RAID SET) can be implemented within an enclosure or expanded beyond a single enclosure
• RAID-6 can co-exist in the same DAE (disk array enclosure) as a RAID-5 and/or RAID-1/0 and/or other RAID types.
• RAID-6 supports global hot sparing like other RAID technologies.
• Supports MetaLUN expansion through concatenated or striped expansion only if all the meta member LUNs are RAID-6 devices (LUNs).
• RAID-6 configuration is possible through Navisphere and naviseccli only.
• With RAID-6 traditionally supported CLI interfaces like Java CLI and Classic CLI have been retired.
• Defragmentation with RAID-6 is currently not supported on Flare Release 26.
• You cannot add new drives to an existing RAID-6 LUN, but you can expand the LUN through RAID-6 MetaLUN technology. Example of this will be, if you have a 6D+2P RAID-6 set and would like to add 16 more drives to the same RAID Group, you cannot accomplish it, but if you manage to create either 2 sets of 6D+2P or 1 set of 14D+2P, and then run a MetaLUN concatenate, you will be able to necessarily achieve the same end result.
• You can have Clariion systems with various different RAID group technologies in the same global domain, but again from a management perspective certain traditional CLI interfaces will not work with RAID-6.
• Using the Virtual LUN Technology with Flare Release 26, now customers can migrate various LUNs (RAID-5, RAID-1/0) to RAID-6 technology. The technology allows the new RAID-6 LUN to assume the exact identity of the previous LUN making the migration process much easy.
• Traditional replication and copy software’s like SANCopy, SnapView, MirrorView, and RecoverPoint are all supported for RAID-6 technology.
• Never use RAID-6 technology with a mix of EFD, FC, ATA and SATA drives in the same RAID Group.
• Never use RAID-6 technology with a mix of various drive speeds like 15K or 10K or 7.2K RPM, drive speed should be exactly similar.

• Oh the most important note: 2 drive failures in the same RAID Group and no data loss or data unavailable (DU / DL), making this a very robust RAID technology. There are some performance overhead related to use of RAID-6 systems with small and random writes. While there is an added penalty with Row Parity and Diagonal Parity calculations on the Clariion.

If you would like to see any further post on RAID-6 workings on Clariion Platforms, please feel free to leave a comment.

To read about other RAID-6 implementations with various platforms, please see below.

EMC Symmetrix RAID 6

SUN StorageTek’s RAID 6

HP’s RAID 6

NetApp’s RAID–DP

Hitachi’s (HDS) RAID 6

Different RAID Technologies (Detailed)

Different RAID Types

© Devang for Gestalt IT, 2009. | EMC Clariion RAID-6 requirements and limitations
Read more posts categorized as Storage

It seems like anytime someone talks about FCoE, they end up also talking about unified fabric. After having read a number of different articles and posts regarding FCoE, I can see where FCoE would be attractive to shops with significant FCP installations. In my mind, though, this doesn’t necessarily mean unified fabric. Given the political differences in organizations—think the “storage team” and the “networking team”—how likely is it that an organization may adopt FCoE, but not unified fabric? Or how likely is it that an organization may adopt FCoE, intending it to be a transitional technology leading to unified fabric, but never actually make it all the way?

So here’s my question: is unified fabric an inevitability?

And here’s a related question: Most people cite VoIP as proof that the unified fabric is inevitable. More so than anything else, I believe VoIP’s success was more a reflection of the rising importance of TCP/IP networking. If so, does that give iSCSI an edge over FCoE? Is iSCSI the “VoIP of the storage world”?

© Scott for Gestalt IT, 2009. | Is Unified Fabric an Inevitability?
Read more posts categorized as All, Storage

It does not cover SAN in any detail and the FC coverage is limited to this is how you present a LUN as a fibre channel device but this led on to some interesting conversations around the complexity of FC vs iSCSI.

I, for sometime, have been saying that the complexity of FC is over-stated and actually it is not really any harder than iSCSI. This often leads to looks of disbelief and complete disagreement, it is almost as if I am spouting heresy. The iSCSI camp think I am mad and the FC camp seem to think that I’m diminishing them.

But, if you take OnTap; there is really very little difference in how you present a iSCSI LUN compared to how you present an FC LUN. It is certainly no harder to do FC from an array management point of view.

And then we go to the host; lets take Windows for example. Now this is where I think a lot of the perception of the simplicity of iSCSI comes in; Microsoft have implemented a very nice software initiator, it’s there and it’s standard. A bit of pointing and clicking and you are there.

Unfortunately, traditional FC cannot be implemented completely in software and needs FC HBAs, hence we need to install additional drivers and software to make it work; these are not part of the standard Windows build and suddenly it all becomes ‘complex’.

If we go to Unix, we end up mucking about configuration files in general for both iSCSI and FC; so really it’s not any harder to do FC than iSCSI.

So if it’s not hard to do at the host level and if it’s not hard to do at the array; where is it hard? And this is where I think it becomes more interesting; it’s the network! Is a large Data Centre IP network harder to set-up than a large Data Centre FC network?

Arguably, the FC network is easier but it is different. In the FC network you have a lot less to worry about; you run less protocols, services, it’s non routeable, the security model is simpler, there is less potential for different workloads to clash, you do not have address space management to worry, you do not have name services to worry about and if I were a Network Admin, I would argue that it’s them who are being diminished by this constant claim that iSCSI is easier.

To do either iSCSI or FC properly is probably equally hard. If you just want to bung a block-level array in and do not care about segregation of traffic, quality of service, don’t care whether your IP back-up traffic and IP storage traffic contend and make your back-ups over-run. If you know you’ve got enough headroom on your existing IP network to carry your block traffic, go ahead with iSCSI; it’s easier because you’ve already got the infrastructure in place.

But if you want put in place a dedicated storage network; the choice is not as clear-cut as people would like to make out. Even when you start looking at cost; yes GbE is cheaper than FC but FC is generally running at 4 gig now and is faster. FC ports are in my experience are significantly cheaper than 10GbE ports. So if you need the throughput, then FC might well be cheaper than iSCSI.

There might be a small saving in the number of FTEs you require as you could have a single Network team but I believe that FC is actually so simple that if you get over the politics, you could have a single Network team which manages both FC and IP. This is purely politics and a turf-war!

If you are a small shop and you have just a few administrators who do everything, iSCSI might also make sense but don’t believe FC is hard; administering a small SAN with a couple of switches might not add a huge amount of additional overhead.

Neither iSCSI or FC are wrong answers but make sure that when you have got to the answer, you show your working. And when a vendor tells you what the answer is, ask them to show your their working and challenge it.

Of course, if I was building a green-field data-centre and could simply start again, I’d probably look at putting in Data Centre Ethernet which would give me the option of FCoE. I would have a single Network team from the get go.

It’ll be interesting to see if Microsoft bundle a software initiator for FCoE into Windows at some point; then I think we’ll see perceptions of complexity change again.

Yes, I’ve completely ignored NAS but actually many of the same arguments apply.

I’d welcome some thoughts; that’s if anyone is still reading and not exploded in apoplectic rage at the heretic!

© Martin for Gestalt IT, 2009. | Protocols, Religions and Heresy!
Read more posts categorized as All, Featured, Storage

The first storage performance horseman is spindles: If you don’t have enough disk units, performance will suffer. I have been laying out storage on enterprise arrays since the dark ages, and one of the first lessons I learned was allocating data to avoid hotspots. I remember spending hours back in the 1990’s hunched over custom Excel spreadsheets trying to get my storage layout just right, balancing the workload across every available disk.

 

This is how we used to avoid hotspots in 1998: Carefully planning every detail of the storage layout.

Each disk drive consists of a spindle of spinning platters with read/write heads move back and forth. Each time you access a piece of data that’s not in cache, the drive moves its arm over the platter to access the correct piece of data. Since each drive can only access one piece of data at once, and since caches can only hold so much data, tuning a system to minimize the number of requests per drive is essential.

Manual storage array layout was an art, but we never fooled ourselves into thinking our designs were optimal. There were just too many intractable problems, so we had to compromise at every turn:

• We usually had no performance data to base our layout decisions on, so we had to rely on guesses and rules of thumb
• Workloads tend to change over time and manual layouts are painful to modify
• The smallest unit of allocation was an entire LUN or drive, so even the best disk layout mixed hot and rarely-accessed data everywhere
• Much of the allocated space was unused, so we used expensive disks to store nothing

One might think that, 10 years later, advances in technology would have solved these basic issues. But for many people using many of the so-called modern mainstream enterprise storage systems, these problems remain.

Like all good systems administrators, I’m a natural control freak. I am uncomfortable letting the system manage itself, having been burned too many times by computers (well, software really) making stupid decisions. It’s analogous to the backlash against anti-lock brakes, traction control, and automated transmissions among racing enthusiasts.

 

Do we allow technology to help us get better performance, or do we try to micro-manage everything? Photo by ClearInnerVision

But the time has come to let go. We don’t have to micro-manage storage anymore, and we have much to gain by letting the array do the work:

• Just as traction control can manage each wheel independently, something a driver could never do, modern virtualized storage systems can allocate small “chunks” to the optimal drive type, creating a better layout than anyone could manage with LUNs
• Dynamic optimization technology can move these chunks around, adapting as loads change
• Thin provisioning can go a step further, not wasting drive capacity for unused space
• Wide striping and post-RAID storage systems have a higher threshold before performance suffers due to spindle hotspots
• Widespread availability of tiered storage, including advanced caches, solid state drives, high-performance SAS and FC, and cheap bulk disks, gives us many more options

As I mentioned, not all systems have these capabilities, and not all implementations are created equal. I’m concerned about misuse of thin provisioning, for example, but it’s hard to argue with its effectiveness in many circumstances. Find out how granular your system’s allocation is – some remain LUN-only, while others are much more effective, using tiny chunks.

These new storage automation technologies really become essential once high-dollar flash storage is added to the mix. If you’re paying 30 times more for a flash drive, you want to make sure you’re making the best use of it that you can! Look at IBM’s recently-announced SAN Volume Controller (SVC) and solid state drive (SSD) combination, for example: It will almost certainly have fine-grained thin provisioning of SSDs, and should be able to dynamically move data between flash and disk storage and even between different storage arrays, but I still have questions on how granular this capability will be. HDS can do similar things with their USP-V. NetApp’s V-Series NAS systems will do dynamic allocation, thin provisioning, and data deduplication to enable a better return on the flash drive investment. I’d love to see 3PAR, Compellent, Dell/EqualLogic, and HP/LeftHand apply their solid dynamic allocation tech to solid state storage as well!

Then there’s the 800 lb gorilla: EMC. More enterprise SSD has probably been shipped out of Hopkinton than every other vendor combined, and both the CX and DMX support (optional/expensive) “virtual provisioning” (aka, thin provisioning) of flash storage. But EMC’s Optimizer is not widely used, and only migrates entire LUNs based on user input – hardly the kind of dynamic and granular technology needed to optimally use all of that flash storage. I’m sure the company is working on addressing this issue, though. Perhaps it will appear in the DMX-5 announcement we are all expecting this year?

© Stephen Foskett for Gestalt IT, 2009. | How I Learned to Stop Worrying and Love Storage Automation
Read more posts categorized as Storage