Image via Wikipedia

I Swear To Tell The Truth

Prevarication. Untruths. Misdirection. Deceit. Call it what you will, lying is ugly. While we expect lying from people (there’s an American TV show built around this premise), we expect our network gear to tell only the truth. The command line is supposed to a bastion of honesty. And indeed, the command line will tell you exactly what its perception of reality is. The magic (and our job as engineers) is determining when the device in question is woefully misguided…when what it thinks it knows is no longer true.

Caches can be guilty of storing bad data. When they first learned their data, they had learned truth. But as a cache’s data ages, the possibility increases that the cached data becomes stale: out of sync with reality. When cache gives you stale data, it’s lying: a stiff penalty we sometimes pay for performance.

In Your Stack, Improving Your Performance

In networking, practically all network stacks have an address resolution protocol (ARP) cache as a standard feature. What is ARP used for? Simply stated, ARP discovers the link-layer (think Ethernet MAC) address when only the network layer (think IP address) is known. Keep in mind that ARP isn’t specific to IP or to Ethernet; various flavors of ARP are used with other network protocols and transports. For this discussion, we’ll assume Ethernet and IPv4, though.

Well okay, I’m with you so far…but WHY do we need to know what the link-layer address is? I mean, who cares? A host can know the IP address of another host he wishes to communicate with, but that’s not enough information to send a packet across the network wire. In addition to the remote IP address, the host must be able to build a link-layer frame of a type appropriate for the network media to which the host is connected. In the vast majority of today’s LANs, that media is Ethernet. So inside of an *Ethernet* frame will go the IP packet. ARP is used to discover the Ethernet MAC address of a remote IP address necessary for the destination MAC component of the Ethernet frame. Still with me? Good.

Next step then…what is ARP *cache* used for? Like any cache, an ARP cache stores MAC-IP mappings, so that the next time the host needs to send an IP packet to that destination IP address, the Ethernet destination MAC is already known. An network ARP request is not required before the creation of the frame wrapper if the requisite destination MAC can be pulled out of ARP cache. Naturally, this improves network performance as a whole.

• ARP requests are broadcasts. Broadcasts, by definition, are sent to all hosts on a network segment. The higher the volume of broadcasts on a segment, the lower the overall segment performance (think aggregate throughput). ARP cache helps reduce the number of broadcasts.
• ARP queries take longer to send and get a response than doing a lookup against a local cache. Again, a performance boost.

Show Me Your ARP Cache

Would you like to see the ARP cache on your network-attached device?

• On a Windows host, type “arp -a“.
• On a *NIX host, type “arp -an“.
• On a Cisco router or layer 3 switch, type “show ip arp” or “show ip arp vlan X” where X is your VLAN number of choice.

I’ll spare you the command output, because they all look similar: a list of IPs with associated MAC addresses that are currently stored in the device’s ARP cache.

Why Did The New Core Switch Break The Network?

Very early this past Saturday morning, I was remotely guiding a core switch upgrade for a small data center several hours distant from me. I’d built the switch and shipped it to the site, where one of the local engineers had racked and powered it. The maintenance window was to move the cables over from the scary 10 year old switch that made me nervous every time I did a “wr mem” to the shiny new one. The core switch being replaced acted as the router for all the VLANs at this site. This was not a “dual core” design, where the core switch was one of a pair of switches acting as backups of one another. Rather, this core switch was *it*; without this switch, intervlan traffic was impossible, as no other device on the network was providing routing services.

Being remote and having no backdoor, I was relying on the local engineer to tell me what was going on. Moving the cables was the easy bit. All the fiber uplinking remote access switches was moved over without incident; all the lights were green. The copper to move over the Internet border firewalls was similarly successful. Shortly thereafter, the IPSEC site-to-site tunnel I needed to access the site reconnected, and I was able to log into the shiny new core switch. After resolving some anticipated speed/duplex and trunk issues, we still found that a number of hosts were unable to talk outside of their local VLAN. Hmm…

Now, this was not my first rodeo. I’ve replaced a few routers and switches in my day. Stale ARP caches telling lies and lies and lies is an annoying issue that can really get your knickers in a twist if you don’t know what’s going on. However, I was expecting (and dreading) this very circumstance.

Let’s say you’ve got host 10.100.100.10 in Vlan 100 needing to talk to some host outside of Vlan 100. Now, before the core switch replacement, this worked fine. 10.100.100.10 knew that to talk to someone outside Vlan 100, he’d need to forward that intervlan traffic to his default gateway, let’s say 10.100.100.254, where it would be routed. 10.100.100.10 also had an ARP cache entry that told him what the MAC of 10.100.100.254 was, because sometime earlier, he’d put an ARP request on the wire, asking for the MAC of 10.100.100.254, to which the core switch had responded. With that information readily in his ARP cache, 10.100.100.10 would merely need to peek inside his cache, pull out the appropriate MAC, and then wrap his outbound IP packet in an Ethernet frame bound for the MAC of 10.100.100.254 to obtain routing services. Life was grand whenever he needed to communicate with far away subnets.

That is, right up until I replaced the ancient core switch with a new one. “Now I done it,” so to speak. While the default gateway IP was certainly the same – 10.100.100.254 – the MAC associated with that IP address was changed. Remember that Ethernet MAC addresses are assigned by manufacturers and are supposed to be unique to every Ethernet interface the world over. As a result, every host on Vlan 100 had a lying ARP cache. Hosts including my imaginary 10.100.100.10 would frame their traffic heading for the default gateway with a stale destination MAC stored in local ARP cache back when the old core switch was in production. They would then put that ill-formed frame on the wire where it would go…nowhere. Well technically, that frame went *everywhere*: the switches in the VLAN would try to deliver the frame, unicast-flooding it out every port, where the frame would be ignored by everyone. No one in the VLAN had that MAC address anymore.

The core switch was new and there was nothing wrong with it, but the network was broken.

Beat Your ARP Caches Into Submission

So how do you, the unicorn-taming, leprechaun-catching network engineer, resolve this issue? With broken ARP caches all over the network, you look a bit of an idiot. “Hey, network guy. Nice new switch that doesn’t work. You gonna fix it? Should we cut back to the old switch? I think I’m gonna call my boss because none of the servers are working. I need a hug!” You’d like to hug him with a good swift smack to the side of his Windows-loving head with the dirty side of your keyboard (you know, all that cheezle and dead skin that falls in between the keys), but you’re better than that. No, you’re going to provide a *solution*.

• Wait. As with any cache, the entries in an ARP cache have a time-to-live. Eventually, the TTL will expire, and hosts will start working again as they put fresh ARP requests on the wire and discover the new MAC. How long? That depends on the host OS, but generally speaking a period of a few hours. I’m too lazy to check at the moment, but 300 minutes comes to mind as pretty standard time.
• Clear ARP caches on critical devices by hand instead of waiting for TTL to expire. Most devices have a way to purge the ARP cache at the command line, forcing the device to put an ARP request on the network wire. This will force the ARP cache to be repopulated.
• The “swatting a fly with a Buick” approach is to reboot the broken system. A reboot is never a graceful answer, but depending on your circumstances and who you’re working with, it’s a solution easy to understand for all involved, albeit not always expedient.
• Use a MAC address that doesn’t change from one switch to the other. I’m referring to a first-hop redundancy protocol, like HSRP. HSRP computes a MAC address that floats between members in a HSRP group; that computed MAC is always the same, varying only by VLAN number. Therefore, if you replace an HSRP switch with another HSRP switch, your hosts using the new switch as a default gateway won’t know the difference. There’s nothing stopping you from running HSRP on your core switch, even if you’ve only got a single core.
• Note that some devices use gratuitous ARPs to notify remote hosts about their new MAC addresses. This is commonly found in hosts using high-availability protocols where there is no floating MAC address, but merely ownership of a MAC shifting from one host to another. For example, F5 Networks load-balancers rely on a gratuitous ARP during a high-availability active/standby failover event to notify hosts of the change in roles from one load-balancer to the other.

So let’s hear from the rest of you clever people. How to you beat the stale ARP cache problem?

© ethan for Gestalt IT, 2010. | Coping Mechanisms For A Lying ARP Cache
Read more posts categorized as All, Networking

To read other Drobo related posts:

Drobo S and DroboElite – Introduced 11/23/2009

Some very interesting articles on Drobo S and DroboElite

The Drobo Math

I totally love my Drobo but

PLEASE SEE THE ATTACHED PDF FOR PERFORMANCE DATA

Here are some performance stats collected with running Drobo as a direct attached storage and using some benchmarking tools to compile these results.  All the performance stats are collected using different criteria’s that can affect the performance of the Drobo. Also used various connection methods including USB and Firewire to obtain these stats.

The entire series of tests have been performed using Mac OSX 10.6.2 host with either USB attached or Firewire attached to the Drobo.

These results are limited to the Drobo and cannot be compared to the Drobo Pro or Drobo Elite or Drobo S.

The test were carried out with using various combination of drives, rebuilt data protection, best-case scenario, worst-case scenario, 1 drive, 2 drives, 4 drives, different drive cache & capacity.

Enjoy!!!

© Devang for Gestalt IT, 2009. | Drobo Performance Stats
Read more posts categorized as Desktop, Featured, Storage

Enhancements to EMC Symmetrix V-Max systems is possibly around the corner (FY09 Q4).

FAST (Fully Automated Storage Tiering) is due this quarter and will be one of the most awaited software release in the enterprise storage space by EMC.

Bundled together with FAST, possibly a new microcode version the enables FAST (its associated features) and other expected enhancements.

Though this will be a major software release and functionality upgrade, I don’t think this would qualify as a 2nd generation EMC Symmetrix V-Max system.

But fully expect EMC to release its FAST v2 and V-Max Gen2 (possibly named like V-Max II or V-Max Ext) somewhere around Mid year 2010.

Here are a few new features to possibly expect on the EMC Symmetrix V-Max Systems this quarter.

1. Introduction of FAST v1, which should allow automated data movement within a single Symmetrix V-Max system. Here are some features of FAST as discussed on GestaltIT and by Barry Burke (TSA) on his blog.

2. FAST v1 data movement should possibly be policy driven around factors like time (how old is the data), SLA (promised SLA’s), Tier (from Tier 0 to Tier 1 to Tier 2) and possibly I/O or IOPS based.

3. FAST v1 should allow automated policy based data movement or prompt a user for manual intervention for data movement.

4. Do not expect FAST v1 to come for free, it will possibly be licensed based on the total number of TB’s in the storage subsystem.

5. Expect some integration between the IONIX platform and FAST v1 and possibly some very tight integration with future releases of FAST and IONIX.

6. Expect FAST and IONIX to integrate very tightly with Atmos through API’s and policies. We should expect to see this with FAST v2 and not with FAST v1.

7. So when does EMC retire Symmetrix Optimizer, with FAST v1 probably not, with FAST v2 probably yes.

8. 2TB SATA II drives will be introduced (According to a Keynote from Joe Tucci in NYC), though Joe Tucci didn’t mention what platforms the 2TB SATA II drives will be available on, it seems the V-Max upgrade would be the most logical platform.

9. The 2TB SATA II drive upgrade should make the V-Max 4 PB total storage (2400 drives x 2TB), possibly the single largest storage subsystem at an enterprise level.

10. RapidIO speed upgrade from 2.5 Gbps to 4 Gbps (interconnects between the engines) upgraded either through MBIE (new processors) and / or through microcode upgrades. EMC currently uses Serial RapidIO which supports 1.25Ghz to 3.0Ghz clocking speeds and can scale upto 10 Gbps I/O though EMC currently utilizes 2.5 Gbps as its interconnect speeds for the engines.

11. Increased drive interconnect speeds. May be denser EFD’s, though I can’t speculate on it.

12. FC and FICON (Host Connects) port speeds upgrade from 4 Gbps to 8 Gbps

13. Interconnect between two separate Symmetrix V-Max Systems (8 Engines each per system) expanding into possibly 16 (max) engines. The more I think about this concept, the more it makes me feel that there are no added benefits of this architecture, rather it will add more complexities with data management and higher latency. Remember how the V-Max was initially marketed with having hundreds of engines and millions of IOPS, the only way to achieve that vision is through interconnects. The longer the distance, the more latency with cache and I/O. If Interconnects end up making in this release, limitation on the distance between two Symmetrix V-Max system bays would be around 100 feet.

14. To the point above, another way of possibly connecting these systems could merely be federation through external policy based engines. Ed Saipetch and myself have speculated that concept on GestaltIT.

15. With the use of larger drive size, possibly expect a cache upgrade. Currently the Symmetrix V-Max supports 1TB total cache (512GB usable), which may get upgraded to 2TB total cache (1024 GB usable).

16. New possible microcode version 5875 that will help bring features like FAST, SATA II drives and additional cache into the Symmetrix V-Max.

17. Processors: 4 x Quad Core Intel processors on V-Max engines may not get an upgrade in this release, it should possibly be with FAST v2 as a midlife enhancement next year.

18. Further enhancements related to FCoE support.

19. Upgrade of iSCSI interface on Symmetrix V-Max engines from 1GB to 10GB (is now available with the Clariion CX4 platforms).

20. Really do not expect this to happen, but imagine RapidIO interconnects change to FCoE. Really not sure what made EMC to go with RapidIO instead of Infiniband 40 Gbps (which most of the storage industry folks think is dead) or FCoE with Engine interconnects, but if the engineers at EMC thought of RapidIO as a means to connect the V-Max engines, there has to be a reason behind it. Enginuity more or less doesn’t care about the underlying switching technology, making a switch from RapidIO to FCoE or Infiniband can be accomplished without a lot of pains. Though for customers already invested into RapidIO technology (with existing V-Max systems), it might be offline time to change the underlying fabric, which in most cases is unacceptable.

21. Virtual Provisioning on Virtual LUNs which is currently not supported with the existing generation of Microcode on V-Max systems.

22. Atmos currently is running as a beta release and we should expect a market release this Quarter. Should we expect to see an integration between V-Max and Atmos. I am not sure of any integration today.

23. A very interesting feature to have in the EMC Symmetrix V-Max would be system partitioning, where you can run half the V-Max engines at a certain Microcode level with a certain set of features and other half can be treated as a completely separate system with its own identity (almost like a Mainframe environment). Shouldn’t this be a feature of a modular storage array.

24. Symmetrix Management Console (SMC) and Vmware integration (like VMware aware Navisphere and Navisphere aware VMware). There is already quite a bit of support related to VMware in SMC for provisioning and allocation.

25. Also a much tighter integration between IONIX, FAST, SMC, Navisphere and Atmos may after all be the secret sauce, which would enable workflow, dataflow and importantly automation. Though do not expect this integration now, something to look forward for the next year.

Summary

Though I am still a bit confused on where FAST will physically sit.

FAST v1 can merely be a feature integrated within the Microcode, configurable & driven through policy within the Symmetrix Management Console.

FAST v2 (Sometime Mid 2010) should support in-box and out-of-box (eg: Symmetrix to Clariion to Celerra to Centera) data movement through policy engine.

Ed Saipetch and myself have speculated on GestaltIT on how that may work. Though after some thoughts, I do believe a policy engine can merely be a VM or a vAPP sitting outside the physical storage system in the Storage environment.

To promote the sales of the EMC Symmetrix V-Max systems, Barry Burke in his blog post talks about Open Replicator, Open Migrator and SRDF / DM (Data mobility) are now available at no cost for customers purchasing a new EMC Symmetrix V-Max system, these are some of the incentives that EMC is offering and further promoting the sales of its latest generation Symmetrix technology.

It remains to be seen the path of success FAST will carve for Symmetrix V-Max systems.

© Devang for Gestalt IT, 2009. | Enhancements to EMC Symmetrix V-Max Systems coming!!
Read more posts categorized as Featured, Storage

From a high level, I have tried to cover the differences in terms of performance and architecture related to the directors, engines, cache, drives, etc

It might be a good idea to also run both the DMX-4 and V-max systems through IOmeter to collect some basic comparisons between the front end and coordinated backend / cache performance data.

Anyways enjoy this post, and possibly look for some more related data in the future post.

EMC Symmetrix DMX-4                                EMC Symmetrix V-Max

More architectural details related to drives, cache, directors, cabinets, Mibe, SIB, Service Processor to come in the V-Max architecture expansion and modularity post over the next week.

Enjoy!!!!

© Devang for Gestalt IT, 2009. | EMC Symmetrix DMX-4 and Symmetrix V-Max: Basic Differences
Read more posts categorized as Storage

Vaulting wasn’t available on the Symmetrix 3, 5 & 8 series of machines neither on the DMX nor the DMX-2 technology. With those machines, as a power hit, power fluctuation or environmental issue would be sensed; the BBU (battery backup unit) power would kick in keeping the machine in an online state for 3 minutes. During this time, all the I/O to and from the host is aborted; anything sitting in the cache is de-staged and written to disk drives. At that point, the entire machine goes into a DD (dead) state leaving itself offline (powered on but non-operational) or turning itself off if it’s a power outage.

With the DMX-3, DMX-4 and V-Max systems, since the number of cabinets has expanded; it becomes crucial that the data be saved before the Symmetrix turns itself offline or off. The concept of BBU has now changed to SPS (Standby Power Supply) modules that are locally attached to DAE’s (in Storage cabinets) and also in the System bay, that will keep the Symmetrix running for 5 minutes. During this time, the Symmetrix cuts off all I/O from the host, then lets the cache in the machine sync for consistency and then de-stage all the data from the cache – memory to the vault drives.

With the DMX-4 and V-Max as the memory is mirrored, it copies all the mirrored memory data to vault drives, creating literary two copies of the same data. In the later part of the blog post we will discuss what the rules of vaulting are and how is this technology deployed within the customers EMC Symmetrix storage environment.

The official definition of vault operation as referenced by EMC is: The vault operation is triggered when the Symmetrix system is powered down or transitioned offline or when environmental conditions initiate a vault situation.

During a power up procedure after the shutdown (vaulting), the hardware initializes and restores all the data in the cache – memory from the vault drives which verifies its integrity. Before another vault operation can be initialized, the SPS (Standby Power Supply’s) will have to be charged, which might take hours.

EMC Hopkinton Manufacturing (remember those HK serial numbers, they stand for HopKinton) The picture as published by Boston Globe

 To read the blog post on EMC Serial Numbers, please see here

 The following are the Vault requirements within the V-Max Systems 

1. Each director pair (2 – odd / even) on the V-Max system will require 200GB of vault space, that is 40 x 5GB chucks of dedicated vault data space
2. The vault drives are M1 devices with not Raid or mirroring protection
3. The vault drive cannot to be used by any host and is reserved for the Symmetrix
4. Vault drives cannot be configured by Timefinder/Snap, virtual or dynamic sparing
5. The data space created by the vault drives will be almost equivalent to the size of the cache – memory installed on the machine
6. As mentioned in the previous blog post on enterprise flash drives, EFD’s cannot be used for vaulting operations
7. For permanent sparing, 5 vault drives per loop are essential

© Devang for Gestalt IT, 2009. | Vaulting on EMC Symmetrix V-Max Systems
Read more posts categorized as Storage

Background

Zumodrive is a cloud storage service.  Data is stored on Amazon S3 and offers all the standard AES encryption for stored data and SSL transfer between the client and Zumodrive’s servers.  Client support covers both Mac and PC and now iPhone and you can access your data from multiple platforms at the same time.  Encryption is important; firstly you don’t want anyone snooping on your file transfers and secondly, your data is being added to a public, shared cloud storage service, so you want to ensure it isn’t accidentally or maliciously accessed by anyone.

Secret Sauce

What claims to make Zumodrive different is that only a portion of files are stored locally on the target devices, while the majority of data is kept in the cloud.  The Zumodrive software “intelligently” caches files, attempting to ensure files are available when needed.    I can’t see how this could be more complicated than a simple Least Recently Used (LRU) algorithm, but I wait to be corrected.  On each of your local devices you can specify the location and size of the cache; for PC and Mac clients, I can see no real benefit caching only a portion of data locally and keeping the rest in the cloud.  Local hard drives are cheap; I’ve found so far that the best way to use similar cloud services is to store the most frequently used data in the cloud and keep the infrequently used data on a PC drive, moving data into the cloud service as it becomes part of my “working set”. However, portable devices form the exception to this rule and I was really excited when I saw the availability of the iPhone client as this is the kind of device where having access to all your files and caching a subset could be good.

Unfortunately, so far, I’ve been disappointed.  Here’s why.

Out of Touch

I don’t have an iPhone, so my testing so far has been on my iPod Touch.  Synchronisation works well; I’ve dumped a number of podcasts onto my Z: drive and they appear on my iPod file list, so no problem there.  The issues come trying to access the files.

• Firstly, I can’t see all of my downloaded podcasts in the “Music” section and I can’t play music files from the “Files” section, so in one view I can see the files exist, but can’t play them there, in the other I can play them but can’t see them all.  Also, the “Music” section doesn’t make it clear whether it lists only locally cached files or not.  
• Second, music files seem to play in a separate Zumodrive music player, not in the standard music player in the Touch.  The Zumo version doesn’t let me pause tracks and restart them where I left off.  More annoyingly, if another application starts while I’m listening to a track, then I lose my position.  This is totally impractical when listening to podcasts.  Also, as I use a Bluetooth dongle which integrates with the standard music player, I have to fiddle with pretending to play a standard music track in order to get this device to work.
• Third, the iPod/iPhone version appears to be read-only.  I can’t delete files from my iPod Touch, which is really annoying.  This feature might be deliberate, but I’d like the option to override this and delete content from any of my devices; as I’ve listened to a podcast, I typically delete it.  I don’t want them hanging around.

Summary

Zumodrive and content caching has a great future and its the visualisation of something I was hoping we’d have maybe 7 or 8 years ago.  As the iPhone client is a first release, I hope they quickly review feedback and fix issues like those I’ve raised.  When they do, Zumodrive will be  a “must-have” application.

I’m starting to keep track of cloud services I review.  You can find a summary spreadsheet of features here.  Let me know if there’s any service you’d like to see compared or features added to the list.

© Chris for Gestalt IT, 2009. | Cloud Storage: Review – Zumodrive
Read more posts categorized as All, Cloud Computing, Storage