You’ve been tasked with interconnecting two ethernet chassis switches. There are lots of reasons you might want to do this. The link you’re building might be between two core switches acting as your main data center routers. The link could be connecting a core switch and distribution or access layer switch. Here’s a brain stream of the pros and cons of various approaches I’ve seen in production environments.
(1) A single cable between the switches
The simplest to implement, but a single point of failure. If the cable fails or the ethernet link on either end has an issue, you’ve lost the link. This could also be a performance bottleneck. Links in between switches tend to carry a lot of traffic. If you’re switches are gigabit, and the interswitch link is 10G, then there might well be enough bandwidth. If the link is only 1G, then it is probably undersized. An oversubscribed, undersized link will result in frame discards, where the switch tries to forward across the link, but was unable to do so before the frame buffer overflowed. Frame discards imply that packets were not delivered to their destination, which in today’s predominantly TCP networks means that packets will be resent. Depending on the severity of the oversubscription, the end user experience will be one of application sluggishness, unexpected disconnects, and other general strangeness. Users might lose response packets (echo replies) when doing ping tests to destination across the link.
(2) Two cables between the switches
In this case, this is not an aggregated link (etherchannel or port-channel in Cisco lingo), but simply two parallel ethernet links. This is an improvement over scenario one, in that you’ve eliminated a single point of failure. However, a loop has been created between the 2 switches. As such, spanning-tree will put one of the two links into a blocked state. Therefore, you’ve got link redundancy, but haven’t done anything to address potential oversubscription concerns.
(3) Two or more aggregated cables between the switches, linecard x to linecard x
Spanning-tree treats an aggregated link like a single interface. Therefore, bundling multiple physical interfaces together using LACP (802.3ad) or PAgP (Cisco proprietary) is common practice for interswitch links. It is possible to bundle links explicitly without an aggregation protocol to save you from yourself (“mode on”), but this is a bad idea as it leaves you prone to inadvertent loops during turn-up if wires are crossed. Crossed wires can happen more often than you might expect, especially if your interswitch link is traversing one or more patch panels.
With an established etherchannel between the switches, the next issue to consider is whether or not there are enough links bundled to adequately handle traffic between the switches. Customarily, etherchannels are built in even numbers of ports. Back in the day, even numbers of interfaces was a requirement to form an etherchannel, but typically you can bundle any number you like today. For purposes of this discussion, let’s say you chose to build the uplink of 8 1Gbps links going from module 1, ports 1-8 on the one switch, to the same on the other side — module 1, ports 1-8.
Remembering that an interswitch link typically carries a lot of traffic and that oversubscription is a concern, it helps to understand HOW your particular chassis switch moves packets around inside. Here’s the big question — does the line card you’re using for the interswitch link have a big enough connection to the chassis backplane to handle the traffic you’re planning to send its way? Typically, you’ll have a certain number of Gbps assigned to each line card slot. How many Gbps will depend on the chassis and the supervisor engine in the chassis. Once you know the speed of the backplane connection, you also need to understand how the line card will allocate those Gbps to the ports. Most line card architecture groups ports, and then gives each port group some amount of Gbps to be shared. Certain ports might be reserved as “non-blocking”. Each card is different, and you’ll need to dig around Cisco’s site to find out how the card you’re working on allocates bandwidth. Here’s a few pages I came across:
- Cisco Catalyst 4500 spec sheet. Classic 4500 chassis support 6Gbps per line card slot. “E” series 4500s support up to 24Gbps. Note that the linecards you put in these chassis have different abilities — taking a classic card out of classic 4500 and popping it into a E series won’t give it 24Gbps worth of backplane speed automatically.
- Cisco Catalyst 4500 module installation guide. Various chassis’, supervisor engines, line cards, and their capabilities are discussed, including specific port groupings for several line cards.
- Cisco Catalyst 6500 data sheet. All the basic chassis specifications are reviewed. Throughput capability of the chassis varies by supervisor engine, but come in 32Gbps, 256Gbps, and 720Gbps flavors per chassis.
- Cisco Catalyst 6500 chassis installation guide.
- Cisco Catalyst 6500 ethernet module installation guide. Includes this chapter, which breaks down the port allocations and oversubscription rate of many line cards.
- Bonus — accessing 6500 backplane utilization via SNMP.
I’ve said all of this about chassis backplane and bandwidth allocation to line card port groups to make a simple point: interconnecting two switches using a bunch of ports on one module (like 1-8) to a bunch of ports on a matching module in the other chassis is risky. Admittedly, you might not have a choice. You’ve got the cards you’ve got. You’ve got the available ports you’ve got. But keep in mind — you’re exposed to a total link failure if the linecard fails. You’re exposed to a total link failure if you’ve uplinked within a specific port group on a linecard, and that port group fails — I’ve seen this happen, where the line card stays up, but a specific port group on the card turns to junk. You’re exposed if the line card is oversubscribed on the backplane; you won’t lose the link, but you might be discarding frames due to busy pipes inbound that the backplane can’t keep up with. If you’ve got the linecards and available ports to deal with, there is another option.
(4) Two or more aggregated cables between the switches, spreading each of the physical links across multiple line cards
In this scenario, you’ve got the benefits of an etherchannel — no spanning-tree blocked ports, and the link stays up as long as at least one physical port is still lit. But by spreading the etherchannel across multiple linecards in the chassis, you’ve reduced your exposure to a linecard or port group failure, and you’ve also made it less likely you’ll run into backplane oversubscription. Of course, this all depends on the architecture of the linecards, chassis, and supervisor engine in play. You need to read up on the specific hardware you’re configuring and come to your own conclusions as far as managing the bandwidth and spreading your risk around.
One final consideration is your etherchannel load-balancing scheme. Very often, the default etherchannel load-balancing scheme will cause one link to see more traffic than the rest of the links in the bundle. I have found that, when available as a load-balancing scheme, hashing on source and destination port number offers a fairly even load distribution, but again, this depends on your switching environment. You need to understand the data flowing across your interswitch link to know what distribution might be the most appropriate…which I suppose is a topic for another article.
Also worth noting is that we focused on making the most of your bandwidth in an oversubscribed world. We didn’t discuss port characteristics, trunking concerns, VLAN pruning, or QoS schemes, all elements that you need to consider when building interswitch links.
