VXLAN is the future of network virtualization. It’s a huge step forward from all of the other encapsulation protocols that we’ve been using before. However, it’s currently being treated like a premium product by traditional networking vendors. How can we build a future proof network without a second mortgage on our existing equipment?
Enter Mellanox
As it turns out, one of the easiest ways to make a cheaper VXLAN switch is to make your solution the best of breed instead of the best of what’s there. Existing vendors marry their hardware solution to their software stack and need to derive value from each component to ensure that the research and development dollars are compensated.
Mellanox takes a different approach. They make hardware. They created the Spectrum series of open Ethernet switches to provide VXLAN support in hardware. The Spectrum chipset allows Mellanox to build a platform that can do full layer 2 and layer 3 mobility as well as layer 3 equal cost multi path (ECMP), as well as multitenancy and over 100,000 VXLAN tunnels.
But, hardware is nothing without software running on it. So, Mellanox gives you the flexility to run different software on top of your platform. Instead of locking you into an OS that requires a license to run VXLAN, Mellanox will let you install their Mellanox Onyx OS to run the switch. They will also support Cumulus Linux or Microsoft SONiC. This means that you can install whatever flavor of OS you’re most comfortable with and make sure that VLAN is still supported without the need for a costly license.
Mellanox also gives you the ability to do true layer 3 VXLAN routing. Some traditional switches that have been upgraded to use software VXLAN termination end points (VTEPs) can on route VXLAN packets by recirculating them through the switch one or more times. This reduces performance and lowers the throughput of the switch. It also creates an environment where changing traffic conditions could cause packet loss. Because Mellanox can do VXLAN routing in a single pass, there’s no worry of failure due to recirculating packets.
Not a RoCE Road
The most exciting part of the Mellanox story is what they’re doing around storage and memory. Because Mellanox has roots in the storage industry as well as networking, and because they can do VXLAN at the chip level, they have the ability to do some exciting things with the protocol in areas that might not have been readily apparent at first.
Mellanox can extend Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE). RoCE is how you can remotely access a storage or memory location via standard Ethernet cabling. While that sounds interesting in theory and all to impractical, you need to look at the bigger picture of where technology is headed today to truly understand why that’s important.
One example is the explosion of GPU-based computing. With NVIDIA and AMD creating more and more complicated GPUs over time and with software developers taking advantage of these GPUs to build computing clusters for things like AI and machine learning, you need to be able to deliver data quickly to them in order to take advantage of their processing power. And there isn’t a single system in the world today that can hold the number of GPUs you want to use to crunch this data.
Enter Mellanox and RoCE with VXLAN. Now you have a network overlay system that can dynamically deliver data to a GPU cluster or a number of GPU clusters and be rapidly reconfigured as needs change. And it does this all at the chip level without the need for expensive licenses in software. You can add hosts and resources to your GPU farms and, thanks to the Mellanox multitenant architecture, ensure that as new hosts and customers are brought on board that their GPU-bound traffic stays separated from everything else traversing the highway.
The sky is the limit with RoCE as well. Since high-speed Ethernet interfaces allow you to access memory and storage as if it were colocated in the same box, you can build huge cloud infrastructures that aren’t geographically dependent. This gives you a huge advantage when building out your resources and helps you shift workloads around to take advantage of things without the need to rearchitect your entire network when the workloads change. That’s the kind of flexibility you want in your underlay.
Bringing It All Together
Fast Ethernet that’s expensive and extensible. It’s a dream right? Now, add on the fact that you can do exciting new things with it and use it as the core of a new type of network overlay infrastructure. You’d be forgiven if you thought I was spinning you a fairy tale. But, as it turns out, this isn’t the case. Mellanox can deliver it all for you today in their Open Ethernet switches built on Spectrum technology. It’s worth a look if you have a switch refresh coming up soon. And for the low cost of entry, you may find you can build out more Mellanox that you thought possible.
