You probably don’t need to be told that networks are getting faster. There’s a ton of reasons why. Cloud-based applications. Changing consumption models. 5G networks. Enhanced Wi-Fi. You name it and it’s going to be a reason why there’s more data flowing through your switches than ever before. You still need to make sure everything is on the up-and-up though, right? Letting all that data fly around without analysis is just asking for trouble.
But how do you sample data at 40Gbps? Or more? How can you possible hope to grab the important stuff out of the stream and not miss something you should have been paying attention to? The metaphor “drinking from the firehose” is not only 100% appropriate but an accurate representation of what happens when you direct that much data to an analytics box of some kind. Anyone that’s worked in networking knows that analytics tricks like SPAN ports can barely keep up. So how do you to do it in the modern networking world?
Enter CounterFlow.ai
I bumped into the CounterFlow.ai team at RSA this year and I must admit that they hooked with with a very simple elevator pitch. They can monitor 40Gbps data flows without dropping packets. I was a bit shocked. I believe my exact words were, “Prove it.”
I got that chance talking to Ashish Malik and Fritz Repich from their team. They laid out their architecture that helps them reduce the dwell time of data. Collecting all the data in the world does no good if you can act on it. Think of it like a gallon of milk. Even if you leave it in your refrigerator without opening it eventually it will spoil. You need to not only buy the milk but do something with it soon. CounterFlow.ai helps you figure out how to best utilize the collected data before it expires.
But how can the collect that much data and analyze it? Well, the first trick is using a custom FGPA in the NIC of their appliance. It helps the system read and write data much faster than a normal NIC would be able to operate. That’s a great way to ensure you’re not dropping packets. Okay, so that tells me how they’re getting the data. But how are they analyzing it high speed?
Pattern Matching
The key is that they’re looking for certain patterns in the data and firing alerts that you’ve configured for quick access. But the rest gets written to disk to be analyzed later. And where the “AI” part of CounterFlow comes into play. They have a machine learning engine built into their platform that gives them the ability to look for patterns in the data after it’s been written to disk. The hard part of trying to do real-time analytics is that you need a massive amount of CPU to make it happen. Because you have to crunch all the data to ensure you didn’t miss anything. But even writing the data to disk for a few seconds or minutes gives your ML engine time to figure out what’s going on and catch things you can’t see with high-speed filters.
You’ve done this yourself. Think about the last time you did a double-take looking at something. Or the last time you were distracted while reading and had to re-read a passage again. You need more processing time, even if just another second or two. CounterFlow.ai ensures you’re going to get that time buy analyzing for more context from the disk and not the data stream. Alerts on easily recognized patterns are the start. But as the ML engine crunches the data you can start to build better alerts for the data stream. Your analytics engine gets smarter thanks to the analysis it’s doing!
And the best part? All that data is stored on the appliance in a familiar format: PCAP. Now you can take all the data and munge it wherever you want. Want to upload it to Cloudshark and let people see what you’re puzzled by? Go for it. Want to share a particular PCAP with support for a device vendor? Export it and run with your hunch. The sky is the limit.
Bringing It All Together
I don’t often have to eat my own words, but I’m happy to say that CounterFlow.ai served them to me with some tasty sauce to boot. They’re not only doing 40Gbps packet capture but they’re applying some tasty AI and ML techniques to make it extra delicious. I’m sure they’re going to have a lot more on their plate in the coming months that will lead their customers to being full and happy.
