Remember when iTunes was first released? It was supposed to be a simple easy to use jukebox that allowed you to sync with your iPod, if you were lucky enough to own one in 2001. It was only available for the Mac, it was simple, and it was fast and responsive. Over time iTunes, gained broader platform support (Windows) and other features such as a store and support for other media formats such as videos and ebooks. Over time what was once a simple, snappy, and reliable application became bloated buggy and sluggish. The quick and speedy roadster was forced to grow into a semi truck so it could carry what Apple thought everybody wanted.
Putting flames on your giant, bulky truck won’t fool anyone into thinking its fast and agile
The Race to the Bottom
We have seen a similar trend in analytics solutions that collect data from applications and infrastructure and promise to provide valuable insights for different purposes such as operations, security and business analytics. As these platform grow the types of data the can ingest and the resulting analytics provide value, often at the cost of performance and usability. Queries that used to take seconds can take minutes, and the learning curve to actually extract value out of the platform is extremely high.
I recently met with Steve Newman, Founder and CEO of Scalyr and learned what makes his product different than other analytics products. Steve is a lifelong programmer and many know him for his most recent company Writely which was acquired by Google and became Google Docs. Steve founded Scalyr just over 7 years ago and the product has become a log monitoring and analysis solution known for extremely fast performance.
A Focus on Speed
Many things about about the way Scalyr is architected lend themselves to an overall faster experience meant to help engineers quickly get the answers and data they need out of their logs. Overall, Newman told me, the product is not that much different than their biggest competitors in terms of capabilities, but they are much faster. In some cases they have seen queries that would take as long as 30 minutes on competitive platforms complete in just 2 seconds.
There are a few key differentiators to Scalyr’s architecture that make this possible. First, they do not use a database, but rather a columnar datastore which enables the product to receive logs in realtime. Logs can be ingested via multiple methods such as syslog, an agent or by API. The platform itself is a distributed scale-out system. Data is distributed across all nodes, so as the data scales, so does the backend infrastructure that is ingesting and processing the data.
Scalyr is also a SaaS product, which fundamentally does not necessarily make the platform faster. But being in control of the entire infrastructure means they are able ensure that all measures are taken to ensure the desired level of performance. For example, all data is currently stored on SSDs only. While there is intention to offer a service tier that will store older data in S3 in the future, primary data needs to be stored in fast storage to deliver the performance the platform is built upon.
Of course quick extraction of data from the platform depends on more than a speedy architecture. A grandiose, multipurpose analysis platform with a powerful UI that has a steep learning curve will slow down customers considerably no matter how fast the underlying architecture is. Thankfully this is not a problem for Scalyr. A lot of work was put into the interface so that its users would not necessarily need to understand its query language. When I saw it in person I was impressed with how quickly queries could be modified and results delivered.
Ken’s Conclusion
The careful, measured approach Scalyr has taken to the design of their platform is refreshing. Too often we see products that started with great promise become an unmanageable mess. Yet 7 years after its inception Scalyr is providing a specific product for a specific audience. Engineers send the platform their logs and it quickly gives them the data they need to understand and troubleshoot their systems.
It’s not necessary for every product to be all things to all people. In fact it is often counter productive. Rather than keeping up with the competition, why not identify ways in which you are different than your competition and focus on those? I am encouraged when I see companies like Scalyr follow this strategy.
