The digital world used to be a keyless black box for most of the world. Data cracked it open, ushering in a kind of transparency that broke precedent. Suddenly opportunities, previously dismissed as ludicrous, were on the horizon.
Modern businesses are obsessed with data because they see opportunity knocking. As AI algorithms penetrate aspects of society, the possibilities have quadrupled.
Today data can tell you the location of a place down to the precise coordinate. It can inform you about events that haven’t happened yet. It can alert you about fraudulent activities happening on an online account while you are away. And the expectations grow every day.
But data also costs money. The banal tools of yesterday are redundant for threshing intelligence from data. It takes expensive third-party services to convert raw information into insights.
Mezmo, a fast-growing Silicon Valley startup, is enabling enterprises to collect and analyze their data within their own walls. At the recent AppDev Field Day event in California, CEO, Tucker Callaway, demoed the solution to the audience.
The Mezmo observability platform centralizes telemetry data from other observability platforms, inspect and enrich it, and accelerate response to intelligence.
The Struggles of Harnessing Telemetry Data
Telemetry data refers to a varied corpus consisting of performance data, metrics and operational data. “In the context of Mezmo, it is your typical MELT metrics – metrics, events, logs and traces – any data that is created by machine,” says Callaway.
IDC released a report that stated that data volume is growing at a rate of 23% year-over-year. But by contrast, observability adoption is down at 10%.
“That necessities a shift in the way people think about observability and cybersecurity. That ultimately plays a role in how we think about telemetry data fueling an LLM.”
A plethora of challenges stand in the way of harnessing telemetry data. There is unprecedented amount of diversity in data format with data coming in from numerous sources and silos. Poor data management translates to a low signal-to-noise ratio which frequently leads to cost blowouts and alert fatigue.
Efforts are being made to put this pool of data to good use, but a rift in communication between teams is making it harder for enterprises.
“People who are controlling the data, who generate a lot of this data, oftentimes, are application developers and the people who are trying to manage the end are platform engineers and SREs. They lack the ability to communicate between themselves, and we find them stepping on each other quite a bit,” Callaway reveals.
Many a times, the skill that affords a high level of data inspectability is missing. “There’s this notion to enrich, transform or reduce data that’s predicated on the assumption that you understand your data. We found that the biggest bottleneck in enterprise isn’t a lack of desire to optimize. The ability to understand data is just not there.”
With data showing no sign of stopping, enterprises are faced with a difficult choice: to turn data over to third-party vendors for analysis and lose control and ownership, or accept the risk that comes with poor visibility.
Callaway argues, “Organizations need to own their own telemetry data. They no longer need to advocate that ownership to the vendors they work with. You can’t just keep it in the vendor-format and have them collect and manage it for you, and store it in their cloud. You need to take control of your data so you can do the things with it that are the most useful to you and your company.”
Mezmo predicts that a shift is underway where cost of telemetry data will grow linearly in the next five years.
Faced with this scenario, teams benefit from an approach that converges telemetry data streams from various sources into one broad pipeline. This pipeline then aggregates and normalizes the data before passing it on to different personas – operations and security – within the organization, in required format.
Observability Maturity with Mezmo
“We take the control point and route the data to the right location. We can manage, normalize, enrich, transform, route and do all the things with the data that matters to your enterprise and that you’re beholden to these vendors,” Callaway says.
Mezmo achieves this by periodically profiling data and detecting repetitive and redundant patterns in it, pointing to possibilities of optimization and insights.
An open platform, Mezmo integrated with a large number of observability solutions at both ends. It ingests telemetry data from platforms like Kafka, Prometheus, Kubernetes and OpenTelemetry. Data is transformed and enriched within the Mezmo Telemetry Pipeline before being routed to various output points like Datadog, Amazon S3, Elasticsearch, Splunk, and Grafana.
Mezmo ensures that insights are distributed without delay, so that teams can make decisions and respond to alerts faster.
One of the key jobs of the Mezmo platform is to increase control of telemetry data and enhance security through analytics and insights. In the process, it reduces cost and overheads of data management, significantly reducing resolution time.
Where optimization decisions happen in the early stages making post-event changes tricky, Mezmo lets operators make responsive changes dynamically after the fact based on the context.
These may sound like what OpenTelemetry does, but Bill Meyer, principal solutions engineer, dispels the confusion, “When it comes to operationalizing the collection of telemetry and operating on it before sending it out to telemetry tools, Enterprise Mezmo helps pick up where OpenTelemetry is currently situated.”
After 2 years in the works, the Mezmo Telemetry Pipeline is at its best today. There are some new capabilities on the horizon that Callaway hopes will drive the solution further into the AppDev world.
One of them is in-stream analysis. Ordinarily, data is brought to a central repository where it is computed. Mezmo’s in-stream analysis will perform correlation and cross-stream analysis on data in motion. In-stream alerts will soon be a function on the platform as well. This will identify variations in data while on the go, based on metrics thresholds, and alert about events that didn’t take place.
Simulation is a capability that will facilitate real-time data operations.
“One of the things really hard about setting up a pipeline is you’re interrupting mission critical data flows.”
Mezmo will have a Simulation mode for creating tests, samples and alerts. This will allow engineers to test the data stream all the way from source, through the processors to the output point in advance before streaming live operational data. This helps them decide the best change management strategy and put controls in place without interrupting the data flow.
Mezmo can be deployed in edge environments where it can be managed from the SaaS control plane with data staying local.
Wrapping Up
There is a lot of value in mature observability practices, but extrapolating from data is not simple. For companies that struggle to derive insights from their telemetry data, Mezmo is a great tool to have. It homogenizes disparate streams of data, detect unseen patterns in them, and spits out contextualized insights, enhancing visualization, and reducing time to resolution, all within the enterprise perimeters.
Watch Mezmo’s presentations from the AppDev Field Day event for more details, and to see the solution in action, be sure to check out the demo in the following session. Also check out Sam Holschuh and Paul Nashawaty’s takes on Mezmo.
