Distributed storage is one of the current mega-trends in enterprise, and so is cloud storage. What happens when you put them together? WekaIO launched their Matrix storage solution this week, combining all the latest concepts in storage to create a massively scalable storage solution.
Local Scale and Performance: Distributed Storage
One of the greatest challenges in storage is the compromise between data sharing and system scalability. Most applications need to have data visible across many servers, with consistency, security, and locking. But companies also want to be able to grow these storage solutions with the needs of their applications. Conventional storage arrays do a good job of sharing but face many limitations when it comes to growth. Indeed, very few enterprise storage solutions can grow past a handful of nodes!
Even if they can scale, most storage arrays run into serious performance roadblocks. Essentially, it takes so much computation and bandwidth to ensure consistency between nodes that systems just can’t scale forever. Solutions like Isilon have addressed this with a very fast RDMA back-end (masking all that work), by eschewing high-performance applications in favor of archiving, or by localizing data and clients.
But some applications need both performance and scale. The traditional solution for these applications is specialized client software that distributes data. This is the sweet spot for SAN filesystems and distributed storage solutions like EMC ScaleIO. And this is the approach taken by WekaIO Matrix.
WekaIO’s Matrix software relies on local SSDs inside the applications servers, delivering high performance and low latency for frequently-used data. Data is localized with applications, yet protected by distributing copies across the cluster. In fact, WekaIO’s “N+4” arrangement looks to beat triple-parity RAID in terms of data availability without incurring the capacity and write overhead of mirroring or the compute penalty of erasure coding. It remains to be seen how exactly WekaIO accomplishes this, but it is the heart of their solution.
Sending Data to the Cloud
WekaIO Matrix is designed to run in an LXC container, allowing nodes to expose data both locally and across the network. And WekaIO can also run in the cloud, providing this same level of scalability in the cloud. The product also includes cloud tiering support, enabling cold data to be moved to S3-compatible object storage.
This means that WekaIO is both a local distributed storage solution and a cloud gateway, suggesting many intriguing deployment scenarios. Here are a few favorites:
- An HPC application could work locally with current data, then move it to an on- or off-site object store
- A company could leverage Amazon AWS scale with high-performance SSD for HPC applications
Although this sounds like the basis of a hyperconverged solution, WekaIO is concentrating on high-performance computing applications right now. This is a ripe target for them, since most current HPC distributed storage solutions include a very complex back-end infrastructure with InfiniBand or Fibre Channel switches and storage and dedicated storage and metadata servers. And by being “in-box”, WekaIO will likely beat all these in terms of performance.
Stephen’s Stance
WekaIO Matrix is an impressive new entrant in the world of high-performance distributed storage, and the cloud connector is a nice complement. The company brings all the latest trends together (distributed storage, NVMe, flash, cloud) in a useful solution. And by targeting HPC, WekaIO has selected a ripe target. We look for great things from this team in the future!