VAST Data began its business with a question; How do you design and build a new type of storage architecture?
The company set out on a journey to build a vastly different architecture that would replace shared-nothing architecture with shared everything architecture. It succeeded and, in doing so, redefined the economics of flash storage.
Its DASE (Disaggregated Shared Everything) architecture is a brand new generation of scale-out storage systems designed to overcome the problems inherent in shared-nothing architecture.
What does all of this mean? In a nutshell, DASE welds together QLC Flash Drives and Intel Optane with stateless, containerized storage services all connected over low-latency NVMe over Fabrics networks. Algorithms are applied to this architecture to improve upon traditional storage efficiency, resilience, and scale.
Before coming to Storage Field Day 20 to present its DASE system developments, VAST Data was announced as a Unicorn, a company worth over $1 billion, and one of the fastest-growing storage companies in its sector.
Disaggregation: A Starting Point
Disaggregation is one of 2020’s storage buzzwords and is the beginning step into VAST’s shared everything architecture platform. Disaggregation is the next level beyond hyper-converged and is a critical piece of this next-gen architecture, but on its own, it is not enough.
By utilizing disaggregation on its own, problems of shared-nothing architecture remain. According to VAST, in most cases where disaggregation is being used, it is a shelf with SSDs detached from the CPU. Often, that shelf contains single ported SSDs, which can become a point of failure. This adds a new problem because if one of these shelves fails, it is not one node that failed; three or four nodes fail.
Disaggregation also reduces growth flexibility with its dedicated SSDs. If a user has dedicated SSDs to each node on the shelf, they cannot grow CPUs at a different trajectory than growing SSDs, because that will cause an imbalance in the system.
So, what is the next step?
Shared Everything
Shared everything is the real key here; once the SSDs are accessible to everyone, the nodes can access all of the SSDs across the shared fabric. Those nodes now can access the same data structure, and they all need to do so in a synchronized manner. The nodes also need to be able to this without having the data structure corrupted.
This requires many process changes to go to shared everything, and while disaggregation was the simple first step, what followed took a lot longer, with much more trial and error. It required a blank sheet of paper; you cannot retrofit an existing system designed as shared-nothing into shared everything.
Once VAST built a shared everything architecture, each node could see the entirety of the network span in a direct-attached manner, even nodes on the other side of the network. The system no longer needs to use RAM, the cache, batteries, or anything that introduces complexity and risk in a shared-nothing architecture.
Intel’s Optane technology allowed VAST to eliminate all of the complexity that comes standard with shared-nothing architecture. Optane allowed VAST to break free from caches, from cache coherency and allowed VAST to keep all of this within the enclosure and the logic in the nodes.
Building DASE required a complete rethink of metadata structures because those structures are, in this architecture, within persistent media that is very fast. Those structures also need to be accessed by hundreds of thousands of nodes simultaneously while maintaining consistency across the plane.
A welcomed side effect of this shared everything architecture is that the system doesn’t need nodes to communicate anymore. The nodes do not talk to each other, all but eliminating east-west traffic within those nodes. This allows better scalability than shared-nothing architectures.
Shared nothing architectures tend to be victims to massive amounts of crosstalk as a user scales; however, with DASE, there is no crosstalk. Everyone sees everything, can read and write directly to the media and scale up to massive levels.
Hyperscaler or Hyperconverged?
DASE is a hyperscale architecture, as it can scale up to hundreds of thousands of nodes. It is hyper-converged in the sense that all of these nodes are eventually software and can run on the same servers as other applications do. It is also hyper-converged and hyperscale in the sense that it is a single network. There is no longer a separate network for storage and for compute; it is a single fabric that is very strong that allows DASE architecture to operate. However, it is not hyper-converged because it is disaggregated and does not sit in a separate physical location than the CPU.
Growth Guaranteed
VAST data has been called the fastest-growing storage company in history. After its first full year of revenue, it is still accelerating a fast as possible. With the company’s new funding announcement towards the end of 2020, there is no sign of it slowing down.
What will be next for VAST Data? Time will tell, but it looks like this independent innovator will be here to stay.
