The pursuit of scalability in enterprise storage has lent to decades of buzzwords and marketing pitches, all without producing the claimed scalability. Myriad, Quantum’s “cloud-native” all-flash storage platform is about to change that.
Myriad represents a significant leap forward, one that adds innovation and predictability to a scale-out storage platform deployable both on premises and in the cloud. Purpose-built for demanding workloads that require transactionally consistent operations, like AI, Myriad is a noteworthy product in the realm of file storage.
When “Cloud-Native” Really Means Scale Anywhere
Over the years, numerous marketing jargons like “web-scale” and “cloud-scale” have popped up to describe the idea of unlimited scalability. Dive into the technology though, and you will find the usual central point of failure, or single bottleneck that limits scale, or other chokepoints that establish a ceiling.
”Cloud-native” is the new addition in the word salad. Quantum Myriad grounds it by establishing a dual meaning. The platform can scale, anywhere you want, and as much as you need.
Myriad’s platform architecture is based on a shockingly open design – shocking because the components of the architecture are fast becoming industry-standard models for scale-out systems.
Kubernetes is deployed on commodity x86 servers for a number of application containers. SONiC-based load balancers distribute and direct the traffic. Compute and storage nodes of “un”-proprietary hardware constitute the platform, with the ‘secret sauce’ being purpose-built application containers distributed amongst cluster members. High-speed networking with 100GbE+ RDMA-based ethernet ties it all together. The openness of the inner workings is quite refreshing.
Given the industry-standard nature of each platform and its software component, these parts are available with most major cloud providers. Their adoption means that Quantum will be able to deploy Myriad in public cloud providers with very little changes, if at all, to the core architecture.
As the world moves towards hybrid IT platforms, having options of where to deploy a given solution is critical to improving platform adoption. In this situation, a solution that works without requiring operational management changes in both the public cloud and on-premises platforms is quite refreshing.
POSIX-Compliant
Historically, object and block storage have been at odds because of varying philosophies built into the platforms. Object storage has been the standard for scale-out storage, while block is usually focused on scale-up and monolithic application stacks.
Where using object storage for the more monolithic workloads has historically been challenging is with transactional consistency. Tasks like IoT data ingestion are no-brainers. Workloads like writing a single large stream of data, a live video stream for example, into a single file, are less of a challenge. The file access could be blocked for other readers while the write is streaming in, but the impact is rather concentrated to just that file.
Unfortunately, not all workloads involve a single file. Consider this. A large database could contain four multi-terabyte data files and one transaction log file. The log file can be thought of as an ongoing ledger that contains, in order, the constant stream of data change. Make a change to a database table, and the change is written to the log file, then distributed to the four data files, all as a single transaction.
If one of the data files has its write operation held up, and then the objects are accessed by a different thread or even replicated to a different site, the files are now out of sync. Failure to keep them back in sync can, and usually will, result in database or object corruption. This can bring a database down extremely fast, ultimately causing a major business disruption.
POSIX compliance is a standard that ensures compliance with attributes such as hierarchical directory structures, atomic file-blocking, and durability for writes. Most notably, the support of a transaction where all storage commands in the group can be written as a set, is critical for this sort of workload.
Providing a transactionally consistent experience for object storage opens opportunities for moving more workloads to this platform which historically it would have been incompatible to.
Distributed CPU Handling is the New Storage Bottleneck
Now that you can scale wider than traditional storage platforms, where is the next bottleneck? In the past, the bottlenecks first started at the storage itself. Spinning media could not keep up. SSDs shifted that bottleneck to the controller. In response, controllers were beefed up with more CPU and memory, which shoved the bottleneck back into the SSD controller and then the interconnects to the storage. NVMe storage relieved the individual drive controller pressure, while RDMA storage improved the interconnect landscape, and 100GbE+ networking shoved the bottleneck back to the controllers.
Now, the bottlenecks to scalability and performance lie in the CPUs of the controllers. Quantum brings innovation in two-fold – by scaling out the controllers via Kubernetes with the load balancing and storage node architecture, and with a custom NVMe protocol.
The physical NIC handles communications, but the load balancers direct traffic, and the storage nodes’ CPUs perform the data movement. Add awareness into the client of the scale-out architecture, and the client’s custom NVMe driver can better direct where to send the data to.
All of these scale and proximity-aware features produce a distributed storage platform that is capable of extremely high-speed storage operations with exceptionally low latencies. And for what it’s worth, low latency operations are great for databases, video and audio systems, real-time analytics, machine learning, and HPC workloads.
Future Roadmap
Quantum is continuing to expand and add functionalities to the Myriad platform to help it appeal to more workloads and markets. Currently, it only supports NFS storage presentation, but SMB and S3 are on the roadmap. Public cloud deployment options too are coming soon.
Conclusion
Quantum Myriad fills a significant void in the object storage space, one that makes it much more appealing to businesses that are generating and retaining exponentially more data every year. For more information, head to Quantum’s website or head over to their Tech Field Day Showcase page.
