Quantum has a new storage system – Myriad all-flash software. Designed to suit large-scale file workloads with low-latency requirements, Myriad is accessible to large numbers of clients using the same storage pool concurrently. Industries with AI and other data-intensive workload needs suited to it include life sciences, genomics, high-performance computing, and specifically, media and entertainment.
Industry Background
Large-scale shared access to file storage has been a constant challenge for organizations since the early days of NFS and SMB protocol access. Since the late 1980s, when NFSv2 was first introduced, much has changed in computing technology. New approaches to network design, access protocols, and workload organization have been developed alongside massive advances in hardware.
Hyper-scale public cloud providers in particular have driven changes in massively parallel access to large-scale data storage. A lot has been learned about how these kinds of workloads perform, and the design trade-offs that work well in practice. Originally considered secondary storage, file systems have emerged from the shadow of block-access storage to be considered another form of primary storage in its own right. Major business systems now rely on storage dedicated to file-based access without adding layers to expensive block-only devices.
Company and Tech
Quantum’s long storage pedigree has allowed it to take a sober look across time and assess which techniques remain relevant and which should be replaced with newer techniques better suited to the times.
One key element in Myriad’s design is the nature of data contention in large-scale, multi-access file storage systems. Most of the time, people who are collaborating, work independently on different things across space and time. Applications no longer assume they have exclusive access to entire datasets for their entire working period, and app developers have learned to share.
This means contention for specific chunks of data is far rarer. Two pieces of code in different parts of an organization are unlikely to try to write to the same few bytes within the same microsecond window. Even within closely related parts of an application, these kinds of collisions are more unlikely than it was even a decade ago.
The massive scale of modern storage systems means Myriad can assume collisions likely won’t happen, and in case of outliers, rare events are to be handled rarely. The storage system code can be simplified and tuned in ways that are difficult, if not impossible, when write-collisions are more common.
Subtle but important insights like this highlight Quantum’s focus for Myriad. Myriad doesn’t seek theoretical perfection at the expense of real-world performance. It is in fact designed to operate in the real world, as experienced by customers.
Product Specifics
Myriad adopts Kubernetes as its internal workload scheduler, leaning on containers and micro-services to create a loosely-coupled, high-performance and scalable design. Being a standard, Kubernetes supports use of commodity hardware and software, allowing Quantum to concentrate on specific areas where proprietary software delivers real value.
Quantum believes there is little value in proprietary hardware for the customer workloads it is targeting and can use the savings to invest where it truly matters.
Ingress to the cluster itself uses BGP-peering, a novel approach borrowed from hyper-scale systems. Instead of the more traditional IP-failover load-balancing techniques that operate at the application layer like DNS, BGP support for Equal-Cost Multipath (ECMP) allows Quantum to push the load-balancing task into the network layer using well-understood network techniques. Myriad can present a single, unified ingress point with robust load-balancing into a customer environment with less complexity.
The Myriad system itself is a combination of NVMe networking nodes and high-performance storage nodes, configured in a scale-out, hyperconverged fashion.
Conclusion
Quantum has carefully looked at AI and the workloads its target customers want to run, and how they function today with current technologies and approaches. Myriad is the result of careful balancing of competing requirements, and navigating constraints that exist in the real, commercial world, not just in the theory of sometimes outdated benchmarks. It is worth exploring for customers with large-scale NFS file storage needs, especially where low-latency is the requirement.
For those with SMB or object storage needs, it’s worth keeping an eye on Myriad as the solution evolves. Some data services—such as replication and HA failover—are not yet available, though Quantum can augment a Myriad system with other products in its portfolio to address advanced resiliency needs.
Similarly, high-security operations features such as immutable snapshots and quorum approvals for sensitive administrator operations are missing, but noted on Quantum’s roadmap.
To conclude, Myriad shows great promise for a variety of storage needs, particularly if Quantum can quickly deliver on its roadmap. To learn more, head to Quantum’s website or to the Tech Field Day Showcase page.
