Modern storage is a complex combination of both hardware and software. While software-only storage options do exist, if you’re after high-end performance, you’ll have to deal with the nuances of hardware sooner rather than later.
And this is especially true when dealing with modern flash storage. Assumptions from the HDD era no longer apply. What works well for spinning rust is often irrelevant, if not outright harmful, when applied to modern flash.
Software Runs on Hardware
Talking directly to flash as it wants to be spoken to removes friction that only exists because of outdated abstractions. A SATA bridge between the outside world and the flash devices was useful when systems needed to speak SATA to the storage. But what if we don’t?
If we move to a higher level abstraction such as iSCSI, SMB or object protocols, why keep using an outdated bridge to nowhere? Translating from S3 to SATA only to re-translate to flash just slows things down. Why not skip the SATA bridge completely? That’s just what Pure Storage did with its DirectFlash approach. Instead of using commodity SSDs, Pure uses flash media directly.
To do this requires knowledge of what the flash hardware layer is doing, and what it is capable of. Changing the software in ignorance of the layer below the abstraction will only lead to pain. But with expert knowledge of how things really work at the hardware level, software can be made to dance. Consider the ways in which the early Apple II was designed to provide color at a time when color monitors were rare and expensive. And yet it required fewer chips to achieve!
These kinds of advances require working with both hardware and software together, rather than in isolation. Software runs on hardware, and understanding the target environment well can provide an edge to those looking for superior features or performance.
Yet as consumers of storage, we would usually prefer not to care too much about how things happen, so long as they do. We want to use abstractions, not talk directly to the flash ourselves.
The Right Amount of Abstraction
The challenge with abstraction is finding the right amount to use. Too little, and we have to wrestle with too many trivial or frustrating details from the underlying systems that were supposed to be abstracted away. Too much, and we’re unable to get our work done. Neither is appealing as a customer, and these issues often reveal themselves at the worst possible time. They’re exposed by edge cases, by things that aren’t part of the day-to-day.
Storage vendors like Pure encounter these edge cases early, often as they’re developing new product features and seeking feedback from customers. They get a variety of perspectives in order to figure out what the right amount of abstraction is in practice, not just in theory.
Yet the ‘right’ amount of abstraction is hard to prove objectively. Most of the time it’s a matter of judgement, taste, and wisdom earned from experience. While there may be some customers for whom manually tuning the compression algorithm or data layout might make sense, most are better served by letting Pure automate things.
By knowing the internals of how and when data is written to the flash hardware, Pure can adjust how the software functions on the fly. Data can be moved, or not moved, depending on the condition of the flash. Compression algorithms can be adjusted to suit the workload, automatically.
Freedom from Friction
Having both hardware and software options available opens up more ways to address design challenges like this. Is a new compression algorithm best implemented in hardware, or in software? Should it be custom hardware, or a commodity part?
Or perhaps, as with the recently announced DirectCompress accelerator card for FlashArray//XL, a combination of software and hardware works best? DirectCompress uses a fairly standard component—an FPGA card—and combines it with Pure’s software expertise to improve inline compression performance. Other choices were available, like ASICs, GPUs, or staying with CPUs, but Pure’s experience and judgement guided the decision.
As customers, we can happily stay using abstractions and concentrate on the results. Does using DirectFlash improve performance? Does an FPGA help our workloads? We have performance and functionality goals to meet, so what ultimately matters is if our storage choices work.
Pure’s expertise with both software and hardware working together means it has more options to succeed, and so do we. Read more on FlashArray//XL on the Pure Storage website.
