A real-world problem that defines “big data”: How do you move massive volumes of data across long distances?
Just dump 300TB of data onto a USB drive and ship it. Sounds like a great idea, but a few problems arose.
1) They don’t really make 300Tb USB drives. One provider has some 20TB USB arrays. We could have done Thunderbolt or USB3 with add-on cards, but we’d still be limited by the fact that…
2) It takes a while to load data off a server’s disks onto some other sort of disk. I usually base my rough math on 20-40MByte/sec to pull data off a disk array. Add this to the time it takes to unload it on the other end, and you realize that “we can send it by Fedex overnight” isn’t exactly the speedy proposition it sounds like.
3) There were some policies that would have complicated the ship-overnight part. But those were the simplest to deal with, and not technological, so not really in my scope.
Read the whole story by Robert Novak: In pursuit of the other kind of Big Data(tm)
- Liveblog: Intel Data-Centric Innovation Day 2019 - April 2, 2019
- Intel’s Crossover Point: A 3D Spork? - March 29, 2019
- Nyansa Wants To Be the IoT Sheriff - March 27, 2019
- Inside-Out vs. Outside-In Hybrid Cloud - January 14, 2019
- Thin Provisioning: The Lies We Tell Our Systems - December 21, 2018
- Why Would Microsoft Buy Mellanox? - December 19, 2018
- Pure Storage Isn’t About All-Flash Anymore (and Never Really Was) - December 13, 2018
- Liqid Takes Composable Infrastructure to a New Level - December 5, 2018
- The fault, dear Brutus, lies not in our machines… - November 26, 2018
- ProtonMail Was Probably Not Hacked - November 16, 2018