“Big Data” is trying to become all things to all people. Right now, as hard drive costs spiral down near the $.01 – $.03 per GB range, all of the other parts of the big data puzzle are becoming more important. The first step to success is determining the size, performance, reliability and connectivity metrics required. Some big data solutions feature high performance and scalability while having issues with reliability while others feature lower performance and cost but deliver proven reliability. There are so many new contenders that many have no HCL metrics to determine proven success BUT there are a few proven, low cost, high performance, virtually unlimited scalability and highly reliable solutions available today if you know where to look.
Some environments are able to rely on fully open source solutions, like Lustre, Swift & Ceph, and for those users we are happy to deliver hardware and reference architecture documentation from our server manufacturer partners. Lustre, arguably Isilon’s less expensive, faster more scalable and bigger brother, is going through an upgrade process to solve it’s data reliability issues as we speak, by adding ZFS as the underlying storage layer, and also allows analytics to be run directly off of your flat data. HGST (the enterprise group within WD) has a turnkey 5PB (in a box) object storage solution that has an Amazon S3 interface that can be used as your private cloud, behind your own firewall, and comes with an optional hardware NAS gateway appliance. Gluster (& now Ceph) is Red Hat’s distributed storage solution that has migrated into Red Hat Storage Services. Lastly, the similarities between Hadoop and object storage are bringing Hadoop to the front of the line of “big data” solutions. The list goes an and on so we are happy to have our engineers discuss your requirements with you and come up with the correct solution.