When Data Pools Become Data Lakes
April 7, 2015, Data Storage Innovation Conference, Santa Clara, CA—Lance Broel from DDN Storage talked about managing and storing massive quantities of data. This new type of storage capability requires a change from blocks and files to object storage.
One growing challenge is to manage the increasing volumes of newly generated data. Much of these new data sets are unstructured; video, images, e-documents, and sensor data, and drive a greater reliance on file storage pools and increased storage islands. As the volumes scale up, the storage becomes a greater challenge with increasing complexity, performance issues, and higher total costs of ownership.
The alternative is to change to an object-based storage system. In theory, this type of system is an infinitely scalable storage pool that can handle trillions of objects in an exascale capacity. The object stores have predictable performance with linear scalability that is unaffected by the number or size of the underlying objects.
Objects are well suited for cloud-scale functions, like data protection. A system can be configured to replicate smaller objects, erasure coding for reads and writes to many locations, and fast data rebuild. The storage system has multi-site capacities, which eases disaster recovery.
Cloud deployments can be as hybrid public or private, although the public variety can be very costly. One consideration is to determine when to move from hybrid public to private clouds, there is a crossover for costs versus Amazon S3 capacity. Using a cloud configuration eases storage migration. Many suppliers have tools and file interfaces to object stores via some gateway. The most popular interface is Amazon S3, but it is possible to get other interfaces.
The basic architecture is a thin layer at the local store and a large object store. Some of the Open Stack configurations support objects through Swift. A path to implementation is to develop, test, and launch in the cloud on S3, then migrate to a private cloud without needing any code changes if you keep the S3 interface. This path allows current systems to grow up to near the limit of the layer 3 cache, while increasing data durability and mobility.
In addition, this migration path supports the legacy and emerging cloud interfaces and can be integrated into existing environments. By converting to an object store, you can reduce complexity and storage administration. Be aware that an object store increases latency, so high sensitivity apps should stay on the local networks.
