Software Defined Storage
September 16, 2013, Storage Developer Conference, Santa Clara, CA—Laz Vekiarides created a working definition for software defined storage and outlined the capabilities and challenges for implementation. Many systems are moving to decoupled software and hardware, and the hardware is moving towards commodity devices.
Starting with the economics, datacenter owners look at the hardware and software costs as inclusive buckets, but should instead consider fine-grained analysis. The hardware includes not only the servers and disks, but also the reliability and qualification costs. Software development, test, and support are part of the software costs. Integration, system-level validation, test, etc. are necessary functions that are usually absorbed into cost of operations. On top of all of these costs comes support for the various devices and systems.
Moving to commodity hardware is an easy consideration if the hardware costs are the only issue, but the other functions cannot be ignored. The move to a software-defined storage system requires hardware for data protection and include the disks, flash, RAID controllers, and some compute entity for object and erasure coding. The data management and services are on top of the hardware and perform functions like controlling volumes, snapshots, security, deduplication, and the apps and operating system integration.
A working definition for software-defined storage is a storage solution comprised of software, commodity hardware (i.e. rack servers, DAS), and standards-based non-commodity hardware that offers storage service not tied to the physical platform SDS is running. The system depends upon virtualization and migration amongst physical platforms and is generally tied to compute virtualization.
In a cloud archive, the growth of data sets requirements for storage, organization, and archive for long periods. The pressure to reduce operating expenses like floor space, cooling, etc. and the unstructured nature of much of the data makes the system optimized for cost and ingest rate and most storage is write only. This is not a good use case for SDS.
For use cses that are amenable to SDS, hosted virtual apps in small and mid sized companies moving to a public cloud storage. The companies are willing to use cloud-hosted SAN’s for applications. Most of the effort is for secondary storage, e.g. replication, and performance requirements are moderate. This market is driven by hosters – implemented as a VM on commodity servers. It is possible to achieve multi-tenant architectures via network virtualization.
A private cloud app is proposed as a replacement in large datacenters for SAN. This is a high density compute environment with over 200 TB of server attached storage and on-demand creation. The performance and availability requirements are rigorous and require deep orchestration and management framework.
In converged workloads at small businesses, companies are looking for simplified entry to virtualization. The footprint constraints for small and remote offices calls for simple integrated appliances with a focus on ease of use for the IT generalists. The system looks like an integrated high availability or cluster with a virtualization stack with the ability to scale out. The storage stack is integrated with the hypervisor. The challenge is that some of these companies have proprietary hardware that can be hard to differentiate from conventional storage arrays.
For systems with unstructured data such as video processing, CAD, logs, and other machine generated content, the performance requirement high enough to rule out a cloud archive solution. Data are too voluminous to store cost effectively in premium solutions because scale out is required. These systems are generally based on open source solutions ZFS, object storage.
The big performance challenges make virtual appliances suspect due to the heavy compute requirements for data services such as RAID, erasure coding, encryption, compression, and deduplication. In these cases, the hypervisor economics can make good old fashioned arrays attractive.
The issues of reliability and availability have to address many challenges. The economics of mirroring versus RAID, shared versus mirrored media, and challenges for the interconnect and network. Standards-based hardware raises other questions: is it virtualized and can I move a virtual machine’s storage? Is the interface interoperable?
The software defined storage complements the move to virtualization, as storage workloads co-mingle with app workloads. Still many challenges face the transition, since the software-defined network is not coincident with the SDS movements. Areas like VM prioritization, I/O path optimization, virtual low latency interconnects, and network configuration and management all need more discussion.
