Fast, Simple, Scalable Storage for HPC and Big Data
September 9, 2014, Intel Developer Forum, San Francisco—The underlying tools and infrastructure software allow users to get high-performance storage to match to their compute and data needs. The Lustre file system allows for both enterprise and cloud configurations to support the storage requirements needed in the datacenter.
The cloud version is optimized for AWS (EC2) and both versions can support and ad on functions to link to Hadoop, use a REST API, and use the many management and monitoring services available from Intel and others.
The system configuration functions start from management target and metadata storage that get accessed by the corresponding servers. The management network talks to the object storage servers which direct traffic to the target storage devices. The goal is to use multiple storage servers to stripe the data to many drives for greater reliability and faster data access. So far, the maximum file size is 32PB due to the striping and the filesystem can handle 10M files.
The special Intel tools allow for simple configuration and switch settings for the entire storage array. Creating a new file system and populating it is a drag and drop operation, or can be done with pull-down menus. The fine system is similar to NFS and includes a scheduler function.
The tools also include multiple views and charts for management functions and implementing policies can be on a directory or file basis. The inherent flexibility in the tool allows for viewing operations, RAM, CPU use, and can show the difference between single and multiple stripes in the storage systems.
The metadata can use multiple targets and a shared directory allows for multiple metadata servers and targets. It is possible to re-stripe the data on the fly, but layout locks can affect the operations. LUSTRE has few fixed policies. Overall robustness depends upon the hardware and other system devices, so a object storage target can lose data if a part fails and the data are non-replicated.
If the target device loses its metadata server, the system can lose its file system. Therefore, it is a good idea to take snapshots to preclude major system failures. The development teams are working on converting the asynchronous replication to synchronous to help reduce the possibility of compromising critical data. It is always possible to trade storage complexity for performance at larger scales.
Some parameters that were measured are storage efficiency is 80-09 percent of the total device at transfer rates of over 1MB per second. In some cases, the file system is more efficient than RAID 6 and is optimized for HPC and big data.
