Massively Scalable File Store
April 6, 2015, Data Storage Innovation Conference, Santa Clara, CA—Philippe Nicolas from Scality talked about the needs to handle increasing data volumes in various tiers of storage. Change is needed as the number and types of files exceeds the file system capacities.
Today, we are seeing massive increases in data volume, and much of it is unstructured. To exacerbate the problem, over 80 percent of these data are inactive after 30 days. The normal file systems cannot address a billion files and TB of data, and the storage networks offer little protection and backup for these inactive files.
Networks which used to have 10,000 clients now have 100 M in the cloud. The number of nodes and data sources will continue to grow as more machine-to-machine communications takes place. The number of nodes is driving the number of files and some big systems have hundreds of billions of files.
Networks need some way to restore elastic resiliency to the file system. At the same time, any changes have to address privacy and security, and provide capabilities to meter, audit, monitor, and otherwise manage the torrent of data. File services and servers will still have to keep the legacy data in some form that is compatible with the new systems, calling for enhanced and extended services that can be distributed into a commodity file server.
A cluster file server is a shared data cluster within an OS domain. All nodes understand the physical, on-disk, file systems, but are stuck with limited scale. SAN file sharing allows users to share files only, and can support hundreds of nodes. The SAN has no OS requirements and can even work with mixed OS with many apps.
The implementation and philosophies for an extendable and flexible file system have to start with standard components and live with existing software. A storage network of the future should have capabilities for self-healing, self-managing, awareness, organizing for all data management services.
The choice for parallel or non-parallel systems will depend upon the various use models as are the options for file versus object-based storage. All will depend upon a new data model with something like a NoSQL database for the metadata. Some examples of working systems include Napster and Gnutella as peer-to-peer systems. These services used shared mutual storage in a peer-to-peer configuration.
In industry, storage networks can be symmetric and may not need a separate metadata server. The architecture will be that of an aggregate independent storage server that only shares the network. One mode will be to have a file on one machine, whie another can be to stripe the file across storage servers. In a parallel to NFS, the storage network will have separate metadata servers and storage access protocols per SAN.
One example is Lustre, an open-source asymmetric object-based storage network that is used in high performance computing. Some models are available at www.pvfs.org and there are others available. Most of these new storage models have some limits on ability to scale.
The social media and search companies had to develop systems to handle 100 million users at a time and address 10-100 PB on line in a billion files. Existing file storage is very limited in this environment, so Google developed GFS and Amazon created Dynamo to address the issues. GFS is an asymmetric service that uses thousands of chunk servers on 64 MB chunks all under one metadata server. Another attempt to address the issue is Hadoop with the HDFS. In HDFS, every node stores a piece of the data and there is local processing at the store.
The emerging store net will have various topologies. A NoSQL relational database as the metadata server is not fully scalable, which will eventually require a new SQL for object and key/value store. Shared nothing is the rule for user mode, except for public networks. The underling systems will be comprised of al commodity hardware and software and will have a high-level of automation integrated into the system.
Key value store will not have nested directories, but a flat name space. There will be a simple relationship between keys and values in a 64-128 bit universal unique ID. This configuration provides scale and flexibility but needs a blank sheet of paper as the starting point. Object stores include the metadata and data as an extension and union of keys. There are no real limits to this type of interface and it can work with a Restful-like interface like the AWS S3 or some other open stack implementation.
The SNIA has developed the CDMI to be the NFS of the Internet. It lacks capabilities for data protection and is hard to back up, because it uses RAID6, which takes too long to rebuild. Alternatives include data replication which has high costs at large scale since it requires 2-3 times the storage capacity or erasure coding which only has a 1.2-1.6 overhead.
At scale, the peer-to-peer or asymmetric systems seem to work. objects allow a unified, global name-space, but add to overall latency. Both are capable of self organizing the storage networks.
