Facebook Describes Flash Infrastructure
August 13, 2013, Flash Memory Summit, Santa Clara, CA—Jason Taylor from Facebook described the structural and architectural implications of flash in the datacenter. Although bulk store is still in HDDs, newer use modes are making flash viable in longer term storage.
Facebook has five main regional server centers and serves 84 percent of its members outside the US. All of the data centers have high data use, so the company spends about $1.24B on capital equipment including servers, storage, construction of data canters, and PCs. Their architecture is structured around network clusters with sub-sections for front-end, server, and back-end tiers.
The front-end servers have 250 racks of servers for Web service. Each server has 144TB of cache. Ad servers take up 30 racks and multi-feed servers use 9 racks. The service tier provides functions like search, messaging, and other peripheral functions. Most of their servers are generic, standard servers that are all installed in the datacenter.
One problem facing the datacenters is heat management. They a always working on efficiency due to the high air conditioning loads. A typical datacenter has a power utilization efficiency of about 1.9, meaning that almost as much energy is used for infrastructure as for computing. They have tried raising the room temperature to reduce the load, and in their Oregon facility, are using natural outside air for cooling. They draw in cool outside air and exhaust the heated air back to the outside. The Oregon facility has a PUE of 1.07.
Other techniques for reducing cooling loads include side-lining UPS and changing the rack power supplies. See more details about their datacenter work at and their results at www.opencompute.org.
Services like the news feed require some servers to be leaf aggregators of the 40 servers in a rack. Data are cached across all 40 servers in a highly optimized indexed store. Incoming requests go to the Web servers and get moved to the memory cache. From there, the requests go to the feed aggregator and the identity check and other administrative functions take place. The queries go to the database for cross-reference across friends and likes, then it goes to another cache for ad insertion before shipping out to the user.
They have restricted their servers to 5 standard types to allow for flexibility and repurposing. The basic servers are configured by function, where 200,000 Web servers have low memory and little disk. In comparison, the Hadoop database servers have lots of low-latency storage to go along with the 144GB of RAM. The storage the storage arrays are 15 x 4 SATA with the driver being cost per gigabyte. The only different servers are called type 6, that use PCIe storage for the index servers.
The index servers have three main uses. In the database functions, they are flash cache and disk alternatives, and in index servers are used as alternatives to RAM. In ’10, all of the database was in RAM and amounted to 1TB compressed before server fail over. This increased in ’11 to 2TB uncompressed and 1.5TB uncompressed. Last year, the index serves went to all flash and can now handle 2.45TB compressed. They are getting better throughput from the database by reducing the number of outliers and reducing overall latency in the PCIe subsystems.
The index servers index the content in the database in flash. the index holds a couple of weeks of stories, with the first two days in RAM and the balance in flash. after 2 weeks, the stories are moved to disk. Complementing the PCIe storage are SSDs for their low latency and high throughput.
One new effort is to implement “cold flash” in the system. These storage arrays are picked for low endurance and good power performance, for operations that are write once, read many with few updates. This WORM data structure matches most of Facebook’s content and is good for high availability and long-term storage. In comparison with cold storage, where you spin down the HDDs, the WORM structures allow for much lower latency and lower power than the HDD structures.
As a result, they are looking for dense and cheap storage, where long write cycles are ok. The low endurance and low IOPS are not a problem since many of the photos & media are actually write once, read never.
Their next plans are to keep drives for bulk storage and use PCIe where possible. Selected flash is helping to disaggregate the racks. In this environment, they can completely flush data when erasure is approved.
