Mythbusting Flash Performance
August 10, 2011, Flash Memory Summit, Santa Clara, CA—Bill Nesheim from the hardware group at Oracle made comments on the use of flash in enterprise applications. Not everything is faster with flash, and enterprises may have other overriding requirements.
In the enterprise, flash can optimize the performance of databases and storage platforms. Last year, companies spent about $3 billion on flash products of which about half were SSDs. Requirements for these platforms are transactional performance, report performance, uptime and serviceability, reliability, data integrity, and visibility. All of these requirements also lead to higher capacities and easier deployment within the enterprise.
NAND is used in L2 read cache and other file system functions, which all performed between 10 and 20 times faster with flash than without. These flash applications are in various forms factors including SSD, PCIe, and flash arrays. OLTP and databases run faster with flash mainly due to reduced wait times for the data accesses shifting from 50 percent to less than 5 percent of the operating time. In a database, a flash cache offers a clean storage for read acceleration, which reduces buffer read I/O.
It takes a lot of work to organize the workload and is not just a drop in change. Most metrics and benchmarks are not relevant to actual applications, so users must tune the hardware, firmware, and software applications. For example, the Exadata database machine is set up for either volume or speed and the choice changes the number of flash cards in the system. The high-speed applications are configured with 100 TB of storage while the volume machines have 336 TB capacity. In either case, the system has 5.3 TB of PCIe flash. The result is an engineered system with 10 to 20 times greater performance compared to one without the flash subsystems.
One of the myths that this application shows, is that flash is always high-performance in a small footprint. The reality is that there is a large performance variation range of two orders of magnitude between various flash configurations. The high densities possible create power and thermal challenges, or the flash subsystems can exceed the power available per hour per slot and are a challenge for the super caps and batteries.
These subsystems blur the line between applications and storage, and the reliability requirements for unpowered data retention constrain architecture decisions. Enterprise storage is not all about IOPS. The real consideration is equal response time per drive, and the variability between drives slows the whole storage subsystem to the lowest common denominator. In systems with limited application threading, the reduced IOPS demand must be traded off for quicker turnaround time.
Storage systems need predictable operations and functions which scale over capacity and time. Ideally, you want balanced read and write times. When comparing predictability and scalability versus time, you always want the configuration that gives easiest upgrade. In most applications, the I/O rate is relatively constant over time, so the most important consideration is low variability for data write operations. The average speeds are not good enough, because the average is not the same as worst-case which is what actually determines system performance.
Slow I/O leads to increased queue depth which results in 20-50 times increases in latency. The latency distribution causes differences in performance that is determined by the slowest 1 percent of all I/O. A two and a half times change in IOPS is due to the outliers in the distribution. Enterprise storage requirements are for constant 24 x 7 uptime for more than three years or a MTBF of greater than 2 million hours. Unfortunately, the firmware for the storage system is affected by storage failures and MLC devices have a much greater cost per endurance than SLC.
Storage systems need to have embedded service policies. For provisioning, the system needs hooks to policies. Some standards exist but mostly there are no standards for these components and applications. Every vendor has unique protocols and management interfaces. Lack of compliance to standards slows down the drives, for example, using 512 byte sector emulation versus a 4K actual right block. This technology is still immature, suffering from high variability and low predictability. With standards and lots of engineering, they can be made to be very effective in the enterprise.
