Solid State Storage Performance Testing
April 4,2011, Storage Network World, Santa Clara, Ca—Eason Ho from Calypso Systems described the emerging SNIA performance test specifications for solid state storage devices. The emerging market is over due for some standardization as there is no common methodology, terminology, nor test environment for SSDs.
Because of the lack of standards, it is difficult to compare test results from different sources, and there are many system attributes that affect the performance numbers. The number of applications, the O/S, the hardware, the nature of the software, and the amount of time in use all affect performance. In addition, MLC-based drives perform differently than SLC drives.
Figure 1: The performance drops over time as the drive fills up. ( FOB is fresh out of the box or purged)
The MLC drives drop faster and further than their SLC counterparts and both trail HDDs over time in IOPS in a 4 k random write saturation test. Despite their advantages of no moving parts, lower power, and faster access, SSDs have some challenges. They have no direct read or write functions and cannot perform a direct overwrite to a location. All flash-based storage is page-based for read and write operations and use a block erase before write. These functions must also exist in an environment where wear leveling is required to manage long-term reliability. As a result, the internal data flow management and control is a key part of the drive. Some of the items that affect SSD performance are write history, access patterns, PC active range, demand intensity, test active range, data patterns, and throttling.
As a result of the numerous interacting variables, the SNIA has developed some common starting points for enterprise-level SSDs. They start by preconditioning drives to clear all pre-existing data and data patterns. Then they start testing until they achieve some steady- state performance, at which point they start recording values. And finally they’ve developed standard reporting formats to enable easy comparisons of devices. These tests are agnostic to the test platforms, although there is a reference platform specified, and use generic test tools for the testing.
The basic tests include some write saturation tests, IOPS, throughput, and latency. Areas not covered by the spec at this time include application workload tests, matching to user workloads, energy efficiency, certification, and device interns availability and data integrity. The energy efficiency portion is part of a different group within SNIA. The basic workflow starts with the purge function, sets operating conditions, preconditions the drive to a known state, run test sequences until steady state is reached, collect data, and create a report.
The purge step starts with a security erase to sanitize the drive which is then formatted. Next, you set the user selectable test parameters such as active range data patterns and demand intensity. Preconditioning is with a series of workload dependent and independent task loops which are run and tell a steady state is reached or until the prescribed maximum number of loops have executed. Steady-state measurement window is defined as variability of a measured parameter stays within 20% of the average for the variable being tracked and trending is within 10% of the average value.
These criteria are less stringent than defining steady-state as an average ± half the window, because the measurement window is allowed to move to encompasses much of the data as possible. The test suite provides information about SSD performance in an environment that is reproducible, and covers the areas that are most critical to system-level operations. Some of the details of these tests may change by the time this specification is finally ratified and released.
Figure 2: The test suite includes tests that will stress the drives while providing some semblance of normal work loads.
Future parts of the standard will include some power measurements, cross stimulus recovery times, response time histograms, and other features that are justified as missing in the current standard. Some form of overall figure of merit might be possible as well as incorporation of other industry specifications and requirements.
Follow on work to this initial enterprise class specification include a similar specification for client-side solid-state storage with constraints to better match the client usage cases. The client specification will address different active footprints and spans, segmentation, and other areas that differ from the other enterprise. They also hope to develop trace-based workloads for finer granularity reporting details in this class of storage device.
