Characterizing and Testing Storage Performance

September 18, 2013, Storage Developer Conference, Santa Clara, CA—Peter Murray from SwiftTest suggested that IOPs is not a good metric for characterizing and testing storage performance. The problem is that this measurement doesn’t acknowledge the contributions of the other systems components to storage performance.

Storage performance validation has evolved from blind reliance on vendor claims, to testing in production and hoping for the best, to validating with freeware tools like IOMeter or IOZone, to testing with something resembling a full workload model. The validation requires a validation appliance, workload emulation, and a storage system that closely resembles the production environment.

The tools enable new feature and technology evaluations, virtualization performance impact assessment, production evaluations and vendor bake-offs, and storage configuration optimization. Using the tools properly will allow the measurements of infrastructure change impact and will aid in troubleshooting storage system problems. The primary requirements for the testing are to emulate application workloads and test the hardware on the target system.

One use for the validation appliance is to compare new products and upgrades. The workload of files, blocks, or objects is input to the appliance, which accesses the candidate solutions and production systems through a switch. This configuration allows for a-b testing and correlation to the production environment.

IOPS has been used as a metric for storage and represents the I/O operations per second. The raw data is normalized to IOPS/W and MB per second /W, but the read and write loads are not specified. Comparisons are likely to be invalid if the I/O sizes and load mix are different. Iometer is one open source tool that measures IOPS and compares overall system performance.

The problem is that IOPS depend upon block/chunk size with a KB taking one request and one response. A 4 KB block also takes a single request, but requires three responses. Larger block sizes require more responses and lower IOPS. The tools and standard definitions do not identify a standard size for the tests.

In addition, real storage does not use a fixed block/chunk size. The sizes are protocol and application dependent and are growing in the newer protocol versions and storage hardware. Read and write operations may be only a small fraction of the storage traffic and over 90 percent of the traffic is related to the metadata. Also, RDMA write/read operations may erode the isolated W/R IOPS measurements.

Metadata operations enable I/O and provide information about the data, such as location, extent, file name, access, attributes, locking, etc. The amount of metadata to perform I/O varies widely and is determined by the access method. The metadata impacts performance based on the application. High volume metadata apps include web and office apps, while lower use apps include database and email. Caching the metadata in RAM, flash, or SSDs helps, but is expensive. The problem is that metadata performance in HDDs is slow, and caching doesn’t eliminate the metadata, it just speeds it up.

Workloads are the IOPS, metadata, and access patterns for an app. Each app has a unique signature with different percentages of read versus write, random or sequential, I/O versus metadata, metadata command frequency, and the use of asynchronous or compound commands. The workload is an intrinsic factor in characterizing performance.

The workload profile helps the engineer understand the total system environment and the effects of configuration changes. It also helps to distinguish system degradation and app failures as well as the various failover modes. The workload model contains access patterns, file systems, I/O parameters, and load properties.

Access patterns include I/O mix, write-read percentages, and metadata percentages. Depending upon the software and the appliance, the reports can include detailed breakdowns down to the command distribution level. The file system reports show the hierarchy and further details like file system depth, files per folder, number of sub-folders, and other file-associated information.

The steps to create a production workload model start with production statistics from workload analyzers, Netstat, NFSstat, etc., pre-built test suites, and packet captures to create an accurate and realistic model. The analysis must include a command mix with the details of the I/O and metadata associated with each command type. The load parameters should indicate temporality, burstiness, and overload conditions.

Next, the workload re-creation has many challenges. It is difficult for vendors and operators to determine service and applications conditions at scale. Big data is database intensive, and virtual environments have many and changing variables. The server to storage traffic is a mix of OS, hypervisor, and app behaviors.

Ideally, a workload emulation would have flexibility to allow users to model a wide range of conditions from exact conversation to blended cloud apps. It would have efficiency and be self-filling, so the engineers don’t have to do many tedious studies, analyses, and test configuration synthesis runs. The emulation should be able to fingerprint and analyze the traffic in a standardized manner to decompose traffic characteristics into quantifiable workload dimensions.

The resulting simulation replicates the operations of dozens of serves, virtual machines, and storage systems in a single box. The full simulation will be very close to the production operations, and the attached switch will allow direct comparisons of the production traffic and the simulated traffic. This configuration also allows the model to run against the target hardware for further validation.

The results go into analysis to help optimize decisions. The analytics provide insight into the details of the operations and show the effects of failure modes, and changes to operations like flash, dedupe, tiering, etc. The results also perform performance analysis to show current and potential response times and throughput for the systems.

In these applications IOPS is a poor benchmark for storage analysis and evaluation. A workload modeling and load generation appliance, and high-fidelity workload modeling can provide production-like workloads and data streams for detailed system analytics. The greater the effort that is put into the emulation, the deeper the understanding and optimizations possible.