Parallel Replication for High Performance Interconnects
August 22, 2013, Hot Interconnects Symposium, San Jose, CA—Xiaoyi Lu from Ohio State presented concepts on parallel replication to improve Hadoop distributed file systems. Their work looks at the constraints built into the file system.
High performance computing apps are moving to datacenters with over 400k cores. Data is transferred using message passing through the multiple Xeon processors in the system. Understanding the interactions of the hardware and software requires a software and programming model as well as an understanding of the storage interfaces.
These heterogeneous systems have many communications planes with many performance tradeoffs. Understanding the data movement patterns helps to avoid the data paths from the MIC to the I/O systems, which also loads down the network. The issue is that the communications have high costs in bandwidth, network utilization, and latency. Minimizing these communications costs requires changes in the communications concepts to classify transfers into must communicate and may communicate.
The alternatives and changes in the architectures to accomplish this reduced communications overhead are: direct, peer-to-peer, and intra-node reduction into base slab and slot. A hybrid control system that separates data and instructions in a staggered control loop allows for better overall performance.
For testing purposes, we used the Hadoop Distributed File System (HDFS). It is a popular choice for Big Data applications due to its reliability and fault-tolerance. HDFS provides fault-tolerance and availability guarantee by replicating each data block to multiple DataNodes.
The current implementation of HDFS in Apache Hadoop performs replication in a pipelined fashion resulting in higher replication times. Such large replication times adversely impact the performance of real-time, latency-sensitive applications.
Changing to a parallel data replication scheme improved performance in all test cases. The alterations to the Hadoop flow requires writing to and from different objects, depending upon whether the design is socket based or RDMA based. The client side now has a single Connection object, so only the first response is needed to proceed, greatly reducing the communications overhead.
The basic results for the parallel replication technique in both socket-based and RDMA-based design of HDFS show significant improvements for high-performance interconnects and protocols like IPoIB, 10 GigE, and RDMA. Parallel replication
can improve HDFS write throughput by up to 12 percent and write latency by up to 16 percent . An HBase Put operation performance is also improved by 17 percent by parallel replication for these high-performance interconnects and protocols. Whereas, for 1GigE, pipelined replication performs better than parallel replication for HDFS write. So parallel replication can benefit latency-sensitive Big Data applications over high performance interconnects and protocols.
