Blue Gene Q Design
IBM Blue Gene Q
August 24, 2011, Hot Interconnects 19, Santa Clara, CA—Philip Heidelberger from IBM described the interconnection and message unit of the latest IBM supercomputer, Blue Gene Q. This machine is currently a R&D project, but will become the third generation of massively parallel computers for large scale scientific and analytic applications.
The design objectives included the ability to have some commercial applications and reduce the total cost of ownership. Operating costs should be reduced by using power efficient chips, packaging, and floor-space efficiency. The performance will be exponentially greater than the previous generation on a flop/watt basis. Reliability is enhanced with ECC throughout the system.
The basic chip has 18 Power ISA 64-bit processors configured as 16 operating, one spare, and one I/O and OS processor. These are all connected through a crossbar to level 2 cache. In addition, there are four double precision floating-point units. L1 cache acts as a pre-fetch interface to this with structures for data load balancing.
The crossbar runs at half of the core frequency or 800 MHz, and has three basic switch functions; request, response, and invalidate. There are 22 master ports and 18 slave ports to provide on-chip bandwidth of 538 GB per second. The L2 cache is 32 MB per processor and a 16 way set associative. This is the point of coherency with multiple versions—transactional and speculative, with software detection of conflicts and recommit. The DDR3 memory interfaces connects to memory chips with extensive ECC permitting single correct and three error detect.
The internal network is actually four separate networks on the chip. The networking is set up in a 5-D torus configuration with all the networks integrated into one torus. Going off chip, these connections use shared physical links plus one extra I/O link. This configuration permits scaling with the size of the system. It reduces latency while optimizing bandwidth and flexible partitions to reduce data jitter and also the number of cables required between cards and racks.
The internal network integrates the function of network and NIC message units: NIC, NDNA, FIFO interface, and the torus acts as an on-chip router with integrated point to point, collective, and I/O connections. Over 85 percent of the network management is clock-gated to minimize power. Signaling is both electrical and optical, with the optical signals going from the plane to plane and account for 20 percent of all the links.
The network is designed as a virtual cut through with 512 byte payloads per packet. The structure results in 90 percent payload bandwidth efficiency. Communications with other processors is through row broadcast with virtual output queuing enabled by multiple FIFO’s. The dynamic routing is zone programmable to move the queues out of memory. Collectives for math operations permit broadcast of operations to neighbor nodes for parallel acceleration.
The network is designed for reliability, all buffers and data paths are ECC protected.. Packet level error detection and hardened latches contribute to reliability. ECC is based on Reed Solomon codes and can detect any five 10-bit symbol errors. There is a modified 8/10 bit code on the optical links to provide single error detection.
The message unit controls DMA transfers through memory, and FIFO put and get through parallel engines and multiple master ports extend this the data of the whole computer. It also supplies memory translations and increases parallelism.
The overall hardware concept was too simpify hardware design, and remove I/O traffic from the core. The resulting system is expected to exhibit ping-pong latency of 40 ns per hop. Collective adds 12 ns per hop, resulting in a 3 µs worst-case transit time.
