PCIe and Interfaces to Storage
September 16, 2013, Storage Developer Conference, Santa Clara, CA—Ron Emerick from Oracle spoke on PCIe and its interface to storage architectures. The new devices and specifications have large potential implications on the storage industry and datacenter infrastructure.
PCI Express is changing the capabilities of storage and is expected to become a greater consideration in new datacenters. The interface connects the processor to I/O and is a standard interface for HBA, NIC, and now storage. The latest specs are increasing I/O speeds, while the replacement of parallel interfaces with serial ones is almost complete.
PCIe is a serial point-to-point interconnect with 4-poins per lane. It uses a packet-based protocol for information transfer, and is scalable by adding lanes. The interface uses the existing PCI constructs to preserve the software base, and can be configured for any level of interface from chips to boards.
The important transactions are memory read or write, and I/O read or write. These transactions transfer data to or from a mapped location, and to support signaling and general-purpose messaging. Generation 3 of the PCIe spec can achieve 8 GT/s per lane and uses 128/130 bit encoding and scrambling. The next generation is in work groups and will double the transfer rate. It is expected to be released late next year.
This high bandwidth in a x16 configuration exceeds the available throughput of 40 Gb Ethernet and its close relatives. The infrastructure is in place with switches, bridges, and signal conditioners readily available.
The latest I/O interfaces include fibre channel as the storage area network standard with data rates of 8 Gb, 16 Gb, and 32 Gb as well as FCoE. Ethernet is a workhorse in the datacenter with data rates of 10, 40, and 100 Gbps and provides a network-based solution for SANs. Infiniband QDR,FDR, and EDR support high-speed process to processor links, and supports wide and fast data channels.
In storage, SAS offers 6 and 12 Gb data rates and is available in HDDS and SSDs. The solid state drives are diverging with some in SAS or SATA and many now available in PCIe cards. The availability of other formats and form factors for solid-state storage clouds the picture even more. The main issue is that the storage and network devices are bandwidth limited by the on-card controllers. NVMe PCIe changes this limit by offering a 64 Gb/s channel in a x8 configuration.
A new backplane connector is helping by supporting SATA Express, SAS. SATA, and PCIe. Single port SATA is defined in the SATA 3.1 revision. Two port SATA Express is waiting for ratification. Dual port SAS supports 6 Gb/s and the new 12 Gb/s drives. Multi-link SAS is for the new 4-lane SAS drives, and the PCIe port can be up to 4 lanes of PCIe.
In the next three years, the datacenter will have more PCIe interfaces, since the PCIe 3.0 root complexes are integrated into the CPUs. The latest processors support multicast and tunneling and can have multiple gen 3 x 16 ports. Networking is moving to dual-port optical 40 GbE and the 100 GbE switch backbones will be coming out in ’16. Gbe will still handle the lion’s share of the traffic in both copper and optical modes.
Storage access will be SAS 3.0 HBAs in 8 and 16 port configurations. 16/32 Gb FC HBAs will have pluggable optics for single and dual port operation, and converged network adapters will operate at 16/32 Gb FC and 40 Gb FCoE. Storage will have a lot of PCIe cards and trays of Flash DIMMS. 2.5- and 3.5-inch 10k rpm SAS drives with 2-4 TB capacity and similar SATA 2.0 drives from 1-8 TB will be the bulk storage.
Ten years out, the datacenter will be very different. CPUs will be dedicated to specific activities: I/O CPU will contain all I/O interfaces, memory CPU contains DRAM and flash controllers, high-speed socket to socket interfaces will operate at 100GT/s and the graphics CPUs will be external to the chassis and connected via a 100 GT/s interface. I/O and storage interfaces will be mostly 100/400 Gb Ethernet and all storage controller will be attached via Ethernet or infiniband. Backside storage will remain SAS and fibre channel.
