Next Generation Storage Networks

April 7, 2015, Data Storage Innovation Conference, Santa Clara, CA—Dennis Martin from Demartek described the evolving storage networks. As data volumes grow, the underlying servers and storage have to grow to accommodate the larger data flows.

In general, servers are improving to address the needs for higher bandwidth and more virtualization. The larger number of virtual machines increases the number of apps in use at any time, even as systems add more servers. The data server complex may be filling up to 40 PCIe lanes with traffic. Blade servers almost all have 10GbE built in and may have as much as 1.5 TB of SSD per server.

One interesting tidbit is that the latest version of PCI express, generation 3, has the number of lanes equal to the Gb data rate. The next generation, due out in 2H 2016, will have double the data rate per lane of the current generation. This rate increase is important, since it has to fee the workhorse Ethernet pipes. 1GbE comes with 4,6, or 8 ports per server, and 10GbE which was first released in ’01 comes in dual or quad port NICs. A quad port 10GbE needs 3 x 8-lane PCIe lanes.

This bandwidth is showing up in blade servers, starting in the rack, where a dual port 10GbE takes 4 x 8-lanes of PICe gen3 or 8 x 8-lanes of gen 2. the good news is that as the number of high-speed ports increases, the costs per port are dropping and approaching $75 per 10 G Ethernet port.

In the same vein, the 40 GbE specification was ratified in June of 2010 and is comprised of 4 x 10GbE lines. In parallel, the 25 GbE is coming out soon to address the higher data speeds and to ease the further aggregation to 100 GbE. A 40 GbE needs 8 x or 16 x 8-lane gen 3 PCIe slots in the server. The 25 G PHY developments are starting and the IEEE is working on the full standards. The Ethernet Alliance has a roadmap that includes some more intermediate speeds of 2.5Gb and 5 Gb per second just to address the need for those functions that don’t need the higher speeds and costs of 10GbE. A 5 GbE line can run on CAT6a cabling, so can preserve existing infrastructure.

Fibre channel (FC) is the other main transport fabric and the latest 16G FC is back compatible with the 4 G and 8 G equipment. The net data rate for 16 G FC is 14 Gbps through a single lane connector, again using up 8-lanes of PCIe gen 3. It is possible to mix FC and Ethernet traffic within a network and the multi-protocol NICs are available.

FC gen 6 has been announced and will be multi-speed. The base is a 32 Gbps single line with a x4 configuration providing 128 Gbps rates. The gen 6 spec calls for forward error correction, energy efficiency, and back compatibility with the 16 G and 8 G lines by throttling back the data rate. The specifications are due out later next year.

The network configurations are converging the LAN and SAN functions to permit a lossless datacenter bridging function for enhanced Ethernet to support FC storage and traffic. This convergence makes the choice of transport fabric a style or preferred protocol issue and not a technology issue. Dual FC and Ethernet NICs at 10 Gbps enable FCoE traffic.

The storage components are also changing. The gen 3 SAS (Serial-Attached SCSI (Small Computer System Interface) can handle data rates up to 12 Gbps and use existing HBA (Host Bus Adapter) and RAID controls. Hele also, the costs are dropping for the cables and other infrastructure. A 12 Gbps line requires 3 x 8-lanes of PCIe gen 3 for data transfers. A new mini-SAS connector provides higher port density and offers other features for connecting storage to the rest of the network.

Another fabric, Thunderbolt, is now available with up to 20 Gbps data rates this interface is mostly directed towards media and entertainment creation functions. The interface supports NVMe devices with a limit of 6 devices on 1 connection and can act as a bridge between MAC and PC. There is a possibility that this interface may move to the datacenter through Thunderbolt-Ethernet bridges.

USB is another 10 Gbps interface. The latest version, 3.1, adds power delivery to the standard. The high speed connections will use a new Type C connector that is non-polarized. Another addition is USAB over WiFi.

NVMe is a software interface that operates on a PCIe interface for flash memory in systems. The hardware-software combination can deliver up to 2 Gbps from a single NVMe SSD. This category is poised to grow and will exceed other SSD interfaces by ’18. the native PCIe hardware in servers will be migrated to other fabrics soon, and will enable sub-10 us latencies in networks.

All of the hardware and software improvements are held back by the cabling. Optical fiber has as 15 to 20-year lifecycle, so any new installation should use OM4 fiber to allow for in-place upgrades over time. Copper is useable at 10 Gbps for short runs of up to 5 meters on standard CAT6a with RJ 45 connectors. This distance is a top of rack function. For longer runs, CAT7 can operate at 10 Gbps up to 100 meters. Unfortunately, CAT 5 is unusable above 5 Gbps. The changes in datacenter designs call for longer cables, which means that users will need more fiber or active copper links. The change to faster cables reduces latency in addition to increasing the bandwidth.