Next Generation Data Center Networks

November 2012 – In an invited keynote at the Server Design Summit, industry icon Andy Bechtolsheim, now of Arista Networks, gave a presentation on the direction and challenges involved with next generation data center networks. The performance improvements for CPU has gone up 64x the pas 12 years, while network bandwidth and performance has only increased 10x during the same period. The issue on networks, is unlike devices which are guided by Moore’s law, I/O and data do not scale systematically and due to pinout limitations, they are constrained by the number of speed of the SERDES that are in a design.

ASICs have been used for networking products historically, due to thier fast top down design flows and time to market. The drawback, is ASIC do not support high speed optimized clock rates. Moving the network processing to a single custom SOC, requires a bottoms up design approach, a longer development cycle and results in the ability to create a higher speed optimized clock rate. The advantage of the single chip custom design, is it restores the throughput to the scaling advantages of Moore’s law (e.g. at the 40nm node).

A typical 10K-100K server data center needs about 10Gbps of bandwidth per server. These are generally accomplished with a non-blocking connection set. For higher performance, a leaf/spine cluster architecture is used. This configuration is currently using a 48 port interface moving to a 96 port interface for the 10G migration to 40G. There are a number of fabrics that can be used to address this migration.

Arista MLAG Network Configuratin

Two of the main methods for these fabrics are : Multi-Chassis Link Aggregation (MLAG) enables the network administrator to use all interconnects in an active/active, Layer-2 topology and Equal-Cost Multi-Path (ECMP) which is a forwarding mechanism for routing packets along multiple paths of equal cost with the goal to achieve almost equally distributed link load sharing that typically uses Border Gateway Protocol (BGP). These are the best alternatives to the option of TRILL – also a Layer-2 topology and Infiniband which suffers from lack of direct ethernet and TCP/IP protocol support.

These methods still need congestion management which handles the flow control, buffer size, traffic balance and routing protocol. Once again there are options based on scaling – CLOS which is multistage network switching fabric, and VOQ (Virtual Output Queueing) . CLOS has been used traditionally due to its small buffer size requirements, which make it fir ASIC based solutions. The challenge is the non-linear scalability of the fabric. The VOQ architecture eliminates head-of-line blocking with a dedicated queue on the ingress side for each output port and each priority level. As a result, network traffic flows through the switch smoothly and without encountering additional congestion points. The VOQ system requires large buffers, and hence custom SoCs for implementation. It however, can provide high speed throughput of any protocol (TCP, UDP, etc) at rates of a non-blocking 10 Terabits-per-second fabric. This fabric supports 384 wirespeed L2/L3 10GbE ports capable of forwarding 5.7 BPPS with a latency of less than four microseconds.