Datacenter Design in the Apps Anywhere Era

April 30, 2014, Ethernet Technology Summit, Santa Clara, CA—Tom Edsall from Cisco talked about the ongoing changes in design requirements for datacenters. The changes are due to new architectures and use cases for data access.

Most datacenters are build to address north-south traffic, from access, to aggregator, to core. The challenge is that much of the traffic in the latest datacenters is east-west, from server to server as many users access many apps and share data, or in a big cluster that is running a job that a single server cannot handle. An example of the big apps would be a Hadoop cluster that needs as many servers as possible.

The datacenter design is changing from a fat tree to a leaf-spine where the emphasis is on server interconnectivity. This topology change helps to increase scale and reduce latency for the jobs in the network, but really only apply to big datacenters. The transition to the newer topologies depends upon the ability to uproot a network and rebuild it in place.

The change to a flatter, regular network calls for testing features in the network as a 2-tier net from the top-down fat tree with many levels. 3 tiers are also possible for containers and other physical constraints. Interconnections across the levels is likely to be in 40G links and, farther out, should move out to 100G links. One consideration is the interconnect costs. Copper is less expensive than optical on a dollars per Gb basis.

A network can be load balanced if the network is multi-path, and a system will see a 10 percent improvement in throughput with 40GbE versus 10GbE. 40GbE can be the link for access leading to 100GbE for connections to the core. The network is more easily managed with an overlay technique rather than a spanning tree, which a enables multiple paths compared to the single path of a spanning tree.

Routing should also be multi-path, but consider that debug is much harder in a multi-path environment, due to a lack of good tools. Putting other connected functions in the network need to be located in places other than the spine router, since these routes are subject to hot spots. It is better to put the routers in the edge for easier traffic management.

Virtualization is becoming the watchword for many systems, and surveys indicate that 70 percent of workloads are virtualized. At the same time, only 20 percent of the servers are virtualized, because some apps like Hadoop need more direct hardware resources and will exist over time.

The datacenter is a combination of hardware, apps and data. The network doesn’t address the apps, but just the hardware, enabling app performance. Therefore, flow completion time is a good proxy for total system performance, and the measure is if all the apps run well. This apps focus forces changes to other areas that must scale in a robust manner while delivering performance.

The tools for operations help to run, debug, grow, shrink, and otherwise manage the network. The primary tools, based on a recent survey, are ping and trace route. The network function is to deliver packets in a timely fashion, but most administrators have no tools for measuring latency or actual throughput. The network must have good security to complement the performance. The design of a scalable, robust network requires that the designer take a balanced approach to development and compromise performance, cost, and reliability for a best overall match to the company’s data needs.