Cooling 2 KW in a 1U System

November 27, 2012, Server Design Summit, Santa Clara, CA—Phil Hughes from Clustered Systems described the capabilities for cooling very high power density systems. The driver for changes in cooling technologies is the relative inefficiencies of air cooling.

As data centers add more computers and GPUs to the mix of servers, the power per slot is going way above the stated maximums for a 1U slot. A GPU adds about 300 to a blade server, and many have more than one GPU card. The thermal densities are approaching critical area safety limits for the semiconductors.

Recently, most systems have used air for cooling. The volume of air required is fairly large, and the cooling for the air amounts to about 10 percent of all the energy for the data center. The limiting factor is the thermal transfer efficiency of air. The big bipolar CPUs in the mainframes needed to have liquid cooling to maintain operating temperatures, and it seems likely that the industry will have to go back to some type of liquid cooling to manage the thermal transfers.

Coolants have to be non-reactive with the electronics, non-conductive, non-inflammable, zero or extremely low toxicity, easy to cleanup if there is a leak, and for easier total systems design, isothermal. Needless to say, the coolant must be environmentally friendly.

The candidates for cooling fluids include oils, water, and refrigerants like R134A. Tests have demonstrated that using oils can reduce the energy for cooling to 2 percent of the total energy, water to 0.5 percent, and refrigerants to less than 0.18 percent. Although energy costs are important, the other characteristics of the coolant are also significant.

Existing liquid coolant systems are designed for water to the rack, and not to the servers. The resulting systems are fairly complex, with lots of plumbing and connectors that are susceptible to leaks and lower reliability. For the oil-based systems, the motherboards are immersed in the cooling oil, making the maintenance and hot-swap fairly problematic. Although the coolants are relatively inert, leaks and spills from transferring a motherboard to another location can cause slippery floors.

Data center operators need to understand the differences across the various solutions and learn which fluids are acceptable in the data center. A different cooling system uses cold plates and refrigerants to cool the electronics. The thermal capacity of the cooing system is over 200 KW per rack and the R134A refrigerant evaporates if a leak occurs. The cold plates replace the normal heat sinks and transfer the heat to the coolant. The only modifications needed are to install the extra connectors for the coolant input and output lines. These lines are low pressure and use commonly available fittings to connect to the rest of the refrigeration system.