Optical Interconnects
September 24, 2012, IEEE Photonics Conference, Burlingame, CA—A. V. Krishnamoorthy from Oracle defined the need to move to optical interconnects in high-end systems. The drivers of area and energy efficiencies will force changes in computer systems in the near future.
The trends and projections for compute systems calls for a continuation of the efforts that have resulted in a 10,000 times improvement in the past 20 years. The drivers were Moore’s law for silicon and the improvements in chip design. As designers change from custom to utility designs, the prices drop and ubiquity increases. Solutions and aggregation provide better controls of compute, and the utility and availability have followed. One clear example is web search. Now we have gotten to a social web with e-commerce, and various functions as a service on the web.
The changes in use models have made changes in pricing models. Scale is driving efficiency in power, transport, etc, compared to custom systems designs. The computers are driving interconnect scaling with their constant need for more bandwidth and lower latencies. The NIC needs to go to 100Gb/s and eventually to 1Tb/second. At speeds of 10 Gb/s and above, you need to go to optical interconnects to have transport match the computer performance.
In addition to the bandwidth, optical provides greater reach. Electrical and optical will co-exist in the slower paths, but at anything above 100Gb/s electrical interconnects cannot cover any distance over a cm. As a result, people will replace wire with fiber, over time, the costs of fiber and optical components will drop.
For compute, the CPUs will not go much faster, but will get higher parallelism for overall throughput. The cost is in higher power, area, and complexity. The increasingly complex pipelines need more area for speculative issues which now dominate the pipeline. In addition, the complex CPUs need larger caches and multithreading. Longer term, the CPUs will get faster through architectural changes so there will be fewer CPUs in parallel, which enables simpler code and increased code reuse.
As a result, concurrency requires on-chip bandwidth for internal communications and more off-chip bandwidth to handle the higher speed peripherals. Eventually, these features will evolve to some type of crossbar for internal and external communications. The I/O scale gap—the number of I/Os needed at 1Tb/s—will have to double in the next two years and we will need 20Tb soon after that. The problem is that the number of pins in a package is fairly static, and presents area and power issues.
The increase in parallelism causes memory latency to be a problem, and the overall low availability of memory defines a requirement for faster serdes between the CPUs and memories. Unfortunately, the physics says that the skin effect gets worse, and the dielectric performance is highly degraded at higher frequencies. Energy per bit increases with the data rate, and the need for compensation increases making the near-far crosstalk management use much more power.
One way to address the area and power issues is to build up stacks of devices with TSV or package on package technologies. These are patch technologies that are hampered by the availability of known good die and act as a barrier to the change from copper to fiber. If you measure the energy in pJ/b/m, the target for the exascale computers is much less than 1 mW/Gb/m or about 10 pJ/b/m. A VCSEL is 100 times better than electrical connections at any extended distance. The industry needs to change to get down to or below 1pJ/b/m, only possible with optical.
All of these issues call for a change in positioning in environment. Systems designers need to rethink computer architectures with respect to link density, and get to Tb/mm2, which can be accomplished with 60 VCSEL/mm2 at 37 Gb/s each. Aggregating all these communications in a chip stack will require all optical connections for the I/Os. This hybrid solution is better than attempting monolithic integration of the links and other components. This macro chip will have a dense core and memory working as a tightly coupled system.
In system case studies, message traffic is most efficient in a point-to-point configuration. Coherent shared memory in a point-to-point mode with lots of small memories with a bidirectional message passing structure to the large memories needs a switch loss of less than 1 dB. Over-provisioned (dark) waveguides are inefficient users of optical power, and adding the ECC and header overhead just compounds the problem. Instead, a simple 32 bit CRC and resent if fail is much more energy efficient than the alternatives. A 32-bit CRC uses 25fJ/b for transport.
The technology for photonics can bridge the major issues. A chip-to-chip interconnect can use a front-side bond to reduce pad capacitance by half at the optical receivers. Electrical measurements of capacitance and load detection are needed, and the issues of signal/noise, and bandwidth need solutions. If the optical systems are optimized, this fixes the signal/noise, and we have the capability to make low power receivers with low capacitance photo diodes.
In our third generation of optical receivers, the circuitry includes clock recovery and low power operation in a noisy environment, with swings of 100’s of mV on the supplies. The transmitters have better modulators which traded other parameters for low power. In the forth generation, transmit power dropped from 400fJ/b to 66fJ/b.
Signal density is improved with wavelength division multiplexing, but the signals need a mux/demux to separate out the signals. You need to reduce power in tuning and increase the thermal cycle times with a heater for stability. The q and tuning range need improvement while a mirror structure on the channel allows better channel spacing. Cascade rings can be tuned in synthetic combs.
The arrays and their performance show that low power optical links are viable, but the laser power is an issue. Ideally, a laser with over 10 percent efficiency that is tunable is needed for these systems. Optical proximity and alignment issues are close to being manageable, by using etched pits and optical balls. To make this move to optical requires scale and testing capabilities. Rework technologies need development to make the yields acceptable and reduce costs. System-level power management and localized heating and cooling to manage the over 100 W in this type system device is important.
