By Casimer DeCusatis
As discussed in my previous blog from the Majorca at MIT microphotonics workshop, it’s clear that data center architectures have become a driving force behind photonic component development (as opposed to telecommunication applications). The second day of this workshop highlighted recent developments in photonic components, protocols, and software, included several presentations from members of the Open Fabric Alliance. I’ve discussed the importance of optics in the data center and open standards in a prior blog post, so let’s skip the background material and jump right into the MIT workshop results.
RDMA Technology
Although Facebook claimed during yesterday’s presentation that remote direct memory access (RDMA) wasn’t a significant part of their current roadmaps, many workshop attendees felt that this technology would be an inevitable part of future networked storage systems. Examples of storage-heavy workloads that would benefit from RDMA included high energy physics, where a supercollider can generate up to 40 exabytes of data per day. After filtering and processing, researchers only keep about a petabyte of this data, so there’s a need to move lots of data around quickly when processing each day’s results. Future high performance computing networks might take advantage of optical links and cross-connects to segregate this type of data. It’s a common misconception that RDMA won’t work without InfiniBand protocols. In fact, as discussed years ago at OFC protocols like RoCE (pdf) can extend this approach to optical Ethernet-based links.
Software and Algorithmic Financial Trading
Software was also featured in discussions on algorithmic financial trading, another high performance computing (HPC) application using optical multi-lane InfiniBand links. Low latency is so critical in these applications that in some cases they have abandoned a Von Neumann architecture entirely, instead making trading decisions in an FPGA at 40-100 Gbit/s line rates (criticizing conventional servers, one attendee noted that anyone who talks about PCI Express and high performance in the same breath should be shot). In order to shave nanoseconds off the processing time, trading decisions can be scheduled and queued for transmission even before the data frames are finished checking their CRC; once the data has been validated, the trade can be executed immediately.
Hardware Breakthroughs in Photonic Switching
Of course, all that software has to run on something, and there was plenty of discussion about recent hardware breakthroughs in photonic switching. There seem to be many design points emerging from research labs around the world, with no clear consensus on the best way forward. Some devices use silicon photonics and quantum dot lasers in the 1.3 micron range for rack-to-rack communication within a data center. This spills over into the realm of quantum physics, where the fundamental technology to build very small devices and characterize their reliability is still under development. The use of nonstandard wavelengths in the 980 – 1100 nm range is also being explored, since this technology has certain dispersion and performance benefits.
Circuit vs Packet Switching
Circuit vs packet switching within the data center remains an active topic of debate. While Facebook claims their data center is Layer 3 only (all routed), some attendees questioned whether IP switches would even exist in five years. Technologies which might supplant them include MEMs-based switches, which were highlighted as the only optical switching technology to feature zero loss in one switching state. This enables construction of crossbar switches where light paths cross multiple switching nodes without attenuation, so efficiencies commonly associated with 3D packaging can be realized in a 2D architecture (or at least a hybrid approach with minimal 3D packaging at the switch cross points). Regular readers of my blog may recall that the world’s largest silicon photonics switch was demonstrated at OFC 2015. This presentation described a 9 mm square chip has been developed with a 50 x 50 optical crossbar switch, using cantilever MEMs and 400 nm silica wires. In order to take advantage of the optical quality flatness of the waveguide fabrication technique (in which the top and bottom of a waveguide are much smoother than the sidewalls, and therefore have lower loss), a 3D cantilever MEM is used. The resulting device reported sub-microsecond switch times, consuming only 10 pJ per switch event, with attenuation as low as 0.5 dB per port. Reliability testing is also promising, including over 40 billion switch cycles and a 48 hour switch hold test. Examples of similar optical switches from research labs at HP, NTT, and IBM were cited, as well as present-day commercial offerings from newcomer Polaris Networks. Although it’s possible to scale such devices to thousands of ports, current applications seem to favor around 100-200 port counts.
Holistic Design
Workshop participants agreed that there was a need to design hardware and software together, to avoid a vicious circle of development in which functions are first implemented in hardware, then ported to software to make them more flexible, then back to specialized hardware for performance reasons. The impedance mismatch between hardware and software development (software is developed much faster than hardware, making it difficult to project future hardware requirements) was also cited as a motivation for disaggregated data centers by Facebook and others. While the push towards concurrent hardware and software development has been referred to as “co-design”, some workshop attendees felt that a better term would be “holistic design” (taking all the application requirements into account early in the solution design phase).
Photonics control software and packaging hardware
Photonics control software and packaging hardware have certainly continued to advance since the original 2010 Majoca meeting. A few weeks ago, at the end of July, It was announced that the National Photonics Initiative has awarded $220 M to develop an integrated photonics manufacturing foundry involving upstate New York in partnership with MIT and with the CIAN consortium. This effort is expected to make significant strides towards the development of optical components and switches, and the Majorca prediction for three-dimensional on-chip packaging is likely to come true. I’ve highlighted the importance of 3D optical packing in previous blogs, and OFC has discussed the topic in many recent panels. Perhaps this research will contribute to unlocking bandwidth bottlenecks at the chip and board level for OCP and other initiatives, which should be a good thing for the photonics industry.
Majorca at MIT certainly covered a lot of ground in only a few days. Let me know if any of these themes resonates with your work (@Dr_Casimer), and don’t forget to check out my third and final Majorca blog as we wrap up key findings and themes of the MIT workshop.
Posted: 15 September 2015 by
Casimer DeCusatis
| with 0 comments