By Casimer DeCusatis, Ph.D.
I just flew in from New York (no, my arms aren't tired), and made my way to the Anaheim Marriot, which will be my base of operations for OFC 2013.
Today I participated in a roadmap workshop sponsored by the Optoelectronics Industry Association (OIDA) on metrics and roadmaps for future “scale-out” data centers. My fellow workshop participants included representatives from Dell, Google, Microsoft, Facebook, U.C. San Diego, the Fraunhofer Institute, APIC Corporation, Finisar, and Ovum.
Last year, OIDA defined several metrics for data center optical networking, in collaboration with the National Science Foundation, The Center for Integrated Access Networks,and the Optical Society of America.
These include link data rate, cost per unit bandwidth, energy per bit (mW/Gbps on the rack-rack or chip-board level), optical switch port count, and optical switching speed. Current values and future targets for these metrics were published to help guide the industry; this meeting included discussions on whether the metrics are complete and correct, and whether there are enablers or inhibitors to their adoption. My talk focused mostly on the first 2 metric, data rate and cost per gigabit within the data center. If we track the growth of Ethernet data rates over time, for example, we see that they are gradually overtaking the raw data rates of other protocols. For example, 10G Ethernet is now faster than 8G Fibre Channel, and 40G Ethernet will be faster than both 32G Fibre Channel and 4X InfiniBand. Once the raw data rates reach this point, Ethernet has potential to disrupt other protocols, especially given its high volumes. This may already be happening with the introduction of link protocols such as FCoE and RoCE. Extrapolating this trend through 2020, we can foresee a time when even the server backplane PCIe connections will be disrupted by 100 – 400G Ethernet.
Virtualization is an interesting impact on these metrics. The number of virtual machines hosted by a dual socket x86 server has steadily increased for the past decade, and network virtualization may be the next big thing for our industry. Virtualizing the network requires treating many switches as if they were one large switch, which is somewhat different from the way we virtualize servers by dividing one large physical server into many smaller entities. Despite this, I feel we can take a lesson from server virtualization; virtualizing the network will create more demand for networking resources, not less, in the long run. This is because people will deploy more bandwidth-hungry applications once they have achieved the lower costs associated with a virtualized network. At the same time, data centers continue to grow larger, according to recent IDC data. This seems to be supporting the notion that link cost per gigabit will continue to fall with higher volumes as an important cost driver.
On the other hand, Google has published some data showing that cost per bit does not naturally decrease with network size. This is because the complexity of pairwise interactions and any-to-any control mechanisms require more advanced control mechanisms. The current lack of determinism in distributed Ethernet management requires over-provisioning, and does not enable scheduling, automation, or optimization of data center fabrics. So perhaps the cost per bit targets for 2020 aren’t achievable after all! Note that these factors could potentially be overcome with a form of software-defined network, as Google has used in their WAN. So it’s not clear which view will prevail, and the impact on data center metrics for the next five years remains a work in progress.
The OIDA meeting was followed by a reception, after which I headed back to the hotel to write up this blog. Time to turn in & get ready for a full day at the conference tomorrow. You can follow my daily blog entries from OFC right here, or drop me a line on Twitter (@Dr_Casimer).
Disclaimer: Opinions, interpretations, conclusions, and recommendations are those of the author and are not necessarily endorsed by IBM.
Casimer DeCusatis, Ph.D. Distinguished Engineer IBM System Networking, CTO Strategic Alliances Member, IBM Academy of Technology IBM Corporation.
Posted: 17 March 2013 by
Casimer DeCusatis, Ph.D.
| with 0 comments