Broadcom data center switching silicon smashes 51 Terabits per second

Broadcom says it has doubled the capacity of its merchant switch silicon with the launch of the 51.2Tbps Tomahawk5 ASIC this week. Broadcom's Tomahawk line targets hyperscale and cloud customers and its latest ASIC promises to deliver a substantial increase in port density while reducing power consumption compared to the previous generation chips. Tomahawk5 also brings the company up to speed with rival Nvidia, which debuted its own 51.2Tbps Ethernet switch at GTC this spring.

Using the ASIC, the company expects OEMs, to which Tomahawk5 is already sampling, will begin shipping switches with 64, 800Gbps ports or 128, 400Gbps, or 256 200Gbps ports by early next year.

"Every two years we've been doubling the bandwidth of the silicon," Pete Del Vecchio, Product Line Manager, told The Register. "Since 2010, we now have an 8x increase as far as the bandwidth, and every generation we also improved the energy efficiency. So now, we're about 95 percent more efficient than we were 12 years ago."

However, it should be noted that like the 25.6Tbps Tomahawk4 it replaces, Broadcom's latest chip is still stuck on a 100Gbps PAM4 Serializer/Deserializers (SerDes). This effectively caps the per-port throughput at 800Gbps when using QSFP-DD optical plugables. So, while Broadcom has managed to double the capacity of its switch silicon, it hasn't actually doubled the port speeds. Not that customers are clamouring for more speed just yet.

According to Broadcom execs, adoption of 400Gbps modules is still in its infancy, even among performance-hungry hyperscalers, making up just 15 percent of the total Ethernet market by revenue. By 2026, Broadcom expects 400G ports to exceed 50 percent of Ethernet revenues.

These figures roughly align with the findings of a recent Dell'Oro Group report, though analysts put market penetration for 400Gbps switching at closer to 10 percent. In the first quarter of 2021, the group reported that shipments of 400Gbps switches had exceeded 800,000 ports, with that number is expected to continue to ramp over the next year, propelled by strong demand from cloud providers.

However, the Tomahawk5 claims more than twice the switching capacity. Improvements to the chip's architecture, including a move to a 5nm manufacturing process enables the monolithic switch chip to achieve significant power savings.

According to Broadcom, the chip consumes less than a watt per 100Gb of capacity. This means that at full tilt, the chip uses less than 500W, which Broadcom says will allow OEMs to continue air cooling switches powered by the chip.

"Doubling the bandwidth actually gives you a 6x reduction as far as cost, power, and complexity," Del Vecchio said, comparing the chip against the previous-gen Tomahawk4.

The ASIC also features a number of features tailored toward hyperscape HPC workloads including network virtualization and segmentation, single-pass VxLAN routing and bridging, as well as "Cognitive Routing" capabilities designed to reduce job completion times by avoiding congestion.

"The traffic characteristics that you have for AI do tend to be different from what you would have for standard compute and storage traffic," Del Vecchio explained, adding that by routing around potential sources of congestion, the switch can reduce latency and improve performance for AI and Machine Learning workloads.

Put together, Broadcom says Tomahawk5 is the highest performance single switch ASIC available on the market, but that claim isn't without caveats. At GTC this spring, Nvidia showed off its 51.2Tbps Spectrum-4 Ethernet switch.

Spectrum-4 is based on technology obtained with the acquisition of Mellanox in 2019, and the switch boasts many of the same capabilities offered by Broadcom's Tomahawk5, including support for up to 64 800Gbps ports on a single chassis. The switch as well as Nvidia's BlueField-3 DPUs are slated to begin shipping later this year.

So, while Broadcom can claim it has beaten Nvidia to market with a 51.2Tbps ASIC, it is technically sampling, those chips still need to be integrated into OEM chassis before they ever make their debut in datacenter networks. This casts some doubt as to which will arrive in customers' hands first, especially given the current state of the supply chain.