“One of the key elements of being able to use NVSwitch is we had to keep all of the servers and the compute GPUs very close together so that we could mount them into a single rack,” Shar Narasimhan, director of product marketing, AI and data center GPUs, at Nvidia, explained. “That allowed us to use copper cabling for NVLink and that, in turn, allows us to not only lower the cost, it also lets us use far less power than fiber optics.”
However, to do so, the rack had to be heavily reinforced to handle the extra weight, and the NVLink spine running vertically down the rack had to hold up to 5,000 copper cables. Nvidia also designed quick disconnect and quick release for the plumbing and cables. Power capacity was upgraded to up to 120KW and 1,400 amps, which Narasimhan said is more than double that of current rack designs.
Nvidia also announced that its Spectrum-X ethernet networking platform and the new ConnectX-8 SuperNIC will support OCP’s Switch Abstraction Interface (SAI) and Software for Open Networking in the Cloud (SONiC) standards. The ConnectX-8 SuperNICs for OCP 3.0 will be available next year.
