The first product of the OmniConnect series under Credo Technology Group aims to enhance memory bandwidth and density to optimize the computational efficiency of accelerators and XPUs in AI inference workloads.
Don Barnetson, Senior Product Vice President of Credo, told EE Times at the press conference that the company’s Weaver is a memory expansion chip designed to overcome memory bottlenecks in AI inference workloads. As the demand for scalability, bandwidth, and efficiency in data centers continues to grow, these bottleneck issues are becoming increasingly prominent.
“AI reasoning poses a series of new challenges to the industry.” He pointed out that “the amount of memory and throughput have increasingly become limiting factors for AI inference workloads, rather than computing power. The challenge faced by inference workloads is that they are different from training workloads and have extremely high memory requirements. Ideally, no matter how large the model is, it is hoped that it can run completely around a computing engine.”
Barnetson stated that LPDRR5 and GDDR memory are limited by bandwidth, density, and power consumption, while high bandwidth memory (HBM) is often too expensive and difficult to obtain for many applications. It is these factors that have driven the development of Weaver.
The technological path to breaking through memory limitations
This memory expansion chip combines advanced 112G ultra short distance (VSR) SerDes technology with Credo’s proprietary design, increasing I/O density by up to 10 times. Barnetson stated that the company adopts an unconventional architecture and utilizes LPDDR5X memory to achieve up to 6.4TB of memory capacity and 16TB/s of bandwidth.
Barnetson used Sora 2 as an example to illustrate the memory limitations faced by current inference workloads, pointing out that Weaver is designed to cope with the memory pressure generated when uploading the entire codebase to the engine. “Sora 2 can only generate video clips of 15 to 20 seconds, as it will run out of memory afterwards.”
He stated that in the past, the solution to this problem was to place LPDDR memory on a large substrate around the inference IC. This method is limited by how much memory can be accommodated on the substrate. It can only configure up to 16 memory stacks, with a maximum capacity of 256GB of memory and 1.3TB of bandwidth. The ultimate result is that all interfaces of the chip are occupied by memory, “Barnetson said.
He added that “Weaver is Credo’s low-cost solution, which moves memory away from the substrate by using SerDes instead of the atypical wide parallel interface to transfer data.
Simplify CXL protocol to achieve low latency
Barnetson introduced that Credo has developed a SerDes with a transmission distance of up to 250 millimeters, with an interface density of about 2 terabits per millimeter and extremely low power consumption of about 1 picojoule per bit.
He stated that the next step is to add a data link layer, which utilizes the CXL protocol. “We have streamlined CXL and placed it on top of SerDes, which has much lower power consumption. We have retained the essence of CXL.” Weaver retains the lower cost CXL FLIT structure, but abandons other features such as speed jumps. Barnetson stated that Credo aims to create an efficient AXI over SerDes mapping while maintaining flexibility – any operation can be done locally through OmniConnect.
“We designed it this way to achieve extremely short latency, with a round-trip delay of about 40 nanoseconds.”
He said that any operation that can be completed through DDR controllers and ASICs can now be achieved through Weaver chips. This expansion chip allows SerDes channels to be connected to chips with memory interfaces. “It implements physical scaling, allowing you to deploy memory at the server level rather than just encapsulating it internally.”
Barnetson stated that this method allows the memory to be placed up to 10 inches away from the chip. “It can increase memory density by about 30 times.”
He added that one of the goals of Weaver and even the entire OmniConnect series is to make it future oriented, as it is difficult to predict the demand for AI in a few years. “Just place a large number of OmniConnect interfaces externally, and three years later when manufacturing the circuit board, you can decide how much memory and how many networks to configure. ”
Integrate optical interconnect technology
Credo recently acquired Hyperlume, a company that develops microLED optical interconnect technology for chip to chip communication. The company’s technology utilizes ultra fast microLEDs and ultra-low power circuits to overcome the inherent energy consumption and bandwidth bottlenecks of traditional electronic interconnects.
Barnetson stated that Credo will integrate Hyperlume’s microLED technology into its product portfolio, further expanding the expansion options for AI networks. “We will support microLED near port optical chips, so some channels can be selected for microLED communication, or standard vertical and horizontal expansion communication. “