r/Cerebras • u/OfficialCerebras • 11h ago
Hardware What Would the Ideal LLM Inference Chip Look Like?
How would you design a chip for faster LLM inference? Follow the engineering choices that took computing from CPUs to GPUs and TPUs, then see why Cerebras built a processor the size of a wafer.
In this Hardware Deep Dive, Alycia Cary from Cerebras DevX builds AI chip architecture from first principles: connectivity, compute, and memory. Learn how parallelism and systolic arrays make GPUs and TPUs effective for training and large-batch inference, and why generating a response one token at a time creates a different challenge.
For a large model serving one response at a time, fetching weights from off-chip memory can leave compute waiting. Cerebras keeps weights in distributed on-chip SRAM beside the compute, while activations flow between layers. We explain how keeping the wafer whole makes this possible and how the design tackles a bottleneck in token generation.
0:00 What are the fundamentals of a communication system?
0:34 Semiconductor fundamentals
1:49 SRAM vs DRAM
2:48 What do CPUs struggle with?
3:34 What made GPUs uniquely positioned for graphics?
5:16 Training vs inference workloads
6:35 TPU’s systolic arrays
8:35 Why was wafer scale hard to build?
9:47 How did Cerebras solve the yield problem?
11:28 Reducing data movement