LLM System | Future Architecture and System Technology for Scalable Computing

LLM System

The deployment of Large Language Models (LLMs) poses significant challenges on cost and design complexity, as training and serving LLMs demand vast computational throughput and enormous memory capacity. Our group conducted research on identifying performance bottlenecks for LLM workloads, integrating heterogeneous hardware components including CPUs, GPUs, FPGAs, and CXL devices into design, targeting on cost-efficient and high-performance LLM systems.

As Large Language Models continue to grow -- now encompassing billions of parameters -- scaling them with GPU-only clusters becomes both costly and power-hungry. Yet in most deployments, CPUs, main memory, and network interfaces remain under-leveraged. By matching different stages of the LLM workload with the hardware best suited to its computational characteristic, we extract far greater efficiency. For instance, Intel’s introduction of Advanced Matrix Extension (AMX) units unlock high-throughput matrix operations directly on the CPU, enabling selective offload of smaller computation kernels from the GPU. Meanwhile, to tackle the huge memory requirements of LLMs, CXL memory expanders are leveraged to increase capacity and bandwidth. In concert, these heterogeneous building blocks yield an LLM framework that not only meets escalating compute and capacity demands but does so with significantly lower cost and power footprints.

CPU-GPU Cooperative LLM Inference

The Architecture of Advanced Matrix Extension (AMX) Unit in CPU.

With the introduction of Intel AMX units starting from 4th Gen Intel® Xeon® Scalable processors, the matrix operations can be completed efficiently on multi-core CPUs. We first provide a comprehensive characterization of AMX, comparing it with Advanced Vector Extension (AVX) units on CPU, NVIDIA P100, V100, A100, and H100 GPU (Kim et al., 2024 ), providing valuable insights on the integration of AMX units in LLM system design.

Building on our characterization results, we propose a cost-effective LLM inference system that leverages a single CPU and a single GPU. By pinpointing data transfer between the CPU and GPU as the primary bottleneck in existing SoTA setups, we aim to offload more computation onto the CPU’s AMX units, thereby minimizing PCIe traffic. A performance model, incorporating GPU and AMX compute throughputs, PCIe bandwidth, and memory bandwidth, determines the optimal offloading strategy. This AMX-aware framework extends Intel® Extension for PyTorch (IPEX) to enable seamless, cooperative CPU–GPU inference (Kim et al., 2025 ).

References

2025

ISCA

LIA: A single-GPU LLM inference acceleration with cooperative AMX-enabled CPU-GPU computation and CXL offloading

Hyungyo Kim, Nachuan Wang, Qirong Xia, Jinghan Huang, Amir Yazdanbakhsh, and Nam Sung Kim

IEEE/ACM International Symposium on Computer Architecture (ISCA) , Jun 2025

Architecture Accelerator System

2024

CAL

Exploiting Intel® Advanced Matrix Extensions (AMX) for Large Language Model Inference

Hyungyo Kim, Gaohan Ye, Nachuan Wang, Amir Yazdanbakhsh, and Nam Sung Kim

IEEE Computer Architecture Letters , Jun 2024

Architecture Accelerator System ML/AI

Abs DOI

The ever-increasing number of parameters in Large Language Models (LLMs) demands many expensive GPUs for both inference and training. This is because even such a high-end GPU such as NVIDIA A100 can store only a subset of parameters due to its limited memory capacity. To reduce the number of required GPUs, especially for inference, we may exploit the large memory capacity of (host) CPU to store not only all the model parameters but also intermediate outputs which also require a substantial memory capacity. However, this necessitates frequent data transfers between CPU and GPU over the slow PCIe interface, creating a bottleneck that hinders the accomplishment of both low latency and high throughput in inference. To address such a challenge, we first propose CPU-GPU cooperative computing that exploits the Advanced Matrix Extensions (AMX) capability of the latest Intel CPU, codenamed Sapphire Rapids (SPR). Second, we propose an adaptive model partitioning policy that determines the layers of a given LLM to be run on CPU and GPU, respectively, based on their memory capacity requirement and arithmetic intensity. As CPU executes the layers with large memory capacity but low arithmetic intensity, the amount of data transferred through the PCIe interface is significantly reduced, thereby improving the LLM inference performance. Our evaluation demonstrates that CPU-GPU cooperative computing, based on this policy, delivers 12.1× lower latency and 5.4× higher throughput than GPU-only computing for OPT-30B inference when both CPU-GPU and GPU-only computing store the model in CPU memory.