Compute Express Link (CXL)

CXL is an open-standard high-speed interconnect technology that enables data transfer between CPUs, SmartNICs, GPUs, and other devices. It offers three protocols—CXL.io, CXL.cache and CXL.memory—to balance I/O, coherent caching and direct memory access.

Modern computing systems have demanded higher memory bandwidth. However, the current parallel interface (e.g., DDR4) can increase neither the number of (memory) channels nor the bit rate of the channels without paying high package and power costs. In contrast, the high-speed serial interface (HSI) can offer much higher bandwidth for the same number of pins and lower power consumption for the same bandwidth than the parallel interface. This allows us to integrate more channels under a pin and/or package power constraint but at the cost of longer latency for memory accesses and higher static energy consumption in particular for idle channels. In our work, we first provided a deep understanding of recent HSI (e.g., PCIe) exhibiting very distinct characteristics from past serial interfaces in terms of bit rate, latency, energy per bit transfer, and static power consumption. To overcome the limitation of using only parallel (e.g., DDR) or serial (e.g., PCIe) interfaces, we proposed a hybrid memory channel architecture-Alloy (the top illustration) consisting of low-latency parallel and high-bandwidth serial channels. At the architecture level, this proposal is the same as the current CXL memory expander in which a CPU is connected to both DDR and PCIe-based CXL (Compute eXpress Link) (the middle illustration).

CXL Memory

CXL-based memory expander technology (CXL.mem) is an extension of the CXL specification that allows systems to expand memory beyond the physical DIMMs installed by connecting external memory devices, such as high-speed DRAM or persistent memory, through the CXL interconnect. Unlike traditional memory expansion, which adds more DIMM slots or increases existing DIMM capacity, CXL.mem enables seamless integration of external memory, appearing as local memory to the system.

It offers high bandwidth and low latency, making it ideal for data-intensive tasks like machine learning and high-performance computing. CXL.mem is still a developing technology, and it is not yet widely available. However, it has the potential to greatly increase memory capacity and performance in a wide range of applications, demonstrated in our works (Sun et al., 2023 ; Sun et al., 2025 ). In addition, we build cost-efficient solution for the use of recycled DRAM modules with thousands of faulty words, exploiting the CXL’s support for near-memory processing and variable memory access latency (Song et al., 2025 ).

CXL Type-2 Device

We delve into three key features of a CXL Type-2 device: cache-coherent device accelerator to host memory, device accelerator to device memory, and host CPU to device memory accesses. Second, using microbenchmarks, we comprehensively characterize the latency and bandwidth of these memory accesses with a CXL Type-2 device, and then compare them with those of equivalent memory accesses with comparable devices, such as emulated CXL Type-2, CXL Type-3, and PCIe devices. Building on these insights, we introduce a cooperative computing framework that integrates Type-2 devices into the operating system and runtime, allowing seamless offloading of heterogeneous tasks with substantial performance and energy-efficiency improvements (Ji et al., 2024 ).

References

2025

ASPLOS

M5: Mastering Page Migration and Memory Management for CXL-based Tiered Memory Systems

Yan Sun, Jongyul Kim, Zeduo Yu, Jiyuan Zhang, Siyuan Chai, Michael Jaemin Kim, Hwayong Nam, Jaehyun Park, Eojin Na, Yifan Yuan, Ren Wang, Jung Ho Ahn, Tianyin Xu, and Nam Sung Kim

2025

System Architecture Memory

Abs DOI

CXL has emerged as a promising memory interface that can cost-effectively expand the capacity and bandwidth of a memory system, complementing the traditional DDR interface. However, CXL DRAM presents 2-3x longer access latency than DDR DRAM, forming a tiered-memory system that demands an effective and efficient page-migration solution. Although many page-migration solutions have been proposed for past tiered-memory systems, they have achieved limited success. To tackle the challenge of managing tiered-memory systems, this work first presents a CXL-driven profiling solution to precisely and transparently count the number of accesses to every 4KB page and 64B word in CXL DRAM. Second, using the profiling solution, this work uncovers that (1) widely used CPU-driven page-migration solutions often identify warm pages as hot pages, and (2) certain applications have sparse hot pages, where only a small percentage of words in each of these pages are frequently accessed. Besides, this work demonstrates that the performance overhead of identifying hot pages is sometimes high enough to degrade application performance. Lastly, this work presents M5, a platform designed to facilitate the development of effective CXL-driven page-migration solutions, providing hardware-based hot-page and hot-word trackers in the CXL controller. On average, M5 can identify 47% hotter pages and offer 14% higher performance than the best CPU-driven page-migration solution, even with a simple policy.
CAL

X-PPR: Post Package Repair for CXL Memory

Chihun Song, Michael Jaemin Kim, Yan Sun, Houxiang Ji, Kyungsan Kim, TaeKyeong Ko, Jung Ho Ahn, and Nam Sung Kim

IEEE Computer Architecture Letters , 2025

Architecture DRAM

Abs DOI

CXL is an emerging interface that can cost-efficiently expand the capacity and bandwidth of servers, recycling DRAM modules from retired servers. Such DRAM modules, however, will likely have many uncorrectable faulty words due to years of strenuous use in datacenters. To repair faulty words in the field, a few solutions based on Post Package Repair (PPR) and memory offlining have been proposed. Nonetheless, they are either unable to fix thousands of faulty words or prone to causing severe memory fragmentation, as they operate at the granularity of DRAM row and memory page addresses, respectively. In this work, for cost-efficient use of recycled DRAM modules with thousands of faulty words, we propose CXL-PPR (X-PPR), exploiting the CXL’s support for near-memory processing and variable memory access latency. We demonstrate that X-PPR implemented in a commercial CXL device with DDR4 DRAM modules can handle a faulty bit probability that is 3.3×104 higher than ECC for a 512GB DRAM module. Meanwhile, X-PPR negligibly degrades the performance of popular memory-intensive benchmarks, which is achieved through two mechanisms designed in X-PPR to minimize the performance impact of additional DRAM accesses required for repairing faulty words.

2024

MICRO

Demystifying a CXL Type-2 Device: A Heterogeneous Cooperative Computing Perspective

Houxiang Ji, Srikar Vanavasam, Yang Zhou, Qirong Xia, Jinghan Huang, Yifan Yuan, Ren Wang, Pekon Gupta, Bhushan Chitlur, Ipoom Jeong, and Nam Sung Kim

2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO) , 2024

System Accelerator

Abs DOI

CXL is the latest interconnect technology built on PCIe, providing three protocols to facilitate three distinct types of devices, each with unique capabilities. Among these devices, a CXL Type-2 device has become commercially available, followed by CXL Type-3 devices. Therefore, it is timely to understand capabilities and characteristics of the CXL Type-2 device, as well as explore suitable applications. In this work, first, we delve into three key features of a CXL Type-2 device: cache-coherent device accelerator to host memory, device accelerator to device memory, and host CPU to device memory accesses. Second, using microbenchmarks, we comprehensively characterize the latency and bandwidth of these memory accesses with a CXL Type-2 device, and then compare them with those of equivalent memory accesses with comparable devices, such as emulated CXL Type-2, CXL Type-3, and PCIe devices. Lastly, as applications that exploit the unique capabilities of a CXL Type-2 device, we propose two CXL-based Linux memory optimization features: compressed RAM cache for swap (zswap) and memory deduplication (ksm). Our evaluation shows that Redis, when running with traditional CPU-based zswap and ksm, suffers from a tail latency increase of 4.5-10.3× compared to Redis running alone. While PCIe-based zswap and ksm still experience a tail latency increase of up to 8.1×, CXL-based zswap and ksm practically eliminate the tail latency increase with faster and more efficient host-device communication than PCIe-based zswap and ksm.

2023

MICRO

Demystifying cxl memory with genuine cxl-ready systems and devices

Yan Sun, Yifan Yuan, Zeduo Yu, Reese Kuper, Chihun Song, Jinghan Huang, Houxiang Ji, Siddharth Agarwal, Jiaqi Lou, Ipoom Jeong, Ren Wang, Jung Ho Ahn, Tianyin Xu, and Nam Sung Kim

2023 56th IEEE/ACM International Symposium on Microarchitecture (MICRO) , 2023

Systems Memory

Abs DOI

The ever-growing demands for memory with larger capacity and higher bandwidth have driven recent innovations on memory expansion and disaggregation technologies based on Compute eXpress Link (CXL). Especially, CXL-based memory expansion technology has recently gained notable attention for its ability not only to economically expand memory capacity and bandwidth but also to decouple memory technologies from a specific memory interface of the CPU. However, since CXL memory devices have not been widely available, they have been emulated using DDR memory in a remote NUMA node. In this paper, for the first time, we comprehensively evaluate a true CXL-ready system based on the latest 4th-generation Intel Xeon CPU with three CXL memory devices from different manufacturers. Specifically, we run a set of microbenchmarks not only to compare the performance of true CXL memory with that of emulated CXL memory but also to analyze the complex interplay between the CPU and CXL memory in depth. This reveals important differences between emulated CXL memory and true CXL memory, some of which will compel researchers to revisit the analyses and proposals from recent work. Next, we identify opportunities for memory-bandwidth-intensive applications to benefit from the use of CXL memory. Lastly, we propose a CXL-memory-aware dynamic page allocation policy, Caption to more efficiently use CXL memory as a bandwidth expander. We demonstrate that Caption can automatically converge to an empirically favorable percentage of pages allocated to CXL memory, which improves the performance of memory-bandwidth-intensive applications by up to 24% when compared to the default page allocation policy designed for traditional NUMA systems.