Intra- and Inter-host Network System and Host Microarchitecture Resource Management

Host servers have undergone a substantial increase in complexity, and cloud deployments now routinely host multiple applications and tenants on the same physical hardware. This convergence makes efficient orchestration of shared resources and low-latency intra-host communication essential for maintaining peak system performance and efficiency. Our research addresses these challenges through dynamic workload balancing across CPU cores and accelerators (e.g., SmartNICs), leveraging self-developed programmable PCIe switch for RDMA performance isolation and communication efficiency improvement, and fine-grained management of host shared resources (e.g., LLC) and microarchitectural features (e.g., Direct Cache Access). Our solutions significantly reduce tail latency, boost throughput for latency-sensitive network applications, and enhance overall system efficiency.

Programmable Intelligent PCIe Switch

We design and implement a novel PCIe hardware, Programmable Intelligent PCIe Switch (PIPS), to enable observability and controllability in the intra-host network. The analogy of this PCIe switch is a programmable switch (e.g., P4-based Tofino switch) in the traditional computer network. The difference is that we design the switch to run on PCIe instead of Ethernet. These new observability and controllability capabilities allow us to explore new debugging/diagnostic tool designs, new system software to enable intra-host network resource management, new solutions for end-to-end network performance guarantees, new management systems for emerging CPU features for fast intra-host communication. We have used this prototype to demonstrate the first microarchitecture-aware RDMA performance isolation solution. Our design, implementation, and results are published in a recent NSDI paper (Lou et al., 2024 ).

Data Direct I/O Technology (DDIO)

Data Direct I/O (DDIO) technology enables I/O devices to directly inject data into the CPU cache. The key intention of DDIO is to put I/O data closer to the CPU cores to reduce latency and memory bandwidth consumption. However, improper use of DDIO, particularly with I/O-intensive workloads, may lead to complex contention scenarios in the cache hierarchy. The IDIO paper investigates unprecedented memory access patterns arising specifically from interactions between DDIO and modern non-inclusive cache architectures. Complementarily, the A4 paper addresses contention scenarios involving various I/O devices and driven primarily by non-inclusive cache microarchitecture (Park et al., 2025 ; Jeong et al., 2023 ; Alian et al., 2020 ).

DMA paths of storage and network I/O devices

SmartNIC and Programmable Network

A typical SmartNIC (SNIC) integrates a processor comprising Arm CPU and accelerators with a conventional NIC. The processor is designed to energy-efficiently execute network functions frequently used by datacenter applications. With such a processor, the SNIC has promised to notably improve the system-wide energy efficiency of datacenter servers. Nevertheless, the latest trend of integrating accelerators into server CPUs for these functions sparks a question on the SNIC processor’s superiority over a host processor (i.e., server CPU with accelerators) in system-wide energy efficiency, especially under given tail latency constraints.

We benchmark cutting-edge SNICs and deliver insights as well as guidelines for SNIC usage in datacenters (Huang et al., 2023 ). Our solutions provide high system-wide energy efficiency without compromising tail latency at any packet rates, consisting of a hardware-based load balancer and an intelligent load balancing policy implemented inside the SNIC (Huang et al., 2024 ). We also offload various applications such as Row Hanmmer defense mechanisms and Linux kernel features to the SNICs to exploit maximum energy efficiency (Song et al., 2024 ; Ji et al., 2023 ).

References

2025

ISCA

A4: Microarchitecture-aware LLC management for datacenter servers with emerging I/O devices

Haneul Park, Jiaqi Lou, Sangjin Lee, Yifan Yuan, KyoungSoo Park, Yongseok Son, Ipoom Jeong, and Nam Sung Kim

IEEE/ACM International Symposium on Computer Architecture (ISCA) , 2025

Microarchitecture System

2024

NSDI

Harmonic: Hardware-assisted RDMA Performance Isolation for Public Clouds

Jiaqi Lou, Xinhao Kong, Jinghan Huang, Wei Bai, Nam Sung Kim, and Danyang Zhuo

21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) , 2024

System Network

Abs

Performance isolation is essential for sharing resources in multi-tenant public clouds. Compared with traditional kernel-based networking, RDMA presents unique challenges especially because RDMA NIC’s complex microarchitecture resources are often hidden from users. Current RDMA isolation methods overlook these microarchitecture resources, leading to insufficient performance isolation. Consequently, a faulty/malicious tenant can exploit these microarchitecture resources to compromise well-behaved tenants’ network performance. In this paper, we introduce the first microarchitecture-resource-aware RDMA performance isolation solution for public clouds, Harmonic. It consists of two key components designed to be conscious of the RDMA NIC’s microarchitectural resources:(1) a programmable intelligent PCIe switch (prototyped with FPGA) and (2) an RDMA-friendly rate limiter. At runtime, these two components allow us to accurately monitor and modulate the RDMA NIC resource usage per tenant. We evaluate Harmonic with a state-of-the-art RDMA performance isolation test suite (Husky) and a popular in-memory database application (Redis). We demonstrate that Harmonic can not only successfully pass Husky but also provide Redis with 1.4× higher throughput than the best alternative isolation solution.
ISCA

Hal: Hardware-assisted load balancing for energy-efficient snic-host cooperative computing

Jinghan Huang, Jiaqi Lou, Srikar Vanavasam, Xinhao Kong, Houxiang Ji, Ipoom Jeong, Danyang Zhuo, Eun Kyung Lee, and Nam Sung Kim

2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) , 2024

Architecture Accelerator System Network

Abs DOI

A typical SmartNIC (SNIC) integrates a processor comprising Arm CPU and accelerators with a conventional NIC. The processor is designed to energy-efficiently execute network functions frequently used by datacenter applications. With such a processor, the SNIC has promised to notably improve the system-wide energy efficiency of datacenter servers. Nevertheless, the latest trend of integrating accelerators into server CPUs for these functions sparks a question on the SNIC processor’s superiority over a host processor (i.e., server CPU with accelerators) in system-wide energy efficiency, especially under given tail latency constraints. Answering this question, we first take an Intel Xeon processor, integrated with various accelerators (e.g., QuickAssist Technology), as a host processor, and then compare it to an NVIDIA BlueField-2 SNIC processor. This uncovers that (1) the host accelerator, coupled with a more powerful memory subsystem, can outperform the SNIC accelerator, and (2) the SNIC processor can improve system-wide energy efficiency only at low packet rates for most functions under tail latency constraints. To provide high system-wide energy efficiency without compromising tail latency at any packet rates, we propose HAL, consisting of a hardware-based load balancer and an intelligent load balancing policy implemented inside the SNIC. When HAL determines that the SNIC processor cannot efficiently process a given function beyond a specific packet rate, it limits the rate of packets to the SNIC processor and lets the host processor handle the excess. We implement a HAL-enabled SNIC with a commodity FPGA and a BlueField-2 SNIC, plug it into a commodity server, and run 10 popular network functions. Our evaluation shows that HAL can improve the system-wide energy efficiency and throughput of the server running these functions by 31% and 10%, respectively, without notably increasing the tail latency.
ASPLOS

TAROT: A CXL SmartNIC-Based Defense Against Multi-bit Errors by Row-Hammer Attacks

Chihun Song, Michael Jaemin Kim, Tianchen Wang, Houxiang Ji, Jinghan Huang, Ipoom Jeong, Jaehyun Park, Hwayong Nam, Minbok Wi, Jung Ho Ahn, and Nam Sung Kim

Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) , 2024

Architecture Memory Accelerator Security

Abs DOI

Row Hammer (RH) has been demonstrated as a security vulnerability in modern systems. Although commodity CPUs can handle RH-induced single-bit errors in DRAM through ECC, RH can still give rise to multi-bit uncorrectable errors (UEs) and crash the systems. Meanwhile, recent work has indicated that the DRAM cells vulnerable to RH are determined by manufacturing imperfections and resulting defects. Taking one step further from the recent work, we first conduct RH experiments on contemporary DRAM modules for 3 weeks. This demonstrates that RH-induced UEs occur only at specific DRAM addresses (RH-UE-vulnerable addresses) and the percentage of such addresses is small in these DRAM modules. Second, to protect the systems from RH-induced UEs, we propose two RH defense solutions: H- and S-TAROT (TArgeted ROw-Hammer Therapy). H-TAROT is a software-based solution running on the host CPU. It obtains RH-UE-vulnerable addresses during the system boot and then periodically accesses such addresses before UEs may occur. Since it accesses only a small percentage of addresses, it does not incur a notable performance penalty for throughput applications (e.g., a 1.5% increase in execution time of the SPECrate 2017 benchmark suite running on a system even with 128GB of DRAM). Yet, it imposes a significant performance penalty on latency-sensitive applications (e.g., a 28.2% increase in tail latency of Redis). To minimize the performance penalty, for a system with a SmartNIC (SNIC), S-TAROT offloads H-TAROT from the host CPU to the SNIC CPU. Our experiment shows that S-TAROT increases the execution time and tail latency of the SPECrate 2017 benchmark suite and Redis by only 0.1% and 1.0%, respectively.

2023

CAL

LADIO: Leakage-aware direct I/O for I/O-intensive workloads

Ipoom Jeong, Jiaqi Lou, Yongseok Son, Yongjoo Park, Yifan Yuan, and Nam Sung Kim

IEEE Computer Architecture Letters , 2023

Abs DOI

The advancement in I/O technology has posed an unprecedented demand for high-performance processing on I/O data, leading to the development of Data Direct I/O (DDIO) technology. DDIO improves I/O processing efficiency by directly injecting all inbound I/O data into the last-level cache (LLC) in cooperation with any type of I/O device. Nonetheless, in certain scenarios with more than one I/O applications, DDIO may have sub-optimal performance caused by interference inside the LLC, resulting in the degradation of system performance. Especially, in this paper, we demonstrate that storage I/O on modern high-performance NVMe SSDs hardly benefits from DDIO, sometimes causing inefficient use of the shared LLC due to the “leaky DMA problem”. To address this problem, we propose LADIO, an adaptive approach that mitigates inter-application interference by dynamically controlling the DDIO functionality and reallocating LLC ways based on the leakage and locality of storage I/O data, respectively. In scenarios with heavy I/O interference, LADIO improves the throughput of network-intensive applications by 20% while maintaining that of storage-intensive applications.
IISWC

Making sense of using a smartnic to reduce datacenter tax from slo and tco perspectives

Jinghan Huang, Jiaqi Lou, Yan Sun, Tianchen Wang, Eun Kyung Lee, and Nam Sung Kim

2023 IEEE International Symposium on Workload Characterization (IISWC) , 2023

Best Paper Runner-up Award

Systems Network Accelerator

Abs DOI

The speed of network interfaces has rapidly increased, while the performance and energy efficiency of CPUs have not, due to the demise of Dennard scaling. As a result, functions processing network packets have become responsible for a rapidly increasing portion of the datacenter tax. To tackle this problem, the industry has developed SmartNICs (SNICs) integrating conventional NICs with inexpensive and energy-efficient processors that can efficiently execute functions widely used by network-intensive datacenter applications. With such processors, the SNICs promise to reduce the total cost of ownership (TCO) for datacenters by increasing energy efficiency of servers and/or decreasing the number of expensive server CPU cores. In this paper, to make sense of using SNICs, we focus on analyzing energy efficiency of a server with an SNIC, especially under service level objective (SLO) constraints which matter for many datacenter applications. To this end, we not only measure the system-wide power consumption of a server but also devise a custom hardware setup that allows us to isolate the power consumption of the SNIC from that of the server. This helps us better understand the impact of using SNICs on energy efficiency of servers. Second, we take popular TCP/UDP, DPDK, and RDMA-based functions, and prepare them to run on an SNIC processor and a server CPU. Then, we measure maximum throughput, tail latency, system-wide energy efficiency of executing the functions on the SNIC processor and the server CPU, respectively. Lastly, based on analyses of the measurements, we make five key observations and three strategies to better use and design SmartNICs in the future.
ATC

STYX: Exploiting SmartNIC capability to reduce datacenter memory tax

Houxiang Ji, Mark Mansi, Yan Sun, Yifan Yuan, Jinghan Huang, Reese Kuper, Michael M Swift, and Nam Sung Kim

2023 USENIX Annual Technical Conference (USENIX ATC 23) , 2023

Systems Network Accelerator

Abs DOI

Memory optimization kernel features, such as memory deduplication, are designed to improve the overall efficiency of systems like datacenter servers, and they have proven to be effective. However, when invoked, these kernel features notably disrupt the execution of applications, intensively consuming the server CPU’s cycles and polluting its caches. To minimize such disruption, we propose STYX, a framework for offloading the intensive operations of these kernel features to SmartNIC (SNIC). STYX first RDMA-copies the server’s memory regions, on which these kernel features intend to operate, to an SNIC’s memory region, exploiting SNIC’s RDMA capability. Subsequently, leveraging SNIC’s (underutilized) compute capability, STYX makes the SNIC CPU perform the intensive operations of these kernel features. Lastly, STYX RDMA-copies their results back to a server’s memory region, based on which it performs the remaining operations of the kernel features. To demonstrate the efficacy of STYX, we re-implement two memory optimization kernel features in Linux:(1) memory deduplication (ksm) and (2) compressed cache for swap pages (zswap), using the STYX framework. We then show that a system with STYX provides a 55-89% decrease in 99th-percentile latency of co-running applications, compared to a system without STYX, while preserving the benefits of these kernel features.

2020

CAL

IDIO: Orchestrating Inbound Network Data on Server Processors

Mohammad Alian, Jongmin Shin, Ki-Dong Kang, Ren Wang, Alexandros Daglis, Daehoon Kim, and Nam Sung Kim

IEEE Computer Architecture Letters , 2020

Abs DOI

Network bandwidth demand in datacenters is doubling every 12 to 15 months. In response to this demand, high-bandwidth network interface cards, each capable of transferring 100s of Gigabits of data per second, are making inroads into the servers of next-generation datacenters. Such unprecedented data delivery rates on server endpoints raise new challenges, as inbound network traffic placement decisions within the memory hierarchy have a direct impact on end-to-end performance. Modern server-class Intel processors leverage DDIO technology to steer all inbound network data into the last-level cache (LLC), regardless of the network traffic’s nature. This static data placement policy is suboptimal, both from a performance and an energy efficiency standpoint. In this work, we design IDIO, a framework that—unlike DDIO—dynamically decides where to place inbound network traffic within a server’s multi-level memory hierarchy. IDIO dynamically monitors system behavior and distinguishes between different traffic classes to determine and periodically re-evaluate the best placement location for each flow: LLC, mid-level (L2) cache or DRAM. Our results show that IDIO increases a server’s maximum sustainable load by up to ∼33.3% across various network functions.