Processing In / Near Memory (PIM/PNM) | Future Architecture and System Technology for Scalable Computing

The performance of modern computing systems is bottlenecked by the off-chip communication bandwidth and its energy cost. Integrating the accelerators within DRAM (i.e., PIM) can mitigate these bottlenecks and additionally expose them to the higher internal bandwidth of DRAM. However, such an integration is challenging, as it should not disturb the highly-optimized DRAM core circuitry. To enable the integration, our work proposed an integration of a SIMD acceleration unit with each DRAM bank (Kim et al., 2022 ).

Later, Professor Kim worked with talented Samsung engineers and researchers to further develop an industry version of an advanced PIM architecture similar to what he proposed (i.e., integration of SIMD units with DRAM banks). For a commercial success, he aimed to develop a PIM architecture that does not demand any change to (1) the existing processor architectures and (2) the existing DRAM interface so that the PIM device can be a drop-in-replacement of standard DRAM device; this development philosophy was inspired by the success of SSD which was initially developed as a drop-in-replacement of HDD although it began to use an interface specialized for SSD later. The PIM architecture was implemented in HBM2 (HBM-PIM) (the middle illustration), and later HBM-PIM was integrated with Xilinx U280 (the bottom illustration), AMD MI-50 GPU, and AMD MI-100 GPU. This PIM architecture became the foundation of a JEDEC HBM3-PIM standard later.

example image — An Industry Prototype by Samsung (Kim et al., 2022 )

We are the first research group proposed a DIMM-based near-DRAM computing (NDA: Near-DRAM acceleration architecture leveraging commodity DRAM devices and standard memory modules, HPCA 2015) and this MCN-based near-DRAM processing work became the foundation for Samsung’s AxDIMM, and Samsung and Meta jointly evaluated its efficacy for accelerating Deep Learning Recommendation Model (DLRM). The AxDIMM has been ued as a popular platform to explore various near-DRAM processing concept until the CXL-based memory modules were introduced to industry researchers.

The physical memory capacity of servers is expected to increase drastically with deployment of the forthcoming non-volatile memory technologies. This is a welcomed improvement for emerging data-intensive applications. For such servers to be cost-effective, nonetheless, we must cost-effectively increase compute throughput and memory bandwidth commensurate with the increase in memory capacity without compromising application readiness. Tackling this challenge, we developed Memory Channel Network (MCN) architecture. Specifically, we first proposed an MCN DIMM, an extension of a buffered DIMM where a small but capable processor called MCN processor is integrated with a buffer device on the DIMM for near-memory processing. Second, we implemented device drivers to give the host and MCN processors in a server an illusion that they are independent heterogeneous nodes connected through an Ethernet link. These allow the host and MCN processors in a server to run a given data-intensive application together based on popular distributed computing frameworks such as MPI and Spark without any change in the host processor hardware and its application software, while offering the benefits of high-bandwidth and low-latency communications between the host and the MCN processors over memory channels. As such, MCN can serve as an application-transparent framework which can seamlessly unify near-memory processing within a server and distributed computing across such servers for data-intensive applications. Our evaluation shows that a server with 8 MCN DIMMs offers 4.56X higher throughput and consume 47.5% less energy than a cluster with 9 conventional nodes connected through Ethernet links, as it facilitates up to 8.17 times higher aggregate DRAM bandwidth utilization. Then, we demonstrated the feasibility of MCN with an IBM POWER8 system and an experimental buffered DIMM (Farmahini-Farahani et al., 2015 ).

Processing-in-memory (PIM) has been proposed to improve the performance of bandwidth-intensive workloads as well as save energy due to reduced compute-memory data movement. To realize PIM, programmable computing units were integrated with memory cores on an HBM2 device to enable parallel processing and minimize data movement. A graphics processing unit (GPU) system equipped with Samsung Aquabolt-XL HBM2-PIM devices improved microkernel general matrix–vector multiplication and speech recognition applications by 8.9× and 3.5×, respectively, and reduced energy consumption by over 60%. In a Xilinx AlveoU280 system, microkernel GEMV and ADD workload performances improved by 2.8×, and long short-term memory workload improved by 2.54×. Simulations show that a performance gain of over 2.3× may be attained in a system with LP5-PIM for certain transformer-based speech recognition with an energy reduction of 86%. In addition, AXDIMM, a DIMM-level PIM with acceleration buffers, exhibits an 80% performance improvement and a 42.6% energy savings over a regular RDIMM system.

Energy consumed for transferring data across the processor memory hierarchy constitutes a large fraction of total system energy consumption, and this fraction has steadily increased with technology scaling. In this paper, we propose near-DRAM acceleration (NDA) architectures, which process data using accelerators 3D-stacked on DRAM devices comprising off-chip main memory modules. NDA transfers most data through high-bandwidth and low-energy 3D interconnects between accelerators and DRAM devices instead of low-bandwidth and high-energy off-chip interconnects between a processor and DRAM devices, substantially reducing energy consumption and improving performance. Unlike previous near-memory processing architectures, NDA is built upon commodity DRAM devices; apart from inserting through-silicon vias (TSVs) to 3D-interconnect DRAM devices and accelerators, NDA requires minimal changes to the commodity DRAM device and standard memory module architectures. This allows NDA to be more easily adopted in both existing and emerging systems. Our experiments demonstrate that, on average, our NDA-based system consumes 46% (68%) lower (data transfer) energy at 1.67× higher performance than a system that integrates the same accelerator logic within the processor itself.

References

2022

2015