Accelerating GPGPU Simulation by Strategically Parallelizing the Compute Bottleneck

GPGPU-Sim is a cycle-accurate simulator for GPUs that supports simulating the execution of applications written using both CUDA and OpenCL. For the purposes of this work, we focus exclusively on CUDA programs. The simulator functions by intercepting the application’s CUDA calls and replacing them with its own calls. This allows it to simulate the target GPU architecture in detail and collect performance statistics. This interception approach provides significant ease of usability, often requiring minimal application modification. GPGPU-Sim also includes a functional-only mode, which bypasses detailed timing simulation. It also incorporates a power model for energy analysis and tools for visualization. GPGPU-Sim’s hardware model is designed to closely model contemporary NVIDIA GPUs. The main architecture features of the model are:

• $\mathrm{■}$

  SIMT cores: These are the main execution units on which thread blocks/CTAs are scheduled and are thus the equivalent of NVIDIA’s Streaming Multiprocessors (SMs).

• $\mathrm{■}$

  SIMT core clusters: Clusters of multiple SIMT cores that share an interface to the memory interconnect, occupying the same place in the organizational hierarchy as NVIDIA’s texture processing clusters (TPCs).

• $\mathrm{■}$

  GPC: A collection of SIMT core clusters. This level incorporates an internal high-speed network (the SM-to-SM network) connecting its constituent clusters without needing to go through the global interconnect. This addition of the GPC structure and its internal network enables the simulation of features like Hopper’s distributed shared memory and thread block clusters.

• $\mathrm{■}$

  Memory Partition: Represents a slice of the device’s memory subsystem, typically including a portion of DRAM and its associated L2 cache slice.

• $\mathrm{■}$

  Interconnect: The interconnect between the SIMT core clusters and the memory partitions. GPGPU-Sim utilizes BookSim [7] to simulate this interconnect.

The component layout is shown in Figure 1. The main driver for larger computing power is the increased component counts, especially the number of SMs, which is the dominant contributor to simulation time (Subsection 3.1). Historically, NVIDIA’s flagship SM count grew from up to 16 (Fermi, 2010) [19] to up to 144 (Hopper, 2022) [1]. This trend is advantageous for our scheme, as it is the dimension along which we parallelize (Subsection 3.2). The simulation is controlled by a central clock, which serially steps through sub-clocks (CORE, L2, DRAM, ICNT, SM_NETWORK). Our optimizations focus entirely on the CORE-clock section, as this handles instruction execution and scales the most with grid size.

The GPGPU simulation landscape extends beyond cycle-accurate simulators like GPGPU-Sim. Machine-learning-based models predict kernel performance [20] or the performance gain of porting applications from CPU to GPGPU [2]. Analytical approaches also statistically model architecture behavior based on execution traces [4, 18, 9]. Despite their utility, researching novel architectural designs often requires the higher fidelity of computationally intensive, cycle-accurate simulators. While valuable for research [13, 15, 6], the slow speed of cycle-accurate simulators like GPGPU-Sim limits study scope and motivates efforts to improve their performance. Early attempts to parallelize GPGPU-Sim, such as by Lee et al. [10], split hardware into shared (memory, interconnect) and parallel (SIMT-Cores) components. This achieved up to $4.15\times$ speedup ($3.39\times$ average with 6 threads) but incurred substantial cycle count errors of up to $5.78\%$.

Zhao et al. [21] implemented a two-level scheme, parallelizing “intra-kernel” (across SIMT-core-clusters) and “inter-kernel” (across machines), estimating a $10-40\times$ speedup with a $0.16\%$ cycle error, though these results are just a theoretical combination of their two separate results without empirical verification of the combined approach. Wang et al. [17] proposed a two-step parallelization: first, evaluating SIMT cores in parallel ($1.61-2.24\times$ speedup, $\le 0.22\%$ error), and second, separating $CORE$, $L2$, $Interconnect$, and $DRAM$ clock domains using a “shadow-clock”. However, such strategies often involve accuracy trade-offs or extensive code changes, hindering adaptability and upstream integration. More recently, Huerta et al. [5] accelerated Accel-Sim using OpenMP with minimal code modification, achieving an average speedup of $5.8\times$ (up to $14\times$) with 16 threads.

The original implementation was profiled using Linux’s perf tool. We profiled our benchmarks on an AMD EPYC 9124 running at $3\text{GHz}$ and up to $3.70\text{GHz}$ in turbo and then evaluate the results by examining the percentage of samples recorded inside each function as a proxy for time spent inside. As anticipated, our profiling results reveal that almost the entirety of the runtime is spent with cycling the simulator clock. On average $98.3\%$ of all recorded samples are within the gpgpu_sim::cycle() function and its child calls (SGEMM: $99.2\%$, SM2SM_HIST: $98.1\%$, MONTE_CARLO: $98.6\%$, VECTOR_ADD: $97.2\%$).

To get a clearer understanding of this, we observe the child calls of gpgpu_sim::cycle(), given its role as the primary simulation clock. Analyzing these child calls reveals a consistent pattern: the same set of functions are the primary contributors to the runtime of gpgpu_sim::cycle() across all benchmarks. Among these, one function consistently emerges as the dominant contributor, although its exact percentage contribution varies per benchmark.

Profiling the major child calls of gpgpu_sim::cycle() (as detailed in Table 1) confirms that the CORE-Clock, handled by simt_core_cluster::core_cycle(), is the dominant contributor to runtime. This confirmation validates our focus and allows us to proceed with the design and implementation of the parallelization scheme.

We’ll now outline our approach to parallelizing the CORE-clock cycle and explain why our method is preferable. Given that profiling identified simt_core_cluster::core_cycle() as the primary contributor within the CORE-clock’s execution breakdown, we must first examine its surrounding code structure to inform our parallelization strategy. The main simulator clock is composed of different sub-clocks, which execute based on which one is active. gpgpu_sim::cycle() represents this main simulation clock in code, managing the execution of the separate sub-clocks, as structurally simplified in Algorithm 1.

Based on the profiling results in Subsection 3.1, the CORE-clock section represented by core_cycle() contains the primary hotspot function. We will now examine the actual internal structure of this core_cycle() section.

As illustrated in Algorithm 2, the compute-heavy portion of each cycle is largely independent across the different SIMT-clusters, with dependencies limited primarily to control logic (scheduling and statistics). Similar to related work covered in Subsection 2.2, we chose to parallelize along this cluster dimension: the set of all clusters is split, and the clst.core_cycle() function for each is evaluated in parallel. This approach is the least invasive method for addressing the identified hotspot and minimizes synchronization overhead compared to parallelizing individual SIMT-Cores within the clusters.

Parallelizing each individual <...>::core_cycle() by launching a new software thread is infeasible due to the substantial overhead from frequent thread creation and destruction in latency-sensitive code. To quantify this overhead, consider the SGEMM $1024\times 1024$ benchmark, which runs for $2.4\text{hrs}$ and involves $1.5\times {10}^6$ CORE-cycles.

Assuming an optimistic $10\text{μ}\text{s}$ overhead per thread-launch, the total wall-time added solely by thread creation across the simulation’s 57 clusters is:

This overhead represents approximately $10\%$ of the total runtime and becomes a bottleneck; any performance improvements must first overcome this initial $\approx 10\%$ slowdown, which is likely underestimated. To address this bottleneck in thread management, we instead opted for a thread reuse scheme, implemented using the C++ thread pool by [16]. This dynamic thread pool approach allows us to control the number of software threads used for cycle execution, enabling precise tuning based on both the host hardware and the simulated hardware model. The pool is initialized once at startup and currently handles only the SIMT-Cluster parallelization.

Algorithm 3 shows the simplified thread pool structure, detailing how tasks are dispatched to worker threads and which components are parallelized. We explicitly wait for all clusters to complete their core cycle before updating statistics based on their finished state. This statistics collection step is kept sequential, as it operates on small pieces of shared state and parallelization is unlikely to provide benefit while potentially introducing race conditions. Access to the shared state manipulated by simt_core_cluster::core_cycle() is primarily managed using std::mutex from the C++ standard library. This static, even partitioning of clusters can be suboptimal under partial system loads. When only a fraction of SIMT-Clusters are active, threads assigned to inactive clusters are underutilized. While detecting and reassigning empty cycling clusters to a single thread could improve speedup in these non-full occupancy scenarios, the additional scheduling logic required would add latency, potentially reducing speedup in the optimal, near-full occupancy case. Given that high-occupancy scenarios correspond to the longest sequential wall-times and are the primary optimization target, we chose to optimize for the highest occupancy rather than implementing dynamic load balancing.

The model configuration used for all tests is designed to replicate the NVIDIA H100 architecture. Specifically, the simulation aimed for a replica of the H100 PCIe $80\text{GB}$ configuration, which features 114 SMs, as briefly discussed in Subsection 2.1. The primary focus is the number of SIMT clusters and SIMT cores (SMs) in the model. Since parallelization occurs across these clusters, the high cluster count is expected to positively influence speedup results.

The benchmarks were executed on a compute node equipped with dual Intel Xeon Gold 6130 CPUs. Unlike the AMD login node used for profiling, these nodes offer a SMP configuration of their cores, which is more suitable for our workload. These nodes provide a total of 32 physical cores (2 sockets $\times$ 16 cores/socket, with hyper-threading disabled) operating at a $2.10\text{GHz}$ base clock speed ($3.70\text{GHz}$ turbo), supported by $192\text{GB}$ of RAM split across the two NUMA nodes. For configurations with $N\le 16$ worker threads, we pinned the process to a single NUMA node using numactl --cpunodebind=0 --membind=0. For $N>16$, threads were allowed to span both NUMA nodes under default Linux scheduling.

To evaluate the performance impact of our parallelization strategy across diverse computational patterns, we selected four CUDA kernels. These kernels represent a range of common GPGPU workload characteristics, from compute-intensive tasks to memory-bound operations.

• $\mathrm{■}$

  MONTE_CARLO: Estimates $\pi$ via Monte Carlo using a simple LCG generating single-precision pseudo-random $(x,y)$ pairs. Results are aggregated within thread-blocks via a shared memory reduction, and final summation happens on the host CPU.

• $\mathrm{■}$

  SGEMM: For our evaluation, we employ an SGEMM (Single-Precision General Matrix Multiplication) kernel based on [3], which operates on two randomized square matrices using 2D shared-memory tiling.

• $\mathrm{■}$

  SM2SM_HIST: A histogram implementation adapted from NVIDIA’s official guide [14] for using the distributed-shared-memory feature. This is our benchmark targeted to check for incurred cycle error in heavy use of the SM-to-SM net model. The thread-block-clusters are statically configured to run at the maximum size possible on the sm90 architecture (16 thread-blocks per cluster).

• $\mathrm{■}$

  VECTOR_ADD: A kernel that performs element-wise addition on vectors of single-precision floating-point values. It was selected primarily because it exhibits high memory intensity, requiring only one arithmetic operation per element.

These benchmarks should equip us to make relatively general statements about the scaling behavior of our implementation of the parallelization scheme.

The primary objective of this work is reducing the wall-time $T_{\text{wall}}$ of GPGPU-Sim simulations, making it the main performance metric. Although factors like total CPU time, utilization, or energy efficiency may be relevant, they are considered secondary to minimizing $T_{\text{wall}}$.

To evaluate the effectiveness and scaling of our parallelization approach, we calculate the speedup $S_N$ achieved using $N$ threads:

where $T_N$ is the wall-time of the parallelized simulator running with $N$ threads. To ensure functional correctness and verify that our parallelization does not alter the simulation semantics, we measure the total number of simulated GPU cycles $C$ reported by the simulator. We compare the cycle-count from the parallel execution $C_N$ with the reported count by the original sequential execution $C_{\text{seq}}$ and measure the absolute percentage deviation:

While GPGPU-Sim reports many more statistics than just the total simulated GPU cycles, this is a good proxy for the other statistics, and also allows us to compare our results with the other works covered in Subsection 2.2.

Using the hardware, software, simulated GPU configuration, and benchmarks described above, we performed the following experiments to evaluate our parallelized implementation and the base version:

1. 1.

   We measured the wall-time for the original sequential version ($T_{\text{seq}}$) and our implementation with 8 worker threads ($T_8$). We set the input-sizes of each benchmark so that ($T_{\text{seq}}$) was at least 60 minutes. We also record the simulated cycles so we can quantify the incurred cycle-error (as described in Subsection 4.4).

2. 2.

   We measured the wall-time for the sequential version ($T_{\text{seq}}$) and our implementation with different worker thread counts, from 2 up-to and including 32 in increments of 2 ($T_2,T_4,\mathrm{\dots },T_{32}$). Due to the increased amount of runs for this benchmark, we reduced the input-sizes slightly. This experiment allows us to observe the performance scaling characteristic of our implementation and potential impacts of the NUMA architecture of our dual-socket machine (as detailed in Subsection 4.2). We also measure the cycle-error in the same way as described above.

The input sizes for each benchmark can be seen in Table 2. For both sets, we average the resulting metrics over three separate runs.

We begin by examining the results for our long-running benchmarks, comparing the base version and our implementation with 8 worker threads.

2(a) presents the measured wall-times for both of these ($T_{\text{seq}}$ and $T_8$) across the different benchmarks. These are averaged over three separate runs, as detailed in Subsection 4.4.

Our parallelization scheme achieved a notable reduction in wall-time over all the benchmarks. The speedups ($S_8=T_{\text{seq}}/T_8$), calculated from these average wall-times, were $3.48\times$ for MONTE_CARLO, $3.57\times$ for SGEMM, $3.25\times$ for SM2SM_HIST, and $4.03\times$ for VECTOR_ADD. The difference between benchmarks is likely explained by the different computational and data-access patterns of the different benchmarks.

Across these four benchmarks, our implementation demonstrated an average speedup of $3.58\times$. This speedup, while remarkable, is below an ideal linear value of $8\times$. This is to be expected due to factors such as inherent sequential portions of the simulation, synchronization overheads, and communication costs between threads, as discussed in Subsection 3.2. The exact scaling behavior of our implementation is discussed below (in Subsection 5.2).

To assess how effectively our parallelization strategy utilizes increasing computational resources, we evaluated its scaling behavior. Figure 3 illustrates the measured speedup $S_N$, relative to the sequential base version, for all the benchmarks as the number of worker threads $N$ was varied from 2 up to 32.

We can see in Figure 3 that our implementation scales well up to a certain amount of worker threads, but starts delivering diminishing returns around $N\ge 8$. The benchmarks peak at different thread counts: MONTE_CARLO and SGEMM both top out at $N=10$ with $3.22\times$ and $3.85\times$ respectively, while VECTOR_ADD doesn’t top out until $N=14$ with a speedup of $4.38\times$ and SM2SM_HIST only tops out at $N=16$ with a speedup of $3.60\times$ over the sequential base version.

We can also see a drop in speedup as the thread count increases above 16. The reason for this is quite clearly the nature of our hardware platform (described in Subsection 4.2) having 2 NUMA Nodes, each with 16 CPU cores. So what we are likely observing here is the increase in NUMA-latency as the program is spread across the two NUMA nodes (we manually restricted the program to a single NUMA node if $N\le 16$).

The diminishing returns effect we observe around 8-10 worker threads is likely due to the increase in synchronization overhead beginning to outweigh the additional speedup of more threads. As mentioned above, we would not expect perfect speedup for any worker thread count, since there are necessary sequential sections to the simulator, and also the mentioned synchronization and communication overhead from the parallelization scheme itself.

As discussed in Subsection 2.2, there have been previous efforts in parallelizing GPGPU-Sim. The comparison reveals distinct trade-offs between performance and accuracy across different parallelization strategies.

Among prior approaches, Work-Group Parallel achieves the highest average speedup at $3.39\times$, but exhibits significantly higher error rates, with maximum errors reaching $5.78\%$. This suggests that while aggressive parallelization can yield substantial performance gains, it may compromise simulation fidelity.

In contrast, both gpusim-para1 and gpusim-para2 prioritize accuracy, maintaining maximum errors below $0.6\%$. However, this reduced error comes at the cost of reduced speedup, with gpusim-para1 achieving $1.91\times$ and gpusim-para2 reaching $2.1\times$ median speedup (the paper only reports median for this specific result). The gpusim-para1 implementation demonstrates exceptional precision with some benchmarks showing zero error, indicating that highly accurate parallel simulation is achievable but requires more conservative parallelization strategies.

We exclude the work of Zhao et al. [21] from this comparison, because the reported $10$ – $40\times$ speedup is not empirically measured but rather a theoretical product of their separate intra-kernel and inter-kernel results.

Comparing implementation complexity, our approach required only 10 lines of critical path changes versus Wang et al.’s [17] 170 lines for their simpler version. This minimal modification approach, combined with our balanced performance-accuracy profile, is particularly valuable for research requiring long-term maintainability and frequent updates to the code base.