DySM: Dynamic Scaling of GPU Streaming Multiprocessor in Spatially Shared Real-Time Embedded GPU Systems

Nvidia GPUs offer various concurrency mechanisms that influence how tasks are scheduled [21]. Previous studies (e.g., [21, 12, 9]) indicate that task behavior is significantly affected by whether tasks are launched from separate processes or from a single process. When tasks are launched from separate processes, they are by default allocated time slices. However, when tasks are launched using Nvidia’s Multi-Process Service (MPS) from separate processes, they are shared spatially [12, 46, 48]. MPS is not used for real-time scheduling because it introduces substantial runtime overhead. MPS reassigns streaming multiprocessors (SMs) only upon task termination, resulting in unpredictable execution delays and violating the timing determinism required in real-time environments [12, 17]. When tasks are launched from a single process, they share a common GPU context and are typically issued into CUDA streams. In this case, kernels may be spatially shared across available SMs, enabling concurrent execution. However, this spatial sharing is opportunistic and not strictly partitioned; kernels may still interfere with one another depending on scheduling behavior and resource contention [40]. To mitigate the contention, Bakita et al. have proposed a hardware-level compute partitioning mechanism that enables true spatial sharing of GPU cores [8]. This approach leverages undocumented mask fields in the kernel’s task metadata (TMD) structure to restrict execution to a specified subset of compute units (e.g., TPCs/SMs). By activating these mask fields prior to kernel submission, the hardware scheduler is constrained to dispatch thread blocks only within the assigned partition. As a result, tasks assigned to separate partitions can execute simultaneously with bounded interference, improving predictability even in the presence of long-running or compute-saturating kernels.

Although spatial sharing can be achieved either through MPS enabled multi-process execution or through single-process stream-based concurrency, we adopt the single-process approach. This design choice mitigates overheads associated with MPS, removes reliance on hardware capabilities that are absent on a substantial fraction of edge GPUs [46, 56], and prevents latency introduced by SM reallocation occurring only upon task completion.

When tasks are launched from the same process, spatial sharing can be explicitly controlled using two primary approaches: (1) persistent threads [60][56] and (2) hardware-level SM masking (via libsmctrl) [8]. Persistent thread technique restructures kernels so that a fixed number of thread blocks remain resident on designated SMs and cooperatively schedules work in software, thereby emulating application-level partitioning. However, this approach requires kernel modification, introduces additional synchronization overhead, and reduces portability across different GPU architectures and programming frameworks.

In contrast, libsmctrl enables spatial partitioning by directly applying hardware-supported TPC masks at launch time, without modifying the application kernels. By configuring the TPC mask fields in the kernel’s task metadata, the hardware scheduler is constrained to dispatch thread blocks only within a specified subset of compute units. Since TPCs represent the native hardware grouping of SM resources, masking at the TPC level aligns with the GPU’s dispatch hierarchy and provides deterministic spatial isolation. This mechanism preserves CUDA scheduling semantics and avoids persistent-thread overhead. Beginning with the Pascal architecture, each Nvidia GPU contains two SMs per TPC; therefore, controlling the number of active TPCs provides slightly coarser-grained yet effective control over SM allocation. In this work, we leverage TPC masking using libsmctrl as the actuator for our scheduler. We use TPC masking to dynamically scale the number of SMs assigned to each task (referred to as SM Scaling).

Once a kernel corresponding to a task instance is launched in a stream with a fixed TPC mask, it executes non-preemptively on the assigned SMs until completion [24, 46]. The assigned partition configuration remains unchanged during kernel execution. Instead, SM reassignment is deferred until the subsequent job release and is implemented by updating the TPC mask before launching the next kernel instance. This reconfiguration incurs negligible overhead, as it only requires modifying mask fields in the kernel’s launch metadata on the CPU side and does not involve reinitialization. When a kernel completes, all per-block state (e.g., registers, shared memory, and warp context) is automatically released by the hardware. Therefore, reassigning those TPCs to another task does not require state migration, context switching, or data movement. Consequently, partition resizing at task boundaries introduces minimal runtime overhead while preserving timing predictability.

To empirically demonstrate the effectiveness of SM scaling, we conduct an experiment on an RTX $3090$ GPU. In this experiment, we vary the number of TPCs assigned to all workloads described in Section 5 and measure their execution times. These workloads (e.g., from Rodinia and CUDA samples) have been widely used in prior real-time and GPU scheduling research [55, 57, 49, 8, 39], and represent a diverse set of compute and memory-bound kernels that are representative of practical embedded and AI workloads.

As shown in Figure 1, increasing the number of allocated TPCs reduces execution time within the scalable region of each workload. However, beyond a certain point, performance flattens because the workload cannot be further parallelized. This saturation occurs due to limited parallelism (i.e., a non-zero serial fraction), finite thread-block availability, and memory bandwidth constraints. As a result, additional TPCs provide diminishing or negligible improvement once the parallel portion is fully exploited. We define the effective scaling region of a workload as the range of TPC allocations over which increasing the number of TPCs yields a non-negligible performance improvement. Specifically, we identify this region empirically as the set of allocations for which the marginal speedup exceeds a small threshold (e.g., $5\text{–}10\%$). Outside this region, additional TPCs provide diminishing returns, reducing the effectiveness of SM scaling for response-time control.

In real-time systems, a task is implemented as a process or thread that is scheduled to run on the target GPU during specific time intervals. Each task has multiple instances, known as jobs. A job’s release time is the point at which the job becomes ready to execute on the GPU, and the system can begin scheduling and running it. In this paper, to simplify the design, we assume all tasks are periodic with implicit deadlines (i.e., the relative deadline equals the period). This assumption is common in embedded and real-time systems, where periodic tasks interact with the environment, such as sensors or cameras [43]. If a job does not complete within its period, the next job is released, and the current job becomes outdated. For example, in autonomous driving systems, tasks such as sensor data processing, decision-making, and vehicle control run periodically to ensure safe and efficient operation [29].

We now introduce the notations used throughout the paper. A system consists of $N$ tasks ($t_i$, $1\le i\le N$), where each task $t_i$ is defined by the following parameters:

• $\mathrm{■}$

  $e_i(k)$: Kernel execution time of task $t_i$ in the $k^{th}$ control period.

• $\mathrm{■}$

  $e_{hdi}$: Host-to-device copy time of task $t_i$.

• $\mathrm{■}$

  $e_{dhi}$: Device-to-host copy time of task $t_i$.

• $\mathrm{■}$

  $s_{max}$: Maximum number of SMs in the GPU.

• $\mathrm{■}$

  $s_i(k)$: The number of SMs assigned to every job of $t_i$ in the $k^{th}$ control period.

• $\mathrm{■}$

  $u_i(k)$ = $s_i(k)/s_{max}$ = normalized allocation (fraction in (0,1]).

• $\mathrm{■}$

  $p_i$: The period of $t_i$, which is a constant without rate adaptation.

• $\mathrm{■}$

  $d_i$: The relative deadline of $t_i$ (assuming $d_i=p_i$).

• $\mathrm{■}$

  $q_i(k)$ = Average response time of all the jobs of $t_i$ that are finished in the $k^{th}$ control period. A job’s response time is its finish time minus its release time.

• $\mathrm{■}$

  $r_i(k)$ = $q_i(k)/p_i(k)$: Relative response time (RRT) of $t_i$ in the $k^{th}$ control period.

• $\mathrm{■}$

  $z_i$: Relative response time set point for $t_i$.

Our task model accommodates a wide range of periodic tasks, represented as $t(e{t}_i,p_i)$, where $e{t}_i$ is the estimated GPU execution time and $p_i$ is the task period. The estimated execution time is $e{t}_i(k)=\frac{e_i(k)}{s_i(k)}+e_{hdi}+e_{dhi}$, where $e_i(k)$ is the kernel execution time, and $e_{hdi}$ and $e_{dhi}$ are the transfer times from host to device and device to host. We assume that $e_{hdi}$ and $e_{dhi}$ depend only on the size of the data and remain constant across jobs and SM allocations. The kernel execution time scales inversely with the number of SM’s. As the number of SM’s decreases, the kernel execution time increases [8][56]. We assume that the total computational demand of all tasks is within the available GPU capacity, i.e., the system is not persistently overloaded. This is a standard assumption in real-time scheduling, as no scheduling or control algorithm can guarantee deadline satisfaction when the aggregate demand exceeds available resources. Furthermore, tasks are assumed to execute independently, with no precedence constraints or shared data dependencies.

In this section, we provide a high-level description of the DySM system architecture. As discussed above, a significant concern in real-time GPU scheduling is the risk of deadline violations. Thus, existing approaches tend to 1) perform offline analysis, where the response time of the task is theoretically computed as an upper bound and schedulability analysis is performed [60, 57, 56, 42], and 2) over-provision GPU resources for each task to ensure that the worst-case execution time (WCET) does not cause any deadline misses. Such practices result in low runtime adaptability [42], which is an issue in many real-time environments, where new tasks may arise from user applications, system events, or sensor inputs [7, 6]. For example, in autonomous vehicles, when an unexpected obstacle, such as a traffic cone, suddenly appears on the road, the vehicle must react immediately to prevent a collision. This event may trigger additional perception or path planning workloads that must execute with tight latency constraints. When this occurs, the system must rapidly reallocate GPU compute resources to accommodate the new task while ensuring that existing safety-critical tasks continue to meet their deadlines. Offline schedulability analysis assumes a fixed and known task set with predetermined resource allocations. When new tasks are introduced dynamically, the original analysis no longer holds because the compute partitioning assumed during analysis may no longer be sufficient. Similarly, over-provisioning resources for worst-case execution leaves little flexibility for reallocating streaming multiprocessors at runtime when unexpected workloads arise. Without the ability to dynamically adjust the number of SMs assigned to each task, the system may either violate deadlines or waste substantial computational capacity. Consequently, rigid schedulability guarantees based on static allocation can break down in highly dynamic environments, motivating the need for adaptive GPU compute partitioning mechanisms.

The overall architecture of DySM is shown in Figure 2. The framework employs a multi-input–multi-output (MIMO) feedback control loop that dynamically performs SM scaling to ensure that each task’s relative response time converges to its specified set point. For a task, a relative response time of $1$ indicates that the job finishes exactly at its deadline, whereas $r_i(k)>1$ implies a deadline miss. For example, with a relative response time set point of $0.9$ and a period of $2$ s, each job should finish at $1.8$ s.

A key feature of our task and system model is its ability to operate in unpredictable runtime environments, where task execution times and interference effects are not known a priori to the scheduler. Instead of relying on offline worst-case execution-time analysis, DySM adopts a closed-loop MIMO feedback control architecture that dynamically adapts GPU resource allocations at runtime to improve deadline adherence under runtime variability. As shown in Figure 2, each task submits its kernels to one or more CUDA streams for concurrent execution. Prior work [9] shows that Nvidia GPUs provide a limited number of compute channels per CUDA context (eight by default on Ampere GPUs). When the number of active streams exceeds the available channels, implicit serialization may occur due to channel sharing. Therefore, we limit the number of concurrent periodic tasks to the available compute channels and assign each task to a dedicated CUDA stream to avoid unintended intra-context scheduling interference. We assume that each task is mapped to a single CUDA stream. Furthermore, we apply a consistent SM/TPC mask across all kernels within a task, ensuring a fixed resource partition throughout execution. Extending the approach to tasks using multiple streams would require additional coordination in the control interface and is left for future work.

In Figure 2, the control loop is invoked periodically at each sampling instant and consists of three main components: a Response Time Monitor, an MIMO Controller, and an Actuator. The control loop operates over a fixed control period, a configurable parameter that need not be an integer multiple of the task period. During each control period, the response-time monitor measures each task’s response time $q_i(k)$ and computes the corresponding relative response time $r_i(k)$. To ensure stable and representative measurements, the control period is chosen to be sufficiently long to include multiple job completions per task. This averaging mitigates transient variations and provides a reliable estimate of task response time for feedback control. The MIMO controller then collects the $r_i(k)$ values of all tasks, compares them against their respective set points $z_i$, and determines the required control actions. Because the GPU’s total SM budget is fixed, allocating additional SMs to one task necessarily reduces the resources available to others. This shared resource constraint introduces intrinsic coupling among tasks, as their response times depend on the same limited pool of SMs. Consequently, independent per-task controllers would ignore this interdependence and may lead to instability or inefficient allocation. Therefore, we adopt a MIMO controller to jointly regulate all task allocations and maintain system-level stability. Based on the controller’s output, the actuator adjusts each task’s compute allocation by updating the corresponding TPC masks (which quantize $s_i(k)$ to the hardware’s TPC/granularity), thereby regulating the task’s response time. In the following subsections, we formally formulate the control problem and present the detailed design of the MIMO controller.

The control problem can be formulated as a constrained optimization problem, as:

where $u_i(k)\triangleq s_i(k)/s_{\max }$ and $u_{\min }=1/s_{\max }$. The objective of the optimization is to minimize the deviation between the desired set point $z_i$ and the relative response time $r_i(k)$. By adjusting the available SM allocations, the controller regulates GPU load and ensures that tasks meet timing constraints when an appropriate set point (e.g., $z_i=0.9$) is selected. Rather than using a traditional offline method to solve this optimization problem, we employ a control-theoretic approach with a closed-loop feedback mechanism. Compared to offline optimization, a control-theoretic solution can dynamically adapt to runtime variations such as workload fluctuations, execution-time variability, and modeling inaccuracies. In this work, we assume the SM budget is typically tight; therefore, for controller derivation, we consider the constraint . This reflects the normalized GPU resource budget, where $u_i(k)$ denotes the fraction of SMs allocated to task $t_i$. This constraint enforces that the total allocation does not exceed the available SM capacity and ensures that the controller redistributes a fixed resource budget among tasks. At each sampling interval, the controller measures the system’s current state and adjusts the number of SMs accordingly. This closed-loop design provides inherent robustness against performance degradation caused by unpredictable execution behavior (see Section 6).

To apply a control-theoretic methodology, we establish a dynamic model that relates the manipulated variables (i.e., normalized SM allocation to every task) to the controlled variables (i.e., relative response time of every task). Our control objective is to manipulate SM allocations so that the relative response time of every task converges to its set point within a finite number of control periods. We define the manipulated variable for task $t_i$ as

where $s_i(k)$ is the number of SMs assigned to task $t_i$ in control period $k$ and $u_{\min }=1/s_{\max }$ and we define $r_i(k)$ as the controlled variable.

We first model the response time of task $t_i$ as $q_i(k)$. In Section 2.2, we have discussed that the response time of a task varies inversely with the number of SMs assigned to the task¹¹1In general, GPU kernels may exhibit non-linear speedup as compute resources increase (e.g., due to memory bandwidth limits, cache effects, or occupancy saturation) [56]. Our model, therefore, serves as a local approximation around the operating region; incorporating explicit non-linear speedup models is left for future work.. For a system comprising $N$ tasks, we first obtain the response-time model for each task $t_i$ via offline profiling within the operating region. We measure the response time of each task $t_i$ under different SM allocations and fit the resultant data to an inverse model

where $s_i(k-1)$ represents the number of SMs assigned to task $t_i$ during the previous control period. The parameters $a_i$ and $b_i$ are determined through least-squares curve fitting. To create a generalized model, we fit the data across all workloads presented in Figure 1 and average the coefficients. This process yields generalized curve-fitting parameters ($a_i$ and $b_i$) that represent the scalable component and the allocation-independent offset, respectively. Using the normalized allocation in Equation (2), we substitute $s_i(k)=u_i(k)\times s_{\max }$ to obtain

Defining $e_i\triangleq \frac{a_i}{s_{\max }}$, the model becomes

In Section 3, we define the relative response time ($r_i=\frac{q_i}{p_i}$) of a task as the ratio of its response time to period. Equation (5) can be converted as follows

The SMs are shared across tasks and must satisfy the global budget constraint stated in the problem formulation in Section 3 (${\sum }_{i=1}^N{u}_i(k)\le 1$), where $u_i(k)=s_i(k)/s_{\max }$ denotes the normalized SM allocation for task $t_i$. This shared constraint induces coupling among tasks, as increasing the allocation of one task reduces the resources available to others.

To facilitate controller design, we next derive a linear discrete-time incremental model by linearizing the $r_i$ expression around the current operating point. Specifically for a system comprising $N$ tasks, we write the incremental form for each task as

where

and $\gamma \in [0,1]$ is a coupling coefficient that captures the strength of interaction induced by the shared SM budget. Stacking all tasks, we obtain the coupled MIMO model as:

where $\mathrm{\Delta }𝐮(k)={[\mathrm{\Delta }{u}_1(k),\mathrm{\dots },\mathrm{\Delta }{u}_N(k)]}^{𝖳}$ and

Independent SISO controllers assume $r_i=f_i(u_i)$ and adjust $u_i$ without accounting for the shared budget. As a result, multiple controllers may issue conflicting control actions, leading to oscillations, saturation, or inefficient resource allocation (see Section 6.1). Hence, the shared SM constraint renders independent SISO control inadequate, necessitating a MIMO formulation that jointly regulates the RRT of all tasks. The analytical model assumes non-overlapping spatial partitions and captures coupling induced by the shared SM budget. In practice, when the requested allocation is infeasible, the actuator may allow bounded overlap as a fallback to maintain progress; we treat this as an implementation-level feasibility mechanism and empirically evaluate its impact (See Section 5).

We present the design and analysis of our MIMO controller DySM. We first derive the state representation of the model. Next, this formulation is transformed into a closed-loop solution, which allows us to design the control algorithm.

A key advantage of control-theoretic design is its ability to provide analytical guarantees on system stability and control accuracy, even in the presence of modeling errors or runtime variations. In this section, we analyze the stability of the proposed MIMO proportional controller based on the discrete-time RRT system model derived in Section 4. We list the steps below for performing stability analysis.

1. 1.

   Compute the control input $\mathrm{\Delta }𝐮(k)$ using the nominal system dynamics defined in Equation (11). Note that the nominal system model is used for our controller design and can be different from the actual system model due to modeling errors or variations.

2. 2.

   Substitute the resulting control law into the actual system model (which may have different system parameters from actual measurements) to obtain the corresponding closed-loop model. This model represents the response of the actual system to the control input derived from the nominal system model.

3. 3.

   Analyze the stability of the closed-loop system by evaluating its eigenvalues; the system is stable if all eigenvalues lie within the unit circle, despite modeling errors or variations.

As example, we consider a system comprising $N$ tasks, where the matrices in the state-space model (11) have the following dimensions: $𝐁\in {ℝ}^{N\times N}$, and $𝐱(k)\in {ℝ}^N$, $\mathrm{\Delta }𝐮(k)\in {ℝ}^N$. From Equation (9), the discrete-time system dynamics are given by

where $𝐁\in {ℝ}^{N\times N}$ is the gain matrix defined in Equation (10), and $\mathrm{\Delta }𝐮(k)$ denotes the increment in the independent normalized SM allocation vector. We define the system state as $𝐱(k)=𝐫(k)$, and the system output as $𝐲(k)=𝐱(k)$. The control objective is to regulate $𝐱(k)$ to a desired reference vector $𝐳$. We employ a discrete-time proportional MIMO control law of the form

where ${𝐊}_p\in {ℝ}^{N\times N}$ is the proportional gain matrix. Substituting Equation (15) into Equation (14) yields the closed-loop system

The closed-loop state transition matrix is $𝐅=𝐈-{\mathrm{𝐁𝐊}}_p.$ As long as the proportional gain matrix ${𝐊}_p$ is chosen such that the eigenvalues of $𝐈-{\mathrm{𝐁𝐊}}_p$ lie within the unit circle, the closed-loop system remains stable. In practice, this implies that the relative response times of all tasks converge to their respective set points without sustained oscillations or divergence, ensuring robust and bounded behavior even in the presence of modeling inaccuracies and runtime variations, without requiring an exact system model. Due to space limitations, the stability range can be derived by following the standard steps listed in [35, 26, 49].

Real-time systems often experience workload fluctuations due to dynamic changes in the surrounding environment [22]. For example, in autonomous driving, an intersection with many traffic lights/signs and surrounding vehicles can significantly increase computational demand [23]. In this experiment, we emulate such real-world workload variations to compare DySM with the three baselines introduced in Section 5. We choose two periodic tasks mm ($29ms$, $45ms$) and stereodisparity ($2.4ms$, $21ms$). At the control period $80$, we increase the input to mm. When the GPU’s computational demands increase dynamically, the controller needs to play a crucial role in adjusting the per-task SM/TPC allocation to maintain consistent real-time performance. We discuss the result of each scheduling solution below.

1. 1)

   SMshare is an open-loop baseline that assigns an equal number of SMs to all tasks and does not adapt allocations at runtime [2]. Specifically, if a system has $N$ tasks and $K$ SMs, each task is assigned $\frac{K}{N}$ SMs, and all $K$ SMs are fully utilized under this equal partitioning. In this example, since there are two tasks SMshare assigns equal SMs to both tasks as shown in Figure 3(c). As shown in Figure 3(a), we observe that the RRT remains below $1$ until control period $80$. Moreover, the RRT is around $0.4$, indicating that the system is over-provisioned (i.e., tasks complete well before their deadlines, leaving GPU resources unused). However, at control period $80$, when the input to mm increases, the RRT exceeds $1$, as shown in Figure 3(a), resulting in deadline misses. Because SMshare uses a fixed allocation and does not perform runtime adaptation, it cannot adjust the number of SMs in response to changing workload conditions. Consequently, it fails to maintain deadlines under increased load, leading to the deadline misses observed in Figure 3(b). We next compare it with STGM.

2. 2)

   STGM is also a state-of-the-art open-loop baseline. We compute the number of SMs required for each task using the algorithm proposed in [42], based on the execution times of the tasks. Following this method, STGM allocates $18$ SMs to mm and 13 SMs to stereodisparity as shown in Figure 3(f), ensuring that both tasks meet their deadlines. As shown in Figure 3(d), STGM maintains a RRT less than $1$, resulting in zero deadline misses. Compared to SMshare, STGM assigns fewer SMs overall (i.e., it avoids equal partitioning across all available SMs) and instead provisions only the minimum number required to satisfy timing constraints. Consequently, the task response times are closer to their respective periods, ensuring tighter resource provisioning with reduced slack. However, because STGM does not adapt SM allocations at runtime, it cannot respond to workload increases. After control period $80$, when the input to mm increases, the RRT exceeds $1$, leading to $100\%$ deadline misses, as shown in Figure 3(e). The results of SMShare and STGM demonstrate that open-loop GPU scheduling solutions cannot adapt to online workload variations. Thus, closed-loop adaptive solutions must be designed.

3. 3)

   Adhoc is a heuristic-based closed-loop solution, as introduced before. In Adhoc, each task is managed by a separate controller, so there are $N$ independent controllers for $N$ tasks. At every control period, if the observed RRT is below the set point, the controller of that task decreases the number of SMs allocated to the task by a fixed step size; if the RRT exceeds the set point, it increases the allocation by the same step size as shown in Figure 3(i). Because each task has its own independent Adhoc controller, actuation decisions are made locally without global coordination. As a result, when adjusting SM allocations, the controllers do not account for overlap in SM partitions across tasks. Consequently, multiple tasks may be assigned overlapping SM sets, leading to contention and unpredictable interference. While the Adhoc controller is simple to implement, its reliance on a fixed step size introduces inherent limitations. If the step size is too large (e.g., 4 TPCs or 8 SMs out of the total available SMs), the controller may overreact to small deviations in RRT, leading to oscillations around the set point. If it is too small (e.g., 1 TPC), convergence becomes slow and may fail to respond promptly to workload changes. Without control theory as a foundation, determining an appropriate step size typically requires extensive offline profiling for each task configuration and GPU platform. In this work, we have experimentally evaluated different step sizes and selected $5$ SMs because it provides a good trade-off between convergence speed and oscillation. However, because the step size is fixed and not derived from system dynamics, it can remain suboptimal when workload characteristics or hardware behavior change, leading to performance variability. As shown in Figure 3(g), because the step size is mainly tuned to make mm have faster response when its input increases at control period 80, and then have small oscillation after 100, this step size causes stereodisparity to have much larger oscillations. As a result, Figure 3(h) shows that such oscillations lead to significant deadline misses for stereodisparity. A key limitation of solutions like Adhoc is that it is commonly hard to find a step size that works well in every case. In addition, the SM/TPC overlap caused by per-task independent controllers can also cause Adhoc to have longer response times. We analyze this reason in more detail in the next subsection.

4. 4)

   DySM adapts promptly to changes in workload at runtime and maintains the RRT close to the desired set point, as shown in Figure 3(j). The precise control of DySM is mainly due to 1) its control-theoretic design provides the right (fractional) SM allocation, and 2) the delta-sigma modulator to approximate this fractional allocation, as discussed in Section 4.2. At control period $80$, when the workload increases, DySM adjusts the number of SMs using the MIMO controller described in Section 4. During actuation, DySM also prevents SM/TPC overlap through coordinated control decisions, which results in fewer deadline misses. As shown in Figure 3(k), the deadline miss rate remains below $1\%$. DySM achieves this by dynamic SM scaling as shown in Figure 3(l) at runtime while preserving isolation among tasks. The overhead of dynamic SM scaling is minimal, as discussed in subsection 6.4.

DySM outperforms Adhoc for two major reasons. First, Adhoc relies on a fixed step size while DySM reallocates SMs based on how far the measured response times deviate from their set points, with control theory as a foundation. Second, DySM is a MIMO controller that integrates the constraint on total available SMs and TPCs in its model, but Adhoc uses a set of independent controllers that make isolated decisions, which can result in overlapping TPCs among tasks and thus GPU resource contention and longer response times.

In this experiment, we examine how overlapping TPC allocations across concurrent tasks can impact task response times. To that end, we run two periodic GPU tasks implemented as tensor-computation kernels (dense 3-way tensor operations). Each task is released every $30$ ms from a dedicated CPU thread and launched on its own CUDA stream. Task $1$ is fixed to TPCs $0$–$7$, while Task $2$ initially executes on the disjoint range $8$–$15$. We then gradually slide Task $2$’s allocation leftward, increasing the number of shared TPCs from $0$ to $8$. At full overlap, both tasks are restricted to the identical set $0$–$7$. For each configuration, we execute $10$ releases and report the average response time.

Figure 4(a) illustrates this progressive overlap pattern, and Figures 4(b)–(c) show the corresponding response times and degradation trends. Performance degradation is computed relative to the disjoint baseline ($\text{ov}=0$), where the two tasks execute on non-overlapping TPC sets. Let $R{T}_i(0)$ denote the average response time of task $i$ measured in this disjoint configuration. The percentage degradation for task $i$ at overlap level ov is defined as ${\text{Degradation}}_i(\%)=\frac{R{T}_i(\text{ov})-R{T}_i(0)}{R{T}_i(0)}\times 100.$ As overlap increases, the response times of both tasks rise steadily. In the disjoint configuration ($\text{ov}=0$), both tasks exhibit nearly identical response times of approximately $39.8$ ms. When the overlap reaches its maximum ($\text{ov}=8$), the response time increases to $65.9$ ms for Task $1$ and $69.2$ ms for Task $2$, corresponding to degradations of approximately $65\%$ and $74\%$, respectively. Even modest overlap levels introduce noticeable slowdowns, and the degradation grows nonlinearly as the number of shared TPCs increases. This degradation occurs because overlapping TPC allocations introduce direct compute contention. When both kernels are eligible to execute on the same TPCs, the GPU hardware scheduler must multiplex thread blocks from both streams onto the shared compute partitions. Since CUDA kernels execute non-preemptively at the block level, blocks from competing kernels contend for the same SMs within overlapping TPCs, leading to increased queueing delays and extended kernel completion times. Similar effects have been observed in studies of hardware compute partitioning, where overlapping partitions lead to unpredictable scheduling behavior and inflated execution times due to contention at the TPC level [8]. Overall, these results demonstrate that even modest overlap in TPC assignments can substantially impact task response times, which is partially the reason for Adhoc’s inferior real-time performance.

Since open-loop solutions cannot handle runtime workload variations as discussed in Section 6.1, in this set of experiments, we evaluate the two closed-loop solutions, DySM and Adhoc, under set points ranging from $0.50$ to $0.80$. The task set consists of three tasks: projection (5.53ms, 39ms), histogram (4.5ms, 22ms), and stencil (5ms, 21ms). We measure the average RRT and standard deviation after the system reaches a steady state to mitigate the influence of any transient states at the beginning of each run.

Figure 5(a) shows that Adhoc cannot converge the tasks to their respective set points in many cases. This is because each task has its own controller that adjusts the number of SMs using a step size chosen offline. In addition, the overlapping TPCs caused by Adhoc increase task response times. As a result, Adhoc cannot precisely converge to the right set points. In contrast, DySM drives the RRTs of all tasks much closer to their specified set points, as shown in Figure 5(a) with significantly smaller deviations. Figure 5(b) shows the corresponding deadline misses for DySM and Adhoc. As the set point increases, response times increase closer to their deadlines, leading to a small number of deadline misses for both solutions, but DySM has much lower miss ratios than Adhoc due to its better control of response times. For example, at set points $0.70$ and $0.80$, the Adhoc deadline miss ratio for histogram is $19.73\%$ and $37.6\%$. For DySM, the deadline miss ratio is just $0.12\%$ and $3.41\%$, respectively. Hence, at set point $0.80$, DySM reduces deadline misses by $90.93\%$ relative to Adhoc. Compared to other tasks, histogram has a relatively higher deadline miss ratio because its higher runtime fluctuations, as shown in Figure 5(a).

The DySM service necessarily introduces overhead. This overhead includes response-time monitoring, control computation, and enabling the TPC masks. DySM is viable only if this overhead remains sufficiently small relative to kernel execution time. To accurately measure overhead, we instrument the control service with timestamps at the entry and exit of each stage and report the corresponding execution times. As shown in prior studies [49], the overhead of the proportional MIMO controller varies linearly with the number of tasks; for a system with $10$ tasks, the overhead is less than $20\mu s$, and for the response time monitor, it is less than $2\mu s$. Hence, here we focus mainly on the overhead of dynamic SM scaling.

We quantify the overhead of dynamically updating the set of active TPCs/SMs at runtime using libsmctrl, and verify that such updates correctly constrain which SMs execute subsequent work. Since these CUDA kernels are not preemptible, a mask update cannot affect a kernel that is already executing; instead, it only influences kernels launched after the update. In our system, each kernel execution lasts on the order of a few milliseconds, while the control period is 1 second. Each task releases one job per control interval, and its execution completes well before the next control decision is applied. Once a kernel corresponding to a task instance is launched in a stream with a fixed TPC mask, it executes non-preemptively on the assigned SMs until completion [24, 46]. The assigned partition configuration remains unchanged during kernel execution. SM reassignment is deferred to the subsequent job release and is implemented by updating the TPC mask before launching the next kernel instance. Since kernel execution time (milliseconds) is much shorter than the control period (1 second), mask updates naturally take effect at job boundaries. As discussed in Section 2.2, the reconfiguration incurs negligible overhead, as it only modifies mask fields in the kernel launch metadata on the CPU side and does not require reinitialization.

Accordingly, our evaluation focuses on (i) the CPU-side overhead of issuing a mask update and (ii) the minimal delay required to observe its effect at the kernel dispatch boundary. We evaluate mask enforcement using a lightweight verification kernel. Each thread block reads its SM identifier (%smid) once and atomically increments a per-SM counter in global memory. The kernel has minimal computation, so that the measured latency is dominated by the mask update and kernel dispatch mechanisms rather than application-level computation. We launch at least 4096 blocks to ensure sufficient parallelism to use all enabled SMs. After kernel completion, the per-SM counters are copied back to the host, and the number of active SMs is determined by counting non-zero entries. We define a flip index as a control step at which the active SM mask is changed. At each flip index, DySM constructs a new bitmask of length $N_{\text{SM}}$, where $N_{\text{SM}}$ is the total number of SMs on the GPU (e.g., 82 SMs on the Nvidia RTX3090). A bit value of 0 enables the corresponding SM for subsequent kernel launches, while a value of 1 disables it. The mask is applied via libsmctrl prior to kernel dispatch. By varying the mask at successive flip indices, we explicitly control the number of SMs available to the workload and observe the resulting dispatch behavior.

In this experiment, we alternate between two configurations at successive flip indices. Specifically, the even-numbered flip indices activate $60$ SMs, while the odd-numbered flip indices activate $20$ SMs. By switching between these two configurations before each new kernel launch, we can explicitly control the number of SMs available to the workload and observe the resulting dispatch behavior. Figure 6 summarizes the overhead and effectiveness of runtime SM mask updates. Figure 6(a) shows the CPU-side latency of issuing a TPC/SM mask update via libsmctrl across successive flip indices. The update cost remains consistently in the sub-nanosecond range, demonstrating that mask reconfiguration introduces negligible control overhead. Minor variation across flip indices is attributable to normal system noise rather than the mask operation itself. Figure 6(b) reports the kernel execution latency required to observe the effect of a mask update at the next dispatch boundary. As expected, the update takes effect only for kernels launched after the flip, and the observed latency remains stable across flips, indicating that the mask update does not introduce additional scheduling or dispatch delays beyond the unavoidable kernel launch boundary. Figure 6(c) validates correct enforcement of the SM mask. The number of SMs that execute at least one thread block closely matches the requested allocation at each flip index, confirming that libsmctrl reliably constrains kernel execution to the enabled subset of SMs. Together, these results demonstrate that runtime SM masking can be applied with negligible overhead while providing precise control over SM-level resource allocation.