Enabling Containerisation of Distributed Applications with Real-Time Constraints

The Linux kernel implements five different schedulers, which are queried in order, thus implicitly implementing a first level of priorities among them. The stop_machine scheduler is queried first, which is used for kernel facilities only, and it is not available to the user. Secondly, the kernel queries the SCHED_DEADLINE scheduler, which provides resource reservation based on the CBS and EDF algorithms (extensively discussed in the following). Then, the fixed-priority schedulers follow, SCHED_FIFO and SCHED_RR, which only differ by the fact that equal priority tasks are executed to completion in SCHED_FIFO while they are assigned a time quantum in SCHED_RR (similarly to weighted round robin). The last one is the general purpose scheduler, implemented by the SCHED_OTHER scheduling class.

Most important to this work is SCHED_DEADLINE. It is a scheduler that implements the CBS resource reservation algorithm [11, 16] and is based on EDF scheduling, providing the assigned workload with the required CPU bandwidth and the worst-case CPU latency. SCHED_DEADLINE reservations enable timing isolation through a budgeting mechanism that relies on two key parameters that need to be configured for each reservation $r_i$: the reservation budget $Q_i$ (also called runtime) and the reservation period $P_i$. Essentially, SCHED_DEADLINE implements a resource partitioning mechanism by supplying $Q_i$ time units to the workload running inside the reservation during each period of $P_i$ time units. SCHED_DEADLINE functions as both a resource partitioning and a resource enforcement mechanism, ensuring that each workload receives no more than its guaranteed fraction of CPU capacity. This approach prevents a faulty or untrustworthy application within a reservation from impacting the timing behaviour of another application due to an overrun caused by a fault or a cyber-attack.

The theoretical properties of SCHED_DEADLINE ensure that, if the reservation servers are schedulable, i.e., if each reservation can actually receive $Q_i$ time units every period $P_i$, the workload running inside each reservation $r_i$ is provided with a fraction of the overall CPU bandwidth ${\alpha }_i$ and a bounded worst-case CPU delay ${\mathrm{\Delta }}_i$, which are a function of the budget $Q_i$ and the period $P_i$. For example, when the reservation is scheduled by only a single core (as under partitioned multiprocessor scheduling), ${\alpha }_i=Q_i/P_i$ and ${\mathrm{\Delta }}_i=2\cdot (P_i-Q_i)$ [53].

The workload running within the reservation can be general Linux processes, including QEMU/KVM virtual processors. A kernel patch to support hierarchical constant bandwidth servers [13] is instead required to manage containers. The patch implements a two-level scheduling hierarchy (named H-CBS [43] from now on). Hierarchical scheduling enables compositional analysis, the ability to analyse each container’s schedulability in isolation. This allows us to leverage existing analyses from real-time literature, avoiding the need to develop new analytical models from scratch. At the root of the hierarchy, the CBS as implemented by SCHED_DEADLINE schedules groups of tasks (named control groups, or cgroups, in the Linux jargon, recalled in the following). Then, a second-level scheduler based on fixed priorities (SCHED_FIFO or SCHED_RR) selects the highest-priority task in the cgroup. In more detail, the root scheduler creates $m_i$ identical CPU reservations (scheduled by SCHED_DEADLINE on $m_i$ different CPU cores) all with runtime $Q_i$ and reservation period $P_i$; this multi-core reservation is often indicated as $(Q_i,P_i,m_i)$. Note that there is no need to change the applications to be able to use SCHED_DEADLINE. Hence, the source code of the applications is not required [31, 13] (see [29] for more detail on how to configure these parameters). The reserved CPU time is shared among all the tasks included in the cgroup, and since containers are generally based on cgroups, this mechanism provides timing guarantees to containers. In other words, H-CBS allows assigning multiple computational activities to the same CPU reservation, while the vanilla SCHED_DEADLINE maps only a single task to each reservation.

The SCHED_OTHER [2] policy is the default scheduling policy used in Linux for general-purpose, non-real-time tasks, that is implemented through a fair-share scheduler (CFS or EEVDF [55], depending on the kernel version; note that CFS is a variant of the Start Fair Queuing (SFQ) [34] algorithm, designed to approximate fair queuing behaviour for general-purpose operating systems). SCHED_OTHER is designed to fairly divide CPU time among all runnable tasks by tracking how much processor time each has used and distributing time slices accordingly. The scheduler uses a red-black tree to track runnable processes, ensuring that the process with the least virtual runtime is always selected next. This design aims to give every task a fair share of CPU access over time, regardless of when it becomes runnable.

SCHED_OTHER does not offer guarantees about when or how quickly a task will execute. Instead, it ensures relative fairness, i.e., tasks are treated equally unless their nice value (a user-defined priority hint ranging from -20 to +19) suggests otherwise. A lower nice value increases a task’s likelihood of being scheduled sooner and more frequently, while a higher nice value means a lower priority for a task.

The scheduler dynamically adjusts task priorities based on their CPU usage and nice levels. While SCHED_OTHER tasks can preempt one another, they are always preempted by higher-priority real-time tasks, such as those scheduled using SCHED_DEADLINE (see Section 2.1). By design, SCHED_OTHER tasks are scheduled only with the leftover CPU capacity, after all real-time tasks have been served [8]. This makes SCHED_OTHER unsuitable for real-time workloads, especially under high system load or interference.

Control Groups (cgroups) is a Linux kernel feature that enables resource management by allocating, prioritising, and restricting system resources such as CPU, memory, I/O bandwidth, and network access among different processes or groups of processes.

Cgroups organise processes into hierarchical groups, applying resource constraints through control group files located in the cgroup filesystem. For real-time scheduling, cgroups expose several configuration files, including:

• $\mathrm{■}$

  cpu.rt_period_us – Defines the real-time scheduling period (in microseconds). For example, a value of 1000000 in the file, means the scheduling period is 1 second.

• $\mathrm{■}$

  cpu.rt_runtime_us – Specifies the maximum CPU time allocated for real-time tasks within a period. For example, the file contents can be a value of 500000, which means the task can run for 0.5 seconds per scheduling period. This file is used to apply a single runtime allocation, meaning that it is valid for all CPUs.

• $\mathrm{■}$

  cpuset.cpus – Defines the specific CPU cores assigned to a group. For example, the file contents can be 0-3,5, which means the process can run on CPU cores 0, 1, 2, 3, and 5.

In the new version of the real-time Linux kernel used in our system, another cgroup file related to real-time scheduling is introduced, aligning well with our resource allocation needs:

• $\mathrm{■}$

  cpu.rt_multi_runtime_us – Unlike cpu.rt_runtime_us, cpu.rt_multi_runtime_us specifies individual runtime values for each CPU. Different from cpu.rt_runtime_us, it allows to selectively allocate budgets on a subset of the physical cores.

  The file format for specifying cpu.rt_multi_runtime_us follows a space-separated list, where each value corresponds to the real-time runtime for a specific CPU core in the system. For example, on a 4-core system, if CPUs 0 and 2 are assigned 200 microseconds of runtime, while CPUs 1 and 3 have no real-time allocation, the file contents are: 200 0 200 0. We use this cgroup file for allocating runtime to applications.

While Kubernetes leverages specific kernel functionalities and mechanisms to implement containers, other components of the software stack run in user space. Kubernetes [3] executes containerised applications over a cluster of nodes divided between control and worker nodes. The former kind host the Kubernetes control plane, while the latter host containers in charge of effectively running the applications. The pod represents the basic execution unit and is formed by one or more containers, network connections, and storage. The Kubernetes architecture includes multiple microservices allocated on different cluster nodes.
Kube-scheduler [4] is the default scheduling component of Kubernetes and plays a central role in determining where newly created pods are deployed within a cluster. It operates on the control plane and continuously monitors the Kubernetes API server for pods that do not yet have a node assignment. Once it detects such a pod, it initiates a scheduling cycle that involves two major phases: filtering and scoring.

During the filtering phase, the scheduler evaluates each available node in the cluster to determine whether it meets the pod requirements. These include checks for sufficient CPU, memory, or other resource availability (e.g., a pod requesting 2 CPUs and 4GiB of memory will not be scheduled on a node with only 1 CPU or 2GiB free); compatibility with node and pod affinity/anti-affinity rules (e.g., a pod may require to be co-located with other pods running a specific microservice, or explicitly separated from them); toleration of taints applied to nodes (e.g., a node tainted with dedicated=gpu:NoSchedule only accepts pods with a matching toleration, ensuring it’s used exclusively for GPU workloads); and compliance with topology spread constraints (e.g., ensuring that pods of the same application are evenly distributed across availability zones to improve fault tolerance). Nodes that do not meet all required criteria are excluded from consideration.

In the scoring phase, the scheduler ranks the remaining eligible nodes by applying scoring functions. These are implemented as scoring plugins, each of which assigns a numeric score to every eligible node. The final decision is based on combined, weighted scores from multiple plugins. The node with the highest score is selected, and the scheduler communicates the decision to the API server by updating the pod specification to bind it to the chosen node.

The scheduler is highly extensible. Kubernetes allows users to define custom scheduling policies or even write custom schedulers. This extensibility makes it possible to support specialised workloads and deployment strategies that are not addressed by default heuristics.
Kubelet is the primary node-level agent and runs on every node in the Kubernetes cluster. Once a pod has been scheduled to a node, the kubelet is responsible for ensuring that the pod is actually created and maintained as specified. It retrieves the pod definition from the API server and communicates with the container runtime, typically via the Container Runtime Interface (CRI), to start and manage the container(s).

The kubelet performs several critical tasks beyond container creation. It monitors the health of running containers, restarting them if necessary according to the specified restart policy. It reports the node and pod status back to the API server, which is essential for observability and control decisions made by the control plane. Additionally, it enforces resource constraints, such as CPU and memory limits, using the CRI. Upon startup, the kubelet registers its node with the Kubernetes API server, advertising its available resources and any attached labels. This metadata is then used by the scheduler for pod placement.

Overall, the kube-scheduler and kubelet form a tightly integrated loop within the Kubernetes control and execution planes. The kube-scheduler makes placement decisions based on a global view of the cluster state, while the kubelet ensures local enforcement of those decisions and continuously manages the operational lifecycle of containers.

Ray [48] is an open source framework for building and scaling distributed applications, which is also popularly used for distributed training and inference of AI algorithms (used by OpenAI, Netflix, ByteDance, and Uber [58, 59, 9], among others). It consists of two fundamental classes of nodes: the head node, which is responsible for the coordination, scheduling, and management of a cluster of nodes, and a set of Ray worker nodes, which are in charge of performing the actual execution and sending the results back to the head node. We consider the KubeRay variant of Ray – an open-source project that integrates Ray with Kubernetes and simplifies the management of Ray clusters using the Kubernetes orchestration features. In this context, the challenge of leveraging SCHED_DEADLINE to schedule Ray workloads involves making KubeRay compatible with Kubernetes, which we address in Section 4. The interaction between Ray and Kubernetes when using KubeRay is graphically shown in Figure 1. In the context of Kubernetes, the Ray head node and Ray worker nodes are, in fact, pods. We use the term node to refer to a node in a Kubernetes cluster, and Ray head and Ray worker to refer to the head node and worker nodes in a Ray cluster, respectively. The Ray Head manages and distributes the workload among the Ray workers. To create the Ray head and workers, a YAML manifest with the required configuration for the Ray head and workers must be submitted through kubectl. The API server of KubeRay translates this to pod configurations and sends it to the kube-scheduler, the Kubernetes component responsible for selecting the appropriate Kubernetes cluster node to deploy pods. Kube-scheduler creates pod scheduling contexts using the configuration of the Ray head and workers to deploy pods.

To leverage the capabilities of SCHED_DEADLINE, various components as well as configuration parameters, need to be modified, as discussed in the following section.

The DRA framework was introduced to overcome limitations in how Kubernetes handles specialised hardware or non-standard resources. DRA provides a dynamic, extensible mechanism that enables Kubernetes to negotiate and allocate custom resources like FPGAs or GPUs during pod deployment. It is particularly useful for workloads that need fine-grained control over resource parameters that Kubernetes cannot natively express. That is, the Kubernetes DRA allows creating custom resource definitions (CRD) not originally supported by Kubernetes. To create a new resource type, we must define two key concepts:

• $\mathrm{■}$

  ResourceClass – Defines the custom resources that a container can request. It specifies how the parameters for a resource should be configured.

• $\mathrm{■}$

  ResourceClaim – Represents a request for a resource by a container. It refers to a specific ResourceClass and requests an instance of that resource for the container.

Kubernetes interacts with the resource driver through its two main components: the centralised controller and the kubelet-plugin. These components have specific predefined interfaces to maintain adaptability and compatibility with Kubernetes. These functions are designed to be general, meaning that their actual implementation heavily depends on the specific type of resource being integrated into Kubernetes.

The centralised controller selects the most suitable node within the Kubernetes cluster that can meet the container resource requirements and must allocate resources to claims.

The kubelet-plugin prepares the allocated resources by producing and forwarding the allocation information in the form of a Container Device Interface (CDI) – a standard mechanism that allows specifying device-specific parameters and configurations for containers – to the kubelet of the corresponding node. This component is specific to each node.

The centralised controller and kubelet-plugin communicate with each other through Node Allocation State (NAS) that maintains the information of the node, such as resources allocated to the claims. This component is specific to each node.

When a user submits a pod that includes a ResourceClaim, the kube-scheduler contacts the DRA centralised controller, as specified in the ResourceClaim, to find a suitable node. The controller checks each node’s NAS to see if the resource is available. Once a node is selected, the controller updates the NAS to record the allocation and allow the kubelet-plugin to access the information. The kubelet-plugin on the selected node reads the allocation state from the NAS and generates a CDI file, which is also recorded in the NAS. The kubelet then reads the CDI and passes it (currently in the form of pod annotations) to the container runtime so that the container can be created and access the requested resource.

We defined a ResourceClass for RT-DRA called rt.kubedeadline.io, which specifies the parameters for real-time CPU reservation and is registered with API group rt.resource.kubedeadline.io following DNS-style naming conventions in Kubernetes. A real-time ResourceClaim is a request for a specific RT-DRA resource and contains three attributes: count, runtime, and period, which denote the number of cores $m_i$, the budget $Q_i$, and the period $P_i$, respectively.

The configurations in Figures 4(a), 4(b), and 4(c) illustrate how a real-time container is deployed using RT-DRA. To request a real-time resource, designers must define a resourceClaim, as shown in Figure 4(a). In Figure 4(b), the ResourceClaimParameters define the scheduling parameters, including count, runtime, and period. Each ResourceClaimParameters must have a unique name (line 11, Figure 4(a)), which allows the corresponding ResourceClaim to reference it (line 4, Figure 4(b)). Here, the name is rtclaimparams. In this example, two cores are requested, each with a runtime of 100 and a period of 1000 time units.

The ResourceClaim must have a name and specify its associated ResourceClass by referencing resourceClassName, which should be set to rt.kubedeadline.io, as previously mentioned. Additionally, it defines parametersRef, which includes a reference (rtclaimparams) to the claim parameters – containing the requested scheduling parameters defined before – along with apiGroup, which is set to rt.resource.kubedeadline.io.

As shown in Figure 4(c), to deploy the real-time container, it must be encapsulated within a pod, as it is the smallest deployable unit in Kubernetes and containers cannot run outside the scope of a pod. Additionally, all resource claims must be referenced in the scope of the pod and then used by the containers. Here, the pod references the required ResourceClaim by name (rtclaim) and sets a name for this reference (rtcpu, line 9). The container inside the pod (ctr0) then associates itself with the claim by referencing the name rtcpu. Each container can have at most one claim, as shown in line 17.

Pods, resource claims, and parameters are defined in YAML format, stored together or separately, and deployed with the kubectl command.

The Node Allocation State (NAS) is in charge of maintaining the information state of each node and allowing the centralised controller and the kubelet-plugin to communicate. It stores the following data:

• $\mathrm{■}$

  Allocated claims: Maintains a mapping of CPU core IDs to resource claims. Each claim has a unique name and ID. The centralised controller writes this mapping to the NAS after an allocation, enabling the kubelet-plugin of the corresponding node to read it and generate the necessary CDI devices, which describes the necessary device-specify configurations of containers.

• $\mathrm{■}$

  Allocated utilisation: Tracks the current utilisation of each CPU core across all nodes. This value is updated by the centralised controller after every allocation or deallocation to reflect the real-time CPU usage across nodes. It stores essential data that is used to ensure a feasible allocation allowing to satisfy real-time constraints [41].

• $\mathrm{■}$

  Prepared claims: Stores the claim IDs that have been prepared by the kubelet-plugin and are ready for allocation, or removes the claim IDs that are unprepared.

Next, we describe the behaviour of RT-DRA during the deployment of a pod requesting real-time resources (e.g., the pod in Figure 4(b)), as described in Algorithm 1 and as shown in Figure 3. Each step of the algorithm describes the functionality of a main component in RT-DRA (centralised controller and kubelet-plugin). Components of RT-DRA communicate with different components in Kubernetes. Hence, the described procedure must comply with Kubernetes DRA specifications to ensure compatibility with Kubernetes without requiring any modifications to the core functionality of Kubernetes itself.

After the CDI devices are prepared by the RT-DRA and sent to the kubelet, the kubelet forwards the CDI devices to containerd without making any changes. Until Kubernetes receives the CDI devices, the pod stays in the pending state.

containerd supports Container Device Interface (CDI) devices; however, it lacks the capability to decode CDI devices generated by RT-DRA. To address this, we modified containerd to properly parse these CDI devices and extract the real-time scheduling parameters, runtime, period, and number of cores. containerd already defines internal variables for the real-time scheduling parameters. Since these variables are natively supported, no modifications were necessary to their definitions. Instead, our modification enables containerd to extract the corresponding values from the CDI device metadata and assign them to these pre-existing variables.

We made a minimal modification to the containerd codebase, adding a total of 19 lines of code in a single file. These changes introduce a helper function that extracts the number of cores, runtime, and period values from the CDI device information embedded in the pod annotations by the kubelet. If containerd detects real-time scheduling parameters in the CDI, it invokes the helper function and applies the extracted values for real-time scheduling. Otherwise, it treats the container as a standard non-real-time workload and schedules it with SCHED_OTHER (the default, general purpose scheduler of Linux). By default, containerd transmits these parameters to runc during container deployment.

runc natively defines and accepts the real-time scheduling parameters, runtime, period, and number of cores from containerd. Additionally, runc supports the creation of cpu.rt_period_us, a cgroup file that specifies the real-time scheduling period.

However, runc does not currently support the creation of cpu.rt_multi_runtime_us, a feature available in real-time kernels that allows for more flexible real-time execution across multiple CPUs. To address this, we introduced a custom function in runc that uses the number of cores and runtime to generate and configure the cpu.rt_multi_runtime_us file. This enhancement¹¹1Modifications in runc extend beyond KubeDeadline, enabling support for general Linux cgroups (cpu.rt_multi_runtime_us); we plan to submit them for future runc versions. in runc allows creating cpu.rt_multi_runtime_us instead of cpu.rt_runtime_us when real-time scheduling parameters are present. Note that cpu.rt_runtime_us is not currently used by Kubernetes.

We modified approximately 116 lines of code across two files in runc. In one file, we introduced two small helper functions to read from and write to the cpu.rt_runtime_us values in the parent cgroups when a container is being created. This step is necessary to enable nested real-time (RT) cgroups, as the Linux kernel requires that a child cgroup runtime does not exceed runtime of its parent.

These helpers were integrated into the existing function that configures real-time cgroup parameters for the container (e.g., cpu.rt_period_us and cpu.rt_runtime_us). We updated this logic to additionally first invoke the functions that update the runtime of parent directories and then handle the cpu.rt_multi_runtime_us for the container. In a second file, we added a function to handle destroying the container, which is called from within the destroy function. This cleanup step identifies the container runtime and recursively removes it from all parent cgroups to avoid accumulated values in the runtime of parent directories.

After runc could successfully create the cgroup files, the resource allocation process is finalised, the pod $P$ is created and its state changes to running (see Figure 3 for the connection among the different components of KubeDeadline).

In the first set of experiments, we evaluate the effectiveness of our approach for sets of synthetic periodic real-time tasks with implicit deadlines. We generated 150 task sets with utilisation ranging from $U=0.6$ to $2.0$ and a number of tasks ranging from $4$ to $12$ using the RandFixedSum algorithm [30]. We assigned the scheduling parameters (budget, period, and number of assigned cores) based on the Multiprocessor Periodic Resource (MPR) real-time analysis [29] using the CARTS tool [50].

The MPR theory guarantees that these scheduling parameters allow respecting deadlines of all containerised tasks. Therefore, by showing that, under KubeDeadline, every task experiences response times always smaller than its period (implicit deadlines), we empirically verify the correctness of our solution. To summarise the response times of all tasks in a single figure, the normalised response time (response time divided by the task period) has been used: if the normalised response time is smaller than 1, then the deadline requirement is met; otherwise, it is violated.

Figure 6 displays the experimental Cumulative Distribution Function (CDF) derived by all the normalised response times of all the tasks of all the task sets (more than $275000$ samples). The KubeDeadline line represents the CDF of the normalised response times experienced with KubeDeadline when properly-designed reservations are used, while the “Vanilla” line represents the CDF of the normalised response times experienced with a standard Kubernetes. The figure shows that the KubeDeadline CDF reaches $1$ for a value of the normalised response time smaller than $1$ (0.9998), indicating that no deadline has been missed, while the Vanilla Kubernetes CDF reaches $1$ for a value of the normalised response time larger than $1$ (about $2.27$), indicating that a standard Kubernetes (using the standard SCHED_OTHER Linux scheduler) is not able to respect all the tasks’ deadlines.

As mentioned earlier, RT-DRA includes the two main components centralised controller and kubelet-plugin. Hence, we measured the timing overhead of each component in deploying a pod. We repeated this measurement over 20 pods, where each container claims one to four cores. The minimum, maximum, and average time overhead of centralised controller are 2 $ms$, 466 $ms$, and 48.5 $ms$, respectively, and for kubelet-plugin are 1.96 $ms$, 52.66 $ms$, and 6.2 $ms$. Histograms of the time overhead of the centralised controller and kubelet-plugin are shown in Figure 7(a) and Figure 7(b), respectively. Note that overhead occurs only during container deployment, not during application execution. Thus, it is deemed acceptable.

The kubelet-plugin performs a constant-time operation involving CDI data generation, resulting in $𝒪(1)$ complexity. In contrast, the centralised controller evaluates resource allocation across all cluster nodes and performs admission tests on each node, yielding a complexity of $𝒪(n\cdot c)$, where $n$ is the number of nodes and $c$ is the number of cores per node.

In this experiment, we focus on comparing, in the context of a case study, the performance of KubeRay extended to support SCHED_DEADLINE with the vanilla KubeRay, which schedules threads with the default SCHED_OTHER scheduling class of Linux, designed for general-purpose applications. The purpose of this experiment is to demonstrate the advantage of real-time containers over non-real-time containers under background workload, highlighting the added value of real-time support in Kubernetes and KubeRay.