PREEMPT-FaaS: Taming Orchestration Times in Latency-Sensitive Serverless Environments

K8s orchestrates containers within a cluster, which is a collection of nodes, i.e., a set of computational resources and storage units. Each node is either part of the control plane or a worker node. The control plane, which can be deployed on a single or multiple nodes, encompasses all Kubernetes internals necessary to properly manage the cluster. Worker nodes host K8s-managed workloads and offer computing and storage resources available to run scheduled containers. An overview of the K8s architecture is given in Fig. 2.

The smallest schedulable unit in K8s is called Pod, which is a set of one or more containers. A pod, along with all other main K8s resources, is considered an Object. Objects are stored in the K8s database, etcd, a key-value store. The vast majority of these resources are deployed in a K8s Namespace, which is an abstraction that isolates workloads and resource instances in the cluster. A K8s object is characterized by its desired and current states; K8s detects a change in the desired states and manages the cluster to match the current and desired states of each object. A manifest is a file used to represent an object in the cluster (often encoded in “.yaml”), which includes three sections: metadata, specification (spec, i.e., the desired state), and status (i.e., the current state). Users or software interacting with K8s can specify the desired state by editing the spec section and can only inspect its status. K8s main components are:

• $\mathrm{■}$

  The kube-apiserver is the central control-plane component. All internal K8s components communicate through it, and every request to or from the cluster is processed via the API server.

• $\mathrm{■}$

  The kube-controller-manager runs a collection of controllers, each responsible for a specific K8s resource. Controllers continuously reconcile the current state with the desired state through control loops. While core controllers are compiled into a single binary, K8s supports extensibility via custom controllers, which can be implemented in any programming language.

• $\mathrm{■}$

  The scheduler assigns pods to worker nodes based on resource availability and policy constraints. K8s allows deploying custom schedulers or extending the default scheduler through plugins.

• $\mathrm{■}$

  The kube-proxy, which configures nodes’ virtual networks. Installed on every node of the cluster, the proxy monitors the changes to Service objects and their endpoints, translating them into network rules inside the node.

• $\mathrm{■}$

  The kubelet is a node-level agent running on each node. It receives pod specifications from the control plane, instantiates the corresponding containers, and continuously reports their status to maintain an up-to-date cluster state.

K8s offers a high degree of customization, making it a flexible software solution for different use cases. In addition to plugins, custom controllers, and schedulers, K8s also offers the possibility of introducing new resources into the system. A Custom Resource Definition (CRD) is used to specify the structure of a new resource type; each instance of this type is called Custom Resource (CR). The CR is usually associated with a custom controller to introduce new logic in a K8s cluster [34].

K8s leverages two specific resources to allow users to deploy an application with multiple instances. A Deployment contains the application’s pod specification along its required number of instances. A Deployment is inherently associated with a ReplicaSet object that keeps track of current and desired number of replicas. The respective controllers ensure that the desired number of pods is always met when sufficient hardware resources are available. This also includes failover procedures when a pod or a node hosting it crashes. A labeling structure in the metadata section of the pods allows the controllers to match Deployments to their respective pods. A Service offers a common endpoint to reach the application’s pods; it can also be seen as a load balancer for a specific application, distributing service requests across all available instances. The same labeling structure is used to redirect traffic to the application’s pods [26].

Knative [20, 22] is one of the most popular serverless platforms [52] specifically designed to operate on top of Kubernetes. We focus on Knative Serving, an HTTP-triggered autoscaling system that handles the lifecycle and scaling of stateless HTTP services. By using K8s resources, such as Services and Deployments, Knative enables users to deploy applications as a Knative Service (KSVC), which is constantly monitored by Knative components to determine, based on the actual request rate, the number of pods necessary to properly serve the workload. Knative Serving (hereon referred to as Knative for simplicity) is characterized by different resources. Knative Services are the first resource in the hierarchy for an application and control the lifecycle of the other resources. Revisions are immutable snapshots of the application’s code and configuration, representing specific versions of it; multiple revisions can coexist, and traffic can be explicitly split among them. Each revision is assigned to a K8s Deployment. Configurations represent the desired state of the applications; each configuration update creates a new revision. Routes map endpoints to one or more revisions. Knative main components are:

• $\mathrm{■}$

  A queue proxy, a container deployed alongside the application in each KSVC pod; it collects and forwards requests and replies to/from the outside. It also collects application metrics and sends them to the Knative autoscaler to determine the necessary number of pods at any given time.

• $\mathrm{■}$

  The activator queues incoming requests when the application is scaled to zero. Upon receiving requests, it signals the autoscaler to scale up the application. Once endpoints are reachable, it forwards the queued requests to them. By default, requests are routed directly to active endpoints, but the activator can be configured to always remain in the request path.

• $\mathrm{■}$

  The autoscaler is responsible for scaling the application up and down, patching revisions’ Deployments, based on incoming requests, internal metrics, and application configuration (e.g., the single-instance level of parallelism drives scaling decisions). The “concurrency” parameter in the KSVC specification can be set to communicate to the autoscaler the single-instance level of parallelism, a configuration that drives the scaling logic of the service. The autoscaler can be customized, and scale-to-zero can be either enabled or disabled. It can leverage the Horizontal Pod Autoscaler (HPA) without the scale-to-zero option, or use the default Knative Pod Autoscaler (KPA) with scale-to-zero enabled. Custom autoscalers can also be employed.

The service request flow and the main components involved are shown in Fig. 3. The typical Knative operational steps are as follows:

1. 1.

   The application is deployed as a KSVC; all related resources are created, including a Deployment and a ReplicaSet per revision.

2. 2.

   An initial application instance is created to verify that the service and its endpoints are healthy; the instance is dropped if no load is detected according to the configured scale-down time window.

3. 3.

   The activator is enabled to buffer new service requests until the required pods are created.

4. 4.

   When new requests arrive, a scale-up operation is triggered.

5. 5.

   Once sufficient pods are ready to respond to requests, the activator forwards the queued requests to the active instances and is disabled.

6. 6.

   The autoscaler scales the application as long as requests continue to be received.

7. 7.

   When no requests are detected within the scale-down time window, the application scales down to 0 instances, the activator is enabled, initial conditions are restored, and Knative awaits new traffic.

In this paper, we address the operation between steps 1 and 2 to apply scaling decisions to custom resources monitored by the PREEMPT-FaaS controller in order to reduce orchestration times for critical services.

The architecture of the PREEMPT-FaaS controller is depicted in Fig. 4. PREEMPT-FaaS preserves the “single queue, multiple servers” paradigm, but, unlike the Vanilla K8s controller, we implement a POSIX multi-priority queue to ensure immediate processing of the high-priority orchestration events. To isolate critical applications from standard workloads, we define a new Custom Resource Definition (CRD), RTResource, which serves as a Deployment-like abstraction enriched with a criticality level. The PREEMPT-FaaS controller exclusively monitors these resources and enforces priority-based orchestration through a preemptive scheduling model implemented at the kernel-thread level. The current implementation of PREEMPT-FaaS leverages SCHED_FIFO thread scheduling policy since our first choice is to manage applications with fixed priorities. We remark that the controller design can be modified to accommodate different scheduling policies. For instance, SCHED_DEADLINE can be employed to introduce a CPU budget control, or if developers want to serve events according to specific deadlines.

In this section, we first describe the RTResource abstraction and its structure, then detail the internal architecture of PREEMPT-FaaS, including its event processing model and scheduling configuration. Finally, we present the minimal modifications required to integrate PREEMPT-FaaS with Knative, enabling transparent switching between default and priority-aware orchestration.

Knative manages resources through direct interaction with the K8s, utilizing its Go codebase and K8s native libraries to manipulate both standard K8s objects and Knative resources. Originally designed to cooperate with Kubernetes Deployments, KSVC instantiation involves the following steps to enable dynamic scaling:

1. 1.

   The user deploys a KSVC.

2. 2.

   Knative creates a standard K8s Service to provide a unified domain name for the application.

3. 3.

   A Knative Configuration and the corresponding Revision are generated.

4. 4.

   Each Revision maps to a Deployment managed by the autoscaler, hosting the required replica count. This Deployment utilizes a pod template propagated from the KSVC.

5. 5.

   The pod template incorporates container information, including the mandatory queue-proxy container, with Knative-generated configuration to automate connectivity between the proxy and the main service.

The Knative metadata annotation validation procedure is modified to recognize a criticality annotation used by PREEMPT-FaaS:

autoscaling.knative.dev/application-criticality-level: <integer-value>

This addition enables seamless switching between Knative default behavior and PREEMPT-FaaS one, employing RTResources instead of standard Deployments. Furthermore, the necessary Go structures are introduced to allow Knative to create, patch, delete, and access RTResources.

A Revision is considered healthy once active endpoints are established, hence the default initialization to a single instance. Crucial to this process is monitoring the health state of the underlying Deployment and its ReplicaSet, and propagating their status to the Revision. This behavior is replicated for the RTResource; the Revision is deemed healthy when the corresponding CR reaches the Ready state.

During RTResource creation, most specifications are derived from the Revision propagated from the KSVC, with the criticality field extracted from the new annotation. Knative independently defines the pod specification, labels, namespaces, and selectors. Since network routing relies on pod labels, employing pod templating allows delegating this setup to Knative, thereby ensuring that PREEMPT-FaaS remains decoupled from KSVC-specific configurations. Finally, the Knative autoscaler was modified to apply scaling decisions directly to the “replicas” field of the CR when the criticality annotation is present.

The Knative patch (including all the mentioned changes) remains compatible with the Knative code generation pipeline, automating standard procedure implementations (e.g., resource finalizers and deep-copy functions). Additionally, Kubernetes authorization options were updated in the Knative installation manifests to grant its components full access to the RTResources API group. The patch required 600 lines of code (LoC), counting both additions and modifications, but excluding Knative auto-generated code and manifest changes.

To answer RQ1, we customized the real-traffic-test benchmark included in the Knative performance tests suite [14]. We compare Vanilla K8s and PREEMPT-FaaS to handle 10 concurrent KSVCs. We deploy these services, each configured with a maximum parallelism of $5$ requests per instance. This parameter is set according to a preliminary sensitivity analysis against our testbed to be sure to trigger scaling operations. To evaluate the prioritization effectiveness of PREEMPT-FaaS over non-critical service (stressload), each service is assigned a distinct priority level (1–10) via the corresponding annotation. The stressload is run with repeated bursts of creating/deleting a set of Deployments (with no priority set) or RTResources (each set to lower priorities than KSVCs)-depending on the controller under test, Vanilla K8s and PREEMPT-FaaS, respectively-each associated with a single pod. To prevent excessive load on the kube-apiserver, interfering bursts are released at 10-second intervals, as most scale-up events occur during the initial transition from a scaled-to-zero state.

The benchmark execution is repeated 10 times for each controller under test for statistical significance purposes. We vary the stressload intensity at 5, 10, and 20 requests of creation/deletion per burst. Each iteration lasts $60$s, during which the workload generator issues 50 RPS (requests per second) towards the 10 deployed KSVCs, with 5 RPS for each service (in a round-robin fashion) to ensure triggering scaling. We ensure that KSVCs are scaled to zero before starting the stress load.

Figure 6 reports the latency distributions for the key orchestration events, measured from the scale-up trigger to each corresponding control-plane stage. The “Starts Processing” distributions show that PREEMPT-FaaS achieves more stable and bounded latencies under all interference levels with a median reduction percentage of $94.4\%$ in the worst case. In contrast, the Vanilla K8s controller exhibits increasing variability as contention grows. Across all scenarios, PREEMPT-FaaS consistently maintains mean latencies below $0.2$ seconds. The “Pods Created” results follow a similar trend. While Vanilla K8s performs comparably under minimal interference, its latency and variability increase significantly as the orchestration load rises, with a median reduction percentage of $88.9\%$ in the worst case. Since pod creation requires repeated interactions with the kube-apiserver, sustained control-plane stress amplifies these delays. The slightly higher latency observed for PREEMPT-FaaS in some high-load cases can be attributed to kube-apiserver saturation, acting as a shared bottleneck, consistent with prior observations [8]. Finally, the “Pods Started” panel indicates that faster orchestration translates into earlier pod readiness when the kubelet is not overloaded. Although pod startup also depends on node-level mechanisms beyond our control, improved control-plane responsiveness accelerates the overall scale-up pipeline.

Furthermore, we repeated the analysis in the worst case (20 creation/deletion as stressload) for the PREEMPT-FaaS, limiting to 5 watchdog threads min/max to match the Vanilla K8s configuration. Table 2 shows that PREEMPT-FaaS outperforms Vanilla K8s in both mean orchestration latency and variance, even with thread scaling disabled. We do not increase the number of control loop threads in K8s (default is 5) since they work in a “single queue, multiple server scheme”, i.e., they pick requests from the same queue (one for each resource type). In [6], the authors demonstrated that increasing the parallelism for K8s control loop threads can lead to high instability.

To quantify the overhead introduced by the monitoring infrastructure, we measured service-level mean latencies with auditing enabled and disabled and 20 interfering resources for both PREEMPT-FaaS and Vanilla K8s controllers. For the Vanilla K8s controller, latency measurements followed a normal distribution with homogeneous variance; therefore, we applied a Student’s t-test ($\alpha =0.05$), which revealed no statistically significant difference attributable to the monitoring pipeline ($p=0.8874$). In contrast, latency samples for PREEMPT-FaaS did not satisfy normality assumptions. We thus employed a Mann–Whitney U test, which likewise showed no statistically significant degradation due to monitoring ($p=0.6232$).

In this section, we evaluate whether reducing orchestration latency translates into measurable improvements in end-to-end service behavior. To this end, we employ the widely adopted vSwarm serverless benchmark suite [55]. Unlike the Knative real-traffic-test benchmark, primarily designed to stress the control plane and measure orchestration dynamics, vSwarm provides realistic application-level workloads, ranging from simple functions to multi-component services, across multiple programming languages and runtimes. This diversity enables us to assess the impact of orchestration optimizations on throughput and latency under representative serverless scenarios. Moreover, vSwarm has been used in several recent studies [40, 46, 48], making it a suitable and reproducible baseline for evaluating application-level performance improvements. Traffic is generated via the default vSwarm invoker script, which provides request latencies and throughput when tests complete.

We chose RNN and Video Analytics benchmark services from the suite. RNN benchmark generates a meaningful text given a language, employing a PyTorch character-level vanilla Recurrent Neural Network (RNN) model. Instead, the video analytics benchmark preprocesses an input video and runs an object detection model (i.e., squeezenet, a lightweight 18-layer convolutional neural network designed for high efficiency in image classification) on a video stored on a MongoDB database. We deploy $5$ identical KSVCs on the designated worker node (see § 4.1), assigning each service the highest priority level under PREEMPT-FaaS. The services are executed alongside an orchestration interference workload consisting of continuous bursts of $15$, $30$, and $45$ resource creation/deletion events. Unlike the previous experiments, which employed a 10-seconds interval between bursts, the continuous interfering loads allows us to further stress the platform, even at lower interfering loads. The stress load is intentionally increased to drive the system into a regime where E2E metrics are noticeably affected by orchestration delays. A burst size of $45$ represents the maximum sustainable load for our cluster; beyond this point, we observe kube-apiserver instability that must be avoided, ensuring that observed performance differences reflect controller behavior rather than control-plane collapse. As such, these parameters should be calibrated per deployment environment to establish a stable interference baseline. We evaluate both the Vanilla K8s and PREEMPT-FaaS controllers under identical conditions, each subjected to their respective interfering resources (Deployments for Vanilla K8s, RTResources for PREEMPT-FaaS), with one pod per resource. The decision to deploy $5$ identical services, rather than a single instance, mitigates stochastic variations and ensures statistical significance. Since the kube-apiserver operates in a best-effort mode, even when processing and dispatching requests, a single scale-up event could be unrepresentative due to transient timing advantages.

We executed $10$ independent runs to ensure statistical significance. In each run, $5$ vSwarm invokers target their respective KSVCs for $30$ seconds at a constant rate of $50$ RPS for the RNN service and $12$ RPS for the video analytics service. A request timeout of $40$ seconds is configured to capture delayed responses without premature termination. Preliminary sensitivity experiments identified $50$ RPS for the RNN services as the scale-up threshold that consistently trigger Knative autoscaling, even when a pre-warmed instance is available. We emphasize that the goal of this evaluation is not to measure the maximum throughput capacity of the services themselves, but rather to assess how orchestration efficiency affects the achieved throughput and latency under controlled scaling conditions when comparing Vanilla K8s and PREEMPT-FaaS in cold-start conditions.

Figure 7 reports the sensitivity analysis of pod creation delays under increasing interference. For single-replica scale-ups, this metric corresponds to the initial control-plane reaction time in Vanilla K8s, and we report the equivalent stage for PREEMPT-FaaS for consistency. Results are aggregated across all KSVCs for each experimental run. Results confirm (as demonstrated in RQ1) that Vanilla K8s exhibits increasing latency and variability as interference grows, failing to maintain bounded orchestration times. In contrast, PREEMPT-FaaS preserves stable and predictable behavior across all stress levels. Although the data might initially appear inconsistent, specifically at lower stress loads, with the results presented in Figure 6, these discrepancies derive from three different experimental configurations. Firstly, the continuous interfering bursts allow us to efficiently stress the kube-apiserver, even when employing a low number of interfering resources. Secondly, the controllers for standard Deployments and RTResources, despite sharing the same overall logic, differ in the implementation of the respective control loops. For instance, whenever PREEMPT-FaaS processes a new event, it triggers an immediate interaction with the Kube-apiserver to update the state of the RTResource, ensuring low-latency synchronization with serverless platforms, e.g., Knative. The last critical factor involves service prioritization. Finally, we remark that while Figure 6 provides aggregated results across multiple scale-up events, Figure 7 focuses on individual scale-up operations per Knative Service (KSVC) within each iteration.

Figure 8 and Figure 9 show the improvements in E2E metrics achieved by a faster orchestration process performed by the PREEMPT-FaaS controller. Figure 8 illustrates the aggregate throughput achieved by the KSVCs, calculated as the sum of the actual RPS served by all deployed applications per experimental run. The results demonstrate a consistent performance gain for each level of interference. Specifically, the PREEMPT-FaaS controller brings a marginal but steady improvement of approximately 5 RPS. When integrated over the full test duration, this leads to a non-negligible increase in QoS. Unlike video analytics, the RNN benchmark does not show a consistent throughput decrease as the interfering load increases. This behavior is primarily attributed to the inherent instability of the invokers, which provide RPS that fluctuate around the nominal configured value. Furthermore, a load of $50$ RPS per KSVC represents a saturation point for these services. In contrast, the video analytics benchmark consistently handles the maximum achievable RPS per KSVC. For this reason, we selected $12$ RPS per KSVC to avoid unnecessary network congestion. Consequently, under lower interfering loads, Vanilla K8s is still able to outperform PREEMPT-FaaS.

However, the most significant contribution is observed in the service-level performance. Figure 9 reports the mean latency, averaged across all active services. The Cumulative Distribution Function (CDF) highlights the following improvements granted by our proposal: while the Vanilla K8s controller exhibits a distribution that shifts significantly toward higher latencies as the interfering load increases, the PREEMPT-FaaS controller maintains a nearly constant profile. At the highest stressload, the RNN benchmark shows for the Vanilla K8s controller a mean latency of $761.45ms$ at the 50th percentile and $7913.29ms$ at the 95th percentile. The respective values for PREEMPT-FaaS are $145.21ms$ and $5716.75ms$, showing an improvement of $81\%$ at 50th percentile and $28\%$ at the 95th percentile. For the video analytics benchmark, 50th and 95th percentiles are $20.58ms$ and $5616.86ms$ respectively for Vanilla K8s, while the same values for PREEMPT-FaaS are $23.37ms$ and $2500.57ms$. PREEMPT-FaaS shows an improvement of $55\%$ at the 95th percentile. This stability confirms that by minimizing orchestration overhead and ensuring service prioritization, the PREEMPT-FaaS controller directly enhances the end-to-end responsiveness of the system, regardless of control-plane congestion.

We evaluate the RNN benchmark in a pre-warmed configuration, with one active instance per KSVC and 10 runs per interference level. By exceeding the default autoscaling threshold, we trigger a scale-up from 1 to 2 replicas to assess the impact of orchestration efficiency on end-to-end (E2E) metrics. Figure 10 reports results for interfering loads of 15 and 30 events. Although experiments were also conducted at a stressload of 45 creation/deletion events, the maximum sustainable load of our cluster, the resulting control-plane and communication overhead led to unstable behavior; therefore, we focus on the two stable interference levels. The results indicate that a single replica can sustain the nominal request rate under moderate load. However, by accelerating the activation of the second replica, PREEMPT-FaaS reduces the time during which requests are queued on a single instance. This earlier load distribution mitigates transient congestion effects, leading to more stable throughput. While average latencies remain comparable, PREEMPT-FaaS exhibits lower variability in end-to-end throughput under contention.