FaaSLoad: Fine-Grained Performance and Resource Measurement for Function-As-a-Service

Authors of [33] studied the commercial platforms of Amazon Web Services Lambda, Microsoft Azure Functions, and Google Cloud Functions from the constrained point of view of customers, trying to understand the distributed architecture and finding limitations on performance and security. By nature, this work is limited by the need to reverse-engineer the inner workings of the platform (e.g., to accurately identify colocated function instances).

A lot of benchmark suites, with some companion tooling, have been published [24, 25, 35, 15, 31]. One most advanced example is ServerlessBench [35], but again, it focuses on the FaaS platforms by evaluating global function performance.

Some benchmarks are provided also as tools to lead experiments. BeFaaS [19] has an interesting focus on fine-grained evaluation of applications, although the goal is to evaluate and compare performance of FaaS platforms. vHive [32] is a experimental, representative serverless platform to experiment on the serverless stack. For both, the focus is on the platform itself rather than understanding the programs running in cloud functions.

FaaSLoad can be a support for benchmarking, but its purpose is to support systems-level studies by injecting a workload and collecting fine-grained data about the functions’ programs.

Other works focused more on the serverless applications. Authors of [23] have added characterization of function properties in a DevOps pipeline. While in a different context than FaaSLoad, their work also include workload generation and profiling of resource usage.

In a more general setting, there is the Serverless Application Analytics Framework [16]. It allows to collect per-function low-level and platform-level metrics, and comes with a tool to lead experiments. By comparison, FaaSLoad avoids function modifications for most data collection, and provides more extended tooling to inject workloads.

Finally, with FaaSProfiler [28], researchers looked into the architectural implications of FaaS. This is close to FaaSLoad’s purpose of being able to profile at low-level the programs running in FaaS platforms. However, FaaSLoad covers more use cases, with more complex workloads, in more representative FaaS platforms.

In a nutshell, the purpose of FaaSLoad is to inject a workload into a FaaS platform, and to collect fine-grained, low-level data about the functions that respond to it. The workload is represented as traces of discrete invocations of functions. This is important: FaaSLoad does not send queries to an application, following given parameters; instead, it triggers individual functions with specified arguments. The point is to gather fine-grained data about functions, instead of wholistic information about an application’s performance; for example, to gather memory or CPU usage of programs, instead of end-to-end latency of an application.

We identify many properties that FaaSLoad’s design must have. First at high level, the design of FaaSLoad is motivated by the need for flexibility. It must be useful to any researcher wanting to have a focused look on the programs in cloud functions in varied contexts. This includes their resource usage and performance profiles, at the levels of the system and the architecture. It must also be agnostic from FaaS platforms.

In addition, FaaSLoad’s design must answer to inherent constraints of FaaS platforms. First is the scalability: invoking thousands of functions at the rate of hundreds of requests per second. Second are semantic features such as cold starts (see Section 3.5), etc.

Furthermore, it is also important to FaaSLoad’s design to be able to inject any shape of workload. Our system is also science-focused, and as such strives to be useful in the context of systems research: it seeks reproducibility and correctness of its workload injection, and includes checkpointing for its long-running dataset generation process. Finally, the data it produces is easy to explore and exploit.

The general process of FaaSLoad injecting a workload is shown in Figure 1.

Offline, (1) a trace builder writes injection traces in FaaSLoad’s format. Upon starting, (2) the system reads those traces to extract information about the cloud functions to invoke, as well as the injection trace points, i.e., invocation requests to send. The main part are the invokers, which (3a) send invocation requests. Actually, they use the platform connector, that invokes the requested functions by connecting to the FaaS platform. The latter eventually schedules the request to a function instance observed by FaaSLoad’s function monitor.

An important point is that the invokers do not wait for any injection; instead, there is an injection monitor that watches over the whole injection process, waiting for every invocation (3b) sent by the invokers, to terminate. When (4) one invocation terminates, it fetches its related data from the FaaS platform through the platform connector, as well as (5) the data about the execution of the function that served the request, from the function monitor. Finally, the injection monitor (6) stores the data to a database for offline use.

As can be seen in this process, there are two main components to FaaSLoad: a workload injector (that gathers the invokers and the injection monitor, and uses the platform connector), and a function monitor. This modularity allows to only uses the necessary parts of FaaSLoad (e.g., using only the function monitor, with a custom client), making it flexible.

We detail below the designs and implementations of FaaSLoad’s workload injector and function monitor. They are illustrated, with their internal components, in Figure 2.

We detail first the workload injector (left side of Figure 2). There are three parts to the workload injector: (i) the injection monitor; (ii) the invoker threads; and (iii) the platform connector (the public interface to the platform in Figure 2).

After covering the design and implementation of the workload injector, we focus on the function monitor that can supplement FaaSLoad’s workload injection (right side of Figure 2).

The role of the function monitor is to collect fine-grained data about the programs running in the cloud functions: CPU and memory usage during the execution, CPU cache misses, etc.; additionally, it watches events related to the management of functions instances (i.e., Docker containers⁷⁷7The function monitor requires Docker to monitor its socket, but could easily be made to work on other container backends that also use a UNIX socket where it could listen for function container events.): creation, deletion, etc.

FaaSLoad is designed for the FaaS context. It includes notions of functions, as well as different users to invoke functions separately on the FaaS platform. Its workload injector invokes functions with parameters through the public interface of FaaS platforms (i.e., it does not bypass the normal path), and it gathers their results, including their potential failure.

Most importantly, FaaSLoad takes into account cold starts, by storing the added initialization time, and marking invocations served by a cold-started instance. Indeed, when a FaaS platform serves a request to a function, it has two choices: spawn a new instance, or reuse a previous idle instance. The former case is called a cold start, while the latter is conversely called a warm start. A cold start increases the response latency of the function, because the FaaS platform must first initialize a new instance [10, 17]. Information about cold starts is provided by the FaaS platform, via the platform connector (see Section 3.3.3).

Finally, FaaSLoad gathers a global view of function instance events (see Section 3.4). In addition, mapping invocations to instances is easily achieved by customizing cloud functions to keep track of their instances, and return this piece of information as a result to be stored by FaaSLoad (see Section 5.3 for another example of function customization).

Injection traces are simple text files containing lines of tab-separated values. An example is given in Appendix B. One invocation is simply one line listing the inter-arrival time from the previous invocation, and the arguments to pass to the function.

FaaSLoad’s traces declare injection points using delays between each other, instead of relative time since the beginning of the trace. This has two advantages: it simplifies implementing the invoker threads explained in Section 3.3.2; and it simplifies implementing trace generators based on statistical distributions of the inter-arrival time (IAT) (see Appendix C).

FaaSLoad allows to inject real workloads into a FaaS platform. This relies on two pieces:

1. 1.

   a dummy cloud function which execution time and memory usage are set by its parameters;

2. 2.

   a converter from the real workload traces to FaaSLoad’s injection traces.

While the first one is straightforward, the second one, converting real workload traces, is much more difficult. Indeed, the canonical data of real workload traces published by Microsoft Azure [29] does not include essential information such as the actual functions that are executed, or even exact invocation dates. First, this comes from the obvious confidentiality requirement of the workloads; and second, the sheer amount of invocations in any sizeable dataset makes collecting complete data prohibitive, so Azure’s data only include statistical properties that summarize features: quantiles of the inter-arrival times for a given time slice, of the memory usage of whole applications instead of individual functions…

First, we evaluate the scalability of FaaSLoad’s workload injector through two experiments. In the first one, we check the scalability of one invoker thread in the face of a bursting injection trace. In the second one, we check the scalability of the whole injector over the number of functions it has to invoke. In both experiments, the metric is the injection timing error, i.e., the time difference between the actual invocation date and the expected invocation date from the trace. Positive errors mean it was late; negative errors denote early invocations.

The setup is as follows: FaaSLoad runs on a dedicated machine, connected to a Kubernetes cluster of 8 nodes. In the cluster, Apache OpenWhisk is deployed, and configured for scalability (2 invokers per node, all memory available). All machines are identical: every machine is a Dell PowerEdge R640 from 2019, with 96 GiB of memory and an Intel Xeon Gold 5220 at 2.20 GHz with 18 cores, running Debian 11 (or Debian 12 for the machine running FaaSLoad), from the distributed testbed Grid’5000 [12]. We provision an OpenWhisk deployment that can serve without overhead the requests sent by FaaSLoad while it is under test, so that the results are not affected by OpenWhisk’s performance.

FaaSLoad was used for building a dataset for machine learning, in a published paper [26]. In this work, we studied the impact of features of functions inputs on their memory usage, and then developed ML models to infer the memory usage. For instance, image processing functions may use different amounts of memory depending on the image size, format, etc. FaaSLoad collected a dataset of memory usage of 12 functions.

To build a dataset with FaaSLoad, the setup is a follows: declare functions under study in a manifest file,⁹⁹9This is inspired by OpenWhisk’s manifest files used by its tool wskdeploy [1]. then use FaaSLoad’s helper tool to load the functions into OpenWhisk; finally to build the dataset, let FaaSLoad run automatically under a reproducibly random injection trace, and with the function monitor configured to collect needed data.

Figure 5 shows a few example results of the data obtained by FaaSLoad. In this work, we successfully trained J48 (C4.5) models [34] with up to 91% of accuracy, thanks to FaaSLoad providing data to identify significant features, and to train models. In addition, data showed how memory usage can vary during a function execution, with different patterns.

Using FaaSLoad, we conduct an experiment on a Java virtual machine (IBM Semeru OpenJ9 VM¹⁰¹⁰10Home page of Semeru: https://developer.ibm.com/languages/java/semeru-runtimes/., used as the default Java VM by OpenWhisk): we examine the Garbage Collector (GC) and the Just-In-Time (JIT) compiler. In the Java VM, the GC runs concurrently to the application, to free memory from the heap by clearing unused objects. As for the JIT, it profiles the application on its first run to find code pieces that would benefit from being executed natively, i.e., compiled to the host architecture, instead of being interpreted.

Specifically, we measure the effects of cold starts on the Java VM itself, beyond the latency they add to function invocations. In this experiment, we check if, and how, cold starts affect the function’s language runtime, by focusing on the Java VM’s GC and JIT.

We ran FaaSLoad to generate a dataset of executions for 3 Java applications: (I) an identity function returning its parameters without any processing, (R) an image resizing function, and (A) an autocompletion algorithm (taking a radix as parameter, it returns the most probable matches from a dictionary). We had FaaSLoad collect the detailed CPU usage of the Java VM. This uses the VM’s specific interface JvmCpuMonitorMXBean [2], that provides the breakdown of total CPU usage between GC+JIT (i.e., the Garbage Collector and the Just-In-Time compiler) and application. Functions have been modified to return this breakdown, and FaaSLoad collects it for data processing.

Figure 6 shows transposed CDFs of CPU usage for the three functions. There is a visible increase in CPU usage and variability between cold and warm starts, both for GC+JIT and for application. About warm executions, this is easily explained: the Java code has been JIT compiled, it is now optimized and simply runs more efficiently. About cold executions, the results show that both the GC and the JIT compiler use more CPU time (both in absolute, and relatively to the total CPU time) in cold executions. By definition, the cold execution of a function instance is the very first one, and this is the one that suffers the JIT overhead the most, as well as some GC overhead of cleaning initialization objects.

As a takeaway, during cold start invocations, GC+JIT CPU usage can represent more than 60% of total CPU time: the cold function instance spends more time optimizing the program, than running it. This trade-off may be beneficial for long-running applications, but in FaaS applications, a function instance may never be reused: then the optimizations are lost, because they are not shared between instances (although some works started looking into that [14]). We conclude that current language runtimes are not FaaS-ready.

In Section 4.2, we showed our conversion algorithm for the Microsoft Azure Functions Dataset from 2019 [29]. In this section, we evaluate their correctness when converted for replaying by FaaSLoad. Note that in this evaluation, we skip building the call graph (see Section 4.2.1), because it was not reliable due to the lack of data from the original traces.

Major challenges are to compute concrete values for Inter-Arrival Times (IATs), and function execution times and memory usage, while keeping the same statistical properties as the original, incomplete and at times inconsistent traces. We evaluate the fidelity of the conversion, by calculating the same parameters as in the companion paper to the traces.¹¹¹¹11Note that the values of the parameters we calculated over the original traces sampled at 100%, are different from those given in the companion paper, because the authors published different traces.

They are shown in Table 1. For execution times, the parameters are the log-mean and standard deviation of the log-normal fit to their distribution; as for memory, they are the parameters of a fitted Burr XII distribution of the memory usage of applications.

As expected, the shorter the sampling window, and the smallest the sampling subset, the less representative the resulting traces. Nonetheless, results would vary on the exact sampled time windows (here, they all start at the beginning of the traces). As shown in Section 5.1.2, FaaSLoad can handle up to around 1000 concurrent functions, so Table 1 indicates that the best trade-offs would be to sample over 1 h and 1% (closest log-mean execution time), or over 10 min at 2% (closest memory statistics), although it of course depends on the objectives.