Real-Time System Evaluation Techniques:
A Systematic Mapping Study

Components are grouped into categories, e.g. UUniFast is an example of a task set generator. In this section we define the categories of components we want to differentiate, roughly ordered from least to highest degree of design complexity:

Microbenchmarks.

In our study, we use microbenchmark to refer to components with the following criteria:

• $\mathrm{■}$

  custom made (not a benchmark suite)

• $\mathrm{■}$

  manually made (not using a generator)

• $\mathrm{■}$

  simplistic design (not a case study)

We use this term even when a component is not specifically referred to as a microbenchmark, but otherwise follows the above criteria. Microbenchmarks are applied to test basic functionalities or provide insights into extra-functional properties, such as memory access times. They are generally not published and do not carry a unique name.

Benchmarks.

Benchmarks are generic example applications, often publicly distributed in larger collections with unique names. We also consider the following criteria:

• $\mathrm{■}$

  not custom made (as opposed to a microbenchmark)

• $\mathrm{■}$

  finalized example applications (as opposed to output of a generator)

• $\mathrm{■}$

  generic system design (as opposed to a case study)

Individual benchmark programs may range from trivial to very high complexity. This is not considered in our categorization. We find that benchmarks can easily be told apart from microbenchmarks and case studies by how they are constructed and distributed.

X Generators.

A generator is a tool which randomly produces a desired kind of output X, in which case we categorize it as an X generator. For example, in real-time systems research, it is common to use task set generators and graph generators. We have also seen specialized network traffic generators. These components follow these criteria:

• $\mathrm{■}$

  automatically generated output (as opposed to manually made)

• $\mathrm{■}$

  output lacks design goal (as opposed to a case study)

Besides using de-facto standard generators with recognizable names, many researchers also implement custom X generators, which may never be published or have a name.

Specifications.

A specification is a collection of extra-functional properties derived from real systems. These are used as reference material to construct a scenario. They are differentiated further by not fitting into any of the other categories presented here:

• $\mathrm{■}$

  not an application (as opposed to (micro-)benchmarks)

• $\mathrm{■}$

  not generated, but may be used with a generator

• $\mathrm{■}$

  not a case study, but may be used to construct a case study

Case Studies.

Case studies consider complete exemplary systems with the following criteria:

• $\mathrm{■}$

  can typically be assigned to a field of study

• $\mathrm{■}$

  designed with intent (as opposed to generated)

• $\mathrm{■}$

  non-trivial design (as opposed to (micro-)benchmarks)

Case studies are typically not generic, but instead exist within a field of study, e.g. the automotive field. Commonly, researchers set up their own systems to construct a realistic scenario, which we consider to be a custom case study. But a case study can also be directly taken from an industrial challenge. We group all kinds case studies based on their field of study.

Sometimes case studies are directly based on various pieces of open source software. However, a deeper look into the underlying design choices of case studies is out of scope for this paper. Again, we are only focusing on whether case studies are being used at all.

In Table 1 we provide the categories of a subset of components from our results. All of the components shown here are popular within real-time systems research and in fact, most of them originated there. Among the examples are several benchmark suites and common generators, for task sets and graphs respectively. If a paper is found to use UUniFast, it can be grouped with others papers using UUniFast, but also with other papers using any kind of task set generator.

IsolBench is a special case worth exploring in more detail: It is a very small benchmark suite with the highly specialized purpose of maximizing interference on multi-core systems. As such, it is to be expected that IsolBench is not used alone, but always in conjunction with other components with which it interferes. Since we are only interested at looking into which components are being used at all, we will not take into account how IsolBench is being combined with other components. We will only provide the complete list of components used per paper.

Real-World Automotive Benchmarks For Free (RWABFF) [18] has been categorized as a specification. In this paper, the authors provide insights into the extra-functional properties of automotive software. No benchmarks and no generator are provided.

Above we have already introduced two optional properties for any given evaluation components: a name and a field of study. Names are easily recognizable and an ideal candidate for our systematic mapping study. The field of study is not as explicit and will need to be determined subjectively. Nonetheless, it is an important characteristic to determine research trends over the years.

Some components, particularly case studies, are characterized by restricted access, i.e. a contractual non-disclosure agreement has been reached between researchers and a third party which sets limitations for distribution. However, if a component merely requires commercial of the shelf products, either hardware or software, we do not consider it to have access restrictions.

Many publication sites now optionally offer artifact evaluation. If authors opt in, their papers are not just assessed based on their contents. Additionally, the experiments are being retraced by at least one reviewer who gains access to the tools and systems used by the authors, which are collectively referred to as an artifact. This way, reproducibility is guaranteed in the review process. Papers which underwent artifact evaluation stand out as they receive a seal on their first page. Authors may choose to publish the artifact alongside the paper, which allows any readers to reproduce the results themselves.

However, there are cases in which artifact evaluation is infeasible. For example, in the field of real-time systems it is common that an evaluation scenario uses specialized hardware or commercial software, which may not be immediately available to any of the reviewers.

We assess the prevalence of both artifact evaluation and the public availability of artifacts. For our study, it is an important distinction that artifact evaluation cannot be attributed to components, but instead only to a paper.

The international real-time systems research community is organized around three major conferences: ECRTS, RTAS, and RTSS. We conduct our study on papers from these conferences released between 2017 and 2024.

To support our selection, we have looked at TACLeBench [9], a benchmark suite for real-time systems research from 2016, close to the start of our considered time frame. In November 2023 we conducted a Google Scholar search to list the publications which have cited the TACLeBench paper and we grouped them by their publication sites.

Table 2 provides a subset of our top search results. We list the names of the publication sites according to the amount of citations of the TACLeBench paper in descending order. We also list the corresponding conference ranking according to CORE [1] and Qualis [2]. CORE uses the ranks A*, A, B, C, listed from excellent to satisfactory. Qualis uses A1 to A4, B1 to B4, and C. The shown ranks are the most recent at time of writing. Our selected conferences are top results in our search and highly regarded in both rankings throughout the entire time frame of our study.

There are several more conferences and journals which do not focus on real-time systems primarily, which we exclude and have omitted from the table. Further, we exclude conferences which have never received a CORE rank of A* or A within our time frame, namely ISORC and RTCSA. RTNS shows up high in the list, however CORE excludes it as a national conference and we have therefore excluded it as well. Journals, such as TECS, publish extended versions of papers previously seen at conferences. Extended papers include the same evaluations as prior publications. We want to avoid double-counting the exact same evaluation and therefore journals are excluded.

Another top contributor to the citations of TACLeBench is the WCET workshop. We have excluded workshops from our search space, as some workshop papers have been followed up with an extended conference paper. This carries the same double-counting risk as seen with journals.

The selected conferences sometimes publish preliminary works or industry papers as part of their proceedings. Preliminary papers are excluded from further analysis to avoid double-counting of the same evaluation. Industry papers are excluded as our interest goes out to the methods and techniques used by researchers to evaluate real-time systems. Papers were excluded based on these keywords in their titles: Work-in-Progress, Work in Progress, Brief Industry Paper, Demo, Demo Abstract, Industrial Challenge.

ECRTS does not publish preliminary papers as part of the proceedings, but does offer separate tracks for them, such as the real-time pitches session. Separate tracks are not considered in our search space.

In Table 3 we break down how many papers per conference and year have been included or excluded, based on the above criteria. In total, we evaluate 677 papers for our systematic mapping study.

The papers were individually read and categorized. This process was set up in multiple phases: In the first phase the evaluation components were collected in human-readable format, inadequate for automated processing. From this first rough draft, we constructed a data layout which represents our findings according to the categories we introduced in Section 2.1.

We determined that a paper can use any number of evaluation components. A component is not necessarily limited to a single experiment. Sometimes it can serve for many experiments, other times multiple components must be combined for a single experiment. We abstract the number of experiments and the design of the underlying scenarios entirely and instead only pay attention to which components are used at all.

Each evaluation component is characterized by up to five data points:

Qualifier.

The qualifier describes how the component is introduced. Viable options are: named, extended, referenced, custom, undetermined.

Category.

The category which the applied component can be assigned to. We consider the categories defined in Section 2.1.

Specifier.

If a component has a name, we store it in the specifier. In some cases, we use unique identifiers as placeholders for missing names. Otherwise the specifier remains empty.

Field.

If a component applies to a specific (as opposed to generic) field of real-time systems research, it is stored here. Otherwise the field remains empty.

Restricted?

In contrast to the other options, this one does not contain descriptors, but is instead a binary decision. If we are certain that the component has access restrictions (as defined in Section 2.2), this is set to true, otherwise it is false.

With the final data layout prepared, the first data set was cleaned and each component was re-evaluated. In a separate process, we analyzed whether the papers in the search space underwent artifact evaluation as part of their reviewing process, and whether the artifacts are publicly available. The format of our data set is represented by the layout shown in Table 4, alongside complete data for two example papers from the dataset. The two papers were selected for their simple, yet representative data points.

For each paper, the following data points are collected first: the conference name, the year of publication, the author name(s), and the title of the paper.

Further classification follows the flowchart shown in Figure 1. In stark contrast to the previous data points, the classification based on the flowchart is up to interpretation. Despite our efforts to delineate clear categories, edge cases may occur.

In the following, we walk through the decision points shown in Figure 1 and highlight some of the challenges we have met during the process.

Does the paper feature an evaluation?

Some papers in our search space do not have an evaluation at all. These include surveys and papers of theoretical nature which present mathematical proofs.

For our study, we will not assign any components to these papers. Therefore, in our dataset the evaluation components section for these papers is entirely empty. These papers are still listed so that we can derive statistics on the number of papers with and without practical evaluations.

Is the evaluation component not described succinctly?

In rare cases we faced great difficulty in determining whether an evaluation component is following established methods or is custom made. This is restricted to papers which use a random task set generator, which is a commonly applied category of component with many variants. If the authors do not provide the name of the algorithm or technique used, nor cite other works providing further details and finally do not succinctly describe the generation process we classified the component as undetermined task set generator.

Regarding this categorization, it is important to keep in mind that our approach is based on cursory reading. In our analysis, we make no distinction between custom task set generator and undetermined task set generator.

Does the evaluation component have a name?

Established components typically carry a name and there is an original resource for them which can be cited. If authors have clearly stated the name in their evaluation, this name was added directly to our dataset.

However, some authors do not use the given name of an evaluation component and instead just reference it. In these cases, the provided reference material was used to help categorize it.

In summary, we categorize the evaluation component based on whether it has a name, not based on whether authors use the name.

In Table 4 we use [16] as an example for a paper which uses a named component, in this case the benchmark suite Autobench [3].

Is the component being extended?

Evaluation components with established names are typically of generic nature. This may be inadequate for a given system model.

In such cases authors may opt to extend the established components in some way. We categorize such components as extended. For example, in [4] the task set generator UUniFast is used in a greater context specific to the system model considered within the paper.

The differentiation between original and extended components is not trivial. Generally, we considered components to be used in their original state, unless it was clearly indicated that they are being extended.

Is the component established in prior work?

As stated above, authors sometimes refer to evaluation components by citing other works. If these other works do not provide any further insights into the name of the component, we assume it to be nameless and categorize it as referenced. We opted to use a unique identifier in our dataset for such components. This allows us to group various papers together, despite the lack of a name. The unique identifier is derived from the last names of the authors of the referenced works introducing the evaluation component. This process is not guaranteed to be free of errors, because there is no guarantee that different papers cite the same paper as original resource for a component.

Default: Custom Evaluation Techniques.

If the above categories are insufficient, we opt to refer to the presented evaluation component as custom. It is important to note that custom components may not have originated with the authors of the paper entirely. For example, it is common for researchers to combine various pieces of open-source software to assemble a custom case study.

We assign all case studies to a field of study as defined in Section 2.2. The fields we identified have a great deal of overlap and making a decision proved challenging. Generally, we tried to extract the field from the authors’ stated goals. If the authors are concerned with the implementation and stability of control systems within real-time contexts, we categorize their case studies as controllers. A drone is a system which requires a flight stability controller, which in isolation is a candidate for the controller field. If however the drone is being considered in a larger context, including e.g. communication with ground stations, we categorize the case study as belonging to the unmanned aerial vehicle (uav) field.

In Table 4 we reference [10] as an example of a paper presenting a custom case study, in this case in the automotive field.

To verify the integrity and quality of our dataset, we have performed checks to validate the reproducibility of the categorization process described in Figure 1 and the comprehensiveness of the entries.

One author has classified all 677 papers according to the categorization process. Afterwards, a random selection of 5% of the papers was drawn and reassessed again in a joint effort by both authors. Of the 34 papers used for this validity check, 17 were an exact match and an additional 7 only had minor differences. These differences can be linked to subjective interpretation, e.g. differentiating fields of study for custom case studies or trying to determine whether a task set generator is truly custom-made or based on UUniFast. For 9 of the remaining 10 papers, we found that the reassessment yielded additional components compared to the first assessment. This could be a result of a greater attention to detail being applied in the validity check, as opposed to a more time-oriented focus during the initial data gathering process. For each of these 9 papers at most one additional component was identified. There was one paper in which our validity check yielded a categorization error. In the initial assessment it was determined that it references the Mälardalen Benchmark Suite. However, during reassessment we found that it instead references a case study concerning autonomous vehicles which was also formulated at the Mälardalen University.

In summary, although we have discovered several minor differences between the original and the validation set, these differences were due to oversight only, whereas the process itself appears to be robust. The additional components were added and the categorization error was fixed in the final data set.

To increase the confidence in the remaining results, we have performed full-text searches for common benchmark names and for terms related to the categories. For instance, we searched all papers for the strings artifact evaluation, UUniFast, TACLeBench, and several spelling variations. We then compared the collected search results with our manual classification. Similarly, we searched for paper citations of common generators and benchmarks, such as [5, 9], as a citation indicates that the specific component has been applied in the evaluation. Interestingly, both the full-text search and citation analysis have only returned a subset of the original dataset, which indicates that these automated approaches are insufficient. Potential reasons include spelling inconsistencies we did not account for, or missing citations, or components only being described indirectly via several steps of references.

Nonetheless, our validity checks give us a high degree of confidence that our dataset is predominantly correct.

We first start with an overview of the general evaluation categories as defined in Section 2.1. In total, we have recorded $1112$ individual evaluation component uses, significantly more than the number of papers we have analyzed. Figure 2 shows how the individual evaluation components are distributed among the papers.

Nearly half of all papers exhibit only one evaluation component. Papers that exhibit more than one component, typically combine one, or even several, case studies with a more generic component, such as benchmarks or generators. Papers with a very high number of components often combine benchmarks from various sources. Although it can be argued that in such a case, the combined set of benchmarks may only serve for one experiment, we have not performed such an in-depth analysis, as we only account for whether a component is used at all. We give the authors the benefit of the doubt that each benchmark suite serves a different purpose, because of these limitations in our methodology.

All $1112$ component instances contribute to the charts in Figure 3 and 4.

More than $30\%$ of all component instances are categorized as a case study, closely followed by benchmarks, task set generators and microbenchmarks. The rest is much less common and includes, among others, various kinds of generators and specifications. Figure 3 shows no clear trends over the years, but some indication that benchmarks are more common at ECRTS than at RTSS, while case studies are more common at RTSS.

The heat map of evaluation components favored by prolific authors, shown in Figure 4, reveals that authors who published more than $12$ papers have all used task set generators. This serves as an indication, that task set generators as a category are de-facto standard practice within real-time systems research. Almost all papers without practical evaluation are co-authored by at least one of the most prolific authors.

In contrast, authors with fewer than 12 publications within our data set did not publish a paper without a practical evaluation, at least not without a more prolific member of our community as co-author.

In Figure 5, we analyze the distribution of fields of study. These fields are only defined per case study instance, hence, the total population in these figures consists of $336$ entries. The fields have significant overlap. One must not interpret Figure 5 to indicate that neural networks are competing for attention with the automotive field. Instead, we draw the conclusion that our community is broadly concerned with cutting edge automotive applications, such as autonomous driving which includes neural networks, e.g. for obstacle detection.

We have not been able to draw conclusive observations from the distribution of research fields to authors. Hence, no heat map regarding fields is provided here.

Figure 6 shows the distribution of instances of named components. This graph subsumes various categories, including benchmarks, task set generators, case studies, and specifications as long as the component is clearly identified by name. In the graph, generators are highlighted with dashes, specifications with grids, and for benchmarks we used dots, rings, and stars. The total population consists of 657 entries.

We have observed a large variety with over $224$ different names²²2Due to different versions of otherwise similar benchmarks, we estimate that the actual number of meaningfully distinct benchmarks or generators may in fact be slightly lower.. Given that we have studied $677$ papers, including some without a practical evaluation, nearly every third paper introduces a new named benchmark, case study or otherwise new evaluation component. Only about $5\%$ of these named evaluation components are used in at least $12$ papers. Instead, $150$ or about $67\%$ are used only once. These results are puzzling. On the one hand, we can hypothesize that this result is a positive indication of the large variety of real-time embedded system models that we cover in our research community. On the other hand, it is also a clear indicator for a lack of commonality. If nearly every paper uses unique evaluation components, the results will generally not be comparable.

The only clear exception, which we may call a de-facto standard, is UUniFast. This task set generator seems to provide common ground for the evaluation of scheduling techniques.

We can further conclude that the industrial challenges, such as those posed at the WATERS workshop and later on at ECRTS, have had only minor impact so far, as there are only $10$ uses within our dataset. This suggests that there may be significant barriers and/or a lack of interest in the research community to adopt a unified and realistic case study.

As described in Section 1, ideally all scientific results are replicable, i.e. a proposed method works under any valid set of conditions. With artifact evaluation we at least have proof of reproducibility, i.e. the method works consistently within equal conditions. With this systematic mapping study, we are not able to provide conclusive evidence about the replicability. We do not only believe that this is out-of-scope of this paper, but also that the effort required would only allow for a replicability study of a very small set of papers.

We can, however, provide an estimate of how difficult it is to recreate an evaluation scenario to analyze the reproducibility. Recreating the very same evaluation scenario also is the requirement for analyzing the very same subject matter which allows for the comparison of experimental results.

First, all evaluation results of papers that have passed artifact evaluation have already been successfully reproduced. Evaluation artifacts are also easy to access and kept in their original state. For other papers, if all components of an evaluation are named, one can at least combine the corresponding benchmarks (or generators, …) in an attempt to reproduce the subject matter. On the other extreme, a custom evaluation can usually only be reproduced if it is comprehensively described. The access to restricted evaluation components is clearly expected to be even more difficult or outright impossible.

In Figure 7 we have categorized all papers that include at least one evaluation component into how easily accessible their components are and, thus, how easy it is to recreate the subject matter. The total size of the population of this figure is $645$. We have used a strict definition in that a single restricted component suffices to classify a paper as restricted and all components must be named for a paper to be classified as named.

Artifact evaluation was available throughout the entire time frame of our study at ECRTS and RTSS. RTAS first offered the process in $2018$. For all three conference it appears that the artifact evaluation has saturated quickly with no clear trends of change throughout the years. Since artifact evaluation is not feasible for every publication, it therefore seems reasonable to assume that all authors who could participate, already do.

Considering reproducibility, nearly every fifth paper passes artifact evaluation and only $3.3\%$ of all papers use restricted evaluation components. On the other hand, roughly half of all papers require the reader to manually rebuild the evaluation scenario based solely on the description within them. We assume this process to be more cumbersome and potentially error-prone than simply using an artifact or even a named component.

Ideally, a benchmark is a permanently fixed piece of software, but in practice all software may contain bugs or require updates for compatibility with new systems. As performance demands are rising, benchmark suites may also require extension to stay relevant. Thus, benchmark suites undergo an evolution and change over time. The same can be said for other tools, such as task set generators. As the experience of our research community grows, older approaches may fall out of favor. We believe that as researchers we must hold ourselves accountable to stay informed and adopt new practices.

In Figure 8, we show the timeline of uses of newer components compared to the ones they are meant to surpass. In the first graph, we track the use of the multi-core task set generators UUniFast–Discard, RandFixSum, and Dirichlet–Rescale. UUniFast–Discard is very inefficient as it tends to generate invalid output, which must be discarded (hence the name). RandFixSum solves this, but constrains the user limiting its applications [8]. Dirichlet–Rescale improves on these usability issues and is state-of-the-art since 2020 [11]. We can see that the community partially adopts the new task set generator immediately following its introduction, but the older generators remain in use nonetheless.

A similar phenomenon is shown in the second graph: The benchmark suite TACLeBench from 2016 contains, among others, the benchmarks from Mälardalen, but with several fixes, extensions, and in uniform format [9]. Nonetheless, Mälardalen was at least equally popular until 2020 and remains in use since then.

In the first two graphs, the adoption of the improved component is slow, but still clearly visible. In the third graph we see an example of no adoption happening at all: The San Diego Vision Benchmark Suite (SDVBS) has been superseded in 2014 by the Cortex Suite, which contains several extensions to meet rising performance demands and to provide a more diverse range of example applications [20]. Yet in our study SDVBS was consistently exactly as or even more prevalent than Cortex Suite.

There may be justifiable reasons to use outdated components, even many years after they have been surpassed. Such justifications are not captured in our study and must be assessed individually in a paper review process. However, it is also possible that both authors and reviewers remain uninformed about changes to the state-of-the-art practices.

Our quantitative analysis has yielded puzzling results prompting further research questions. For example, the specification Real-World Automotive Benchmarks For Free (RWABFF) based on realistic automotive systems was used much more commonly than complete automotive industrial challenges posed at the WATERS workshop. It is possible that the industrial challenges are deemed too restrictive by researchers and that in comparison a specification such as RWABFF offers more degrees of freedom. This is supported by the frequent use of custom case studies, which indicates that researchers are able and willing to deal with highly complex systems. In summary, more research is needed to determine which demands must be met to provide a unified, realistic case study to the real-time systems research community.

Our systematic mapping study provides the ground work for these research questions. We plan to tackle them by undertaking a systematic review, in which we narrow down on specific evaluation practices in more depth, by analyzing not just whether they have been applied at all, but also the specific methodology and conditions in detail.