Research Infrastructures and Tools for Collaborative Networked Systems Research

A science is defined by a set of encyclopedic knowledge related to facts or phenomena following rules or evidenced by experimentally-driven observations. Computer Science and in particular computer networks is a relatively new scientific domain maturing over years and adopting the best practices inherited from more fundamental disciplines. The design of past, present and future networking components and architectures have been assisted, among other methods, by experimentally-driven research and in particular by the deployment of test platforms, usually named as testbeds. However, often experimentally-driven networking research used scattered methodologies, based on ad-hoc, small-sized testbeds, producing hardly repeatable results. We believe that computer networks needs to adopt a more structured methodology, supported by appropriate instruments, to produce credible experimental results supporting radical and incremental innovations. This presentation reports lessons learned from the design and operation of test platforms for the scientific community dealing with digital infrastructures. The SLICES initiative is introduced as the outcome multi-year process of conceptual evolution for a networking test platform transformed into a scientific instrument. Challenges, requirements and opportunities are addressed that our community is facing to manage the full research life cycle necessary to support a scientific methodology. Further details are provided in [1].

This group discussed goals of reproducibility initiatives, current pitfalls/gaps and how we could address them, and role of research infrastructures in supporting reproducibility.

This breakout group focused on defining workflows and software needs for research infrastructures to ensure reproducible experimental research. Two key assumptions were established: $(i)$ correctness of experiments remains the responsibility of the researcher, and $(ii)$ infrastructure policies are expected to be followed, with logging and monitoring serving as diagnostic tools rather than enforcement mechanisms.

The working group was set up to address the following questions: What needs to be considered when creating Digital Twins by means of AI-based Data and Workflow management? Does the development of next-generation Foundation Models (FM) for networking lead to a new kind of “collaborative networked systems research”?

The working group started with analyzing the creation of a Digital Twin within a testbed. A guiding example for which the creation of a digital twin would be desirable is autoscaling (scale-out/scale-in) of a Kubernetes Network Function Virtualization (NFV), e.g., as part of a 5G/6G use case. The digital twin would allow to analyze the behavior of the autoscaling function for different input and different configuration parameters. In general, to create a digital twin for a given task, there exist different approaches. One important traditional approach is to create a (hand-crafted) simulation; a more recent approach leverages Machine Learning (ML) and Artificial Intelligence (AI). When applying AI/ML, different AI/ML methods can be used, in particular:

1. 1.

   black-box learning (e.g., supervised learning),

2. 2.

   grey-box learning (e.g., explainable AI), and

3. 3.

   Foundation Models (FM) that use self-supervised learning and unlabeled data for pre-training (phase 1) and then perform application-specific finetuning (phase 2).

In order to assess the different approaches of generating a digital twin, it is important being able to assess the prediction quality (concerning what-if questions) of a digital twin. Scientific questions in this context include:

• $\mathrm{■}$

  What is the “right” training data for a given problem?

• $\mathrm{■}$

  What are possible root causes of inaccurate or wrong inferences? (E.g., are root causes due to a specific machine learning method used, or due to bad training data?

• $\mathrm{■}$

  Decision tree (DT) models are appealing because they are inherently interpretable, but does their simplicity limit their applicability?

• $\mathrm{■}$

  Do DT models miss out on important hidden context?

• $\mathrm{■}$

  How to quantify the confidence in the output of a trained DT model?

• $\mathrm{■}$

  How to test for model quality beyond accuracy?

• $\mathrm{■}$

  Is the generated model generalizable? (An example for a non-generalizable model would be that the prediction is only accurate for a specific testbed, which was used to generate the data.)

• $\mathrm{■}$

  What is a good FM?

Domains for which such digital twins could be generated for answering scientific questions include performance and security. With regard to model quality, mediocre models will be able to answer specific questions with sufficient accuracy, while we can expect that from a good model, a domain expert can learn from its outcomes. While large FMs may have the potential to be a basis for good models, realizing them using currently employed workflows or pipelines poses significant challenges, including high costs (e.g., specialized hardware), high energy consumption (e.g., computing power requirements in the terraFLOPS range), and long model training times (e.g., days or weeks). Other open problems associated with FMs include whether different FMs need to be generated for different domains (e.g., for wired networks, for wireless networks, for cybersecurity, etc.), and how to create good FMs (e.g., by creating hybrid models that include domain-specific hand-crafted models). With respect to good datasets, testbeds are considered valuable (i.e., necessary but not sufficient) to generate realistic data sets.

The working group started looking at testbed coverage and futures, discussing whether the current set of testbeds provides all necessary scientific instrument functionalities. The discussion covered gaps in current testbeds, emerging use cases, and the role of testbeds in advancing research. Participants explored testbed categorization, research question formulation, and the potential alignment with methodologies used in other scientific disciplines.

A scientific instrument is a research infrastructure designed to enable systematic experimentation and validation of scientific theories. In digital sciences, this means testbeds should go beyond simply demonstrating technology – they must support rigorous experimentation, reproducibility, and fundamental scientific inquiry, similar to physics research facilities.

There are two approaches to research testbed design in computer science:

• $\mathrm{■}$

  Testbed-Driven Research: Researchers use available testbeds and define research questions accordingly.

• $\mathrm{■}$

  Research-Driven Testbed Design: Research questions are defined first, and testbeds are designed or selected based on scientific needs.

The working group emphasized the importance of the second approach, similar to physics research, where experimental facilities are built to answer fundamental scientific questions rather than just demonstrate technology feasibility.

The group identified three main categories of existing testbeds:

1. 1.

   General-Purpose, Flexible Testbeds – Adaptable platforms for a wide range of research topics.

2. 2.

   Domain-Specific Testbeds – Built for targeted experimentation in realistic but controlled environments, including simulation & emulation.

3. 3.

   Unique Capability Testbeds – High-cost infrastructures supporting specialized experiments (e.g., quantum networks, satellite communications).

These testbeds have several gaps in their functionalities:

• $\mathrm{■}$

  Scalability: Large-scale experiments remain challenging due to cost and infrastructure limitations.

• $\mathrm{■}$

  Data Accessibility: Many research questions, especially in AI-driven networking, require real-world data that is not publicly available.

• $\mathrm{■}$

  Technology Evolution: Rapid advancements in 5G/6G and AI make it difficult for testbeds to remain relevant over time.

• $\mathrm{■}$

  Reproducibility and Observability: Unlike in physics, digital testbeds often struggle with experiment reproducibility and privacy-related constraints.

• $\mathrm{■}$

  Reconfigurability vs. Stability: Some research requires testbeds to be highly modular, while others demand a stable and reliable setup.

Moreover, several research challenges demand improved testbed capabilities:

• $\mathrm{■}$

  AI-driven network automation: requires testbeds with real-world telecom data and high reliability.

• $\mathrm{■}$

  Digital twin-based testbeds: essential for replicating complex, real-time network conditions.

• $\mathrm{■}$

  Open, decentralized networking research: enables investigation beyond industry-driven technologies.

• $\mathrm{■}$

  Large-scale software emulation: facilitates cost-effective scalability for networking experiments.

Based on the discussions the working group made the following recommendations:

1. 1.

   Testbeds should be designed to serve as scientific instruments, ensuring they support: (1) systematic experimentation and theory validation, (2) reproducibility and rigorous scientific inquiry, and (3) scalability and adaptability to future research challenges.

2. 2.

   Develop a research-first methodology for testbed design guiding researchers to (1) define research questions independently of available testbeds, (2) identify the necessary testbed characteristics (e.g., scalability, modularity, real-world data integration), then (3) determine whether existing testbeds can be adapted or if new infrastructures are needed.

3. 3.

   Establish large-scale and open research testbeds; given industry constraints on testbed access, academia should explore independent initiatives such as:

   • $\mathrm{■}$

     “Build Your Own Network” (BYONET) – A modular, open-source testbed for networking research, free from industry limitations.

   • $\mathrm{■}$

     Academic Cellular Network for AI Research – A dedicated testbed for large-scale telecom data collection and AI-driven networking studies.

   • $\mathrm{■}$

     Scalable Software-Based Emulation – Tools that allow large-scale experiments without requiring physical infrastructure.

The working group concluded that while existing testbeds provide essential functionalities, significant gaps remain in scalability, sustainability, and alignment with emerging research needs. Addressing these challenges requires better research question formulation, modular and reconfigurable testbeds, large-scale academic-led initiatives, and closer collaboration with industry. By adopting structured methodologies and leveraging lessons from physics and engineering disciplines, the research community can ensure testbeds continue to serve as powerful scientific instruments for future innovations.

The working group was focused on a range of topics on Improving User Experience (with testbeds). The participants were Terry Benzel and Paul Ruth – organizers, Jim Kurose and Adam Wolisz.

The initial topics for exploration included testbed evaluation, testbeds in the research lifecycle, i.e., how to make them a place where new technology is developed as opposed to mostly testing and evaluation of semi-mature research technology; eliciting input from the testbed user community early in the development of testbeds, i.e., how do testbed users and developers come together on needs for testbed services; how to balance research versus educational use of research infrastructures, identifying examples of specific types of testbeds in hardware and communications (5G/6G); balancing testbed access models on-site versus remote and sustainable testbeds and testbeds for sustainable IT. Note not all topics were discussed, given time limitations.

The discussion began by suggesting that the community needs a “catalog” of testbeds and some means of identifying features, and by posing the question if there is a way to loosely standardize or create an ontology of testbeds and can this lead to approaches to common (but not mandated interfaces). Some concerns were raised about the term “ontology” as possibly being too prescriptive.

A more general discussion ensued around defining models that could be reusable on arbitrary testbeds, though that might be too ambitious. To realize this vision, there is a need for basic building blocks that can be easily tied to sets of research problems. These need to be structured through layers of abstraction. Abstractions are best created by user communities for specific use cases. One approach to developing packages is to crowd-source through user-packaged and shared experiments. Shared experiments should include scenarios and stories that show how to create an experiment using these examples. As always, there are challenges in bridging user communities from the novice to the expert users. Each may need different building blocks and scenarios. There is no one-size-fits-all. Some newer research infrastructure projects are developing different use “portals” for the different user communities. To scale, there is a need for large project use cases integrated with people, processes, and design experts. These are not trivial recommendations and typically go beyond the scope of most research infrastructure project support.

The group then turned to the question of how research infrastructure developers/operators can attract user communities. Understanding who the users are and matching research infrastructure to different communities is important. Some are interested in low-level technologies, while others are designing (or using) applications. This is related to the initial discussion about building blocks, layers of abstraction, and ontology.

At present, each testbed engages in “traditional” approaches to outreach, including presenting at conferences, recruiting users to present at conferences, and a certain level of evangelism. Ideas to shift these approaches to more coordinated, interactive, and high-visibility recruitment include competitions, grand challenges, and laboratories for education classes. Recruiting, maintaining, and growing user communities is vital to research infrastructure success. It is strongly recommended that testbed funding should include resources for creating and marketing users and educational exercises/experiences.

The final point of discussion was an exploration of roles for research infrastructure/testbeds outside of research labs and educational institutions. In these cases, the demand needs to come from the organizations, but this becomes a catch-22: if they don’t know about research infrastructure, how can they understand how it fits in their workflow? In general, R&D organizations in major tech companies have or can develop research infrastructure. On the other hand, companies further down the tech innovation chain may not have cycles to develop and operate exploratory research infrastructure. We recommend adopting initiatives to extend publicly funded research infrastructure through private funding while maintaining both open and proprietary use. There are challenges in moving from testbeds as exploratory tools, to mature technology transfer and production systems. One approach to balancing these competing demands is to create a research infrastructure as a service model.

The discussion first focused on the issue that it is very difficult to run experiments with 5G/6G industrial requirements on academic research infrastructures, and how academia and industry can collaborate on 5G/6G research. One of the main issues is that the capabilities of testbeds built by academic researchers lag behind the capabilities even of current production RAN systems, let alone provide a platform for research on future generations of mobile networks. Academia thus struggles to access high-quality 5G/6G hardware for experiments. Most industry research in that area takes place behind closed doors. Open platforms like SLICES exist, but given their characteristics they are of limited use for industry. It is a very difficult question how to design testbeds that industry will actually use. Some operators, like Deutsche Telekom, are open to working with universities, but vendors are more hesitant because of confidentiality and proprietary technology. ITU-T has proposed distributed testbeds, but they are complex and hard to implement. That’s why many researchers rely on simulations, emulations, or small private 5G setups with OpenAirInterface (OAI) as a more practical solution.

While a potential idea for academia is to work with industry on industry-internal testbeds, this is not very realistic due to the sensitivity and confidentiality of such testbeds, and comes with many challenges like NDAs and limited access to new technology. One way forward is to form research partnerships and public-private collaborations to align research goals. A possible approach is to create shared testbeds that both academia and industry can use, with realistic network traffic to bridge the gap between experimental and real-world scenarios.

The second topic discussed was the role that universities can play in standardization, and specifically in 3GPP. 3GPP standardization is almost entirely controlled by industry and is highly driven by “political” and economic aspects rather than purely technical, making it hard for universities to contribute. Unlike Internet standards, which evolve through open collaboration, 3GPP operates largely behind closed doors. Institutions like Fraunhofer and EURECOM are involved, but most universities are not. Timing is a big issue—by the time academic research produces results, industry has usually moved on. Many academic testbeds are already outdated by the time they could influence the standard. To make an impact, researchers would need to look beyond current 5G setups and focus on what comes next, which is difficult. One possible area of contribution is the higher-layer Service Architecture (SA), especially in network automation, AI-driven management, and security. Open RAN (O-RAN) could also be an opportunity, since it promotes modular and flexible network design, making it easier for academic research to have an impact.

The discussion made it clear that it would be desirable for academia and industry to collaborate more closely, with better access to open testbeds and a stronger effort to keep research relevant to future 5G/6G developments. However, there are significant obstacles. Universities need to engage with industry early, work together on research plans, and align with standardization timelines. Public funding can help to some degree by supporting joint projects and open testbed initiatives. However, ultimately the biggest hurdle is the reluctance of industry to collaborate due to confidentiality concerns, the difficulty to match timelines and oftentimes unclear direct benefit of such collaborations. Academia needs to show industry why its research matters and find ways to integrate it into the development process.

The discussions of this theme had the goal to identify suitable criteria to evaluate and compare different testbeds. This includes the question of what are good metrics. Additionally, it was discussed how to evaluate plans for future testbeds.

A number of quantifiable criteria can be used to measure and compare the performance of its components and amount of resources, economic value, impact and quality of different testbeds. One also has to be aware that testbeds are highly diverse and serve different communities, which partly have largely different sizes, so quantifiable criteria must be used carefully.

While measuring the amount of resources and the associated economic value means using a limited number of well-understood metrics, a fairly large number of metrics can be used for measuring impact and quality.

One aspect of impact are scientific results. This includes the list of huge discoveries, which, however, is frequently empty. Relevant metrics to measure scientific results are the number of publications that cite a testbed (however, one must be aware that publication may be much later than testbed usage), the number of of citations of these publications (however, citations may take up with a long delay). From these citations, it is possible to calculate an H index for testbeds (while its usefulness is debatable).

Another aspect of impact relates to usage of a testbed. Metrics include the number of single users (active per month and cumulative), the number of organizations to which users belong, the number of experiments (active per month and cumulative), user retention (measuring the share of users that stick around), and “monthly active users” (MAU), which is a standard statistic for web platforms. The number of users can be related to the size of community addressed by a testbed, to derive the usage in percent within that community. Usage also relates to education activity, where suitable metrics are the number of educational events, and the number of students trained. Usage also relates to industry involvement, with number of industrial users; number of SMEs, and percentage of industrial usage as suitable metrics.

Another aspect of impact relates to artifacts available in the context of a testbed. Metrics include the number of software artifacts, including contributions to open-source (cf. GitHub metrics), and the number of experiments available. Further metrics can be derived to assess standardization relevance, based on supported standards (evaluation and prototyping), and the number standard contributions with artifacts in the testbed.

To assess the quality of a testbed, usability metrics can be used. An innovative usability metric can be defined by measuring TTFE: Time to first experiment, i.e., how much time does an average user need to realize a representative experiment.

As testbeds are highly diverse, different different categories of testbeds can be defined. It was concluded that different sets of evaluation criteria are needed for different categories.

Long-running experiments were the main topic of this discussion group. Experiments in this category typically run for weeks or months. Typical platforms for such experiments were the now defunct PlanetLab [2], or current platforms such as Ripe Atlas [3] and FLOTO [1].

We identified three areas for long-running services on testbeds: (1) observational studies, (2) monitoring, and (3) research infrastructure as a service provider:

Observational studies monitor networks, such as the Internet. The main purpose of these experiments is to record data for research. Typical parameters to observe are the performance of services, including temporary or long-term changes in performance, the reliability or availability of services, their energy consumption, or the human interaction with these services.

Monitoring is a different use case for testbeds. The observed parameters are similar to observational studies; monitoring applications typically check the availability and potential performance degradations of services. However, the main purpose of the data collection is not research but maintenance. Testbeds can be used to monitor internal and external purposes, i.e., to observe the services of the testbed or to monitor services hosted outside the testbed. Despite of the original focus on maintenance, the collected data may eventually be used for research.

Research infrastructure as a service provider uses the facilities of a testbed to host services. A simple service of a testbed-run service would be a webserver. Another notable example of such a service was CoDeeN [4], a content delivery network hosted on the PlanetLab testbed.

We further identified requirements typical for long-term experiments. The computing and bandwidth resources that are consumed by these experiments are typically limited. The energy consumption and the storage requirements for data recording can be significant. Long-term experiments can prevent the testbed from being switched off, increasing its power consumption. Specific APIs may be needed for specific measurement tasks, e.g., for energy measurements. At the same time, shared resources must be managed to ensure a fair resource distribution between the experiments running on a testbed. Based on these preconditions, we deducted recommendations for testbed providers to host this type of experiment.

A major difference concerns the usage of resources. Observational studies run over weeks and months, monitoring services or services on research infrastructures may even run for years. The runtime of these experiments may be even longer as researchers extend measurement times to increase the amount of collected data. Another typical usage pattern is periodically scheduled experiments that run their task according to a service schedule. To handle such experiments, the usage policy of a testbed should be defined clearly. A possibility to minimize interference with other experiments is an automated scheduler. Such a scheduler can shift execution times to less attractive time slots where the testbed usage is lower. Another recommendation is to make the experiments reproducible. The process of reproducing experiments should be fully automated to make it as easy as possible [5]. This automation can then be used to recreate the measurement process on the testbed without needing the assistance of the experimenters.

The nature of long-running experiments requires a different approach to data validation. Typically, data is validated at the end of an experiment, when data collection is complete. However, for long-running experiments, the collection process may never be complete. Therefore, we recommend shifting towards an ongoing validation process that constantly or periodically checks the validity of collected and processed data. This validation process must notify the experiment in case of an error during the runtime of an experiment. We further recommend informing the experimenter about the resource usage or the underutilization of resources. This serves two purposes: it incentivizes responsible use of resources and helps identify potential errors.

The observational data measured by long-term experiments depends on the current state of the observed system. This state of the system, e.g., the Internet, is constantly changing, making the observed data unique for a specific point in time and vantage point. The data can not or not easily be recreated, which gives an inherent value to the measured data. The analysis of data and publications based on this analysis create additional value for scientists that is recognized and incentivized in the scientific community. However, this process prevents the sharing of data before a publication is accepted. To mitigate this problem, we recommend that high-profile conferences and journals should accept publications on data sets to motivate people to share data as early as possible. We further recommend sharing intermediate data sets that can be extended over time.