Computer Science Methods for Effective and Sustainable Simulation Studies

Parametric verification for stochastic models can be expressed as checking the satisfaction probability of a certain property as a function of the parameters of the model. Smoothed model checking (smMC) [1] aims at inferring the satisfaction function over the entire parameter space from a limited set of observations obtained via simulation. As observations are costly and noisy, smMC is framed as a Bayesian inference problem so that the estimates have an additional quantification of the uncertainty. In [1] the authors use Gaussian Processes (GP), inferred by means of the Expectation Propagation algorithm. This approach provides accurate reconstructions with statistically sound quantification of the uncertainty. However, it inherits the well-known scalability issues of GP. In this paper, we exploit recent advances in probabilistic machine learning to push this limitation forward, making Bayesian inference of smMC scalable to larger datasets, enabling its application to models with high dimensional parameter spaces. We propose Stochastic Variational Smoothed Model Checking (SV-smMC), a solution that exploits stochastic variational inference (SVI) to approximate the posterior distribution of the smMC problem. The strength and flexibility of SVI make SV-smMC applicable to two alternative probabilistic models: Gaussian Processes (GP) and Bayesian Neural Networks (BNN). The core ingredient of SVI is a stochastic gradient-based optimization that makes inference easily parallelizable and it enables GPU acceleration. In this paper, we compare the performances of smMC [1] against those of SV-smMC by looking at the scalability, the computational efficiency and at the accuracy of the reconstructed satisfaction function.

Darema in 2000 as a paradigm where measurement data from an operational system is dynamically incorporated into an execution model of that system, and computational results from the model are then used to guide the measurement process. Independently, Symbiotic Simulation [2] paradigm proposed in Dagstuhl Seminar on Grand Challenges for Modelling and Simulation in 2002 solves the what-if problem by having the simulation system and the physical system interact in a mutually beneficial manner. Since their inception, many techniques have been developed to support DDDAS and symbiotic simulations. Digital Twin, originally proposed by Michael Grieves in 2002 and popularized in recent years due to the rise of IoT and AI, is “a real-time virtual representation of a real-world physical system or process that serves as the indistinguishable digital counterpart of it for practical purposes, such as system simulation, integration, testing, monitoring, and maintenance” [3]. Digital twin is a concept. DDDAS and symbiotic simulation techniques form the basis for the realization of digital twins. They emphasize on the interaction between the virtual and physical systems.

Moving beyond DDDAS and symbiotic simulation, data-driven and machine learning (ML) techniques should be integrated in various stages of M&S. For instance, in addition to use sensor data from physical system to calibrate simulation models, ML techniques can be used to extract useful knowledge and insight from the data to facilitate model development. Data analytics and ML techniques can also be used to manipulate or steer simulation experiments on the fly (see Figure 2).

Some examples of our recent works along this direction include: Our PADS’21 paper [4] is about how to use ML approach to create a car-following model (instead of using traditional physics-based model) and how to dynamically calibrate the model using online data. Our recent research work focuses more on using data-driven approach to improve performance of simulation execution and simulation-based optimization. Our PADS’20 & PADS’22 papers [5, 6] are about dynamically analyzing simulation state to determine level of details to be used in the model of a simulation entity during simulation execution. The objective is to reduce the simulation runtime while maintaining accuracy of the simulation results. We applied the approach to a semi-conductor manufacturing simulation. And our WSC’18 paper [7] uses an approach to dynamically predict the usefulness of a simulation run. If the results of a simulation run won’t contribute to the overall optimization objective, then the simulation run can be terminated early. In this way, the total number of simulation runs required in a simulation-based optimization process will be reduced.

Problems involving societies and their interactions with cybernetic systems in the context of physical restrictions represent a paradigmatic case of Complex Adaptive Systems (CAS) involving emergent behavior and micro-macro loops.

The study of dynamic CAS lack analytical solutions thus requiring simulation as the only means for quantitative research.

If we add a layer of goal-seeking governance to inform real-world policy-making, legitimately contradictory worldviews must be factored in. This paves the way to reaching what has been termed as Wicked Problems, those that do not accept “correct” definitions nor “optimal” solutions, but rather require discursive processes to reach a consensus about the models themselves.

We postulate Planning for Sustainable Egalitarian Development as an embracing, flagship case study that pushes the envelope of sustainable simulation, thus challenging the state of the art and practice of several computer science disciplines (HPC, model checking, validation and verification, visualization, to name a few)

The proposal includes building on control theory to include simulation models in the loop of decision-making processes, creating a simulation-assisted arena to experiment with interventions, thus obtaining “feasible future developments” and analyzing them with advanced visual analytics.

We envision a framework within which planners (human intelligence) and algorithms (artificial intelligence) inform each other to obtain better strategies to use the simulation models as demonstrators of feasible paths of development.

An advanced facility, such as an interactive and immersive visualization room for complex data and reactive simulations shall integrate and boost capabilities for participatory model-based design and evaluation of sustainable complex systems.

Advances in digital twin technology are creating value for many companies. We consider, from a sustainability perspective, how these digital twins can be better developed. At first glance, the energy consumption of the twin during its life cycle can be described as follows: $E_{total}=E_{design}+E_{local}+E_{networking}+E_{cloud}+E_{update}$. Decomposing this formula allows us to see several challenges in the design and operation of a digital twin.

The ASTRÉE analyzer [2, 3, 4] aims at statically proving the absence of Run-Time errors in critical embedded software. Following the abstract interpretation framework [1], it is based on a formal semantics of C. Then various abstract domains are available to abstract this semantics. The correctness of the approach is proven by construction. ASTRÉE has been used successfully to certify the absence of run-time errors in the primary flight control software of the A340 (2003), the primary flight control software of the A380 (2005), and a C version of the animatic docking software of the Jules Vernes ATV (for the International Space Station). Since 2009, it is commercialized by Absint Angewandte Informatik (Saarbrücken) and is used mainly in automotive software, but also, in avionic and nuclear software.

Universities faced a difficult decision in summer 2020: whether to reopen for in-person instruction despite the pandemic and how to protect students, employees, and the surrounding community if they did. Simulation was critical to this decision at Cornell University in the USA, which successsfuly reopened for in-person instruction in fall 2020 under the protection of an asymptomatic screening that tested all undergraduate students twice per week. This talk discusses key factors that helped simulation modelers earn credibility and influence: transparency, designing a testing strategy that was robust to the unknown, explaining analysis in a simple clear way, understanding stakeholders’ incentives, responsiveness to stakeholder questions, and a focus on providing value.

Modern energy systems are increasingly adopting Internet technology to boost control efficiency, which unfortunately opens up a new security frontier. As a result, extensive applications have been proposed to enhance the cyber resilience and security of those critical infrastructures. Incorporation of new technologies in such systems is very challenging because of strong real-time requirements, continuous system availability, and many resource-constrained legacy devices. Therefore, a testing platform targeting such cyber-physical systems is strongly needed for the research community to evaluate the new application and system designs and their impact on the power systems before the real deployment.

We develop a unique testbed, DSSnet [1], combining container-based network emulation and power system simulation using a novel Linux-kernel-based virtual time system. DSSnet enables the modeling of a modern power distribution system and simulates the Intelligent Electrical Devices that make it up. DSSnet also enables high-fidelity analysis by allowing real networking applications to run in the network emulator and interact with the power simulator. DSSnet is composed of the following main components: (1) a container-based network emulator, Mininet [2] that allows execution of real software and communication network applications, (2) an electrical power distribution system simulator, OpenDSS that enables power flow simulation studies [3], (3) a unique Linux-kernel-based virtual time system [4] for synchronization of the two sub-systems, which significantly enhances the temporal fidelity issues in ordinary co-simulation or hardware-in-the-loop testbeds; (4) two coordinators for interfacing with the cyber- and physical-side modules and the virtual time system; and (5) a distributed software-defined networking (SDN) control environment, ONOS [5] that provides high-level abstractions and APIs for power grid control applications to manage, monitor and program the emulated communication network.

One key challenge is synchronizing the execution of the power simulator and the container-based emulator. This is because all the processes in the emulator execute real programs and use the system clock to advance experiments, while the simulator executes models to advance experiments with respect to its simulation virtual clock. To address this issue, we developed an independent and lightweight middleware in the Linux kernel to support virtual time for Linux container [4]. Our system transparently provides the virtual time to processes inside the containers, while returning the ordinary system time to other processes. No change is required in applications. Next, we expanded the capability of the testbed with a distributed virtual time system [6] that enables processes and their clocks to be paused, resumed, and dilated across embedded Linux devices through the use of hardware interrupts and a common kernel module. The distributed system architecture uniquely consists of a common virtual time Linux kernel module and three communication channels, one for virtual time synchronization using general-purpose-input-and-output (GPIO) hardware interrupts, one for connecting the embedded Linux devices, and one for interfacing with the physical system simulation that performs an offline computation. Additionally, we modeled and analyzed the temporal error during non-CPU operations, such as disk I/O, network I/O, and GPU computation, and developed a barrier-based time compensation mechanism to enable accurate virtual time advancement with precise I/O time measurement and compensation [7].

In summary, we present DSSnet, a testing platform that combines an electrical power system simulator and a communication network emulator using a virtual time system. DDSnet can be used to model and simulate power flows, communication networks, and smart grid control applications, and to test and evaluate the effect of network applications on the smart grid.

Performance is one of the core motivations in the field of parallel and distributed simulation. Contributions for new methods and optimizations frequently rely on custom models, parametrizations, and baseline implementations. This makes a direct comparison between methods and approaches difficult. We present our vision and initial steps towards COMPADS, a benchmark model and repository for reproducibly comparing the performance of parallel and distributed simulators and their respective algorithms. COMPADS[1] is short for COMparing Parallel And Distributed Simulators. The first results include a novel deterministic-by-design synthetic benchmark model inspired by PHOLD and La-pdes. The benchmark output is a checksum that attests to the correctness of an implementation and its execution. So far, implementations exist for the simulators ROOT-Sim and ROSS.

The role of modelling and simulation in the scientific domains has had a long history and computational X is now a well-established area in Physics, Chemistry, Biology and more. During the last decade with the increase in detailed data, and the development of novel modelling techniques, the application of modelling and simulation has become more commonplace in the social sciences. In some cases, these models provide scientists and policymakers with unique ways to reason about sociological challenges (e.g., Polarization, Segregation and Inequality).

In this talk, I present a sample of current work [1] in which we develop models to understand the process of primary school choice and school segregation with the Municipality of Amsterdam. In the talk, I present an agent-based model where households face residential decisions depending on neighbourhood compositions and make school choices based on distance and school compositions. Using a global sensitivity analysis we demonstrate that the observed excess (the level of school segregation compared to residential segregation) segregation in schools occurs for a wide range of parameters and demonstrate that asymmetric preferences (residential vs. school selection) are not a requirement for excess school segregation.

Using this case study I highlight a number of challenges and opportunities for modelling and simulation within the social sciences. Firstly, the use of models for theory building and testing in social sciences, in particular how simple models with clear assumptions can demonstrate potential causes for social phenomena. Secondly, using techniques from simulation-based inference I demonstrate how novel calibration methods offer promising solutions to calibrate city-scale models of social dynamics using microdata. Finally, I present some real-world cases where a “digital twin” of the city can be used to answer important policy questions for the municipality of Amsterdam and help them statistically estimate the likely outcomes for different interventions.

I conclude the talk by highlighting a number of initiatives [2] within Amsterdam and the Netherlands where new computational infrastructure presents unique opportunities to conduct computational social science and modelling simulation at a city and country-wide scale.

Parallel discrete-event simulation (PDES) refers to the class of techniques and tools for efficiently running discrete-event simulation on parallel and distributed computing platforms. Applications of PDES include performance modeling and simulation of large systems. Simulation of large systems exacts high computational demand, and successful PDES efforts must be able to cope with both the scale and complexity of the target systems on modern parallel computing architectures. In this talk, we summarize some of our research efforts for developing and applying parallel simulation of various systems.

Traditional PDES research has been largely focused on examining efficient parallel synchronization algorithms and incorporating those in simulation tools for general applications. Previously, we developed DaSSF [1], a simulator implemented in C and C++ that incorporates a composite parallel synchronization algorithm [2]. The algorithm was extended to run on both shared and distributed memory architectures and was implemented in a simulator called MiniSSF [3]. We have also explored the use of scripting languages (including Python, LUA, and Javascript) to simplify model development. The simulator, called Simian, was shown to be able to achieve good, and in some cases, even superior performance by taking advantage of just-in-time compilation [4]. One latest effort was the development of a simulator with full-fledged support for the process-oriented simulation world-view in Python for fast development cycle [5]. Abraham Maslow once said: “If you have a hammer, everything looks like a nail.” These and other simulators have been used for simulations of large-scale computer systems and networks, including the Internet [6], mobile ad-hoc networks [7], high-performance computing interconnection networks [8], and parallel files systems [9].

PDES not only enables large-scale simulation of complex systems, but can also be incorporated with real-time and interactive simulation of large systems due to its superior simulation performance. We previously designed and implemented a real-time network simulator to run on parallel and distributed platforms [10]. With real-time simulation, simulated network protocols, such as TCP, can seamlessly interact with real network entities in a hybrid simulation and emulation setting [11]. One can also control and steer the network experiments in real time – for example, by injecting network events and observing the results via a remote dashboard during the live experiment [12].

Many large complex systems feature a huge number of components and processes that may inter-operate across multiple layers of the system hierarchy and at different time granularity. A fine-grained simulation may not be able to scale up to the required size even if PDES could achieve linear speedup. Solving problems in many cases does not require brute force. George Box once said: “All models are wrong but some are useful.” In this case, one must be able to use multi-resolution models to balance the trade-off between simulation performance and accuracy. A case in point is the hybrid network traffic modeling, which combines fluid traffic models (e.g., based on ordinary differential equations) and packet-oriented simulation to achieve faster-than-real-time performance for large-scale network simulation [13, 14]. Another example is network simulation and emulation symbiosys. To allow high-fidelity high-performance large-scale network experimentation, one can run a full-scale detailed network model on high-performance parallel computing platforms, and an emulation system, which executes unmodified applications in a virtual machine environment configured to represent the target system. Both systems need to represent the same traffic behavior. We applied model reduction techniques to scale down the model complexity both in emulation to improve its performance and in data exchange between the two systems to reduce the synchronization and communication overhead [15, 16].

We observe PDES research has evolved by leaps and bounds over the last three decades. Many PDES techniques and tools have matured in various domains, although the community continues to discover new techniques, tools, and applications, many coinciding with the emergence of big data systems, machine learning techniques, and data-driven applications. We predict PDES will continue to play an important, and sometimes indispensable, role in modeling dynamic and complex systems, in many cases though combining with new techniques in order to provide its unique capability in solving problems.

Mobile networks evolve roughly every ten years, each generation bringing new services and capabilities. By 2030, 6G will bring more capable, intelligent, reliable, scalable, and power-efficient communications. This will enable new applications such as holographic telepresence, collaborative autonomous driving and personalized body area networks [3]. The 6G era will help realize the hyper-connected world of people, data and things – the Internet of Everything (IoE). It will also enable a focus on small data, generated by the plethora of edge devices embedded in our everyday world. The IoE will create a need for efficient processing of massive amounts of small data and edge intelligence, a process where data is collected, analyzed, and insights produced near the end user. The promise of 6G and the emergence of edge intelligence will provide users with actionable real-time information. An extension of this concept is to also provide users with actionable real-time predictions. The intersection of edge computing, sensor networks, artificial intelligence and online predictive simulations enable a new vision called Simulation at the Edge.

Computer simulation has become an established method for assessing uncertainty by conducting what-if analyses across a variety of disciplines and domains. It allows us to observe the behavior of a system under different circumstances and to investigate the potential consequences of different actions or interventions without actually interfering with the system we want to analyze. During the Covid-19 pandemic, for instance, we could see that a great number of models was developed already in the first months after the initial outbreak of the crisis [1]. Figure 5 shows that models were developed for all different kinds of potential interventions and to investigate how they possibly might affect transmission processes and the overall dynamics of the pandemic.

In practice, however, many researchers experienced that their models and the results generated by their simulation studies were not considered by policy makers when deciding upon which interventions to pursue and implement. In addition to this, other shortcomings in the development and application of simulation models and the conducting of simulation studies could be observed. Most researchers, for instance, did not reuse existing simulations and instead have chosen to start developing new models and to conduct new studies, which affects responsiveness in crisis situations. Potentials reasons might be the limited availability of models, ambiguous documentations of the models and previously conducted studies, or a general lack of trust in simulations studies conducted by others. This leads to a series of questions regarding why simulations sometimes fail to support what they were actually intended for namely to provide valuable insights and to facilitate decision making processes.

In this regard, when we discuss how to improve the effectiveness and sustainability of simulation studies, it seems that there might be different perspectives on this very issue. A simulation that we as developers and modellers consider highly insightful, comprehensive, and trustworthy might not be convincing, helpful, or plausible for the decision makers the study was intended for. Therefore, when discussing the effectiveness and sustainability of simulation studies, we should not forget about the stakeholders that might be involved in the process of a simulation study. How can we make sure to develop and generate what decision makers actually need? Why do we prefer to conduct own (new) studies instead of reusing the models and results of others? And how can we improve the rigor and trustworthiness of our studies?

Computational science experiments and simulation studies often require significant computational and human resources, making it impractical or impossible to repeat (i.e., reimplement and/or re-execute) these studies for reproducibility purposes. Transparency, on the other hand, is arguably always desirable [8] and can be achieved by revealing and laying open the assumptions, general approach, computational methods, codes, parameter settings, runtime environments, and – last not least – the data sources used to conduct a study. Regarding the latter, precisely identifying the correct subsets of data that were used in a computational experiment is already a difficult challenge [3, 12].

Provenance (a.k.a. lineage) information captures the origin and processing history of data products [5, 7, 6], i.e., it is a form of metadata that provides the technical means to support transparency [10]. While the existing W3C standard PROV [11] provides a minimal baseline for exchanging provenance information, domain-specific extensions need to be developed by and for the community to capture more of the application context, the semantics of data and parameters, and assumptions of simulation models and scientific workflows [13, 16].

Transparent research objects [1, 15, 4, 10, 9, 14], with their data, computational, and provenance artifacts, and research papers (which tell the “science story” of a paper) will continue to co-evolve and should ultimately converge towards open, transparent, and reproducible research tales [2]. – Declaremus et calculemus!

Visualization and interactive visual analysis have been used to explore and analyze simulation data for a long time. At the beginning, results of single run simulation have been visualized. With advancement of storage and computing technologies, the models became more and more complex and, at the same time, ensemble summation or simulation experiments – multiple simulation runs using variations of the same model – became possible. In case of computational fluid dynamics simulation the ensembles contain a relatively small number of members. Due to large and complex spatial-temporal data and long computing time it is not feasible to compute many models. Wang et al. [6] provide a recent survey on visualization of such ensemble simulations. In cases where simulation can be computed fast it is possible to compute hundreds, thousands or even tens of thousands of simulation runs. Matković et al. [4] provide an overview of such approaches.

The main idea here is to deploy coordinated multiple views that visualize multidimensional parameter space and complex simulation results at once. Interaction is used to support exploration and analysis. The user can brush, i.e. interactively select a subset of data in any view, and the items correspond to the brushed subset will be highlighted in all views. The idea functions if the number of parameters does not exceed half a dozen, or so. A parameter space of a higher dimensionality requires many runs in order to be sufficiently covered. Interactive simulating ensemble steering can be used in such cases. A kind of automatic analysis can be integrated in order to guide the expert.

In the case of multi-model simulation, there is little available support from the visualization community. Simultaneous exploration of simulation results computed with models of different levels of detail remains a challenge for the visualization. Large data, hierarchical data structure, and a need for fluent switching of context depending on level (and maybe even providing the overall context across the levels at the same time) is subject of future research. However, based on successful deployment of interactive visualization in the simulation community up to know, it is plausible to reason that interactive visualization can become a key interface in navigating in complicated multi-level models and simulation results, and a great support in comprehending underlying phenomena for different stake-holders. Scalability often represents a serious problem in visualization for simulation. In case of multi-model simulation, scalability will also represent a challenge. Finally, provenance tracking will gain in importance, as user actions across multiple levels need to be stored and recalled on demand. Finally, Dimara and Stasko [1] recently reported on the missing link between user tasks in visualisation taxonomies (e.g., sensemaking) and the high-level task of decision-making. One of the key causes of this mismatch is a lack of interdisciplinary approaches. A close collaboration with simulation experts represents a valid approch to establishing the missing link.

In order to support future requirements from the simulation community we need further advances in interactive visualization. We expect the novel approaches to be inspired by several directions of interactive visualization, depending on the simulation methods and corresponding models, as well as on the identified explorative tasks. Besides coordinated multiple views, the promising pillars of future research are certainly focus and context techniques [3], comparative visualization [2], tree [5], networks and graphs visualization, as well as plethora of existing visualization for simulation methods and provenance tracking.

From the reliability in a wireless sensor network to the formation of traffic jams, spatio-temporal patterns are key in understanding how complex behaviors can emerge in a network of locally interacting dynamical systems. One of the most important and intriguing questions is how to describe such behaviors in a formal and human-understandable specification language. A possible approach consists in using formal methods and in particular logic languages. In this talk, we show how a logic specification can be used to specify and analyse complex behaviours of large-scale spatially-distributed systems. Furthermore, we briefly show how we can use the logic for parameter synthesis, anomaly detection and the automatic feature extraction from spatio-temporal data.

Introduction: The last 40 years have witnessed an explosion of methodologies and techniques related to speculative PDES [8]. Looking at the history of this research line, while performance aspects have always been paramount, each decade has seen its hot points. Roughly speaking, in the first decade, algorithmic solutions were proposed that enabled significant speedups with relatively little effort (see, e.g., [12]). The second decade began to propose solutions for the adaptivity of particular simulation aspects (see, e.g., [6, 10]). The third decade focused on effectively supporting models’ programming by hiding the complexity of Speculative PDES (see, e.g., [23]). The fourth decade showed the technique’s robustness even on new hardware/software architectures [2, 13, 21].

We can conclude that Speculative PDES brings together effective solutions and methodologies for executing many classes of models, offering significant speedups and making tractable problems that otherwise would not be.

However, these results do not appear to be generally exploited. Market penetration of these techniques is severely limited and often, with some exceptions, confined to the academic sphere. The research community has already shown that there are various scenarios in which exploiting speculative PDES can bring benefits (e.g. Agent-Based Models [1], Spiking Neural Networks, SNN [20, 16], Traffic [11] and Hardware Architecture Simulations [25]). However, the most widely used simulation software does not consider this methodology at all. To give a few examples, in traffic simulation, SUMO [3] is single-threaded; in SNN simulations, most simulators are sequential (Brian [22]) or, if parallel, are based on conservative/time stepped algorithms (NEST [9]). Architecture simulators, such as gem5 [4], are also sequential.

Over the past few years, we have been wondering what the reasons might be for this poor uptake of methodologies that the scientific community has proven to be effective in many fields. From these motivations, we have tried to identify some challenges that we believe are relevant to make Speculative PDES more attractive to other fields of research and industry.

Diversity and Heterogeneity of man-made systems increase rapidly and so do the costs spent for them. Measuring efficiency and effectiveness becomes more and more elaborate but is an urgent need. Development of new methods, models and technologies is needed to support analysing, planning and controlling. The quantity and quality of available data strongly increase and therefore facilitate the description and analysis of systems like health care. Bringing together necessary technologies is an enormous challenge.

Data-based Demographic models have to be combined with models for the spread of diseases. Dynamic modelling concepts must be parametrised with complex data sets from various sources. For system simulation, an important aspect is the possibility to implement changes inside the system, like interventions within the computer model, and analysing their effects. As a recent example see Covid-19 Modelling at TU Wien [1]).

On basis of experiences of the Austrian DEXHELPP Competence Centre for Decision Support in Health Policy and Planning [2], which started in 2014 a concept was developed how large, interdisciplinary teams can handle these complex processes in the future and what are similarities and differences between health systems and other complex man-made DEXHELPP developed. DEXHELPP developed an innovative research infrastructure with (1) a flexible virtualised health system, (2) methods to cope with data, (3) an adaptive analysis and simulation methods pool and (4) stakeholder-oriented interfaces to enable researchers and other stakeholders to share data and methods for research and decision making. (see Figure 6).

“Ten Concepts to Integrate” were identified and first presented at Invited Talks at the University of Rostock [3] and University of Stuttgart [4]. The ten concepts are (1) Methods to Assess and Improve Quality of Data, (2) Potential to Integrate Missing Data, (3) Modular & Efficient Solutions, (4) Reproducible Processes, (5) Different Methods for Different Research Questions, (6) Comparability of Results, (7) Possibility of Communication of Advanced Simulations, (8) Open and Independent Solutions, (9) Priority for Data Security and Stake Holder Interests and (10) Broad Applications like Health System, Energy, Industry, Energy, Mobility and Infrastructure Planning and Usage (see Figure 7).

“Methods to Assess and Improve data Quality of Data” and “Potential to Integrate Missing Data” (summarized as “Data” in Figure 7) offer the possibility to find wrong data and correct them, ideally also during the simulation process. For this purpose, methods of interactive visualisation and statistics are used, among others, to preprocess data collected unilaterally, e.g. sensor data or reimbursement data or to link data that are unstructured or have different structures without direct linkage, e.g. [5].

To develop “Modular & Efficient Solutions” using “Different Methods for Different Research Questions” and maintain “Comparability of Results” (summarised as “Model” in Figure 7) includes sustainable, modular models that can be quickly adapted to new problems and concepts for comparing, combining and linking models (qualitatively and quantitatively) in order to demonstrate the benefits and limitations of the concrete model. As an example the DEXHELPP Population Model GEPOC (Generic Population Concept) was used for Covid-19 Modelling in order to guarantee rapid and quality-assured implementation, e.g. [6] also to compare it in the ECDC Covid-19 Scenario Hub [7].

“Reproducible Processes” and the “Possibility of Communication of Advanced Simulations” (Blocks 4 and 7 in Figure 7) are essential to guarantee the credibility and usability of the models. We need tools to manage and share data (e.g. [8] and models and to communicate not only the simulation results, but also the modelling process and model construction.

Last but not least, guidelines, standards that go beyond the concrete implementation are crucial (Blocks 8-10 in Figure 7). Here, the concepts of “Open and Independent Solutions”, “Priority for Data Security and Stake Holder Interests” and “Broad Applications” are crucial. The possibility of publication is limited, for example, by (justified) economic or data protection interests, which, however, leads to a lack of comparability of different models and thus jeopardises quality. This requires fundamental regulations such as those addressed in the General Data Protection Regulation and Data Governance Act. And clear and transparent processes are necessary for every project (even before the start of a simulation development) as well as the reuse of models is necessary to ensure quality and sustainability over time.

It is planned to publish the process description and examples as a White Paper within the European Umbrella Organisation of Simulation Societies EUROSIM [9].

With the large amount of data written by large-scale scientific simulations, efficient parallel I/O is critical. However, leadership class systems have a number of factors that make developing applications difficult, from complex storage hierarchies to different parallel filesystems with different configurations. The Adaptable I/O System (ADIOS) [1] aims to simplify I/O, while achieving high performance on leadership class supercomputers and providing a simple API for developers. ADIOS is an open source C++ library that also provides bindings in C, Fortran, and Python. ADIOS is used by applications in various scientific fields, such as plasma physics, earth science, and aerospace engineering. Figure 8 shows the data lifecycle of a large-scale scientific campaign. ADIOS can be used to transfer data from simulations or sensors to persistent storage, couple simulations to exchange data, provide checkpoint/restart capabilities, or transfer data to other systems for analysis workflows.

ADIOS has a single, simple API to perform I/O, whether data is to be written to disk or streamed over a network. ADIOS provides this capability through different engines that can be selected at runtime. File engines include a self-describing format called binary packed (BP), as well as a HDF5 engine. There are a number of engines that stream data over a network, such as the Sustainable Staging Transport (SST) engine which can be used for exchanging data between loosely coupled jobs running on either the same compute cluster or over a wide area network. There is also an Inline engine that provides in process coupling of simulation and analysis/visualization for in situ analysis. Regardless of the engine chosen, it is selected in an XML configuration file and ingested by ADIOS at runtime, so data can be handled in different ways without needing to change the source code or using different API calls.

Recent work with ADIOS has involved integrating it into visualization tools such as ParaView [2] and the Visualization Toolkit (VTK) [3]. The Fides library was developed for this purpose and provides a data model schema which enables users to describe their data in a simple JSON format mapping ADIOS variables to the mesh and field characteristics of a high-level data model. With ADIOS and Fides, simulations can use ParaView out of the box to visualize their data with post hoc, in situ, or in transit methods, without needing to write specialized adaptors or having a deep understanding of the VTK data model.

To the best of our knowledge, ADIOS is not currently being used in parallel discrete event simulations (PDES), but it could be useful in several ways for PDES. For instance, a recent US Department of Energy roundtable report [4] discusses the need for interoperability of multiple simulators that is portable and efficient on emerging platforms. ADIOS can be used for coupling simulations and provides portability and efficient parallel I/O.

The roundtable report also discusses incorporating machine learning (ML) frameworks into PDES engines/models. Some future work planned by the ADIOS team is to extend Fides metadata and provide data transformation utilities for ML data, which would enable ADIOS codes to integrate ML methods with their simulations. In addition to extending Fides metadata for ML data, we would also like to extend it for non-spatial data such as performance data and provide Python bindings. This would enable the use of other visualization and analysis tools besides scientific visualization tools like ParaView.

A variety of phenomena (such as the spread of diseases, pollution in rivers, etc.) can be studied as diffusion processes over networks (i.e. the diffusion of the phenomenon over a set of interconnected entities). There are different methods for studying diffusion processes, but most are based on various kinds of entities that are interconnected (networks). Two main approaches have been used to model the diffusion process over the network: Differential Equations (DE) and Agent-Based Modeling (ABM). The main advantage of nonlinear DE is that they can include a wide range of feedback effects (i.e. how the current value of a parameter of the system affects its future value, as in closed-loop systems). However, when they are used to study diffusion processes, one typically needs to aggregate nodes into fewer states or categories. Instead, ABM uses different attributes in each category, and different nodes in the same category may have different behavior. The network structure is clearly defined, and the behavior of each node is modeled individually at an increased computational cost. Likewise, neither of these two approaches provides well-established modeling and simulation (M&S) mechanisms for incorporating diffusion algorithms into multiplex dynamic networks and running simulations. The limitations of DE and ABM to studying diffusion processes pose the following question: how can we study diffusion processes in Multiplex Dynamic Networks to overcome the limitations of DE and ABM? Is there a framework that allows us to do this? To answer these questions, we introduced a method to study diffusion processes in multiplex dynamic networks, maintaining separations of concerns in all phases of modeling, implementation, and experimentation. The results include: a) an Architecture to study Diffusion Processes in Multiplex dynamic networks (ADPM); b) a systematic Process to define, implement and simulate diffusion processes over such networks. We use Network Theory formal specifications to define the topology of the diffusion process, Agent-Based Modeling (ABM) to define the behavior of the entities involved, and a formal specification of both for simulation modeling. This research uses the Discrete Event System Specification formalism (DEVS) to define the formal simulation model. The Architecture (and development process) to simulate Diffusion Processes in Multiplex dynamic networks (ADPM), is presented in Figure 9. The architecture is generic and can be used to study several types of diffusion processes.

ADPM includes the Requirements and Assumptions Document of the problem; a Network model of the relations among components; an Agent-Based model of behavior; the Diffusion Abstract Model (DAM) (Figure 10), a formal representation based on the Network and Agent models; a Diffusion Computerized Model (DCM) of the DAM; the Simulation Logs and a Results Analysis Report.

The DAM is defined using a formal specification, DEVS, in our case. This solves some of the limitations of Network Theory, such as the lack of a formally verified simulator to simulate a diffusion process over the network. By including ABM, we can also define the behavior of both the nodes and links in the network. The formal DAM confers ABM rigor while separating modeling, verification, and experimentation. The main advantage of combining Network Theory, ABM, and DEVS is that we can use the most appropriate method to model the various aspects of the problem. Network Theory is well suited to model the relations between components; ABM is better suited to model behavior. DEVS provides a formal specification to define the whole model as components with hierarchical and modular specifications, which can be executed using well-established abstract simulation algorithms, which are proven to execute models correctly. This combination allows us to separate concerns and clearly differentiate each part of the problem, as well as separating models from simulation engines and experiments that are independent software entities.

Modeling is an important technique in many engineering disciplines. Modern modeling languages and tools allow developers to early concentrate on key aspects of the product and thus frontload quality assurance e.g. through simulation.

Equally important, explicit models of system requirements, technology independent function and architecture models, as well compact software and system component models enable reuse and variability management. Composition, tracing and refactoring assist evolutionary development and simulative quality assurance in a way that greatly reduces development cost for products of all kinds.

Using explicit models in appropriate modeling languages, like SysML or even domain specific languages, and an integrated, highly automated tool chain for construction, analysis and simulation is key to integrate all forms of system components. A holistic development approach needs a clear decomposition and decoupling and thus also well defined integration techniques.

Furthermore, models are a good basis for the construction of a digital twin, because a digital twin shares a lot of characteristics with a model. A digital twin is like a real twin: it is an active instance that interacts with the real system, allows to share real operative data, but also to simulate what the real twin would do and thus predict a systems behavior.

We examine the current state and problems of modeling for cyberphysical systems. In particular, we discuss how to make use of models in large development projects, where a set of heterogeneous models of different languages needs is developed and needs to fit together. A model based development process (both with UML/SysML as well as a domain specific modeling language) heavily relies on modeling core parts individually and composing those through generators to early and repeatedly cut simulative and productive code as well as a digital twin infrastructure from these models.

Recently, the ready availability of “big data” has led to the adoption of data mining methods by organizations around the globe, as they seek to sift through massive volumes of data to find interesting patterns that are, in turn, transformed into actionable information. Yet a key drawback to the big data paradigm is that it relies on observational data, limiting the types of insights that can be gained.

The simulation world is different. Simulation models are integral to modern scientific research, national defense, industry and manufacturing, and in public policy debates. These models tend to be extremely complex, often with hundreds of thousands of potential factors (inputs or embedded parameters) and many sources of uncertainty. A “data farming” metaphor captures the notion of purposeful data generation from simulation models using efficient designed experiments.

Efficient design of experiments are required because it is literally impossible to examine even moderate numbers of factors by brute force. When the first supercomputer with petaflop performance (a quadrillion operations per second) was launched in 2008, the New York Times stated that machines such as this had “the potential to fundamentally alter science and engineering” by letting researchers “ask questions and receive answers virtually interactively” and “perform experiments that would previously have been impractical” [3]. Fourteen years later, the “Frontier” leads the world as the first exaflop machine [8], capable of over ${10}^{18}$ or a quintillion operations per second. But let us take a closer look at the practicality of a brute-force approach. Suppose a simulation has 100 factors, each factor has two levels (low and high) of interest, and we decide to look at all combinations of these 100 factors. A single replication of this brute-force experiment would take over 40 millenia on the Frontier, even if each of the $2^{100}$ simulation runs consisted of a single machine instruction! Efficient design of experiments can break this curse of dimensionality where expensive hardware cannot; experiments involving 100 factors can be completed in hours to weeks even for simulations with runtimes of minutes or hours [7].

With a data farming mindset, we can achieve tremendous leaps in the breadth, depth, and timeliness of the insights yielded by simulation. Large-scale experiments let us grow the simulation output efficiently and effectively. Modern statistical and visual analytic methods help us explore massive input spaces, uncover interesting features of complex simulation response surfaces, and explicitly identify cause-and-effect relationships [7, 2, 4]. Yet despite the current benefits offered by data farming, opportunities remain for advancing both the practice and research of simulation studies. A brief list of a few opportunities and challenges follows: see [1, 6, 5] for more.

From the practical standpoint, decision makers in many areas (climate change, economics, public health, to name a few) face increasingly complex problems where computational models are better than simple analytic models at capturing the complexity of the underlying system. At the same time, decision makers are increasingly comfortable with computerized and computer-based decisions based on observational big data, such as machine learning, artificial intelligence, and the use of digital twins. The rapid evolution of data science means that a greater number of simulation and non-simulation professionals are becoming more adept at scripting, modeling, graphical and statistical displays. Decision makers may, similarly, be less likely to shy away from using model-driven data to inform their decisions if they are made aware of the potential benefits – particularly as they seek solutions to complex problems. By varying inputs in carefully chosen ways and exploring or building metamodels of the input/output relationships, we use simulation as an inferential (rather than descriptive) decision support tool. Rather than simply answering ‘what is?’ and ‘what if?’ questions, we can explore ‘what matters?, how?, and why?’

From a research perspective, the portfolio of methods suitable for addressing complex questions needs to be expanded. Further research is needed on multi-objective procedures; exploitation of parallel computing; adaptive sequential design methods; and methods that leverage the structure of inferential big data, rather than observational big data, for analysis and visualization tools. Regarding simulation optimization and other adaptive search techniques, it may be that we should be doing optimization on metamodels, rather than on the simulations themselves – and that we need automated ways of reoptimizing as these metamodels evolve over time. Adaptive sequential procedures are particularly important if a simulation study is viewed as an ongoing process, rather than a terminating one.

Systems operating in military operations and crisis situations usually do so in contested and dynamic environments with poor and unreliable network conditions. Individual nodes within these systems usually have an incomplete, local and changing view of the system and its operating environment, and as such optimizing how nodes communicate in order to improve decision making is critical. Reinforcement learning approaches have been very successful at solving problems in dynamic environments and in some cases when dealing with incomplete information. In this talk, we discuss the challenges of training and executing reinforcement learning approaches within such environments, in particular when considering communication middelwares.

Simulation-based inference aims to address the question of linking simulation models with empirical data by designing statistical inference procedures that can be applied to simulators [1]. Data assimilation is an inference method, in which the observed data are assimilated into the model to produce a time sequence of estimated system states [2]. Data assimilation was initially developed in the field of numerical weather prediction. However, simply inserting point-wise measurements into the dynamic weather models results in large instabilities in the simulation, rendering the forecasts meaningless. Hence, data assimilation is developed to initialize a model using observation data while making sure to maintain stability in the model by iteratively selecting the best system estimation.

The initial concept of data assimilation relies on the initialization of an existing model. However, changes in the observation data may also result from dynamic structural changes in the system. This means that the original simulation model built from the past system will be invalid; initialization of an outdated model will render the simulation results useless. Instead of assimilating the observation data into an existing model, we propose to build the model concurrently with the data assimilation procedure.

Considering a smart factory with Internet-of-Things (IoT) sensors to monitor physical events occurring in the factory. Process mining is frequently used to analyze operational processes based on event logs. In process mining, process discovery aims at constructing a process model as an abstract representation of an event log [3]. The goal is to build a model (e.g., a Petri net) that provides insight into the behavior captured in the log. A petri net is a class of discrete event dynamic system, which can be used to simulate discrete event systems, such as the manufacturing process in a factory.

Due to dynamic changes in the customers’ orders, there will be frequent changes to the production operations in the factory. A new approach to combine data assimilation with process mining is proposed to accurately model the dynamic manufacturing processes. By monitoring the sequence of physical events, these events are captured into snapshots of the event lists for each event’s arrival. Concurrently for each event, the real system performance is also measured, e.g., factory throughput. Process discovery will be performed on these snapshots to obtain the process models for each snapshot. Process enhancement will be used to enrich these process models by mining additional operational behaviours from the event logs, e.g., queuing or batching operations, to achieve a more realistic discrete event model of the factory. Each process model is simulated to obtain the system performance. Data assimilation will be performed by comparing the simulation performance with the real system performance and selecting the process model with the best estimated system states. By iterating discovering new process models and selecting the most accurate process model, the proposed approach is able to adapt to changes in the system structure while producing a calibrated simulation model at the same time.

Modeling means organizing knowledge about a system of interest. In the talk, we will extend this citation, which has been attributed to Bernhard Zeigler, to the various artifacts of the modeling and simulation life cycle, as well as the life cycle itself. Thus, we will consider not only simulation models, but also models about simulation experiments, simulation methods, or entire simulation studies. With the subject of modeling the means of modeling varies as well, and includes approaches as diverse as domain-specific languages, formalisms, meta-models, ontologies, and logics. We will exemplify the interrelations between the subject of the model and the modeling approach. The second part of the talk is dedicated to the challenges that developing and applying model-based approaches face in modeling and simulation studies. Examples of such challenges including expressivity trade-offs, computational efficiency, reusability, accessibility, explainability, and evaluation are discussed.

Indoor factors like thermal, visual, acoustic, and chemical exposure all impact occupants’ wellbeing. One of the main approaches to improve wellbeing is to have continuous monitoring of the building. Similarly, building design issues or technical flaws in the building software can have negative impact on occupants’ health [1]. Modern building systems include sensors, actuators, and control devices including a tight coupling of hardware and software features. Advanced building systems collect data to improve building performance, operation, and maintenance. Building systems today use a variety of state-of-the-art equipment, such as embedded hardware, wide-area connectivity, and software for decision making [2]. We use cloud computing to increase collaboration among various building stakeholders and to display real-time data from buildings systems for performance and maintenance analysis. Using cloud services, designers can store large Building Information Models (BIM), historical data or simulated data. BIM allows managing the digital representation of building components, their digital geometry, metadata, relationships, and the parametric rules to manipulate them [3]. We propose a system architecture and a workflow called SUSTAIN (Sensor-based Unified Simulation Techniques for Advanced In-building Networks). SUSTAIN integrates fault tolerance models, a text-mining framework and BIM to improve building system reliability. The overall software architecture is shown in Fig.11.

The data collected from SAFE is integrated into BIM through Autodesk Forge, a cloud-based platform. Autodesk Forge allows to store and access various BIM and includes an integrated visualization component to visualize the real-time data through various web services. The BIM displays the sensor locations and ties the timeline data to its location. The aggregated data is represented as a shader to the room volume depending on the data parameters like temperature, humidity, or CO2.

The VSIM BIM is a replica of the actual building with all the building elements (windows, doors, desks, etc.) and building systems (sensors, HVAC, etc.). We show a case study using temperature sensors with an operating range 0°C – 50°C (with an accuracy of ±2?). We conduct simulation studies with different temperature readings within the measuring range were used. To evaluate the SAFE framework, faults were deliberately injected in both simulation and experimentation using different scenarios. The BIM of the lab was used to fabricate a maquette at a scale of 1:20 using laser cutting on acrylic sheets and adding control hardware (Fig. 12).

This physical model mimics the real-world setting of the research lab, which consists of six workstations, a fixed window, an operable blind, automated HVAC, and lighting controls. Fig. 13 shows the integration of the various sensors. Although the physical model does not behave as the real-world setting (e.g., the temperature in the maquette becomes steady faster than in the real building; however, when they are both in steady state, their behavior is similar), the objective here is different. SUSTAIN is focused on the development of the building control software, the integration with 3D visualization, simulation and support in a cloud environment. The physical model permits conducting a variety of experiments useful during the software development cycle, and this can be done safely and without affecting the actual building operations.

Simulation experiments play a vital role during a simulation study, be it for calibration, validation, or exploration [1]. Many simulation experiments are used repeatedly throughout a simulation study or across related simulation studies. For instance, models are successively calibrated and validated after each major model refinement. Similarly, if a model is built by extension of a previous study, cross-validation experiments are carried out. And in the case where different models represent alternative hypotheses about a system, or the same model has been implemented for different modeling and simulation platforms, again slightly adapted simulation experiments are used to compare these model alternatives.

The Reuse and Adapt framework for Simulation Experiments (RASE) supports modelers in conducting these repeated experiments in a more systematic, effective, and efficient manner [2]. Based on predefined activity patterns and inference rules, key user activities are automatically detected (e.g., model refinements), whereupon suitable, previously executed simulation experiments (e.g., validation experiments) are selected, adapted, and executed in the context of the new simulation model, possibly reusing other information such as data or requirements. The framework is founded on the notion of provenance, i.e., information about the various entities (the research questions, simulation models, simulation experiments, simulation data, input data, requirements, qualitative model, assumptions, theories, etc.) and how they participated in or were generated by the diverse activities of modeling and simulation [3]. This information needs to be captured during a simulation study and stored as a directed acyclic graph based on the PROV-DM standard [4, 5] to be accessible to the pattern detection mechanism of the framework.

Currently, a new approach for capturing the provenance of simulation studies on-the-fly with minimal user involvement is being developed [6]. However, also the means for formalizing the context of a simulation model (provenance meta-data) need to be enhanced for provenance to be fully machine-interpretable. Here, we might take advantage of existing model-based approaches, such as signal temporal logic for specifying requirements [7], whereas, for the specification of other entities, such as assumptions or theories, no approaches yet exist.