Safety Assurance for Autonomous Mobility

The talk delves into the emerging challenges and recent advancements in the Verification and Validation (V&V) of autonomous mobility systems. Focused on enhancing safety assurance, the discussion encompasses a broad range of cyber-physical systems, including smart cities, medical devices, and autonomous driving systems (ADS). The research emphasizes the critical role of requirements verification, validation, testing, certification, and correct-by-design approaches. The talk provides a number of methods for safety verification of machine learning-enabled CPS, safe planning and control of heterogeneous multi-agent systems, and human-robot interaction. The team aims to provide formal guarantees on system functionality and performance under uncertainty. The presentation showcases tools and case studies, highlighting the importance of rigorous V&V processes in realizing the future of autonomous mobility safely and efficiently.

This presentation demonstrates how safety critical systems are designed, developed, and certified in the railways. Safety is about failing systems. Failing parts to consider are exposed: wrong specification, wrong program, wrong binary, wrong execution, bad hardware (hardware caonatains other functions/interface that described in the datasheet), failing hardware (entropy leading to dysfunctional gates, drifting clock, etc.), wrong environment specification, and wrong exploitation procedure. Failure, either systematic or random, have an impact on the behaviour of the system. Safety is about keeping the probability of catastrophic failure below a treshold. Safety demonstration has to convince an independent expert that the feared events are not going to happen more frequently than expected. CLEARSY is using formal methods and related tools at different levels (software, data, system) to complete this safety demonstration, in accordance with the safety standards. The introduction of ML-based technologies to enable autonomous driving is raising safety issues. AI is at the moment not recommended to develop the most critical function in the railways, the main argument geing the lack of explanability of the ML black box function. UIC (http://uic.org) has launched a project titled “New methods for safety demonstration” that is aimed at helping human certifiers to envisage the certification of functions developed with AI or using IoT/Cybersecurity. Several projects of autonomous trains have been initiated in France, some are related to existing lines (freight, passengers, high-speed) while others are targeting low traffic, regional lines with specific/adapted infrastructure. These projects are expected to contribute to the standards, based on the results obtained.

Learning-enabled systems promise to enable many future technologies such as autonomous driving, intelligent transportation, and robotics. Accelerated by the computational advances in machine learning and AI, there has been tremendous success in the design of learning-enabled systems. At the same time, however, new fundamental challenges arise regarding the safety and reliability of these increasingly complex systems that operate in unknown dynamic environments. In this tutorial, I will provide new insights and discuss exciting opportunities to address these challenges by using conformal prediction (CP), a statistical tool for uncertainty quantification. I will advocate for the use of CP in systems theory due to its simplicity, generality, and efficiency as opposed to existing model-based techniques that are either conservative or have scalability issues.

Proving the safe operation of autonomous driving is one of the biggest challenges in the automotive domain. Conventional test methods are not sufficient due to the large amount of test kilometers that are needed to argue about system safety guarantees. Therefore, scenario-based testing is a promising and widely studied method to address this challenge in simulation. In this talk, I will discuss critical scenario identification (CSI), one of the ways to perform scenario-based testing. We will refer to requirements from the safety standard ISO 21448 (SOTIF) and focus on discovering scenario “unknown unknowns” using CSI methods. We will mainly focus on intelligent testing methods within CSI, using optimization-based testing, and we will discuss open challenges, e.g., the selection of appropriate input parameters and the appropriate design of the scenario. The talk will contain illustrative examples and results to demonstrate the effectiveness of the CSI approach for the safety assurance of autonomous driving.

Benefits to automated vehicles are greater road safety, cost savings, more productivity and reduced use of fuels, thus contributing to a green future. End users are ready to invest in autonomous vehicles, however, 64% demand increased safety for autonomous vehicles. Customers expect that autonomous systems cause no dangerous situations and do not influence the traffic flow, however, most systems currently on the market cannot handle complex situations. So, how to make sure that the automated vehicle behaves correctly in EVERY situation? In my talk, I presented a framework for close-loop testing and verification of automated vehicles that are under development in the AVL List. I presented ways how to avoid expensive full factorial testing, by using the Active DoE approach, followed by a comprehensive overview of the co-simulation environment for its execution. The reliability of the test depends on the fidelity of the 3D environment, which needs to support close-loop testing and the possibility of creating rare events. I finished my talk with the progress AVL made in modelling sensors and pedestrian behaviour.

The talk covered several topics in the area of safety assurance for automated driving systems. Based on a model to reason about the interaction of required, specified and implemented behaviors as well as the definition of the Operational Design Domain, the purposes and scopes of two most prevalent safety standards for road vehicles have been discussed. Furthermore, the Pegasus-VVM framework for scenario-based safety assurance as well as a specific method for open context domain analysis have been introduced.

This working group discussed important problems in the field of autonomous robotics. The topics discussed can roughly be divided into problems that arise during design time and problems faced after deployment of an autonomous actor.

The discussion in this working group started off with problems that occur during design time. The first hard problems that have been discussed concerned the design and definition of concrete requirements and specifications, as well as assumptions that can be taken at design time. Clear definitions of safety and performance are often highly dependent on the specific application area. The group discussed several examples from industry, an example to highlight the intricacy of the problem are autonomous delivery robots in a japanese hospital. The robots are used to pick up and deliver e.g. blood samples and are therefore allowed to use elevators. Since the hospital is in an area prone to earthquakes it is of special interest that robots do not hinder the fast evacuation of patients and staff in case of an earthquake. The group put a special emphasis on the arising complexity w.r.t. the different application areas of autonomous robots. In contrast to the problems faced in autonomous mobility, an area in which the legislative body has already compiled a manifest that autonomous systems need to adhere to. The group discussed a second issue that arises during the design time with respect to simulation software, especially the robot operating system (ROS). The issue raised regarding ROS is its dependency on the scheduling of the underlying operating system. The simulation might not be faithful as it cannot ensure real-time execution due to this dependency. This issue has been discussed in the seminar group, we refer the reader to the abstract regarding discussed solutions.

The discussion continued with topics related to the deployment of autonomous robots. The discussed problems heavily relate to the problem mentioned above regarding the different application areas, especially under the presence of humans. The problem mentioned here is twofold: It is an open problem, how a robot should behave to be the least harmful towards humans in safety critical situations, and it is a formidable problem to incorporate arbitrary human behaviour, such that autonomous robots can behave appropriately. Additionally, the problem of reliable perception has been labeled as very interesting for industry partners.

Lastly, the group wanted to include security as a hard problem. The topic has initially been mentioned, but has not been discussed.

The group followed up on the discussion about hard problems for autonomous robotics by discussing potential solution methods. The complexity of generalizing specifications regarding safety and utility in the domain of autonomous robotics has been identified as a key problem. This stems from the wide range of application areas. Therefore, the need for formal definitions of Operational Design Domains (ODDs) per applications has been identified as key driver during the discussion. The group highlighted the different aspects that are to be taken into account, specifically with regards to human interaction. In order to be able to allow safe behavior there is the need for dynamic modelling of human behaviour. The group mentioned data-driven approaches as a solution. Apart from a model of potential human behaviour, specifications of the types of intended interaction with an autonomous robot needs to be part of an ODD.

The group continued by discussing methods that enhance the explainability through counterfactual analysis.

Regarding open problems related to the faithfulness of simulations, the group discussed how parts of the ROS can be formally verified, and the capabilities of ROS to allow real-time simulations.

The development of machine learning has created great promises, not only for autonomous driving but also for other domains. Within the railway domain, projects such as safe.trAIn aim to uncover the potential of ML-based perception systems for increasing the Grade of Automation (GoA) via enabling driverless regional trains.

When aligning railway applications with urban autonomous driving, one can find some challenges sitting on the common ground. The first is the positive contribution of ML in safety arguments. The second is the introduction of security attacks. The third is the offering of guarantees to cover sufficiently the environmental conditions. The fourth is the ability to explain why ML fails by generalizing a single prediction error.

Nevertheless, there are also some railway-specific challenges. The first challenge comes with the problem where trains are operated on high-speed and have a long stopping distance. Thus, precisely characterizing acceptable behavior by considering aspects such as safety margin is a concern. The long stopping distance also implies that the decision making requires (1) a different sensor suite that allows look ahead way further than autonomous driving, and (2) considering high uncertainties associated with prediction due to detected objects being very far. The final challenge is related to the use of synthetic data. In contrast to autonomous driving where obtaining data is relatively straightforward, for railway applications such as defect inspection or rail track covered by muds, they occur very rarely, implying a must to use synthetic data. Arguing whether the degree of simulation fidelity is “enough” remains a challenging issue.

In the intricate world of autonomous train safety, a blend of traditional and innovative solutions have been discussed which can ensure a secure railway environment.

Traditional safety measures like guard-rails, isolation protocols, and radio signals act as the backbone, guiding trains safely on their tracks and preventing collisions. These time-tested methods establish a reliable foundation for safe railway operations. Safety is further fortified through sensor diverse redundancy, where an array of safety-focused sensors act as vigilant guardians. Redundancy ensures that even if one sensor encounters an issue, others are ready to step in, contributing to fail-safe detection capabilities. This principle of sensor redundancy has been extensively studied and discussed in sensor fusion for autonomous car engineering utilizing different sensor technologies and their detection ranges. The introduction of telescope cameras for instance adds a futuristic dimension to safety measures, allowing us to “see the future” of train travel. This innovation poses interesting challenges, particularly in the context of Railway Vehicles (RV), and has been a subject of exploration in the field of railway safety research. In alignment with the philosophy of designing for verifiability verification processes and approaches can be streamlined such as when tracks are only designed as straight-line tracks. This design approach simplifies safety checks and addresses railway infrastructure design. Furthermore, if full specifiability can be reached then there is no need for complex Deep Neural Networks (DNNs).

Looking towards promising and innovative solutions online runtime monitoring has been identified as a beacon for ensuring safety in real-time operations. The emphasis on making these monitors as safe as possible is a foundational principle, underscored in safety studies on autonomous systems. This real-time vigilance serves as a safeguard against potential risks, ensuring the continuous safety of autonomous trains throughout their journeys. However, the advent of the autonomous train era brings forth new challenges, notably the domain’s distribution shift. Placing monitors in the safety loop requires thoughtful consideration, as an excessive presence might overwhelm the system. This delicate balance between effective monitoring and system efficiency is a topic to be further explored on autonomous system architectures. Striking the right equilibrium becomes pivotal in managing the distribution shift effectively without compromising safety.

Another aspect discussed is the adapting infrastructure as a forward-thinking strategy, heralding a promising future for autonomous trains. Rather than solely replacing the driver, reshaping the environment in which trains operate is a paradigm shift discussed on autonomous transportation infrastructure. This approach ensures a holistic enhancement of safety measures while leveraging existing infrastructure investments. The characteristic of the rarity of failures in autonomous train systems sparked the idea of injecting artificial failures to address the synthetic data gap. This proactive approach to failure simulation aims to bridge the gap in rare failure occurrences, ensuring comprehensive safety testing. Rigorous envelope protection, with an emphasis on minimum braking capabilities, establishes another robust standard for safety, ensuring autonomous trains possess the necessary safeguards to handle unforeseen circumstances. Finally, the integration of neuro-symbolic approaches, incorporating temporal stability, consistency across modalities, and physics-assisted techniques, is seen as cutting-edge methodology to filter potential safety issues in autonomous train operations. Neuro-symbolic approaches leverage a combination of symbolic reasoning and neural networks, offering a sophisticated methodology to enhance safety in autonomous train operations.

For aerospace system design, the discussion initially centered around the critical balance between safety and utility, the quantification of risk, and the pursuit of more modular and principled design methodologies. The discussion further centered around the multifaceted challenge of reconciling the inherent trade-offs between ensuring safety and achieving optimal performance and utility. Central to these discussions is the notion of risk, traditionally quantified as the product of severity and probability. This classical framework, however, prompts a reevaluation in the context of both offline design and runtime adaption of aerospace systems. The discourse underscored the necessity for nuanced conceptions of risk (such as the conditional value of risk) that are responsive to the dynamic operational environments and the uncertainties that pervade these phases. The discussion also focused on the complexities of flight certification, with risk-friendly and risk-averse approaches, and challenges due to the sim2real gap. Specifically, models in aerospace may be highly nontrivial and not correct due to lack of data. Another point of discussion was on academic strategic repositioning towards emphasizing the economic benefits of safety verification (“Safety does not sell”) in accelerating system development and deployment. Marie brought up another point of discussion, that of the regulatory landscapes of NASA, ESA, and private entities like SpaceX, the latter being in absence of regulatory frameworks. In the end, the conversation touched upon educational and research methodologies, discussing the value of integrating courses on assurance cases and system safety into the aerospace (or more broadly any engineering) curriculum. This is to equip future engineers with the skills necessary to navigate the complex interplay of safety, risk, and performance in aerospace design and operation and to prepare them best for real world challenges. Lastly, the discussion centered on the development of case studies tailored to the aerospace and space sectors that link academia and industry more closely (further bridging the gap between theoretical safety tools and their application in real-world scenarios).

In the working group dedicated to low-level control, we discussed the integration of various autonomous mobility components, such as perception, planning, and low-level control, and the types of guarantees that can be provided for safe execution. A key focus was on the architectures used in system design, including the distinction between functional and physical architectures and how structure is defined through interfaces. We explored the use of languages and standards such as AADL, SysML, and Autosar for specifying system architecture from various perspectives, including functionality, safety, behavior, timing, and enabling different types of analyses. Contract-based verification was highlighted as a critical method for ensuring system reliability. Challenges included integrating systems composed of many components, particularly when dealing with legacy systems and proprietary models, especially interfaces to low-level control through application programmer interfaces (APIs) that often may necessitate crossing cross-layer boundaries between high-level and low-level control, similar to cross-layer optimizations in network protocols. Overall, the discussion covered the need for a more flexible and adaptive approach to system design, one that can accommodate changing requirements and negotiations between stakeholders. Various techniques for software development and verification were discussed, including structure, composition, certification, schedulability analysis, modeling, and verification.

The discussion also touched upon the aerospace industry’s adoption of architectures defined by the customer, similar to Autosar, underlining the importance of system architecture in the design and verification process. The challenge of low-level control not being fully accessible to the control designer was discussed, emphasizing the gap created by hidden system models and the necessity for system identification, and the use of lookup tables (LUTs) to manage nonlinearities, as well as approximations of large LUTs with other methods, such as via neural networks. More broadly, there was discussion on verification of these types of systems, ranging from the classical designs in low-level control like LUTs, to the usage of artificial intelligence (AI) and machine learning surrogates for these and broader tasks, where robustness of these components are critical.

We delved into the complexities of managing systems with time-varying parameters, such as battery degradation and the effects of aging, and discussed the potential for systems to self-identify and monitor changes. The conversation also covered the concept of conducting verification not only a priori but online, through the generation of conditional evidence and dynamic assurances, so that overall system-level and component-level specifications are monitored for assurance during operation.

The ability to refine high-level architectures through methods such as optimization modulo theories was considered crucial for overcoming system integration challenges, particularly the issue of abstraction leading to loss of detail and mismatches at the low level. We debated the importance of establishing low-level metrics that aid in reasoning about system-level integration and verification, including considerations for common mode failures, logical sensor fusion, and the balance between safety and availability.

Risk modulation and data-driven approaches were discussed for measuring risk either fleet-wise or at the individual vehicle level, with considerations on scoring individual artifacts, Safety Integrity Levels (SIL), and blame analysis. Questions were raised about the systematic approach to decision-making, such as selecting driving routes based on safety assessments.

Finally, the session touched upon the role of falsification in continuous integration and continuous delivery (CI/CD) processes, underscoring the evolving nature of safety and verification in the context of autonomous mobility. The discussion illuminated the multifaceted challenges and innovative strategies involved in ensuring the safe execution of autonomous robots, cars, planes, and trains, with a particular emphasis on the crucial role of low-level control in the broader context of autonomous system safety, all of which will require collaboration in teams of varied expertise to address.

The main topic of the discussion was the perception module and its role in the safety of smart mobility applications. The perception module is a complex system that typically relies on multiple sensors (e.g. camera, lidar and radar in the automotive domain), which are fused together, sometimes even using additional information extracted from other sources such as maps. The type and configuration of sensors used for the perception is domain-specific. For instance, the railways domain relies more on the infrastructure sensors rather than the train sensors, due to the large breaking time for trains. Modern perception systems must also have predictive capabilities (e.g. predict the next action of the detected pedestrian). Finally, perception has to deal with multiple sources of uncertainty. The participants identified three sources of uncertainty: state uncertainty, classification of uncertainty and existential uncertainty. Perception contracts were proposed as a potential solution towards dealing with uncertainty and building safe perception with error bounds on the state estimation.

The participants noted that the problem of testing perception is quite different from operational perception. Testing perception is notoriously hard and opens many challenging problems – from the synthetic generation of realistic inputs to sensors to the integration of perception in closed-loop testing with the ability to catch rare scenarios. There is also the problem of storage when testing perception. For instance, recording one hour of raw sensor data in a car requires several TBs of storage. There is hence a need to pre-process and filter this data on the edge, before sending it to the cloud. When looking at the perception module in isolation, it is not clear what is the right set of properties to verify. The participants noted that testing perceptions shall be done on a system level, since the important part is the functional impact of a misperception on the overall system. Perception contracts were proposed as a potential solution for building safe perception with error bounds on the state estimation.

The discussion concentrated primarily on the crucial role of Machine Learning (ML) in effective planning, emphasizing its indispensable nature in trajectory planning. ML’s ability to consider the conduct of a multitude of actors in scenarios such as switching lanes in traffic jams is indeed invaluable. One of the core discussion points was the significance of re-evaluating our predictions about the environment. This point underscores the necessity for dynamically adapting to changes that were previously unforeseen.

In terms of alternatives and security, the group outlined the importance of having a Plan B to fall back on when assumptions are not met while planning. They also presented the idea of having logical safeguards in place, alongside a Simplex architecture, to ensure that the planning process is robust and able to handle unexpected changes or challenges. A substantial portion of the discussion was devoted to explainable planning, highlighting the need for ML and symbolic planning to work in parallel. The symbolic planner, they detailed, would be capable of elucidating decisions when inquired, an essential feature for understanding and improving the plan’s operation.

The group also shed light on how planning often involves juggling various objectives that might be conflicting. Addressing this challenge, the concept of minimal-violation planning was introduced, which emphasized having different property levels, treating safety and performance at distinct levels.

The discussion then directed toward handling risks and uncertainties, recognizing the crucial role of conformal prediction in this context. Motion Planning, as deterministic as it may appear, also contains a slew of uncertainties, often arising due to perception nuance. Therefore, developing strategies to evaluate and handle these risks plays a part in successful planning. The involvement of symbolic and sub-symbolic elements in end-to-end modular learning, which encompasses perception, planning, and control, was clarified. The group also elaborated on the utilization of Belief-Desire-Intention (BDI) agents to increase explainability and transparency.

Practical aspects like the costs to monitor, the timing, and the demands towards optimization engines were explored, with a particular focus on time budgets. The meeting acknowledged that there are numerous different solutions, and the challenge lies in how to combine them effectively. Emphasis was laid upon verification aspects and how to propagate results between modules—an essential part of improving and refining the planning process.

Lastly, the discussion proposed possible solutions, advocating for the use of all the ML resources, including black-box models, yet enclosing them within a safety net. This safety net could comprise Hoare logic-based controllers, safety filters, as well as rulebooks.

The discussion centered on the use of LLMs in the development of safety-critical autonomous mobile vehicles. During this discussion, we used ChatGPT to guide our discussion. We asked questions including: What are the hot topics regarding the use of LLMs in the development of safe autonomous systems (for mobility), what are specific applications of LLMs in the development of safety-critical autonomous mobility systems, how can LLMS be used in the development process of safety-critical autonomous mobility systems, what makes the use of LLMs in this domain unreliable and untrustworthy, what role can LLMs have in the development process of planning for autonomous mobility systems, and what are the dangers of using LLMs in this domain?

It is clear that LLMs bring both benefit and potentially create undesirable outcomes when used in the development of safety-critical autonomous mobility systems. Some of the positive ways that LLMs can be used are as follows. LLMs can improve the natural-language understanding of autonomous vehicles, potentially improving the safety of the interactions that the system has with the user. For example, responding accurately to verbal commands. They can also be used to provide clear instructions to passengers in an emergency. LLMs can be used to recognise and interpret traffic signs, signals and road markings. They can potentially even be used in adaptive navigation by assisting in dynamic route planning. One particular strength of LLMs in this domain is to provide fast incident reporting and documentation which can be used in insurance claims, compliance, and post-incident analysis. During the development process itself, LLMs can be used to generate requirement specifications and associated documentation. Other uses include the provision of a natural language interface for design and configuration. This would facilitate intuitive communication with system designers and engineers during the design phase. LLMs can be used to summarise regulatory compliance and standards documentation to help the developers to demonstrate compliance. Along this vein, LLMs can be used to carry out risk/hazard analyses by processing relevant pre-existing documentation. They can also provide useful ways of generating user manuals and training materials as well as improve communication with stakeholders. Each of these positive perspectives can contribute to the overall safety and reliability of these systems.

However, this all comes with some major caveats. LLMs can struggle to gain a deep understanding of context in real-world, dynamic situations and current documentation that LLMs are trained on is likely incomplete. Since LLMs are sensitive to ambiguities in natural language, misinterpretation of cues or context can lead to incorrect, potentially dangerous situations. The age-old adage of “garbage in, garbage out” holds true and LLMs are likely to inherit biases in training data that could inspire inequitable decision-making which would raise ethical concerns. They are highly dependent on the quality and diversity of their training data but these models may not accurately reflect real-world conditions, leading to unsafe responses. LLMs are vulnerable to adversarial attacks that could be exploited by malicious agents and compromise the safety of the system. The environments that mobile autonomous systems operate within are rapidly changing and LLMs may struggle to adapt to dynamic scenarios such as road closures. Although LLMs are great at producing explanations, they do not possess a deep understanding of the operation at hand, merely they produce reasonable explanations based on data they have seen. However, understanding natural-language and its nuances is not trivial and it is possible that LLMs may misinterpret instructions that are given by the user. Though LLMs produce explanations, the LLM itself is not transparent and does not explain how it made the decisions that it made, this lack of transparency and explainability can make it difficult to trust LLMs and the output that they generate.