Computational Models of Human-Automated Vehicle Interaction

It was 15 years ago that I first wrote a computational cognitive model of driving. The model was used to make performance predictions for a range of different dual-task interleaving strategies that a driver could potentially adopt when entering a number to a mobile phone while driving. This work was published at CHI 2007 and it provided a great foundation for a lot of future research, including Chris Janssen’s PhD thesis (who is now one of the organisers of this meeting).

At this meeting, I’ve been reflecting on three significant advances that have happened since 2007.

First, many modern cars have advanced driver support systems. This includes cameras that monitor the road so that the car itself can manage speed and lateral control. These systems have become quite wide spread, and show that the basic demands of the driving task are changing.

Second, mobile devices have got better. In 2007, the iPhone had only just been released. Now many cars have in-car displays to show route directions, and integrated voice user interfaces that can be used to play music or send messages. The secondary tasks that drivers can do are also changing.

Third, the basic science of modelling drivers has advanced. In 2007 we were just beginning to explore ways to use reinforcement learning methods to get our models to adapt strategies for driving and secondary task interactions. At the time we opted for a simple “black box” approach. Since this time the great AI Spring has delivered modern machine learning techniques that can be use by our models. The work being done by Jussi Jokinen and Antti Oulasvirta is now realising what we only hoped would have been possible 15 years ago. The models that we can develop are changing.

The aim of this meeting then is to reflect on the advances that we’ve seen in these three inter-related areas that are relevant for developing Computational Models of Human-Automated Vehicle Interaction: the driving task, the secondary task, and the basic science of cognitive modelling. I hope as a result of this meeting we can set out a research agenda for the next 15 years of change.

Modeling an environment or human behavior, in essence, is looking at the world as a computer. But the world is complex, and within the dynamism we operate, it is often hard to quantify the specific dimensions of interest that provoke a certain behavior we observe. A complex world emerges from a complex interaction from an unquantifiable explosion of parameters, which tend to make any mathematical solution intractable. The job of a cognitive/ computation model is to simplify this quagmire of parameters and attempt to make some sense of this complexity with some informed guess, or in other words, illuminate the black box. Empirical research can help in systematically investigating the hierarchy of effect of the different parameters.

In the world of interactions within automotive human factors, we can say a lot about situations and negotiations, but we are not able to draw boundaries on problems and definitively say how to solve them. Furthermore, when we attempt to answer these questions through models, the expected reaction of the environment (e.g. driver, pedestrian, vehicle, etc.) are often not captured. Applying this to the context of my current research domain of eHMIs (external Human-Machine Interfaces) that facilitate AV-road user communication, linear models tell us how pedestrians will react and how a car should interact. However, the interaction from a car can cause further behavioral adaptations in a pedestrian, which current models are often unable to capture. The concept of interdependence – collaboration, coordination, and teamwork – is a grand challenge that seems to emerge as a gap in the state of the art.

Specifically in the context of eHMI research, the biggest potential benefit of a modeling approach emerges when tackling the problem of scalability. The state of the art has been mainly confined to a “one-car-one-pedestrian” setup in conducting empirical research. This is primarily because when investigating scalability of interactions, even at the minimum viable condition of testing scalability (i.e. just two pedestrians), the complexity introduced leads to the previously mentioned explosion of parameters, which make systematic empirical studies untenable. A potential approach is therefore to use a variety of theory- and naturalistic-data driven models to identify and decode the interactions in multi-agent scenarios, and conduct empirical studies with a more informed, limited set of parameters.

Another application for models is as a filtering tool. If a set of data does not fit a proven model derived from data or theory, it can be used as an indication of an outlier, which can be applied in practice as an alerting system. However, model interpretability is an important and non-trivial part of the equation if a model is to be used in that way: a machine-learned black-box model needs to be interrogated by a surrogate interpretable/ cognitive model. Furthermore, the question “how do we evaluate a model” stays wide open. What is the metric of a good model? And what is the appropriate thing to model for? The present? The future? A specific, imagined future? Points to ponder.

For some applications, like real-time driver state detection, the power of black-box Machine Learning (ML) tools is indisputable over other types of driver models. These ML tools enable the predictive modeling of complex nonlinear systems and can accommodate the modeling of a large number of predictor variables and their interactions. However, such complex models also require large datasets, which may or may not be directly available to researchers.

Caution should be taken when applying ML approaches to small datasets that empirical researchers traditionally collect. Although data may be collected at high frequency from each participant, experimental/observational scientific studies are generally limited in sample size (i.e., number of participants). Traditional statistical modeling techniques (model building and validation) were developed for small samples, whereas ML techniques assume large amounts of data. For example, researchers who have data from an experiment may have to split their training/test datasets within participants rather than across participants. But this approach can create data leakage resulting in ML models being rewarded for identifying participants rather than signals associated with the phenomenon of interest.

Caution should also be taken when researchers are lucky enough to obtain large datasets from other sources (e.g., OEMs). Real world data is messy, especially if there is not much control on how data is collected/selected for model training – leading up to examples such as facial recognition algorithms that are inequitable. Empirical researchers who design their own data collection are in a good position to apply their expertise and not fall into the same pitfalls that others do, by questioning how the data was collected, whether it is representative, and whether correlations exist in the data.

Out of the many extremely interesting topics that were discussed, what is of greatest interest to me is the question of how to combine cognitive models with learned (ML-based) models. How can we theoretical/mechanistic approaches improve machine learning-based approaches and vice versa? An effective combination of both could be the key to addressing the accuracy vs. Interpretability trade-off in HCI. In my opinion, an interesting way to go is to explore how complex interactions can be broken down into multiple subtasks/modules that only describe a small part of human behavior (glance allocation, pointing time…). These modules need to be restricted such that they only operate within certain mechanistic/cognitive boundaries. By doing so one can understand the different mechanisms of human behavior that in their combination describe a “complete” human-machine interaction without sacrificing accuracy due to abstract (but theoretically sound) models that lack prediction accuracy. However, even though this idea might be tempting, each module would still be trained in a specific context which leads to the problem of generalizability and raises the question of uncertainty prediction. I think that the effective usage of large natural data can be a solution to this problem. Leveraging large amounts of data we can represent different driving situations and contexts that would lead to an increase in generalizability.

For decades, language models were rule-based and theory-driven. Language data is and has been plentifully accessible – models were not theory-driven in the 1960s because in-the-wild language data was not available, but because we didn’t have the compute power to make use of large amounts of language data. The last decade has been very different for language models, with very large models like BERT and GPT-3 which, by utilizing tremendous power to train the models, modelling language in a quite complex, data-driven way. These data driven models have surpassed theory-driven models in many language tasks. They have been deeply flawed in other ways however, because of attempts to generalize data-driven models when the sample of language data is not general or ought not be generalized. This has resulted in applications, like chatbots that use racist language, or applications said to be capable of making moral judgments despite much of the training data coming from a Reddit advice forum. As new types of in-the-wild human behavioural data becomes available for contexts like human behaviour in SAE Levels 3 and 4 of driving, it is crucial that we examine the biases inherent to our data sources – the things that either will not or ought not generalize – and to remember that these biases can be caried through in our models. We must choose model architectures which are sensitive to biases in are data and we must be careful in how generally we try to apply models trained on idiosyncratic human behavioural data.

Intelligent design of human-automated vehicle interaction requires computational models of the human driving behavior and cognition. Particularly holistic models that can explain and predict multiple aspects of SAE Level 0 human driving behavior simultaneously seem equally important as modelling tool during the design of driving automation systems and their certification, and as a component of the system itself. Unfortunately, there is still a lack in commonly agreed upon models or architectures that can bear such a title.

As extensively discussed during the seminar, there is a common request for open-access and agreed-upon datasets, scenarios, benchmarks, and competitions, and first plans emerged to address this issue in the future. I’d argue that this endeavor should be accompanied by attempts to define (and provide software support for) an open-access architecture for a holistic model of human driving behavior that could evolve into a kind of gold-standard of driver models.

For a recent starting point, [1] provide a framework for a unified visuomotor model of a driver’s lateral control during simulated driving that couples gaze and steering control in a three-layered architecture. It would be interesting to see this framework extended and harmonized with theories for e.g., longitudinal control, intention formation, route planning, and situational awareness, to name a few.

Such an architecture could start as a collection of (potentially competing) models that implement different aspects of human driving behavior (e.g., [2] two-point visual control model of steering) that could be plugged together and interfaced with the currently envisioned datasets, scenarios, benchmarks, and challenges in open-access driving simulators.

From the perspective of design and safety analysis of highly critical human-cyber-physical systems, like automated driving at SAE levels 4 and 5, the demands on epistemic validity and accuracy of human models running as human state predictors in the car might seem prohibitive and the search for appropriate human models elusive. In my contribution to the panel, I made the point that this may not actually be true: if the models generate predictions which permit reliable (yet probably pessimistic) inferences in case of uncertainty then these models can seamlessly be embedded into safety-oriented system architectures when resorting to techniques of safe planning and control under uncertainty. Control theory as well as formal methods in computer science have in the past exposed various fundamental approaches to rigorous handling of uncertainties, which the human models and their corresponding execution mechanisms would then, however, need to be able to interface to.

These approaches rely on comprehensive mathematical representation of uncertainties concerning system state and evolution, building on possible world semantics in that they represent a system by sets or distributions of possible states at any point of current and extrapolated future time. This may seem a minor variation from the currently prevalent state-based models, given that the latter frequently come in stochastic variants which implicitly define the required distributions. The interfacing problem, however, results from the execution mechanisms employed for computing state extrapolations over these models: simulation-oriented mechanisms generating state traces, even if they do so using means of randomized simulation for stochastic models, are not suitable, as approximating a distribution via massively iterated randomized simulation is grossly inadequate for embedding in hard real-time into, as necessary for an online mechanism. What we instead need are mechanisms directly computing distributions in a time-resolved manner.

We argue that using such mechanisms, quite unreliable or, more precisely, uncertain human models -in experiments we have used such with just 61% predictive accuracy- could effectuate significant safety gains in embedded control applications. Technically, the human model here acts inside the embedded control as an online proxy of the human. This proxy enables predictive evaluation of the consequences of possible control actions, thus permitting the technical system to pursue rolling-horizon model-predictive control that is provident to the human.

Such an online embedding of human models would, however, require that the models to be embedded feature rigorous real-time guarantees, that they can incorporate the (uncertain) state evidence provided by in-situ measurements, that they can exploit the latter for computing best-possible estimates of current state, and that the provide time-resolved distributional state predictions over reasonable horizons of the imminent future. All these are prerequisites for interfacing seamlessly to rolling-horizon model-predictive control. Current models or rather their execution mechanisms fail to meet these by primarily being targeted to simulation, i.e., to trace generation rather than to state estimation by distributions.

With higher levels of automation, the human’s role shifts from actively steering the vehicle to passively monitoring the system. However, as long as fully automated driving (SAE L5) is not available, humans will have to intervene in the driving task at times. Scenarios in which transitions are likely to occur should therefore receive special attention. For example, when exiting a highway, or when entering urban traffic. Starting with SAE L3 of automation, the human driver may disengage from the driving task and perform other non-driving related activities. When a TOR is issued, he or she may be out-of-the-loop and incapable of understanding the situation and acting appropriately to traffic events, especially in safety-critical situations with small time-budgets. In order to support the human during transitions in automated driving, we need to predict the driver state. Does the human have enough situation awareness to safely take over control of the vehicle? With situation awareness, I don’t mean being able to consciously reproduce knowledge of the current situation, but having a well enough understanding to make appropriate driving decisions – implicit rather than explicit knowledge. Computational models may inform us about driver state by considering metrics relevant to the build-up of situation awareness, for example eye-tracking (behavioral) or physiological data (e.g., heart rate). The question arises as to what data could or should be fed to the model in order to make valid predictions about the state of the driver and his or her ability to take over the driving task, and how the model can be validated if the variable of interest, the implicit understanding of the situation, is not measurable.

A key step to understanding driving behavior are the underlying cognitive processes while driving. With the help of computational cognitive models, we aim to not only better understand points of human failure but also provide theory-driven predictions for different driving scenarios. Combining theory-driven white box models like ACT-R models with data-driven machine learning techniques (“black box models”) is an intriguing approach towards creating predictive models which are usable in autonomous vehicles. In the Dagstuhl Seminar we identified key areas in which black box approaches are more useful either from a computational point of view (e.g., lower computation time in highly time-critical scenarios) or when it comes to estimating Driver states/traits. On the other end, white box approaches can be useful when the model needs to be robust to handle unseen scenarios or when the aim of the model is to understand the cognitive processes responsible for the behavior. Hybrid models, which integrate machine learning techniques to restrict the behavior of a cognitive model seem promising.

With a rapid transition to a new future of work, the traditional definitions of where, how and when people can get their best work done is continuously evolving. An important finding of the years of the pandemic showed that people appreciated not having to commute as they had to work remotely and found that they could rather use the time previously spent on commute more productively while working from home. Yet with hybrid work practices, many workers will be returning to some form of commute at least during some of their work week. A key research question is how workers can effectively use the commute time – be it for their productivity or personal wellbeing needs without jeopardizing driving safety. In hybrid work scenarios, this becomes an even more important question, as efficient use of time for personal and collaborative productivity and wellbeing will be paramount for workplace performance. I propose a research agenda looking at the challenges of making the car a temporary worksite, where people can safely and comfortably get work done or attend to their personal needs without the worry of “wasting time in commute”. I envision three stages – 1) fully manual (present day), 2) Level 3 where the driver is still in charge of driving with some autonomy support from the car, 3) fully autonomous – and the nature of work that people can get done will be very different from one stage to the next. The value of having three stages is that we can start experimenting right away and inform vehicles of the future of the general challenges of working in the car – limited attention span, environmental constraints such as motion sickness, lack of large workspaces – and extend and adapt to the unique needs of a particular stage. This work cross cuts both the domain of Future of Work, as well as Autonomous Vehicles and will facilitate innovations of how to get work done in non-traditional worksites.

It is an exciting time for the field of computational cognitive models of automated driving, and that excitement is palpable at this Dagstuhl meeting — even when attending it remotely. The field of cognitive modeling has been around for at least five decades (depending on how you count, but for example going by this commentary of Allen Newell [1]), and has seen a lot of progress. The last few years have seen a growth in the types of AI and modeling techniques that have been developed for and in Human-Computer Interaction [2]. Moreover, there is excitement to include these techniques in the Automotive domain [3]. A challenge is though that with the breadth of techniques, and the breadth of application areas it is easy to get lost in translation. Following a preceding Dagstuhl [4], we had already identified that it is valuable to distinguish between simulations of the agent, its environment, and scenarios [5]. Similarly, at this Dagstuhl we had various discussions on how to push the field forward: refine and integrate techniques for modeling into the automotive domain. My hope and expectation is that it will be a win-win-win situation. First, through modeling the automotive field will gain better insights into its users and can better design for it. Second, the social sciences and related fields will have an area to test their models, to see if they also work in more practical sessions and to identify where refinement is needed. Third, through this endeavor a solid scientific community will form. I can’t wait to see what the future brings!

Computational models are used to simulate and predict human behaviors by quantifying the relationship between research parameters and behavioral outcomes. In this process, precision of prediction and efficiency of the modeling process might be necessity for modeling, rather than capabilities. Once these fundamental components are satisfied, we can consider adaptability or expandability of modeling, which we can call compatibility. The first technical capability the model should have for compatibility would be sensitivity. It is about whether the model can be sensitive to changes in resource demand. The next one is selectivity. It is about whether the model can be sensitive only to differences in specific resource demand. For example, if task difficulty can be resolved by multiple resources (e.g., secondary task while driving), but the model has only visual components, it might not be able to describe the entire human behavior. Then, the research question would be whether the model can be expanded by having other resources, such as auditory (speech and non-speech) components. The next capability for compatibility would be diagnosticity. It is about whether the model can indicate when human behaviors vary and can indicate the cause of variation. For example, the cognitive architecture framework may not capture other types of human states–e.g., emotions, fatigue, trust, mind wandering while driving. So, the research question would be whether the model can be combined with other constructs to predict and explain other than cognitive constructs. Finally, in terms of precision of the model, as long as the model can postulate the same stimulus-response mapping, it should work. This is the original meaning of ecological validity. Therefore, the simulation does not necessarily have high fidelity software and hardware compatibility (which is more about external validity), but the important question would be whether the model can have the psychophysical similarity. These technical capabilities (sensitivity, selectivity, diagnosticity, and ecological validity) will be useful to assess the compatibility of computational models in future.

To ensure driving safety, efficiency, and comfort in mixed traffic environment with autonomous vehicles (AVs), these vehicles must be able to understand the plans and states of partner road users, and can communicate and coordinate their actions with them. With the increasing number of AVs being tested and operating on roads, external HMI (eHMIs) are proposed to facilitate interactions between AVs and other road users. Taking eHMI for crossing pedestrians as an example, many methods like user interview with images & videos, Wizard of Oz (WoZ), virtual reality, the Delphi method, on-road experiment, etc have been conducted in studies focusing on the effect of eHMIs. But different methods are subject to different biases, even yielding conflicting results. This problem should be carefully considered. As there are cultural differences in the way how goals and states are expressed by HMI as well as how HMI is interpreted by others, it is important to understand these differences to allow the integration of AVs into different cultural contexts.

The benefit of using computational cognitive models is that they force the modeller to make theoretical and practical assumptions explicit. For instance, in the case of semi-autonomous vehicles, understanding how the human driver adapts to the presence of different automatic driving assists helps to design safer and efficient driving. Moreover, models can be used to generate predictions of how drivers adapt to various design choices. However, modelling is time-consuming and hard, and involves a learning curve that may be unacceptable. In order to facilitate proliferation of computational cognitive modelling, the modelling workflow must be made usable. This involves creating tools for modelling, but also benchmarks for model testing, and “canonical scenarios” for trying out design and modelling ideas in familiar environments.

To build computational models for human-AV interaction, we need a lot more data. Of course, we have all heard that “all models are wrong, some models are useful.” To date the models I have seen in the space of human-AV interaction do not account for the cultural variation, contextual variation and temporal variation that we know for certain would affect any predictions for how people will respond to an AV.

Because of the complex and unpredictable nature of traffic with human road users, I argue that in order to be safe, driver should monitor the behavior of the car and the traffic environment up to SAE level 3 of automated driving (maybe even at level 4). This requirement is decreased as the automation level increases and that is actually a big problem as the driver’s attentional capacity is freed but not fully. SAE levels are also somewhat problematic in this sense as a real car given a level 3 status may not always function as a level 3 car should (e.g., always know when the driver should be alerted to take over).

Current driver attention monitoring systems are really impressive in their object and behavior classification performance but I would argue that it is not enough to monitor the driver or insides of the cabin to know if the driver is inattentive towards driving. Drowsiness is probably the only form of inattention that can be reliably detected by monitoring the driver only. The attentional demands of driving vary based on traffic situation, surroundings, upcoming situation, etc. and there is often so-called spare attentional capacity in driving. And even more when the level of automation increases. But how much, that is the question? Definition of these requirements might also get harder when we go up in the level of automation. As this task is fairly complex, we would need computational models for this.

For these reasons, and because driving is a safety-critical context, we’ll need also prescriptive computational models, not only descriptive and predictive. For instance, we would need a normative criterion to define when the driver is inattentive, depending on situational and driver-specific variables

A driving scenario that promises to be important for human-automation interaction in vehicles in the near term is bumper-to-bumper traffic. In this scenario the vehicle is relatively easy to control by automation, and there is a relatively low risk of injury in case the automation makes a mistake. At the same time there would be a large positive impact on the driver if they could use the time that the vehicle is in bumper-to-bumper traffic to engage in non-driving tasks.

How can drivers use their time if the car can drive itself in bumper-to-bumper traffic? The answer is: short bursts of non-driving activity with rapid returns to driving. The reason is that bumper-to-bumper traffic doesn’t last forever – it’s likely to often take only minutes. And transitions back to driving will need to be very quick, on the order of seconds, not minutes [1].

It is likely that those short bursts of activity will very often involve manual-visual tasks. We recently conducted a time-use study with 400 knowledge workers who commute by driving [2]. We asked them what they would like to do in a future, safe automated vehicle. They provided us with tasks they are interested in, and we assessed the tasks in terms of the need for various cognitive resources (using the Wickens multiple resources model). We found that our participants wanted to do more manual-visual tasks, both for work and for personal tasks. Thus, we expect more typing and browsing, in contrast to silent reflection or listening to music.

Computational models (including but not limited to cognitive models) are playing an essential role to enable safe automated driving and to make it a reality on tomorrow’s roads. We are looking at four challenges in the field.

1) Modeling driver engagement and driver availabilty.

Driver monitoring becomes a must in the homologation of assisted and automated driving. We need real-time models estimating whether the driver fulfills his supervision task in Level 2 (engagement) and whether he will be able to take over if a request occurs in Level 3 (availability).

2) Take-over performance models.

Respecting the current status of traffic, vehicle, and driver we need models to predict the temporary and quality aspects of the take-over activity and to decide if his/her actions are intentional and supportive for safe driving.

3) Shared control / cooperative driving.

As humans are still needed for subtasks in automated driving (e.g. for deciding on lane changes and other tactical aspects)driver’s computational models should be used during system development to analyze and validate the successful cooperation between driver’s actions and automation control.

4) Good driving behavior as a role model for automation.

Behavioral models describing the human driving process in traffic are a good way to serve as a role model and to be mimicked by future automated and even autonomous driving systems. Separating “good” and “bad” driving can help that autonomous driving will be able to realize a chauffeur-like safe and comfortable driving style.

Computational models of human cognition and behaviour have applied uses in, and in some cases will be indispensable for, development of safe and human-acceptable vehicle automation. At lower levels of automation (e.g., SAE level 2-3), models of how humans monitor the automation and take, receive, or share the vehicle control can be used (1) as part of online, real-time algorithms to infer driver state and predict driver actions, and to adapt vehicle behaviour accordingly, or (2) in offline computer simulations to evaluate different automation design alternatives. The same division into use in both online algorithms and offline testing applies also for higher levels of automation (e.g., SAE level 4-5), but then with respect to modelling the cognition of behaviour of other road users around the automated vehicle. One exciting prospect and grand challenge, across all of those application areas, is to find the balance and combinations between mechanistic/cognitive models on the one hand, and data-driven/machine-learned models on the other, to enable extensible models that users (automated vehicle designers/engineers) can generalise to new contexts and data sets, without direct involvement from the original model developers.

Current thinking around human-machine interaction in autonomous vehicles is often through the lens of untrained everyday users. We may often think that our models of human behavior are generalizable. However, after training, people may develop new ways of acting, especially with autonomous systems. ,A potential area of work that is underexplored in the space of autonomous vehicles is how trained operators will act. For example, a bus operator or a semi-truck driver will have more training in both driving and with the systems they are using. As such, cognitive models for these people may be different than those of a lay population. The research community should consider research on trained operators and should capture data in experimental settings and develop models that are specific for them. Such work may lead to better predictive models that can be used in real-world autonomous vehicle sooner than in vehicles with a lay population. This line of research may also identify good mental models that could then be taught to lay drivers to help improve their interactions with autonomous vehicles.

“Simulation Intelligence” brings together first principles models, probabilistic programming and machine learning. It has a lot of potential for supporting a more principled approach to creation of computational models, which can help support and direct scientific progress in the field. Moving from correlational fits to data towards causal models which really represent the actual behaviour will be vital for reliable behaviour in novel contexts. Using forward and inverse inference mechanisms, as is common in other areas of science, gives a potentially more reliable way of formulating the problems, and managing models for components of larger models. Closed-loop aspects are also critical – they affect how we acquire data, and also provide challenges associated with the interdependence of human behaviour on the dynamics of the vehicle (e.g. crossover models), but this seems to be underconsidered at the moment.

How do people interact with computers? This fundamental question was asked by [1] with a proposition to frame it as a question about human cognition, in other words as a question of how information is processed in the mind. Recently, the question has been reframed as a question of adaptation: how do people adapt interaction to the limits imposed by cognition, device design and its environment? The core assumption of computational rationality is that users act according to what is best for them given the limits imposed by their cognitive architecture and their experience of the task environment. The theory can be expressed in computational models which explain and predict interaction and be therefore used for design and adaptive systems.

In many discussions I have had over the last years with my students (in the interdisciplinary program User Experience Design), within my research group but also with colleagues (e.g. in the context of Dagstuhl Seminar 16262), it has turned out again and again that there is a different understanding of what a model is and how or for what it can be used. The main reason was that we have different backgrounds (engineering, computer science, design, social sciences, psychology). We need to find ways and means to make sure we (“the system designer”) mean the same thing… Predictive models could be a common basis – but of course they are only a simplification of real behavior [1]. If we focus on human-machine (or human-AV interaction), it is particularly necessary that both parties (human, machine) understand each other and can mutually assess how far the partner is trustworthy in a certain situation [3]… And this is particularly important in the recent research field “Cooperative Driving” – where automation and human work together as team players [2]. In a team, each agent must be informed about the other agents’ current activities, their strategies, the status of their efforts, if they are having problems, and their intentions for planned actions. The Object, View, and Interaction Design (OVID) model [1] (Roberts, 1998) contains three design models based on Don Norman’s notion of cognitive engineering [4] and clearly visualizes the problem: We cannot really design a user’s conceptual model, as this is highly inter-personal (but also intra-personal), based on experience, and exist only in the brain of a person… Furthermore, user interface and vehicle designers as well as user experience practitioners are often challenged with the question for which user group they are designing for – each with different needs, different interests, and very different ways of interacting with technology. In vehicle production, we do not have the luxury of focusing on only one group (at least not so far), i. e., designing vehicles for specific age groups or cultures, that’s why interaction designers and engineers must learn to recognize and reconcile the needs of their main user demographics. This problem will remain even with fully automated driving when using the car as a place for relaxation, entertainment or work. Defining characteristics, differences, and tensions between individual user groups might help to account for different individuals. An important question in this regard is, whether or not there is a single system suitable for all (or at least most) customers and stakeholders. Is it axiomatic to target user groups differently? (HMI and system configuration allow to…). Dagstuhl Seminar 22102 was the perfect place to discuss with all the participants from different backgrounds (my access is hypotheses-driven experimental research) the possibilities of combining existing models, extending/improving them, or developing new models to better represent the wide variety of variants in human-AV interaction/cooperation and human diversity. I hope that results can be derived from this week’s collaboration that will help the community….

The next big challenge for computational models of human-Automated Vehicle Interaction is, from my perspective, to integrate more aspects and understanding of the human perspective. I do not think that one modelling method needs to capture all possible aspects of a scenario and related entities to be a good model. What is needed are good models that can capture specific aspects and a good concept that can integrate this information with emerging transparency – how different aspects have led to a specific understanding of the situations. A modular approach might be helpful to achieve this in case some module results have a symbolic form. What is needed is a tighter connection of human and technical system understanding. It is not sufficient to measure some mental states, it is also (in some situations) necessary to understand what the cause of the measured state is and how to address it. I claim for more research on models that anticipate the human driver in the dynamic situation and include mental representations e.g., for “Situation Awareness” or “Mental models” or specific “Expectations” about what the human considered relevant now and what is expected to happen next. This would enable a better mutual understanding and a more natural (less effortful) shared control of the car or the situation and to avoid misunderstanding.

The advances in automation and machine learning are changing the designs of in-vehicle user interfaces from static command-response “buttons & displays” to dynamic adaptive interfaces that adjust themselves to user behavior and preferences. From a modeling perspective, however, this can be a challenge to integrate these new “emerging behaviors” into the models. On one side, having simplistic models allow us to scale them to different scenarios and interaction forms, on the other side, it restricts our ability to precisely describe/predict the adaptive and dynamic interactions between the user and the automated vehicle.

Another challenge is answering the question of whom we are designing for and in which scenario? The first step towards that is defining the sides of the interaction, i.e., the human and the automated vehicle. Identifying the capabilities and responsibilities of each of these partners across different levels of automation can help us understand what factors are required to be considered in the models. For example, for an urgent takeover situation in level 3, factors that define users’ situational awareness, the urgency of the situation, and contextual factors might be determinants of the behavior. However, for a level 5 scenario, where the interaction goal is mostly to assure users’ comfort and wellbeing, the determining factors stem from users’ psychological needs and affective state rather than her situational awareness. It is, therefore, necessary to first define users’ tasks at each level of automation and in different scenarios. And then elicit the factors that are required to fulfill these tasks’ goals. This allows us to create a modular model of tasks, levels of automation, and required factors that assist us to identify the important elements that should be addressed in the design of in-vehicle user interfaces.

Computational models can help to ease the cost of experimental studies. Discussions have been developed under the consensus that various individual or external factors affect distribution of driving responsibility between the human and the automated vehicle. These factors range from personal preferences, trust, experience, social and legal norms to environmental conditions. While it is crucial that these factors are recognized, it may not be vital for computational models to include all possible parameters. It is still a matter of discussion whether their benefit is limited by their simplistic qualities. While some might consider that they might have to be complex enough to provide generalizability, they could still be helpful to investigate individual factors with limited complexity.

With the introduction of driving automation systems [1], the subtasks of the DDT are being performed by the (human) driver, by the driving automation system, or by both. This means that under certain circumstances, human and automation have become collaborative partners in executing (parts of) the driving task.

Although the view of driver and automation acting as collaborative partners has been well established, means to assess their teamwork are still lacking. Moreover, available evaluations are usually addressed either from a technical stance or from a human factors viewpoint, which does not comply with a general acknowledged view of a unified driver-vehicle system. This stance on teamwork and a description of our aim to evaluate driver and vehicle cooperation by means of a framework can be found in [2] .

Our work is currently being followed up by a roadmap – which is shared as soon as possible – aiming to deliver guidelines on the framework’s practical implementation, potentially supporting monitoring, evaluation and design activities concerning driver-vehicle cooperation.

This abstract is a call to get involved in our activities regarding the objective evaluation of driver-automation cooperation.

Six years have passed since my first Dagstuhl Seminar. At that time, the first deadly crash with an automated vehicle happened. It became clear that the trust relationship between drivers and automated vehicles will become a key issue for guaranteeing safety in the future since drivers are not necessarily automation experts. Thus, we intensively discussed the topic of trust in automated vehicles and proposed to research “personalized trust models” so that in-vehicle HMIs can adapt to drivers’ states, strengths, and weaknesses. A great variety of trust studies have been conducted and published since that seminar, revealing many relevant factors influencing this multidimensional psychological construct. However, a sophisticated trust model has still not been developed. Thus, I am very happy for the invitation to this modeling seminar, as I got the chance to (1) talk to modeling experts from my domain and (2) discuss with colleagues concrete, actionable steps to move towards the goal outlined above. I am confident that it will not take another six years to integrate all the past study results and develop an initial functional version of the proposed model.

In order to better understand the interaction process between humans and automated vehicles as well as develop adaptive assistance systems to support drivers, computational models that can precisely predict driver behaviors are needed. However, with the change of automation levels, there are challenges while developing computational models. With the change of driver’s role from SAE L2 to SAE L3, the focus of modeling changes from monitoring driver’s states, to modeling driver’s takeover reactions which are more safety critical compared to SAE L2. From SAE 4 or SAE L5, the transition of responsibility can be relevant. While modeling takeover process from SAE L3,indicators like takeover timing as well as takeover quality such as maximal acceleration and TTC, subjective measurement such as workload can be taken into account. Facing the complex scenario in the mixed traffic, the combination of cognitive approach with black-box models is needed, which can maximize the advantages of different models, but also ecologically save the cost of developing complex cognitive models. The cognitive theory, evidence or empirical research should be made use of to help to develop casual black-box models, which can help to interpret the relations between relevant factors.

As applications and application scenarios make their way into the research stream mostly via use cases and benchmark examples, the discussion in the group focused on how to obtain representative domain coverage via benchmarks and use cases. The lack of a reasonably broad and generally accepted set of such benchmarks and use cases was identified as an impediment both to rigorous and comparative assessment of the current state of research and to the identification and prioritization of open research problems. An informed, problem-driven advancement of the state of the art would thus require to find, collect, and publicize benchmarks that collectively obtain representative coverage of the relevant applications and application scenarios and that facilitate rigorous evaluation of and friendly competition between modelling approaches.

This quest in turn induces the need for an intelligible characterization of benchmarks that permits a mapping of individual benchmarks as well as benchmark sets concerning their particular contributions, thereby supporting demand-driven selection and adoption of existing benchmarks. For this mapping of the area, we came up with the following dimensions and explications of dimensions:

1. 1.

   The dimension of paradigm of use of the human models, distinguishing between human models as proxies of human behavior in model-based design and the embedding of human models into an operational system.

2. 2.

   Means of parametrisation and validation, namely observed in real-life / on real road / by physical test track driving or through simulator studies.

3. 3.

   Focus of the embedded human model, ranging over driver only (possibly including in-car passengers also), other traffic participants (in alter cars, in environment – especially VRUs), or both, including reflection of mutual reactive behavior.

4. 4.

   A broad set of quality criteria for the benchmark specification, including among others heterogeneity of driver behavior and coverage of driver types as well as societal groups (e.g., elderly), non-discrimination, explicit identification of observable/measurable correlates for latent or hidden phenomena of interest, strong correlation of benchmark-represented/benchmark-measurable features to real-world effects and effect strengths, accurate coverage of variability, not just of nominal behavior, as well as means for automated variant generation to avoid overfitting to particular benchmarks, test-retest reliability, inter-rater reliability, and repeatability and reproducibility.

5. 5.

   Aim of benchmark, covering a wide range of attributes from incentivizing research through friendly competition to facilitation of rigorous relative evaluation and ensuring coherency of methods, quality criteria, etc. between scientific communities. The last dimension identified was 6. technical prerequisites like executability on common hard- and software platforms, convenient packaging (e.g., via containers), and issues of open access and open science.

The discussion group holds the belief that such a classification would enable consolidated and collaborative research, stabilize and align the pertinent quality criteria across domain-relevant disciplines, and thereby advance scientific progress. Necessary first steps to render it reality would be to, first, generate a questionnaire permitting individuals and research groups to locate their respective benchmarks in the above multidimensional criteria set, second, to establish a platform for publication of benchmarks, including video tutorials and supportive contacts, and, third, to foster industrial contribution.

We discussed whether there can be a benchmark test, or evaluation method, to better compare different models. We were inspired by how competitions are done in the language community. In that community, there are multiple competitions and test frameworks. Not any of those is the “killer test”, but across tests one can see where a model or framework is stronger or relatively weaker. This moves the community forward.

In the field of computational modeling for human-automated vehicle interaction there are perhaps two challenges:

1. 1.

   What is a good model that can perform well on an evaluation test (or tests)?

2. 2.

   What is an appropriate (large enough, diverse) dataset to test these models on?

We foresee that this can be approached in three steps, that each require attention:

1. 1.

   Identify what the possible criteria are to evaluate a model on and define what scenarios are needed to test these.

2. 2.

   Collect data in an empirical scenario where the above features emerge

3. 3.

   Develop models that can then be tested on this dataset / scenario.

In later discussions we delved deeper into question 1. As a constraint we set that the model should exhibit human-like behavior. We looked at other papers that have defined model criteria, in particular papers by Anderson and Lebiere [1] and Taatgen and Anderson [2]. For these papers, we went through the criteria they defined and determined if and how they apply to the driving domain. We’ve identified that some modifications and specifications are needed. We also foresee that additional criteria – such as the ability to perform driving tasks and driving specific tests in a human way – need to be added. We plan to further refine these ideas in a follow-up project.

In the beginning of the discussion, it became quite obvious, that we can’t write down all important and relevant scenarios. We searched for a framework-like setup, where a scenario is composed of a multitude of independent factors. Some sort of building blocks to construct a specific scenario.

Several of these factors are influenced by the environment and the actual situation. They are relevant, but not directly linked to the automation. We identified

• $\mathrm{■}$

  Traffic environment. Freeway (number of lanes, exit section, tunnel, construction site) / Rural road (curvature) / Urban traffic (crossing, pedestrian, cyclist, parked vehicles)

• $\mathrm{■}$

  Criticality of situation (mostly described by dynamic behavior of others)

• $\mathrm{■}$

  Vehicle Type (truck, passenger car)

• $\mathrm{■}$

  Driver characteristics (stable factors as age / driving experience / knowledge about automation, dynamic factors as time pressure / fitness level)

• $\mathrm{■}$

  Boundary conditions (weather, daylight/night, road surface)

When it comes to human-AV interaction, the main ingredient of the scenario comes from the automation itself. There is the temporary aspect of the current phase of automation, which can pose specific questions: activating the automation, during automation, transfer-of-control. Within these phases, again we see multiple factors, and we partly need to separate carefully between the levels of automation, as some of the effects are relevant for a specific level only.

• $\mathrm{■}$

  Level 2 asks for continuous driver engagement and anticipatory behavior. The transfer of control is usually initiated by the driver.

• $\mathrm{■}$

  Level 3 needs driver availability during automation. Fast set-up of situational awareness after a take-over request. Assessment by take-over time & quality (including safeguard, e.g. mirror glances).

• $\mathrm{■}$

  Level 4 changes the time scale more to minutes than seconds. Some requirement of perceptibility. On system level, there may even be the need for a driver lock-out avoiding unintentional or undesired actions.

• $\mathrm{■}$

  For level 3 and 4 reaching the ODD limit or possible system failures becomes most relevant situation.

• $\mathrm{■}$

  Information design and communication on all levels (dependent of specific situation)

• $\mathrm{■}$

  Clear definition of visual behavior requirements (level 2), cognitive requirements (level 2 and level 3)

• $\mathrm{■}$

  Mode shift with transfer between levels (e.g., from level 3 to level 2) and corresponding challenges of mode awareness

• $\mathrm{■}$

  Aspects of trust towards the system.

The presentation of the group’s work in the plenum added the question, where models can be part of the system. This augments the perspective, that knowledge represented in driver models comes partly as a requirement from legal perspectives and regulations. Consequently, the constraints from a legal perspective have to be respected carefully.

In this focused session we discussed, which specific system design possibilities are offered within the different SAE levels of automation. We came out with a sketch of a framework to clearly describe the user tasks in the different levels. This description has the potential to serve as a backbone for the system design.

With respect to the levels, we concentrate on the levels 2, 3 and 4. By this, we are adapting somewhat the terminology used by the German Federal Highway Research Institute BASt of assisted (level 2 and below), automated (level 3) and autonomous (level 4 and above) driving, see https://www.bast.de/EN/Automotive_Engineering/Subjects/f4-user-communication.html. The time scale for the user’s tasks varies as well within these levels, from below seconds (level 2) via few seconds to minute (level 3) to minutes (level 4).

During the automation phase, the main task is shaped by monitoring. Monitoring means to monitor the environment, to monitor the system, and to monitor yourself. The monitoring serves the goal to decide whether I (as the driver) need to intervene, which would end the monitoring phase, and replace it by an action.

We systematically added the view for the phase transitioning from on to off. The possible interventions can be differentiated as an adjustment of the automation behavior, a system-initiated taking back of the control (as a result of a request-to-intervene), and a driver-initiated taking back of the control (might be necessary for safety reasons or just a current driver’s preference). As during the automation phase, depending on the level of automation, the concrete task description varies again. We are explicitly looking at the “what?”, “why?” and “when?” of the transition.

The two main pictures compiled by the working group are added to this report. The group had the clear impression to have a solid ground for an overall framework, but being clearly away from being complete. The focus was so far from the user’s perspective only, the system’s perspective needs to be added. The group plans to pick it up at a later stage, and to compile a full view out of it. Replacing detailed settings in scenarios by a comprehensive task-oriented view. To be continued.

In this focused session, we discussed a number of different scenarios involving multiple human road users that could benefit from the application of human behaviour models. Our discussion converged towards a scenario, centered on platooning – wherein a number of vehicles (cars and/or trucks) follow each other on a highway, at close distance in automated mode, typically with the expectation that drivers are available to resume vehicular control upon demand. This rich setting gave rise to a number of different control transition scenarios. In particular, we discussed the requirements for enabling the platoon to temporarily break apart to change lanes – e.g., due to an upcoming lane closure – and reassemble later. After discussing the potential contributions of different types of human behaviour models, we agreed on three main types of uses: First, models of the drivers in the lead and following vehicles can inform online algorithms in the platooning vehicles. This would guide real-time decisions how to break up the platoon, by predicting likely human responses to different alternative vehicle actions, and by choosing the control approach that is predicted to give the best outcome in terms of safety, efficiency, and comfort. Second, similar (but possibly not identical, since they would serve slightly different purposes) driver behaviour models could be useful in offline simulations. These would assist an engineer in designing systems by providing possible platoon breakup scenarios for automation implementations and testing – for example, to test lead time requirements for breaking up the platoon. Such tests would optimize safety and comfort, as a function of platoon speed, traffic density, etc. Third, we discussed that models could be useful on a conceptual level. Such models would help system designers and decision-makers better understand the behaviour and limitations of human drivers across different situations.

In this focused discussion we attempted to provide as broad as possible an overview of currently existing computational models which have been applied to the study of human interaction with automated vehicles, or could be applied in such a context. We listed a large number of references (which will be included in a separate document by the seminar organisers), and as we did so we structured the existing work in a rough taxonomy including the following dimensions: (1) Type of traffic scenario addressed, e.g., automation control transitions, shared control, multi-agent interaction, teleoperation, or driver support systems. (2) Type of (intended) use of the model, e.g., use of models to conceptually guide automation design, use in online algorithms for prediction or decision-making as part of the automation itself, use in offline simulated testing of automation, or as human safety benchmarks for automation. (3) Aspect(s) of human cognition/behaviour modelled, e.g., vehicle control, visual behaviour, individual characteristics, or unobservable mental/cognitive states. (4) Type of model, e.g., machine-learned, cognitive architecture, control-theoretic, and models focused on decision-making. These dimensions are to a large extent independent, but we also discussed that especially different model types, not least distinguishing between machine-learned, black-box models and mechanistic, white box models, might map more or less naturally to different parts of the other dimensions in this taxonomy. We discussed the strengths of these different types of models with respect to time critical decisions, understanding, drivers mental states or cognitive states, subjective and objective complexity, flexibility, driver prototypes and traits. For example, if conceptual understanding is important for the application at hand, white box models have a clear advantage, whereas for example when the complexity of the scenarios that need to be addressed grows, black box models, which can be learned from large naturalistic datasets, may be more appropriate.

In this focused session, we discussed the need to develop modeling workflows specific for this domain. In contrast to empirical sciences (e.g., [1]), models in human-AV interaction need to be, eventually, deployed in interactive systems or as decision-support systems in design. Applicability gains emphasis. However, despite the practical aims of modeling efforts, it is important to retain a level of theoretical and biological plausibility. It was also observed, that because of the safety-criticality of this domain, simulation studies are preferred prior to empirical studies in the wild.

Based on these and other observations, four design principles for a revised modeling workflow were set out: 1) put iteration to the heart of model development; 2) engage in both simulated and real-world studies, 3) aim for robust, verified modeling outputs, and 4) constant contact with theory (in cognitive sciences, biology, neurosciences).

The attached figure shows the revised modeling workflow proposed by the team.

In this focused session, the topic of how to prioritize possible types of predictions was discussed. Prediction is the essential feature of computational models, and it is essential to move beyond merely isolated tests. (Think: Newell’s 1973 paper “You can’t play 20 questions with nature and win.”).

The group agreed that prediction should not be confused with other uses of models, such as description, explanation, and counterfactual reasoning. These are linked to each other but different. The essence of prediction is that it involves some verifiable statement about a future state or unseen situation, given some observational data. What distinguishes driving as a domain is the need to predict what happens in previously unseen future situations, for example when doing verification of an AI’s behavior across different situations, some of which can be very rare.

It was also noted that predictions of driver behavior are needed in different ways in different processes, such as user-centered design, safety engineering, and for AI development. Also, different SAE Levels pose different requirements for prediction; like Level 3 poses a requirement for predicting driver’s availability. It was noted that an exercise should be carried out to assess these needs from a cognitive modeling perspective.

It was agreed that there is a multiplicity of predictions needed in this domain. This reflects the fact that computational models are used in widely different driving contexts, with different inputs, purposes, and time-ranges. For example, predicting whether a pedestrian is likely to step on a crosswalk occurs in the timespan of a few seconds, while route prediction occurs in the timespan of minutes to hours. There are micro-level and macro-level predictions that are needed, for example arousal states and states related to social interaction with other traffic users. And these interact. There are predictions of latent states that are not directly measurable but measurable via indirect physiological reaction (skin conductance for example). In real applications, we sometimes need to predict behavioral outcomes in driving, but sometimes we need to estimate a latent factor like the level of workload or stress. The group discussed what would be the most valuable thing to predict. There was some consensus that it should be the future state of the driver+vehicle+scene system within some time horizon. This system includes the driver, the vehicle, and the driving environment including the pedestrians.

The need for academic reseach to consider cross-validation practices was discussed. It was perceived as a harder measure of a model’s predictive capability. In statistics-based models, parameters are often fit to the same dataset on which fitness metrics are computed. However, the cross-validation practices that are followed in machine learning may not be directly useful for this domain. For example, we sometimes need to predict what happens with a new driver that has not been included in the training dataset (leave-user-out, or LOU, cross-validation). In many machine learning pipelines, the training dataset might contain samples from each user, thus ’leaking’ information in a way that is not plausible from a deployment point-of-view in this domain. It was further remarked that safety-related incidents are rare, which complicates prediction, the acquisition of training/reference data for computational models, as well as validation.

Prediction is almost always done under uncertainty, and uncertainty should be factored in the system’s action. Therefore a prediction might be more valuable if it also contains a measure of confidence intervals. The need for a probabilistic representation of outcomes is the greater the further in to the future the prediction tries to reach. However, when it comes to the idea of showing this to the driver, or an uncertainty visualization, the problem is non-trivial. Showing uncertainty to a driver can be confusing, as some earlier studies suggest. On this note, “a prediction is an intervention”: showing a prediction to a driver affects the driver’s future behavior. In multi-agent systems research, there are some formalisms to describe such interactions, but they are rarely done with computational models of humans.

We also discussed the need for updating the way we do models, i.e .the modeling workflow, with methods like sensitivity analysis and what is called parameter recovery.

In this focused group, we discussed different types of models and their content. Our understanding of a computational model is an executable, precise model that is defined through an unambiguous formulation (mathematical, logical, code, etc.) and is replicable. Computational models can cover different types of models such as cognitive, psychological, social behaviour. To understand what model to use, we first need to define the purpose of the model, e.g., predictive or descriptive. In modelling interaction between humans and automated vehicles, black-box models seem promising. However, the drawback is the lack of knowledge they provide about causal relationships. Therefore, we require to integrate machine learning approaches with causal models, predict the uncertainty of models, and test them in different contexts to understand their validity. This helps us move from black-box data-based models of behaviour towards causal first-principles models, which can generalise better, due to capturing key aspects of the system.

Another topic discussed in this focus group was predicting human behaviour while they adapt to changes in the design of the system. Predicting these emerging behaviours without currently existing data seems challenging. We proposed applying closed-loop models that allow the integration of the adaptive behaviour of humans. Moreover, descriptive models may help us develop causal models that allow predicting emergent behaviour or future behaviour that we currently do not have a lot of data about.

We have started by thinking possible scenarios for each level of automation, and soon we realized that we could not possibly list all the scenarios. To have a better structure of the discussion we have focused on categorizing these scenarios from different angles. This brought up whether a taxonomy regarding possible scenarios existed in research. Eventually, we have categorized factors originated by the user and the external world. This way, we presented 2 x 2 table of factors regarding the user and external world in SAE levels 1-4 and in level 5. On SAE levels 1-4 User factors included purpose of commute, driving experience, fatigue and engagement. Trust has been found related to all levels. Level 5 included personal preferences such as sustainability, costs and favorable routes. External or contextual factors were mentioned in all levels. These included social norms, environmental factors such as light and weather conditions and cultural or regional norms and formal rules.

We discussed potential approaches to model the concept of “trust in automation” in the driving domain. In particular, we discussed an experimental design that could allow us to model trust as a function of user interventions and monitoring (i.e., “occlusion” of the driving environment) behavior when a driver cooperates with a lane-keeping system (i.e., an SAE level 2 driving automation system).

Assuming that both monitoring and intervention behavior are related to a users’ subjective trust, the experiment could work the following way:

• $\mathrm{■}$

  A participant experiences an imperfect lane keeping system, i.e., the automation keeps the vehicle in the lane center but frequently starts to deviate from the ideal trajectory, slowly towards the lane boundaries.

• $\mathrm{■}$

  We assume that a driver observing this process would at some point start to correct the maneuvering of the vehicle to maintain a trajectory in the lane center again.

• $\mathrm{■}$

  While observing the situation and frequently correcting the steering, the participant needs to complete a (comparably short) trust questionnaire multiple times during the experiment.

• $\mathrm{■}$

  The collected data will then be analyzed to determine if the deviation that a driver allows the vehicle to leave the ideal trajectory is linked to the subjective trust ratings. We hypothesize, that a driver with low trust would correct the vehicles’ maneuvers earlier and more frequently, than a driver who trusts the automation more.

• $\mathrm{■}$

  Additionally, we could include a secondary task to extend the principle to monitoring behavior. In other words, a high-trusting driver may monitor the driving automation system less often than a low-trusting driver.

• $\mathrm{■}$

  The data observed in both conditions (i.e., intervening/monitoring) could then be used to model the subjective trust of a driver. Given that the assumptions described above are valid, a drivers’ monitoring and intervention behavior can then be used to derive their trust levels.

The panel discussed technical capabilities required of computational models in this domain. Although the nature of different modeling frameworks and different studies might differ, what do we consider the core functionality? For example, there are capabilities related to:

(1)

Compatibility: To what degree do models need to be compatible with simulator software (e.g., to test a “virtual participant”), hardware (e.g., be able to drive a car on a test track), and other models of human thinking?

(2)

Adaptive nature: Computational models aim to strike a balance between precise predictions for more static environments and being able to handle open-ended dynamic environments (like everyday traffic). How can precision be guaranteed in static and dynamic environments? How can models adapt to changing circumstances?

(3)

Speed of development and broader adoption: The development of computational models requires expertise and time. How can development speed be improved? How can communities benefit from each other’s expertise?

The panel consisted of the following speakers: Gustav Markkula, Martin Fraenzle, Myounghoon Jeon (Philart), Jelmer Borst, Otto Lappi, and Nele Russwinkel.

Gustav Markkula discussed the compatibility of software and hardware, which is critical at the very end of an applied project, but much less considered earlier in the model developemnt stage. It is therefore critical to study the ultimate application context at least roughly. Similarly from an applied perspective, the speed of development and the flexibility of modelling is important. Models should be extensible with new data, new features, new capabilities etcetera.

Martin Fraenzle followed up and discussed how the idea of running models ”in situ” might be out of scope at the moment.

Myounghoon Jeon brought up Distract-R as a great example of modeling, which allows different parameters to be simulated and rapid prototyping while being hardware and software agnostic and yet have some ecological validity. He echoed the need for adaptability and extensibiliy. He further brought up the topic of motions, which is presently missing from cognitive models although clearly important.

Jelmer Borst discussed the desiredata of modeling, one of which is that they should be able to drive the car. However, ACT-R is far from that goal according to the panelist. It is too brittle. There is a need for hybrid models that combine cognitive architectures and machine learning to overcome this issue. This may allow a cognitive model to operate in the background while the user is driving and warn users if something out of line is taking place, such as overloading.

Otto Lappi proposed a more philosophical perspective. He claimed that the first question to ask is who you are modeling for: the designer, the engineer, the scientist, or someone else? This has significant implications to the capabilities we want from models, and also our validation criteria.

Nele Russwinkel followed up on the topic of what is required of a model during driving, pointing out that ACT-R may be useful if it can anticipate the driver, even if now able to drive the car. She pointed out that latent variables, such as those related to situation awareness, are important for a car to understand. However, presentday models mostly do not afford this in a real-time system, due to computational intensity. She pointed out that a lot of modeling is still missing when it comes to the different levels of automation in vehicles. Moreover, some of the events we are interested in are rare and therefore inherently difficult to predict.

Discussion with the audience followed up on the need for better simulation environments and hybrid models that use ML. There was disagreement on whether simulation environments are too simplistic and can or cannot contain the richness of perceptual and other cues that drivers exploit in driving. The audience gravitated toward the need for developing a roadmap for cognitive models. Participants agreed that one of the outstanding goals is to understand the different SAE levels better, especially from the perspective of their psychological consequences and, therefore, their consequences on cognitive models.

Future topics that were raised included:

1)

How to allow drivers update models about themselves?

2)

How to trade off predictive accuracy and computational costs in such a way that we can allow cognitive models to run in real-time systems?

3)

How to go from mental models research to scene understanding, which is critical in driving.

The fourth panel discussed how computational models that are developed for Human-Automated Vehicle Interaction can benefit from advances in AI while also avoiding some of its pitfalls. The summary below is made by Chris Janssen, building on crowd-sourced community notes of the sessions and snippets of panel members’ presentations or text where possible.

Before the meeting, the organizers had identified that there are many developments in AI that computational models can benefit from. Three examples are advances in (1) simulator-based inference (e.g., [1]) to reason about possible future worlds (e.g., varieties of traffic environments), (2) reinforcement learning [2] and its application to robotics [3] and human driving [4], and (3) deep learning [5] and its potential to predict driver state or behavior from sensor data. At the same time, incorporation of AI techniques also comes with challenges that need to be addressed. Three potential challenges are for example:

1. 1.

   Explainability: Machine learning techniques are good at classifying data, but do not always provide insight into why classifications are made. This limits their explainability and is at odds with the objective of computational models to gain insight into human behavior. How can algorithms’ explainability be improved?

2. 2.

   Scalability and generalization: How can models be made that are scalable to other domains and that are not overtrained on specific instances? How can they account for future scenarios where human behavior might be hard to predict [6]?

3. 3.

   System training and corrective feedback: if models are trained on a dataset, what is the right level of feedback to correct an incorrect action to the model? How can important new instances and examples be given more weight to update the model’s understanding without biasing the impact?

During the panel, these and other themes were discussed. We started with short pitches by the panelists, which are summarized first.

Roderick Murray-Smith discussed among others the following points.

(1)

A priori insight can be placed into machine learning models in multiple ways. Forward and inverse modelling components were discussed as being particularly relevant.

(2)

The advent of deep learning models has helped to place more complex models of perception into control models.

(3)

This has knock-on effects of explainability: is trying to explain a complex policy model a good idea? Or is it better to ensure that the model learns a value function that is relevant to the human and then optimized?

(4)

Engineering approaches often follow a modular approach, where multiple components are solved from first-principles, before being placed in bigger systems. The question is whether such an approach is scalable to a more complex context such as driving, where a closed -loop simulation might not be feasible.

Mark Eilers discussed that the best models for driving an automated vehicle do not necessarily need to be models that drive in a human way, or based on a human-like model. Mark saw potentials and possibilities in the following fields and areas.

(1)

robotics, for example imitation learning,

(2)

reinforcement learning and optimal control,

(3)

deep learning, for example the use of new unsupervised learning techniques.

All these techniques can be adapted to fit existing models and contexts. Mark identified as pitfalls that there might be an identifiability problem: data alone can’t always tell the researcher which of the many different approaches or models is the correct one. Theory and experiments together must inform the model and the modeling community.

Duncan Brumby reflected on his own past work in preparation for the meeting. When he modeled driving 15 years ago, he used a relatively simple model, but it was useful and valuable (e.g., [7]). At the time, there was talk of how reinforcement learning models can one day give even more insight, and it is good to see that that potential has come to fruition (e.g., [4]). Duncan now sees three areas where science of AI has lead to great developmens and still has potential to grow further.

(1)

Models of the driver: science can help to create better models

(2)

Models for the car itself: current semi-automated vehicles have lots of technology and models in it, for example to go around a corner and break when a car in front of it stops. However, the models for these tasks are developed by companies, and the question is how it will reach science.

(3)

The technology with which people interact in the car (e.g., secondary devices) have evolved. The tools that people use are different.

Overall, due to these changes, some of the parameters that underly older models might need updating. For example, when do we call something distracting or not differs between settings.

Birsen Donmez looked at the topic of this panel from a statistics and experimentation perspective. Some machine learning approaches such as Deep Learning are very powerful, as they enable to model complex non-linear systems. Capturing multiple predictors and their interactions go beyond the model typical traditional statistical or cognitive models. However, they do require big data sets or samples. More traditional statistical tools do not always require such larger sets, but can also work on smaller samples. There are challenges for both the big data set (and machine learning) and small data set (and statistics) approaches. For the bigger data sets the general challenge is that these are often collected by industry and not available for researchers. How can this be changed? When we collect our own datasets they might also not always be big enough, and when using data sets of others, that might also not always be trivial. For smaller data sets there is a limited sample problem, when applying machine learning techniques. It requires insight to for example split the dataset appropriately into training set and test set. Birsen sees potential for situations where researchers do get hold of bigger data sets. They have the right background to be critical about the data. For example, to ask questions such as: is the data representative? How was data collected? Do correlations exist?

Jussi Jokinen discussed a model of how humans can interact with an automated vehicle. A cooperative AI can for example observe human user behavior, and then try to identify posterior probabilities of user states (in a Bayesian fashion). There are however potential pitfalls in that such a Bayesian inference process can be slow and the results of a modeling framework are sometimes surprising and don’t seem plausible. Given that the model is not always correct, it is useful to explicitly consider (un-)certainty. In Jussi’s own work (e.g., [4]), he benefits from applying the computational rationality framework (e.g., [8, 9]). Such a model expresses how latent variables impact behavior. A reinforcement learning agent is developed that works within constraints of a cognitive architecture and task environment to come up with the optimal policy that if architecture and task are correctly defined should be similar to user’s behavior. It is cool when it works, but there are lots of moving parts. Jussi sees opportunities (and challenges) in refinement of the ability to use inference techniques in these models: how can one be confident that the learned model is indeed the correct model?

Justin Edwards drew parallels with the history of language models and his experience in the Conversational User Interfaces (CUI) community. Coming up with language generation and processing models long proved a challenge to the field. Initially there were many rule-based models. The last decade has seen a wider range of models that use large datasets and are data-driven. Data-driven models have surpassed the more theory driven models in quite a lot of tasks. However, they also are occasionally deeply flawed. The samples on which training takes place are not always general, but are used for generalization purposes. This showed for example in the Delphi system, which was meant to be capable of moral judgements, but used data from reddit chatboards for training.So, the big lesson learned for the automotive community is: as new technology becomes available, be aware of where the data comes from. What are the biases? Those might carry through in surprising ways.

Moritz Held focused mostly on two themes. First: explainability. Hybrid models that combine machine learning with theory-driven models might be fruitful in this regard. For example, combining ACT-R models with techniques from Bayesian spatial networks. The second topic was scalability of models. Moritz was surprised by some discussions in previous days regarding scalability. He would expect that there is always some robustness in a model. There is no need for (for example) and ACT-R supermodel to handle every driving situation. But, he would expect it to at least handle different driving scenarios.

Chris Janssen talked about four points. First, it is important to think about function allocation: how are tasks divided between humans and the automated vehicles. Models can play a role in this regard (see e.g. [10]). Second, in such discussions, it is tempting to think of simple heuristics such as “men are better at [some tasks] and machines are better at [other tasks]” [11]. However, humans and machines are dynamic (they can both learn and adapt), and they operate in varying contexts. Therefore, static heuristics might be useful at first, but might not always be appropriate for these dynamic contexts. Cognitive models should take such dynamics into account. Third, explainability is becoming increasingly important inside and outside of science. Computational models can tie in with this by combining theory-driven and data-driven models. In other words: it is important that models are connect to theory (not just data), but also that theory is connected to practice.

During the discussion that followed, we talked among others about the following aspects:

1. 1.

   What creates distributional shifts in models? How can models be made more robust against this?

2. 2.

   How can the designers and researchers of modelers better communicate what the goals and limitations of models are? For example, for which level of automation they are designed.

3. 3.

   There seems to be a balance between generalization of a model versus having a more structured / fixed model. Models differ in which aspects they keep fixed (as assumptions, framework, or architecture) and which aspects they train or learn (for example, based on data).

4. 4.

   There was also a discussion about how computational models can be designed such that designers and developers (of cars) can better benefit from them. There is a potential discrepancy here in that designers want to test multiple, flexible designs whereas models are often trained on or designed for more fixed tasks.

5. 5.

   There is a need to be more systematic about the model selection process.

6. 6.

   There is an open question about for what types of research questions small or large datasets are needed.

Computation models are only as good to the degree that they can describe and predict phenomena in the real world. In particular, many current computational models capture the results of a single experiment based on the current context of what is being modeled. However, behavior might change with more exposure to and experience with automated technology. To make models most useful over time, empirical testing is of paramount importance, especially in order to evaluate models that look at current behavior, as well as models that are based on projections of future behavior. The panel on empirical research insights focused on the question above, as well as related questions such as how to study phenomena where computational models are not commercially available, and how to map findings from simulator tests to real world scenarios.

Patrick Ebel started the conversation with a focus on mapping real world data to input for computational models. Clearly there is a need for models to be used in industry and leveraging existing models can save time and money but there seems to be a barrier in adaptation. A couple of possible causes could be accuracy of models, especially since real world application needs to consider many confounding factors which cannot be typically captured in experimental models. Patrick also raised the question of the comparability of data that come from different simulators and how to reconcile for those differences.

Wendy Ju discussed the generalizability of empirical studies that are done in controlled settings in the lab – where the studies themselves are challenging to run and have additional limitations (e.g. cultural, temporal variation). She proposed the alternative of gleaning results from understanding naturalistic data from real-world driving. However, this will require instrumentation of vehicles where there needs to be a common agreement of what data is collected, when and how. The data itself can be a contribution to the field, and once we have these datasets, ML can be used to scale up and detect patterns in the diverse data sets that are collected.

Fei Yan talked about how to extract insights from empirical research and black box models and posed the core question of determining when a model is really needed to solve a research question. Empirical research can help understand causal relationships, and black box models can look at theory (e.g. social psychology on human-human interaction) which can be then used to form hypotheses that can be tested in empirical research. Fei proposed using empirical research to understand relevant factors before making the black box model. An example scenario could be the use of a predictive model in a lane change support system that can adapt to a driver’s states of uncertainty. But it is difficult to know what models are most applicable in these scenarios. For some domains, ML or Bayesian models might be enough, but depending on the level of accuracy needed, those might not be most appropriate for other domains. Fei also talked about the small sample size in empirical research which helps with deeper dives into relationships between factors.

Tuomo Kujala shifted gears towards modelling driver attention. Driver attention monitoring is required for cars in level 3 or even up to level 4 in order for a car to respond safely and effectively to takeover requests. The challenge is that most attention systems monitor the inside of the car and how the driver’s attention is impacted by the internal environment that can be detected. However, a driver’s attention can be impacted by factors that are not detectable directly – for example, how much ‘free attention’ or cognitive capacity is available for the driver and whether it is enough to handle an upcoming scenario in driving. Appropriate attention modelling can decrease the uncertainty. Tuomo proposed that we need prescriptive or normative computational models, not only descriptive models.

Phillipp Wintersberger focused on the specific scenario of what is needed to make Level 3 automated driving possible and presented the argument that simulator results may not be as limited as we assume them to be. The trade off for a data-driven model is that a huge amount of naturalistic data is needed, connecting back to Wendy’s point – we as a field do not have established standards on how to get this data. For example, doing a study in a test track may not yield useful data if the test track is not similar to the real world, but similar studies can be designed to be more similar to real world scenarios. Phillipp also talked about how we can get massive data to investigate development of trust in such systems. Is gamification a possible path? He also questioned the need for really big data sets and whether smaller datasets can be combined to get bigger sets.

Shamsi Iqbal talked about the target user – who we are designing for, the current user or the future user who will have different technological capabilities, exposure and experience. Can we effectively extrapolate from the current user to project what the future user will look like? Empirical research typically falls into three buckets – lab experiments, model building approach and naturalistic driving. While naturalistic driving gives the opportunity to learn more about what challenges people face in real world scenarios, there are also privacy and security concerns around collecting massive amounts of data from people. For lab experiments in the other hand, we have the opportunity to test innovative ideas that push the boundary of what might be possible. The challenge is then to determine how to generalize the findings to real world scenarios, where we may not even know what the future scenarios might be.

Linda Boyle brought up the counter point that model-based and empirical research are not mutually exclusive and naturalistic and simulated data should not be mutually exclusive either. Research needs data from separate dimensions, not just comparative studies. Even in a single lab – different types of data can be collected. The goal should not be just to create a data repository, but also to use results for scenarios such as Bayesian inferences/priors for future models and experiments. Linda emphasized that while field studies are great there is a lot of variation. Lab studies are a must – especially to test rate situations and edge cases that we may rarely or never see in real world data, but still need to be accounted for in our models.

Ben Cowan emphasized that the key to empirical research is observation of phenomena. In his view, the role of empirical research in automated vehicles are: a) to assess how concepts we deem important to automated driving (e.g. situational awareness, attention) manifest in and are impacted within the experience of automated driving. We can learn a lot from conducting qualitative studies on automated driver experience (e.g. observation studies, interviews) to assess the importance of concepts like trust, potential challenges and barriers to adoption etc. Quantitative empirical work, particular lab studies can also be used to test how specific concepts are causally influenced by design or events during the drive to get a clearer picture of what are the key areas for us to focus on (e.g. the design of communication strategies for pedestrians- Lanzer et al, AutoUI 2020 on politeness strategies positive effect on trust and acceptance). They also allow us to experiment with identifying cognitive concepts of importance.

To support the notion of what to model and to give our initial models a test run: Models should be related to the real world. But we need to perhaps incubate these first. Empirical lab based work is important to determine what is and is not important in that real world for the concepts we deem important to research. That is not to say that we cannot and should not do lab-based work, devise, test and assess models for concepts on lab based data to determine how the cognitive concepts may behave within more real world driving contexts.

A key consideration is also in the measurement quality for measuring the concepts we wish to measure. In particular when using questionnaires to assess concepts like trust, it is important that we stay true to any base questionnaires used or conduct the design of our own psychometrics to ensure what we are using are valid, reliable and sensitive to the concepts we measure. On a wider point as a community of researchers we must also ensure that the empirical work we are doing is replicable and open. We need to create an infrastructure and reward system that allows us to embed replication in our empirical research activities (either through replication when publishing the initial study, or in specific tracks at our key conferences and journal publication venues).

In the discussion, Patrick talked about triangulating empirical research and model based research using the following approach: 1) Use model based systems that are built upon naturalistic data. Can help to see patterns in the data and suggest empirical tests, and 2) Use models as a sandbox and test bed approach. How good does a model need to be? If model already shows a study is not worth the time, might be useful to test. Another important topic that emerged from the panel discussion is whether a ‘replicate’ track should be added to the Auto-UI conference. Key questions that arose included what is the definition of replication, what is the expected rigor, how to address the differences in the experimental setup that might impact the replication, how to share data, how to present the results at the conference and what kind of benchmarks should be setup.

As possible future topics, the following points were extracted from the panel discussion:

• $\mathrm{■}$

  How to connect models even better to needs of industry? They want them, but don’t always use them. Why? (e.g., Patrick)

• $\mathrm{■}$

  How to move towards “data is a contribution” and replication (Wendy)

• $\mathrm{■}$

  Do we need models even? (e.g., Fei)

• $\mathrm{■}$

  Prescriptive models are needed; not just descriptive (Tuomo)

• $\mathrm{■}$

  To understand attention, more is needed than just in-car glances (Tuomo)

• $\mathrm{■}$

  Do we need realistic studies? Or is simulator sometimes better (Philipp)

• $\mathrm{■}$

  Opportunity of gamification (Philipp)

• $\mathrm{■}$

  Need for a place to store online data (Philipp)

• $\mathrm{■}$

  Who are we designing tech / experiment / model for? Today’s user or tomorrow’s user? (Shamsi)

• $\mathrm{■}$

  How to make tools accessible to wider community? For models and tools and experiments. (Linda)

• $\mathrm{■}$

  How to motivate and reward replication systems? (Ben)

During the conference all the attendees identified various areas that can be part of a longer-term research agenda to move the field forward. Below, we list some of these research agenda items very briefly. They are clustered within each of the broader challenges that were a core panel discussion. Of course, these challenges are only an incomplete subset of the many research questions that are out there.