Human in the Loop Learning through Grounded Interaction in Games

With the increasing deployment of language technologies to users, the need for a deeper understanding of the impact of natural language processing models on our society and user behaviors has grown. Designing culturally responsible, equitable, and inclusive language technologies that can benefit a diverse population is ever more important. Toward this goal, this brief presentation highlights efforts to address linguistic injustice and promote fairness, with a special focus on sign language technologies. By integrating insights from cognitive science, social sciences, and machine learning, and engaging communities through co-design, we explore pathways for advancing inclusive and innovative language technologies that support diverse communicative needs.

The literature is flourishing with surveys on this topic taking Large Language Models (LLMs) as the main viewpoint. We summarized the classification reported in these surveys and the future directions they highlight. It becomes clear that Interactive Games, within 3D environments, and multi-agent situations will play a crucial role in the near future. Such games are a good tool to evaluate LLMs and to advance from models to agents, and from virtual environments to real-world situations. From such an overview, it also emerged the importance of paying attention to evaluation methods. In particular, the next challenges are in evaluating reasoning and planning capabilities, as well as the social intelligence of AI systems. After this overview, we focused on reporting the roles LLMs have currently in games, the limitations they are facing, and the ethical issues of using LLMs in games. Furthermore, we reviewed the existing virtual and 3D environments. Finally, we identified open questions focusing on the evaluation methods for aspects of multimodal interaction currently not evaluated, for testing what would be gained by moving to multimodal interactive games and what are the limitations of current games and platforms.

The talk motivated the use of a modular approach for specializing language models (LMs) more efficiently. For instance, Low-Rank Adapters (LoRAs) can be learned independently on diverse set of tasks and later be combined during inference via model weights merging. The second part of the talk motivated the need for more interactivity during LMs training since this how LLMs are being deployed nowadays. Also, a brief overview of TW-Bench was given, showcasing the current SoTA performance of LLMs at playing text-based games.

Current conversational AI agents have remarkable ability to understand written language and static images. In this first State-of-the-Art presentation, we introduced a distinction between language-only tasks, tasks requiring multi-turn inteeractions (as in chatbots), vision-language tasks (such as captioning), and embodied tasks (real or simulated). We then surveyed the extensive work devoted to testing models in these different tasks – e.g., for referring expression interpretation and intention recognition (theory of mind). Aspects of grounded interaction covered in some detail included grounding (the distinction between “perceptual grounding” and “meaning grounding” was made) and the extent to which agents ar aware of the ambiguity of NL expressions in visual tasks.

In the traditional textual coreference resolution task, a model identifies whether two or more linguistic expressions in a text refer to the same entity, object, or event. In a multimodal setting, these “referring expressions” are linked, or grounded, to their corresponding visual counterparts. Our work focuses on the interplay between visual information and coreference, investigating how visual cues can assist models in resolving complex coreferential usages.

Today, we present results from a pilot study evaluating textual coreference tools on multimodal data. We also outline our roadmap for developing a model that integrates visual and textual features to enhance coherence. Our approach addresses traditionally challenging cases, such as plural pronouns, and involves annotating additional data to capture a broader range of phenomena. We aim to generate text using coreference chains tailored to specific target populations and aimed at mitigating hallucination. Finally, we aim to work with other genres, including dialog.

The talk introduces a new type of dialogue game, in which two players collaborate to jointly solve an NP-hard optimization problem (e.g. two-player Traveling Salesman). These games bring together two lines of my recent work: (a) dialogue games that are designed to elicit balanced role-taking, as in Jeknic et al. 2024, and (b) the use of dressed-up NP-hard problems as test suites for the reasoning capabilities of LLMs. Preliminary results indicate that current LLMs can play some optimization games successfully, especially if the graphs are small, but lose track of the dialogue state after a few turns and are unable to solve the problems.

In recent years we have witnessed an explosion of research in three converging directions. The first is a decisive move in Artificial Intelligence (AI) towards human-centered intelligence and AI models that can learn through interaction and are able to act as intelligent assistants. The second is equally substantial process in conversational agents and dialogue systems on grounded interaction – developing models aware of the situation in which they operate (a physical world in the case of robots, a virtual world in the case of artificial agents) and able to, e.g., understand / produce references to this situation. The third is the appearance of virtual world games in which conversational agents impersonating characters can learn to perform tasks, or improve their communicative ability, by interacting with human players in platforms such as Minecraft. The objective of Dagstuhl Perspectives Worskhop 24492 was to review progress in these directions, and map the most promising future research directions. In this introductory presentation, I summarized motivations and objectives of the workshop, and presented the preliminary schedule.

In this talk, I presented an overview of a planned community challenge that is aimed as fostering research on “learning in interaction”. I started by presenting a motivation for why one might want to look into learning in interaction (as opposed to learning from observation): Ecological plausibility, strong learning signal. I then talked about some challenges in realising this idea, stemming from the sample inefficiency of our current learning mechanisms, which make it difficult to set up “human-in-the-loop” learning. The challenge starts from the observation that the situation has changed somewhat: We now have available to good (albeit not perfect) “teacher simulators” that do not tire: Large Language Models. The idea of the challenge is that so-called “frontier models” (the current largest and most competent ones) can be used to guide the learning process of smaller models, leading to a relative improvement that is larger than that achievable by other learning methods on the same amount of data. The challenge will build on the clembench project, to provide participants with an environment in which they can bring their learner models in interaction with stronger teacher models, in a range of conversational games.

The most recent breakthroughs in large language models have been possible thanks to the development of Transformer-based models that can be trained at scale using self-supervised learning objectives, which are dependent upon access to internet-scale datasets. Current models learn from these datasets assuming that they are “observers” – lacking any form of interaction. In this working group, we discussed the importance of interaction with the world and with other agents as an important means for learning competent artificial agents that can solve a variety of real-world tasks that involve coordination and planning of actions. We first provided a concrete definition for “interaction” and what can be learned through interaction. We presented a list of prior works for learning in interaction ranging from the pre-LLM era to cutting-edge research. Interestingly, most of them leverage language in one form or another to provide/guide the learning process. As a result of our investigation, encompassing recent literature on reinforcement learning, we found that current learning approaches are predominantly focused on single-turn interaction and that very few deal with the intricacies of real-world interactions required to complete complex real-world tasks. This includes an inability to quantify uncertainty in beliefs relating to the environment or interaction partners or to request clarifications. Some rely on human preference data collected off-policy – not through interaction with the learner –, which leads to models that struggle to reflect human preferences, particularly over many turns of interaction. In essence, we argue that future work should develop learning methods that are able to leverage language use in interaction to acquire more relevant signals for learning more robust agent policies.

Real human conversation is messy and flexible, and its dynamics are influenced by interaction design. In this talk, I describe a new corpus of goal-directed conversations between human players of the cooperative 3D puzzle game, Portal 2, and outline a number of complex phenomena that arise in real human conversations in the corpus.

This talk discussed reference games as a tool to study notions of context and context effects on language use in grounded interaction. As our starting point, we introduced classic reference games where the speaker’s task is to refer to a target object, in the context of a set of distractor objects. As an example, we showed a reference game from a basic color domain, where even recent multimodal LLMs do not resolve references to simple color patches correctly when the referring expressions are highly context-sensitive. We compared such synthetic toy settings to reference games where more realistic scenes, e.g. images, are available. These contexts are more akin to scenes that humans act in in their everyday lives. We argue that these scenes broaden the types of context and context effects that can be studied. Based on [1], we show that scene context can be relevant for recognizing target objects in referring expression models, but also show that current models still seem to lack a holistic scene and situation understanding. We discuss examples of ongoing data collections, where these ideas are extended further to test the effects of scene-level expectations and violations on reference and the effects of scene context on conceptualization.

The Complex Interaction working group identified complexities of language-based interactions that arise from interaction design, including incentive mechanisms and participant affordances (in action and communication). They enumerated a number of such complex phenomena, and for each, define the phenomenon, provide some background on historical approaches to addressing the phenomenon, the state of current models in handling the phenomenon, and any recent approaches addressing them. For each of these, they also described what one might learn in interaction when the phenomenon occurs. Finally, they provided recommendations on different philosophies in designing systems that are able to manage such complexities in interactions with humans.

Advances in language modeling have led to an explosion of interest in testing agents in games using game platforms and benchmarks for grounded interaction. Existing games and platforms are extremely diverse in their setting, goals, and complexities, and they are being developed in different subfields of AI, NLP, and computational linguistics. Furthermore, within these subfields, games are designed for different purposes and, to some extent, classified using different taxonomies.

The working group on Game Design for Grounded Interaction identified four key research questions arising from this explosion and diversity of games that we suggest should be explored in future research:

• $\mathrm{■}$

  Q1: How can games and game benchmarks be designed more systematically, such that they lead to a deeper understanding of games and the skills that games are testing? How do we generalize skills and agents’ abilities across games?

• $\mathrm{■}$

  Q2: How important it is to maximizing human engagement and fun in the design of these games?

• $\mathrm{■}$

  Q3: What role does the complexity of the game is? And how do we measure it?

• $\mathrm{■}$

  Q4: How do we evaluate agents within and across games? In particular, how do we evaluate whether the skills trained / tested with a game transfer to real world applications?

Regarding Q1, the WG identified as a promising research direction to develop inventories of skills or taxonomies that make it easier to compare games and agents’ skills across different games, and that allow to disentangle agents skills at an intermediate level.

There has been little investigation into how fun and engagement impact the effectiveness of game designs and presence of different linguistic phenomena in grounded interactions (Q2). While it is less clear whether fun and engagement are important for game design with agent-in-the-loop approach (e.g., agent-led tasks), these two aspects are very important for games that incorporate a human-in-the-loop approach. The WG recommended that developers of grounded interaction games focus more on evaluating the connection between the game engagement level and the intended game outcome. It is also important to involve professional game designers in the process of game world building and to continuously collect and integrate feedback from players.

Although many grounded interaction games / tasks in which conversational agents occur, or that are used to test / train them, are very simple in design and only involve simple forms of interaction others are very complex. An important cluster of questions that need researching is the extent to which the complexity of a grounded interaction game affects its usefulness for the purposes of testing and evaluation (Q3). Such questions include:

1. 1.

   How complex a game needs to be to be useful for testing / training?

2. 2.

   Is there a link between complexity and transferability?

The WG’s recommendations are that

• $\mathrm{■}$

  More research should be carried out questions 1 and 2.

• $\mathrm{■}$

  Developers should be more explicit / aware about the complexity in their game and how that may affect its usefulness.

Finally, regarding Q4 – evaluation often prioritizes performance of an agent in the game itself (intrinsic evaluation), while placing less emphasis on how in-game performance translates to downstream applications, other games, or real-world scenarios (extrinsic evaluation). Both intrinsic and extrinsic evaluations are complementary and equally important.

Evaluation, much like game design, benefits greatly from a multidisciplinary approach to ensure robustness and reliability. For instance, the outcomes of human evaluation studies can be influenced by factors such as the personality traits of the recruited participants. Incorporating expertise from fields like psychology during the design of evaluation methodologies can help mitigate potential biases or adverse effects arising from such factors. Similarly, expertise from HCI can play a key role in ensuring that the UX and UI elements needed for human evaluation are designed correctly.

Transferability is an important dimension of evaluation that warrants special emphasis. It examines the extent to which skills learned in one game can be effectively applied to another experimental setting, including both other games and real-world environments. This has implications for the development of generalizable, adaptable learning agents.