Mixed-Initiative Dynamic Autonomy Through Variable Levels of Immersion and Control (MIDA-VIC): A New Paradigm for Collaborative Robotic Teleoperation in Space Exploration

Given the complex and collaborative nature of robotic teleoperation in space exploration, research in human factors (HF), human-computer interaction (HCI), and human-robot interaction (HRI) has aimed to reduce the mental, physical, and temporal demands of teleoperation. For example, innovative designs of user interfaces, visualizations, interaction techniques, and control room layouts are intended to leave teams with more mental, physical, and temporal resources to develop a good collective situation awareness (SA).

SA is commonly defined after Mica R. Endsley as “the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” [8]. In other words, SA is a person’s real-time understanding of “what is going on” around them and what might happen next. Endsley’s model breaks SA into three levels that can be applied to our telerobotic scenario:

• $\mathrm{■}$

  Level 1: Perception – Being aware of the relevant cues in the environment. For a rover operator, this means perceiving key data from the robot and its surroundings: e.g. receiving camera images, seeing obstacle locations, reading the rover’s speed, battery level, or telemetry warnings. Without perceiving these status elements, an operator cannot move to higher awareness.

• $\mathrm{■}$

  Level 2: Comprehension – Understanding what those perceived elements mean in context. In teleoperation, the operator must interpret the sensor data correctly – for instance, recognizing that “the rover’s wheel slip is high”, which could indicate that the terrain is soft sand. Alternatively, a certain rock in the camera view might be a science target of interest. Comprehension implies that the operator synthesizes different data points into a holistic picture (“the rover is tilted on a slope and losing traction”). This level of awareness allows the human to assess the significance of the current situation relative to mission goals.

• $\mathrm{■}$

  Level 3: Projection – Anticipating the future status of elements in the environment. In the rover scenario, this is the ability to predict what will happen next: if the rover continues on its path, will it get stuck? If the battery is at 20% now, will it last through the next planned drive? Good SA means that the operator can project ahead a few minutes or hours based on the current trends. This foresight is critical for decision-making, such as planning maneuvers to avoid an obstacle or scheduling a return to base before sunset.

Endsley’s framework is highly relevant in this context because teleoperators operate in dynamic environments (another planet’s surface) where they must continuously perceive changes, understand their implications, and predict the outcomes to make correct decisions. However, achieving robust SA in a teleoperation setting is challenging. The operator is separated from the rover by distance and relies on information constrained by limited bandwidth, which can degrade awareness. Endsley and others point out several factors that can undermine situation awareness for remote operators: poor sensory information, communication delays, and interface complexity are all obstacles to building a clear picture [9].

Virtual Reality (VR) for robotic teleoperation in space has been proposed since the 1990s [5] and potentially improves SA by providing more sensory information and reducing interface complexity. However, isolating operators in VR from their real-world team members and command center can also undermine SA and efficient collaboration. In our own previous work on collaborative Transitional Interfaces [25] and using Augmented Reality (AR) for HRI [3], we have examined different strategies for immersive technologies that could support operators without such disadvantages.

For example, our research on Transitional Interfaces explores how 2D graphical user interfaces (GUIs) can be seamlessly integrated with AR and VR to support teams. We have examined new ways for human collaborators to use variable levels of immersion, i.e., seamlessly switching between (1) personal, immersive, and stereoscopic views with good depth perception in VR during spatial tasks, (2) less immersive, yet stereoscopic, 3D displays of spatial information in AR that do not isolate operators from their co-located human team members in the real world, or (3) “flat” 2D maps and charts on shared vertical or horizontal displays for efficient overview and group discussion. By this, we intend to better support teams during phases of tightly-coupled collaboration as well as more loosely-coupled parallel work and increase overall workspace awareness [25]. As we discuss in [24], technologies using variable levels of immersion could also help operators to better deal with communication latencies and asynchronous collaboration over long distances in space.

In HRI, we have explored approaches to increase SA by providing the operator with contextual visual information that augments the robot in such teleoperation scenarios. In particular, we were able to show how such visual augmentations can improve the user’s ability to perceive and comprehend the robot’s position with respect to its surroundings and potential target objects [3]. We also found a clear trade-off regarding visual complexity that needs to be taken into account when designing such augmentation [4]. Regarding the third level of SA – Projection, a literature review on understanding robot intent has shown that it is imminent to communicate not only motion but also current state or guide the user’s attention [21]. It also showed that the level of immersion can vary quite a bit between different communication strategies, ranging from direct LED feedback on-robot joints [26] to visual path predictions in AR/VR [27], which all come with certain advantages and drawbacks, highlighting the need for variable levels of immersion. Regarding the question of how to realize such variable levels of immersion, we developed the AdaptiX framework, which provides a technical architecture that allows HRI to be designed along the full Mixed-Reality Continuum [20].

Good situation awareness, including projection and anticipation of future states of the environment as well as the individual actors, is vital for the teleoperation of robots due to the roundtrip communication delays of radio signals. These communication latencies are typically a few hundred milliseconds in low Earth orbit (LEO), but can reach between 5 to 14 seconds in Earth-Lunar or up to 40 minutes in Earth-Mars communication [19].

As latencies increase, control paradigms have to change as well. Manual control (or direct teleoperation), involving the complete control of the actuators on the robot by the operator, must be increasingly enhanced with supervisory control, in which the operator sends intermittent commands to a (semi-)autonomous robot and intervenes only when necessary. Manual and supervisory control may be utilized by the same telerobot, as evidenced by the popular Mars Curiosity Rover [15]. Due to the high latency of Mars communication, operators of Curiosity could choose between three layers of control to drive the rover [18], applying a concept also known as traded control [12]:

1. 1.

   Low-level motor control commands that specify exactly how much to rotate each drive wheel and turn each steering wheel actuator.

2. 2.

   Directed driving primitives for driving along circular arcs (of which straight-line driving and turn-in-place are special cases).

3. 3.

   Autonomous path selection for reaching a goal while reacting to unexpected changes along the way.

Another perspective on these different layers is to look at the type of tasks the user takes on in such situations, i.e. rather on a strategic level (3) or more towards tactical (2) or operational (1) involvement. In particular, on the tactical and operational level, the possibility of shared control arises [11]. Such approaches aim to combine human input with robot operations without a clear traded control scheme, i.e., both operator and human act or respond, and the overall resulting action is a combination or fusion of these different inputs.

Approaches from previous research and literature show that this can often be used to optimize human input through the use of machine learning and deep neural networks, e.g. CNNs [10, 22, 6, 23]. This shift necessitates the development of advanced human-robot autonomy frameworks [12, 1] but also mixed-initiative control strategies to maintain effective operation despite communication delays.

A possible direction in the context of variable control paradigms is the integration of Large Language Models (LLMs) to use natural language as a primary interface for human-robot communication [7, 2, 17]. With their ability to understand and generate context-aware dialogue, LLMs enable robots to be collaborative agents that are capable of participating in mission workflows as embodied conversational team members. Equipped with an LLM, a robot can report its internal state, explain its behavior, and reason about its environment using natural language. This enables a more natural interaction where operators can describe high-level goals or ask situational questions (e.g., “Are there any promising rock formations nearby?”), and the robot can respond accordingly (e.g., “I detected two interesting objects 15 meters northwest. Would you like me to investigate them?”). Such capabilities transform the robot into an active collaborator that can proactively provide suggestions, communicate uncertainties, or propose alternative aspects of the mixed-initiative paradigm that we propose. Future research should explore how LLMs can serve not only as conversational interfaces but also as reasoning layers for agent-like behavior in robotic systems. This includes dynamically adjusting language to support the operator’s requirements, reasoning about high-level goals and constraints, and engaging in the negotiation of roles between humans and robots in long-term missions.