Gaze Beyond Limits: Integrating Eye-Tracking and Augmented Reality for Next-Generation Spacesuit Interaction

A Head-mounted Display (HMD) is a device that directly projects information into the user’s field of view [29]. HMDs were initially developed for military and aviation applications, primarily serving as targeting systems in fighter jets and helmet-mounted sighting systems to enhance operational efficiency and pilots’ situational awareness. With technological advancements, HMDs have expanded into various fields, including healthcare, industrial applications, and consumer electronics [22].

In space exploration, as human missions become more complex, HMDs are emerging as a critical component in modern spacesuit design to facilitate real-time information display. Particularly for the upcoming Artemis lunar mission, NASA has designated solutions such as wrist-mounted displays and HMDs as key requirements for the development of the next-generation extravehicular mobility unit (xEMU) [1]. Due to the unique challenges of lunar environments – such as the difficulty of depth perception caused by uniform terrain and unidirectional sunlight – astronauts often struggle to accurately estimate distances or navigate effectively. The integration of HMDs aims to provide astronauts with vital information, including biometric monitoring, navigation assistance, and mission-related data, thereby enhancing autonomy and reducing reliance on ground control [1].

Building upon traditional HMD technology, AR Head-Mounted Displays (AR-HMDs) integrate virtual content with real-world environments by using optical combiners and advanced display systems. Unlike conventional HMDs, which primarily focus on overlaying digital data, AR-HMDs enable real-time interaction with the environment, making them particularly suitable for complex operational settings such as space exploration. Optical waveguides and freeform prism optics allow for a lightweight and transparent display, ensuring critical mission data can be seamlessly integrated into the astronaut’s field of view without obstructing vision [10]. Additionally, AR technology provides a more intuitive and immersive interface compared to wrist-mounted displays [26], addressing key challenges such as limited dexterity and complex information retrieval during EVAs. Recognizing these benefits, the NASA/Johnson Space Center has conducted multiple feasibility development programs exploring the integration of HMDs into Extravehicular Mobility Units (EMUs) [25]. These studies have resulted in prototype binocular and biocular HMD systems utilizing conventional optical elements (e.g., glass lenses and beamsplitters) and holographic optics. Furthermore, research efforts have explored the potential for integrating voice recognition capabilities, allowing astronauts hands-free access to mission-critical information through their HMDs.

Eye-tracking technology has evolved as a fundamental tool in applied cognitive science and human-computer interaction (HCI), enabling real-time monitoring of visual attention, cognitive load, and user intent [4]. By capturing gaze direction and pupil dilation, eye-tracking systems facilitate hands-free interaction and real-time data capture, providing insights into cognitive processes. These capabilities have been widely adopted in various fields, including performance psychology and human factors research, to measure task performance, visual attention, cognitive load, and affective states.

The European Space Agency (ESA) conducted experiments aboard the International Space Station (ISS) to investigate how microgravity affects astronauts’ eye movements and balance (Figure 1) [13]. Researchers developed a helmet equipped with high-performance image-processing chips to track eye movements without interfering with the astronauts’ tasks. The findings revealed that weightlessness impacts balance and eye movement control, indicating that our sensory-motor systems rely on gravity for orientation. These insights are crucial for designing eye-tracking systems that function effectively in space environments.

Spacesuit bulkiness and rigidity impose severe restrictions on astronauts’ mobility and dexterity [23]. During every movement of a spacesuit joint within the pressurized environment of the spacesuit, the volume within the suit is compressed or extended, which creates significant resistance against movement, making even simple tasks physically demanding. One of the most critical limitations is the lack of fine motor control, especially in gloves, which affects astronauts’ ability to perform delicate operations such as instrument adjustments, sample collection, and tool handling.

In microgravity, astronauts face additional difficulties due to the lack of stable ground support, which increases the physical strain of maneuvering while counteracting inertia. On planetary surfaces such as the Moon and Mars, the added complexity of navigating uneven terrain while wearing a stiff suit – and managing the inertia of the suit’s mass – further increases the risk of falls and inefficient locomotion.

Voice commands are crucial for hands-free control during EVAs, allowing astronauts to operate systems without manual input. However, they are highly susceptible to interference from ambient noise generated by life support systems, radio disruptions, and microphone inconsistencies [16]. These factors can cause misinterpretations, delays, or failed commands, reducing operational efficiency and increasing workload.

Moreover, spacesuit gloves, designed for protection and pressurization, significantly restrict finger dexterity [23]. This makes precise interactions with buttons, switches, and touchscreens difficult or even impossible, further complicating equipment operation.

The helmet design of current spacesuits significantly limits astronauts’ peripheral vision and provides minimal real-time digital information display. As a result, astronauts often rely on written procedures or pre-memorized sequences learned through repetitive training, which increases the risk of spatial disorientation and reduces situational awareness during complex EVAs. Additionally, astronauts retrieve real-time environmental data based on mission control or pre-memorized details, which is inefficient, especially in deep-space missions with long communication delays [11].

Furthermore, current spacesuits incorporate very few embedded sensors, and many bioinstrumentation signals, such as electrocardiogram readings and body temperature data, are transmitted exclusively to ground-based medical teams, restricting astronauts’ ability to monitor their own vitals in real-time [17].

Traditional gaze-based interaction systems often suffer from the Midas Touch problem [33], where bottom-up fixations and false detections due to involuntary eye movements may trigger unwanted commands, necessitating a more robust mechanism to distinguish top-down gazes from stimulus-driven ones. To address this challenge, our system will employ a two-stage approach to reduce the impact of the Midas Touch phenomenon on the gaze-based system.

Eye-movement modality recognition (EG-NET) and zonotope set-membership filtering (ZSMF) are two algorithms to enhance gaze detection accuracy [36]. EG-NET differentiates between fixations, saccades, and outliers by integrating eye image features and gaze dynamics, filtering out involuntary blinks and tracking losses. To further refine accuracy, ZSMF eliminates noise and stabilizes gaze signals in real time, leveraging an unknown but bounded (UBB) noise assumption for robust performance in dynamic space environments. Together, these methods ensure that only deliberate gaze fixations are processed, providing a reliable foundation for gaze-based segmentation and interaction.

Pupil size is widely recognized as a physiological indicator of cognitive load and attentional engagement [32], which refers to the burden on the brain in processing information, decision-making, and problem-solving. [32] proposed a methodology for integrating an online modeling algorithm that fits task-evoked pupillary response (TEPR) curves to identify temporal variations in cognitive workload. The study indicated that TEPR, which is often used to distinguish between conscious and unconscious blinking, is more strongly driven by luminance changes than by cognitive load, thus a smart filtering mechanism is needed to eliminate light-induced noise and ensure only intentional interactions are captured. By continuously monitoring variations in cognitive load, it can dynamically differentiate between top-down and bottom-up gaze behaviors. As a result, this adaptive filtering mechanism reduces the frequency repositioning of the gaze point, thereby minimizing unnecessary updates and preventing visual clutter.

Furthermore, to reassure that the point of top-down gaze is accurate, the system will first show the steady gaze point after a two-second threshold, followed by a voluntary blink to confirm the steady gazing area. Research [7] has demonstrated that a longer trigger duration can improve this problem, but durations tend to be inconsistent across studies due to different individuals. Blinking is a natural interaction and does not cause Midas Touch problems. Although prior study [24] demonstrated that the double-blink trigger yields the best performance, is had also highlighted that blink frequency tends to increase under physical fatigue or high cognitive load, which may result in unintentional activations. Building upon these insights, our system will build a blink-gaze hybrid control system that ensures accurate and intentional interactions while mitigating issues related to uncertain trigger duration and fatigue-induced blink variability.

Once the astronaut’s gaze is accurately detected and stabilized, the system will utilize the Segment Anything Model (SAM) [34] to automatically segment the Area of Interest (AOI) based on real-time eye-tracking data. By transforming gaze fixation points into input prompts, SAM generates highly precise pixel-level segmentation masks, effectively isolating relevant objects within the astronaut’s field of view, such as Martian terrain, control panels, scientific instruments, or potential hazards.

SAM operates on a prompt-based segmentation mechanism, where the user’s gaze coordinates serve as input prompts to guide the segmentation process. The model consists of a Vision Transformer (ViT)-based backbone, which first encodes the input image into deep feature representations. When a gaze-based prompt is received, SAM processes it through a prompt encoder, mapping the fixation point onto the pre-computed image embedding. The model then generates a segmentation mask that outlines the AOI with high accuracy. This approach enables dynamic object segmentation without requiring predefined labels, making it highly adaptable for unstructured extraterrestrial environments [34].

Unlike traditional segmentation methods that rely on manual selection or predefined object categories [35], SAM allows the real-time identification of previously unseen or unstructured objects in extraterrestrial environments. This capability is particularly valuable for autonomous navigation, geological analysis, and operational decision-making, where rapid object recognition and interaction are crucial.

Following the segmentation of AOIs, the system will employ Visual Language Models (VLMs) to provide real-time semantic understanding of the astronaut’s surroundings [21]. This step enhances situational awareness by interpreting segmented objects, generating descriptive annotations, and delivering contextual information overlays directly to the astronaut’s AR helmet display.

At the core of this process, the VLM leverages a multi-modal transformer architecture, which fuses visual embeddings with pre-trained language representations to generate meaningful descriptions of the segmented AOIs. The workflow begins with image encoding, where helmet-mounted cameras capture real-time visual input. A Vision Transformer (ViT) extracts high-dimensional features, which are then processed through a multi-modal fusion network to align the visual and textual representations. To improve accuracy and relevance, the system incorporates context-aware prompting, adjusting output information based on the astronaut’s gaze behavior, mission objectives, and environmental conditions. This ensures that VLM-generated insights are task-specific and contextually appropriate, reducing cognitive overload in high-stakes EVA scenarios.

Insights from AR-based contextual learning research [21] highlight the advantages of real-time, adaptive information delivery, demonstrating that dynamic, gaze-based content adjustments significantly enhance task comprehension and user engagement. Experimental findings in AR-assisted language learning show that multi-modal interactions, where visual cues are seamlessly integrated with textual feedback, lead to deeper contextual understanding and improved retention of information. By incorporating similar principles, our system will optimize knowledge acquisition and decision-making efficiency for astronauts by ensuring timely, context-relevant semantic explanations.

The system automatically provides options that can be acted upon based on the results of the VLM analysis. It will allow astronauts to navigate the operation menu with eye movement. Building on the previously described blink-gaze hybrid control mechanism in 3.1 (Step 1: Gaze Detection), this step applies the same principle to select action options, ensuring consistent and intuitive user input across different interface elements.

To further optimize usability, we focus on the spatial arrangement and sizing of interactive objects (IOs), preventing accidental activations while maintaining an efficient and ergonomic design. Research [24] has shown that IOs with a 55.5 mm diameter and 33.3 mm spacing significantly enhance selection accuracy and response time by reducing fixation instability and minimizing unintended selections. In our system, these parameters will guide the layout of AR-based control panels and mission-critical interfaces, ensuring that astronauts can effortlessly target the correct option without excessive cognitive strain.

Additionally, visual feedback mechanisms, such as subtle highlights or translucency adjustments, help astronauts differentiate between active selections and background elements, reducing cognitive overload in complex operational scenarios.

Beyond simple menu navigation, the system will enable astronauts to interact with mission-critical interfaces and robotic systems in real-time. For example, when operating a control panel, a sustained gaze highlights a button, and a voluntary blink confirms selection, eliminating the need for manual input. Similarly, in robotic arm operation, astronauts can lock onto an object using gaze fixation, and a blink command triggers grasping or manipulation, facilitating efficient sample collection or tool handling.

To evaluate the effectiveness of our proposed eye-tracking and AR-embedded spacesuit system, we are in the process of designing a highly realistic virtual experimental environment using Unreal Engine 5, allowing participants to experience simulated EVA scenarios. It will replicate planetary exploration tasks, including terrain analysis, robotic arm control, and control panel interactions, ensuring a comprehensive assessment of gaze-based interactions. The four-step interaction process will be fully implemented, allowing the system’s cognitive load assessment and adaptive UI adjustments to be tested under realistic workload conditions. Furthermore, to analyze the constraints imposed by the bulkiness and rigidity of modern spacesuit designs, we have developed a Hard Upper Torso (HUT) simulation platform (Figure 4). The HUT system is a full-scale, modular EVA suit mockup designed to replicate lunar surface operations’ ergonomics when operating in a HUT and includes a constrained field of view from the visor and sun shield. It can further integrate a shoulder cage, soft goods for limb assemblies, and onboard telemetry instrumentation.

Participants will wear VR headsets equipped with integrated eye-tracking, interacting with virtual interfaces that closely resemble the AR overlays and control mechanisms that astronauts would use within their helmets during real operations. They will complete specific EVA tasks within the VR environment using the HUT simulation platform under varying suit configurations and operational constraints, which can validate the utility of gaze metrics as input for suit design optimization and crew training procedures.

The study will compare the efficiency and cognitive workload of our eye-tracking-based interaction system against traditional hand-based control methods in specific operational scenarios, such as control panel manipulation and exploration of unknown terrains. The following performance metrics will be assessed:

• $\mathrm{■}$

  Task Performance Quality: Measured through accuracy in button selection, correct identification of segmented objects (e.g., rocks, equipment), and precision in robotic arm operations.

• $\mathrm{■}$

  Task Completion Time: Comparing the speed of gaze-based selection and confirmation via blink gestures against traditional manual input.

• $\mathrm{■}$

  Cognitive Workload Assessment: Utilizing pupil dilation variations, physiological signals, and subjective workload surveys (e.g., NASA-TLX scale [15]) to analyze psychological stress and mental effort.