Human-AI Interaction in Space:
Insights from a Mars Analog Mission with the Harmony Large Language Model

SpaceCHI [45, 46, 63] represents an initiative of the HCI community to investigate interactive computer systems in space, as a representative ICE environment, by “designing new types of interactive systems and computer interfaces that can support human living and working in space and elsewhere in the solar system” [45, p. 1]. SpaceCHI emphasizes the diversity of topical coverage in space exploration requiring HCI knowledge and expertise [3], including software for crew collaboration [54], mission planning [63], human-system resilience and design for maintainability in space [37], participatory design for space systems engineering [40], food experience design for space travel [41], and examinations of the influence of extraterrestrial conditions on designing interactive systems [16, 17, 30].

However, the UX of interactions with computer systems in space has been briefly addressed in the scientific literature. For example, a self-scheduling application evaluated during a Mars analog mission [49] reported UX varying across pragmatic and hedonic dimensions, while Graphical User Interfaces (GUIs) for astronauts have started to be the subject of systematic UX design and assessment [19, 53]. Other research has looked at specific interactive computer technology in space. For example, interviews with astronauts and space experts regarding the capabilities of virtual environments facilitates user-centered approaches to operational performance [40]; MoonBuddy [7] consists of a voice-based VR system for extravehicular activities (EVA); Telemetron [16] is a musical instrument to play in ICE environments with low gravity; and Minerva [11] applies user-centered design for human exploration.

AI technologies are increasingly utilized in space stations to enhance crew safety, productivity, and autonomy [15, 44]. Recent research has explored Human-AI interaction (HAI) in various contexts of use, including space applications. For example, a semi-formal representation of HAI using a set of interaction primitives and patterns was proposed in [56] with applications to text summarization [8] and, at the International Space Station (ISS), AI models have proven valuable for data analysis and automation [43]. Specific AI technologies deployed at the ISS include NASA’s Robonauts for assisting crew members and ATLAS for asteroid detection. AI has also been explored for providing medical aid to astronauts, such as in the absence of a doctor for the treatment of a fractured tibia [32], data analysis [43], and support for exploratory operations [21]. In this context, critical HAI aspects emphasize the need for mutual trust and reliability [44]. Since astronauts cannot access and manage the vast and diverse knowledge required to conduct space missions, AI assistants [5, 8, 12, 15, 21, 43, 44, 56] could help with finding answers to their questions in context. For example, Bensch et al. [5] combined information retrieval techniques, knowledge graphs, and Augmented Reality (AR) cues in their AI assistant for spaceflight operations [6]. These advancements enhance astronaut autonomy [4] while simultaneously addressing safety and assurance considerations [44], critical for practical implementations.

The recent interest of the HCI community in contributing to humanity’s quest for reaching [34], working [31], and living [57] in ICE environments for space exploration [63], sets the context for our examination reported in this paper, where we focus on the UX of analog astronauts interacting with AI assistants in such environments. While proper UX design [22] can optimize the interaction between users and systems in conventional contexts, it requires careful and extensive examination in extraterrestrial environments [39]. Related research has primarily focused on technical aspects, such as hardware and software reliability or communications latency [45, 46, 63, 9] and, when tackling UX, on effectiveness and performance aspects. In this context, the need for research on how the specific challenges of ICE environments impact interactions and UX remains largely unaddressed [60]. Moreover, the growing interest in integrating AI in applications demands examination of the UX resulting from increasingly frequent human-AI interactions. Our work lies at the intersection of SpaceCHI, AI, and UX.

The software architecture of Harmony is composed of four modules, each responsible for a specific task in the processing pipeline (Figure 2 and Table 1), as follows:

1. 1.

   Audio Capture: Audio input is captured using a microphone and activated with a mouse click acting as a push-to-talk control. This interaction simplifies hardware requirements and allows for flexible input in low gravity conditions [17].

2. 2.

   Speech-to-Text Transcription: The OpenAI Whisper ASR model, a robust and accurate English speech recognition engine, runs locally and transcribes the astronaut’s question into text without requiring an Internet connection.

3. 3.

   Query Generation and Answering: The transcribed question is processed and sent to an LLM based on GPT-3.5-turbo, which formulates a concise, mission-adapted response. This module operates either locally, with a fine-tuned model, or via the Google Cloud API when a communication link with Earth is available.³³3An LLM capable of running locally on the onboard computer was initially implemented but, during the mission, we considered an LLM based directly on GPT-3.5-turbo for faster and more relevant responses.

4. 4.

   Text-to-Speech Response: The generated answer is converted back into audio using the Google Text-to-Speech gTTS engine, a Python library and Command-Line Interface tool to interface with gTTS API, allowing astronauts to receive direct vocal feedback.

This modular architecture ensures that each component can be independently updated or adapted based on mission requirements. For example, the LLM backend could be swapped for an onboard model in deep space scenarios with communication latency, while a more advanced Text-To-Speech module, such as Tacotron 2 or Coqui TTS, added to the architecture to improve perceived naturalness of voice-based feedback.

To foster trust and reliability [44], Harmony employs a conversational user interface [48] designed based on the following principles [1]: transparency to explain the reasoning behind the provided suggestions to ensure astronaut confidence, adaptability to personalize responses based on individual crew preferences and roles, and error management to detect and mitigate inaccuracies through validation mechanisms, e.g., reinforcing or deforcing an answer. Figure 3 presents a concrete system walkthrough illustrating how Harmony operates in a real-world scenario. This step-by-step description follows the typical interaction between astronauts and Harmony, from voice input to a vocal response: a context of is first provided to the analog astronaut with a goal to achieve; subsequently, the astronaut formulates mentally a question corresponding to the goal ($\mathrm{\Delta }{t}_1$); the astronaut enters the question with voice input ($\mathrm{\Delta }{t}_2$); the question is transcripted and sent to the LLM to produce an answer that is converted to audio to preserve the conversational style; lastly, the astronaut rates their satisfaction regarding the provided answer to improve the LLM training.

The experiment took place at the Mars Desert Research Station (MDRS), a simulated Mars-inhabited environment in Hanksville, UT, USA, serving as an international research facility for studying human factors and conducting experiments for future Mars missions [18, 47]. MDRS is composed of several modules, including the habitat, the EVA preparation room, and the science dome with a science laboratory; see Figure 5 and a 3D navigation in VR.

The experiment was carried out with a crew of seven analog astronauts (four women and three men), aged between 21 and 31 years (M=24.75, SD=2.63), who spent two weeks at the MDRS with multiple daily reporting and protocols. Participants had different backgrounds, such as astronomy, biology, chemistry, computer science, geology, and engineering.

Based on [49, 53], we defined a series of contexts of use, each associated with a situation and a corresponding question, as follows (see Figure 4 for an example):

• $\mathrm{■}$

  $Q_1$. Medical assistance in case of injury: Your teammate has just dislocated their shoulder on a mission. Formulate a question to the AI to learn how to react and what first aid measures to apply.

• $\mathrm{■}$

  $Q_2$. Cooking in space with dehydrated products: You want to prepare pancakes using only dehydrated ingredients. Ask the AI for a suitable recipe.

• $\mathrm{■}$

  $Q_3$. Entertainment: You want the AI to play the role of a general knowledge quiz host. Formulate a question so that it suggests an interesting challenge.

• $\mathrm{■}$

  $Q_4$. Communication with a foreign colleague: You meet a foreign colleague and have difficulty communicating with them. Ask the AI for translation or communication tips.

• $\mathrm{■}$

  $Q_5$. Recipe ideas with limited resources: You have harvested basil and you want to use it to prepare your meal. Ask the AI for two or three dishes ideas.

• $\mathrm{■}$

  $Q_6$. Language learning in your spare time: You want to learn a few words in a new language in your spare time. Ask the AI a question, which will teach you useful vocabulary.

• $\mathrm{■}$

  $Q_7$. Physical exercise in a confined space: You are confined to a small space for a long time and want to keep fit. Ask the AI for a physical exercise routine that can be performed without specialized equipment.

• $\mathrm{■}$

  $Q_8$. Project management skills development: You want to improve your project management skills in your spare time. Ask the AI for resources or a learning plan.

• $\mathrm{■}$

  $Q_9$. Using natural resources: You are looking to harness the natural resources of your environment. Ask the AI how to identify and use these resources effectively.

• $\mathrm{■}$

  $Q_{10}$. Team building: To strengthen team cohesion, you want to organize a team-building game. Ask the AI a question to get ideas tailored to your mission.

• $\mathrm{■}$

  $Q_{11}$. Water purification in the event of a technical problem: A technical problem affects your water supply. Ask the AI what alternative purification methods you can implement.

• $\mathrm{■}$

  $Q_{12}$. Night navigation without instruments: You are lost without a compass and want to find your way using the constellations. Ask the AI a question to learn the basics of astronomical navigation.

• $\mathrm{■}$

  $Q_{13}$. Setting up a recycling or composting system: With limited resources, you want to set up a small recycling or composting system. Ask the AI for advice on how to do it.

• $\mathrm{■}$

  $Q_{14}$. Stress management and relaxation: You are experiencing a lot of stress and would like to practice a relaxation technique. Ask the AI for instructions for a relaxation session.

• $\mathrm{■}$

  $Q_{15}$. Recognizing and treating hypothermia: You are confronted with extremely cold conditions and suspect hypothermia. Ask the AI how to identify symptoms and what treatment to apply with the available means.

To evaluate UX, we relied on the UEQ+ method, a modular extension of the User Experience Questionnaire (UEQ) [51, 52], which covers both pragmatic and hedonic UX dimensions and is supported by analysis instruments and published norms [38] for interpreting results [25]. Based on prior work [59, 60], we selected the following UX dimensions for our experiment:

• $\mathrm{■}$

  Attractiveness: The overall impression concerning Harmony. Do users like it or not?

• $\mathrm{■}$

  Efficiency: The impression that tasks can be successfully performed with Harmony without unnecessary effort. Can users solve their tasks efficiently?

• $\mathrm{■}$

  Perspicuity: The impression that Harmony is easy to learn how to use. To what extent do users find Harmony easy to learn?

• $\mathrm{■}$

  Dependability: The impression to be in control of the interaction with Harmony. To what extent does Harmony give users the feeling that they are in control of the interaction?

• $\mathrm{■}$

  Stimulation: The impression that Harmony is interesting and fun to use. Do users find Harmony exciting and motivating?

• $\mathrm{■}$

  Novelty: The impression that Harmony is creative and original. To what extent do users appreciate Harmony as creative? Does it catch their interest?

• $\mathrm{■}$

  Adaptability: The impression that Harmony can be easily adapted to personal preferences or working styles. To what extent does Harmony appear adaptable?

• $\mathrm{■}$

  Trust: The impression of the users that their data are safe with Harmony and not misused to harm them. Do users feel confident that their data is secure and handled appropriately?

• $\mathrm{■}$

  Usefulness: The impression that using Harmony is beneficial. Do users perceive any advantages in interacting with Harmony?

• $\mathrm{■}$

  Value: The impression that Harmony looks professional and valuable. To what extent does the design of Harmony convey a sense of professionalism and quality?

• $\mathrm{■}$

  Visual aesthetics: The impression that the graphical interface of Harmony looks beautiful and appealing. Do users find the visual design of Harmony attractive and engaging?

• $\mathrm{■}$

  Intuitive use: The impression that Harmony can be used immediately without any training or help. Can users start using it right away without needing help or instruction?

• $\mathrm{■}$

  Trustworthiness of content: The impression that the information provided by Harmony is of good quality and reliable. To what extent is the information provided by Harmony perceived as accurate and trustworthy?

We refer to UEQ+ [51, 52, 38, 25] for further details about these dimensions. Following prior research [60, 59], we report for each dimension the Scale-Mean-Score (SMS) as the average score obtained on all its subscales, ranging from $-3$ (negative experience) to $+3$ (positive experience), and the Scale-Mean-Importance (SMI), representing the average weight of importance of a given UX dimension; see Vanderdonckt et al. [59] for calculation details. Furthermore, to compare the scale of a target, e.g., an extreme condition such as Mars, to the corresponding scale of a baseline such as Earth, we use:

Participants’ answers were computed with the UEQ data analysis tool and interpreted according to Schrepp et al.’s [52] recommendation: “the standard interpretation of the scale means that values between $-0.8$ and 0.8 represent a neutral evaluation of the corresponding scale, values superior to 0.8 represent a positive evaluation.”

We also measured:

• $\mathrm{■}$

  Quality-Score: a numerical variable representing the perceived quality of the answer provided by Harmony to the question prompted, from 0 (failure) to 1 (lowest quality) to 5 (highest quality for complete and correct answers).

• $\mathrm{■}$

  Success-Factor: a numerical variable defined as the number of iterations needed to complete a task successfully.

• $\mathrm{■}$

  Thinking-Time: a numerical variable defined as the time elapsed between the moment when the task was presented and the moment when the participant started the interaction with Harmony, measured in seconds with a stopwatch ($\mathrm{\Delta }{t}_1$ in Figure 3).

• $\mathrm{■}$

  Production-Time: a numerical variable defined as the time needed to formulate and record the question, measured in seconds with a stopwatch ($\mathrm{\Delta }{t}_2$ in Figure 3).

• $\mathrm{■}$

  Confidence: an integer variable expressing the degree of confidence attributed to the answer provided by Harmony for the given context, ranging from 1% (no confidence) to 100% (maximum confidence).

Harmony ran on an Apple MacBook Air with a 13” Retina screen (2560$\times$1600 pixels), 8-core 3.2 GHz CPU, 8 GB RAM, and 256 GB SSD. The Apple AirPods Pro 2 were used for audio input and output. We devised four tasks to be carried out in the experiment:

• $\mathrm{■}$

  A discovery task, in which the participants received a tutorial on using Harmony, lasting between 5 and 15 minutes, and then interacted freely for another 10 to 15 minutes.

• $\mathrm{■}$

  A practice task, in which the participants were instructed to perform a representative task with Harmony to become familiar with it, lasting about 20 minutes.

• $\mathrm{■}$

  A domain task, in which the participants received five stimuli ($Q_1$ to $Q_5$, $Q_6$ to $Q_{10}$, and $Q_{11}$ to $Q_{15}$ in random order). We suggested, but not imposed, a time limit of 10 minutes.

• $\mathrm{■}$

  An evaluation task, in which the participants filled out the UEQ+ questionnaire.

The discovery and practice tasks were performed once before the mission in a dedicated tutorial room (see Figure 5, left). The domain and evaluation tasks were performed four times during four subsequent sessions, as follows: a first session S1 one week before the mission in the tutorial room (Figure 5, left), a second session S2 after four Mars days⁴⁴4A Mars-day, or a Sol, constitutes a solar day on Mars, i.e., the apparent interval between two successive returns of the Sun to the same meridian as seen by an observer on Mars, which is approximately 24 hours, 39 minutes, and 35 seconds on Earth.(Sol 4) in the science laboratory (Figure 5, middle left), a third session S3 after eight Mars days (Sol 8) outside the station and involving light equipment (Figure 5, middle right), and a fourth session S4 after twelve Mars days (Sol 12) outside the station involving heavy equipment, stress, and high fatigue due to the ICE environment and continuous exposure to its constraints (Figure 5, right).

The panel charts in Figure 6 present the Scale-Mean-Scores (SMS) and Scale-Mean-Importance (SMI) for the various UX dimensions. Overall, all scores fall into the positive experience zone, above the 0.8 threshold. This phenomenon is rarely observed in such evaluations [50], which suggests that participants assessed their interactions with Harmony positively throughout the study. While the score range remains mostly similar across sessions, the scales receiving the minimum and maximum scores differ: Visual aesthetics ($M=1.25$, SD$=1.02$) and Attractiveness ($M=2.18$, SD$=0.71$) define the range in S3 whereas Adaptability ($M=1.32$, SD$=1.39$), Attractiveness ($M=2.18$, SD$=0.80$), and Stimulation ($M=2.18$, SD$=0.66$) in S4. The following trends can be distinguished for the observed UX:

• $\mathrm{■}$

  V-shaped curves undergo a noticeable drop, represented by a local minimum of the SMS starting in the first session on Mars, then gradually rising to S4; see Perspicuity, Trust, Value, and Intuitive use ($\frac{4}{13}=31\%$). For example, Value started from a mean score of $M=1.82$ (SD$=0.76$) in S2 and reached $M=1.79$ (SD$=0.86$) in S4.

• $\mathrm{■}$

  Inverted V-shaped curves represent the opposite trend with a local maximum and ending with a lower score; see Efficiency ($\frac{1}{13}=8\%$).

• $\mathrm{■}$

  Overall upward trends progressively increase from the first to the last evaluation session; see Attractiveness, Stimulation, and Visual aesthetics ($\frac{3}{13}=23\%$). For example, Stimulation increased from S2 ($M=1.64$, SD$=1.04$) to S4 ($M=2.18$, SD$=0.66$).

• $\mathrm{■}$

  Overall downward trends progressively decrease from the first to the last evaluation session; see Dependability, Novelty, Adaptability, Usefulness, and Trustworthiness ($\frac{5}{13}=38\%$).

These results should be interpreted based on the importance attributed by the participants, measured with the Scale-Mean-Importance (SMI) scores, as detailed below:

• $\mathrm{■}$

  V-shaped curves ($\frac{4}{13}=31\%$) for Value, Visual Aesthetics, Intuitive use, and Trustworthiness.

• $\mathrm{■}$

  Inverted V-shaped curves ($\frac{5}{13}=38\%$) for Efficiency, Perspicuity, Dependability, Adaptability, and Novelty.

• $\mathrm{■}$

  Upward curves ($\frac{3}{13}=23\%$) for Attractiveness, Stimulation, and Trust.

• $\mathrm{■}$

  Downward curve ($\frac{1}{13}=8\%$) for Usefulness.

Figure 7 shows Quality-Score evaluations of the answers provided by Harmony. A one-way ANOVA revealed a statistically significant difference across the conditions ($F_{14,90}=2.22$, ${}_{}{}^{\ast }p=.013$) with a large effect size (${\eta }^2=0.26$). Tukey’s HSD Test for multiple comparisons found that the mean value of Quality-Score was significantly different between $Q_2$ and $Q_6$ ($p=.027$, 95% CI=[0.16, 5.55]) and between $Q_6$ and $Q_{13}$ ($p=.046$, 95% CI=[0.02, 5.40]). Wilcoxon signed-rank tests among the group of five questions in each session (Sol 4, Sol 8, and Sol 12) revealed Sol 4 and Sol 8 significantly different (z=1.90, $p=.028$, $r=.22$).

Figure 8, top shows the average Thinking-Time participants needed for the various tasks, with no statistically significant difference ($F_{14,90}=1.01$, $p=.056$, n.s.). However, when we considered the total Thinking-Time per session, a Wilcoxon signed-rank test revealed that the Sol 4 and Sol 12 conditions were significantly different (z=2.58, $p=.0049$) with a medium effect size ($r=.31$), as well as Sol 8 and Sol 12 (z=2.62, $p=.0043$). This finding suggests that the participants benefited from a learning effect. We did not find any other significant difference between the other sessions and between the five orders of questions, suggesting that the participants needed a similar time to address questions. Figure 8, right shows the average Production-Time participants needed for the various tasks. A one-way ANOVA revealed a statistically significant difference across the conditions ($F_{14,90}=4.32$, $p\le .001$) with a medium effect size (${\eta }^2=.40$). The average production time significantly increased from one session to the next ($F_{2,102}=14.11$, $p\le .001$) with a small effect size (${\eta }^2=.22$). This result suggests that the participants were progressively more careful in how they entered the vocal command to implement the interaction with Harmony.

Figure 9, left shows how many trials were performed for each question: on average, the questions were assessed as satisfying after the first trial (71%), the second trial (14%), and the third trial (15%), respectively. The answers provided by Harmony to $Q_2$, $Q_7$, $Q_8$, and $Q_{12}$ were judged satisfactory during the first trial mainly due to our participants’ familiarity with the question domain (e.g., food, management). Figure 9, right shows the Confidence evaluations of the answers provided by Harmony, ranging from a minimum of 57.97% for $Q_3$ to a maximum of 71.14% for $Q_{13}$. The mean of 65.61% suggests that the participants had moderate confidence in the answers provided by Harmony, probably due to a lack of traceability and explanation, in line with the Trust scores. Participants repeatedly expressed doubts about the system’s answers, e.g., $Q_{11}$ about water purification and $Q_{15}$ in medicine, except in the case of participants whose area of expertise was relevant. Figure 10 shows the average number of words per question with $Q_{13}$ and $Q_{14}$, $Q_3$ and $Q_5$ standing out with the highest (30) and the lowest number of words (12), respectively.

Table 2 provides a summary of the main UX trends identified in our experiment:

• $\mathrm{■}$

  Attractiveness: The value and importance associated with this pragmatic dimension increased over the evaluation sessions, suggesting that participants developed a more favorable overall impression and came to recognize the significance of this dimension.

• $\mathrm{■}$

  Efficiency: This pragmatic dimension was the most affected by the ICE conditions, which was an expected result since performance represents the primary criterion in such contexts [34, 35, 36], while its importance remained consistent across sessions.

• $\mathrm{■}$

  Perspicuity: This dimension showed a decline in both value and importance across the sessions, though not significantly. This finding suggests that further attention is needed to ensure transparency of system functionality for increased learnability and ease of use.

• $\mathrm{■}$

  Dependability: This pragmatic dimension revealed the greatest decline in value after Efficiency, including in terms of participants’ perceived importance. Moreover, participants’ sense of control over the LLM decreased over time, likely due to the similarity of the responses received and the lack of variation in the level of detail.

• $\mathrm{■}$

  Stimulation: This hedonic dimension revealed a growing importance surpassing what is typically observed in a conventional Earth-like setting. Stimulation was strongly affected from the beginning to the end, where participants reported feeling increasingly stimulated when using the LLM, due to the growing importance also felt in terms of Value.

• $\mathrm{■}$

  Novelty: The scores of this dimension decreased progressively across the evaluation sessions as participants become more accustomed to Harmony, resulting in a reduction in both its perceived value and importance.

• $\mathrm{■}$

  Adaptability: While participants appreciated the consistency of the answers, they expressed concerns that Harmony did not provide any means to adapt responses to their individual needs, preferences, or level of expertise. For less experienced users, responses could benefit from progressively increasing levels of detail according to the request, whereas the more experienced participants preferred concise summaries. These findings indicate a clear need for adaptive response mechanisms according to the context of the question, including its urgent or safety-critical nature.

• $\mathrm{■}$

  Trust: Participants reported feeling increasingly secure when using the system, mainly due to the quality of the responses. Confidence was also reinforced by the reduced number of trials, an observation consistent with findings reported in previous work [44].

• $\mathrm{■}$

  Usefulness: This dimension, initially recognized for its benefits, declines over the evaluation sessions, a finding that highlights the need for strategies to maintain perceived usefulness over time.

• $\mathrm{■}$

  Visual aesthetics: This dimension became increasingly valued over the sessions with a stable level of perceived importance. As such, it does not need major changes.

• $\mathrm{■}$

  Intuitive use: This dimension remained constant, but warrants further attention due to its growing perceived importance.

• $\mathrm{■}$

  Trustworthiness: This dimension requires increased attention as participants perceived it as deteriorating over sessions, casting doubt on Harmony’s interaction capabilities.

The dimensions requiring significant improvement are Efficiency, Dependability, Adaptability, Usefulness, and Trustworthiness (the latter to a lesser extent since it is somewhat compensated by Trust). In contrast, Attractiveness, Stimulation, Trust, and Visual Aesthetics showed positive development over time, despite the ICE conditions becoming more constraining, and do not warrant immediate action. Lastly, Perspicuity and Trustworthiness could be slightly improved, whereas Value and Intuitive Use received consistently positive evaluations.