A Postcard from Mars: Exploring Interplanetary Communications in Virtual Reality

ISEs [20], a methodology originating from other non-immersive HCI speculative methods. The aim of ISEs is to use the results of the analysis of the user from these VR studies to provide insights that can potentially be applied in the real-world and to stimulate discussion on these topics.

Conventional speculative methods include Design Fictions [3] are a method that consists in describing a possible future or artefact, and using this opportunity for subsequent reflection. User Enactments [18] represent a more hands-on approach in which the possible future is enacted in the real-world. In Speculative Enactments, use speculative methods and real actors to engage participants in reflecting about a fictional artefact or scenario in the real-world. ISEs combine these methods but use VR to provide a realistic representation of the possible future the experimenters had in mind. This approach benefits the limitless potential of VR in allowing participants to experience a vivid recreation of the speculative scenario, but also “locks in” the imagined future to how the experimenters designed it, whereas in other conventional methods user imagination can play a bigger role.

Other examples of ISEs include using VR to analyse the usability of an artifact [28], to evaluate a system to support users in remaining aware of out-of-view incoming traffic [28] and to investigate increasingly more distant future scenarios and technologies [6, 20].

The intersection of HCI practices and their application in the domain of space exploration constitutes a nascent trend. With the increase of the capabilities provided by contemporary VR headsets, a number of studies have sought to explore how VR can be used for this purpose. Nilsson et al. [16] also used VR to simulate a lunar lander and involved astronauts and other related professionals to critically evaluate the design choices, in a simulated lunar environment. Their results align with the ones we obtained through this ISE, in identifying the limitations of the level of immersion provided by VR. In a related study [7], the use of physical gloves to enhance the believability of the simulation was also evaluated, showing positive effects on presence. Using more conventional means, Freitas et al. [11] developed Conversational User Interfaces to support astronauts operating in extraterrestrial habitats, using a Wizard of Oz to provide guidelines for such systems. In [27], Vanderdonckt et al. investigated the suitability of existing interaction design frameworks for application in extra-terrestrial settings, suggesting future research to take a greater focus on the analysis of these themes.

The design of the interior habitat (see Figure 3) was inspired by a NASA-sponsored 3D Printed Habitat competition ran in partnership with “America Makes”. The goal of the challenge was to design habitats that could be built using 3D printers which would use local materials or even a mission‘s waste products. It must be noted that designs for deep space habitats are often envisioned as cylindrical. However, to utilise the space available in our VR lab to its fullest extent, we decided to model the room in 1:1 scale with the physical bounds where the ISE would take place, a room of $4\mathrm{m}\times 4\mathrm{m}$. Although the presence of a glass wall showing the exterior might not be realistic for an habitat built with today’s knowledge, we decided that seeing the planet’s exterior might reinforce the illusion of being there.

Transition to the exterior was handled by touching a button placed next to the airlock. This would trigger a teleportation to the exterior area. In order to reuse the physical space, users found themselves rotated ${180}^{\circ }$ after the teleportation. This allowed us to reuse the $4\mathrm{m}\times 4\mathrm{m}$ physical area for the exterior. This area, delimited with railings, contained the manual override panel to control the communication dish’s orientation – the objective of the task (see Figure 5). Once outside, the user’s view was modified to give the semblance of wearing a helmet, but due to time and resource constraints we did not implement a realistic airlock transition nor suit-wearing interactive behaviour.

Where conventional enactments use materials such as short videos, in-person events, brochures, etc. to prompt the speculation [8], the immersive analogue consisted in a sandbox VE. The VE, through its graphical design, reinforces the boundaries of the enactment, while the interactive elements of the VE allow the users to express themselves and improvise their reaction. These elements together provide the lens through which we can observe a glimpse of their speculative behaviour in such a situation.

Since the time spent by users who decided to engage with the task would represent only a minor proportion of the total immersion time, we brainstormed about what kind of activities a prospective space colonist would be interested in doing, while waiting from a reply from Earth. The design of the environment had to support the open-ended nature of the enactment. While the narrative suggested a “mission” the user had to complete, they remain free to completely disregard it.

For the design of the interactive aspects, We were inspired by contemporary science-fiction games such as Deus Ex: Human Revolution or Prey (2017), which portray sequences in otherwise ordinary environments. For example, the initial “fake” apartment of the protagonist in Prey, or Adam Jensen’s or V’s apartments in Deus Ex or Cyberpunk 2077. With some notable exceptions, these do not usually provide the player with major gameplay activities or puzzles, but rather they provide contextually appropriate interactive elements that players would expect to find in such settings. For example, food items, reading materials, watching the TV, checking the mail on a computer, taking a shower, etc.

Similarly, we added several interactive objects that could be manipulated via the natural interaction metaphors [15] commonly found in VR games that leverage the availability of 6-DOF controllers. Thus, small objects were not static, but could be grabbed by the user and used to interact with the environment (e.g., a pen to write on the whiteboard) or inspected more closely (e.g., an ebook reader, a tablet with instructions on the task). The environment also presented various other interactive objects such as: a music player, drawers that could be opened by grabbing the handle and pulling towards oneself, screens that could be switched on and off, reporting readings from the nearby hydroponics farm (which was inaccessible) and solar panel array (see Figure 3-4).

The scenario depicted in the ISE had users play the role of colonists on another planet. The experience was purposely set in the hypothesised early stages of human space exploration. Participants were not intended to represent the “first humans” on Mars nor the the first to return to the Moon after the 20^th century. In the scenario, it is assumed that a (semi-) permanent presence on either Mars or the Moon has been established and that some initial infrastructure has been built as well.

We created a plausible backstory that we relayed to each participant before starting the experiment. Participants play the role of technical staff being sent to perform routine maintenance to an automated outpost. A distinction between an imagined larger settlement and a more isolated outpost was made in order to explain why participants would find themselves alone in this location.

The VR experience begins inside the habitat. After one minute in the VE, a notification of an incoming message will be shown on the screens (Figure 3). The message urges the participant to walk towards the “VR Communication station” (Figure 4) and use the headset provided. This additional “virtual” VR system provides a VR-in-VR experience, where they will be able to play the message back, and eventually reply. The motivations behind the choice of using a second layer of VR are discussed successively. The message instructs the participant to go outside and operate the manual override of the external communication dish’s control panel (see Figure 5). The mission control on Earth requires this to be done in order to correct the course for an incoming delivery on the surface of the planet. The message contains an animation of a sequence of buttons they need to press on the device’s control panel located outside, in order to re-orient it. It also informs them that a reference of this information can be found on a (virtual) tablet located on one of the desks, which they can pick up to remind themselves of the correct sequence.

Once outside, they can open the panel by holding the trigger and performing a pulling motion with the controller. They can press the buttons by colliding the controller with each one. By entering the correct sequence, the dish will rotate to the needed position. A series of coloured LEDs informs the user whether the sequence they have entered is correct or wrong, in which case it resets and they can try again. After performing the task, participants are instructed to return inside, send a response through the VR-in-VR system, and then wait to receive a confirmation back. In the Lunar environment this confirmation arrives almost instantaneously, whereas in the Martian environment the communication was artificially delayed until 10 minutes had elapsed. This duration was chosen to simulate a situation in which the two planets were relatively close, to keep the duration of the experiment manageable.

We opted for this solution for two reasons, one practical and one design-oriented. Firstly, because the only actual example of communication between parties located on different celestial bodies is represented by the Apollo radio exchange. In hard science-fiction media (that is, media that attempts to maintain a degree of scientific realism – such as in The Expanse books and TV show, as opposed to the Star Wars universe), the thopos of interplanetary (or even interstellar) communication is represented by asynchronous video message fragments that are recorded at one end, then sent to the receiving end. Due to the lack of resources necessary to hire professional actors to record the messages originating from Earth, we considered this alternative.

We considered the feasibility of such a VR communication system based on current technological capabilities. For instance, NASA’s Curiosity rover can establish a direct link with Earth at a rate of 32kB/s. When routing data through the Mars Reconnaissance Orbiter, transmission speeds can reach up to 2MB/s, whereas the Odyssey orbiter supports rates ranging from 128kB/s to 2MB/s. However, these connections are constrained by power availability and orbital alignments, limiting the transmission window to just eight minutes per Martian sol. The Mars Reconnaissance Orbiter then forwards data to Earth at a maximum rate of 750kB/s.

Given these constraints, the direct streaming of audiovisual data from Mars to Earth would likely be impractical in the early stages of human colonisation, particularly for non-essential communication. The establishment of a deep space Interplanetary Internet might alleviate this problem [1]. Transmitting short encoded messages, which would then be interpreted and reconstructed locally through the help of an embodied or conversational user interface might feel more “natural” for the user, as also investigate dby Goncalves et al. [11].

In our implementation we assume that the outpost provides such an “AI” system. Upon receiving a message from Earth, the AI, personified by a three-dimensional avatar (see Figure 6) can voice it and thus relay any instruction to the user of the VR station. This introduces an additional VR system in the VR experience itself (a similar approach was used as a metaphor for a VR transition technique [17]). Participants can “log into” this VR-in-VR system by walking towards the VR station, grabbing the virtual headset and moving it towards their head. They can remove it by performing a similar, but inverted, motion. The VR station is shown to the right side of Figure 4, under the green light. Once in the system, a 3D User interface allows the user to interact with the local “AI” who reports the received message and communicates it through synthesised speech recordings.

Although the data rate will be less of a problem in Earth-Moon transmission, to provide similar conditions across the two scenarios, the VR-in-VR system is also present in the Moon scenario.

We recruited 26 volunteer participants (19 Male, 7 Female; $M=29.23,SD=5.43$) randomly assigned to one of the two groups (13 in the Martian environment, 13 on the Moon). They were asked to rate their average experience with VR technologies on a scale from 1 (use VR rarely) to 7 (use VR multiple times per week), with a mean self-reported rating of $3.19(SD=1.79)$.

The study protocol was reviewed and approved by our University‘s Ethics Review board. Participants were instructed that they would receive a communication from Earth asking them to perform an activity outside. We told them that they were free to act as they wished. We added a maximum time-limit of 20 minutes after receiving the first message, in the event that participants refused or were not able to complete the task as described. However, this limit was never reached and every participant was able to finish the task.

To avoid the so-called “wow factor” experienced by people being immersed in VR for the first time, participants were first required to undergo a tutorial session instructing them on how to interact with the objects, how to move, and how to open doors. The assumption is that, were they actually in that situation, they would not be unfamiliar with the equipment present therein. After completing this tutorial, the actual experiment begins and was then concluded after receiving confirmation from Earth in the Martian scenario, or after spending a further ten minutes in the Moon scenario, in order to have a comparable duration between the two conditions.

We recorded each participant’s first-person interactions in VR. Explicit consent for the treatment and use of the video and audio recordings was collected. After the conclusion of the experiment, participants were required to fill-in two questionnaires: the Slater-Usoh-Steed Presence questionnaires [25] and Kennedy’s Simulator Sickness questionnaire [13]. Successively, we conducted semi-scripted interviews with the participants based on the following list of topics, and asked further clarifications where necessary.

1. 1.

   Their experience of the VR-in-VR system.

2. 2.

   Their behaviour in VR.

3. 3.

   Whether they felt present in VR.

4. 4.

   How they felt emotionally while performing the task.

5. 5.

   Whether they would behaved differently if the scenario were actually real.

This theme focuses on the inclination of participants to suspend their disbelief and behave in a way adherent to the scenario’s setting. We associated 41 of the total 224 items to this theme, which contains two codes: 1) those participants who did not a priori suspend their disbelief of the ISE (23); 2) those who approached it with a “gaming mindset” (18).

Seven participants explicitly expressed that they were sceptical of the potential of the VR experience to provide a believable rendition of an extra-terrestrial scenario. They approached it with a detached attitude. Participant #24 said: “It’s just VR, the worst thing that can happen is that it’s game over”, also echoed by P25: “I think it’s a game, so I came to it like a game. Doesn’t matter much.” P9 stated that “I did not really think [my actions] would really set something in motion.”

Again, P24 questioned whether we should even try to make VR experiences more believable, as the “the feeling of failure [could] escape the VR and bring it from the outside to inside.” Referring to the potential of negative VR experiences transcending the boundaries and affecting our real-life emotional well-being. For example, the backlash from failing a training simulator drill. P6 motivated their behaviour due to the knowledge that the experimenters were present in the room: “I would have acted different if I knew that people weren’t watching me. I would have talked more with myself.” Further, we did not initiate any conversation, but some participants asked for permissions. For example, P9 said “I was a bit hesitant to pull that [lever]. If I had not asked anyone if I could pull it, I would have left it alone.” P9 instead considered the element of time as being crucial to the suspension of disbelief: “Within a small time fragment, how much can someone get convinced that this is real?”.

P21 said: “I was basically just trying everything out.”, echoed by P8: “Like because I game also so I am used to try to explore a little bit of everything.” These statements reflect the behaviour of those players who enjoy exploring a game to its fullest extent and exhibit completionist or hoarder attitudes [4, 12]. Inevitably, this play style will confront players with the limitations of the game or system. Indeed, due to time constraints, only the objects described in the Design of the Outpost section were manipulable. The scene, however, showed several others that might have been interactive. P8 wanted to pick up all the objects in the cabinets (see Figure 4), while P13 wished they could have been able to use the computer (see Figure 3). P13 lamented the difficulty in using the controllers to perform precise movements such as using the crayon to draw on the whiteboard. P18 attributed their desire of interacting with the objects due to a sense of loneliness: “The feeling that you are on this planet alone. […] There’s this sense of loneliness. That’s why I was looking at other things to do, like the radio, the books…”

P24 and P26 highlighted the impossibility of sitting down as a strong cause of breaks-in-Presence, perhaps because a chair was shown (see Figure 3). P26 stated: “[…] one thing that I miss because I really wanted to do is that you cannot sit down. That’s something that disconnects you a lot. Because it’s something you would normally do. We are not usually standing up in real life.”

This theme describes the reasons participants gave to explain why they felt they behaved in a way that was different from how they thought they would behave were the situation real. We associated 53 items to this theme, which comprises two codes: 1) those who associated urgency to danger (35); 2) those who associated it to a lack of consequences (18). In the following, we discuss the reasons behind this contrast.

The majority of participants acknowledged that, were they really on Mars or on the Moon, they would have behaved differently. Twenty-six items described the reactions they would have had in a real scenario. Overall, participants felt they would have exhibited a more mundane behaviour, saying that they would not have tried to explore and interact so much. P8 and P18 would have read the book during the wait; P9 and P21 would have switched to other tasks; others reported that they would not have interacted with the objects and perhaps just sat down to wait.

P2 and P5 motivated their behaviour by hypothesising that in real-life they would be already familiar with the environment depicted in the outpost or others similar to it, and thus being already experienced, interaction with objects would hold no novelty for them. P24 mentioned the issue of authority: “I would know whether [exploring or interacting with objects] was allowed or something like that. If it was allowed I would probably do it.”

Two participants said they would not spend so much time outside the outpost. P4 explained: “I know it’s a game about Mars, but I have to go outside. It will be the only experience I will ever have of Mars. I have to go out to see what it’s like on Mars.” P8 found it a scary experience: “I wanted to experience Mars because I won’t be able to do so at home any more and yet it was a thrill as well because it is so scary.”

This theme resumes the feedback users gave on the VR-in-VR system and the concept of communicating between different planets. Fifty items were associated to the theme.

The system received mixed feedback depending on whether its purpose was understood or not, that is, the idea of reproducing locally the received text communications in place of video or other more demanding media in terms of bandwidth. Those who considered it superfluous would have preferred a video message. Others appreciated the presence of a human virtual character. P7 said: “I think the fact there was a figure and not just the sound of a voice was helpful.” The feedback we received also highlights issues in interplanetary communications: the temporal validity of these messages. Both parties react on messages that by the time they are delivered refer to a situation already in the past. P26 mentioned that it was not possible to have feedback on the progress of the task or what to do in case the orders they had begun to carry out had been superseded by an evolving situation back on Earth, whose information was still travelling towards them. Four participants did not understand there was a virtual tablet that resumed the information they got via VR, despite it being mentioned in the message. Having redundant information that actively attempts to catch the user’s attention seems essential in ensuring the information is conveyed and understood correctly.

Based on the results of the thematic analysis, enhancing objects’ physics behaviour might improve the believability of the experience. Sjolie et al. state that the VE should generally avoid violating the assumption that it is depicting a truly real experience [21]. Simulating object behaviour to a realistic extent would be a further element supporting this assumption. If being “careless” or behaving in a way that is not adherent to the scenario’s context resulted in breaking or damaging objects and thus rendering them unusable, it is conceivable that participants could have behaved differently and tried to be more careful. Especially if these objects were critical to the mission.

In games, those that allow users to manipulate objects usually treat them as indestructible (e.g., Ultima 9 [24], Skyrim [23]). In “The Gallery” [9], a VR game, some objects shatter when thrown. In a non-VR game, users manipulate objects through intermediary expedients (such as pressing a button to “equip” an object), thus breaking an object could be seen as a source of frustration since the user is not directly responsible [10]. Players of egocentric VR games are instead usually directly expressing agency on an object through natural interaction, and thus might find it more acceptable. Further exploration of these insights might have significant implications on player experience and user behaviour. Designing uncomfortable interactions [2] as a component of ISEs (such as having to carry weight or wear garments simulating a spacesuit) might also positively reinforce behaviour that is adherent to the scenario.

Differently from conventional enactment, ISEs present a stark contrast between the moment before immersion, and immediately after. Literature has investigated the moment of entering [22, 26] and exiting VR [14]. Introductory sequences are a mainstay of digital games. Analogously, we think that designers should use them to ease the immersion of the participant from a previously real and mundane environment, into the world of a possible future. With reference to this ISE, an example introduction could be a sequence where the participant is aboard a vehicle travelling towards the outpost. Such sequences could also give designers opportunities to relay instructions to participants. Performing the stage-setting directly through the ISE can remove links that tie the participant to the outside world, and thus potentially lower their immersion.

The ISE presented participants with one specific solution to this challenge, the VR-in-VR system. Based on our observation of how participants used it, the two most critical aspects that future designers should focus on are the recording and conversation support phases. The AI assistant was only used to relay messages, however participants indicated that talking to someone would have been preferable, based also on their own experience of sending voice clips in contemporary instant messaging apps. Using the AI as a listener could provide a less artificial setting. Further, in addition to providing feedback on the delivery of their own messages, such systems should estimate a time window of when a reply might be received. Indeed, we observed participants going back in the VR system or glancing at the screen for such information.

It would also be beneficial to integrate new AI embodied conversational agents based on Large-Language Models that could provide further clarifications based on the communication received so far. Ideally, such an agent would be “trained” on pre-existing mission knowledge. Incoming broadcasts would update the current status of the mission and allow the extra-terrestrial operative to ask further questions, before formulating an actual response to send back. By clearly distinguishing between the real communications and those extrapolated by the conversational agent, the operative could use this as an opportunity for further reflection and analysis. It would also be useful to replicate this approach on the other side, with a conversational agent simulating what is known about their status back on Earth. Future research should investigate these more complex form of exchanges to provide further insights on interplanetary communications.