Underwater VR for Astronaut Training

Building a budget VR headset from scratch is something, that, ever since Google launched the well-known, but now discontinued Cardboard [4] in 2014, has been the playground for many Makers around the world. As a starting point for this work, the open-source project Relativty [9] was chosen, since the recommended displays offer 2 K resolution at 120 Hz and the kit includes a Mobile Industry Processor Interface (MIPI) to Mini Display Port (mDP) controller board, which is powered using Micro USB, while being still affordable with around 200 USD. To test the basic functionality, a small mock-up was designed and 3D printed, cf. Fig. 2. Two Fresnel lenses with a focal length of 50 mm are mounted on a sliding mechanism, in order to adjust for different interpupillary distances and visual acuities. The displays are connected via the controller board to a desktop PC. This mock-up was used to provide an easy and simple handheld device to work on the virtual environment and the tracking, as described in the next two sections, while adapting the VR headset onto a diving mask.

The diving mask chosen is the Interspiro Divator Full Face Mask [5], due to it being widely spread amongst professional divers, as well as being used by the European Space Agency (ESA) for their underwater astronaut training. The design consists of six different parts, shown in Fig. 2. The first part in dark grey on the right is specifically modelled to fit onto the shape of the Divator visor. It is glued onto the mask and sealed off with a waterproof silicone to fill all remaining gaps between the visor and the part. Next, shown in light grey, is a laser-cut 2 mm silicone gasket. The part in medium grey is a mounting plate for the displays, which is screwed onto the left part, again in dark grey, that contains the controller board and an outlet for the power and display cables. The casing is bolted together with 16 stainless steel screws and nuts, and 32 stainless steel washers to redistribute the clamping force. The two black parts show in Fig.2 are attachment plates for reference markers. To safely guide the power and display cables to the surface, a 22 mm silicone tube is clamped to the cable outlet. One of the biggest challenges of underwater VR is water tightness. Therefore, the whole casing in rev1 was 3D printed with 100 % infill using a FDM (fused deposition modeling) printing. Additionally, any grooves on the connecting surfaces touching the silicone gasket are filled with a waterproof epoxy, sanded down, spray coated, and again sanded down for maximum smoothness. In rev2 resin printing, also known as SLA (stereolithography) has been used. Rev3 was made all-alluminium using milling. Here we have replaced the silicone gasket with an O-ring.

Since all electronics are enclosed and bolted down in the watertight housing and adjusting the lens-to-lens and lens-to-screen distances would be tedious if integrated within the housing, they needed to be mounted inside the mask itself. The Interspiro Spectacle Kit is a solution by Interspiro to use custom lenses with a diving mask. It comes with a mounting frame, which is put inside the mask, and a lens holder, which attaches to the mounting frame via an up-and-down-slideable rubber piece. Since the lens holder is angled and does not accept sound lenses, a custom lens holder was designed and 3D printed to accommodate the Fresnel lenses. To adjust for different interpupillary distances, multiple pieces with 60 mm to 65 mm center-to-center distance are available. To adjust for different visual acuities, the connecting pins are 8 mm long. Also, the lens holder is designed such that the lenses are mounted parallel to the displays. The mounting frame and the lens holder are shown in Fig. 3.

The complete VR diving mask is shown in Fig. 3. The connected cables are an optical mDP to DP cable and a Micro-USB Cable, both 10m long, to accommodate for diving depths of 3 m in a 3$\times$3 m pool while providing enough length to connect to a PC, located next to the pool.

The two scenes shown in Fig. 4 are implemented in the Unity game engine. To create the necessary output stream for the VR headset, a dummy body with two cameras with an interpupillary distance of 60 mm as ”eyes”. The video stream sent to the VR headset has a resolution of 2880$\times$1440 pixels, split vertically into two equal parts for the left and right eye.

One of the goals defined by ESA was to have the Lunar Gateway Space Station as the central objective in the modeled game environment. As there already exist numerous 3D models on the Internet under open licenses, one of these models was selected to be utilized for this project. The chosen model¹¹1https://www.artstation.com/marketplace/p/5j9Jv/gateway?utm_source=artstation&utm_medium=referral&utm_campaign=homepage&utm_term=marketplace provides Unity Universal Render Pipeline (URP) files to embed it directly into Unity. The exact dimensions and positions of the objects in relation to each other have been preserved in Unity, so they match the dimensions of the real objects.

Head tracking systems are used in various applications, including virtual reality (VR), augmented reality (AR) and human-computer interaction, to estimate the movement of a user’s head. In our setup, four cameras are placed on opposite sites around the workspace on aluminium bars with a camera-to-camera distance of about 1.8 m and a bar-to-bar distance of about 3.4 m. On the underwater headset, AprilTag [8] fiducial markers are fixed to allow 6-DoF tracking of the head pose if at least one marker is visible. The cameras are mounted approximately 0.3 m above the ground, tilted inwards by 30 ^∘ and upwards by 15 ^∘ to ensure good coverage of the workspace. A schematic overview of the setup is shown in Fig. 5, top.

The cameras are integrated into underwater housings with acrylic viewports and are connected via Ethernet to a computer outside the pool for image processing and motion estimation. In water the viewport acts like an optical element. Due to the change in refractive index between water, glass and air, refraction occurs. In order to minimize the effect of refraction, dome ports are employed. Dome ports are built from a hemispherical glass shell and aim to minimize the geometric changes of the light path. If the camera center is aligned with the center of the dome port, the ray from the object through the center of projection to the image point is perpendicular to the spherical media interfaces. Therefore, no refraction occurs, and a near-perfect central projection is provided. The advantage of the dome port is that the field of view and image distortions are unchanged if the camera is centered. However, the depth of field increases slightly, and the focus distance is different compared to air. The spherical interface acts similarly to a negative lens element. Hence, the objects captured through the dome port appear to be closer.

Examples of a dome port and flat port underwater housing are shown in Fig. 6. The top left image shows a dome port housing that is rated for a depth of 750 m. The flat port housing depicted in the top right images features a 19mm thick glass window and a depth rating of 1750 m. In the two housings, two industrial machine vision cameras are integrated. Below the housings, example images of the two underwater cameras are shown. The images are captured with the dome port and flat port partially submerged in water.

The top half of the images is in air, while the lower half is in water. The dome port image shows little geometric distortion of the captured chessboard, which suggests that the camera is well-centered in the hemispherical glass window. The sharpness of the image is noticeably different for the part of the image in air and water since the camera needs to be focused at different distances. Here, the camera-lens system was focused for the in-water conditions. The bottom right image visualizes the different magnifications of the flat port interface in air and in water. The chessboard appears closer in the lower half of the image captured in water. The magnification and focal length of the combined camera and housing system are increased.

We align the camera based on the manufacturers specification of the position of the entrance pupil of the lens. Smaller errors in alignment are compensated by in-situ camera calibration [1]. We do not apply a specific underwater camera model but assume that the errors are absorbed by the radial and tangential distortion model. The cameras are individually calibrated in water using a calibration board. The extrinsic parameters of the cameras are estimated online using AprilTag markers placed on a fixed structure in the center of the workspace. This way, the setup can be easily reconfigured without performing re-calibration. The cameras are synchronously triggered and deliver an image stream with up to 80 Hz. For each image set, the head pose is computed by minimizing the reprojection errors of the projection of all visible AprilTag marker corners on the individual camera sensors. After head pose estimation, filtering is applied to smooth the pose output and avoid erroneous fast movements due to tracking failure, which would negatively affect the user.

All structures are build from aluminium bars with aluminium and stainless steel connectors, the cameras are mounted with 3D-printed attachments made from PET. The center piece is a 1$\times$1$\times$0.42 m cubic shaped structure. 8 mm PVC foam boards with printed on makers are mounted on all four sides and the top. On top of the structure, a 30 mm diameter aluminium rod is mounted about 0.5 m above ground and acts as a handrail to perform basic EVA movements.

The data processing of the camera streams runs on a dedicated workstation PC (i9-13900KS, 64 GB RAM). All cameras are connected to a 10 GBit/s Switch, to allow for the simultaneous transfer of four 2.3 MP video streams at up to 80 Hz. The data processing is done with the Robot Operating System (ROS) [10, 7], which provides the pose of the tracked headset via a ROS-to-Unity bridge. The Unity simulation listens to the data processing for updates on the current headset pose and moves the virtual cameras accordingly.

For the final version of the headset, i.e., rev3 as described above, an inertial sensor is used in addition to optical tracking, which is also integrated into the headset. For this purpose, the BNO085 IMU breakout board from Adafruit was selected, which is equipped with the BNO085 inertial measurement unit from CEVA Technologies, Inc. The sensor employs the same hardware as the Bosch Sensortech BNO055, but with an optimized firmware specifically designed for AR/VR applications. An Arm Cortex M0 processor is integrated in the sensor, which, when utilized in conjunction with the firmware from CEVA Hillcrest Laboratories, provides a sensor fusion of the accelerometer, gyroscope and magnetometer. The BNO085 is connected to the Adafruit QtPy RP2040 microcontroller via an I2C connection. The QtPy RP2040 is connected over a USB UART serial connection to the workstation, thereby providing the sensor data for subsequent processing. Consequently, the microcontroller is responsible for the initialization of the IMU, the access of the data and the transmission to the tracking system. As previously referenced, the DisplayPort cable and power connections, in conjunction with the USB cable for the IMU breakout board, are routed through a watertight tube that extends from the water to the computer and the external setup.

ROS 2 Iron is chosen as the software framework foundation of the project to process the various sensor and camera inputs and to determine the head pose. Different ROS nodes are utilized for the sensor data acquisition, the sensor fusion and the transmission of the pose to Unity. The Spinnaker Camera Driver for the FLIR BFS-PGE-23S3C-C cameras is embedded as a ROS node in the project. Each of the camera ROS nodes employs the AprilTag library to identify the various fiducial tags present in the raw images. The detections are evaluated and the positions of the cubic structure and the headset in relation to the camera are determined using the AprilTag library. The tracking node updates the message whenever a new position and rotation is computed and published as ROS topic.

The software for the headsets in rev1 and rev2 was designed in a syncronous fashion, i.e., it waited for all four cameras to transmit new image data and subsequently using the combined image data to calculate the position and orientation of the headset. We call this optical tracking system OTS old. With OTS new we denote the tracking, that makes the first processing step of the camera data in the image node independent of each other for which we achieve faster tracking times. It also includes the IMU built in rev3 of the headset.

The CEVA Hillcrest Laboratories firmware SH-2 optimized for AR/VR applications is running on the BNO085 inertial measurement unit. This firmware offers different data outputs that result from internal sensor fusion and calibration algorithms for AR/VR purposes.

The AR/VR Stabilized Game Rotation Vector delivers a self-correcting position estimation utilizing the onboard accelerometer and gyroscope measurements. Accumulated errors in the integration of the gyroscope measurements are corrected by estimations from roll/pitch angles delivered by the accelerometer data. Another data output utilized for this project is the Linear Acceleration. This output results from the accelerometer data which has been improved by calibration. Whenever new sensor events are available, those are received by the QtPy microcontroller and serialized for further transmission, corresponding to the data output of either rotation vectors or linear accelerations. Due to the onboard filtering of the rotation vector related data of the SH-2 firmware, the outputs received on the microcontroller result in stable values without noticeable jumps or wrong values. Therefore, the rotation vector data is forwarded directly to the PC without further processing on the microcontroller using the UART serial protocol. A low-pass filter implemented on the QtPy microcontroller serves the function of smoothing the overall data and eliminating the higher frequencies that appear as noise in the dataset.

Since the IMU data is transferred to the PC and the OTS system calculates the pose in a ROS node utilizing the AprilTag library for object tracking, the final determination of the player pose is now carried out based on the IMU and OTS data. After transforming the coordinate frames and calibrating of IMU and OTS rotation vector a Kalman filter in employed for sensor fusion, to achieve the objective of a robust head tracking with minimal delay and lag. To determine the pose of the headset, we use the IMU rotation vector output for orientation determination, and OTS and IMU Kalman filter sensor fusion for position determination. A lightweight solution for a one-way transfer of the pose from ROS to Unity is presented.

The overall workflow of the data is visualized in Fig. 3 and works similar to the Unity Robotics Hub which provides the ROS-TCP-Connector, but without serializing every available ROS topics over TCP. The TCP Server Socket is a ROS node that hosts a TCP socket for a client to connect to. Upon the publication of each new pose the TCP server node is responsible for receiving the pose and calibration information. Subsequently, the server node disseminates this information to the connected client via TCP. In this particular instance, the TCP client is initiated through a Unity script, which is attached to the Unity game object player. It extracts the pose information and adjusts the position and orientation of the player accordingly. The calibration information is used to determine whether the artificial horizon should be displayed.

The objective of the robustness evaluation tests is primarily to assess the performance and general functionality of the underwater VR headset. The evaluation process was primarily divided into two categories: (1) Delay, (2) Robustness.

In a campaign at the European Astronaut Center in Cologne the underwater VR headset has been tested within the pool of the Neutral Buoyancy Facility (NBF). This visit resulted in an occasion to have the VR headset assessed by a group of test persons, including the ESA astronauts Matthias Maurer and Marco Sieber, along with VR experts in the domain of astronaut training purposes and an astronaut training instructor (referred to as test subjects in the following test evaluation).

The cameras and cubic structure are mounted on a height-adjustable platform in the NBF, enabling a system testing in the dry prior to deployment. The system is then lowered exactly to the desired water depth of 1.5 meters, at which depth the tests are conducted, as seen in Fig. 1.

To ensure a consistent experience for all test subjects, a sequence of movements and tasks was defined in advance. This sequence is discussed with the participants during a pre-test briefing and also communicated via the headset during the dive. The initial task of the test dive is to grip the handrail as a reference point and to become familiar with the virtual environment. Subsequently, the calibration between the IMU and the OTS needs to be performed.

The subsequent step in this process is to take a closer look around and to become more familiar with the surroundings. One such example is the observation of the moon. The goal is to identify any differences between the tracking robustness before and after the calibration step. Given that the tracking is based purely on the optical system prior to calibration, comparisons are made with the performance improvements of the sensor fusion. Subsequently, movements such as translation in the $x$, $y$, $z$-direction and rotations in roll, pitch, and yaw are carried out separately in order to gain further impressions of the tracking. The next task requires moving beyond the left and right edges of the cubic platform to be able to observe the lateral surfaces. The ESA logo is randomly distributed on one of both sides for each participant. The task at hand is to locate the logo and report its position via the headset communication to the diving instructor. Subsequently, the subjects are instructed to orient their heads above the four colored markers positioned at the corners of the platform, while continuing to hold onto the handrail. The sequence of the markers is communicated via the headset by the diving instructor. Finally, each participant is given a period of time for unrestrained movement within reach of the cubic platform, thereby further experiencing the simulation of floating in the virtual environment in zero gravity. After carrying out the procedure, the participants are interviewed and asked to share their impressions and experiences during the test of the underwater VR headset.