Document Open Access Logo

Gaze Beyond Limits: Integrating Eye-Tracking and Augmented Reality for Next-Generation Spacesuit Interaction

Authors Jiayu He, Yifan Li, Oliver R. Runswick , Peter D. Hodkinson , Jarle Steinberg, Felix Gorbatsevich , Yang Gao

PDF HTML 5

Files

Thumbnail PDF
PDF
OASIcs.SpaceCHI.2025.29.pdf
  • Filesize: 7.94 MB
  • 15 pages
HTML5 Logo HTML (experimental)
OASIcs.SpaceCHI.2025.29.html

Document Identifiers

Subject Classification

ACM Subject Classification
  • Human-centered computing → Human computer interaction (HCI)
Keywords
  • Augmented Reality (AR)
  • Eye-Tracking
  • Cognitive Load/Workload
  • Segment Anything Model (SAM)
  • Visual Language Models (VLMs)

Metrics

Abstract

Extravehicular activities (EVAs) are increasingly frequent in human spaceflight, particularly in spacecraft maintenance, scientific research, and planetary exploration. Spacesuits are essential for sustaining astronauts in the harsh environment of space, making their design a key factor in the success of EVA missions. The development of spacesuit technology has traditionally been driven by highly engineered solutions focused on life support, mission adaptability and operational efficiency. Modern spacesuits prioritize maintaining optimal internal temperature, humidity and pressure, as well as withstanding extreme temperature fluctuations and providing robust protection against micrometeoroid impacts and space debris. However, their bulkiness and rigidity impose significant physical strain on astronauts, reducing mobility and dexterity, particularly in tasks requiring fine motor control. The restricted field of view further complicates situational awareness, increasing the cognitive load during high-precision operations. While traditional spacesuits support basic EVA tasks, future space exploration shifting toward long-duration lunar and Martian surface missions demand more adaptive, intelligent, and astronaut-centric designs to overcome current constraints. To explore a next-generation spacesuit, this paper proposed an in-process eye-tracking embedded Augmented Reality (AR) Spacesuit System to enhance astronaut-environment interactions. By leveraging Segment-Anything Models (SAM) and Vision-Language Models (VLMs), we demonstrate a four-step approach to enable top-down gaze detection to minimize erroneous fixation data, gaze-based segmentation of objects of interest, real-time contextual assistance via AR overlays and hands-free operation within the spacesuit. This approach enhances real-time situational awareness and improves EVA task efficiency. We conclude with an exploration of the AR Helmet System’s potential in revolutionizing human-space interaction paradigms for future long-duration deep-space missions and discuss the further optimization of eye-tracking interactions using VLMs to predict astronaut intent and highlight relevant objects preemptively.

Cite As Get BibTex

Jiayu He, Yifan Li, Oliver R. Runswick, Peter D. Hodkinson, Jarle Steinberg, Felix Gorbatsevich, and Yang Gao. Gaze Beyond Limits: Integrating Eye-Tracking and Augmented Reality for Next-Generation Spacesuit Interaction. In Advancing Human-Computer Interaction for Space Exploration (SpaceCHI 2025). Open Access Series in Informatics (OASIcs), Volume 130, pp. 29:1-29:15, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/OASIcs.SpaceCHI.2025.29

Author Details

Jiayu He
  • STAR LAB, Centre for Robotics Research, King’s College London, UK
Yifan Li
  • STAR LAB, Centre for Robotics Research, King’s College London, UK
Oliver R. Runswick
  • Department of Pychology, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, UK
Peter D. Hodkinson
  • Spacesuit Physiology Laboratory, Centre for Human and Applied Physiological Sciences, King’s College London, UK
Jarle Steinberg
  • Nåva Space, Oslo, Norway
Felix Gorbatsevich
  • PaleBlue, Stavanger, Norway
Yang Gao
  • STAR LAB, Centre for Robotics Research, King’s College London, UK
  • Centre for AI Robotics for Space Sustainablity (CAIRSS), Hong Kong University of Science & Technology, Hong Kong

References

  1. Brian K. Alpert and Brian J. Johnson. Extravehicular activity (eva) framework for exploration - 2019. In 49th International Conference on Environmental Systems, Boston, Massachusetts, July 2019. NASA. NASA/TP-2019-220304, NASA Technical Report Server. URL: https://ntrs.nasa.gov/citations/20190028714.
  2. Liz Altmiller, Taylor Campbell, Tyler Chapman, Dean Cohen, John Garrison, Graham Hill, Daniel Lambert, Brenna Leonard, Katelyn Schuettke, Marie Shirley, Olivia Thomas, and A. J. Trantham. Arsis 2.0: Augmented reality space informatics system. Boise State University Undergraduate Research and Scholarship Conference, 2019. URL: https://scholarworks.boisestate.edu/under_conf_2019/95.
  3. Harald Köpping Athanasopoulos. The moon village and space 4.0: The ‘open concept’as a new way of doing space? Space Policy, 49:101323, 2019. Google Scholar
  4. Mihai Bâce, Sander Staal, and Andreas Bulling. Accurate and robust eye contact detection during everyday mobile device interactions. arXiv preprint arXiv:1907.11115, 2019. URL: https://arxiv.org/abs/1907.11115.
  5. Blaze Belobrajdic, Kate Melone, and Ana Diaz-Artiles. Planetary extravehicular activity (eva) risk mitigation strategies for long-duration space missions. npj Microgravity, 7(1):16, 2021. Google Scholar
  6. Shanqing Cai, Subhashini Venugopalan, Katie Seaver, Xiang Xiao, Katrin Tomanek, Sri Jalasutram, Meredith Ringel Morris, Shaun Kane, Ajit Narayanan, Robert L MacDonald, et al. Using large language models to accelerate communication for eye gaze typing users with als. Nature Communications, 15(1):9449, 2024. Google Scholar
  7. Hubert Cecotti. A multimodal gaze-controlled virtual keyboard. IEEE Transactions on Human-Machine Systems, 46(4):601-606, 2016. URL: https://doi.org/10.1109/THMS.2016.2537749.
  8. Eva Yi-Wei Chang. Fashion styling and design aesthetics in spacesuit: An evolution review in 60 years from 1960 to 2020. Acta Astronautica, 178:117-128, 2021. Google Scholar
  9. Ching-Ju Chao, Hao-Chiang Koong Lin, Cheng-Hung Wang, and Min-Chai Hsieh. Eye tracking for evaluating an ar-based learning system on monocotyledons/dicotyledons. In IEEE International Conference on Consumer Electronics, 2011. URL: https://api.semanticscholar.org/CorpusID:219781409.
  10. Dewen Cheng, Qiwei Wang, Yue Liu, Hailong Chen, Dongwei Ni, Ximeng Wang, Cheng Yao, Qichao Hou, Weihong Hou, Gang Luo, et al. Design and manufacture ar head-mounted displays: A review and outlook. Light: Advanced Manufacturing, 2(3):350-369, 2021. Google Scholar
  11. Kristine N. Davis and Tymon E. Kukla. Nasa advanced space suit xemu development report-helmet and extravehicular visor assembly (evva). In 51st International Conference on Environmental Systems, St. Paul, Minnesota, July 2022. International Conference on Environmental Systems (ICES). Google Scholar
  12. ESA/NASA. Eye tracking in space, 2014. Image courtesy of ESA/NASA. © ESA Standard Licence. Accessed: 2025-03-27. URL: https://www.esa.int/ESA_Multimedia/Images/2014/11/Eye_tracking_in_space.
  13. European Space Agency (ESA). Eye-catching space technology restoring sight, 2024. Accessed: 20 March 2025. URL: https://www.esa.int/Science_Exploration/Human_and_Robotic_Exploration/Research/Eye-catching_space_technology_restoring_sight.
  14. Monika Gisler and Didier Sornette. Exuberant innovations: the apollo program, 2009. Google Scholar
  15. Sandra G Hart. Nasa-task load index (nasa-tlx); 20 years later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50(9):904-908, 2006. URL: https://doi.org/10.1177/154193120605000909.
  16. Yiteng (Arden) Huang, Jingdong Chen, and Shaoyan (Sharyl) Chen. Integrated spacesuit audio system enhances speech quality and reduces noise. Technical Report LEW-18405-1, NASA Glenn Research Center, November 2009. Google Scholar
  17. Richard S Johnston, Lawrence F Dietlein, and Charles Alden Berry. Biomedical results of Apollo, volume 368. Scientific and Technical Information Office, National Aeronautics and Space …, 1975. Google Scholar
  18. Jacob Keller, Lanssie Ma, Matthew Miller, Skye Ray, Daren Welsh, Lauren Brady, Forrest Porter, Joseph Vacca, Paromita Mitra, and Matthew Noyes. Using virtual reality to envision deployment of spacesuit-compatible augmented reality displays for lunar surface operations. In 52nd International Conference on Environmental Systems (ICES), 2023. Google Scholar
  19. Thomas Kosch, Mariam Hassib, Daniel Buschek, and Albrecht Schmidt. Look into my eyes: using pupil dilation to estimate mental workload for task complexity adaptation. In Extended abstracts of the 2018 chi conference on human factors in computing systems, pages 1-6, 2018. Google Scholar
  20. Douglas N Lantry. Man in machine: Apollo-era space suits as artifacts of technology and culture. Winterthur Portfolio, 30(4):203-230, 1995. Google Scholar
  21. Hyungmin Lee, Chen-Chun Hsia, Aleksandr Tsoy, Sungmin Choi, Hanchao Hou, and Shiguang Ni. Visionary: Exploratory research on contextual language learning using ar glasses with chatgpt. In Proceedings of the 15th biannual conference of the Italian SIGCHI chapter, pages 1-6, 2023. URL: https://doi.org/10.1145/3605390.3605400.
  22. Hua Li, Xin Zhang, Guangwei Shi, Hemeng Qu, Yanxiong Wu, and Jianping Zhang. Review and analysis of avionic helmet-mounted displays. Optical Engineering, 52(11):110901-110901, 2013. Google Scholar
  23. Seamus Seamus Joseph Holt Lombardo. Evaluating the effect of spacesuit glove fit on functional task performance. PhD thesis, Massachusetts Institute of Technology, 2020. Google Scholar
  24. Guo-Rui Ma, Jia-Xin He, Chun-Hsien Chen, Ya-Feng Niu, Lan Zhang, and Tian-Yu Zhou. Trigger motion and interface optimization of an eye-controlled human-computer interaction system based on voluntary eye blinks. Human-Computer Interaction, 39(5-6):472-502, 2024. URL: https://doi.org/10.1080/07370024.2023.2195850.
  25. Jose A Marmolejo. Helmet-mounted display and associated research activities recently conducted by the nasa johnson space center. In Helmet-and Head-Mounted Displays and Symbology Design Requirements, volume 2218, pages 281-291. SPIE, 1994. Google Scholar
  26. NASA. Apollo 16 cuff checklists. https://www.nasa.gov/history/alsj/a16/cuff16a.html. Accessed: 2025-03-20.
  27. DJ Newman, PB Schmidt, and DB Rahn. Modeling the extravehicular mobility unit (emu) space suit: physiological implications for extravehicular activity (eva). Technical report, SAE Technical Paper, 2000. Google Scholar
  28. Florian Saling, Andrea Emanuele Maria Casini, Andreas Treuer, Martial Costantini, Leonie Bensch, Tommy Nilsson, and Lionel Ferra. Testing and validation of innovative extended reality technologies for astronaut training in a partial-gravity parabolic flight campaign. arXiv preprint arXiv:2410.14922, 2024. URL: https://doi.org/10.48550/arXiv.2410.14922.
  29. Daryl J Schuck. Development of a spacesuit helmet mounted display testbed system. In 43rd International Conference on Environmental Systems, page 3458, 2013. Google Scholar
  30. Ryan T Scott, Erik L Antonsen, Lauren M Sanders, Jaden JA Hastings, Seung-min Park, Graham Mackintosh, Robert J Reynolds, Adrienne L Hoarfrost, Aenor Sawyer, Casey S Greene, et al. Beyond low earth orbit: biomonitoring, artificial intelligence, and precision space health. arXiv preprint arXiv:2112.12554, 2021. Google Scholar
  31. Statista. Duration of moonwalks during nasa missions from 1969 to 1972, 2024. Accessed: 20 March 2025. URL: https://www.statista.com/statistics/1028544/length-of-moonwalks/.
  32. Moritz Stolte, Benedikt Gollan, and Ulrich Ansorge. Tracking visual search demands and memory load through pupil dilation. Journal of Vision, 20(6):21-21, 2020. Google Scholar
  33. Boris Velichkovsky, Andreas Sprenger, and Pieter Unema. Towards gaze-mediated interaction: Collecting solutions of the “midas touch problem”. In Human-Computer Interaction INTERACT’97: IFIP TC13 International Conference on Human-Computer Interaction, 14th-18th July 1997, Sydney, Australia, pages 509-516. Springer, 1997. Google Scholar
  34. Bin Wang, Armstrong Aboah, Zheyuan Zhang, and Ulas Bagci. Gazesam: What you see is what you segment. arXiv preprint arXiv:2304.13844, 2023. URL: https://doi.org/10.48550/arXiv.2304.13844.
  35. Weiyao Wang, Matt Feiszli, Heng Wang, and Du Tran. Unidentified video objects: A benchmark for dense, open-world segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10776-10785, 2021. Google Scholar
  36. Bo Yang and Jian Huang. Outlier-robust gaze signal filtering framework based on eye-movement modality recognition and set-membership approach. IEEE Transactions on Biomedical Engineering, 70(8):2463-2474, 2023. URL: https://doi.org/10.1109/TBME.2023.3249233.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact