Extended Reality Accessibility

Dr. Steffen Puhl, Coordinator for Accessibility and Inclusive Information Technologies at the IT-Service Center (HBZ), presented on the efforts to enhance accessibility in virtual and augmented reality (VR/AR) environments. Since 2020, accessibility has been a key focus for the VR/AR working group at Universität Gießen. In 2021, the Lab for Innovative Teaching launched the NIDIT project⁵⁵5https://www.uni-giessen.de/en/study/teaching/projects/nidit/network-for-impactful-
digital-teaching-skills-nidit?set_language=en, and by 2022, initial results on improving VR access were shared with colleagues from ACCESS@KIT⁶⁶6https://www.access.kit.edu/index.php.

The primary focus of these efforts has been the inclusion and participation of blind and visually impaired (BLV) students and teachers in VR learning scenarios. The approach involves identifying the needs of this group, developing relevant content, and programming and evaluating a demo application. The overarching goals are to make VR usable for the target group and to make existing systems accessible, thereby creating immersive experiences that are tangible for BLV users.

The team examined various available platforms, including Spatial, Engage, 3 Spin Learning, Mozilla Hubs, Unity, Unreal Engine, and Blender. However, they found that existing platforms like Spatial and Hubs were not flexible enough, necessitating the creation of new solutions. Basic requirements for the learning scenarios were identified, such as audio, navigation, walking, leaning interfaces, teleportation, and traveling.

Understanding the structure of a room and who is present when entering a VR space is crucial. In real life, BLV individuals can explore their surroundings using a cane, hands, or by asking a sighted guide. Equivalent methods need to be developed for VR. The next steps include securing funding for technical equipment and experts in spatial audio engineering, 3D modeling, programming, learning architecture, and audio description. Additionally, the research will expand to include other fields, such as transitions for people with higher residual vision and other disabilities.

In a presentation titled “Making Virtual Reality More Accessible for Persons with Balance Impairments,” Dr. John Quarles discussed the challenges and potential solutions for individuals with balance impairments, particularly those caused by conditions such as multiple sclerosis (MS). Quarles, who uses a wheelchair due to his inability to walk long distances, highlighted the significant balance issues he faces, especially in virtual reality (VR) environments. Following a severe MS attack, he found VR usage nearly impossible without falling over, a problem that extends to all users to some degree due to the inherent imbalance issues VR can cause.

To address these challenges, Quarles explored the effects of various auditory feedback techniques on improving standing balance in VR. He conducted a user study with 42 participants, half of whom had balance impairments primarily due to MS or diabetes. The study involved a standing reach and grasp task under different conditions: a baseline with no VR, and several VR conditions with different types of auditory feedback. These included spatial audio (spatialized white noise), static rest frame audio (white noise in HMD headphones), rhythmic audio (white noise beats at one-second intervals), and center of pressure (CoP) audio (pitch and stereo adjustments based on balance board data).

The results indicated that VR without feedback worsened balance compared to the baseline. However, the CoP and spatial audio techniques significantly improved balance, providing more information about balance than what is typically available in the real world. This improvement was observed in both participants with and without balance impairments, demonstrating a “curb cut effect” where accessibility features benefit all users. Follow-up studies replicated these findings with tactile and visual feedback, as well as walking tasks, showing similar results. Multimodal feedback (e.g., tactile and audio) yielded stronger results but also introduced potential information overload.

Quarles concluded that spatial and CoP auditory feedback were the most effective, followed by rhythmic and static rest frame audio, with no audio being the least effective. This research assists developers in creating more accessible and usable VR experiences for individuals with balance impairments. Future work will focus on busy, stimuli-rich virtual environments and adaptive feedback tailored to individual needs and virtual contexts.

The presentation also touched on the potential for longitudinal studies, where systems could be installed in users’ homes to provide continuous support. Questions were raised about the feasibility of using platforms like Kinect or software-based solutions to eliminate the need for balance boards. Additionally, the discussion included the importance of supporting disabled researchers and speakers through fellowships, scholarships, and volunteer opportunities, as well as the need for more alignment and mentorship across conferences to foster long-term engagement and support for accessibility in VR.

Dylan Fox presented on XR Access, a research initiative of Cornell Tech. Led by Executive Director Shiri Azenkot and Director of Operations Dylan Fox. Its goal is to make Extended Reality (XR) accessible to people with disabilities. After reviewing the ongoing research and advocacy work of XR Access, Fox led a discussion on how it could help the gathered researchers conduct and share their research.

Diane Nahabedian’s presentation on “Envisioning Access” highlighted her personal motivation driven by witnessing disability impacts within her household. She discussed the evolution of her 45-year-old organization, originally Helping Hands: Monkey Helpers, which transitioned from training monkeys for spinal cord injury support to exploring technological solutions due to legislative changes. The organization now focuses on testing and incubating technologies to aid disabled adults in returning to work or school, collaborating with companies like Cognimate on innovations such as a glove for stroke survivors, robotic canes, exoskeletons, and exploring AR, VR, and AI. She called for developers to collaborate, emphasizing their base in Boston, Massachusetts, and their nationwide client reach.

Dr. Tobias Langlotz’s presentation on the “Sensory Input Matrix” focused on addressing the needs of individuals with low or no vision by exploring the design space for vision augmentation. The matrix he discussed outlines possible remappings of sensory inputs, such as converting auditory signals to haptic feedback, which can be applied to both accessibility and assistive technologies. This approach is independent of specific applications and aims to compensate for missing sensory cues. Langlotz emphasized the importance of considering both input and output modalities, suggesting that VR could be used to simulate disabilities and evaluate accessibility. He also highlighted the need for collaboration among researchers to refine the taxonomy and explore multimodal aspects, while noting the potential benefits of publishing separate papers to cover different areas comprehensively.

Dr. Anthony Steed’s presentation on “Towards Accessible XR: Technical Opportunities for Developing and Appropriating Applications” highlighted the potential for enhancing accessibility in XR by integrating code at the XR runtime and signal composition levels. He discussed how tools like WalkinVR⁷⁷7https://www.walkinvrdriver.com/ and Ubiq⁸⁸8https://ubiq.online/ operate at the sensor hardware and driver levels, making applications unaware of their presence. Steed emphasized that functionalities like teleportation should be managed by the runtime rather than individual applications. He pointed out the need for collaboration with OpenXR to make scene compositors available, which would allow for more extensive modifications and accessibility features. Steed also addressed the challenges and opportunities with different platforms like SteamVR and Quest, noting the limitations and security concerns of each. He suggested that a more open composition layer could facilitate better accessibility without requiring game developers’ permission. Additionally, he mentioned tools like VR Hook and RenderDoc for low-level system access and data collection, including real-time cybersickness metrics.

Dr. Gregory F. Welch, Professor at the University of Central Florida, presented on the Virtual Experience Research Accelerator (VERA)⁹⁹9https://sreal.ucf.edu/vera/ project that is a human-machine system for carrying out human subjects research related to extended reality (XR), including virtual reality and augmented reality, for the XR researcher community. The project also seeks to foster a vibrant, diverse, and inclusive VERA research community, and to foster the professional growth of individual researchers. The mission of the VERA project is to serve the community of XR researchers by providing tools to accelerate and improve the quality of their human subjects research, building and sustaining a diverse and inclusive collaborative community of XR researchers, and fostering the next generation of XR researchers. The discussion focused on accessibility in the scope of the VERA project. The discussion will be continued at the upcoming ISMAR 2024 conference in the form of a tutorial organized by Dr. Welch.

In the working group session on “Social Interaction in XR,” the discussion was divided into two main themes: “The Box” and “Avatar Representation and Control.”

In the first working group session on “Accessible Tangible Games”, several key points were discussed to enhance the accessibility and enjoyment of games for all players. The primary focus was on ensuring that games are not only usable but also easy, intuitive, and thus fun to play. This is crucial because, unlike other applications, people do play games for enjoyment. Therefore, the challenge lies in creating games that bring people together, regardless of their varying degrees of accessibility needs.

One of the central questions raised was the role of immersive technologies (XR) in making physical board games more accessible. The group debated whether XR is necessary or if traditional board game elements could be adapted to address broader accessibility issues. For instance, integrating aspects of board games into video games could make the latter more accessible. This includes synchronizing virtual and physical board states to ensure remote accessibility and using transitional interfaces like ego- and bird’s-eye views. Additionally, smartphones could be employed for sonification and spatial audio, enhancing the gaming experience for players with visual impairments.

The discussion also touched on the nature of gameplay, considering both collaborative and competitive formats, concluding that collaborative formats avoid balancing issues of different interface types and thus might be the better choice to be considered further. The group highlighted the potential of Dungeons and Dragons (D&D) as a first use case, given its existing accessibility: Most parts of the game are imaginary and communities already address accessibility extensions such as Knights of the Braille¹¹¹¹11https://knightsofthebraille.com/.

Incorporating spatial audio, where players can hear their teammates talking on the board, was suggested as a way to create an inclusive, multimodal experience. This approach not only enhances gameplay but also offers opportunities for roleplay as part of therapy, benefiting disabled groups such as Knights of the Braille.

Finally, the group identified the need for further research and funding opportunities to support the development of accessible games. This includes surveying existing tools, identifying gaps in current systems, and exploring the potential of generative AI in creating customizable and inclusive gaming experiences. By addressing these challenges, the goal is to create games that are not only accessible but also enjoyable for all players, fostering a sense of community and inclusivity. Ultimately, the group identified possibilities for abstracting from board games to other scenarios re-using approaches for XR-based accessibility in case groups are collaborating on tasks, that incorporate a single physical space (such as a table).

In the second working group session, participants explored the integration of physical and digital elements to create inclusive gaming experiences. The discussion highlighted the potential of combining remote and in-person gatherings around a common scene, applicable not only to games but also to fields like architecture and urban planning.

In the working group session on “Low and High Hanging Fruits for Accessibility in XR,” participants identified various strategies and challenges to enhance accessibility in extended reality (XR) environments. The discussion was categorized into three main areas: low-hanging fruit, mid-hanging fruit, and high-hanging fruit or grand challenges.

The working group session on “Non-Visual Interaction in XR” explored various facets of enhancing accessibility and inclusivity in extended reality (XR) environments. One of the discussions centered around the capabilities of scene weaving and the use of VR sighted guides. These technologies enable users to understand and navigate through virtual environments, which is crucial for creating a more inclusive XR experience. The group emphasized the necessity of establishing standards for non-visual interaction in XR, highlighting the importance of metadata and hooks for non-visual systems to ensure that solutions are not proprietary and can be universally applied.

The session also addressed the inherent challenges of VR as a predominantly visual medium. To overcome these challenges, the group proposed the development of multisensory interaction and representation of information to make virtual worlds more accessible. This includes the use of haptic feedback, where vibrations on different parts of the body can communicate various messages, akin to gestures in communication. Additionally, the integration of virtual assistants and the adjustment of sensory information to match the capabilities of users were discussed as essential components for enhancing non-visual interactions.

A significant point of discussion was the need for a hierarchical system to prioritize relevant information. For instance, an AI assistant should be able to differentiate between important and trivial details, such as identifying people in the room who can be spoken to versus describing every individual present. The group also explored the potential of selectively sonifying objects to indicate their size, velocity, and type, as well as their affordances, such as whether an object can be opened or interacted with. The working group underscored the importance of designing with users, not just for them, to ensure that the solutions meet the actual needs of those with sensory impairments. They referenced existing guidelines, such as the Game Accessibility Guidelines and the W3C’s WCAG 3.0 draft, as frameworks to build upon. The group also discussed the potential for motion sickness with visual feedback and the potential to explore haptic and auditory interactions to mitigate this issue.

In conclusion, the session highlighted the necessity of creating inclusive XR environments by leveraging multisensory interactions and establishing universal standards. The group proposed a roadmap for understanding and implementing these standards, including consulting existing literature, running studies to investigate accessibility, and organizing workshops and calls for papers to foster innovation in this field. By focusing on both accessibility and inclusion, the group aims to create XR experiences that are immersive and accessible to all users, regardless of their sensory capabilities.

Further, the group prepared a workshop proposal for the IEEE VR 2025 conference, including a tentative Call for Submissions.

The working group on “Low Vision Interaction,” discussed various strategies to raise awareness and improve accessibility for individuals with low or no vision. The group emphasized the importance of advocacy and collaboration with organizations dedicated to supporting these groups. One proposed approach is to engage in lobbying efforts by sending representatives to advocacy groups and inviting these groups to research institutions and conferences. Identifying key organizations in the USA and finding equivalent associations in the EU, such as the EuroXR Association, is crucial for fostering international collaboration.

The group also considered the potential impact of publishing papers that outline the issues and call for funding and research. Targeting high-impact journals like Nature or Science, as well as engaging in science news and podcasts, could help raise public awareness. Two types of papers were suggested: opinion pieces and research articles. Involving multiple disciplines and stakeholders, including industry partners, is essential for securing research funding and ensuring the practical application of findings.

Collaboration across countries, was discussed as a way to combine different perspectives on low vision as well as different approaches to increase accessibility. Tobias Langlotz work for example is focused on changing the visual appearance of the world to transform visual cues into perceivable ones for those with low vision, while Torsten Kuhlen is exploring the use of virtual agents as personal assistants. Bobby Bodenheimer focused on building better low vision simulations in head-mounted displays (HMDs) with eye tracking to understand real-world behaviors and deficits, and Katharina Krösl uses her low vision simulations primarily to increase empathy and understanding for people with vision impairments. Sarah Creem-Regehr emphasized the importance of understanding how low vision changes the visual information available for everyday tasks, suggesting that augmented reality (AR) could highlight crucial cues.

The group also discussed the potential of XR as a tool to help make the real world more accessible by identifying perceptual challenges. This can facilitate universal design, which aims to create environments usable by all people, rather than just inclusive design. In this context the group discussed whether some current technology follows a universal design approach and is accessible enough for low vision groups and whether devices are adjustable for different contexts. Drawing an analogy to hearing aids, they considered whether visual devices could be similarly fitted and adjusted via smartphone apps.

A significant research question emerged: What can be done right now with smartphones and AR to improve accessibility, and what problems need to be solved for always-on displays? Suggestions included using filters on displays to improve contrast, outline edges, or magnify content. The group noted that while voiceover functions on phones work well for blind users, holding both a cane and a phone can be challenging. Future devices could incorporate different controls and input methods to enhance usability.

Social acceptability and form factors were also discussed, with examples like Samsung glasses that resemble sunglasses but contain displays. The group concluded that developing devices with integrated sensors that are easy and quick to use is essential for mass adoption. Overall, the session underscored the need for interdisciplinary collaboration, innovative research, and practical solutions to improve accessibility for individuals with low vision.

In the working group session on “XR Taxonomy for Accessibility,” participants explored various definitions and frameworks to enhance accessibility in extended reality (XR) environments. The session began by examining definitions from international standardization organizations. According to the ISO 9241-171:2008 standard, accessibility refers to the usability of a product, service, environment, or facility by people with the widest range of capabilities. This definition emphasizes that accessibility is not limited to individuals formally recognized as having a disability but addresses the full spectrum of user capabilities. The usability-oriented concept of accessibility aims to achieve high levels of effectiveness, efficiency, and satisfaction, considering the specified context of use.

The ITU’s F.791 standard and the W3C’s definition of web accessibility further elaborate on this concept. Web accessibility ensures that websites, tools, and technologies are designed so that people with disabilities can perceive, understand, navigate, interact with, and contribute to the web. These definitions underscore the importance of inclusivity and the need for accessible design in digital environments.

The working group identified several key outcomes or axes of accessibility, including actions such as access, navigation, interaction, and communication. These actions form the basis of a taxonomy of accessibility, which organizes the different aspects, types, or components of accessibility across various fields. This taxonomy helps structure the understanding of how accessibility can be applied to technology, services, environments, and products, ensuring inclusivity for people with disabilities or other barriers to access.

The taxonomy of accessibility can be broadly categorized into several areas:

Types of Disabilities:

This includes visual, auditory, motor/mobility, and cognitive/neurological disabilities. Each type requires specific accessibility solutions, such as screen readers for visual impairments, closed captioning for auditory impairments, voice-controlled interfaces for motor disabilities, and simplified content for cognitive disabilities.

Digital Accessibility:

This encompasses web accessibility, software accessibility, and mobile accessibility. Ensuring compliance with standards like the Web Content Accessibility Guidelines (WCAG), providing customizable user interfaces, and designing mobile applications with larger touch targets and simplified gestures are crucial for digital accessibility.

Physical Accessibility:

This involves removing barriers in physical spaces, such as providing ramps, elevators, accessible signage, and wheelchair-accessible vehicles. Public facilities must also be designed to be accessible, with features like accessible restrooms and emergency exits.

Technological Accessibility:

This includes the development of assistive technologies, such as hearing aids, prosthetic devices, screen readers, and eye-tracking systems. Ensuring that virtual and augmented reality technologies are accessible through inclusive design and haptic feedback is also essential.

Legal and Regulatory Accessibility:

Compliance with accessibility laws and guidelines, such as the Americans with Disabilities Act (ADA) and Section 508, is critical. Ethical considerations must also be taken into account to ensure equality and fairness in accessibility.

Social and Economic Accessibility:

Ensuring that products and services are affordable and widely available is important for social and economic accessibility. Designing solutions that are sensitive to cultural differences and language barriers is also necessary.

A well-structured taxonomy of accessibility provides a comprehensive framework for understanding, developing, and implementing inclusive designs and services across various domains. This approach ensures that accessibility is integrated into all aspects of XR environments, making them more usable and enjoyable for everyone.

In the working group session on “AI as Enabler for XR Accessibility,” participants explored the potential of artificial intelligence (AI) to enhance accessibility in extended reality (XR) environments. The discussion focused on several key areas where AI can play a transformative role.

We have formed a group that will persist in its advocacy for bringing awareness to issues surrounding accessibility with XR. The committee is made up of attendees who elected to be part of this organization initially for a workshop proposal review committee (see more on this below). However, the steering committee also has plans to advocate for making conferences more accessible to those with impairments or disabilities. There are plans to write letters to the organizing bodies of many of our interdisciplinary conferences (e.g., Association for Computing Machinery or Institute of Electrical and Electronics Engineers) in order to ask for more funding to support attendance at conferences and to generally bring awareness to issues surrounding accessibility with XR. We also will try to expand our steering committee to include people not in attendance at this seminar, but who may be interested in the topic of XR accessibility, such as members of the XR Access community who did not attend the seminar. One of our attendees (D. Fox) is the lead of this organization, so expansion of our group to include individuals from like-minded organizations is already underway. Further, one of our attendees (A. Olivier) is a General Chair for the IEEE VR 2025 conference. She is advocating for changes to accommodate accessibility for the conference. Finally, some members of the workshop who are located within the EU or the UK (led by A. Steed) plan to propose an Marie Curie innovative training network (ITN) on XR accessibility. We see this steering committee as an enduring outcome of the seminar that we hope will have wide-ranging effects on the community. We set up a Discord server for the seminar with various channels that we are now using to organize ourselves for further planning, along with the Dagstuhl Seminar email listserv.

In order to introduce more researchers to ideas around XR accessibility, we have planned and proposed workshops at some of the major conferences for XR researchers. Specifically, a small group of us (led by B. Bodenheimer) proposed a workshop on XR Accessibility for the IEEE VR 2025 conference to be held in St. Malo, France from March 8th to 12th, 2025¹²¹²12https://sites.google.com/view/xraccess/home. This workshop was accepted and it will feature stakeholder talks, poster presentations, and discussions around the topic of XR accessibility. The aforementioned steering committee will be reviewers for the submissions and seminar attendees are lined up to help with on-site organization of the workshop itself. Another workshop on XR accessibility is planned (by D. Zielasko) for submission to the CHI Play conference. Finally, one of our attendees (S. Stefania) is involved in organizing a conference next year (EAI ArtsIT) and has suggested featuring XR accessibility as special session at the meeting.

The seminar resulted in many plans for future publications. Specifically, the working group on taxonomy for accessibility in XR realized that a larger paper with definitions and terms would likely be helpful for organizing the field. There was also recognition of a need for more papers and research discussing standardization of techniques for designing for accessibility. In order to aid in writing these types of overview papers, we set up a Zotero library of papers that was shared with all attendees of the seminar and can be made public to share with others who are interested in research in this field. A specific paper is planned that will discuss how to layout the design space for different types of sensory loss that is based on a paper already published on deficiencies in vision by attendee T. Langlotz. We believe that other papers will likely emerge as our group expands and attends workshops or conferences to further advance our ideas.

Despite the many outcomes that we are or have pursued thus far, there are challenges that we see in growing our advocacy and impact going forward. Ideally, we would like to secure some funding to continue to host meetings and seminars for those interested in advancing XR accessibility. Finding funding agencies that are interdisciplinary and also will work internationally to support groups is difficult. We have plans to talk to program officers in the United States in order to continue to pursue funding for meetings there. Some attendees are also pursuing an accessibility ITN as mentioned above. We also know that getting funding from organizing bodies in the XR field will not be easy. We will need to advocate heavily for more attention to be paid to accessibility as an issue for conference attendance, but also as a major topic worthy of research funding. Overall, we hope to use the steering committee and our growing community to continue to pursue these goals.