Architecture and Design Study with Analytical Validation of a Mixed Reality Software System for Pilot Training with Real-Time Visual Scene Substitution

Abstract

This paper presents the design and architectural formalization of a mixed reality (MR) system for pilot training that supports selective, region-level visual scene substitution. The proposed approach enables context-aware replacement of semantically meaningful regions within the pilot’s egocentric view - such as cockpit displays and external windows - while preserving direct physical interaction with real cockpit controls and the pilot’s hands.
The core contribution is a modular real-time architecture that integrates semantic perception, structured scene representation, and GPU-accelerated mask-based compositing into a unified dataflow pipeline. The system is designed with explicit latency-aware modelling to support analytical evaluation of performance under strict real-time constraints (targeting below 20-30 ms).
A functional prototype was implemented and evaluated under different hardware configurations. Experimental results demonstrate that the GPU-accelerated version running on an NVIDIA RTX 3060 achieves an average end-to-end latency of 26.3 ms (approximately 38 fps) with improved temporal stability. These results confirm the practical feasibility of the proposed architecture under controlled conditions.
Adopting a design-science perspective, this work combines analytical modelling, architectural design, and preliminary prototype validation. The study addresses the need for stronger technical grounding in XR-based aviation training research prior to large-scale human-subject evaluation. The presented approach provides a reproducible foundation for the development of hybrid training systems that combine the sensorimotor fidelity of physical cockpits with the instructional flexibility of real-time visual substitution.