A Mixed Reality Generator for Real-World Environments in Real Time

Devrim Akça, Cagin Torkut, G. Kemper, Armin W. Grün

Abstract. By integrating computer vision, photogrammetry, UAV and Extended Reality (XR) solutions, the proposed innovative Mixed-Reality (MR) photogrammetry system enables real-time 3D visualization, interaction and measurement of real-world environments. Eliminating the need for physical presence, the system enhances safety, efficiency and accuracy in tasks like assessing structural integrity, tracking construction progress, and observing environmental changes over time. At the core of the system is a UAV equipped with a stereo camera rig and onboard processing capabilities. Operated on-site by an operator, the UAV captures high-resolution stereo imagery processed in real time through a centralized Rest API running on cloud infrastructure. Experts and engineers located anywhere in the world can connect to the system using VR headsets or a web-based application, gaining immersive access to a 3D stereoscopic view with full photogrammetric measurement functionality. The system supports multi-user collaboration, enabling synchronized analysis and data sharing across different locations. This seamless integration of hard- and software components represents a significant advancement in real-time stereoscopic visualization. Key features include a 2-camera stereo rig mounted on the UAV, mixed-reality visualization software for VR, and a web-based interface. Combining real-time data acquisition, processing, and visualization, the system provides unparalleled insights, surpassing the limitations of traditional on-site inspections. Designed with focus on cost-effectiveness and safety, this innovative approach is particularly valuable for inspecting large-scale, critical structures in challenging or hazardous environments. The proposed system offers a transformative tool for remote monitoring, ensuring timely and informed decision-making in diverse applications.