Augmented and Mixed Reality Scene Alignment Through 3D-to-3D Learning-Based Cross-Source Point Cloud Registration

Juan Sardi-Barzallo, Norbert Haala, Volker Coors

Abstract. Rapid advancements in reality capture technology and increasing accessibility to devices capable of generating point cloud data have led to a greater prevalence of applications requiring the interaction and integration of cross-source Point Cloud Data (PCD). Augmented and Mixed Reality (AR/MR) technologies are similarly pivotal for integrating digital and physical environments by overlaying Digital Twin (DT) models onto real-world contexts. While AR/MR devices can generate real-time 3D point clouds, integrating this data with high-precision sources, such as LiDAR, remains a research challenge—particularly regarding scene alignment and camera localization. Accurate positioning of AR/MR users within large-scale pre-scanned environments enables the deployment of digital content at predefined locations without prior field preparation. However, conventional vision-based methods are often vulnerable to environmental variations, complicating reliable localization. This work proposes an exclusively 3D-to-3D-based methodology for AR/MR scene alignment and camera localization, addressing cross-source registration in scenarios with significant size disparity. By combining a learning-based Voxel Representation and Hierarchical Correspondence Filtering (VRHCF) method with the Truncated Least Squares Estimation And SEmidefinite Relaxation (TEASER++) algorithm, our approach effectively manages asymmetric heterogeneous point cloud data. The results demonstrate promising gross registration performance, particularly in extensive indoor settings. Although computational constraints and outlier sensitivity in outdoor environments warrant further investigation, this study underscores the potential of advanced 3D-to-3D methodologies to enable seamless interaction between digital and physical worlds.