A Dual Digital Twin Framework for Reinforcement Learning: Bridging Webots and MuJoCo with Generative AI and Alignment Strategies

Algirdas Laukaitis, Andrej Šareiko, Dalius Mažeika

Deep reinforcement learning (DRL) has shown potential for robotic training in virtual environments; however, challenges remain in bridging simulation and real-world deployment. This paper introduces an extended reinforcement learning framework that advances beyond traditional single-environment approaches by proposing a dual digital twin concept. Specifically, we suggest creating a digital twin of the robot in Webots and a corresponding twin in MuJoCo, enabling policy training in MuJoCo’s optimized physics engine and subsequent transfer back to Webots for validation. To ensure consistency across environments, we introduce a digital twin alignment methodology, synchronizing sensors, actuators, and physical model characteristics between the two simulators. Furthermore, we propose a novel testing framework that conducts controlled experiments in both virtual environments to quantify and manage divergence, thereby improving robustness and transferability. To address the cost and complexity of maintaining two high-fidelity models, we leverage generative AI agents to automate the creation of the secondary digital twin, significantly reducing engineering overhead. The proposed framework enhances scalability, accelerates training, and improves the reliability of sim-to-real transfer, paving the way for more efficient and adaptive robotic systems.