From Grasping to Speaking: Generative AI-Based Environment-Grounded VR Communication Training for Autistic Individuals

Ziming Li, Roshan L. Peiris

Autistic individuals often face barriers in workplace communication, where soft skills are embedded within ongoing tasks and surrounding environment context, not in isolated verbal exchange. Recent work has introduced LLM-driven agents into VR-based communication training and proposed prompting schemas that let agents generate dialogue grounded in the VR environment and the user's hand-based interactions. Building on this work, we explore how different levels of environmental grounding influence the training experience of autistic trainees and job coaches. We conducted an exploratory study with 9 autistic trainees and 7 job coaches across three modalities: conversation-only (C), conversation with environmental objects (C+O), and conversation with objects and grasp interactions (C+O+G). Usability and workload were comparable across modalities, while both trainees and coaches preferred the more interactive and environment-grounded condition (C+O+G). Participants viewed C+O+G as helpful for sustaining engagement and integrating communication practice into task performance. We discuss design considerations for more flexible, interactive, and workplace-relevant VR communication training for autistic individuals.