Human Tracking Integration for Adaptive Robot Behavior in VR/MR Environments
Context
We work at the intersection of human-robot interaction, Mixed Reality interfaces, and collaborative cyber-physical systems. Modern robotics needs seamless collaboration between humans and robot swarms in shared workspaces. This project operates within the SwarmOps research framework, focusing on human-sensing based MLOps for collaborative cyber-physical systems. Current developments in VR/MR technology provide new opportunities for rich human tracking that can inform adaptive robot behavior.
Motivation
Current robot systems lack sophisticated human awareness in shared workspaces, leading to inefficient and potentially unsafe human-robot collaboration. Existing solutions either ignore human presence entirely or rely on basic proximity sensors that provide minimal context about human intentions, activities, or spatial relationships. Meanwhile, VR/MR environments capture rich human behavioral data through advanced tracking systems, but this information remains disconnected from robot behavior systems. This creates a gap where valuable human context data could dramatically improve robot adaptability and collaboration effectiveness, but no systematic integration exists between VR/MR human tracking and robot behavior adaptation.
Goal
The student will develop a human tracking system that integrates VR/MR interfaces with robot behavior adaptation. The system will include a human tracking pipeline using VR/MR headset sensors that extracts pose, gaze, gesture, and spatial positioning data. Privacy-preserving feature extraction will ensure no raw video or personally identifiable information is transmitted. A real-time communication bridge will connect VR/MR tracking systems with ROS2 robot networks. Adaptive robot behavior algorithms will respond to human presence, intentions, and activities. Demonstration scenarios will show robots modifying their behavior based on human tracking data. Documentation and testing framework will be suitable for extension into thesis research.
Requirements
- The student should have basic programming experience in C# or Python, with willingness to learn Unity 3D development.
- Familiarity with computer vision concepts is helpful but not required as training will be provided.
- Understanding of basic robotics principles and ROS2 concepts would be beneficial but can be learned during the project.
- The student should be comfortable working with VR/MR hardware and have interest in human-computer interaction.
- Strong problem-solving skills and ability to work independently while maintaining regular communication with supervisors are essential.
- Access to VR/MR headset will be provided for development and testing.
Pointers
- Meta Quest SDK and OpenXR documentation provide cross-platform VR development resources.
- MediaPipe and OpenCV libraries support human pose estimation and gesture recognition.
- ROS2 tutorials and geometry_msgs documentation help with robot integration.
- Privacy-preserving machine learning techniques and differential privacy resources address data protection needs.
- Unity3D Mixed Reality Toolkit (MRTK) supports MR development.
- Research papers on human-robot interaction and collaborative robotics provide theoretical background.
- SwarmOps project documentation and related publications offer project context.