ARMADA: Augmented Reality for Robot Manipulation and Robot-Free Data Acquisition

Nataliya Nechyporenko
Ryan Hoque
Christopher Webb
Mouli Sivapurapu
Jian Zhang
Full-text
Check out our recent posts!

Abstract

Teleoperation for robot imitation learning is bottlenecked by hardware availability. Can high-quality robot data be collected without a physical robot? We present a system for augmenting Apple Vision Pro with real-time virtual robot feedback. By providing users with an intuitive understanding of how their actions translate to robot motions, we enable the collection of natural barehanded human data that is compatible with the limitations of physical robot hardware. We conducted a user study with 15 participants demonstrating 3 different tasks each under 3 different feedback conditions and directly replayed the collected trajectories on physical robot hardware. Results suggest live robot feedback dramatically improves the quality of the collected data, suggesting a new avenue for scalable human data collection without access to robot hardware.

System Architecture

Human skeletal data is sent over websockets to an external compute device, which runs a live robot simulation. The robot then executes a command and its proprioceptive data is sent back to Vision Pro for AR visualization. The full loop runs real-time at 30 Hz.

User Interface

To enable new users to interact with the system, we develop ARMADA, a user-friendly software application that can be used by anyone with access to an Apple Vision Pro to collect data.

User Study

In a user study with 15 participants, including participants with no prior experience using a virtual reality device, we collect a total of 675 robot-free demonstrations on 3 tasks with our system. Results suggest that demonstrations collected with real-time AR feedback can be directly replayed on physical robot hardware, dramatically increasing the average replay success rate from 1.3% to 71.1% when compared to demonstrations collected without such feedback.

Outlook

By enabling in-the-wild data collection from anyone with a Vision Pro, ARMADA may facilitate the creation of large datasets with tens of thousands of hours of manipulation data. Such datasets may enable imitation learning at unprecedented scale, an essential ingredient for generalization across tasks, environments, and robot hardware

BibTeX Citation

@misc{nechyporenko2024armada, title={ARMADA: Augmented Reality for Robot Manipulation and Robot-Free Data Acquisition}, author={Nataliya Nechyporenko and Ryan Hoque and Christopher Webb and Mouli Sivapurapu, and Jian Zhang}, year={2024}, eprint={2412.10631}, archivePrefix={arXiv}, primaryClass={cs.RO}, url={https://arxiv.org/abs/2412.10631}, }

Contact

If you have any questions, please feel free to contact Nataliya Nechyporenko or Ryan Hoque.