The Human And Robot Multimodal Observations of Natural Interactive Collaboration (HARMONIC) dataset is a large multi-modal dataset of human interactions with a robotic arm in a shared autonomy setting designed to imitate assistive eating.
The dataset provides human, robot, and environmental data of 24 different people engaged in an assistive eating task with a 6-degree-of-freedom (6-DOF) robot arm. Human data include egocentric RGB videos, eye gaze positions relative to these videos, infrared (IR) videos of both eyes, stereo, third-person video of the participant, and EMG recordings on the joystick-controlling arm. Robot data include joystick control inputs from the user, the control input and belief distribution calculated by the assistance algorithm, and the robot joint positions. Environmental data include the 3D locations of the food morsels as well as the locations of fiducial markers. Several features were also extracted directly from these recordings, namely eye gaze fixations in the egocentric camera frame, and body pose, hand pose, and facial keypoints.
The dataset is intended for research in human–robot collaboration and multimodal human behavior analysis, particularly in studying intention prediction, human mental state modeling, shared autonomy, and the human–robot dynamics of assistive teleoperation.
Verified
@article{NewmanHARMONIC2021,
author = {Benjamin A. Newman and Reuben M. Aronson and Siddhartha S. Srinivasa and Kris Kitani and Henny Admoni},
title = {HARMONIC: A multimodal dataset of assistive human–robot collaboration},
journal = {The International Journal of Robotics Research},
month = {Dec},
year = {2021},
doi = {10.1177/02783649211050677},
url = {https://doi.org/10.1177/02783649211050677}
}
