Clione

HARMONIC is a large multi-modal dataset of human interactions in a shared autonomy setting. The dataset provides human, robot, and environment data streams from twenty-four people engaged in an assistive eating task with a 6 degree-of-freedom (DOF) robot arm. From each participant, the authors recorded video of both eyes, egocentric video from a head-mounted camera, joystick commands, electromyography from the participant's forearm used to operate the joystick, third person stereo video, and the joint positions of the 6 DOF robot arm. Also included are several data streams that come as a direct result of these recordings, namely eye gaze fixations in the egocentric camera frame and body position skeletons. This dataset could be of interest to researchers studying intention prediction, human mental state modeling, and shared autonomy. Data streams are provided in a variety of formats such as video and human-readable csv or yaml files.

Due to computational load, certain data streams may have periodic dropouts. The stats directory contains some info on when and how often these occur. The missing data are particularly exacerbated for the Myo signal due to the data recording software failing to start. Finally, due to permissions restrictions, unedited ZED video capture is available for 10 participants, de-identified video (video with faces blurred) is available for 13 participants, and video for 1 participant is unavailable for release. Within the released participants, some initialization failure means that videos of certain trials are occasionally missing.