Egocentric hands, in 3D

Robots need actions in 3D to learn, but egocentric videos only contain what a person sees. MacroHands, our hand-tracking model, takes these videos and reconstructs 21 joints for each of the wearer's hands, along with where the head-mounted camera is and where it points in every frame. Below, you can see how these predictions look over the video, as well as in 3D space.