MacroHands: 3D hands and camera pose from egocentric video

• We introduce MacroHands: our state-of-the-art model family for 3D hands and camera path from egocentric video, with monocular (mono) and stereo variants
• MacroHands-Mono beats every open-source baseline we tested, including all stereo baselines (32.2 mm on SHOW3D [1] vs 55.9 for HaWoR [2], and 46.4 for the best stereo baseline)
• MacroHands-Stereo brings further improvements over Mono, with 2.5× lower error (13.1 vs 32.2 mm on SHOW3D, mostly due to better depth estimates)
• MacroHands handles partial and even brief full occlusions (on clips where the hands are often partly hidden it still finds 80% of hands vs at most 59% for baselines)
• MacroHands distinguishes wearer from other hands (0.2% of detections land on someone else's hands, vs ~24% for HaWoR), works on footage with gloves and in other common real-world scenarios where most models fall short
• Our camera pose is also state of the art, even more so when IMU is available (~4× less drift than DROID-SLAM [3] on Monado [4] and Aria Gen 2 [5])
• Send us a small sample of your footage (mono or stereo) and we'll run MacroHands on it for free, so your team can compare it with your current hand tracking.

Bigger and more capable robotics models keep getting released, trained on ever larger amounts of data. In our latest blogpost, we went deep into egocentric data, a cheaper and more scalable way than robot teleoperation to get training data, using simple head-strapped cameras.

Unfortunately, most of the 3D hand tracking models and pipelines available today still suffer from the same issues we had found: partial occlusions make footage unusable, hands become unreliable when operators wear gloves or work in crowded settings, or even when bare feet are visible (which many models mistake for hands). Overall, the estimates, especially depth (the distance from the camera), are far from good enough, despite the "sub-mm precision" claims from some vendors.¹

We spent the last few months tackling this problem, and today we are announcing MacroHands, our family of 3D hand tracking and camera pose models for egocentric video. MacroHands comes in two variants: Mono, which works on footage from any single camera, and Stereo, which uses both views of a stereo rig.

MacroHands‑Mono beats every open-source baseline we tested, including the stereo ones on SHOW3D, and Stereo cuts the error by another 2.5×. Both keep tracking through occlusions and stick to the wearer's hands. They also estimate the camera pose, meaning where the head-mounted camera is and where it points in every frame, so the hands can be placed in the world as the wearer moves. Our camera pose beats leading open-source SLAM (Simultaneous Localization And Mapping) systems, and improves further when an IMU (Inertial Measurement Unit: the accelerometer and gyroscope built into most headsets and phones) is available.

The first step in tackling a problem like this is establishing a benchmark with accurate ground truth labels. Unfortunately, for 3D hand positions and camera pose, getting ground truth labels after the fact is next to impossible, even for a human. A person can roughly place 2D keypoints, but 3D depth is extremely tricky. Occluded hands are another big issue, and so is estimating how the camera moved. You can see for yourself how hard this is by guessing them in Figure and Figure below.

So the ground truth has to be captured while the footage is recorded. For hands, datasets use motion capture: ARCTIC (⁠Fan et al., 2023⁠) and HOT3D (⁠Banerjee et al., 2025⁠) track small markers on the hands with a studio camera system, and SHOW3D (⁠Rim et al., 2026⁠) uses a backpack rig of outward-facing cameras, synchronized with the headset, and no markers. For camera pose, the reference comes from the device's own closed-loop tracking (Aria Gen 2, Meta Reality Labs Research, 2025) or from an external tracking system (SteamVR Lighthouse in Monado, de Mayo et al., 2025). All of this needs a studio or special hardware, which is why only a handful of such datasets exist.

We used the following datasets as benchmarks:

Metrics:

• Cam-MPJPE (mm): the average distance between predicted and true joints in camera space, with no alignment, reported as mean / median / 90th percentile.
• Recall: the share of ground-truth hands the model finds. A hand counts as found when the model labels it correctly as left or right, and the box around its predicted joints (projected into the image, no padding) overlaps the box around the true joints with IoU > 0.5.
• PA-MPJPE (mm): the joint error after aligning each predicted hand's rotation, translation, and scale to the ground truth, so it measures hand shape only.

For every frame we also need the camera pose: the camera's rotation and translation in a fixed world frame (the camera-to-world transform). It is what places the hands from every frame in one shared 3D space, so the motion of the hands can be told apart from the motion of the head.

We used the following datasets as benchmarks:

Metrics:

• RPE 1 s (cm / °): the error of the camera's motion over every 1-second step, translation and rotation, with no alignment.
• Scale error (%): for every 1-second step where the camera moved more than 5 cm, we divide the predicted distance traveled by the true one. The scale error is how far the median of these ratios is from 1 (e.g. 10% if the camera's motion comes out 10% too long or too short).
• ATE (cm): the camera position error (RMSE) over the whole recording, after one rigid alignment of the whole path with rotation and translation only, no scale (often called ATE-SE3).

MacroHands‑Mono needs only one camera: from ordinary video, it predicts the 21 joints of each of the wearer's hands directly in 3D, in the camera's own coordinates.

On the benchmarks (Figure), MacroHands‑Mono is the most accurate model on two of the three datasets and finds nearly every hand on all of them. The clearest case is SHOW3D, which none of the models compared here has trained on: our joints land 32.2 mm from the truth on average, against 55.9 mm for HaWoR, the best open-source baseline, which also misses about one hand in five. We fall slightly behind only on ARCTIC, where ACE‑Ego‑Hand, trained on ARCTIC itself, reaches 18.7 mm to our 19.4 mm. MacroHands has never seen an ARCTIC frame, not even from its training split.

The only 3D hand benchmark where a baseline has lower error than ours is ARCTIC (18.7 vs 19.4 mm), and that baseline, ACE‑Ego‑Hand, trained on ARCTIC's training split. This is a small example of a bigger problem: benchmark contamination. Early on, when we tested hand-tracking models for our own use, we kept seeing models post great numbers on the benchmarks they had trained on, then lose depth entirely on footage from a different device. We think the cause is scarce data: there are only a handful of labeled 3D hand datasets, so most models train on the few that can be used as benchmarks. The cameras, people, and rooms in a benchmark's training split match its test split, so a model only has to learn the typical hand sizes and distances of those recordings to score well.

Testing for this properly means recording your own held-out ground-truth dataset in a motion-capture studio, which takes time and money. But the gap is often so large that a far simpler test shows it. Take a phone with known camera intrinsics and a tape measure, and turn off video stabilization (since it crops and shifts the image, invalidating the intrinsics). Then record yourself placing your wrist at 10, 20, 30, 40, and 50 cm along the tape (50 cm is a stretch, but doable), run the model with the phone's intrinsics, and compare its wrist depth with the tape.

Your measurement will only be good to 1–3 cm, but that should be plenty. Many of the other hand-tracking models we tried were not off by a few centimeters, but by 10 to 20 cm, and the error grew the further the hand moved from the camera. If you are benchmarking a hand-tracking model, we think this is the first test you should run.

Beyond depth, the second problem we kept running into on real egocentric data was how well models find and track the right hands in the first place. Unlike benchmark videos, real footage is full of things that trip them up: other people's hands reaching into the frame, the wearer's bare feet (which can look a lot like hands), the wearer's own hands disappearing behind the tools they hold, and work gloves, which fully occlude the hands.

The benchmark. To measure how well our model and others handle these cases, we built an internal benchmark of egocentric clips that none of the models trained on. Every frame is hand-labeled with a bounding box for each of the wearer's hands, marked left or right. We also label the hands of other people in view (about a third of all labeled hands), and a model that reports one of them as the wearer's is marked wrong. Clips are also tagged by situation: feet in view, occlusion, gloves, and no wearer hands.

Scoring. We score every model the same way, from the 3D hands it outputs. We project its 21 joints into the image and take their outline, padded by 15% (our labeled boxes are hand-drawn and a bit larger), as the box. A detection is correct when its score is above 0.5 and its box overlaps the labeled wearer hand by at least 50% (IoU ≥ 0.5). AP50, the average precision at that overlap, combines precision and recall over all confidence thresholds; higher is better.

Everything else is an error. A box that is not correct goes to the labeled hand it overlaps most, if that overlap is at least 20%: the correct hand (the box is only partly on it), the wearer's other hand (left/right swapped), or someone else's hand. A box that overlaps no labeled hand by 20% is a false detection. A wearer hand with no correct box is missed. The chart below shows both sides: what happened to each of the wearer's hands, and where each box the model reports lands.

MacroHands finds 83% of the wearer's hands at 84% precision. The strongest baseline, HaWoR, finds 51% at 66%.

The gap is widest when someone else is in the shot. On clips with long stretches of no wearer hands, often with another person's hands in view, the right answer is to output nothing. There, none of MacroHands' boxes land on another person's labeled hands, while HaWoR, HaMeR, and WiLoR put 89% of their boxes there.

The baselines also lose the wearer's hands exactly where the footage gets hard. With gloves on, the best of them finds 48% (MacroHands: 79%). Under occlusion, 59% (MacroHands: 80%).

We can't release this benchmark because it contains internal data, but Figure shows examples from open-source videos.

Everything so far used one camera. While MacroHands‑Mono is fairly accurate, it can still get depth wrong. Stereo cameras help with depth by adding a second, slightly shifted view of the same scene (see Figure). We trained a variant of our model specifically to ingest stereo footage: MacroHands-Stereo. MacroHands‑Stereo's error drops from 32.2 mm to 13.1 mm on SHOW3D, two and a half times lower, and from 12.2 mm to 10.6 mm on HOT3D. The hardest frames improve just as much: the 90th-percentile error falls from 65.0 mm to 24.7 mm on SHOW3D, and from 23.8 mm to 19.2 mm on HOT3D.

Only two of the benchmarks originally used for mono have stereo footage: SHOW3D and HOT3D, both recorded with the two front cameras of a Quest 3. ARCTIC's egocentric view comes from a single head-mounted camera, so it appears only in the mono results. Mono models are scored on the left camera, on the same hands. Surprisingly, the other models built for stereo do little better than the mono baselines. UmeTrack (⁠Han et al., 2022⁠), Meta's multi-view hand tracker and the best of them, is at 46.4 mm on SHOW3D against 55.9 mm for HaWoR with one camera, and at 46.2 mm on HOT3D against 48.4 mm for HaWoR*.

The gain is almost entirely depth. After Procrustes alignment (PA‑MPJPE), the two models are level on SHOW3D, 8.5 mm with two cameras and 8.4 mm with one. What a second camera adds is how far away the hand is, which a single view can only infer from appearance. So if you're choosing a capture rig, we recommend stereo. If you already paid for large amounts of mono footage, MacroHands‑Mono still beats every stereo baseline we ran.

As we showed in our previous post, knowing how the hands move relative to the camera is not enough. Robot training and scene reconstruction both need to know where the camera itself is going, so we trained our model to also estimate camera movement.

The camera path matters at two scales: over a whole recording it must not drift, so the scene can be rebuilt or turned into a simulation, and over a single second it must be precise, since that is the horizon over which policies are typically trained to predict action chunks. We designed our pipelines for both uses and report the error at each scale. Monado and Aria Gen 2 also record an IMU (motion sensor), and MacroHands uses it when available. On HOT3D, which has no IMU, every method uses only the cameras and MacroHands still leads (0.96 vs 1.22 cm RPE in stereo).