Turning Egocentric Video into Robot Actions

The scale of ordinary egocentric video comes with a tradeoff: it gives us pixels of a human demonstration, but not the robot actions that a policy should predict next. To turn egocentric video into action data, we need to reconstruct the 3D position and movement of the operator’s hands in physical space.

That is the problem we tackle in this post: how to turn raw egocentric video into 3D hand-action trajectories, without any additional sensors. We will build the pipeline from open-source components, optimize its performance, and ensure that it runs at 15+ FPS on an H100.

At Macrodata Labs, we build embodied data intelligence that transforms raw physical-world data into rich training datasets for robotics. While this post focuses on open-source models, we are also developing proprietary hand-tracking methods that outperform this pipeline on challenging real-world footage. If open-source hand tracking falls short on your egocentric data, get in touch.

An egocentric video records a human performing an action, but a robot policy ultimately needs commands specific to its own body, such as end-effector poses or joint positions. These robot-specific commands can be obtained by first extracting the physical 3D motion of the person’s hands, including how the wrists and fingers move through metric space while the camera moves with the wearer. We use this as a common representation of the demonstrated action that can be adapted to different robot embodiments.

The resulting trajectory is directly usable for robotic policy training if treated as a co-embodiment. It could also be transformed into aligned robotic actions but this is a separate problem, which we do not evaluate here. Robot retargeting introduces robot kinematics, inverse kinematics, joint limits, and controller design on top of the visual reconstruction problem.

There are multiple ways to represent a hand trajectory. A compact representation can treat the hand like a gripper, while a detailed representation can retain the motion of the wrist and fingers.

We choose the same general level of detail as ViTra, but use a more directly geometric target. We represent each hand as 21 metric 3D joint positions—one wrist joint and 20 finger joints—rather than MANO parameters or robot-specific joint angles. This keeps the target robot-agnostic and makes the resulting trajectory directly measurable, while allowing smaller representations to be derived later. We also report wrist position accuracy separately.

Choosing the 21 joints tells us what each hand pose contains. We must also determine how those poses relate across a video recorded by a moving camera.

Hand predictions are initially expressed in the coordinate system of each frame’s camera. These coordinate systems move with the wearer’s head. If the wearer keeps their hand still but turns their head to the left, the hand appears to move right in the image. This apparent motion is visible in Figure. Comparing the per-frame predictions directly would therefore mix hand motion with camera motion.

To separate the two, we estimate how the camera moves and transform all hand predictions into a shared world coordinate system. Every frame is then measured relative to the same fixed reference, allowing us to recover the motion of the hands independently of the camera.

To evaluate these trajectories in a way that reflects how they would be used for VLA training, we organize them into one-second action chunks. We start a chunk at every frame $t$ for which the complete future horizon is available. For each chunk, we transform the future shared world hand trajectory into the coordinate system of the camera at frame $t$. This coordinate system remains fixed throughout the chunk, while keeping the trajectory relative to the observation from which the actions would be predicted.

The previous section defined the target: metric 3D hand trajectories that remain coherent as the egocentric camera moves. To evaluate whether a pipeline recovers them correctly, we need a dataset that provides:

• Calibrated egocentric RGB video.
• Independently measured camera poses.
• Metric 3D positions for both 21-joint hands.
• Hand-visibility labels indicating when the ground truth is reliable.
• Long episodes that test camera motion, occlusions, and temporal consistency.

Several datasets come close, but they differ both in the geometry they provide and in how that ground truth is produced:

HOT3D contains recordings from Project Aria and Quest 3, calibrated camera streams, dense metric 3D hand and camera trajectories, and hand-visibility confidence. Its hand and object poses are measured using an optical motion-capture system with physical markers. Because it provides the closest match to our requirements, we chose HOT3D as our primary benchmark.

The main limitation of HOT3D is diversity. Its controlled recordings cover a narrower range of environments and hand appearances than large-scale, in-the-wild egocentric video.

We evaluate our pipelines on ten complete HOT3D Project Aria episodes: 19,350 native 30 Hz RGB frames, or 10.75 minutes of video in total—about 65 seconds per episode. Using complete episodes rather than short clips tests whether a system can maintain a coherent trajectory through camera motion, occlusions, and tracker fragmentation.

Most hand-tracking pipelines attach a confidence score to each predicted pose. If the confidence is below a chosen threshold, the pipeline discards that pose as unreliable, leaving the hand without a prediction for that frame. This creates a scoring problem. Excluding missed frames rewards a pipeline for predicting only the easiest cases, while assigning every miss the same fixed error can make the result depend more on an arbitrary penalty than on the quality of the reconstructed hands.

To assign an error to these missing frames without inventing a fixed penalty, we fill them with temporary pose estimates. When a gap lies between two direct predictions, we linearly interpolate between them. When frames are missing before the first or after the last direct prediction, where interpolation is impossible, we reuse the closest available pose. We compare the resulting dense trajectory with HOT3D’s ground truth. Separately, the pipeline must reach at least 75% direct-prediction coverage on frames where HOT3D marks a hand as visible.

Speed is the second eligibility requirement. We require at least 15 FPS on an H100. Assuming the input videos run at 30 FPS, this means that one hour of video requires at most two H100‑hours to process, keeping the compute cost below $10 per hour of video. We measure end-to-end throughput across all ten episodes. We exclude model loading and video download, but include detection, camera reconstruction, hand reconstruction, fusion, and post-processing (without any caching).

Mean per-joint position error (MPJPE) is a standard metric for evaluating 3D pose estimates. It measures the Euclidean distance, in millimeters, between each predicted joint and its ground-truth position, then averages across the joints. It is widely used in 3D human-pose estimation, including egocentric 3D hand-pose evaluation (⁠Ohkawa et al., 2023⁠).

Standard hand-pose MPJPE evaluates one pose at a time, often relative to the wrist. We extend the same idea to the one-second, camera-relative hand trajectories defined in Separating hand and camera motion. We call this trajectory-level metric Action MPJPE.

We start a one-second action chunk at every possible frame. Each start frame produces separate left- and right-hand chunks. We include a hand chunk when HOT3D provides its starting camera pose and marks the hand as reliably visible in at least one future frame. Within an included chunk, only the future frames where the hand is visible contribute to the score.

For each chunk starting at frame $t$, we anchor the prediction and ground truth separately. We transform the predicted future joints from the pipeline’s reconstructed world frame into the coordinate system of its predicted camera at frame $t$. Independently, we transform the HOT3D future joints from the HOT3D world frame into the coordinate system of the ground-truth camera at frame $t$, aligning both trajectories to the camera axes of the same physical observation so we can directly measure the Euclidean distance between corresponding joints. This removes each trajectory’s arbitrary global origin and orientation while preserving errors in metric scale, relative camera motion, and hand motion. Action MPJPE first averages these errors over the visible frames and 21 joints within each hand chunk, then averages all eligible hand chunks equally:

Here, $\mathcal C$ contains the eligible left- and right-hand chunks, while $\mathcal V_{t,h}$ contains the HOT3D‑visible future frames for hand $h$ in the chunk beginning at $t$. The index $j$ runs over the hand’s 21 joints. The values $\hat{\mathbf p}_{t,h,i,j}$ and $\mathbf p_{t,h,i,j}$ are, respectively, the predicted and HOT3D positions of joint $j$, $i$ frames into the chunk, after both have been expressed relative to the camera at frame $t$. We report the result in millimeters.

For example, suppose the predicted and ground-truth index-fingertip positions differ by 10 mm horizontally, 20 mm vertically, and 10 mm in depth. The error for that joint is $\sqrt{10^2 + 20^2 + 10^2} = 24.5$ mm. We repeat this calculation for every joint in every visible future frame, average those errors within the hand chunk, and finally average all eligible chunks.

We score the trajectory exactly as the pipeline produces it, without shifting, rotating, or rescaling it before comparison. Both trajectories are measured relative to the camera at the start of the chunk. Errors in the predicted hand or camera motion therefore produce a larger error value. Lower Action MPJPE is better.

We start from HaWoR (⁠Zhang et al., 2025⁠), an end-to-end pipeline for reconstructing 3D hand trajectories in a common world coordinate system from egocentric RGB video.

The HaWoR pipeline begins with the hand detector from WiLoR (⁠Potamias et al., 2024⁠). It processes each frame independently and predicts a bounding box, confidence score, and left- or right-hand label for each detected hand. Boxes below a chosen confidence threshold are discarded. HaWoR then links the remaining boxes across consecutive frames when their handedness and image locations are consistent. This produces separate left- and right-hand tracks: sequences of accepted crops that appear to belong to the same hand. An overview of the complete pipeline appears in Figure.

HaWoR then reconstructs each hand track with a temporal network that uses neighboring frames rather than treating every crop independently. For every frame in the track, the network predicts the wrist’s position and orientation relative to the camera, together with MANO hand-shape and finger-pose parameters. MANO (⁠Romero et al., 2017⁠) decodes the shape and finger pose into a 3D hand mesh, while the predicted wrist pose places that mesh relative to the camera. This produces the 21 camera-space joint positions $\mathbf p^C_{t,h,j}$, where $t$ is the frame, $h$ identifies the left or right hand, $j\in1,\ldots,21$ identifies the joint, and the superscript $C$ denotes camera coordinates.

The pipeline next uses DROID‑SLAM (⁠Teed & Deng, 2021⁠) to estimate how the egocentric camera moved. DROID‑SLAM does this by finding the same visual points in consecutive frames and measuring how their positions changed. This works for the static background, but hands move independently and could be mistaken for camera motion. HaWoR therefore uses the reconstructed hand meshes to identify and ignore the hand pixels, then estimates camera motion and scene depth from the background.

To verify that masking matters, we reran DROID‑SLAM twice on the same 1,500-frame HOT3D sequence with identical settings. The only change was whether wearer-hand pixels were masked. Against the HOT3D camera ground truth, masking reduced mean camera-trajectory error from 0.247 m to 0.220 m.

While this monocular SLAM approach can recover camera rotation and the shape of the camera path, it cannot determine the path’s real-world scale. The same image motion could result from a camera moving one meter through a large scene or ten centimeters through a smaller version of it. DROID‑SLAM therefore recovers the camera trajectory only up to an unknown global scale: its translations are internally consistent, but are not yet measured in meters.

In a sensor-equipped recording, this scale can come from measured geometry: an RGB‑D (color-and-depth) camera, LiDAR, or a calibrated stereo rig, as is common in instrumented demonstration setups such as UMI. In our RGB‑only setting, none of those measurements are available at inference time. We have only the video, so the pipeline must estimate metric depth from the images themselves.

To recover this scale from RGB alone, Metric3D (⁠Yin et al., 2023⁠) estimates a dense depth map in metric units for each SLAM keyframe. HaWoR compares that depth with DROID‑SLAM’s unscaled depth on static-scene pixels, excluding the rendered hand masks. Each keyframe provides an independent estimate of the conversion from DROID‑SLAM’s internal scale to meters. Because Metric3D’s depth predictions are noisy, these estimates vary across keyframes. HaWoR takes their median and applies that single scale to the entire sequence. The resulting camera-to-world transform at frame $t$ consists of a rotation $R^{C\rightarrow W}_t$ and metric translation $\mathbf t^{C\rightarrow W}_t$, both defined relative to the fixed shared world frame $W$.

We can now combine the hand and camera trajectories. For joint $j$ on hand $h$, its world-space position at frame $t$ is obtained by applying that frame’s camera-to-world transform to its camera-space position:

The rotation expresses the camera-space point using the world axes, and the translation adds the camera’s position in the shared world. Repeating this for every directly reconstructed joint and frame places all observed hand poses in one shared 3D trajectory, even while the wearer moves their head. This only works once the camera trajectory has metric scale: otherwise the camera translation and the metric hand reconstruction would use incompatible units.

Some frames still lack a hand reconstruction because the hand is occluded, leaves the image, or falls below the detector’s confidence threshold. After placing the available predictions in the shared world, HaWoR fills these gaps with a learned motion infiller. The infiller does not inspect the images; it receives the incomplete MANO‑based trajectories for both hands, together with a mask indicating which frames contain direct reconstructions.

Using a 120-frame window around each gap, the infiller expresses each hand relative to its wrist pose at the start of the window, predicts the missing motion, and transforms the result back into the shared world. It fills only the missing frames, leaving direct reconstructions unchanged. The completed representation produces the same 21 world-space joints used throughout the pipeline.

We evaluated the original HaWoR pipeline on the ten HOT3D episodes, both with its motion infiller enabled and disabled. With the infiller disabled, missing frames were filled using the benchmark rule from Section; with it enabled, HaWoR’s learned completions were used instead.

The learned infiller increased Action MPJPE by 1.59 mm, despite significantly increasing coverage. On these episodes, the benchmark’s gap-filling rule produced a lower error than HaWoR’s learned completions. Both configurations exceeded the 75% direct-prediction coverage requirement.

The main issue with the original pipeline is its slow speed: 3.34 FPS, well below the required 15 FPS. Its runtime distribution is shown in Figure.

Camera reconstruction and metric-scale recovery consumed 61.7% of the runtime, more than all other stages combined, making it the pipeline’s main computational bottleneck.

HaWoR provides a complete end-to-end baseline, but it falls well short of the runtime requirement. In the following subsections, we examine how to make it substantially faster and more accurate.

Because the baseline’s DROID‑SLAM and Metric3D stage was its main computational bottleneck, we first evaluated alternative systems for camera reconstruction and metric-scale recovery.

There are two broad approaches to this problem. The first uses sequential reconstruction systems. We already used one such system in the baseline: DROID‑SLAM. As frames arrive, it repeatedly adjusts the estimated camera motion and scene depth so that visual points observed in different frames remain geometrically consistent. ViPE (⁠Huang et al., 2025⁠) follows the same sequential strategy, but maintains a selected set of keyframes and combines visual point tracks with metric-depth estimates. Because these systems extend an existing reconstruction as new frames arrive, they naturally maintain one shared coordinate frame across a long recording. The trade-off is the cost of repeatedly updating that reconstruction.

The second approach uses feed-forward geometry models such as VGGT‑Omega (⁠Wang et al., 2026⁠) and MapAnything (⁠Keetha et al., 2026⁠). Rather than maintaining an online reconstruction, these models jointly predict camera poses and scene geometry for a supplied set of frames through learned network inference. They are straightforward to batch and can be considerably faster. The limitation appears when a video is too long to process at once: we must divide it into windows, and each window is reconstructed independently. Consecutive windows can therefore disagree in position, orientation, or scale.

To compare these approaches, we evaluated several systems from both groups as replacements for the baseline's camera-reconstruction and metric-scale-recovery stage:

Although VGGT‑Omega had the worst action error in this first comparison, it was the only system that satisfied the runtime requirement. More importantly, this was not yet a fair test of what VGGT could do over a long recording.

VGGT takes a fixed window of K frames, which was 60 frames in our initial configuration, and predicts camera poses, depth, point maps, and intrinsics for that window together. It has no persistent world state beyond those K frames. To process a longer video, our first pipeline simply split it into consecutive windows with no overlap and concatenated their outputs.

That strategy treated every window as if it already used the same world coordinate system. It did not. Each window could reconstruct the local scene correctly while choosing a different global position, orientation, or scale from the window before it. Concatenating those independent reconstructions therefore introduced trajectory discontinuities in both the camera and, after world fusion, the hands.

We therefore decided to work on VGGT-based camera-pose estimation because it was the only approach that could realistically fit our compute budget. The next experiments were about turning its independent window reconstructions into one coherent long-video trajectory, inspired by Streaming VGGT (⁠Zhuo et al., 2025⁠).

To improve coherence between VGGT camera-pose windows, we introduced overlap. Instead of ending one window and starting the next immediately afterwards, we processed a short set of source frames in both windows.

More precisely, imagine two consecutive windows, A and B, that share the same overlap frames. VGGT reconstructs those frames twice: once as part of window A, and once as part of window B. The images are identical, but the predicted cameras and 3D points are expressed in two different local world coordinate systems.

We use the duplicate reconstruction of those overlap frames to estimate how window B's coordinate system must be rotated, translated, and potentially rescaled to match window A's coordinate system.

For a camera center at overlap frame $t$, window A predicts $\mathbf c_t^A$ and window B predicts $\mathbf c_t^B$. They represent the same physical camera location in two different coordinate systems. We fit a transform that makes those corresponding locations agree:

Here $\mathbf R$ is a 3D rotation, $\mathbf q$ is a 3D translation, and $s$ is one scale factor. After fitting it from the overlap, we apply the same transform to every 3D prediction from window B, including camera centers, scene points, and hand joints.

The alignment variants differed in the evidence used to establish correspondences and the transformation fitted from those correspondences.

We tested two transformation families. SE(3) is a rigid transformation: it rotates and translates one reconstruction into the other but does not change its size. It therefore assumes that both windows already use the same scale. Sim(3) additionally estimates one uniform scale factor, allowing one reconstruction to be enlarged or reduced before it is rotated and translated. This is useful for monocular reconstruction, where independently processed windows can assign different scales to the same scene.

We also tested two sources of alignment evidence. Camera-center alignment uses the camera positions predicted for the corresponding frames in the overlap. Depth-derived alignment uses each window's predicted depth and intrinsics to back-project corresponding overlap pixels into paired 3D scene points.

Combining these choices produced four variants: camera-center SE(3), camera-center Sim(3), depth-derived SE(3), and depth-derived Sim(3).

Beyond alignment, we also had to choose what to do with the duplicate predictions inside the overlap. Once the later window had been moved into the earlier window's coordinate system, both windows still produced a camera pose and geometry prediction for every shared frame.

We used two modes:

Linear blend: the earlier window received more weight near the start of the overlap and the later window received more weight near the end.

First-window prediction only: we kept the earlier window's camera and geometry predictions for every shared frame, then switched to the transformed later-window predictions immediately after the overlap ended.

We first fixed the overlap at 30 frames and tested these alignment methods:

Depth-derived Sim(3) produced the best alignment result. It was slightly slower because VGGT's depth head is optional and had not been enabled in the earlier camera-center experiments. We therefore kept this alignment method fixed and varied only the overlap size.

Increasing the overlap consistently lowered action MPJPE, at the cost of a slower pipeline because more frames were processed twice. We chose a 40-frame overlap, the best configuration in the sweep, because its throughput was still comfortably within our compute budget.

Beyond overlap, the second important parameter is the window length. Larger windows give VGGT more temporal context for estimating camera motion and scene geometry, and create fewer coordinate-system boundaries across a video. Those benefits come at a higher GPU‑memory cost.

The final parameter to choose is VGGT's input resolution. VGGT was trained with inputs up to 512 pixels, so a higher-resolution input should preserve more visual detail: small scene features, sharper depth boundaries, and more reliable image correspondences for camera reconstruction. Unlike window length, it does not add temporal context or reduce the number of window boundaries. The trade-off is higher GPU‑memory use.

We fixed the window length at 195 frames and the overlap at 40 frames, then varied the VGGT input bucket:

The higher-resolution bucket consistently improved the action score. Its measured throughput was still above our 15 FPS requirement, so we retained 416 px.

After all these experiments, our VGGT‑Omega‑based camera-reconstruction system used 200-frame windows, 40-frame overlap, a 416-pixel input bucket, depth-derived Sim(3) alignment, and linear blending. The full pipeline still ran at 15.53 FPS end to end.

More importantly, it reached 55.60 mm Action MPJPE, improving on every other camera-reconstruction system we tested, including the original HaWoR pipeline at 59.12 mm.

The geometry system from the previous section can only turn a hand trajectory into a stable world-space trajectory after we reconstruct that hand in each camera frame. To do that, we first need to find the hands in the video, track them across frames, and then infer MANO parameters from the resulting image crops.

We focus on that first step here: deciding which boxes to reconstruct and how to connect them through short detection failures.

HaWoR starts with the WiLoR hand detector (⁠Potamias et al., 2024⁠). For every frame it predicts hand bounding boxes, a confidence score, and a left/right label. HaWoR then uses BoT‑SORT (⁠Aharon et al., 2022⁠) to associate boxes over time. BoT‑SORT predicts where an existing track should appear in the next frame and links it to a compatible new detection, using box overlap and camera-motion compensation.

We kept the WiLoR detector, but began with a lighter association rule. For each side, we retain the highest-confidence box in a frame and link it only to nearby boxes of the same side. This is deliberately conservative: a false recovery is more costly than a short gap.

Our first question was whether a detector trained on a broad hand dataset could replace WiLoR. We compared WiLoR with a YOLOv10n (⁠Wang et al., 2024⁠) detector trained on HaGRID (⁠Kapitanov et al., 2022⁠), a large hand-image dataset. Both used a confidence threshold of 0.25.

HaGRID missed almost half of the visible egocentric hands, despite its low threshold. Since it was far below our 75% coverage requirement, we did not run the full hand-reconstruction pipeline with it.

Every WiLoR box has a confidence score. Lowering the threshold gives HaWoR more crops to reconstruct but increases the risk of false predictions, while raising it yields fewer direct predictions and more interpolation.

The best action score occurred at a threshold of 0.75. As the explorer shows, continually raising the confidence threshold does not improve the score. Our metric penalizes a method that predicts only easy frames: for missing frames, we linearly interpolate the hand trajectory before comparing it with ground truth.

A strict 0.75 threshold leaves short gaps where WiLoR produces a weaker but plausible box. To fix this, we only admit a low-confidence proposal when it occurs inside a short same-side gap with high-confidence WiLoR boxes immediately before and immediately after it.

For each proposed box, we linearly interpolate the two high-confidence anchor boxes and calculate IoU, the overlap divided by the combined area, between that expected box and the weak proposal. We accept it only when that IoU exceeds a fixed gate.

A four-frame gap and 0.20 IoU gate improved both action error and direct coverage. It is the selected detector/tracker configuration.

Finally, we evaluated several other ways to connect a hand through a missed detection.

ByteTrack (⁠Zhang et al., 2022⁠) first associates boxes above its high-confidence threshold with existing tracks. It then tries to attach lower-confidence boxes using the track's predicted next position and box overlap. Unlike our rule, it does not require a high-confidence WiLoR box immediately before and immediately after the missing interval. It can therefore keep a track alive through longer uncertain periods, but also accepts more ambiguous boxes.

BoT-SORT follows the same general approach, but first estimates camera motion between frames so that box association can account for a moving egocentric camera.

SAM2 (⁠Ravi et al., 2024⁠) starts from trusted hand boxes, propagates a hand segmentation mask through the gap, and derives a new box from that mask. EdgeTAM (⁠Zhou et al., 2025⁠) is a lighter temporal segmentation method that uses the same prompt-and-propagate approach.

Unfortunately, none of these methods improved the quality-speed frontier. They recovered more boxes, but those boxes came from the most occluded or ambiguous frames, where a plausible crop was still not sufficient for accurate 3D hand reconstruction.

Our final detection and tracking recipe is deliberately conservative:

• Run the WiLoR detector on every frame.
• Keep direct boxes with confidence at least 0.75.
• Preserve left and right hands as separate tracks.
• For gaps of at most four frames, consider 0.10-0.75 proposals only when high-confidence detections exist on both sides.
• Accept a weak proposal only if its IoU with the interpolated high-confidence box is at least 0.20.

This retains 81.23% direct hand coverage while avoiding the larger 3D reconstruction errors created by more aggressive tracking.

Now that we know where the hands are in each frame, the remaining visual task is to turn those detections into camera-space 3D hand trajectories.

For HaWoR, a trained temporal model receives a short sequence of hand crops rather than treating every image independently. It predicts the parameters of a MANO hand model: finger and wrist rotations, hand shape, and the hand translation relative to the camera. MANO is a parametric 3D hand model. Forward kinematics then converts these parameters into the 21 hand joints we use throughout this evaluation.

Our first hand-reconstruction experiment was simply to try other models. We kept the rest of the pipeline fixed and replaced only the component that converts a hand crop into 3D joints.

We tested five different ways to turn a detected crop into 3D joints.

• HaWoR takes a sequence of crops and jointly predicts a temporally consistent MANO trajectory, including finger pose, wrist orientation, hand shape, and camera-relative translation.
• WiLoR predicts a MANO hand from each crop independently. We converted its predicted parameters into the same camera-space 21-joint representation used by the rest of our pipeline.
• HaMeR (⁠Pavlakos et al., 2024⁠) also predicts a MANO hand from an individual crop.
• HaPTIC (⁠Ye et al., 2025⁠) is another learned parametric hand-reconstruction model. We tested it as a direct replacement for HaWoR, but its inference cost was too high to run a meaningful end-to-end comparison.
• MediaPipe Hands (⁠Zhang et al., 2020⁠) predicts 2D landmarks and a relative 3D hand rather than a metric camera-space trajectory. We used depth at the wrist to set its scale and position, testing depth from VGGT and DA3 Metric Large (⁠Lin et al., 2025⁠).

HaWoR was better by a large margin. WiLoR produced plausible single-frame hands, but its independent predictions were less stable once converted into a long camera-space trajectory. MediaPipe with DA3 was substantially better than MediaPipe with VGGT depth, but it still reached 87.80 mm and had poor coverage because MediaPipe frequently failed to detect the hands. We therefore retained HaWoR.

HaWoR is a temporal model that reconstructs sequences of 16 frames at a time. Consecutive sequences can produce slightly different predictions near their boundaries, so we tested overlap between neighboring hand-reconstruction windows. Unlike VGGT, these predictions are already expressed in the same camera coordinate system, so no Sim(3) alignment is needed. For every duplicated frame, we linearly blend the two joint and translation predictions: the earlier window has more weight at the start of the overlap and the later window has more weight at the end. We blend rotations with spherical interpolation instead of averaging rotation matrices directly.

Overlap helped, but its effect was small compared with the geometry-window stitching changes in the previous section. Eight frames gave the best result in this sweep, so we retained it for the later end-to-end configuration.

We kept HaWoR, using 16-frame crop windows with eight-frame overlap and overlap blending. WiLoR was faster but less accurate over a long video; HaMeR and HaPTIC were too slow; and MediaPipe landmarks did not become reliable metric trajectories after depth lifting.

The overlap result is deliberately modest. It reduces disagreement at hand-model window boundaries, but most of the larger improvement came from making the camera trajectory coherent across the video. With both components chosen, we can combine their outputs into a shared 3D hand trajectory.

Although the preceding stages already produce a complete hand trajectory, post-processing can use temporal and geometric consistency to reduce prediction error and jitter. By jitter, we mean rapid frame-to-frame changes in the reconstructed joints that do not correspond to the underlying hand movement.

Alongside action MPJPE, we therefore report acceleration error, a standard diagnostic for jitter in pose reconstruction (⁠Wei et al., 2022⁠). For every three consecutive valid frames, we compare the predicted second difference of each joint with the ground-truth second difference. We report it in mm/frame² because all videos in this benchmark use the same 30 Hz frame rate.

Acceleration error is a diagnostic rather than our optimization target. A method can obtain a very smooth trajectory simply by removing real movement. We therefore select configurations using action MPJPE, then use acceleration error to distinguish similarly accurate trajectories. In total, we evaluated 45 post-processing configurations. Most did not improve action accuracy, so below we focus on the representative failures and the methods that survived.

Although HaWoR is a temporal model, its reconstructed joints can still be visibly jittery. We first tested two direct smoothing methods. A centered mean filter replaces each joint with the average over a short temporal window. A Gaussian filter performs the same operation but gives the current and nearby frames more weight than frames farther away. We swept mean-filter windows from 3 to 17 frames and Gaussian widths from 1.5 to 2.5.

Both filters reduced acceleration error substantially, but every setting worsened Action MPJPE. The Gaussian filter nearly halved the acceleration error while adding 0.72 mm of action error; wider mean filters removed progressively more real movement. We therefore disabled direct joint smoothing in the selected pipeline.

VGGT's estimated camera position can also contain small frame-to-frame changes. Because camera translation is added to every world-space hand joint, this noise appears as movement of both hands. We therefore tested a centered binomial filter on the camera translation while leaving camera rotation unchanged.

For the selected three-frame filter, the camera positions at frames $t - 1$, $t$, and $t + 1$ receive weights $1$, $2$, and $1$:

The three-frame filter produced a small improvement in both metrics. The wider filter began removing useful camera movement, so we retained the three-frame version.

The wrist translation predicted from one crop is the least constrained part of a monocular reconstruction. Its image position tells us the direction from the camera to the wrist, but its distance along that ray remains uncertain. We therefore assign one adjustable wrist depth to every reconstructed frame in a continuous hand track. Changing this depth moves the complete hand towards or away from the camera without changing its 2D projection or finger articulation.

We optimize all of these depths together after transforming the wrists into the shared world coordinate system. The objective balances two terms. The first keeps each optimized depth close to HaWoR's original prediction. The second penalizes abrupt world-space acceleration, measured from every three consecutive wrist positions. A smooth movement with a similar displacement in each frame has little penalty; an unsupported jump towards the camera followed by an immediate jump back has a large penalty.

The acceleration weight, denoted by λ, controls the balance between these terms. Our sweep selected λ = 0.2, which gave the best action MPJPE while substantially reducing acceleration error.

The first term is also weighted by detector confidence. High‑confidence crops keep their original depth more strongly, while uncertain crops are allowed to move farther when that removes an abrupt trajectory jump. In the selected configuration, we normalize confidence by the median confidence of the track, clamp it to 0.5–1.5, and raise it to the eighth power. The optimization is run independently for each continuous left- or right-hand track and changes depth only; it neither fills missing frames nor creates new hand poses.

Larger weights continued to reduce acceleration, but eventually did so by removing real movement. Weight 0.2 gave the best action MPJPE and was therefore used in the final combined configuration.

HaWoR predicts both a hand shape and its camera-space translation. If we stabilize bone lengths over a clip but leave wrist depth untouched, we create a physically inconsistent hand: its bones change size while its distance from the camera does not. We instead measure a clip-level hand scale, then apply the same bounded correction to wrist depth along the original camera ray. The 2D projection remains unchanged.

Mean bone lengths worked better than the median. The combined configuration used a 3.5% correction bound; larger corrections overreacted to frame-level shape noise.

We also tested the following adjustments, but all of them regressed the action metric:

• Longer mean and Gaussian joint filters produced smoother but less accurate trajectories.
• Smoothing wrist rotation, MANO pose, or MANO shape did not help.
• Using the clip-median bone length was worse than using the clip mean.
• Bone/depth corrections wider than the selected bound increasingly overcorrected individual frames.
• Applying a temporal median filter to the bone/depth scale slightly regressed the action score.
• Root-relative articulation averaging did not change the resulting joint metrics.
• Direct VGGT scene-depth corrections failed badly.
• HaWoR's learned motion infiller and EgoInfinity's (⁠Wang et al., 2026⁠) temporal infiller did not provide a reliable advantage over applying fixed linear interpolation inside the evaluator.

We retained the narrow camera-translation filter, bounded scale/depth correction, and ray-constrained wrist-depth optimization. Together, post-processing improved Action MPJPE from 53.7045 mm to 52.0736 mm and cut acceleration error from 12.7329 to 7.2182 mm/frame² because these changes preserve the image projection of every direct hand observation. We reject generic hand smoothing because it removes real hand movement.

The current output therefore contains only direct 3D hand predictions. Missing poses remain explicitly missing in the saved annotation, while the benchmark interpolates them consistently during evaluation.

After all of these experiments, the final RGB-only system retained the WiLoR detector and HaWoR hand reconstruction, but replaced HaWoR's original world reconstruction with the windowed VGGT‑Omega pipeline. It uses a detector threshold of 0.75, with low-confidence proposals admitted only inside same-side gaps of at most four frames when their interpolated-box IoU is at least 0.20. HaWoR reconstructs overlapping 16-frame crop sequences.

VGGT‑Omega runs at the 416-pixel input bucket in 200-frame windows with 40-frame overlap. Consecutive windows are aligned using depth-derived Sim(3) and linearly blended in their shared frames. The resulting camera translation is filtered over three frames. Finally, we apply the bounded clip-level bone-scale correction and optimize wrist depth along the original camera ray with an acceleration weight of 0.2 and detector-confidence weighting.

The table below reports the fully certified end-to-end results. These are the numbers used for the final comparison; the earlier tables isolate individual components through controlled rescoring experiments.

Compared with the original HaWoR pipeline, the final system reduces action error by 12.0% while increasing measured throughput from 3.34 to 15.53 FPS. Compared with our initial VGGT‑based baseline, action error falls by 31.3%. The lower direct coverage remains a real limitation. Interpolation does not excuse missing predictions; it assigns them a trajectory derived from surrounding observations so they still incur error rather than disappearing from the evaluation.

The final question is which parts of the pipeline still produce the largest errors. To separate camera reconstruction from hand reconstruction, we repeated the action evaluation using only frames with a direct HaWoR prediction and visible ground truth, deliberately excluding non-predicted frames. We then replaced the camera trajectory and each camera-space hand component with ground truth and used Shapley attribution to distribute their interactions:

On frames where the pipeline reconstructs a hand, its camera-space prediction is the dominant remaining error source. Wrist translation, particularly monocular depth, is the largest component. Replacing only the camera trajectory with ground truth reduces the error from 39.20 to 34.87 mm, while replacing the complete hand-model prediction reduces it to 12.67 mm.

Using the same direct-prediction subset, we decomposed the action error along the axes of the camera at the start of each action. The horizontal x-axis contributes 11.52 mm, the vertical y-axis contributes 10.86 mm, and the depth z-axis contributes 16.82 mm. Depth accounts for about 43% of the 39.20 mm direct-only action error, although the two image-plane axes contribute more when combined.

Although camera pose contributes less error than hand reconstruction, we also studied the effect of VGGT window boundaries on prediction quality. We replaced every directly predicted hand with its ground-truth camera-space pose while retaining VGGT's estimated camera trajectory. VGGT window boundaries remain visible but are not dominant: actions crossing a boundary score 13.29 mm, compared with 12.42 mm elsewhere. Larger camera motion has a much stronger relationship with camera error. Actions with less than 2 cm of camera translation score 5.74 mm, while actions above 10 cm reach 25.09 mm. Similarly, error rises from 5.79 mm below 2 degrees of camera rotation to 16.50 mm above 10 degrees.

Finally, we tested the open-source pipeline on data from our clients. The HOT3D benchmark does not cover several deployment-specific challenges, including wearer identification, gloves, and wrist-mounted cameras. Although the pipeline works fairly well in clean environments without heavy occlusion, the most significant issues on client data are typically related to camera-space hand prediction. In particular, the detector sometimes identifies hands belonging to people other than the camera wearer, or fails entirely when gloves or wrist-mounted cameras are present. Because of these limitations, we have developed our own detection system.

In this blog, we introduced, improved, and diagnosed the current state of open-source hand-tracking pipelines.

The strongest system was not a single new reconstruction model. It combined conservative hand detection, temporal HaWoR reconstruction, long-window VGGT camera estimation, metric window alignment, and narrowly targeted trajectory corrections. Together, these changes reduced the one-second action error of our initial VGGT‑based system from 90.73 to 52.04 mm while keeping the complete pipeline above 15 FPS.

The benchmark also exposes where current open-source systems still fail: camera-space hand reconstruction, reliable detection, and camera-pose estimation during larger head movements.

At Macrodata, we are building our own proprietary hand-tracking pipelines for real-world robot and egocentric data. They combine the strongest components evaluated here with our own models and optimizations for challenging real-world footage. We believe solving these remaining limitations is critical to making egocentric video a practical source of robot-training data.

Send us a small video sample and we’ll run hand-action extraction on it for free, so your team can evaluate the results.