EgoExplorerA Vision–Motion–Touch Observatory for Human Loco-Manipulation In Situ

Humanoids are built for human spaces, and almost nothing useful in those spaces happens standing still. A person fetching a reagent walks to the bench, leans in, adjusts their stance so both hands can reach, and keeps the grasp while they turn away. The walking and the manipulation are not two phases. They are one behaviour.

That is the behaviour we want robots to learn from, and it is the behaviour today's portable datasets record least well. Egocentric video scales beautifully and carries enormous visual and semantic breadth, but the body that produced the view is unobserved. Wearable trackers recover the body, but usually stop at the wrist. Glove and contact datasets recover the fingers, or the touch, and usually drop the walking. Each one is a true recording of part of the behaviour, and the part they all drop is the coupling between the parts.

The datasets that do carry body and view together are mostly captured in instrumented space, or carry the body without the hands. We built EgoExplorer to record that coupling: an observatory of human loco-manipulation — 100+ hours, growing by about five hours every day, 2,715 recordings — captured with a wearable rig light enough to walk between a chemistry bench, a factory bay and a kitchen. Four streams — what the person saw, how the whole body moved, what every finger did, and where the hands actually pressed — are recorded together, on one clock, in rooms we did not instrument.

04 / RECONSTRUCTION

From five poses to a whole body

Sparse sensing moves the cost from hardware to reconstruction, and the bill comes due immediately. A tracker pose is the product of two unknowns — where the device sits on the limb, and how the limb is articulated — and a great many combinations of the two explain the same measurement. Fit them together and a mounting error quietly becomes a pose error. Everything between the five anchors is not observed at all.

Our pipeline, C³, refuses to solve those unknowns at the same time.

• Calibrate. A short prescribed sequence, matched by phase against a reference motion library, gives body shape, metric scale and each sensor-to-anchor transform. Solved once per mounting configuration, and fixed before any motion is inferred.
• Complete. A temporal prior — AvatarPoser extended from three anchors to five and trained on AMASS — turns the calibrated anchors into a first whole-body trajectory, supplying the joints no tracker observes.
• Constrain. That trajectory is plausible but not yet faithful. Three layers refine it against the measurements, against the body's own surface, and against the floor.

Only the body is reconstructed. Finger articulation and contact are measured directly: the glove reports twenty-one hand points and, in the tactile configuration, seventeen regions of pressure per hand, at roughly 120 Hz. The pipeline's job for the hands is alignment, not estimation — the calibrated wrist transform is what carries the glove's own frame onto the solved arm, so the fingers and the pressure land on the same body, on the same clock.

The third stage is the one that is easy to under-build. Matching a wrist target says nothing about whether the arm passed through the chest on the way, and an ankle above the floor says nothing about the heel. So the surface layer constrains oriented clearance on the participant's own posed mesh, and the ground layer runs over every posed vertex. All three optimise one trajectory through one shared objective, not one after another.

Against tracker-fitted references this reaches 18.0 mm body-point error, against 64.1 mm for the strongest baseline we evaluated — AvatarPoser retrained on all five anchors. The number we actually watch is the one at the seventeen joints no tracker ever sees: 20.3 mm, barely worse than the average over the whole body. A method that merely snapped to its anchors would show the opposite — tight at the five, loose at the elbows and the knees — so it is that near-flat margin, not the headline figure, that says the articulation itself improved.

07 / CITE

Citation

If EgoExplorer is useful in your work, please cite it as:

@misc{zhou2026egoexplorer,
      title={EgoExplorer: A Vision-Motion-Touch Observatory for Human Loco-Manipulation In Situ},
      author={Yunsong Zhou and Xuekun Jiang and Haiqi Shen and Hangxu Liu and Chaojian Shi and Yujie Liu and Jingyuan Li and Haoran Yang and Jiafei Cao and Dongzhan Zhou and Dong Wang and Yao Mu and Chunhua Shen and Weinan Zhang},
      year={2026},
      eprint={2610.18872},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2610.18872},
}

The dataset, the capture toolkit and the reconstruction code will be released.