EgoXperience

Learning World-Grounded Egocentric Representations
from Human Experience

Xinhao Ji1,3,*,★Hui En Pang*,★Hao Li1,*,‡,★Kairui Hu1,★Jiner Wang1Mingze Sun2,★
Ziang Cao1Yalun Dai1,★Thang Luu1,DQin Ma1,DHongyi Chen1,DYukang Cao1
Fangzhou Hong★Zhaoxi Chen★Ziwei Liu1,†,★
★ Ropedia1 Nanyang Technological University2 Tsinghua University3 Peking University

* Equal contribution‡ Project leaderD Data collection† Corresponding author

Nanyang Technological University Singapore
Tsinghua University — purple seal and Chinese/English name
Peking University — seal and Chinese/English name
CodePaperVideoDataModel
↓

INTRO

TheEgoXperienceParadigm

EgoXperience is a full-stack paradigm spanning real-world capture, aligned annotation, unified reconstruction, and benchmarking, turning human experience into world-grounded representations for embodied intelligence.

EgoX-Engine
EgoX-Data-6M
EgoX-Bench
EgoX
test

01 · ENGINE

EgoX-Engine

EgoX-Engine transforms captured experience into eight synchronized modalities: RGB, depth, camera poses, MANO hands, object masks, object 6DoF trajectories, metric object meshes, and task captions.

01 · OBJECT ASSET

Metric object mesh

Multi-view object scanning cage with a green object
Multi-view cage scan
Complete geometryMetric scale

02 · EGO CAPTURE

HOMIE Capture Suite

HOMIE Gen2 egocentric capture device
Ego capture device
RGBDepthCameraCaptionMANO

03 · MASK TRACKING

Identity-safe masks

First-mask anchor↓SAM3 propagation↓Identity recovery

04 · 6DOF

Verified 6DoF trajectory

FoundationPose proposals↓
Evidence verificationDepthSilhouetteCoverageMotion
↓Temporal state update

05 · OUTPUT

Aligned annotations

  • Object 6DoF trajectory
  • Object mask sequence
  • Object mesh
  • Ego-view depth
  • Camera poses
  • Caption
  • MANO

02 · DATA & BENCHMARK

EgoX-Data-6M&EgoX-Bench

EgoX-Data-6M contains 50 hours and 5.4 million multimodal frames across 270 tasks and 101 objects, spanning workspace, cross-room, and outdoor activities. EgoX-Bench is a quality-controlled one-hour benchmark for joint reconstruction.

Source annotations
50hours
5.4Mframes
270tasks
101objects

TASK COVERAGE

Task distribution

270tasks
  • Single-area 49.0%
  • Cross-room 33.8%
  • Outdoor 17.2%

CAPTURE SETTINGS

Region distribution

Cross-room33.83%
Outdoor17.18%
Tabletop15.21%
Kitchen14.62%
Dining8.42%
Living room8.20%
Bedroom5.20%
Bathroom1.88%

OBJECT DIVERSITY

101 objects, six families

Kitchen 48Storage 23Home 12Office 8Cleaning 8Toys 2

03 · MODEL

EgoX

EgoX extends the VGGT feed-forward reconstruction paradigm from scenes to egocentric video. From monocular observations and a first-frame object mask, it jointly reconstructs the full system in a shared metric world frame with a single forward pass.

Input
Egocentric video
First-frame object mask
→
EgoXNeural network1.2B parameters
→
Shared metric geometry
Hands
Object 6DoF
Depth
Camera

Training data

  • Real-world53.2%
  • Simulated46.8%

Training sources

Real-world ≈151.0 h

  • EgoX-Data-6M 50 h
  • ACE-Data-0
  • Xperience-10M
  • AgiBot World 2026
  • DexYCB
  • HO-Cap
  • HOT3D
  • HOI4D
  • H2O
  • H2O-3D

Simulated ≈133.1 h

  • SceneNet RGB-D
  • Aria Synthetic Environments
  • BEHAVIOR-1K
  • Dynamic Replica
Performance Examples

04 · EVALUATION

EgoXPerformance

We evaluate hand, object, camera, and depth reconstruction on HOT3D, EgoX-Bench, HOI4D, and H2O.

05 · CITATION

BibTeX

@article{ji2026egoxperience,
  title   = {EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience},
  author  = {Ji, Xinhao and Pang, Hui En and Li, Hao and Hu, Kairui and Wang, Jiner and Sun, Mingze and Cao, Ziang and Dai, Yalun and Luu, Thang and Ma, Qin and Chen, Hongyi and Cao, Yukang and Hong, Fangzhou and Chen, Zhaoxi and Liu, Ziwei},
  journal = {Ropedia Technical Report},
  year    = {2026}
}