Holi-World: Building Holistic World Intelligence from Human-Ego Experience

// Paper Video

// Introduction

Holi-World turns egocentric recordings into static 3D objects, dynamic 4D tracks and grounded training data, without manual labels.

Foundry auto-annotationData ~80K grounded QAVL-4B modelBench human-annotated eval

Static 3D · ScanNet

0.55AP₅₀

4.6× the strongest baseline

Dynamic tracking · ADT

54.31% HOTA

vs. 36.00% with GT initialization

Reasoning · Holi-World-Bench

78.02% F1

4B model beats zero-shot 27B

Grounded supervision

~80KQA

from ~100 scenes

The Ecosystem

From experience capture to world intelligence

Five parts, two verification loops.

Hardware

Ego Capture Device

Stereo RGB + IMU, in the wild

Pipeline

Holi-World-Foundry

Reconstruction · 3D/4D annotation

Dataset

Holi-World-Data

Geometry · semantics · QA

Model

Holi-World-VL

Spatial-temporal reasoning

Held-out captures

Human Annotation

Independent labels

Benchmark

Holi-World-Bench

Reference for both loops

data flowverification

Captured in the wild

Mapping walks record the room. Interaction sessions record the hands at work.

~100

Scenes

444

Ego-videos

1.32M

RGB-D frames

Holi-World-Foundry

An engine that writes world records

Reconstruct once, then split: static objects persist, dynamic objects move.

Input

Stereo RGB + IMU

01 · Shared prep

Geometry & semantic cues

VI-SLAMDA33DGSVLM

02 · Static

Persistent 3D objects

detectmergeverify

03 · Dynamic

4D object tracks

associatetracklift

04 · Output

Unified world state

IDsboxestracksevidence

/01Shared preparation

Build the world first

Poses, depth and a Gaussian scene, then VLM cues decide which branch each object takes.

VI-SLAM posesDA3 depth3DGS sceneVLM routing

/02Static perception

Two detectors, one VLM inspector

Real counts from one room:

Boxer 2D lifts · left + right1,684

Associated + TR3D, merged30

After point-support filter22

VLM check✓ 17✎ 4✕ 1

Static 3D objects21

Static object perception pipeline: dual-path candidates, merge and filter, VLM validation with one fallback retry.

/03Dynamic perception

Same cup, same ID, even after it leaves the view

Interaction groundingIdentity associationSAM mask tracking3D back-projectionPer-frame OBB4D track

Dynamic object perception pipeline: identity association, mask propagation, back-projection, per-frame OBBs and 4D tracks.

/04Unified world state

One record, every label traceable

One office, one coordinate frame:

880

Camera poses

95

Static objects

A→B

Tracked trajectory

Holi-World-Data

Four levels of grounded supervision

Every example links back to the objects, times and frames behind it.

1

Raw data

Scenes: ~100
Ego-videos: 444

2

Geometry

3DGS scenes: ~100
RGB frames: 1,322K
Depth maps: 1,322K
Camera poses: 1,322K

3

Semantics

2D boxes: ~90K
Static 3D boxes: 2,466
Dynamic 4D tracks: 2,095

4

Spatial intelligence

3D grounding: ~4K
Captions: ~80K
Static QA: 13,647
Dynamic QA: 64,409

QWhat is behind the bed?

AThe curtain.

static 3D boxescamera pose

30 QA types across static mapping, trajectories and captions, including hard negatives.

Experiments

Accurate annotations, real learning value

Best AP on all three datasets.

4.6×

AP₅₀ on ScanNet

ScanNet · AP₂₅+0.28

Holi-World0.68

Map-Det3D0.40

01.0

ScanNet · AP₅₀×4.6

Holi-World0.55

Map-Det3D0.12

01.0

ScanNet++ · AP₂₅+0.07

Holi-World0.30

Boxer0.23

01.0

Bench · AP₂₅+0.08

Holi-World0.24

Boxer0.16

01.0

Full table · 9 methods × 3 datasets

	ScanNet			ScanNet++			Holi-World-Bench
Method	AP₂₅↑	AP₅₀↑	mIoU↑	AP₂₅↑	AP₅₀↑	mIoU↑	AP₂₅↑	AP₅₀↑	mIoU↑
Model-based methods
SpatialLM	0.18	0.09	0.52	0.03	0.01	0.50	0.04	0.00	0.23
LLaVA-3D	0.02	0.00	0.34	0.00	0.00	0.34	0.00	0.00	0.00
SceneScript	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Map-Det3D	0.40	0.12	0.48	0.18	0.05	0.48	0.12	0.01	0.37
3DGS-based methods
M3-Spatial	0.00	0.00	0.37	0.00	0.00	0.60	0.06	0.00	0.28
LangSplat	0.02	0.00	0.45	0.01	0.00	0.33	0.01	0.00	0.10
Spatial pipeline frameworks
Boxer	0.32	0.09	0.47	0.23	0.05	0.46	0.16	0.07	0.52
Holi-Spatial	0.33	0.10	0.47	0.17	0.05	0.45	0.07	0.01	0.37
Holi-World (Ours)	0.68	0.55	0.72	0.30	0.13	0.56	0.24	0.12	0.53

[0, 1] scale. AP at 3D IoU 0.25 / 0.50; mIoU over pairs matched at IoU 0.25. best second best.

Qualitative static 3D detection comparison of Boxer, Holi-Spatial, Holi-World and ground truth on ScanNet, ScanNet++ and self-collected scenes.

Baselines get ground-truth help at test time. Foundry gets none, and still leads.

+18.3

HOTA pts on ADT20

ADT20 · HOTA+18.31

Holi-World54.31

IT3DEgo*36.00

0100%

ADT20 · AssA+18.34

Holi-World71.94

IT3DEgo*53.60

0100%

Bench · HOTA+28.01

Holi-World43.14

SAM+LMK*15.13

0100%

Bench · PCL+21.05

Holi-World57.89

EgoSeg3D†36.84

0100%

Full table · 5 methods × 12 metrics

	ADT20						Holi-World-Bench
	Bounding box			Trajectory			Bounding box			Trajectory
Method	IoU↑	GTC↑	PP↑	HOTA↑	AssA↑	PCL↑	IoU↑	GTC↑	PP↑	HOTA↑	AssA↑	PCL↑
Instance segmentation and tracking
EgoSeg3D^†	–	–	–	21.89	37.11	36.47	–	–	–	5.76	21.78	36.84
IT3DEgo^*	–	–	–	36.00	53.60	34.64	–	–	–	2.44	4.36	7.89
SAM+Whareformer^*	–	–	–	30.76	40.99	31.26	–	–	–	14.63	20.97	36.84
3D object perception and tracking
SAM+LMK^*	7.96	21.27	14.53	26.10	30.76	21.45	20.28	50.84	21.30	15.13	14.35	23.68
Holi-World (Ours)	30.88	43.03	56.25	54.31	71.94	47.95	41.79	53.18	76.49	43.14	56.54	57.89

% at a 0.1 m trajectory tolerance. ^*GT first-visible mask initialization; ^†GT category queries. best second best.

Qualitative dynamic tracking comparison of IT3DEgo, Whareformer and Holi-World in three scenes.

Trained only on Holi-World-Data, a 4B model beats a zero-shot 27B on every metric.

16→78

Token F1 (%) · Bench

Bench · Token F1+61.67

VL-4B (ours)78.02

Qwen 27B43.67

Qwen 4B16.35

0100%

EgoDynamic4D · Token F1+48.10

VL-4B (ours)66.39

Qwen 27B42.93

Qwen 4B18.29

0100%

Bench · Numerical+71.04

VL-4B (ours)81.84

Qwen 27B66.13

Qwen 4B10.80

0100%

EgoDynamic4D · Yes/No+40.20

VL-4B (ours)99.67

Qwen 27B61.07

Qwen 4B59.47

0100%

Full table · 3 models × 8 metrics

	Holi-World-Bench				EgoDynamic4D
Model	F1↑	Y/N↑	MCQ↑	Num.↑	F1↑	Y/N↑	MCQ↑	Num.↑
Qwen3.5-4B	16.35	34.86	62.00	10.80	18.29	59.47	79.16	0.02
Qwen3.8-27B	43.67	82.85	75.00	66.13	42.93	61.07	57.54	0.16
Holi-World-VL-4B	78.02	93.52	94.00	81.84	66.39	99.67	94.92	16.33

%. F1 is Token F1; Y/N, MCQ and Num. are yes/no, multiple-choice and numerical accuracy. best second best.

Ablations

Each part covers another's blind spot

image lift → small objectspoint cloud → large extentsVLM → right labelslabel continuity → stable IDsSAM masks → smooth tracks

Qualitative ablation results for static and dynamic components.

Citation

BibTeX

@article{holiworld2026,
  title   = {Holi-World: Building Holistic World Intelligence from Human-Ego Experience},
  author  = {Li, Hao and Zheng, Xiyuan and Gao, Yuanyuan and Dai, Yalun and Ji, Xinhao and
             Chen, Xuan-Li and Zhao, Jing and Hong, Fangzhou and Chen, Zhaoxi and Liu, Ziwei},
  journal = {arXiv preprint arXiv:2610.26549},
  year    = {2026}
}