ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video

ReViV reconstructs viewer-centric human motion (body, hand, and gaze) and view-centric scene geometry (camera and depth) from a single egocentric RGB video in a unified feed-forward model. For visualization, the initial camera pose and camera intrinsics are estimated with GeoCalib.

Overview

Abstract

Egocentric devices, such as wearable front-facing cameras, provide a unique perspective for capturing the continuous interaction between a human viewer and the surrounding environment. A holistic and efficient multimodal model capable of reconstructing this 4D representation is therefore highly desirable. However, existing approaches often rely on auxiliary inputs such as pre-computed camera trajectories, treat scene perception and human ego-motion modeling as separate problems despite their strong interdependency, and suffer from slow inference time. To address these limitations, we present ReViV, the first unified framework for holistic egocentric 4D reconstruction that extracts both viewer and view dynamics from a single monocular RGB video. We formulate the task as learning the full joint probability distribution over multimodal signals, including RGB video, camera trajectory, gaze direction, full-body motion, hand motion, and depth. Powered by a Masked Generative Egocentric Transformer, ReViV operates within a single feed-forward architecture to simultaneously reconstruct the temporally consistent 4D reconstruction across the viewer and the view with fast inference speed. Extensive experiments on diverse benchmarks, including HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO, demonstrate that ReViV achieves state-of-the-art accuracy and efficiency across holistic ego-body, hand, and gaze reconstruction, camera tracking, while maintaining highly competitive egocentric depth estimation, without relying on heavy task-specific priors.

Method

ReViV architecture with modality-specific tokenizers, masked generative modeling, and multimodal reconstruction — **ReViV architecture.** Modality-specific VQ-VAEs discretize heterogeneous viewer signals (gaze, hand, and body) and view signals (RGB, depth, and camera) into a unified token sequence. A Masked Generative Egocentric Transformer (MGET) learns the joint distribution by predicting randomly masked tokens, capturing both temporal dynamics within each modality and correlations across modalities. During inference, MGET conditions on the observed RGB tokens and iteratively reconstructs the unobserved human and scene states. A lightweight floor-fitting stage aligns the viewer and the view within a shared metric 4D coordinate system.

Experiments

Egocentric Body Motion Reconstruction

ReViV establishes a new state of the art for monocular egocentric body motion reconstruction. Even without camera trajectories as input, it outperforms baselines that rely on ground-truth cameras across local pose accuracy, semantic correspondence, and motion realism, while being 100× faster than EgoAllo and over 10× faster than UniEgoMotion — demonstrating that global ego-motion can be effectively learned implicitly from pure monocular video.

Baselines given a VIPE camera trajectory

Columns from left to right: Input RGB · Ground Truth · Ours (RGB only) · EgoAllo (RGB + VIPE camera) · UniEgoMotion (RGB + VIPE camera).

Baselines given the ground-truth camera trajectory

Columns from left to right: Input RGB · Ground Truth · Ours (RGB only) · EgoAllo (RGB + GT camera) · UniEgoMotion (RGB + GT camera).

Each row merges one clip into a single strip. ReViV uses RGB only in both settings, while the baselines are additionally provided a camera trajectory.

Egocentric Hand Motion Reconstruction

ReViV outperforms both frame-based (HaMeR) and dynamic (Dyn-HaMR) baselines while being 100× faster than HaMeR and over 400× faster than Dyn-HaMR. It leverages strong spatiotemporal priors learned from the joint distribution to produce biomechanically plausible, temporally consistent hand movements under severe occlusion, even when the hands are entirely unobservable from the egocentric view.

Left: Input RGB. Remaining columns from left to right: Ground Truth · ReViV (Ours) · HaMeR · Dyn-HaMR. Within each column, the top view is the 3D hand reconstruction rendered from a third-person viewpoint and the bottom view is that reconstruction reprojected onto the egocentric frame; the ground-truth hands are overlaid in every method column for reference.

Two ARCTIC sequences. For each, the enlarged input RGB video is shown on the left, followed by the ground truth and the three reconstructions. The 3D renderings (top) and their egocentric reprojections (bottom) show that ReViV recovers biomechanically plausible, temporally consistent hand poses that align closely with the ground truth, staying stable under severe occlusion and out-of-view intervals from RGB input alone.

Egocentric Camera Tracking

Unlike methods that rely on explicit geometric constraints, ReViV predicts camera poses directly from egocentric RGB input end-to-end. It achieves the lowest relative translation and rotation error among all methods, predicting camera poses in a single feed-forward pass without geometric post-processing.

Camera trajectory result for sequence one — Ground-truth and predicted trajectories are visualized for representative egocentric sequences.

Camera trajectory result for sequence two — Ground-truth and predicted trajectories are visualized for representative egocentric sequences.

Each video compares ReViV with EgoM2P, EgoMono4D, and VIPE on the same egocentric clip while preserving the original trajectory visualizations above.

Egocentric Gaze Estimation

ReViV predicts 2D gaze locations from egocentric video and achieves significantly lower error compared to EgoM2P on unseen data.

Gaze estimation result one — Predicted gaze follows the wearer's visual attention across changing scene context.

Gaze estimation result two — Predicted gaze follows the wearer's visual attention across changing scene context.

ReViV tracks gaze over time from egocentric RGB input, showing stable attention estimates under changing scene context.

Egocentric Video Depth Estimation

Trained from scratch without depth-expert priors, ReViV significantly outperforms EgoM2P on the unseen dataset. While it falls behind EgoMono4D — which inherits strong depth-specific priors — ReViV is substantially more efficient, achieving over 20× faster inference.

Columns from left to right: Ground Truth, Ours, EgoM2P, EgoMono4D.

ReViV produces temporally consistent depth predictions from the same RGB stream used for the other reconstruction tasks.

Citation

@inproceedings{lyu2026reviv,
  title     = {ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video},
  author    = {Lyu, Xiaozhong and Li, Gen and Qian, Zhiyin and Zhang, Xucong and Pollefeys, Marc and Tang, Siyu},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

ReViV is built upon EgoM2P and EgoGen. Please also consider citing these works:

@InProceedings{li2025egom2p,
    author    = {Li, Gen and Chen, Yutong and Wu, Yiqian and Zhao, Kaifeng and Pollefeys, Marc and Tang, Siyu},
    title     = {EgoM2P: Egocentric Multimodal Multitask Pretraining},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2025},
    pages     = {10830-10843}
}

@InProceedings{li2024egogen,
    author    = {Li, Gen and Zhao, Kaifeng and Zhang, Siwei and Lyu, Xiaozhong and Dusmanu, Mihai and Zhang, Yan and Pollefeys, Marc and Tang, Siyu},
    title     = {EgoGen: An Egocentric Synthetic Data Generator},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {14497-14509}
}