Source-linked AI summary
Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipates
Hui Wei, Licai Sun, Guoying Zhao
TL;DR
Human-centric image models are strong at static perception but do not represent motion or anticipate the future. Human-JEPA adapts a video JEPA with frozen-initialization dense anchors and a past-to-future forecasting split, leading specialists on pose and person re-identification with fewer parameters while avoiding degradation from its released predictor head.
Problem
Existing human-centric models excel on static human perception but represent no motion and predict no future, leaving anticipation unaddressed.
Method
Human-JEPA specializes a video JEPA through anchored forecasting, pinning dense targets to a frozen initialization and replacing block masks with a pure past-to-future split.
Results
Under frozen probes, Human-JEPA leads pixel-anchored specialists on pose and person re-identification at 2.7× fewer parameters, and its predictor head does not degrade anticipation.
Takeaways & Limitations
A single safely adapted human-centric model serves both present perception and immediate-future anticipation within the demonstrated frozen-probe setting.
Takeaways & Limitations
At the tested observation fraction, rollout adds essentially nothing over the encoder alone, so the demonstrated value is representation consistency rather than extra anticipatory signal.
Abstract
from arXiv · showhide
Machines that understand humans should perceive the present and anticipate the future. Existing human-centric vision model are pretrained on human images, set the state of the art in static dense perception, so motion and anticipation are out of reach. Here we present Human-JEPA, a human-centric vision model trained on video by anchored forecasting: dense targets are pinned to a frozen copy of the initialization, preventing a silent collapse of dense perception, and block masks are replaced by a pure past-to-future split, avoiding a five-point action tax and a seventeen-point re-identification collapse. Under frozen probes, Human-JEPA leads the pixel-anchored specialists on pose and person re-identification at 2.7 times fewer parameters, conceding high-resolution dense parsing, and its released predictor head is the first that does not degrade anticipation. A single safely adapted model thus serves both halves of understanding humans.
1 INTRODUCTION
Human-JEPA addresses the gap between static human perception and future anticipation by adapting a video JEPA with anchored forecasting. Under frozen probes, it leads human specialists on pose and identity while also supporting anticipation.
- Motivation: Human-centric image models excel at static perception but represent no motion and predict no future.They support tasks such as parsing, pose, and re-identification, while anticipation remains out of reach.
- Approach: Human-JEPA combines human-centric perception and anticipation in one video model through anchored forecasting.Dense targets remain tied to a frozen initialization, while a pure past-to-future split makes the predictor model scene evolution.
- Results: 0.620 AP pose performance leads every pixel-anchored specialist, including the 2.7× larger Sapiens2-0.8B.The comparison uses frozen probes over two pretraining seeds.
- Results: Human-JEPA leads pixel-anchored specialists on pose and person re-identification at 2.7× fewer parameters, while its predictor head does not degrade anticipation.The model is evaluated across seven frozen-probe tracks under one protocol.
2 RELATED WORK
Human-centric representation learning has progressed from task-specific systems toward shared models, while self-supervised learning has developed contrastive, distillation, masked-modeling, and JEPA paradigms. JEPA predicts latent target-region embeddings from context, extending from images to video.
- Human-Centric Vision: Human-centric vision moved toward general-purpose representations shared across pose, parsing, re-identification, and related tasks.Unified models jointly train across human-centric datasets using task-specific queries or projectors.
- Self-Supervised Learning: Self-supervised learning learns transferable representations from massive unlabeled data through contrastive learning, distillation, and masked data modeling.These three paradigms organize the related self-supervised methods discussed here.
- Self-Supervised Learning: Contrastive methods learn by separating instances or aligning multiple views of the same scene.The cited progression includes instance discrimination, CMC, and SimCLR.
- Self-Supervised Learning: Distillation methods match outputs between networks, with variants using positive pairs, stop-gradient, or momentum teachers.BYOL, SimSiam, and DINO illustrate this paradigm.
- Self-Supervised Learning: Masked data modeling learns representations by reconstructing missing inputs or predicting latent representations from masked views.MAE and data2vec exemplify reconstruction- and latent-prediction-based approaches.
- JEPA: JEPA predicts target-region embeddings from context embeddings in latent space, avoiding augmentation-dependent positives and low-level reconstruction biases.I-JEPA introduced the paradigm for images, followed by video variants including V-JEPA 2.1.
3 HUMAN-JEPA
Human-JEPA keeps a released video JEPA’s architecture and initialization while specializing it for human video through anchored dense targets and forecasting masks. The design preserves dense features and directs adaptation toward temporal prediction without person-level masking or pose input.
- 3 HUMAN-JEPA: Human-JEPA preserves the released video JEPA’s architecture and size, producing a checkpoint swap with a transferable encoder and predictor.It is trained self-supervised on curated human video and person crops.
- 3 HUMAN-JEPA: Longer training on human data can silently collapse dense features and spend adaptation capacity on a copy task rather than temporal representation.The standard masked objective is identified as a source of this specialization tax.
- 3 HUMAN-JEPA: Human-JEPA addresses these problems with an anchored context stream and a forecasting mask family while retaining the architecture and initialization.It omits person-level masking and pose input; a two-phase schedule is optional.
- 3 HUMAN-JEPA: The student encoder processes unmasked video tokens, while the predictor estimates masked-token latents supervised by an EMA teacher.A context loss also supervises visible positions with multi-layer targets, supporting dense features.
- 3.1 ANCHORED CONTINUED PRETRAINING: Setting the context-target network to a frozen initialization preserves the dense representation, while an image branch supplies full-body appearance from person crops.Stock training instead allows dense targets to drift with the student.
- 3.2 FORECASTING MASKS, AND MIXING THE FAMILIES: A pure past-to-future split replaces repeated spatial blocks, making continuation require a model of scene evolution rather than appearance copying.The first half of the tubelets is context and the second half is target; the dense stream remains anchored.
- 3.2 FORECASTING MASKS, AND MIXING THE FAMILIES: Per-batch mixtures expose an appearance-versus-dynamics trade-off between forecasting and block mask families.The forecasting probability can be constant or follow a two-phase schedule, with family selection occurring across batches.
4 EXPERIMENTS
Frozen-probe experiments show that Human-JEPA preserves human perception while adding anticipation, with forecasting and anchoring determining the central trade-offs. The model leads specialists on pose and identity, while dense parsing remains a supported limitation and continued pretraining can silently collapse without preservation mechanisms.
- Experimental scope: The evaluation uses frozen probes across perception, identity, anticipation, ablations, and causal partner-use tests.The experiments apply a common frozen-probe protocol across tracks and include systematic design attribution.
- Data: 536,699 Kinetics-700 clips are filtered to 164,431 full-body clips and 82,696 clips with at least two spatially engaged people, with AIST++ and LUPerson-T adding motion and image data.The full-body gate retains 30.6 percent of clips; the image branch uses 958k LUPerson-T images.
- Perception results: 0.620 pose AP and 0.4635 Market-1501 mAP place Human-JEPA ahead of pixel-anchored specialists, while it concedes 0.9 points of parsing and 1.9 of DensePose against its base.The pose result is 2.9 AP ahead of Sapiens2-0.8B, and ReID improves 2.7 mAP over the frozen base.
- Anticipation results: The released predictor improves the full bundle by 4.06 NTU-120 points over the base bundle, and its own head does no harm at +0.06.The base’s released head instead costs 2.98 points; at the tested observation fraction, the rollout adds essentially nothing beyond the encoder alone.
- Anticipation results: 0.873 cosine agreement with the true continuation exceeds 0.798 for holding the last observed latent fixed across 20 held-out clips.The margin over the static reference holds through two seconds, supporting a forecast rather than a copy.
- Ablations and limitations: Pure past-to-future masking avoids the block-mask action tax and ReID collapse, while anchoring and image co-training prevent silent dense-perception collapse.Block masks lose about five action points and 17 ReID mAP; naive continued pretraining drops pose AP from 0.614 to 0.110 despite decreasing training loss, whereas no repair closes the dense gap near 0.767 versus 0.74.
5 CONCLUSION
Human-JEPA combines human-centric perception and anticipation by preserving dense features during video specialization and replacing inpainting with forecasting. The paper concludes that pixel-anchored pretraining remains strongest for static high-resolution appearance, while Human-JEPA specializes people in time.
- Human-JEPA is the first human-centric vision model reported to perceive the present and anticipate the immediate future.
- Continued pretraining silently collapses dense perception through target drift; anchoring dense targets to the frozen initialization with an image branch prevents it.
- Forecasting rather than inpainting removes the specialization tax associated with the mask family.
- Person-level prediction is falsified in all nine tested forms by a causal partner-ablation probe.
- Pixel-anchored pretraining owns static appearance at high resolution, whereas Human-JEPA owns people in time.