PaperlayerView exact revision on arXiv

Source-linked AI summary

Human-Centric Intelligence in the Era of Foundation Models: A Survey

Yang Chen, Tianqi Wang, Xiaorui Jiang, Yilei Man, Yihua Shao, Mengyuan Liu, Zhi Chen, Xiaofeng Cao, Qibin Zhao, Chi Harold Liu, Albert Y. Zomaya, Nicu Sebe, Jingren Zhou, Dacheng Tao, Song Guo, Jingcai Guo

arXiv:2608.18184v1cs.CV

TL;DR

Human-centric intelligence remains fragmented across tasks, modalities, and research communities, with unclear conceptual and methodological connections. This survey introduces a six-level human context taxonomy and synthesizes methods, evaluation resources, advances, and challenges to provide a coherent framework for the field.

  • Problem

    Human-centric intelligence remains fragmented across tasks, modalities, and research communities, leaving its conceptual and methodological connections unclear.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    The survey organizes human-centric intelligence through six contextual levels and synthesizes its data, architectures, optimization strategies, methods, datasets, benchmarks, and metrics.

    Source: §1 · pp. 2–3, §2 · p. 3

  • Results

    The survey provides a full-spectrum account connecting human perception, generation, interaction, and action across the six human-context levels.

    Source: §8 · p. 38, §9 · p. 40

  • Takeaways & Limitations

    The taxonomy and organized resources provide a coherent framework and practical reference for advancing human-centric intelligence in the foundation-model era.

    Source: §1 · p. 2, §9 · p. 40

    The survey notes that convincing world-model futures do not guarantee decision utility, because planning requires actionable interventions and calibrated uncertainty.

    Source: §6.1.2 · p. 24

Abstract

from arXiv · show

Human-centric intelligence is evolving in the foundation-model era, with growing emphasis on scale, transferability, and general-purpose modeling. Yet it has not fully integrated with foundation models to achieve the comparable progress seen in them. More importantly, recent advances across this broad landscape remain fragmented across tasks, modalities, and research communities, leaving their intrinsic conceptual and methodological connections unclear. To bridge these divides and rethink human-centric intelligence in the foundation-model era, we introduce a full-spectrum human context taxonomy that integrates six interconnected levels by viewing humans as observable subjects through visual appearance and spatial geometry, as dynamic actors through kinematic dynamics and interaction modeling, and as situated agents through world simulation and embodied agency. We next present the methodological foundations of the field, covering human-centric data families, computational architecture paradigms, and representative training and inference optimization strategies. We then systematically review representative methods across these levels and organize the associated datasets, benchmarks, and evaluation metrics. We further discuss open challenges and promising research directions toward human-centric intelligence that is scalable, trustworthy, physically grounded, and deployable, aiming to provide a coherent framework and practical reference for advancing the field. Finally, we provide a systematically organized and continuously updated collection of human-centric AI literature and resources on our project page.

1 Introduction

This survey reframes human-centric intelligence for the foundation-model era through a full-spectrum taxonomy spanning observable subjects, dynamic actors, and situated agents. It organizes methods, evaluation resources, challenges, and future directions into a unified reference for the field.

Motivation

  • Foundation models shift human modeling toward large-scale pretraining, reusable backbones, multimodal interfaces, and cross-task or cross-domain adaptation, motivating this survey.Earlier approaches progressed from hand-crafted descriptors to data-driven deep learning but remained constrained by predefined assumptions, datasets, or objectives.Source: §1 · p. 2

Scope

  • The survey covers human-specialized and foundation-era methods using scalable learning, reusable pretraining, multimodal interfaces, cross-task generalization, and transferable human priors.Its scope includes humans as observable subjects, dynamic actors, and situated agents while excluding broader human-computer interaction, human-oriented language modeling, and policy studies.Source: §1 · p. 2

Taxonomy

  • The taxonomy connects visual appearance, spatial geometry, kinematic dynamics, interaction modeling, world simulation, and embodied agency across expanding human contexts.It progresses from observable subjects and spatial structure to motion, interactions, world dynamics, and physical execution or agentic intelligence.Source: §1 · p. 2, Figure 2 · p. 3

Methodological Foundations

  • It organizes methodological foundations around human-centric data families, computational architecture paradigms, and training and inference optimization strategies.The taxonomy guides examination of how representative methods develop human-centric capabilities in the foundation-model era.Source: §1 · p. 2

Evaluation Protocols

  • The survey consolidates datasets, benchmarks, and commonly used evaluation metrics to clarify how different human-centric capabilities are developed and evaluated.These resources provide the empirical foundations for comparing methods across the full spectrum of human-centric intelligence.Source: §1 · p. 3

Frontiers and Resources

  • It discusses open challenges and promising directions and releases systematically organized resources to support continued progress in human-centric AI.The survey is organized from taxonomy and methodology through methods, evaluation, challenges, future directions, and conclusion.Source: §1 · p. 3

2 Human Context Taxonomy

The taxonomy organizes human-centric intelligence by six interconnected human-context levels, spanning observable subjects, dynamic actors, and situated agents. It distinguishes perceptual and structural human properties, temporal and relational behavior, and environment-aware or executable capabilities while assigning multi-context methods by primary modeling target and evaluation objective.

2 Human Context Taxonomy

  • The six-level taxonomy spans visual appearance, spatial geometry, kinematic dynamics, interaction modeling, world simulation, and embodied agency across three perspectives: observable subjects, dynamic actors, and situated agents.The levels are interconnected rather than mutually exclusive, and methods spanning multiple contexts are assigned according to their primary modeling target and evaluation objective.Source: §2 · p. 3

2.1 Observable Subjects

  • Observable-subject modeling separates visual appearance, which captures visible and identity-bearing properties, from spatial geometry, which represents articulated body structure, shape, pose, and 3D spatial organization.Together, these levels provide perceptual and structural foundations, with appearance methods increasingly unified through large-scale pretraining and multimodal interfaces, and geometry methods moving toward reusable reconstruction and avatar systems.Source: §2.1 · p. 3, §2.1.1 · p. 4, §2.1.2 · p. 4

2.1.1 Visual Appearance

  • Visual-appearance tasks cover generalist perception, identity recognition and retrieval, and controllable human generation, while spatial-geometry tasks cover pose and mesh recovery and renderable avatar modeling.The taxonomy includes tasks ranging from pose estimation, parsing, recognition, and synthesis to body reconstruction, animatable avatars, multi-view synthesis, and novel-view rendering.Source: §2.1.1 · p. 4, §2.1.2 · p. 4

2.2 Dynamic Actors

  • Dynamic-actor modeling distinguishes kinematic dynamics, which captures the temporal evolution of individual body states, from interaction modeling, which explains behavior jointly shaped by objects, scenes, and other people.Kinematic methods connect motion understanding, generation, editing, and animation, whereas interaction methods organize human-object, human-scene, and social behavior while retaining contact, affordance, compatibility, and coordination constraints.Source: §2.2 · pp. 4–5, §2.2.1 · p. 5, §2.2.2 · p. 5

2.3 Situated Agents

  • Situated-agent modeling distinguishes world simulation, which predicts or generates coupled human-environment evolution and action consequences, from embodied agency, which turns human-centered knowledge into executable, transferable behavior.World simulation supports human-conditioned future generation, state-transition modeling, and planning; embodied agency covers humanoid control and human-to-agent skill transfer, with physical executability and embodiment mismatch remaining challenges.Source: §2.3 · p. 5, §2.3.1 · pp. 5–6, §2.3.2 · p. 6

3 Preliminary

Section 3 establishes three methodological foundations for human-centric intelligence: data specify learnable information, architectures transform it into capabilities, and optimization governs capability acquisition, adaptation, and elicitation.

3 Preliminary

  • Three foundations organize human-centric intelligence: heterogeneous human-centered data, computational architectures, and optimization strategies.They respectively define what information can be learned, determine how it becomes capabilities, and govern how those capabilities are acquired, adapted, and elicited.Source: §3 · p. 6
  • These foundations apply to human-specialized foundation models and adaptations of general-purpose pretrained models, reflecting scaling, transfer, multimodal integration, and general-purpose modeling.Source: §3 · p. 6

3.1 Human-Centric Data and Signal Families

The section organizes human-centric data by acquisition mechanism and preserved information into five signal families spanning visual observation, spatial structure, sensorimotor dynamics, wireless sensing, and linguistic or acoustic communication. Together, these families provide complementary capabilities while exposing modality-specific limitations in reliability, scalability, privacy, semantic detail, and alignment.

3.1 Human-Centric Data and Signal Families

  • Five human-centric signal families organize heterogeneous observations and structured representations by acquisition mechanism and the information they preserve.The taxonomy is illustrated in Fig. 4 and spans visual imaging, spatial-structural, sensorimotor, wireless and ranging, and linguistic-acoustic signals.Source: §3.1 · p. 6, Figure 4 · p. 7

3.1.1 Visual Imaging Signals

  • Visual imaging signals provide scalable, semantically rich observations through exocentric and egocentric RGB, infrared and thermal, and event-camera data, but remain vulnerable to occlusion, viewpoint and illumination changes, motion blur, and privacy concerns.Event cameras preserve fast human motion at high temporal resolution, while infrared and thermal imaging extends observation to low-light conditions.Source: §3.1.1 · p. 6

3.1.2 Spatial-Structural Signals

  • Spatial-structural signals explicitly represent human geometry through depth, poses, skeletons, point clouds, meshes, parametric bodies, and neural renderable formats, supporting geometry-aware modeling but requiring costly acquisition or error-prone estimation.These representations can be captured by depth cameras and scanners, reconstructed from multiple views, or estimated from visual observations.Source: §3.1.2 · p. 7

3.1.3 Sensorimotor Signals

  • Sensorimotor signals capture gaze, body dynamics, action trajectories, control commands, tactile contact, and physiological activity, offering temporally precise and privacy-preserving information but facing deployment and data-scale constraints.Limitations include sensor placement, calibration, drift, individual variation, synchronization, and limited data availability.Source: §3.1.3 · p. 7

3.1.4 Wireless and Ranging Signals

  • Wireless and ranging signals, including WiFi, radar, and LiDAR, provide non-contact motion and spatial observations when RGB imaging is unreliable or undesirable, but offer limited semantics and transfer poorly across sensing systems.Their performance depends on sensor configuration and environmental conditions.Source: §3.1.4 · p. 7

3.1.5 Linguistic and Acoustic Signals

  • Linguistic and acoustic signals express human behavior through captions, instructions, dialogues, narrations, speech, and nonverbal sound, enabling cross-modal understanding and controllable generation while indirectly describing geometry and motion.Their informativeness depends on semantic specificity, recording quality, transcription accuracy, and temporal alignment with other modalities.Source: §3.1.5 · pp. 7–8

3.2 Computational Architectures

Human-centric computational architectures are organized into four paradigms according to how encoded inputs are transformed into capability-specific outputs. They span single-step mapping, sequential factorization, iterative generation, and hybrid composition.

3.2.1 Single-Step Mapping Architectures

  • Single-step mapping architectures use one feedforward computation for representations or outputs, supporting perception, reconstruction, alignment, and 3D modeling but potentially struggling with target ambiguity.They include encoder-only [156], encoder-decoder [150], and JEPA-style [43] designs.Source: §3.2.1 · p. 8

3.2.2 Sequential Factorization Architectures

  • Sequential factorization architectures model variable-length text, motion, and actions [297] through next-token prediction, capturing long-range dependencies while incurring latency and error propagation.Autoregressive LLM and MLLM implementations factorize outputs as ordered conditional sequences.Source: §3.2.2 · p. 8

3.2.3 Iterative Generative Architectures

  • Iterative generation architectures progressively transform an initial state toward the target distribution, with diffusion and flow-based models capturing complex human-centric distributions at higher inference cost.Representative designs include LDMs [191] [252], DiTs [47] [86], and MM-DiTs [44] [169].Source: §3.2.3 · p. 9

3.2.4 Hybrid Architectures

  • Hybrid architectures combine multiple essential paradigms, such as autoregressive reasoning with diffusion realization, perception or language reasoning with control, or latent-state prediction with action generation.They may compose mechanisms sequentially, through shared representations, in parallel, or via iterative feedback; conventional modules alone do not qualify.Source: §3.2.4 · p. 9

3.3 Optimization Strategies

Optimization strategies govern how human-centric capabilities are acquired during training and elicited during inference. The survey organizes training mechanisms by parameter updates and optimization signals, and inference methods by test-time intervention mechanisms.

3.3.1 Training Mechanisms

  • Training updates all or selected parameters before deployment to learn, adapt, transfer, or align human-centric capabilities.The survey distinguishes scratch training, full-parameter fine-tuning, parameter-efficient tuning, instruction tuning, reward-based tuning, knowledge distillation, and task-head tuning, which may be combined in multi-stage pipelines.Source: Optimization Strategies · p. 9, §3.3.1 · p. 9

3.3.2 Inference Optimization

  • Inference keeps learned parameters fixed and improves capability elicitation through semantic augmentation, guided sampling, iterative refinement, preference-based selection, and retrieval augmentation.These mechanisms enrich inputs, steer sampling, correct outputs, select among candidates, or retrieve external knowledge, and may be combined within one inference process.Source: Optimization Strategies · p. 9, §3.3.2 · p. 10

3.3.1 Training Mechanisms

  • Scratch training forms human-specialized capabilities from random initialization but requires large-scale data and substantial computation, whereas full-parameter fine-tuning adapts pretrained models at high computational cost and may weaken general capabilities.Parameter-efficient tuning updates selected parameters or lightweight components, largely preserving general knowledge under limited compute, memory, or training data.Source: §3.3.1 · pp. 9–10
  • Instruction, reward-based, distillation, and task-head tuning respectively improve controllability, optimize underspecified behavior, transfer capabilities while reducing inference cost, and efficiently map frozen features to task outputs.Instruction tuning exposes capabilities through a shared language-facing interface; distillation can transfer across architectures, modalities, or perception-generation-control stages; task-head tuning also evaluates feature transfer across tasks.Source: §3.3.1 · p. 10

3.3.2 Inference Optimization

  • Inference interventions reduce ambiguity, improve controllability and consistency, satisfy geometric or physical constraints, select higher-quality candidates, and add instance-level knowledge absent from the input.They respectively rewrite semantic conditions, guide sampling, iteratively correct outputs, rank multiple candidates, and use retrieved examples or priors without changing parameters.Source: §3.3.2 · p. 10

4 Human Visual Appearance and Spatial Geometry

This section examines human visual appearance and spatial geometry as complementary perspectives on observable subjects, covering generalist perception, identity understanding, controllable generation, structured geometry modeling, and renderable avatar reconstruction. It emphasizes reusable human priors, flexible interfaces, human-specific structure and data, and the need for semantically plausible, visually consistent, metrically accurate, spatially coherent, uncertainty-aware, and animation-ready models.

4 Human Visual Appearance and Spatial Geometry

  • Visual appearance generalizes perception, identity understanding, and controllable generation, while spatial geometry seeks spatially consistent human states for reconstruction, animation, and rendering.Appearance captures image-space cues; geometry represents the body as an explicit or renderable structure.Source: §4 · p. 11, §4.1 · p. 11

4.1.1 Generalist Human Perception

  • Human-specialized pretraining and unified interfaces broaden generalist perception across dense, video, multimodal, and facial tasks, but task coverage must expand without sacrificing performance.HumanBench, HAP, Sapiens [150], Sapiens2, DAViD [285], THFM [324], Hulk, L-Man, FaceXFormer [253], FSFM, Face-MLLM [303], GroundingFace, and UniF2ace [180] exemplify this progression.Source: §4.1.1 · pp. 11–12

4.1.2 Discriminative Identity Understanding

  • Identity modeling is shifting toward transferable recognition across heterogeneous observations and query modalities, while separating persistent identity from changing appearance remains essential.SapiensID [156], SapiensID 2.0 [301], PLIP, Instruct-ReID [108], Instruct-ReID++ [109], ChatReID [256], and ReID5o support this shift, but biometric identity differs from descriptive grounding and pedestrian localization.Source: §4.1.2 · p. 12

4.1.3 Controllable Human Generation

  • Controllable human generation increasingly uses human-specific data, anatomical structure, factorized references, and multimodal unification to preserve identity and compose appearance changes.CosmicMan [191], FoundHand [40], Visual Persona [252], Voost [169], Tstars-Tryon 1.0 [44], Oxygen-TryOn [227], UPGPT [52], DreamActor-M1 [239], and Archon [12] illustrate this progression, while controllability remains harder than realism.Source: §4.1.3 · pp. 12–13

4.2.1 Structured Geometry Modeling

  • Structured geometry modeling scales explicit body recovery through larger data and models while preserving fine articulation, spatial context, temporal stability, and suitable spatial or temporal granularity.SMPLer-X [30], SMPLest-X, HaMeR [268], PEAR [349], and DanceHMR [292] demonstrate this trajectory.Source: §4.2.1 · p. 13
  • Scene-level geometry models unify multi-person recovery, camera estimation, tracking, prompting, and in-the-wild full-body reconstruction instead of relying on fixed detection-and-cropping pipelines.Multi-HMR [81], DETRAM [167], PromptHMR, and SAM 3D Body preserve scene context, temporal identity, or user-specified targets.Source: §4.2.1 · p. 13
  • Multimodal and hybrid methods expose human geometry to foundation models for pose reasoning, expression control, preference-guided mesh recovery, and body-shape reconstruction, while specialized geometric components retain responsibility for spatial prediction.These approaches include shared image–pose–language spaces, language-conditioned SMPL interaction, pose-aware synthesis, expression generation, vision-language preferences, and VLM-guided geometric recovery.Source: §4.2.1 · p. 14
  • Semantic or perceptual plausibility does not guarantee metric accuracy, spatial consistency, or physically valid structure, motivating explicit geometric grounding and uncertainty representation under incomplete observations.Foundation priors are therefore positioned as complements to, rather than replacements for, verifiable human geometry.Source: §4.2.1 · p. 15

4.2.2 Renderable Avatar Modeling

  • Renderable avatar modeling replaces prolonged per-subject optimization with amortized feed-forward reconstruction and population-level pretraining, enabling faster, reusable avatar creation across diverse inputs.IDOL reconstructs photorealistic humans from single images, LHM predicts animatable Gaussian avatars with a large transformer, and HumanNOVA combines image evidence, body priors, and large human-asset collections.Source: §4.2.2 · p. 15
  • Population-level avatar priors infer unobserved content and support controllable generation, but plausible or view-consistent outputs can still have unreliable geometry, animation, and uncertainty.Methods use canonical formulations, generated views, large-scale human-asset collections, 3D tokenization, specialized head representations, and generative conditioning, underscoring the need for coherent, controllable, animation-ready models.Source: §4.2.2 · pp. 15–16

5 Human Kinematic Dynamics and Interaction Modeling

The section frames kinematic dynamics and interaction modeling as the dynamic-actor perspective of human-centric intelligence, advancing from reusable temporal priors toward multimodal and relational foundation-model interfaces. It covers motion evolution, video animation, and interactions with objects, scenes, and people, while emphasizing persistent gaps in physical, temporal, and reciprocal grounding.

5 Human Kinematic Dynamics and Interaction Modeling

  • Foundation-era human-centric intelligence is organized around temporal body dynamics and relations with objects, environments, and other people, moving beyond isolated task-specific pipelines.The section identifies reusable temporal priors, multimodal interfaces, and general relational models as the main direction of progress.Source: §5 · p. 16

5.1 Kinematic Dynamics

  • Kinematic dynamics covers scalable motion modeling and human video animation, linking structured temporal representations with photorealistic realization while preserving appearance and temporal coherence.The subsection distinguishes modeling body dynamics as temporal signals from realizing them in pixel space.Source: §5.1 · p. 16

5.1.1 Scalable Motion Modeling

  • Scalable motion models establish transferable motion-language interfaces, expand across data and model scale, and unify understanding, generation, control, and refinement across observations and tasks.However, scaling only partially captures physical validity, temporal causality, and subtle human variation, making transferable continuous kinematics the central unresolved goal.Source: §5.1.1 · pp. 16, 18

5.1.2 Human Video Animation

  • Human video animation specializes video foundation models for audio-, pose-, motion-, and multimodal-conditioned generation, but realistic frames do not guarantee temporally faithful, anatomically consistent, long-horizon dynamics.Unified models increasingly co-model video and structured motion, while precise correspondence between driving bodies and generated people remains necessary.Source: §5.1.2 · pp. 18–19

5.2 Interaction Modeling

  • Interaction modeling expands from local human-object relations to scene compatibility and social coordination through semantic knowledge, multimodal reasoning, and reusable generative priors.Its three forms are human-object interaction, human-scene interaction, and social interaction.Source: §5.2 · p. 19

5.2.1 Human-Object Interaction

  • Human-object interaction models increasingly transfer relational knowledge across objects and tasks, but semantic plausibility can still miss contact, state transitions, affordances, and physical feasibility.Promising hybrids combine large pretrained models with geometric optimization, explicit affordances, and physics-based control.Source: §5.2.1 · pp. 19–20

5.2.2 Human-Scene Interaction

  • Human-scene interaction treats environments as persistent behavioral constraints, combining open-vocabulary scene use, scene-grounded motion, and language interfaces for unified understanding and generation.Language-level scene semantics still require alignment with the geometric and functional details governing feasible human activity.Source: §5.2.2 · p. 21

5.2.3 Social Interaction

  • Social interaction models reciprocal timing, participant attribution, multi-person conditioning, and online audiovisual response, but scale alone does not ensure reciprocity, adaptation, cultural understanding, or calibrated uncertainty.Future progress requires participant-aware temporal memory and explicit mutual-influence modeling.Source: §5.2.3 · pp. 21–22

6 Human World Simulation and Embodied Agency

This section frames situated human-centric intelligence through world simulation and embodied agency, distinguishing perceptual generation from actionable planning and reusable physical control. It organizes embodied agency around generalist humanoid control and human-to-agent skill transfer, while highlighting persistent gaps in causality, actionability, and embodiment alignment.

6.1 World Simulation

  • World simulation connects human behavior to evolving environments, separating human-centered world generation from actionable world planning that supports decisions and predicts action-dependent futures.This distinction separates visually plausible simulation from models whose internal variables or outputs can support embodied action.Source: §6.1 · p. 22, Figure 10 · p. 24

6.1.1 Human-Centered World Generation

  • Human-centered world generation adapts video priors to human motion, interaction, goals, scene views, and body dynamics, but its outputs remain perceptual rather than executable.Progress requires persistent world state, causal response, and long-horizon controllability beyond visually plausible continuation.Source: §6.1.1 · pp. 22–23

6.1.2 Actionable World Planning

  • Actionable world planning conditions predictions on actions, jointly models hand actions and videos, transfers interaction knowledge to robots, and makes prediction part of action selection.Decision utility additionally requires intervention responses, separation of consequential changes from incidental variation, and calibrated uncertainty.Source: §6.1.2 · p. 24

6.2 Embodied Agency

  • Embodied agency comprises generalist humanoid control and human-to-agent skill transfer, increasingly connected through multimodal interfaces, behavioral pretraining, and large-scale human experience.The two directions convert human-centered knowledge and experience into executable behavior across physical embodiments.Source: §6 · p. 22, §6.2 · p. 24

6.2.1 Generalist Humanoid Control

  • Generalist humanoid control is shifting from isolated skills toward reusable motor priors and foundation-style policies that connect perception, language, and action.Broad generality still requires physical reliability, and it remains unclear how far shared policies can generalize without weakening execution stability.Source: §6.2.1 · pp. 24–25

6.2.2 Human-to-Agent Skill Transfer

  • Human-to-agent skill transfer uses human video as scalable supervision, bridging missing robot actions through latent action abstractions, embodiment alignment, and motion retargeting.Human observations provide broader coverage than costly, narrow robot trajectories, but transferability across bodies, environments, and control spaces remains uncertain.Source: §6.2.2 · pp. 25–26

7 Datasets, Benchmarks, and Metrics

This section reviews representative datasets, benchmarks, and commonly used metrics that support model development, evaluate capabilities, and reveal limitations in current evaluation practices.

7 Datasets, Benchmarks, and Metrics

  • Representative datasets and benchmarks define what human-centric models learn from and how their capabilities are evaluated.Commonly used metrics further organize evaluation, while existing practices remain incomplete.Source: §7 · p. 26

7.1 Datasets and Benchmarks

This section organizes human-centric datasets and benchmarks by four human-context groups—subjects, dynamics, interactions, and embodiment—while tracing a shift from observation and recognition toward language-grounded, physically situated, and embodied evaluation. Across groups, resources broaden capabilities but remain constrained by limited diversity, physical grounding, long-horizon context, transfer, and governance.

7.1 Datasets and Benchmarks

  • Datasets provide development observations and annotations, while benchmarks standardize tasks and protocols across human subjects, dynamics, interactions, and embodiment.The taxonomy groups heterogeneous resources by the primary human context they support rather than solely by modality or task.Source: §7.1 · p. 26

7.1.1 Human Subject Resources

  • Human subject resources progress from visual and multisensory observation to identity correspondence and renderable 3D geometry, increasingly testing language grounding and physically grounded recovery.They cover appearance, multisensory sensing, cross-condition identity matching, and deformable or animation-ready human assets, while facing controlled collection, privacy, diversity, and annotation constraints.Source: §7.1.1 · p. 27

7.1.2 Human Dynamics Resources

  • Human dynamics resources expand from video generation and behavior understanding to structured motion, gait, sport, and virtual try-on, but evaluation remains limited for subtle motion, causal reasoning, physical grounding, and transfer.Many benchmarks remain label- or template-driven, while kinematic resources lack long-horizon intent, social interaction, and reliable transfer from synthetic or mocap data to in-the-wild motion.Source: §7.1.2 · pp. 28–29

7.1.3 Human Interaction Resources

  • Human interaction resources progress from egocentric procedural activities to human-object, human-scene, and social interaction data, emphasizing continual assistance, contact geometry, scene affordances, and coupled behavior.Persistent gaps include long-tail and causal annotations, object and force coverage, environment diversity, multi-person interaction, privacy, cultural coverage, long-horizon context, and intent annotation.Source: §7.1.3 · pp. 30–31

7.1.4 Human Embodiment Resources

  • Human embodiment resources connect scalable human experience to robot learning and evaluate dynamically feasible, human-like behavior through physical control and behavior-oriented assessment.Transfer remains constrained by embodiment mismatch, sparse physical sensing and failures, limited environment and morphology diversity, sim-to-real gaps, and underdeveloped manipulation, recovery, and safety evaluation.Source: §7.1.4 · pp. 31–32

7.2 Metrics

The section organizes human-centric evaluation into five metric families—reconstruction-, semantic-, generative-, interaction-, and efficiency-centric—covering outputs from geometric structure and meaning to physical behavior and deployment cost. Together, these metrics evaluate fidelity, semantics, diversity, temporal and physical validity, task performance, human judgment, and computational practicality.

7.2 Metrics

  • Five metric families organize evaluation by target: reconstruction, semantic, generative, interaction, and efficiency, with shared notation and formulations summarized in Tables 9 and 10.The taxonomy spans geometric agreement, semantic meaning, generated-output quality, relational and physical behavior, and computational cost.Source: §7.2 · p. 32, Table 9 · p. 33, Table 10 · p. 36

7.2.1 Reconstruction-Centric Metrics

  • Reconstruction-centric metrics progress from articulated joint and mesh accuracy to surface geometry and image-rendering fidelity.Representative measures include MPJPE, PA-MPJPE, PVE/MPVPE, PCK, AUC, F-score, Chamfer Distance, depth errors, PSNR, SSIM, LPIPS, and perceptual or warping distances.Source: §7.2.1 · pp. 32–33

7.2.2 Semantic-Centric Metrics

  • Semantic-centric metrics evaluate recognition, retrieval and verification, and cross-modal alignment through correctness, ranking, overlap, embedding similarity, and generated-language quality.The measures cover accuracy and Top-k accuracy, precision/recall/F1, AP/mAP, IoU/mIoU, Rank-k and Recall@K, CLIP- and DINO-based similarities, and BLEU-family language scores.Source: §7.2.2 · pp. 33–34

7.2.3 Generative-Centric Metrics

  • Generative-centric metrics jointly assess fidelity, diversity and coverage, temporal dynamics, audio-visual synchronization, and identity or appearance consistency.FID, FVD, KID, and FGD measure distributional fidelity; diversity, MultiModality, coverage, and nearest-neighbor measures address variation; trajectory, smoothness, synchronization, and identity metrics capture temporal, audio-visual, and subject consistency.Source: §7.2.3 · pp. 34–35

7.2.4 Interaction-Centric Metrics

  • Interaction-centric metrics evaluate relational and physical validity, embodied task execution, and subjective quality from contact and collision through task completion and perceived naturalness.They include contact precision/recall/F1, contact distance, penetration and collision measures, physics-based scores, success and completion rates, return and reward, planning errors, robot-suite metrics, preference rates, ratings, and naturalness tests.Source: §7.2.4 · p. 35

7.2.5 Efficiency-Centric Metrics

  • Efficiency-centric metrics assess practical deployment through inference speed, latency, throughput, model size, memory, parameter count, FLOPs, and computational complexity.FPS, runtime, and frequency measure processing capacity, while resource metrics relate model footprint and computation to hardware-limited deployability.Source: §7.2.5 · p. 35

8 Open Challenges and Future Directions

The section identifies six directions for advancing human-centric intelligence beyond specialized models: scalable and trustworthy data, unified representations, physical grounding, embodied world models, integrated evaluation, and efficient deployment.

8.1 Scalable and Trustworthy Human Data

  • Synthetic data can scale human-centric pretraining, but future work must separate data quantity from quality while preserving provenance, consent, and representation of underrepresented populations.Synthetic data improves recognition and generation when combined appropriately with real data, yet may also amplify generator artifacts and biases [78] [211].Source: §8.1 · pp. 38–39

8.2 Unified Human-Centric Foundation Models

  • Human-centric foundation models should learn shared representations across contexts while retaining modality-specific precision and demonstrating transfer with limited adaptation without negative transfer or catastrophic forgetting.Joint pretraining and cross-context alignment are proposed routes toward a shared representation space spanning observable humans, behavior, interaction, and embodied action [12] [178] [324].Source: §8.2 · p. 39

8.3 Physical Grounding for Next-Generation Human Representation Learning

  • Physical information should be central to human representation learning, using biomechanics, physics-informed learning, and differentiable simulation to improve validity in understanding, generation, interaction, and control.Evaluation should test physical validity rather than relying only on perceptual and kinematic similarity [35] [92] [122] [104] [166].Source: §8.3 · p. 39

8.4 Human-Centric World Models and Embodied Action Intelligence

  • Human-centric agents should connect causal world prediction with action selection, physical outcomes, and human responses while evaluating decision quality, motor execution, and preservation of human agency.Current video world models may be visually plausible without learning action consequences, while HumanCLAW and ComBodied Agents motivate closed-loop and agency-centered evaluation [293] [339] [361] [192] [62].Source: §8.4 · p. 39

8.5 Evaluation for Generalization and Integrated Capabilities

  • Evaluation should combine task-specific diagnostics with shared protocols for cross-context generalization, output consistency, planning, and closed-loop performance instead of relying on a single aggregate score.The proposed outcome is a structured evaluation profile spanning integrated capabilities [28] [323] [75] [92] [380].Source: §8.5 · p. 40

8.6 Efficient, Modular, and Deployable Human-Centric Systems

  • Efficient deployment may require modular agentic systems that route among foundation models, specialized human models, and external tools while managing latency, privacy, reliability, error propagation, and coherence.Such systems could support perception, garment modeling, motion generation, and video understanding through a shared semantic interface [209] [15] [184] [405] [444].Source: §8.6 · p. 40

9 Conclusion

The survey unifies human-centric intelligence through a six-level human context taxonomy and methodological review, while identifying incomplete cross-context consistency, physical grounding, and transfer to executable behavior as central challenges.

Contribution

  • The survey connects visual appearance, spatial geometry, kinematic dynamics, interaction modeling, world simulation, and embodied agency across observable subjects, dynamic actors, and situated agents.It uses this taxonomy to organize methodological foundations, advances, datasets, benchmarks, and evaluation metrics.Source: §9 · p. 40

Conclusion

  • The foundation-model era emphasizes scalable human data, reusable priors, multimodal interfaces, transferable capabilities, and connections among perception, generation, reasoning, and action.The survey characterizes this shift as broader than simply increasing model size.Source: §9 · p. 40

Open Challenges

  • Key challenges remain in preserving human consistency across contexts, obeying physical constraints, and transferring knowledge into executable behavior.These gaps limit progress toward human-centric intelligence that is scalable, trustworthy, physically grounded, and deployable.Source: §9 · p. 40

Related papers

Loading 2608.18184v1…