PaperlayerView exact revision on arXiv

Source-linked AI summary

The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook

Xinlei Yu, Zhangquan Chen, Yongbo He, Tianyu Fu, Guanting Dong, Cheng Yang, Chengming Xu, Yue Ma, Xiaobin Hu, Zhe Cao, Jie Xu, Guibin Zhang, Jiale Tao, Jiayi Zhang, Siyuan Ma, Kaituo Feng, Haojie Huang, Youxing Li, Ronghao Chen, Huacan Wang, Chenglin Wu, Zikun Su, Xiaogang Xu, Kelu Yao, Kun Wang, Chen Gao, Yue Liao, Ruqi Huang, Tao Jin, Zhucun Xue, Cheng Tan, Jiangning Zhang, Wenqi Ren, Yanwei Fu, Yong Liu, Yu Wang, Xiangyu Yue, Yu-Gang Jiang, Shuicheng Yan

arXiv:2604.02029v2cs.AI

TL;DR

Language-based models are still commonly framed around explicit token generation, although critical computation increasingly occurs in continuous latent space and existing reviews do not unify the broader literature. This survey defines the scope of latent space, traces its evolution, and organizes research by mechanism and ability, concluding that latent space is a broad computational paradigm with major interpretability challenges.

  • Problem

    Existing reviews mainly focus on latent reasoning, leaving latent space insufficiently unified across modalities, mechanisms, scenarios, and capabilities.

  • Method

    The survey synthesizes latent-space research through five sequential perspectives and a two-dimensional taxonomy spanning four mechanisms and seven abilities.

  • Results

    The survey shows latent space research has expanded from latent reasoning into a broader paradigm spanning language, vision, memory, collaboration, and embodied action.

  • Takeaways & Limitations

    Latent space may serve as a general computational and systems paradigm for next-generation intelligent language-based models.

  • Takeaways & Limitations

    Latent representations remain difficult to evaluate, control, and interpret directly because they are high-dimensional, distributed, and entangled.

Abstract

from arXiv · show

Latent space is rapidly emerging as a native substrate for language-based models. While modern systems are still commonly understood through explicit token-level generation, an increasing body of work shows that many critical internal processes are more naturally carried out in continuous latent space than in human-readable verbal traces. This shift is driven by the structural limitations of explicit-space computation, including linguistic redundancy, discretization bottlenecks, sequential inefficiency, and semantic loss. This survey aims to provide a unified and up-to-date landscape of latent space in language-based models. We organize the survey into five sequential perspectives: Foundation, Evolution, Mechanism, Ability, and Outlook. We begin by delineating the scope of latent space, distinguishing it from explicit or verbal space and from the latent spaces commonly studied in generative visual models. We then trace the field's evolution from early exploratory efforts to the current large-scale expansion. To organize the technical landscape, we examine existing work through the complementary lenses of mechanism and ability. From the perspective of Mechanism, we identify four major lines of development: Architecture, Representation, Computation, and Optimization. From the perspective of Ability, we show how latent space supports a broad capability spectrum spanning Reasoning, Planning, Modeling, Perception, Memory, Collaboration, and Embodiment. Beyond consolidation, we discuss the key open challenges, and outline promising directions for future research. We hope this survey serves not only as a reference for existing work, but also as a foundation for understanding latent space as a general computational and systems paradigm for next-generation intelligence.

1 Introduction

The survey frames latent space as a machine-native substrate whose importance extends beyond latent reasoning to multimodal and systems-level capabilities. It addresses this fragmented literature through a sequential five-part narrative and a two-dimensional taxonomy organized by mechanism and ability.

  • Motivation: Language-based models are increasingly studied as computing through continuous latent space rather than only human-readable token sequences.The survey connects this shift to reasoning, perception, memory, communication, and action.
  • Research gap: Research has expanded from latent reasoning into a broader paradigm spanning modalities, interaction settings, and design choices.Existing reviews mainly focus on latent or implicit reasoning, leaving a broader computational and systems perspective missing.
  • Organization: The survey follows five sequential questions covering Foundation, Evolution, Mechanism, Ability, and Outlook.These sections move from conceptual grounding and historical development to operation, capabilities, and future directions.
  • Taxonomy: Its technical synthesis uses Mechanism and Ability as complementary axes for comparing diverse latent-space methods.Mechanism covers Architecture, Representation, Computation, and Optimization, while Ability covers seven capability domains.
  • Scope and resources: The survey clarifies latent space against explicit or verbal space and latent spaces studied in generative visual models.It also provides figures, structured tables, links, and repositories to support research and community engagement.

2 Foundation: What is Latent Space?

Latent space is a continuous, machine-native representational space in language-based models, distinct from explicit token space and from visual generative-model latents. Its continuity and flexibility support richer computation and generalization, while reducing direct human evaluability and interpretability.

  • Concept: Language models map tokens into continuous internal representations that encode and manipulate information not verbalized at the token level.This latent space differs from explicit space, where inputs and outputs are expressed as discrete linguistic symbols.
  • Representational properties: Latent representations are machine-native, high-dimensional real-valued vectors that avoid direct human legibility and additional encoding or decoding overhead.Individual dimensions do not straightforwardly correspond to human-interpretable semantic, structural, or perceptual features.
  • Representational properties: Compared with explicit token sequences, latent space is continuous and flexible, avoiding discrete tokenization and linguistic rendering of semantic information.Explicit generation can introduce redundancy, representational transformation inefficiency, and sequential inefficiency, while tokenization can impose a quantization bottleneck.
  • Functional capabilities: Latent variables can preserve rich continuous information across computational steps and support multimodal structure and cross-domain generalization.The survey also identifies operability, expressiveness, scalability, and generalization among latent space’s functional capabilities.
  • Functional capabilities: Latent space makes granular human evaluation, control, and interpretation inherently difficult because its representations are high-dimensional, distributed, and entangled.This opacity complicates tracing transformations, explaining conclusions, and locating errors or misalignment.
  • Comparison with generative visual models: Language-model latent spaces focus on linguistic semantics, whereas visual generative-model latents preserve explicit spatial and temporal structure.Visual latents commonly organize image patches and video motion; language-model latents lack spatial topology or physical dynamics.

3 Evolution: How Did Latent Space Develop?

Latent-space research progressed from prototype demonstrations of continuous reasoning to theoretical systematization, multimodal and embodied expansion, and increasingly specialized architectures. Each stage resolved earlier questions while exposing new needs for broader evaluation and consolidation.

  • Prototype: The prototype stage established the feasibility of moving reasoning from discrete token sequences into continuous latent representations.Early work compressed reasoning or fed hidden states back as continuous thoughts, while recurrent and plug-in methods broadened the design space.
  • Prototype: The prototype stage lacked systematic explanations, comparisons with explicit CoT, and evaluation beyond isolated proof-of-concept demonstrations.These limitations motivated theoretical formalization, benchmark construction, and more principled technical design.
  • Formation: Formation-stage theory formalized superposition, parallelism, and recurrent-depth expressiveness, while technical work developed representation and optimization strategies.The stage remained primarily text-centered but began extending toward multimodal and embodied settings.
  • Formation: Early multimodal efforts demonstrated that latent-space methods could extend beyond textual reasoning across modalities and embodiment types.Visual latent tokens and related approaches provided initial evidence for a broader latent-space paradigm.
  • Expansion: Expansion diversified latent methods into visual, embodied, and multi-agent applications, but produced fragmentation across architectures, objectives, evaluation criteria, and latent interfaces.The resulting bottleneck shifted from finding applications to unifying the field.
  • Outbreak: The outbreak stage shifts the frontier toward consolidation through standardized interfaces, cross-modal evaluation, interpretability-efficiency alignment, and integration with agentic systems.Specialized architectures, latent communication, and latent mechanisms in VLA systems are becoming increasingly central within this broader transition.

4 Mechanism: How Does Latent Space Work?

The survey organizes latent-space mechanisms around how latent variables are instantiated, represented, computed through, and optimized. Its formalization contrasts token-space generation with generation augmented by a continuous latent channel.

  • Mechanism taxonomy: Mechanism classifies latent-space methods along Architecture, Representation, Computation, and Optimization.These axes organize where latent variables are introduced, how they are represented, how computation uses them, and when they are shaped.
  • Formalization: Standard autoregressive generation operates through a token-to-token interface, despite relying internally on continuous hidden states.The distinction is between internal continuous computation and externally expressed discrete generation.
  • Formalization: Latent-space methods add a continuous representation z to condition generation alongside the observable input x.The latent channel can encode global semantics, multimodal features, intermediate reasoning states, structural constraints, or other task-relevant factors.
  • Formalization: The taxonomy focuses on how latent space is instantiated and integrated into generation rather than merely whether latent variables are present.This mechanism-oriented perspective supports comparison through shared design principles.

4.1 Architecture

Architecture-level methods integrate latent computation into backbones, plug-in components, or auxiliary models. These designs range from native recurrent and iterative computation to modules that preserve the backbone while adding latent functions.

  • Architecture taxonomy: Architecture-driven methods are classified as Backbone, Component, or Auxiliary Model according to where latent space is integrated into the model.Backbones provide native latent capacity, components preserve the main skeleton, and auxiliary models supply supervision or intermediate features.
  • Backbone: Backbone methods embed latent computation in the primary generative architecture through recurrent, looping, recursive, or otherwise structured transitions over latent states.The backbone itself performs the latent operation without an additional component.
  • Backbone: Parameter-shared backbones reuse a smaller set of modules across repeated depth steps, while iterative backbones allocate variable or learnable computation through repeated hidden-state updates.Examples include fixed recurrent reuse, elastic loop counts, and Jacobi-style parallel refinement.
  • Backbone: Augmented backbones use architectural additions such as hierarchical encoder-decoders, compressed concept spaces, sparse attention mixtures, and low-rank attention.These designs broaden how backbone architecture can organize latent computation.
  • Backbone: Backbone-oriented methods improve efficiency through parameter reuse, flexibility through adaptive iterations, and architectural scope through augmentation.The survey presents this progression as a basis for more computation-aware and cognitively expressive generative systems.
  • Component: Component methods preserve the backbone while adding plug-in modules for constructing, transforming, aligning, controlling, or storing latent representations.Their five functional families are Generation, Projection, Alignment, Control, and Storage.
  • Component: Generation components synthesize intermediate latent objectives, subgoals, reasoning states, embeddings, activation features, or steering vectors for downstream decoding.Examples include encode-think-decode mechanisms, embedding prediction, sparse-feature directions, and MLP-generated guidance vectors.
  • Auxiliary Model: Auxiliary models guide the host model with supervision signals or intermediate features, supporting complex reasoning, multimodal understanding, and embodied decision-making.This paradigm introduces latent guidance externally rather than embedding it directly in the backbone.

4.2 Representation

The representation taxonomy classifies latent representations by their structural source and parameterization, yielding Internal, External, Learnable, and Hybrid paradigms. These paradigms range from parameter-free backbone activations to trainable and auxiliary representations for multimodal and specialized settings.

  • Foundation: Latent representations encode information in continuous, high-dimensional space rather than a fixed discrete vocabulary, increasing semantic expressivity.
  • Taxonomy: The taxonomy uses two orthogonal axes: whether representations are native to the backbone or externally generated, and whether they use fixed states or trainable modules.Their intersection yields four representation paradigms.
  • External: External representations inject signals from frozen, structurally independent auxiliary systems as conditioning inputs or supervision targets.This approach supports modality bridging without modifying backbone parameters.
  • Learnable: Learnable representations use dedicated trainable modules optimized end-to-end, enabling non-verbal modalities and high-bandwidth inter-agent collaboration.Unconstrained optimization can cause manifold overfitting and reduce zero-shot generalization, motivating regularization.
  • Hybrid: Hybrid representations first acquire task-specific latent states through learnable modules and then inject them as structured external conditioning signals.They target multimodal or domain-specialized settings where purely internal or purely learnable approaches leave bottlenecks unresolved.
  • Internal: Internal representations derive parameter-free aggregations of backbone activations, including hidden states, weighted embeddings, and key-value caches.These representations use endogenous activations from the standard forward pass and bypass the discrete vocabulary bottleneck.

4.3 Computation

Latent computation is organized by the operation performed on representations: Compressing, Expanding, Adapting, or Interleaving computation. Across these categories, methods trade explicit length, fixed budgets, or homogeneous token streams for denser, broader, selective, and heterogeneous computation.

  • Overview: Latent computation shifts part of inference away from fixed-depth, token-by-token generation toward more flexible, efficient, and scalable mechanisms.
  • Taxonomy: The taxonomy distinguishes Compressed, Expanded, Adaptive, and Interleaved operations according to their underlying computational behavior.The categories compare different ways of changing computation volume, allocation, or generation media.
  • Adaptive: Adaptive computation allocates resources conditionally across depth, width, semantic units, or control, spending more on difficult or uncertain cases.The category generalizes fixed-budget recurrent reasoning into input-conditioned computation.
  • Interleaved: Interleaved computation alternates symbolic outputs with latent computation or other heterogeneous media to support non-textual generation and module or agent coordination.This design can avoid verbalizing every intermediate step while connecting explicit-latent, language-vision, reasoning-memory, or planning-perception processes.
  • Compressed: Compressed computation maps verbose traces, internal states, or cross-modal features into denser latent representations while preserving information needed for downstream decoding.Its central objective is to reduce computational and memory overhead without eliminating inferential content.
  • Expanded: Expanded computation increases latent capacity along depth, width, or structure through recurrent refinement, parallel hypotheses, or richer topologies.These designs trade additional latent computation for stronger ability, faithfulness, or adaptability.

4.4 Optimization

Latent-space optimization operates at pre-training, post-training, or inference, with the optimized object shifting from model parameters to inference-time latent states. The survey distinguishes these stages by their supervision signals and search strategies for shaping latent representations and trajectories.

  • Overview: Latent optimization spans pre-training, post-training, and inference, where parameters are optimized during training while latent states or trajectories may be optimized at inference.
  • Pre-training: Pre-training learns latent capabilities jointly with model parameters from large-scale data, primarily using scalable autoregressive supervision with optional auxiliary or reinforcement objectives.The survey characterizes continuous-state autoregressive prediction as the dominant approach, with auxiliary losses shaping latent geometry.
  • Post-training: Post-training refines latent representations on pretrained models using supervision such as task losses, distillation, contrastive alignment, or reward-based feedback.A central design choice is whether latent variables receive explicit targets or are shaped implicitly through output-level objectives.
  • Inference: Inference-time optimization freezes model weights in the usual case and directly manipulates latent states as the optimization variable.The final output is generated conditioned on the optimized inference-time state.
  • Inference: Inference methods differ by search strategy: scaling selects reward-guided trajectories, optimization updates latent variables by gradients, and guidance imposes structural or contrastive constraints.

5 Ability: What Does Latent Space Enable?

Latent space enables language-based models to perform capabilities through continuous internal representations rather than explicit token-by-token verbalization. The survey organizes these abilities around implicit inference, compact traces, continuous refinement, branching paths, modal generalization, planning, self-inspection, and richer expression.

  • Reasoning: Latent reasoning performs deduction and relational computation through continuous internal representations instead of explicit token-by-token verbalization.
  • Reasoning: Latent reasoning supports implicit inference, compact traces, continuous refinement, branching paths, and modal generalization beyond text-only settings.Continuous thought vectors can encode multiple reasoning paths, while latent states can compress long chains and support iterative revision.
  • Reasoning: Parallel latent reasoning explores several candidate trajectories simultaneously, reducing wall-clock latency while maintaining quality.
  • Reasoning: Latent reasoning generalizes across linguistic, visual, heterogeneous, chemical, narrative, geometric, and temporal settings.
  • Planning: Latent planning supports controllable exploration, geometry-guided search, adaptive computation budgets, and sequential decision-making across recommendation, retrieval, tool use, and multimodal agents.Adaptive planning adjusts reasoning depth and termination according to input complexity, while latent trajectories support sustained state tracking and intermediate-decision optimization.
  • Modeling: Latent-space computation also enables richer expression and direct inspection of internal reasoning dynamics, with an associated efficiency–interpretability trade-off.Expressiveness analyses describe simultaneous representation of multiple search frontiers, while probing and visualization methods examine whether latent states encode genuine reasoning.

6 Outlook: What is Next?

The survey frames latent space as a machine-native substrate whose mechanisms and capabilities may reorient language-based intelligence beyond token-centric computation. It identifies major challenges in evaluability, controllability, and interpretability, and points toward theories and architectures that coordinate latent and explicit spaces.

  • Perspective: Latent space is presented as a primary computational substrate that reduces redundancy, discretization bottlenecks, sequential inefficiency, and semantic loss in verbalized computation.The survey characterizes this shift as moving from human-aligned generation toward machine-optimal computation.
  • Mechanism: The survey organizes latent-space progress around co-designed architecture, representation, computation, and optimization dimensions.This mechanism taxonomy emphasizes how latent variables are instantiated, operated through, and optimized.
  • Ability: Latent computation is reported to support reasoning, planning, modeling, perception, memory, communication, and embodiment across modalities, timescales, and agents.The survey presents latent space as a common substrate for coordinating heterogeneous information rather than merely replacing textual tokens.
  • Challenges: Latent trajectories remain difficult to evaluate because they are inaccessible to direct inspection, while fragmented benchmarks and inconsistent metrics impede fair comparison.The survey highlights missing mature protocols for supervision and evaluation of faithfulness, robustness, and internal consistency.
  • Challenges: Reliable latent manipulation is constrained by low controllability and the difficulty of mapping high-level semantic intentions onto precise, machine-actionable internal processes.The stated challenge extends beyond local steering to robust mechanisms connecting goals, safety requirements, and resource constraints with computation.
  • Future Directions: Future research should develop theories explaining latent computation and its relation to explicit space while making hidden processes evaluable, controllable, and interpretable.The proposed framework treats explicit language as an interface and latent space as an internal workspace for reasoning, memory, simulation, and planning.

7 Conclusion

The survey presents latent space as an increasingly unified paradigm for language-based models, organized through mechanism and ability taxonomies. It also emphasizes unresolved challenges in evaluating, controlling, and interpreting continuous hidden computation.

  • The survey reviews latent space through five perspectives: foundation, evolution, mechanism, ability, and outlook.These perspectives provide the paper’s overall synthesis of the field.
  • The survey argues that latent space has developed from early latent reasoning studies into a broader paradigm spanning language, vision, memory, collaboration, and embodied action.
  • Its taxonomy classifies latent-space research by four mechanisms and seven abilities.Mechanisms are architecture, representation, computation, and optimization; abilities are reasoning, planning, modeling, perception, memory, collaboration, and embodiment.
  • Internalizing more cognition into continuous hidden computation makes these processes harder to evaluate, control, and interpret.The paper therefore calls for stronger theory, benchmarks, supervision protocols, and transparent, controllable latent mechanisms.
  • The survey concludes that latent space has the potential to become a foundational principle for language-based models.It positions the survey as a foundation and reference for future research.
Loading 2604.02029v2…