Source-linked AI summary
Show, Don't Tell: Morphing Latent Reasoning into Image Generation
Harold Haodong Chen, Xinxiang Yin, Wen-Jie Shu, Hongfei Zhang, Zixin Zhang, Chenfei Liao, Litao Guo, Qifeng Chen, Ying-Cong Chen
TL;DR
T2I systems often lack dynamic self-refinement, and explicit fixed-step reasoning can lose information and add computational overhead. LatentMorph performs adaptive reasoning in continuous latent spaces through lightweight components that guide generation without textual chain-of-thought. It improves fidelity and abstract reasoning while reducing inference costs and aligning invocation with human evaluators.
Problem
Existing T2I generators have limited deliberation and self-refinement, while explicit fixed-step reasoning introduces information loss, inefficiency, and cognitive mismatch.
Method
LatentMorph interleaves latent reasoning with generation using visual-memory condensation, latent translation, control shaping, and an adaptive RL-trained invoker.
Results
LatentMorph improves Janus-Pro by 16.0% on GenEval and 25.3% on T2I-CompBench, outperforms explicit paradigms on WISE and IPV-Txt, and reduces inference time and token consumption by 44.3% and 51.0%.
Takeaways & Limitations
Continuous latent reasoning supports adaptive self-refinement while balancing generation fidelity and efficiency across reasoning-augmented T2I settings.
Takeaways & Limitations
LatentMorph optimizes Janus-Pro post-training, inherits the base model’s safety guardrails and biases, and may be misused for misleading photorealistic content or deepfakes.
Abstract
from arXiv · showhide
Text-to-image (T2I) generation has achieved remarkable progress, yet existing methods often lack the ability to dynamically reason and refine during generation--a hallmark of human creativity. Current reasoning-augmented paradigms most rely on explicit thought processes, where intermediate reasoning is decoded into discrete text at fixed steps with frequent image decoding and re-encoding, leading to inefficiencies, information loss, and cognitive mismatches. To bridge this gap, we introduce LatentMorph, a novel framework that seamlessly integrates implicit latent reasoning into the T2I generation process. At its core, LatentMorph introduces four lightweight components: (i) a condenser for summarizing intermediate generation states into compact visual memory, (ii) a translator for converting latent thoughts into actionable guidance, (iii) a shaper for dynamically steering next image token predictions, and (iv) an RL-trained invoker for adaptively determining when to invoke reasoning. By performing reasoning entirely in continuous latent spaces, LatentMorph avoids the bottlenecks of explicit reasoning and enables more adaptive self-refinement. Extensive experiments demonstrate that LatentMorph (I) enhances the base model Janus-Pro by $16\%$ on GenEval and $25\%$ on T2I-CompBench; (II) outperforms explicit paradigms (e.g., TwiG) by $15\%$ and $11\%$ on abstract reasoning tasks like WISE and IPV-Txt, (III) while reducing inference time by $44\%$ and token consumption by $51\%$; and (IV) exhibits $71\%$ cognitive alignment with human intuition on reasoning invocation.
1. Introduction
Existing T2I systems have limited explicit deliberation and self-refinement, while reasoning-augmented methods often impose discrete, fixed-step thought cycles. LatentMorph addresses this gap with adaptive latent reasoning and reports gains in fidelity, abstract reasoning, efficiency, and cognitive alignment.
- Prior T2I generators primarily map text to pixels, with limited explicit deliberation or self-refinement during generation.
- Fixed-step explicit chain-of-thought introduces information loss, decode–encode inefficiency, and mismatch with continuous human judgment.
- LatentMorph interleaves implicit reasoning with generation to dynamically monitor visual states and refine outputs without explicit textual chain-of-thought.
- Its condenser, translator, shaper, and RL-trained invoker summarize visual states, convert latent thoughts into guidance, steer image tokens, and adaptively schedule reasoning.
- 16.0% on GenEval and 25.3% on T2I-CompBench improve Janus-Pro, while inference time and token consumption decrease by 44.3% and 51.0%.
- 15.6% and 11.3% gains on WISE and IPV-Txt accompany 71.8% alignment with human evaluators for adaptive reasoning invocation.
2. Related Work
Related work incorporates LLM-style reasoning into image generation through external-loop and internal-loop paradigms, while latent-reasoning research carries deliberation in compact continuous representations.
- External-loop approaches couple an (M)LLM with a generator for planning, prompt optimization, verification, revision, or editing.
- Internal-loop approaches incorporate reasoning within the generation process, complementing the external-loop family described in related work.
- Latent-reasoning methods replace verbose textual rationales with compact latent representations to preserve richer intermediate deliberation and potentially improve efficiency.
3. Preliminary: Problem Formalization
The formalization models autoregressive image-token generation alongside multimodal understanding, with interventions that use partially generated images and prompts to refine subsequent generation.
- The UMM contains an autoregressive T2I generation branch UMMg and a multimodal understanding branch UMMu.
- Given prompt T, UMMg generates image-token sequence X autoregressively and decodes it into the final image.
- The objective maximizes an alignment reward R(Î, T) over a prompt distribution D.
- At intervention step k, partial tokens are decoded into an intermediate image, encoded with T, and passed to UMMu for reasoning and prompt refinement.
- Reason-before and reason-after methods intervene at k = 0 or k = |X|, whereas reason-while-generation methods intervene at fixed intermediate steps.
4. Methodology
LatentMorph interleaves implicit latent reasoning with autoregressive image generation, monitoring evolving states and intervening adaptively without explicit text decoding. Its condensers, invoker, translator, and shaper summarize visual history, decide when to reason, and inject guidance into generation.
- LatentMorph: Generating with Latent Reasoning: LatentMorph interleaves implicit reasoning directly into autoregressive generation, enabling adaptive self-refinement without decoding intermediate thoughts into text or images.The framework addresses fixed-step explicit reasoning and repeated decoding or re-encoding by operating in continuous latent spaces.
- Morphing Reasoning in Latent Space: The translator converts latent thoughts, long-term memory, and prompt information into control signals, while the shaper injects control tokens into the generator’s KV cache.This directly modifies subsequent token predictions while preserving guidance from the evolving generation context.
- Learning to Invoke Reasoning with Visual Memory: A short-term condenser compresses the latest w hidden states into compact visual memory for monitoring the evolving generation state.The resulting state representation supports the invoker’s decision process using recent generation dynamics.
- Learning to Invoke Reasoning with Visual Memory: Figure 3 shows adaptive invocations aligned with critical semantic transitions, correcting object omissions or counting errors seen in baseline generation.The case study uses blue stars to mark the reasoning interventions.
- Learning to Invoke Reasoning with Visual Memory: The invoker combines semantic consistency, prediction uncertainty, and temporal changes to choose between continuing generation and invoking reasoning.Its policy samples CONTINUE or REASON from the monitored state and is trained to balance task performance with reasoning efficiency.
- LatentMorph: Generating with Latent Reasoning: When reasoning is invoked, a long-term condenser summarizes global generation history into visual memory for latent reasoning in the understanding branch.The long-term condenser retains informative tokens and supports efficient processing of arbitrarily long histories without pixel-space decoding and re-encoding.
5. Experiments
LatentMorph is evaluated against ten baseline strategies across five benchmarks spanning general, compositional, and complex image-generation tasks. The experiments examine fidelity, abstract reasoning, efficiency, and alignment between adaptive invocation and human cognitive timing.
- Experimental Setup: LatentMorph is compared with ten baseline strategies across five benchmarks covering general, compositional, and complex generation.The evaluation includes Janus-Pro generation-only, reason-before/after-generation, and reason-while-generation strategies.
- Main Results: LatentMorph establishes superior fidelity in general and compositional generation, including a 7.28% margin over TwiG-RL in T2I-CompBench’s Non-Spatial category.It also exceeds T2I-Copilot and MILR by 8.86% and 8.49%, respectively, in the reported comparison.
- Effectiveness Analysis: Latent reasoning preserves visual information lost through textual decoding, improving performance on abstract and counterintuitive prompts such as those in IPV-Txt.The latent variant activates attention around subtle textures and lighting that are difficult to verbalize explicitly.
- Efficiency Analysis: LatentMorph reduces redundant computation by operating as a single-pass stream with compact visual memory and direct control injection.The adaptive invoker triggers reasoning only when necessary, avoiding repeated image decoding and explicit text tokenization.
- Cognitive Alignment: Invocation frequency correlates positively with task complexity, while user evaluations rate LatentMorph’s timing as more natural and necessary than fixed-step baselines.The reported average invocation frequency is 1.14 on GenEval, where prompts are simpler.
- Ablation Study: The learned invoker consistently outperforms random and fixed-step injection strategies, indicating that when to reason matters alongside how to reason.Sensitivity analyses also examine the condenser, translator, shaper, and invoker hyperparameters.
6. Conclusion
LatentMorph integrates implicit latent reasoning into autoregressive text-to-image generation, bypassing explicit textual decoding. Its visual-memory condensation, latent translation, adaptive control injection, and RL-based invoker jointly target fidelity, efficiency, and human-like intervention timing.
- Conclusion: LatentMorph integrates implicit latent reasoning into autoregressive text-to-image generation.The framework performs continuous reasoning and refinement within high-dimensional latent spaces.
- Conclusion: LatentMorph combines visual memory condensation, latent translation, adaptive control injection, and an RL-based invoker.The invoker dynamically aligns intervention with human-like cognitive rhythms by intervening only when necessary.
- Conclusion: The framework achieves a balance between generation fidelity and efficiency while providing a foundation for cognitively aligned and creatively robust visual generation.This conclusion follows the paper’s stated characterization of the proposed paradigm.
Impact Statement
LatentMorph is presented as reducing reasoning-augmented generation overhead and supporting more intuitive creative workflows, while inheriting the base model’s safety properties and biases.
- LatentMorph is aimed at enhancing text-to-image fidelity and efficiency.
- The framework is described as reducing computational overhead, supporting more energy-efficient and environmentally sustainable AI systems.
- LatentMorph is presented as a more intuitive tool for artistic expression and creative workflows through closer alignment with human cognitive processes.
- The method optimizes Janus-Pro’s latent states post-training without adding pre-training data, so it inherits the base model’s safety guardrails and potential biases.
- Photorealistic generation creates potential misuse risks, including misleading content and deepfakes.
A.3. Compatibility Analysis
LatentMorph is designed as a model-agnostic latent interface for autoregressive generators, using hidden-state readout and control-signal injection without changing generator weights.
- LatentMorph’s model-agnostic design supports external-loop and internal-loop autoregressive generator settings.
- The framework interacts with generators through latent-space read and write operations implemented by condensers and the shaper.
- Condensers use accessible hidden-state histories, with a projection into reasoning space and cross-attention that is independent of the generator’s layer or head count.
- The shaper injects control signals into attention for later steps while preserving positional consistency with Rotary Positional Embeddings.
- KV-cache-based generators are compatible with the injection method without structural changes or altering model weights.
- For external-loop generators, learnable adapters bridge the generator’s pixel/token space and the reasoning model’s semantic space.
B.2. More Details of Evaluation Benchmarks
The evaluation covers general alignment, compositionality, abstract reasoning, and cognitive alignment, including a controlled study comparing invocation decisions with human judgments.
- GenEval evaluates general alignment across diverse text-to-image prompts.
- T2I-CompBench evaluates compositional generation, while T2I-CompBench++ adds dimensions including numeracy.
- WISE tests world knowledge with abstract and complex prompts suited to reasoning-augmented image generation.
- IPV-Txt tests counter-intuitive physical phenomena and whether models capture physical laws rather than merely fitting training distributions.
- The user study used 10 experienced evaluators and 20 complex prompts, pausing generation every 64 tokens for binary Invoke-or-Continue judgments.
- Human Alignment measures the percentage of intervals where the model’s action matches the evaluators’ majority vote.
C.2. More Analysis of Condensers
Increasing either short-term or long-term latent-memory capacity improves generation quality, suggesting benefits from richer local monitoring and more comprehensive history summaries.
- Length of Short-Term Memory: Short-term memory lengths ns ∈ {2, 4, 8, 16} show consistent generation-quality improvements as memory length increases.
- Length of Short-Term Memory: Larger short-term memory buffers provide the invoker with more discriminative features for decision-making about local generation dynamics.
- Length of Long-Term Memory: Long-term memory lengths nl ∈ {4, 8, 16, 32} yield progressive performance gains as global memory size expands.
- Length of Long-Term Memory: A larger long-term memory alleviates the information bottleneck by giving the reasoning core a more granular summary of generation history.
C.3. More Analysis of Invoker
LatentMorph’s RL-trained invoker adaptively decides when to trigger reasoning, outperforming stochastic and fixed intervention schedules. Its monitoring design balances responsiveness and stability by combining an intermediate check window with multiple state signals.
- Invocation Timestep: Adaptive invocation consistently outperforms random injection and fixed schedules by triggering reasoning when semantic drift or high uncertainty is detected.The authors attribute this advantage to context awareness, avoiding wasted computation during trivial phases and missed critical turning points.
- Window Size of Checking: A check window of w = 64 provides the best trade-off between noisy local decisions at w = 32 and delayed detection at w = 128.The authors report that smaller windows capture token-prediction noise, whereas larger windows miss rapid compositional shifts.
- Multi-dimensional Signals: All four state-vector signals—semantic consistency, uncertainty, temporal dynamics, and stability—are essential for robust invocation decisions.Removing uncertainty or semantic consistency causes the largest performance drops, identifying them as the primary indicators.
C.4. More Analysis of Translator
Translator ablations show that latent thoughts require both visual history and prompt grounding for precise control. The shaper’s additive KV-cache injection preserves the original generation objective while incorporating refinement guidance.
- Control Signals: Latent thoughts alone are insufficient for precise control; visual context and textual grounding are both indispensable.The translator fuses latent thoughts with long-term visual memory and the original prompt embedding.
- Impact of Visual Memory: Removing long-term visual memory causes a marked performance decline because historical context aligns abstract reasoning with the visual content generated so far.Without this memory, control signals risk disrupting global visual consistency.
- Impact of Prompt Embedding: Omitting the prompt embedding produces inferior alignment scores because it removes the semantic anchor to the user’s original intent.The authors associate prompt grounding with preventing reasoning from drifting into unconstrained generation.
- Control Injection: Additive KV-cache injection outperforms direct replacement of prompt embeddings by steering attention while preserving the global generation objective.The shaper appends control tokens as soft modulation rather than overwriting the prompt representation.