PaperlayerView exact revision on arXiv

Source-linked AI summary

From Chain-of-Thought to Loops: Non-Autoregressive Latent Reasoning via Looped Transformers

Gerard Grau García, Arnau Padrés Masdemont, Niccolò Grillo, Jordi Ros-Giralt, Arash Behboodi, Victor Conchello Vendrell

arXiv:2610.11472v1cs.AI

TL;DR

Explicit chain-of-thought makes additional reasoning computation a sequential text-generation process, while many latent-reasoning methods still generate latent vectors left to right. LLoCoT iteratively refines a shared latent workspace and performs on par with Reasoning SFT on aggregate coding benchmarks, with lower reasoning and end-to-end latency.

  • Problem

    Explicit chain-of-thought ties additional reasoning computation to sequential natural-language generation, and most latent-reasoning approaches retain left-to-right dependencies among latent states.

    Source: §1 · p. 1

  • Method

    LLoCoT repeatedly applies a shared transformer to refine a fixed latent workspace, then jointly generates latent thoughts to condition autoregressive answer decoding.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4

  • Results

    Across four code-generation benchmarks, LLoCoT performs on par with Reasoning SFT and outperforms the base model, answer-only SFT, and NF-CoT on aggregate, while achieving the lowest measured end-to-end latency.

    Source: §5.2 · p. 9

  • Takeaways & Limitations

    The results suggest recurrent latent-workspace refinement can replace autoregressive latent-thought generation while retaining competitive aggregate coding-benchmark performance.

    Source: §5.2 · pp. 8–9

    Experiments use a 1.7B-parameter backbone and code-generation tasks, leaving extension to larger models and other reasoning domains unclear.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

Chain-of-thought (CoT) reasoning often improves language-model performance by giving models additional computation before answering. However, explicit CoT expresses this computation as a sequence of autoregressively generated tokens. Latent reasoning replaces these tokens with compact continuous states, but most autoregressive latent-reasoning methods retain a left-to-right dependency among latent vectors. We introduce LLoCoT: a looped latent-reasoning framework that replaces left-to-right latent generation with iterative refinement of a compact latent workspace. A shared transformer is reapplied for a small number of refinement iterations, jointly updating the latent slots based on the prompt and the evolving workspace state. Using the refined state, a probabilistic head predicts a distribution from which latent tokens are sampled in parallel and used to condition an autoregressive decoder for answer generation. Training uses continuous representations derived from explicit CoT together with a final-answer prediction loss and likelihood-based supervision of the latent states. Across HumanEval and MBPP, LLoCoT achieves the highest mean among the evaluated methods, performing on par in accuracy with Reasoning SFT, our explicit-CoT baseline, while outperforming the base model, answer-only SFT and NF-CoT. Relative to Reasoning SFT, LLoCoT reduces time to the first answer token by approximately $36\times$ and reasoning-phase latency by approximately $42\times$, while increasing end-to-end throughput by $9.2\%$. This design replaces serial thought generation with parallel latent-slot refinement while retaining probabilistic latent modeling and autoregressive answer decoding.

1 INTRODUCTION

Explicit chain-of-thought adds computation through sequentially generated text, while latent reasoning avoids verbalizing every step but often retains left-to-right latent generation. LLoCoT replaces that dependency with iterative refinement of a shared latent workspace and performs competitively on coding benchmarks with lower reasoning latency.

1 INTRODUCTION

  • Across several coding benchmarks, LLoCoT performs on par with Reasoning SFT while outperforming the base model, answer-only SFT, and NF-CoT on aggregate.The authors also report that iterative refinement consistently improves over NF-CoT.Source: §1 · p. 2
  • LLoCoT replaces autoregressive latent-thought generation with recurrent refinement of a fixed-size latent workspace.A shared transformer refines the workspace, and the final Gaussian head jointly generates latent thoughts for autoregressive answer decoding.Source: §1 · p. 2, Figure 1 · p. 2
  • LLoCoT substantially shortens reasoning latency relative to textual and autoregressive latent reasoning, with the lowest end-to-end latency among evaluated methods.Its parallelizable refinement brings time to the first answer token close to direct answer generation.Source: §1 · p. 2

2 RELATED WORKS

Prior latent-reasoning work moves intermediate computation from text into continuous states, with approaches differing in how those states are generated. Looped transformers provide a related approach based on repeatedly updating a shared hidden state.

2 RELATED WORKS

  • Latent-reasoning methods avoid verbalizing every intermediate step; NF-CoT uses VAE- and flow-based continuous thoughts but generates them autoregressively, whereas LLoCoT jointly refines a fixed workspace.LaDiR instead uses diffusion to iteratively denoise blocks of latent thoughts.Source: §2 · p. 3
  • Looped transformers reuse a shared block to iteratively manipulate hidden states, with prior work exploring algorithmic procedures, structured reasoning, and general-purpose language modeling.Reported structured reasoning tasks include Sudoku, maze planning, and ARC-AGI.Source: §2 · p. 3

3 LLOCOT: LATENT LOOPED CHAIN-OF-THOUGHT

LLoCoT uses a fixed latent workspace, repeatedly updates it with a shared transformer, and uses the resulting latent thoughts to condition answer generation. Training targets come from explicit reasoning traces, while inference uses prompt-conditioned refinement and joint latent generation.

3 LLOCOT: LATENT LOOPED CHAIN-OF-THOUGHT

  • LLoCoT replaces left-to-right latent generation with iterative refinement of learned workspace slots, whose final latent thoughts condition an autoregressive answer decoder.The main configuration uses K = 64 slots, hidden width 2048, latent width 2560, and R = 2 refinement iterations.Source: §3 · p. 3

3.1 NF-COT-DERIVED TARGETS

  • A separately trained, frozen VAE encodes each explicit reasoning trace as K latent vectors that jointly summarize the complete trace rather than individual tokens or reasoning steps.A target-space flow adapts the VAE representation for LLM-facing latent training targets; both components are training-only.Source: §3.1 · pp. 3–4

3.2 LOOPED HIDDEN-STATE REFINEMENT AND MEAN FEEDBACK

  • The shared transformer updates all K slots in parallel at each iteration, while the iterations recur sequentially over the evolving workspace.Non-final Gaussian means are projected back into the workspace to guide later refinement, without adding sampling noise to that trajectory.Source: §3.2 · p. 4

3.3 FINAL LATENT THOUGHTS, TRAINING OBJECTIVE, AND INFERENCE

  • Training combines final latent likelihood, auxiliary supervision of non-final predictions against the same target, and autoregressive answer loss.For R = 1, intermediate supervision is defined as zero; the target VAE representation is detached from gradients.Source: §3.3 · p. 5
  • At inference, the prompt alone drives refinement, after which the final Gaussian head generates all latent thoughts jointly for answer decoding.The selected latent thoughts are projected into the decoder dimension; target construction, intermediate supervision, the VAE, and target-space flow are absent at inference.Source: §3.3 · p. 5

4 RESULTS

Across four coding benchmarks, LLoCoT has the highest mean and performs on par with Reasoning SFT. Its ablations show that depth alone does not guarantee better performance, while its looped refinement sharply reduces inference latency.

4.1 ACCURACY RESULTS

  • 63.6 is LLoCoT’s highest four-benchmark mean on HumanEval, HumanEval+, MBPP, and MBPP+, 1.6 points above NF-CoT; it is on par with Reasoning SFT.LLoCoT’s mean is 6.3 points above the base model and 0.9 points above Base SFT.Source: Table 1 · p. 6, §4.1 · p. 6, §4 · p. 6

4.2 SELECTED ABLATIONS

  • The strongest four-benchmark mean occurs at a relatively small loop depth; deeper configurations do not consistently improve either benchmark family.Performance is not monotonic as the number of causal no-feedback refinement loops increases.Source: §4.2 · p. 6, Table 2 · p. 7
  • Combining mean feedback with auxiliary intermediate supervision yields the strongest result, improving the no-feedback baseline while preserving performance across both benchmark families.Directly conditioning the answer decoder on the predicted latent mean causes a large performance drop, while auxiliary supervision alone does not improve the overall mean.Source: Table 3 · p. 7, §4.2 · p. 7

4.3 INFERENCE EFFICIENCY

  • 0.09 seconds is LLoCoT’s time to first answer token, versus 3.17 seconds for Reasoning SFT and 2.01 seconds for NF-CoT.Its 0.06-second refinement phase is over 40× faster than Reasoning SFT’s textual reasoning phase and over 30× faster than NF-CoT’s latent generation. LLoCoT matches NF-CoT’s peak memory at 6.3 GiB but has higher analytical FLOPs than the other reasoning methods; the reported wall-clock advantage comes primarily from parallelizable refinement.Source: Table 4 · p. 8, §4.3 · p. 8

5 LIMITATIONS AND CONCLUSIONS

LLoCoT uses recurrent refinement to improve reasoning efficiency, with benchmark performance on par with Reasoning SFT. The conclusion also identifies limits in model scale, task scope, resource use, and dynamic adaptation.

5.1 LIMITATIONS

  • Experiments use a 1.7B-parameter backbone and code-generation tasks; workspace size and refinement depth must be selected in advance, and LLoCoT uses more peak memory and analytical FLOPs.The reported speedup comes from replacing serial reasoning with more parallelizable computation, not from reducing total computational cost.Source: §5.1 · p. 8

5.2 CONCLUSIONS

  • Across four code-generation benchmarks, LLoCoT performs on par with Reasoning SFT and outperforms the base model, answer-only supervised fine-tuning, and NF-CoT on aggregate.It has higher point estimates than NF-CoT on every benchmark and the lowest measured end-to-end latency.Source: §5.2 · p. 9

AI USE STATEMENT

  • Generative AI tools assisted manuscript drafting and editing and code writing; an author checked reported values, reviewed and tested assisted code, and takes responsibility for the final content.All reported experimental values were recomputed from logged outputs.Source: AI USE STATEMENT · p. 9

ETHICS STATEMENT

  • Latent reasoning does not expose intermediate thoughts, which may reduce interpretability and make reasoning harder to inspect or monitor.The paper recommends external evaluation, testing, and appropriate safety controls for deployment.Source: ETHICS STATEMENT · p. 9

REPRODUCIBILITY STATEMENT

  • The paper describes its methods and evaluation protocols; a repository is being prepared with implementation and reproduction materials, and licensed checkpoints or model links may also be provided.The planned repository contents include scripts, configurations, dependencies, experiment metadata, and documentation.Source: REPRODUCIBILITY STATEMENT · p. 9

A IMPLEMENTATION DETAILS

The appendix specifies the NF-CoT-interface VAE used to produce fixed continuous targets and reports its configuration and training. LLoCoT uses the cached posterior means while keeping the VAE encoder frozen.

A IMPLEMENTATION DETAILS

A.1 TARGET VAE

  • The continuous targets come from a transformer-based VAE using Qwen/Qwen3-1.7B-Base, with a causal encoder, K = 64 learned query slots, and D = 2560-dimensional latents.The VAE is not prompt-conditioned and does not use block segmentation.Source: §A.1 · p. 11
  • The VAE objective uses β = 0.01, and training runs for 180,030 optimizer steps with an effective batch size of 32.Training uses AdamW with learning rate 10^-5, two GPUs, bfloat16 precision, and a maximum sequence length of 3119.Source: §A.1 · p. 11
  • Latent extraction stores deterministic posterior means as fixed targets; the VAE encoder remains frozen, and Gaussian dequantization noise with standard deviation 0.3 is added during LLoCoT training.The VAE decoder is not used during LLoCoT training or inference.Source: §A.1 · p. 11

A.2 LLOCOT ARCHITECTURE

LLoCoT reapplies its transformer to learned workspace slots, predicts latent distributions, and feeds intermediate means back between refinement loops. A target-space flow adapts the targets during training but is absent at inference.

A.2 LLOCOT ARCHITECTURE

  • The answer model uses K = 64 loop slots, D = 2560 latent thoughts, and R = 2 refinement loops, reapplying the full transformer backbone at each loop with causal attention.The backbone has hidden width 2048, 28 layers, 16 attention heads, and 8 key–value heads.Source: §A.2 · p. 11
  • A shared diagonal Gaussian head maps each slot independently to the mean and scale of one final latent thought.Means and log-scales are soft-clipped with clipping value 4.0; scales use softplus with a minimum of 10^-5.Source: §A.2 · p. 11
  • After each non-final loop, the intermediate predicted mean is projected into the hidden dimension and added to the next loop's hidden states.The projection is shared with answer conditioning, and the intermediate Gaussian objective has weight 0.25 with stop-gradient targets.Source: §A.2 · p. 11
  • The target-space flow uses five causal affine blocks and alternates latent-slot order between identity and reversal; it is trained as an adapter and not run at inference.Source: §A.2 · p. 11

A.3 PARAMETER STATUS AND TRAINING STAGES

Training freezes the VAE encoder and uses a two-stage schedule that first freezes then unfreezes the Qwen backbone. The reported generation setup samples final latent thoughts jointly before autoregressive answer decoding.

A.3 PARAMETER STATUS AND TRAINING STAGES

  • The VAE encoder stays frozen, while the target-space flow is trained jointly; Stage 1 freezes the Qwen backbone and trains loop modules, then Stage 2 initializes from Stage 1 and unfreezes it.The intermediate latent target is detached for the auxiliary objective.Source: §A.3 · pp. 11–12

A.4 OPTIMIZATION

  • Both LLoCoT stages use an effective global batch size of 64; Stage 1 runs 1,563 optimizer steps and Stage 2 runs 45,222 steps.Stage 1 uses learning rate 10^-4 and Stage 2 uses 5 × 10^-5; the loss weights are λNF = 1, λCE = 1, and αintermediate = 0.25.Source: §A.4 · p. 12

A.5 GENERATION

  • All final latent thoughts are generated jointly, while answer generation remains autoregressive; accuracy evaluations use n = 16 completions per task.Latent sampling uses temperature 0.9 and CFG scale 1.0; answer generation uses temperature 0.6 and top-p = 1.0.Source: §A.5 · p. 12

B.1 BENCHMARK AND DECODING PROTOCOL

The appendix evaluates four coding benchmarks using fixed task sets and a shared evaluator, and specifies protocols for decoding, failure handling, and uncertainty reporting.

B.1 BENCHMARK AND DECODING PROTOCOL

  • Evaluation covers HumanEval, HumanEval+, MBPP, and MBPP+ using fixed task sets, a shared answer-extraction procedure, and the benchmark evaluator.Prompt rendering follows model-specific protocols.Source: §B.1 · p. 12
  • Stochastic evaluations generate 16 completions per task; answer decoding uses temperature 0.6, top-p 1.0, and a 2048-token limit.Native-thinking models have an 8192-token reasoning limit; generated-latent NF-CoT and LLoCoT use latent temperature 0.9 and CFG scale 1.0.Source: §B.1 · p. 12

B.2 FAILURE HANDLING AND INTEGRITY CHECKS

  • Invalid, empty, truncated, malformed, rejected, or failed completions remain in the fixed 16-sample denominator and count as zero.Source: §B.2 · p. 12
  • Missing or duplicate records, invalid task identities, corrupted evaluator artifacts, or inconsistent outputs block result promotion rather than being repaired or replaced.Source: §B.2 · p. 12

B.3 PASS@1 AND CONFIDENCE INTERVALS

  • Reported pass@1 averages per-task success rates across 16 samples, while stochastic accuracy entries include symmetric approximate 95% fixed-task sampling intervals.The intervals quantify uncertainty over the fixed benchmark tasks, not task selection, training seeds, or model selection.Source: §B.3 · p. 13
  • The main-results mean is the unweighted arithmetic mean of four benchmark point estimates and receives no additional confidence interval.Source: §B.3 · p. 13

B.4 EVALUATION RECORDS AND REPRODUCIBILITY

The appendix records evaluation metadata and defines the conditions and endpoints used for inference-efficiency measurements.

B.4 EVALUATION RECORDS AND REPRODUCIBILITY

  • Evaluation records include benchmark and task identities, stable seeds, prompt variants, decoding parameters, checkpoints, evaluator outputs, and completion status.Promotion requires complete expected records, unique sample identities, valid evaluator artifacts, and complete point-estimate and interval artifacts.Source: §B.4 · p. 13

B.5 INFERENCE-EFFICIENCY PROTOCOL

  • Efficiency measurements use 128 fixed prompts, batch size one, and a single NVIDIA H100 80GB HBM3 GPU in bfloat16.Each prompt is measured three times after ten warm-up runs, yielding 384 measured records per endpoint; warm-up, tokenization, sampling, network, and benchmark-evaluation time are excluded.Source: §B.5 · p. 13
  • Reasoning-phase endpoints are the first </think> boundary for Reasoning SFT and serial latent generation or looped latent refinement for NF-CoT and LLoCoT.Conditioning or latent selection and answer decoding are measured separately when applicable.Source: §B.5 · p. 13
  • Time to first answer token runs from model execution start to that token, while end-to-end latency includes measured reasoning, conditioning, and answer-generation phases.Peak allocated memory is recorded during each run; analytical FLOPs are deterministic estimates.Source: §B.5 · p. 13
  • Forward-pass counts are averaged over measured prompt–run records and may be non-integer; reasoning or latent calls and total transformer calls are reported separately.These counts describe invocation structure and do not replace analytical FLOP estimates.Source: §B.5 · p. 13
Loading 2610.11472v1…