PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking Before Constraining: A Unified Decoding Framework for Large Language Models

Ngoc Trinh Hung Nguyen, Alonso Silva, Laith Zumot, Liubov Tupikina, Armen Aghasaryan, Mehwish Alam

arXiv:2601.07525v2cs.CLcs.AI

TL;DR

LLMs can reason flexibly but lack guaranteed output structure, while early constraints may restrict reasoning. In-Writing separates reasoning from formatting in one call and improves performance across reasoning and classification benchmarks.

  • Problem

    LLMs lack guaranteed adherence to predefined output structures, limiting their applicability to schema-based extraction, structured question answering, and industrial use cases.

  • Method

    In-Writing decouples unconstrained reasoning from structured final-answer generation within a single inference call using a trigger token.

  • Results

    Across reasoning and classification benchmarks, In-Writing improves performance, with gains of up to 32% compared to CRANE.

  • Takeaways & Limitations

    The framework treats constrained decoding as a post-reasoning parser and corrector rather than requiring reasoning entirely within a constrained grammar.

  • Takeaways & Limitations

    The study does not optimize prompts for In-Writing's desired output format.

Abstract

from arXiv · show

Natural generation allows Large Language Models (LLMs) to produce free-form responses with rich reasoning, yet the lack of structure makes outputs difficult to verify. Conversely, constrained decoding ensures standardized formats but can inadvertently restrict reasoning capabilities by imposing constraints too early in the generation process. We propose a hybrid approach, namely In-Writing, that combines free-form reasoning and structured generation in a single call. The model first performs unconstrained reasoning and only applies structured decoding after a trigger token is generated, explicitly decoupling reasoning from formatting. We establish that our trigger-token strategies are able to virtually eradicate premature triggering, a failure mode in which constrained decoding interrupts on-going reasoning. Evaluations across diverse datasets covering classification and reasoning tasks demonstrate that our approach outperforms the state-of-the-art by achieving accuracy gains of up to 27% over natural generation. Our code are available at: https://github.com/Nokia-Bell-Labs/InWriting.

1 Introduction

LLMs support diverse applications but often lack structural guarantees, while grammar-constrained decoding can impair expressiveness and fluency. In-Writing addresses this tension by separating free-form reasoning from structured decoding through trigger tokens and evaluates the approach across models and classification and reasoning benchmarks.

  • Motivation: LLMs perform diverse tasks, but their lack of strict structural guarantees limits applicability to schema-based extraction, structured question answering, and industrial use cases.
  • Limitations of Existing Methods: Grammar-constrained decoding ensures syntactic correctness by masking invalid continuations, but may reduce expressiveness and fluency, harming generalization.
  • In-Writing: In-Writing lets reasoning proceed freely until trigger tokens such as <eos> or { activate structured decoding, decoupling reasoning from formatting.
  • Evaluation: The framework is evaluated across Qwen, Llama, Gemma, DeepSeek, and SmolLM models ranging from 1.5B–14B parameters on classification and reasoning benchmarks.Evaluation uses accuracy, parsability, and token efficiency as metrics.
  • Contributions: The contributions include minimal token overhead, constrained decoding as a parser and corrector, trigger-token strategies against premature triggering, and evidence against reasoning entirely within constrained grammar spaces.

2 Preliminaries

This section introduces finite automata, also called finite-state machines (FSMs), as central components of constrained decoding. FSMs represent regular expressions and guide token selection, with each automaton defined as a 5-tuple of states, alphabet, transitions, start state, and accepting states.

  • Finite Automata: Finite-state machines represent target regular expressions and guide decoding by allowing only regex-consistent tokens at each step.They are identified as central components in constrained decoding.
  • Finite Automata: A finite automaton is formally defined as the 5-tuple (Q, Σ, δ, q0, F).Q is the finite state set; Σ is the alphabet; δ is the transition function; q0 is the start state; and F is the set of accept states.

3 Related Work

Related work contrasts unconstrained autoregressive sampling, tool-based structured generation, and hard constrained decoding. While constrained methods improve structural control, tool calls remain autoregressive and hard constraints can degrade LLM performance.

  • Unconstrained Generation: Autoregressive sampling lacks structural constraints, limiting its applicability when outputs must follow strict formats.Each token is sampled from the conditional distribution given previous tokens, but standard sampling remains stochastic.
  • Tool Calling: Tool calling supports external API invocation and structured outputs such as JSON, but autoregressive tool calls do not guarantee strict structural correctness.
  • Hard Constrained Decoding: Hard constrained decoding enforces structured generation through decoder logit masking and can formulate regex- or grammar-guided generation as a finite-state machine.The finite-state-machine formulation enables controllable initialization and termination.
  • Hard Constrained Decoding: Hard constrained decoding has been shown to degrade LLM performance on certain tasks.

11 end

The paper positions In-Writing as a unified hybrid constrained-decoding framework that balances reasoning expressiveness, strict format compliance, and computational overhead. It addresses limitations of natural generation, hard-constrained decoding, CRANE, and NL-to-Format through flexible reasoning with minimal overhead.

  • Related work: NL-to-Format uses two stages: natural-language answer generation followed by conversion into the target format.This approach is presented as a recent hybrid method.
  • Related work: CRANE interleaves free-form and grammar-constrained generation through delimiter-based (« ») switching.The paper identifies CRANE as another recent hybrid approach.
  • Motivation: The paper frames a gap between reasoning expressiveness, format guarantees, and computational overhead in existing generation approaches.Hard-constrained decoding and CRANE restrict free-form reasoning, while natural generation and NL-to-Format lack strict structural enforcement.
  • Contribution: The proposed unified framework preserves reasoning flexibility, ensures strict format compliance, and does so with minimal overhead.This is the paper’s stated contribution for addressing the identified gap.

4 In-Writing Method

In-Writing separates unconstrained reasoning from structural formatting constraints. It preserves reasoning expressiveness while applying structured decoding only after a trigger token signals that reasoning is complete.

  • Formalization: Conventional constrained decoding conditions both reasoning and answer generation on the formatting constraint F.Logit masking formalizes this as P(A | R, q, F)P(R | q, F).
  • Formalization: Hard-constrained decoding can discard logically valid reasoning traces because of minor structural violations.An indicator function removes reasoning paths that do not comply with F.
  • Formalization: In-Writing delays formatting until reasoning is complete, making reasoning independent of the formatting condition.Its formulation is P(A | R, q, F)P(R | q), separating reasoning from formatting constraints.
  • Operational decoding: In-Writing retains CoT reasoning expressiveness and hard-constrained formatting guarantees.Algorithm 2 first generates an unconstrained reasoning trace until a trigger token, then switches to structured generation that parses the final trace into structured output.

17 end

In-Writing unifies free-form reasoning with constrained formatting in one call, applying strict syntax only to target output slots. It provides valid structured outputs with robustness and negligible formatting overhead.

  • In-Writing decouples reasoning and formatting within a single call, eliminating external verification.The approach allows free-form generation before enforcing constraints on target output slots.
  • Regex- or grammar-based constraints enforce syntactically valid, schema-compliant outputs during decoding.
  • Outputs are reliably mapped to structured formats without complex prompting or implementation, while formatting introduces negligible latency.

5 Experimental Setup

The experiments evaluate In-Writing across diverse reasoning and classification benchmarks using prior-work preprocessing and splits, with accuracy and parse rate as primary metrics. They also test trigger-token robustness across open-source models and fixed-format comparison settings.

  • Benchmarks: Evaluation spans numerical, symbolic, and textual reasoning and classification benchmarks using prior-work preprocessing and data splits.Reasoning tasks include GSM8K, GSM-Symbolic, Last Letter Concatenation, and Shuffled Objects; classification tasks include DDXPlus, MultiFin, Sports Understanding, and NI - Task 280.
  • Metrics: Accuracy and parse rate are the primary metrics, with answers evaluated by exact string match and parsability determined by expected prefixes or syntactic validity.Natural Language uses answer-prefix extraction, while NL-to-Format, Constrained Decoding, and In-Writing parse structured outputs directly.
  • Robustness: In-Writing robustness is evaluated across trigger-token sets to analyze premature triggering and whether a sole <eos> trigger mitigates reasoning truncation.In-Writing-Base uses trigger_token_ids = {<eos>, {}, while InWriting* uses <eos> alone.
  • Models: The study evaluates 18 open-source models from five families, ranging from 1.5B to 32B parameters and run locally with Transformers.The model families are Qwen, Llama, Gemma, DeepSeek, and SmolLM.
  • Comparison settings: Comparison benchmarks retain original fixed-format templates, leaving In-Writing without output-format guidance to test mapping reasoning into required formats.NL-to-Format uses natural-language prefixes, whereas CRANE uses delimiter-based outputs.
  • Implementation: Constrained Decoding and In-Writing use Litelines, while NL-to-Format uses Qwen3-32B as the second-stage parser model.Litelines extends Outlines with allow_preamble and trigger_token_ids, enabling free-form reasoning before schema-constrained generation.

6 Results and Discussion

Across reasoning and classification evaluations, In-Writing* improves structured extraction while preserving unconstrained reasoning, mitigating premature triggering and achieving reliable formatting with low overhead. It also outperforms competing constrained approaches, including CRANE, on symbolic reasoning tasks.

  • Extraction performance: In-Writing* yields gains of up to 27% over LLM-based parsing while sharing identical reasoning traces with natural-language and NL-to-Format methods.The performance difference therefore comes from answer extraction rather than reasoning.
  • Premature triggering: Over 30% degradation occurs on GSM8K when premature triggering truncates reasoning in In-Writing-Base compared with In-Writing*.The impact is minimal on classification tasks.
  • Overall effectiveness: In-Writing* generally outperforms NL-to-Format and vanilla constrained decoding across model scales, achieves 100% format validity, and adds only 5–20 tokens.Setting <eos> as the unique trigger token mitigates premature triggering without requiring larger models or explicit formatting guidance.
  • Formatting reliability: 100% parse rate is achieved by constraining only formatting while leaving reasoning unconstrained, with minimal inference overhead.The reasoning segment can be controlled with a maximum token limit, while the formatting component uses constrained construction.
  • Symbolic reasoning: Up to 32% improvement over CRANE is achieved on GSM-Symbolic by allowing unrestricted reasoning and constraining only final answer extraction.CRANE’s grammar-restricted reasoning can prevent necessary operators such as round() and encourage incorrect use of permitted operators such as int().

7 Conclusion

In-Writing unifies expressive reasoning with guaranteed output formatting in one inference call by applying constrained decoding as a post-reasoning parser and corrector. Experiments show improved performance with comparable or lower computational overhead, while the study identifies prompt optimization and few-shot behavior as important limitations.

  • 7 Conclusion: In-Writing decouples reasoning expressiveness from guaranteed output formatting within a single inference call.It reframes constrained decoding as a post-reasoning parser and corrector rather than a generation method.
  • 7 Conclusion: Experiments across reasoning and classification benchmarks show improved performance with comparable or lower computational overhead, even with misaligned prompts.
  • 7 Conclusion: The study deliberately evaluates In-Writing without prompt optimization or explicit guidance for the desired output format.Future work should explore prompt optimization to improve performance and stability.
  • 7 Conclusion: Increasing few-shot examples improves Baseline performance but degrades both NL-to-Format and In-Writing.The effect stems from greater adherence to the “answer is:” prefix, not differences in reasoning traces; some marginal cases show NL-to-Format succeeding while In-Writing fails.

A Appendix … A.4.4 Empirical Contradictions to the CRANE Framework

The paper presents In-Writing as a hybrid method that reasons freely before switching to constrained decoding, while revisiting prior evaluations reveals prompt, dataset, parser, and framework limitations. Its results suggest In-Writing can recover performance losses associated with constrained reasoning, although it remains imperfect and CRANE’s empirical robustness is questioned.

  • A.1 Illustration of the In-Writing Framework: In-Writing generates unconstrained reasoning in state −1, then switches to FSM-guided decoding after a trigger token.The constrained phase restricts outputs to regex- or grammar-valid tokens.
  • A.2 NL-to-Format Comparison: Experimental Setting Discussion: Prior evaluations contain inconsistent answer-choice instructions, format variations, few-shot examples, and prompting settings, motivating a zero-shot Shuffled Objects evaluation.The study otherwise preserves prior settings such as seed=1, temperature=0, and do_sample=False, and uses the same prompt template for direct comparison.
  • A.3 CRANE Comparison: Experimental Setting Discussion: The CRANE comparison appends an instruction requiring the final answer as a symbolic expression in JSON while retaining the original Chain-of-Thought prompt and few-shot examples.This setup is intended to extract the final answer from reasoning marked by « ».
  • A.3.2 GSM-Symbolic dataset: GSM-Symbolic contains operator misuse, omitted operations, and ambiguities that can make correctness ill-defined when symbolic variables receive concrete values.The discussion contrasts // or int() with / or round() in examples of division and rounding.
  • A.3.3 Evaluation of GSM-Symbolic Experimental Results: The evaluation manually compares parsed formulas with gold answers while treating semantically equivalent division and rounding operations as equivalent.Ambiguous currency-number expressions are conservatively marked incorrect.
  • A.4 Experimental Results and Discussion: The experiments re-evaluate prior models, add extension models, measure parsability, analyze overlap with NL-to-Format, assess token efficiency, and compare In-Writing with CRANE.Results span GSM8K, Last Letter Concatenation, Shuffled Objects, and GSM-Symbolic settings.
  • A.4.2 Limitations of LLMs as Parsers: LLM-based parsers can be costly, modify predictions, and behave unstably because of prompt dependence.Examples include correcting whitespace or hyphens, and adding conversational prefixes before otherwise correct diagnoses.
Loading 2601.07525v2…