PaperlayerView exact revision on arXiv

Source-linked AI summary

ORION: Teaching Language Models to Reason Efficiently in the Language of Thought

Kumar Tanmay, Kriti Aggarwal, Paul Pu Liang, Subhabrata Mukherjee

arXiv:2511.22891v1cs.AIcs.CLcs.LG

TL;DR

Reasoning models often produce verbose chains that increase latency and redundancy. ORION trains models to reason in compact symbolic Mentalese and uses SLPO to favor shorter correct solutions while allowing longer reasoning when needed. Across evaluations, ORION pairs compressed traces with strong benchmark results and lower training costs.

  • Problem

    Large reasoning models rely on long, verbose reasoning traces, motivating a compact reasoning style that reduces redundancy and latency.

    Source: §1 · p. 2

  • Method

    ORION aligns models to compact symbolic Mentalese traces and applies SLPO, which rewards shorter correct rollouts while allowing more steps when needed.

    Source: §1 · pp. 2–3

  • Results

    Across math and out-of-domain evaluations, ORION pairs compressed reasoning with strong results; ORION-AG-SLPO 1.5B beats DeepSeek-R1 1.5B by 7 pp with 7× shorter reasoning.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present concise, structured reasoning as a pathway toward deploying reasoning capabilities in real-time, resource-constrained settings.

    Source: §6 · p. 10

    In experiments, direct SLPO on raw base models collapsed after approximately 300 steps, whereas an intermediate Mentalese SFT stage maintained stability.

    Source: §5 · p. 10

Abstract

from arXiv · show

Large Reasoning Models (LRMs) achieve strong performance in mathematics, code generation, and task planning, but their reliance on long chains of verbose "thinking" tokens leads to high latency, redundancy, and incoherent reasoning paths. Inspired by the Language of Thought Hypothesis, which posits that human reasoning operates over a symbolic, compositional mental language called Mentalese, we introduce a framework that trains models to reason in a similarly compact style. Mentalese encodes abstract reasoning as ultra-compressed, structured tokens, enabling models to solve complex problems with far fewer steps. To improve both efficiency and accuracy, we propose SHORTER LENGTH PREFERENCE OPTIMIZATION (SLPO), a reinforcement learning method that rewards concise solutions that stay correct, while still allowing longer reasoning when needed. Applied to Mentalese-aligned models, SLPO yields significantly higher compression rates by enabling concise reasoning that preserves the benefits of detailed thinking without the computational overhead. Across benchmarks including AIME 2024 and 2025, MinervaMath, OlympiadBench, Math500, and AMC, our ORION models produce reasoning traces with 4-16x fewer tokens, achieve up to 5x lower inference latency, and reduce training costs by 7-9x relative to the DeepSeek R1 Distilled model, while maintaining 90-98% of its accuracy. ORION also surpasses Claude and ChatGPT-4o by up to 5% in accuracy while maintaining 2x compression. These results show that Mentalese-style compressed reasoning offers a step toward human-like cognitive efficiency, enabling real-time, cost-effective reasoning without sacrificing accuracy.

1 INTRODUCTION

The introduction argues that reasoning models’ verbose traces and costly training motivate a more compact reasoning style. It presents Mentalese alignment and SLPO as a framework for concision while preserving correctness, and reports compressed reasoning with strong benchmark results.

1 INTRODUCTION

  • Reasoning-focused reinforcement learning can improve performance, but rollout generation is costly and R1-style traces are often verbose and redundant compared with short human thought steps.The introduction grounds its proposed compact reasoning style in the Language of Thought hypothesis.Source: §1 · p. 2
  • ORION-AG-SLPO 1.5B surpasses GPT-4o, Claude 3.5 Sonnet, and Llama 3.3 70B by an average of 6 pp in mathematical reasoning, and beats DeepSeek-R1 1.5B by 7 pp with 7× shorter reasoning.ORION-DS-GRPO 1.5B achieves 14× compression relative to DeepSeek-R1 1.5B; on out-of-domain tasks, ORION improves over the base model by 1 pp with 15× compression.Source: §1 · p. 3
  • Mentalese alignment restructures LLM reasoning into compact, faithful symbolic reasoning, while SLPO adaptively balances brevity and correctness without rigid length penalties.SLPO rewards shorter correct rollouts, allowing concise reasoning on simpler tasks and more steps when needed.Source: §1 · pp. 2–3
  • MentaleseR-40k is a released dataset of ultra-compressed reasoning traces for 40k math problems, generated under symbolic constraints inspired by the Language of Thought hypothesis.Source: §1 · p. 3
  • The experiments compare GRPO and SLPO to identify their different compression levels and the settings in which each is most effective.Source: §1 · p. 3

2 RELATED WORKS

Prior work improves reasoning through longer or multiple chains, but long natural-language traces can be redundant. This paper situates its compact symbolic reasoning and adaptive preference objective among efforts to control reasoning length and change reasoning formats.

2 RELATED WORKS

  • CoT prompting, parallel sampling, tree search, planning, and iterative refinement scale reasoning, while extended traces can also bring redundancy, latency, and compounding errors.Source: §2 · p. 3
  • Length-control approaches include fixed budgets, difficulty-adaptive budgets, reference pruning, and rewards for shorter outputs; SLPO complements them by preferring concise correct reasoning without penalizing necessary longer derivations.Source: §2 · pp. 3–4
  • Alternative reasoning formats include symbolic or structured CoT and latent reasoning; Mentalese CoT adds a cognitively motivated symbolic language, paired with SLPO to target compression and reliable reasoning.Source: §2 · p. 4

3 METHODOLOGY

The methodology combines symbolic reasoning alignment with reinforcement learning that favors concise solutions while allowing necessary longer reasoning. It defines Mentalese as a structured symbolic format and describes its construction and training use.

3 METHODOLOGY

  • The pipeline aligns models to Mentalese with supervised fine-tuning, then applies GRPO or SLPO with verifier feedback; SLPO rewards brevity without penalizing necessary longer reasoning.Source: §3 · p. 4

3.1 MENTALESE: MENTAL LANGUAGE OF THOUGHT

  • Mentalese translates the Language of Thought hypothesis into compact symbolic primitives intended to capture core logical operations without verbose natural-language explanations.Source: §3.1 · p. 4
  • A Mentalese step pairs an operator with an expression, and a valid trace is well-typed, executable, and ends with exactly one terminal ANS:e; step.Source: §3.1 · p. 4
  • Unlike free-form CoT, Mentalese uses canonical OPERATION:expression; steps joined by semicolons; the stated advantages include up to 10× shorter reasoning, faithfulness, and cognitive alignment.Source: §3.1 · pp. 4–5
  • MENTALESE-40K contains traces for 40k+ math problems from AIME, Omni-Math, and STILL; GPT-4.1 generated them with structured prompts, and 65 malformed cases were removed.The traces were used for supervised fine-tuning; RLVR instead used the original question-answer pairs with a verifier.Source: §3.1 · p. 5

MENTALESE FRAMEWORK

The framework defines Mentalese as compact symbolic reasoning, then combines supervised alignment with verifier-based reinforcement learning. GRPO supplies group-relative optimization, while SLPO adapts brevity preferences to correctness.

DEFINITION

  • Mentalese represents reasoning as compact symbolic logic steps separated by semicolons, with no natural language and all core logic preserved.Source: DEFINITION · p. 5

3.2 GROUP RELATIVE POLICY OPTIMIZATION (GRPO)

  • GRPO samples responses for each prompt, normalizes their rewards into group-relative advantages, then optimizes a clipped surrogate objective with a KL penalty.The relative advantages compare responses within the group rather than relying on an absolute critic model.Source: §3.2 · pp. 5–6

3.3 SHORTER LENGTH PREFERENCE OPTIMIZATION (SLPO)

  • SLPO favors shorter correct traces when multiple valid derivations exist, while preserving uniquely correct long solutions and avoiding reward distortion when none are correct.Its preferences are relative to the observed range of correct reasoning lengths for each problem instance; α=0.1 was used for a stable balance across benchmarks.Source: §3.3 · p. 6

3.4 MENTALESE ALIGNMENT THROUGH SFT FOLLOWED BY RLVR

  • Training first uses supervised fine-tuning on Mentalese traces to anchor a compact single-chain format, then verifier-based reinforcement learning refines it and can restore accuracy while limiting verbosity.RLVR may use either GRPO or SLPO as its reward formulation.Source: §3.4 · pp. 6–7

4 EXPERIMENT DESIGN

The experiments examine Mentalese compression, recovery of performance through reinforcement learning, and training efficiency and stability across mathematical and out-of-domain evaluations. They compare ORION models with distilled, length-controlled, and frontier-scale baselines using accuracy, response length, and compression metrics.

  • The study asks whether Mentalese compresses reasoning while preserving performance, which RL algorithm best recovers performance, and how algorithm suitability and training stability vary.Source: §4 · p. 7
  • The primary comparison was DEEPSEEK-R1-1.5B, alongside AGENTICA-24K and L1-Max; GPT-4o, Claude 3.5 Sonnet, and LLaMA-3 70B provided frontier-scale references.The passages identify the sources as DeepSeek-AI et al. (2025), Luo et al. (2025b), Aggarwal & Welleck (2025), OpenAI et al. (2024a), Anthropic (2024), and Grattafiori et al. (2024).Source: §4 · pp. 7–8
  • Approximately 7,800 tokens for Agentica-24k, 150 for ORION-AG, and 300 for ORION-AG-SLPO produced similar AIME 2024 accuracy in the reasoning-trace comparison.Source: Figure 4 · p. 8
  • Evaluation covered six in-domain mathematics datasets and three out-of-domain benchmarks to assess generalization beyond mathematical reasoning.Source: §4 · p. 8
  • Performance was reported as Pass@1, average response token length, and compression rate relative to DEEPSEEK-R1-1.5B.Source: §4 · p. 8

5 DISCUSSION

The discussion reports that Mentalese SFT sharply reduces accuracy, while RLVR recovers most of it with shorter traces. Intermediate SFT also lowers RL training costs and stabilizes training, but large generation budgets can undo compact reasoning.

  • SFT on Mentalese lowered average accuracy by 35.5 p.p. relative to the base model, while RLVR regained most lost accuracy with traces only a few hundred tokens longer.SFT encourages a single symbolic path; RLVR restores adaptive reasoning behavior within the compact format.Source: §5 · p. 10
  • Intermediate SFT reduced RL training cost by 7–10× while achieving performance close to the base model with reasoning traces 10× shorter.Direct RLVR on base models took 5–6 days for 1,500 steps; long reasoning chains were the main latency bottleneck.Source: §5 · p. 10
  • Direct SLPO on AGENTICA-24K collapsed after about 300 steps, while intermediate Mentalese SFT kept training stable throughout.The collapse followed response lengths expanding to the 8k-token limit and NaN gradient norms.Source: §5 · p. 10
  • With 4k- or 8k-token generation limits, RLVR could drift from compact reasoning and sometimes collapse; 1k-token GRPO and 2k-token SLPO limits preserved the learned format.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents Mentalese plus SLPO as a compact reasoning framework that balances brevity and correctness. The authors report over 10× shorter traces with accuracy close to verbose large reasoning models, alongside lower training and inference costs.

  • The proposed framework combines compact symbolic Mentalese reasoning with SLPO, which adaptively balances conciseness and correctness.The authors report over 10× compression while maintaining accuracy close to verbose large reasoning models and reducing training and inference costs.Source: §6 · p. 10

A.1 METRICS FORMULATION

The section defines Pass@1 as single-sample correctness, Token Length as average response length, and Compression Rate as shortening relative to DeepSeek-R1-1.5B.

A.1 METRICS FORMULATION

  • Pass@1 measures the fraction of problems solved correctly under single-sample decoding.The metric counts correct first-attempt solutions across the test problems.Source: §A.1 · p. 18
  • Token Length is the average number of generated tokens per response for a benchmark.The mean output length is computed across all questions in the dataset.Source: §A.1 · p. 18
  • Compression Rate (CR) compares response length with DeepSeek-R1-1.5B, with higher values indicating greater compression.A CR of 10 means responses are ten times shorter than DeepSeek-R1-1.5B.Source: §A.1 · p. 18

A.2 HYPERPARAMETERS

The section identifies separate key-hyperparameter tables for supervised fine-tuning and RLVR training.

A.2 HYPERPARAMETERS

  • Table 3 presents key hyperparameters for supervised fine-tuning.The supplied table passage gives its scope but not the parameter values.Source: Table 3 · p. 18
  • Table 4 presents key hyperparameters for RLVR training.The supplied table passage gives its scope but not the parameter values.Source: Table 4 · p. 19

A.3 MENTALESER-40K EXAMPLES

The examples demonstrate Mentalese’s compact symbolic steps on varied mathematics problems, while a figure reports substantial response-length reductions from Mentalese-based training.

A.3 MENTALESER-40K EXAMPLES

  • A custom operation is encoded as a definition followed by calculation steps and a final subtraction yielding −2/3.The example defines a ⊗ b = a^2/b and evaluates both groupings before subtracting.Source: §A.3 · p. 19
  • The window example turns pane dimensions and border widths into two total-dimension expressions, equates them, and obtains a side length of 26.Its symbolic steps represent two rows and four columns of panes.Source: §A.3 · p. 19
  • The polynomial example encodes repeated values at residue-class inputs, counts 3n+1 points, and concludes n = 1.The provided Mentalese trace uses the degree-plus-one point count to form its equation.Source: §A.3 · p. 20
  • Mentalese-based methods reduce Agentica-24k response lengths by 10–20× across six benchmarks, approaching a balance between efficiency and performance.The figure caption notes some performance degradation, with the Mentalese training pipeline using RLVR methods offering the best trade-off.Source: Figure 6 · p. 21

A.4 PROMPT STRUCTURE

The appendix prompt specifies symbolic, compact Mentalese output, and its model-response examples contrast response lengths and reasoning styles on a shared math question.

A.4 PROMPT STRUCTURE

  • The prompt requires Mentalese CoT as compact symbolic logic steps separated by semicolons, preserving core reasoning.It also specifies OP:params or OP:result notation and says steps should be necessary and sufficient.Source: §A.4 · p. 21
  • The prompt requests a single JSON object containing the mentalese_cot output.It frames the task as generating a detailed solution and compressing it into the requested format.Source: §A.4 · p. 21

A.5 MODEL RESPONSES

  • ORION-AG responds in 122 tokens, versus 1,512 tokens for Agentica-24k on the same smallest-cube question.The appendix also lists 526 tokens for Agentic-24k-SLPO on that question.Source: §A.5 · pp. 22–23
  • The ORION-AG example reaches 27 by expressing the sum as a cube, deriving k = 3, and verifying 8 + 9 + 10.Its response is presented as 122 tokens.Source: §A.5 · pp. 22–23
  • For the same question, ORION-AG-SLPO uses 351 tokens and ORION-AG-GRPO uses 236 tokens.The appendix presents both as model-response examples.Source: §A.5 · p. 22
  • For three consecutive integers centered at n, their sum is 3n; requiring a cube k^3 forces k divisible by 3, and the smallest positive such cube is 27, from 8 + 9 + 10.Source: §A.5 · p. 22
Loading 2511.22891v1…