Source-linked AI summary
General Intelligence Requires Reward-based Pretraining
Seungwook Han, Jyothish Pari, Samuel J. Gershman, Pulkit Agrawal
TL;DR
LLMs show broad practical utility but remain fragile when transferring algorithmic reasoning to novel contexts, partly because knowledge and reasoning are coupled through passive next-token pretraining. The paper proposes reward-based reasoning pretraining, synthetic-task curricula, and a short-context architecture with external memory; its central conclusion is that deliberately decoupling knowledge from reasoning may support more robust generalization. The paper also notes that synthetic-task design and the relationship between knowledge and reasoning remain important limitations and open questions.
Problem
LLMs can succeed on familiar reasoning tasks yet struggle to generalize algorithmic understanding across novel contexts, raising a gap between AUI and AGI.
Method
The paper proposes reward-based pretraining, synthetic-task curricula, and a decoupled reasoning network that dynamically reads and writes to external memory through short context windows.
Results
The paper reports that current models perform poorly on esoteric programming-language tasks, averaging ∼12% accuracy in Brainf**k and ∼29% in Befunge, with o1 as a notable outlier.
Takeaways & Limitations
Deliberately separating reasoning from memorized knowledge is presented as a route toward more transferable reasoning across new problems and knowledge domains.
Takeaways & Limitations
Synthetic pretraining tasks may fail to transfer when they are poorly designed, narrow, or omit reasoning skills required by natural-language tasks.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated impressive real-world utility, exemplifying artificial useful intelligence (AUI). However, their ability to reason adaptively and robustly -- the hallmarks of artificial general intelligence (AGI) -- remains fragile. While LLMs seemingly succeed in commonsense reasoning, programming, and mathematics, they struggle to generalize algorithmic understanding across novel contexts. Our experiments with algorithmic tasks in esoteric programming languages reveal that LLM's reasoning overfits to the training data and is limited in its transferability. We hypothesize that the core issue underlying such limited transferability is the coupling of reasoning and knowledge in LLMs. To transition from AUI to AGI, we propose disentangling knowledge and reasoning through three key directions: (1) pretaining to reason using RL from scratch as an alternative to the widely used next-token prediction pretraining, (2) using a curriculum of synthetic tasks to ease the learning of a reasoning prior for RL that can then be transferred to natural language tasks, and (3) learning more generalizable reasoning functions using a small context window to reduce exploiting spurious correlations between tokens. Such a reasoning system coupled with a trained retrieval system and a large external memory bank as a knowledge store can overcome several limitations of existing architectures at learning to reason in novel scenarios.
1. Introduction
The paper argues that LLMs’ useful capabilities do not guarantee robust reasoning across novel contexts because next-token pretraining can couple reasoning to memorized knowledge and spurious token correlations. It proposes reward-based reasoning pretraining, synthetic-task curricula, and a decoupled memory-reasoning architecture to improve transferability.
- Motivation: LLMs perform well on familiar tasks but struggle to transfer algorithmic understanding to novel contexts, revealing a gap between artificial useful intelligence and artificial general intelligence.Esoteric programming-language tasks are used to isolate reasoning from memorization.
- Motivation: Next-token prediction can reward superficial token correlations or memorized answers rather than learning reasoning algorithms that generalize beyond the training distribution.For example, a model may learn that a number after '+' is followed by '=' instead of learning addition.
- Reward-based pretraining: Reward-based pretraining is proposed as an alternative that generates reasoning traces through iterative, step-wise problem solving rather than passive answer prediction.The approach is intended to support more robust reasoning by exposing intermediate problem-solving processes.
- Synthetic-task curriculum: A curriculum of synthetic tasks with reduced token spaces is proposed to learn a transferable reasoning prior before bootstrapping reasoning in natural language.Task difficulty and token-space complexity can increase gradually while preserving structural properties such as commutativity, associativity, and in-context learning.
- Architectural bias: The proposed architecture separates an external memory bank from a reasoning network, uses short context windows, and trains dynamic memory reading and writing to reduce spurious correlations and support new knowledge domains.The paper hypothesizes that this structured approach improves generalization by decoupling knowledge from reasoning.
2. Background and Notation
The paper distinguishes supervised pretraining on passive Internet text from reward-based pretraining based on online interaction and cumulative reward. It formalizes the former as next-token likelihood maximization and the latter as expected-reward maximization.
- Supervised pretraining: Current LLMs are pretrained on large, unlabeled Internet corpora that usually contain answers without intermediate reasoning steps.The corpus is represented as token sequences, and the model predicts each token from preceding tokens.
- Supervised pretraining: Supervised pretraining maximizes the log-likelihood of each token conditioned on its preceding tokens.The paper calls this task-agnostic next-token objective supervised pretraining, or SPT.
- Reward-based pretraining: Reward-based training instead gathers data online from the model’s own interactions and optimizes expected cumulative reward.The model receives a scalar reward at each step, which may depend on partial or complete outputs.
3. Evaluation of Reasoning Separate From Knowledge
The evaluation tests whether LLMs can transfer reasoning to simple tasks in unfamiliar esoteric programming languages rather than rely on familiar programming knowledge. Models generally perform poorly, especially on Brainf**k, although in-context examples help on limited Befunge tasks and o1 performs notably better.
- Benchmark design: The benchmark uses Brainf**k and Befunge because their unconventional rules and limited exposure reduce the influence of memorized programming knowledge.Both languages are Turing-complete but differ radically from familiar languages such as Python, enabling evaluation of reasoning transfer to novel domains.
- Evaluation setup: The evaluation covers printing, sorting, copying, factorial, and related tasks across models from multiple families and scales using standardized prompts with language rules and examples.Problems vary inputs for each task, such as changing the characters in printing problems.
- Results: ∼12% average accuracy on Brainf**k and ∼29% on Befunge show that current models generally struggle on these simple algorithmic tasks.Brainf**k performance improves by only about 4% on average when in-context examples increase from 1 to 10.
- Results: In Befunge, in-context examples raise printing accuracy to ∼70-90%, but models fail on the Fibonacci and factorial tasks.This contrast indicates that example-based assistance helps for some task types but does not yield broad transfer across the benchmark.
- Results: o1 vastly outperforms other models, consistent with its extensive post-training with RL for solving reasoning tasks.Even o1 scores only 1% accuracy on sorting five elements with 10 in-context examples.
4. Proposed Directions
The paper proposes replacing passive supervised pretraining with reward-based reasoning, synthetic-task curricula, and architectures that separate reasoning from stored knowledge. Evidence from Go and synthetic mathematics supports the concern that supervised training can constrain exploration and generalization.
- 4.1. Pretraining for Reasoning with RL: The first proposed direction integrates RL directly into pretraining instead of pretraining on passive data and applying RL only during finetuning.In this formulation, the model learns from online interactions and scalar rewards rather than a passively collected corpus.
- 4.1. Pretraining for Reasoning with RL: A 92% win rate against SPT + RFT with KL coefficient 0.5, versus 66% with coefficient 0.1, shows that tighter reliance on pretrained knowledge hinders exploration while looser constraints improve RL strategies.RPT also achieves a 100% win rate against SPT; without KL constraints and with infinite training, both paradigms can eventually converge to the same optimal policy.
- 4.1. Pretraining for Reasoning with RL: Pure RFT outperforms SFT-then-RFT on vector orthogonality, while SFT-then-RFT reaches 100% training accuracy but only 80% test accuracy through overfitting.The authors attribute this pattern to SFT models predicting answers directly rather than generating intermediate reasoning steps needed for later RL exploration.
- 4.2. Pretraining for Reasoning from Scratch (RPT) using Synthetic Tasks: Synthetic tasks or games with reduced token spaces are proposed to make reasoning-trace exploration tractable before transferring learned reasoning priors to natural language.The proposal targets the combinatorial difficulty of searching nearly unconstrained natural-language spaces and leaves task selection and transfer as open questions.
- 4.2. Pretraining for Reasoning from Scratch (RPT) using Synthetic Tasks: Structured environments such as code may support transferable reasoning because their precise syntax, deterministic semantics, and modularity provide logic and problem-solving priors.The paper cites code pretraining and cognitive studies of programming as evidence that structured environments can facilitate transfer beyond coding.
- 4.3. Decoupling Knowledge and Reasoning to Generalize Across Domains: The proposed architecture decouples knowledge and reasoning through an external semantic memory, learned retrieval, and a small working context for selective processing.New-domain knowledge can be inserted into semantic memory, while reasoning operates over a smaller context intended to reduce overfitting to token correlations.
5. Discussion and Alternative Views
The paper acknowledges that knowledge and reasoning may be inseparable in practice, while arguing that systems should retain a mechanism to separate them. It also identifies scalability and task-design challenges for reward-based and synthetic pretraining.
- Knowledge and reasoning may be coupled for efficiency, but the paper argues models should still separate them to avoid relying only on experiential correlations.The coffee-completion example illustrates the ambiguity between memorized associations and inference.
- RL-based pretraining is computationally expensive and may become infeasible as model sizes and training demands grow.
- Poorly designed synthetic tasks or narrow task distributions may fail to transfer to natural-language reasoning.Tasks whose reasoning skills differ from real-world language demands may require compensatory mechanisms.
6. Related Works
Prior work reports strong LLM reasoning abilities, but also documents fragility and overfitting to training patterns. Fine-tuning can improve both memorization and generalization, while inference-time scaling is identified as another reasoning direction.
- LLMs show strong reasoning yet remain fragile and often overfit to training patterns, including in esoteric-language, counterfactual, and symbolic-reasoning evaluations.
- Fine-tuning has been shown to improve both memorization and generalization.
- Inference-time scaling is presented as a related direction for reasoning in LLMs.
A.1. Example of Brainf**k Program +++++++[>++++++++++<-]>+++++.
The appendix provides a Brainf**k program example and a Befunge factorial-program listing. These examples illustrate the esoteric-language materials used in the paper.
- Listing 3 presents a Brainf**k program intended to print the character ’K.’
- The examples use esoteric programming languages as concrete program artifacts in the evaluation materials.
- Listing 4 presents a Befunge program for calculating a factorial.
A.3. Details on Experimental Protocol
The experimental protocol supplies prompts, language rules, examples, and task-specific program pairs for Brainf**k and Befunge. It covers printing, sorting, copying, arithmetic, and control-flow behaviors across the two-dimensional and tape-based languages.
- Brainf**k operates with a zero-initialized memory array and data pointer, using commands for movement, cell updates, output, and input.
- Brainf**k loops jump forward or backward according to whether the current data cell is zero, enabling iteration with matched brackets.
- The protocol includes Brainf**k prompts whose instruction-program pairs vary with the number of in-context examples, while sorting and copying change only the task-specific parts.
- Befunge uses a two-dimensional grid, directional instruction-pointer movement, stack operations, arithmetic, output commands, and @ for termination.
- The examples demonstrate Befunge self-modifying operations, string output, addition, countdown control flow, and explicit program termination.
A.6. Synthetic Task Details
The synthetic math task was designed so the base model could sometimes produce correct reasoning while still making errors, enabling efficient RL exploration. The section also records the SFT and RL training configurations used for comparison.
- The synthetic task used 100 training and 100 test examples and was designed to give the base model a non-trivial chance of generating correct reasoning traces.Custom verification was developed to evaluate the model’s outputs.
- The training setup included LoRA with r=256, alpha 32, q/k/v projection targets, 0.05 dropout, and no bias.
- Table 4 is identified as presenting the SFT training results, but the supplied passage does not report its numerical values.
- Both SFT and base models underwent 31 RLOO steps with batch size 36, one PPO epoch per iteration, and KL coefficient 0.5.The RLOO algorithm is attributed to Kool et al. (2019).
A.7. Prompt Format and Sample Responses
The appendix illustrates the prompt format and reasoning responses for a vector-orthogonality multiple-choice task. The examples show both concise answer-only output and step-by-step dot-product verification leading to option (d).
- A.7. Prompt Format and Sample Responses: The prompt instructs the assistant to think step by step concisely and answer in a specified solution format.
- A.7. Prompt Format and Sample Responses: The detailed response rejects [3, 5, 0] because its dot product with [1, -1, -1] is -2, not zero.
- A.7. Prompt Format and Sample Responses: The detailed response rejects [-5, 2, 2] because its dot product with [1, -1, -1] is -9, not zero.
- A.7. Prompt Format and Sample Responses: The response computes a zero dot product for [1, 2, -1], identifies it as orthogonal, and concludes that it is the only valid vector.
- A.7. Prompt Format and Sample Responses: The SFT sample gives the concise response “solution: (d)” for the orthogonality question.
B.1. PT + RL Model Training Details
The PT + RL model is trained in a curriculum-guided setup that combines supervised Go-game pretraining with a reasoning model, external memory, and staged tasks.
- Stage 1 trains on 80,824 professional 9×9 Go trajectories, yielding roughly 1M next-move examples.Training uses 10 epochs, batch size 1024, learning rate 10^-3, and weight decay 10^-4.
- Algorithm 1 initializes a reasoning model, external memory, and an Easy-to-Hard curriculum before iterating over sampled tasks.
- At each task step, the reasoning model appends observations to context and queries actions that can read, write, or reason using external memory.