Source-linked AI summary
Reasoning Like Program Executors
Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Qiang Fu, Yan Gao, Jian-Guang Lou, Weizhu Chen
TL;DR
Language models remain weak at natural-language reasoning, while clean reasoning data is difficult to obtain. POET pre-trains them to imitate program executors using programs and execution results, and experiments show transferable gains across numerical, logical, and multi-hop reasoning tasks. The approach is strongest when executor and downstream reasoning skills overlap.
Problem
Existing language models have limited advanced reasoning ability, while large amounts of clean natural-language data containing explicit reasoning are difficult to obtain.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
Program executors provide a data-driven source of reasoning knowledge that can transfer from formal programs to broader natural-language reasoning scenarios.
Source: §1 · p. 2Transferability depends on overlap between reasoning skills learned during pre-training and those required downstream.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Reasoning over natural language is a long-standing goal for the research community. However, studies have shown that existing language models are inadequate in reasoning. To address the issue, we present POET, a novel reasoning pre-training paradigm. Through pre-training language models with programs and their execution results, POET empowers language models to harvest the reasoning knowledge possessed by program executors via a data-driven approach. POET is conceptually simple and can be instantiated by different kinds of program executors. In this paper, we showcase two simple instances POET-Math and POET-Logic, in addition to a complex instance, POET-SQL. Experimental results on six benchmarks demonstrate that POET can significantly boost model performance in natural language reasoning, such as numerical reasoning, logical reasoning, and multi-hop reasoning. POET opens a new gate on reasoning-enhancement pre-training, and we hope our analysis would shed light on the future research of reasoning like program executors.
1 Introduction
POET addresses the gap between strong language understanding and weak reasoning in language models by pre-training them to imitate program executors. It uses executable programs and automatic execution results to transfer reasoning knowledge to natural-language tasks.
1 Introduction
- Existing pre-training relies mainly on natural-language modeling, yet language models remain inadequate on numerical and logical reasoning tasks.Neural-symbolic systems improve reasoning but leave the reasoning mechanism in an external symbolic component, limiting reuse on unseen tasks.Source: §1 · p. 1
- POET pre-trains language models on programs and their execution results, aiming to internalize symbolic reasoning and transfer it across representation systems.The framework takes a program and program context as input and learns to produce the correct execution result.Source: Figure 1 · p. 1, §1 · p. 2
- Programs offer scalable, high-quality reasoning data because their formal grammar enables systematic synthesis and avoids many shortcomings of noisy natural-language corpora.The paper argues that programs can be made arbitrarily complicated and generated at scale with automatic supervision from program executors.Source: §1 · p. 2
- POET is instantiated across different program types, including math, logic, and SQL, to target varied reasoning capabilities.The introduction presents three exemplary instantiations and frames program executors as sources of formal reasoning knowledge.Source: §1 · p. 2
- On six reasoning-focused downstream tasks, POET transfers learned reasoning ability to broader natural-language scenarios and achieves competitive performance.The authors also report comprehensive analytical studies intended to clarify reasoning transfer and guide future work.Source: §1 · p. 2
2 Related Work
Prior work injects reasoning through specialized models, natural-language pre-training, or program execution, but each line has distinct scalability or transfer limitations. POET instead pre-trains on executable programs to transfer program-executor knowledge to diverse natural-language reasoning tasks.
Reasoning Skills
- The paper targets numerical, multi-hop, hybrid-context, and logical reasoning while leaving commonsense reasoning for future work.This scope distinguishes the reasoning abilities emphasized by POET from those outside the paper’s focus.Source: §2 · p. 3
Reasoning via Specialized Models
- Specialized reasoning modules can work well on individual datasets but are difficult to scale to complex reasoning scenarios.These methods typically select task-specific components, such as a count module, based on predicted answer types.Source: §2 · p. 3
Reasoning via Pre-training
- Natural-language reasoning pre-training is constrained by the difficulty of collecting large, high-quality reasoning text and synthesizing fluent sentences for complex operations.POET addresses this data challenge by using programs that can be synthesized at arbitrary scale with high quality.Source: §2 · p. 3
Reasoning in Giant Language Models
- Unlike prompting methods that mainly benefit giant language models, POET is presented as applicable to models ranging from millions to billions of parameters.The paper identifies combining POET with prompting strategies as an open direction.Source: §2 · p. 3
Program Execution
- POET differs from prior program-execution work by transferring executor knowledge through pre-training rather than merely learning a neural representation of an executor.Its scope includes Math, Logic, and SQL and applies to multiple natural-language downstream tasks.Source: §2 · pp. 3–4
3 Reasoning Like Program Executors
POET treats reasoning as executable symbolic computation and trains language models to imitate program executors using programs, contexts, and execution results.
3 Reasoning Like Program Executors
- POET frames reasoning as executable symbolic computation that can transfer across input modalities into natural-language tasks.The paper motivates this view by describing how humans abstract information into executable symbolic systems.Source: §3 · p. 4
- During POET-Math and POET-Logic pre-training, the model receives concatenated program and context inputs and learns to output the execution result.Figure 3 depicts this shared pre-training pattern.Source: Figure 3 · p. 4
- A program is a formally governed symbol sequence, including logical forms, code, or mathematical expressions, whose meaningful output reflects its computational procedure.POET accepts programs as long as they return meaningful outputs.Source: §3 · p. 4
- A program context is the execution environment containing variables accessible to the program, analogous to the relationship between a question and its reading-comprehension passage.This analogy connects program execution inputs with natural-language reasoning contexts.Source: §3 · p. 4
- Program executors deterministically run programs within their contexts, providing the reasoning supervision that POET uses to train language models.The executor is treated as a teacher, while the language model learns from its outputs.Source: §3 · pp. 4–5
4 POET with Singleton Executors
POET's singleton-executor instances target arithmetic and logical reasoning separately, and preliminary experiments show transfer of both skills to natural-language benchmarks.
4 POET with Singleton Executors
- The singleton-executor experiments begin with separate reasoning capabilities before later moving to integrated executors.The section distinguishes the singleton setup from the broader POET-SQL design.Source: §4 · p. 5
4.1 Learning from Math Calculators
- POET-Math injects basic addition and subtraction skills into language models through training on math-calculator execution.Its target skill matches downstream questions requiring arithmetic relations between two numbers.Source: §4.1 · p. 5
- POET-Math trains an encoder-decoder model to calculate expressions from floating-point variables and an addition/subtraction program, producing outputs such as 180.7.The corpus contains 4 million randomly generated examples with expressions using up to 2 operators and 3 variables.Source: §4.1 · p. 5
4.2 Learning from Logic Solvers
- POET-Logic injects necessary-conditional reasoning by determining whether a conclusion is implied by first-order-logic premises.Its binary True/False execution result is suited to encoder-only models.Source: §4.2 · p. 5
- POET-Logic synthesizes 1 million premise-conclusion examples and uses Z3 to obtain implication labels, with nearly 16% labeled True.The Boolean statements are sampled from implication forms over five variables.Source: §4.2 · p. 5
4.3 Preliminary Observation
- 9.0% EM gain on DROP follows POET-Math pre-training for BART, while POET-Logic yields a 2.2% EM improvement on LogiQA for RoBERTa.These results support transferring executor-derived reasoning knowledge to natural-language scenarios; Figure 4 reports EM performance on DROP and LogiQA.Source: Figure 4 · p. 5, §4.3 · p. 6
5 POET with Integrated Executors
POET-SQL uses an integrated SQL executor to expose language models to multiple reasoning skills, adapting query-result prediction to encoder-decoder and encoder-only architectures.
5 POET with Integrated Executors
- POET-SQL is designed to master multiple reasoning skills simultaneously because SQL queries encompass a wide variety of computational procedures.This integrated executor addresses the task dependence of POET-Math and POET-Logic.Source: §5 · p. 6
- Encoder-decoder models generate SQL query results, whereas encoder-only models select result tokens from flattened databases through IO sequence tagging.The selection task restricts outputs to query results appearing in the database.Source: §5 · p. 6, Figure 5 · p. 6
- The SQL pre-training corpus contains 5 million generated examples for query-result generation and nearly 2 million filtered examples for query-result selection.Examples instantiate SQUALL templates over WIKISQL databases, with large databases truncated below 450 flattened tokens.Source: §5 · p. 6
6 Experiments and Analysis
POET-SQL improves backbone language models across reasoning benchmarks and transfers execution-trained reasoning from program contexts to natural-language tasks. Analyses indicate that execution pre-training is data-efficient and that its necessity differs from SQL language modeling.
6.1 Experimental Results
- 11.5% on DROP EM and 21.1% on SVAMP EM are the improvements POET-SQLBART achieves over vanilla BART across downstream reasoning tasks.The comparison covers five benchmark datasets and reports significant gains across encoder-only and encoder-decoder backbones.Source: §6.1 · p. 7, Table 3 · p. 7
- POET-SQLBART exceeds PReasM initialized from T5-Large by 8.3%, while POET-Math+SQLBART slightly outperforms the single-executor model.These comparisons support competitive performance against prior reasoning-enhanced models and a modest benefit from mixing program executors.Source: §6.1 · pp. 7–8, Table 4 · p. 7
6.2 Pre-training Analysis
- Replacing program execution with SQL language modeling produces trivial performance variance, corroborating that execution rather than SQL modeling is necessary.The ablation masks SQL queries and trains BART to reconstruct them from masked queries and databases.Source: §6.2 · p. 8
- Both pre-training and downstream performance rise toward an asymptote with more steps, while additional data accelerates convergence.The Figure 6 experiments vary pre-training steps and corpus scale.Source: §6.2 · p. 8, Figure 6 · p. 8
- 10% of the corpus, or 500k examples, approximately reaches the full-data asymptote, indicating high data efficiency and rapid convergence.The full POET-SQL corpus contains 5 million examples.Source: §6.2 · p. 8, Figure 6 · p. 8
7 Discussion and Open Questions
Discussion experiments examine whether POET scales to giant models, preserves language understanding, depends on program naturalness, and transfers bidirectionally between natural-language reasoning and program execution. The results generally support transfer beyond lexical similarity, while revealing a small MNLI drop.
Does POET improve reasoning skills at the sacrifice of NL understanding abilities?
- POET-SQLRoBERTa performs almost equally to RoBERTa on SQuAD and QuoRef, while MNLI drops by 1.2%.The authors suggest joint language-modeling and program-execution pre-training as a possible mitigation.Source: §7 · p. 8
Will POET be affected by naturalness of program context or program?
- Changing program or program-context naturalness does not significantly affect POET, with unnatural programs reducing DROP EM only from 77.7% to 76.9%.The result indicates that the model learns abstract reasoning capabilities rather than merely lexical associations.Source: §7 · p. 9, Table 5 · p. 9
Does pre-training on NL reasoning benefit model learning on program execution?
- DROP pre-training lowers train and dev perplexity for SQL-execution learning, suggesting relative independence between reasoning mechanisms and symbolic representations.Execution accuracy is expected to increase as perplexity decreases.Source: §7 · p. 9, Figure 8 · p. 9
Can POET boost reasoning abilities of giant pre-trained language models?
- POET-SQL still boosts numerical reasoning in T5-11B, although the improvement is less pronounced than for smaller language models.The result is reported across the datasets evaluated for T5-11B and POET-SQLT5.Source: §7 · p. 9, Table 6 · p. 9
8 Conclusion & Future Work
The paper concludes that POET transfers reasoning knowledge from program executors into language models across numerical, logical, and multi-hop reasoning. It positions this transfer as a direction for future research while noting that its mechanism remains insufficiently understood.
8 Conclusion & Future Work
- POET is a pre-training paradigm that boosts language-model reasoning by imitating program executors.The approach is presented as a general paradigm for transferring reasoning capability through execution behavior.Source: §8 · p. 9
- Across six datasets, POET significantly improves numerical, logical, and multi-hop reasoning, with its best model reaching highly competitive performance against specialized systems.The authors identify investigating the causes of reasoning transfer as future work.Source: §8 · p. 9
Limitations
POET’s transfer depends on overlap between pre-training and downstream reasoning skills, and its instantiated templates may limit program diversity. The paper also studies how program context design affects transferability.
Limitations
- POET requires overlap between the reasoning skills learned during pre-training and those required downstream for execution learning to transfer.Removing math-operation programs from POET-SQL produces poor DROP performance.Source: Limitations · p. 10
- Using instantiated program templates instead of probabilistic context-free grammars may reduce program diversity and generalization, although grammar-based synthesis is more complex.This limitation concerns the current program-generation strategy.Source: Limitations · p. 10
A Program Context Analysis
- POET-Math variants are used to test how different program-context designs affect reasoning transferability.The analysis is motivated by an analogy between program-to-context structure and sentence-to-natural-context structure.Source: A Program Context Analysis · p. 14
A.1 The Necessairty of Program Context
The analysis finds that program context is essential for POET-Math, while irrelevant variables are used to approximate noisy natural-language contexts. It also specifies the evaluated datasets, baselines, and task-specific setup for POET-SQLRoBERTa.
A.1 The Necessairty of Program Context
- A dramatic performance drop without program context verifies its importance for POET-Math.Source: §A.1 · p. 15, Table 7 · p. 15
A.2 The Variables Design in Program Context
- Program contexts contain necessary and irrelevant floating-point variables, mirroring irrelevant information commonly present in natural-language passages.Source: §A.2 · p. 15
- POET-Math can learn numerical reasoning without irrelevant variables, but 30 irrelevant variables improve BART-Large more than 10.The authors limit the setting to 30 because it approaches pretrained models’ maximum input length.Source: §A.2 · p. 15, Table 8 · p. 15
B.1 Dataset Setup
- The evaluation covers DROP, HotpotQA, TAT-QA, EQUATE, LogiQA, and SVAMP, spanning numerical, multi-hop, hybrid-context, quantitative-inference, logical, and arithmetic reasoning.HotpotQA uses the distractor setting because retrieval is outside the study’s focus.Source: §B.1 · pp. 15–16
B.2 Baseline Setup
- Baseline comparisons include specialized reasoning systems, language models, and reported state-of-the-art models selected separately for each benchmark.Source: §B.2 · p. 16
C.1 POET-SQLRoBERTa on Different Datasets
- POET-SQLRoBERTa is adapted to each benchmark with task-specific heads or architectures, including TAGOP substitution on TAT-QA and zero-shot EQUATE evaluation after MNLI training.SVAMP uses no encoder-only model because its answers are out of context.Source: §C.1 · p. 16
C.2 Pre-training Details
The paper reports optimizer and training configurations for POET pre-training, model implementation, and benchmark-specific fine-tuning. These details vary by POET instance, model family, dataset, and computational setting.
C.2 Pre-training Details
- POET pre-training uses AdamW with weight decay 0.05 and fp16, while POET-Math, POET-Logic, and POET-SQL use different step budgets and input limits.POET-Math runs 10,000 steps, POET-Logic 5,000, POET-SQLBART and POET-SQLRoBERTa 50,000, and POET-SQLT5 20,000.Source: §C.2 · p. 16
C.3 Fintuning Details
- The models are implemented with Transformers, fairseq, and DeepSpeed, and HotpotQA passage retrieval uses a classifier with 97.2% Hits@3.Source: §C.3 · pp. 16–17
- DROP and SVAMP use character-level number representations and reverse decoding to support arithmetic modeling.Source: §C.3 · p. 17
C.4 Fine-tuning Hyperpameters
- Fine-tuning generally uses three random seeds, except for T5-11B and POET-SQLT5 because of computational limits.Source: §C.4 · p. 17
- DROP fine-tuning differs across architectures, using span-only training for RoBERTa variants and broader dataset coverage for BART-Large and T5-11B.Source: §C.4 · pp. 17–18
- TAT-QA uses TAGOP’s official implementation, while HotpotQA, EQUATE, and LogiQA use dataset-specific step, batch-size, truncation, and learning-rate settings.Source: §C.4 · p. 18
D Fine-grained Analysis
Fine-grained results show that POET-SQL improves numerical reasoning across DROP and EQUATE while preserving other abilities, and transfers from table-like pre-training to hybrid table-and-passage reasoning.
D Fine-grained Analysis
- 15.3% on DROP number questions, 9.8% on date questions, and around 5% on span questions are the gains of POET-SQLBART over vanilla BART-Large.POET-SQLRoBERTa gains 1.9% on span and 3.2% on spans, while POET-SQLT5 gains 2% on number, 2.2% on span, and 0.8% on spans.Source: D Fine-grained Analysis · p. 18
- Around 10% accuracy improvement appears for both POET-SQLBART and POET-SQLRoBERTa on EQUATE’s NR ST subset.Both models also improve on RTE-Q; POET-SQLRoBERTa alone improves on RedditNLI, while other subsets remain approximately comparable.Source: D Fine-grained Analysis · p. 18
- POET-SQLRoBERTa transfers from table-like pre-training to TAT-QA’s hybrid table-and-passage setting.Source: D Fine-grained Analysis · pp. 18–19
E NL Understanding Performance
- The natural-language understanding evaluation uses SQuAD v1.0, matched MNLI, and QuoRef, with sequence-tagging or sequence-classification formulations.Source: E NL Understanding Performance · p. 19, Table 12 · p. 19