Source-linked AI summary
Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems
Wang Ling, Dani Yogatama, Chris Dyer, Phil Blunsom
TL;DR
The paper addresses the difficulty of inducing arithmetic programs directly from algebraic word problem question-answer pairs. It generates answer rationales as scaffolding for program structure, introduces a 100,000-problem dataset, and reports improved rationale fluency and problem-solving ability over existing neural models.
Problem
Inducing arbitrarily complicated arithmetic programs directly from algebraic word problem question-answer pairs is difficult.
Method
The model generates natural-language rationales interspersed with mathematical expressions, then uses heuristic search to infer programs that produce the rationale and answer.
Results
The method outperforms existing neural models in rationale fluency and problem-solving ability, while baseline accuracy remains near chance at 20%.
Takeaways & Limitations
Answer rationales provide indirect supervision that supports inducing arithmetic programs for algebraic word problems.
Takeaways & Limitations
Generating complex rationales correctly remains unsolved because each additional step increases inference and decoding complexity.
Abstract
from arXiv · showhide
Solving algebraic word problems requires executing a series of arithmetic operations---a program---to obtain a final answer. However, since programs can be arbitrarily complicated, inducing them directly from question-answer pairs is a formidable challenge. To make this task more feasible, we solve these problems by generating answer rationales, sequences of natural language and human-readable mathematical expressions that derive the final answer through a series of small steps. Although rationales do not explicitly specify programs, they provide a scaffolding for their structure via intermediate milestones. To evaluate our approach, we have created a new 100,000-sample dataset of questions, answers and rationales. Experimental results show that indirect supervision of program learning via answer rationales is a promising strategy for inducing arithmetic programs.
1 Introduction
The paper makes algebraic word problems more tractable by generating answer rationales that expose intermediate solution structure while producing the final answer. It introduces a rationale-centered program-induction approach and a dataset containing questions, answers, and rationales.
- Algebraic word problems require mapping real-world goals, entities, and quantities into mathematical form, computing a solution, and mapping it back to the world.
- The paper models programs that generate natural-language rationales interspersed with algebraic expressions before selecting the answer.The rationale provides a coarse guide to the arithmetic program's structure.
- Rationales serve as latent variables that give rise to answers, making them more faithful to computational steps than post-hoc plausible explanations.
- The authors created a dataset with more than 100,000 algebraic word problems containing answers and natural-language rationales.
- The proposed learner uses heuristic search because the target program is not provided and the rationale-constrained program space remains large.
- State-of-the-art sequence-to-sequence models perform near chance, whereas the proposed model doubles baseline accuracy.
2 Dataset
The dataset combines seed multiple-choice math problems with crowdsourced variants and annotates each example with a question, options, rationale, and correct answer. After heldout removal and filtering, it contains 100,949 training problems spanning varied topics, difficulties, and lengths.
- The dataset contains 100,000 problems annotated with a question, answer options, rationale, and correct option label.Examples include equation solving, multi-step arithmetic, and testing answer options.
- The construction begins with 34,202 seed multiple-choice math questions covering broad topics and difficulty levels.The seeds include questions from exams such as the GMAT and GRE.
- Crowdsourced workers create similar questions while changing the answers and rationales to avoid direct paraphrases.The resulting items include minimally altered replicas and more radically different problems in the same topic.
- The collection comprises 104,519 problems, including 34,202 seed and 70,318 crowdsourced problems, before heldout removal.
- After removing 500 development and test problems and filtering near-duplicate training instances, 100,949 problems remain for training.
- Most examples contain 30 to 500 total tokens, while some exceed 1,000 tokens.
3 Model
The model generates rationales by executing latent programs that interleave natural-language output with arithmetic operations, rather than solving only for the final answer. Programs select operations, arguments, and destinations while using copying and memory to support multi-step mathematical reasoning.
- Model motivation: Rationale generation requires explaining intermediate algebraic steps, not merely obtaining the final solution.For example, solving (27x + 17y)/(x + y) = 23 requires generating intermediate equations before deriving x/y = 3/2.
- Problem definition: The model predicts a rationale and answer from the problem and answer options, appending the selected option after the rationale.The options are included because some problems can be solved by elimination or by testing candidate answers.
- Program representation: A latent instruction sequence z is executed to generate the rationale output y, using the input x and memory buffer m.The program can access x, y, and m, placing generated tokens into the output vector or memory during execution.
- Operation set: The system defines 22 operations, including arithmetic, mathematical, conversion, and checking operations for constructing executable rationales.Frequently used operations include Add, Subtract, Multiply, Divide, Power, Log, Sqrt, trigonometric functions, Factorial, and Choose.
- Instruction structure: Each instruction contains an operation, ordered arguments, a result destination, and the computed value.The operation comes from a predefined set, and its argument count determines the required argument sequence.
- Argument generation: Arguments are generated from the vocabulary, copied from the input, or copied from previously generated output and memory values.A latent variable selects among SOFTMAX, COPY-INPUT, and COPY-OUTPUT for each argument.
4 Inducing Programs while Learning
Because the instruction sequence generating the rationale and answer is unobserved, the model marginalizes over programs and uses rationale progression to constrain program induction. It further filters executable instruction paths during learning and decoding, while random sampling remains unlikely to find answer-generating programs.
- Latent program induction: The instruction sequence z that generates the observed output y is unobserved, so training optimizes a marginal probability over possible programs.Programs are grouped by whether execution generates y; exact marginalization is intractable and is approximated using generated samples.
- Latent program induction: Randomly sampling instructions from the RNN distribution is unlikely to produce a program that generates the target answer.
- Rationale-guided search: Rationale progression reduces the latent search burden by making each successive rationale token require fewer operations than generating the final answer directly.The rationale is assumed to solve the problem step by step, such as computing combinations before the final probability.
- Rationale-guided search: The model filters candidate instructions to those with at least one path capable of generating the next rationale token.Candidates are tested using instruction combinations with at most one level of indirection.
- Executable decoding: During decoding, each hypothesis instruction is executed and removed when it causes an error, preventing unexecutable code such as log(0).
5 Staged Back-propagation
Long rationale programs create memory and computation challenges because attention and copy mechanisms must process growing instruction sequences. Staged back-propagation addresses this by training on token slices while retaining global sequence context.
- Challenge: Rationales longer than 200 tokens are common, and intermediate instructions can make the program length z exceed 400.These lengths create a practical training challenge.
- Challenge: Attention and copy mechanisms require operations involving an exponential number of matrix multiplications as the attendable sequences grow.
- Method: Staged back-propagation trains on slices of K tokens instead of unrolling the full instruction sequence at once.For |z| = 300 and K = 100, training uses three mini-batches covering successive 100-token slices.
- Method: The slice length K is adjustable so memory-intensive attention and copy computations fit within the available computation graph memory.
- Global context: Unlike truncated back-propagation, the method retains global context by constructing the full state sequence before extracting a slice for attention.The attention vector is incorporated after state propagation, a strategy called late fusion.
6 Experiments
The experiments evaluate rationale quality and answer accuracy against sequence-to-sequence baselines. The program-driven model improves rationale generation and solves problems above chance, but complex multi-step rationales remain difficult.
- The evaluation measures rationale quality with perplexity and BLEU, and answer performance with accuracy.
- Perplexity: Input copying substantially improves rationale perplexity by allowing models to reuse values from the question, while output copying adds only a slight further improvement.Output copying helps when values recur after their first occurrence.
- Perplexity: The proposed model improves over baselines because algebraic manipulation lets it generate new values that copying mechanisms cannot obtain.Examples include intermediate values such as 0.025, 0.023, 0.002, and the answer E.
- BLEU: The regular sequence-to-sequence model has low BLEU, copying improves it by enabling variable definition, and the proposed model achieves the highest BLEU by generating intermediate and final values.
- Accuracy: Baseline accuracies are close to chance at 20%, whereas the proposed model solves problems at a significantly higher rate.Correctly solved problems generally require only one or two operations.
- Discussion: Generating complex rationales remains unsolved because each additional step increases complexity during inference and decoding.
7 Related Work
Related work includes algebraic problem solvers, sequence-to-sequence and program-generation models, and systems that produce explanations. This work differs by using rationales to guide latent program induction rather than generating post-hoc explanations.
- The model is presented as a single generative system that solves the problem while explaining its approach.
- The approach combines sequence-to-sequence transduction with semantic parsing and program-generation ideas, including external memory, operators, and copying.
- Unlike prior explanation systems that generate descriptions after predicting an answer, this work uses rationales to guide program induction.
8 Conclusion
The paper addresses math problems by generating both correct answers and descriptions of the solving method. Its approach combines rationale generation with arithmetic operations during decoding and outperforms existing neural models in rationale fluency and problem-solving ability.
- 100,000 question-and-rationale pairs were collected for math problems requiring both correct answers and descriptions of solution methods.
- The proposed model generates natural language and performs arithmetic operations in the same decoding process.
- The method outperforms existing neural models in rationale fluency and ability to solve the problem.