PaperlayerView exact revision on arXiv

Source-linked AI summary

Solving Math Word Problems via Cooperative Reasoning induced Language Models

Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Ruyi Gan, Jiaxing Zhang, Yujiu Yang

arXiv:2210.16257v5cs.CL

TL;DR

Math word problems require PLMs to reason and adapt from limited data, but direct generation lacks sufficient supervision. CoRe couples a System 1-like generator with System 2-like verifiers that provide feedback during reasoning. Across multiple mathematical reasoning datasets, it consistently outperforms competing baselines, with up to 9.6% improvement on MultiArith.

  • Problem

    Directly applying PLMs to math word problems can lack sufficient supervision for fast, flexible adaptation from limited data.

    Source: §1 · p. 1

  • Method

    CoRe uses a generator to produce reasoning paths and verifiers to evaluate them and provide feedback throughout solution generation.

    Source: §1 · p. 2

  • Results

    CoRe consistently outperforms competing baselines across multiple mathematical reasoning datasets, with up to 9.6% improvement on MultiArith over state-of-the-art baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CoRe’s cooperative System 1–System 2 framework strengthens reasoning and generalization, including on models dozens of times smaller than competing baselines.

    Source: §4.2.1 · p. 5, §6 · p. 8

    The self-thinking procedure uses only GSM8K’s 7.5K training examples, and experiments do not include larger models such as GPT-3 or PaLM.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large-scale pre-trained language models (PLMs) bring new opportunities to challenging problems, especially those that need high-level intelligence, such as the math word problem (MWPs). However, directly applying existing PLMs to MWPs can fail as the generation process lacks sufficient supervision and thus lacks fast adaptivity as humans. We notice that human reasoning has a dual reasoning framework that consists of an immediate reaction system (system 1) and a delicate reasoning system (system 2), where the entire reasoning is determined by their interaction. This inspires us to develop a cooperative reasoning-induced PLM for solving MWPs, called Cooperative Reasoning (CoRe), resulting in a human-like reasoning architecture with system 1 as the generator and system 2 as the verifier. In our approach, the generator is responsible for generating reasoning paths, and the verifiers are used to supervise the evaluation in order to obtain reliable feedback for the generator. We evaluate our CoRe framework on several mathematical reasoning datasets and achieve decent improvement over state-of-the-art methods, up to 9.6% increase over best baselines. Our codes are available at https://github.com/TianHongZXY/CoRe

1 Introduction

CoRe addresses the difficulty of making PLMs reason and adapt flexibly on math problems by coupling generation with verification. It models this cooperation through System 1-like generators and System 2-like verifiers, and reports stronger results across mathematical reasoning settings.

  • PLMs struggle with fast, flexible reasoning from limited new data because their success can depend on memorized patterns from very large parameter counts.The introduction contrasts this behavior with human reasoning and adaptation.Source: §1 · p. 1
  • CoRe is motivated by the dual-process account in which System 1 provides fast responses and System 2 performs deliberate reasoning through interaction.The paper maps these systems onto generation and verification.Source: §1 · pp. 1–2
  • CoRe introduces feedback during solution generation by pairing a generator for reasoning paths with verifiers that evaluate those paths.The framework includes token-level and sentence-level verification across the solution-generation lifecycle.Source: §1 · p. 2
  • 9.6% improvement on MultiArith over state-of-the-art baselines demonstrates CoRe’s headline gain.The comparison includes zero-shot and fine-tuning methods, including baselines that are dozens of times larger.Source: §1 · p. 2

2 Related Work

Related work uses intermediate reasoning, verifiers, and dual-process ideas to improve language-model reasoning. CoRe extends these directions while targeting smaller models and reducing reliance on extremely large PLMs.

  • Dual-process approaches distinguish a generating System 1 from a deliberate System 2 verifier that checks feasibility and correctness collaboratively.CoRe follows this division while developing a distinct verifier model.Source: §2.1 · p. 2
  • Prior work improves reasoning by training verifiers, generating intermediate processes, or prompting models with Chain-of-Thought examples.These approaches include verifier training and zero-shot or exemplified intermediate reasoning.Source: §2.2 · pp. 2–3
  • Existing Chain-of-Thought and Zero-shot-CoT methods rely on extremely large models and are reported to be unhelpful for smaller models.The paper positions CoRe as capable of working across model sizes while reducing computational cost and inference time.Source: §2.2 · pp. 2–3

3 Cooperative Reasoning

CoRe makes solution generation cooperative by pairing a generator with step- and path-level verifiers, then using their feedback during inference and self-training. The framework comprises cooperative training, cooperative inference, and self-thinking.

  • CoRe comprises three sequential stages: cooperative training, cooperative inference, and self-thinking.Source: §3 · p. 3
  • The generator produces reasoning paths, while step and path verifiers evaluate them at token and whole-path levels.Source: §3.1 · p. 3, §3.2 · p. 3

Cooperative Training

  • During cooperative training, the generator is fine-tuned on question–reasoning-path–answer samples, then generated paths are used to train verifiers.Source: §3.2 · p. 3

Cooperative Inference

  • Cooperative inference uses MCTS to guide generation, combining verifier scores into node rewards and updating rewards through backup.Source: §3.3 · pp. 3–4

Self-Thinking

  • Self-thinking filters generated reasoning paths using verifier scores and generator perplexity, merges selected data with the original dataset, and repeats training and inference until performance saturates.Source: §3.4 · p. 4

4 Experiments

CoRe is evaluated across zero-shot, fine-tuning, guidance, self-thinking, and search settings, consistently improving mathematical reasoning over strong baselines. The experiments attribute gains to verifier feedback, cooperative search, and self-generated training data.

4.2.1 Zero-shot Results

  • CoRe achieves superior zero-shot performance on SingleEq and MultiArith despite baselines using models several dozen times larger.The authors attribute this result to activating reasoning ability in smaller PLMs and using self-produced data to teach Systems 1 and 2.Source: §4.2.1 · p. 5, Table 1 · p. 5

4.2.2 Zero-shot v.s. Fine-tuning

  • CoRe surpasses previous fine-tuned state-of-the-art methods on all four evaluated datasets in a zero-shot setting.The generator alone underperforms most prior methods, while cooperative reasoning substantially boosts the language model's reasoning ability.Source: §4.2.2 · p. 5, Table 2 · p. 6

4.2.3 GSM8K Results

  • In fine-tuning experiments, CoRe outperforms a GPT-3 350B state-of-the-art baseline with fewer parameters, less computation, and shorter inference time.The comparison uses 40 searched paths for CoRe versus 100 sampled solutions for the baseline.Source: §4.2.3 · p. 6, Table 3 · p. 6

4.3.1 Is guidance important during path searching reasoning?

  • Verifier guidance raises SingleOp accuracy from 59.6% to 82.9% and MultiArith accuracy from 92.3% to 96.8%.Increasing the step-verifier reward weight improves both datasets, while path-verifier guidance alone gives a 20.6% absolute gain on SingleOp.Source: §4.3.1 · p. 6, Table 4 · p. 6

4.3.2 How much does self-thinking boost the reasoning ability of a language model?

  • Self-thinking consistently improves performance across search strategies, with increasing iterations improving both greedy decoding and self-consistency on GSM8K.The model reaches saturation after one round in the reported setting, suggesting faster learning from collaborative self-generated data.Source: §4.3.2 · p. 7

4.3.4 How performance varies as the number of search iterations for different search strategies changes?

  • CoRe's accuracy grows faster with search iterations than self-consistency, whose gains quickly saturate after 40 sampled paths.CoRe starts lower because its heuristic continues exploring previously generated paths, then improves with a slower saturation rate.Source: §4.3.4 · p. 7

5 Discussion

Although CoRe is fine-tuned only on GSM8K because few datasets provide intermediate rationales, its zero-shot arithmetic results show transfer to unseen datasets. The discussion therefore identifies cross-dataset generalization within the same domain as a supported scope of the approach.

5 Discussion

  • Fine-tuning only on GSM8K still yields transferable reasoning ability across unseen arithmetic datasets.The authors connect this result to collecting a limited number of question-answer pairs with reasoning paths for a new domain.Source: §5 · p. 8

6 Conclusions

CoRe mimics human dual-system cognition by combining a generator as System 1 with verifiers as System 2, using mutual reinforcement to improve mathematical reasoning.

  • CoRe combines a generator as System 1 with verifiers as System 2, conducting reasoning through their mutual reinforcement.Source: §6 · p. 8

Limitations

CoRe's self-thinking procedure is limited by its reliance on one dataset and by the absence of experiments with larger language models.

  • Self-thinking uses only GSM8K, whose training set contains 7.5K examples because high-quality reasoning-path data are scarce and costly to collect.The authors identify data scarcity, computation costs, and human-resource costs as constraints.Source: Limitations · p. 9
  • The study does not evaluate larger language models such as GPT-3 and PaLM because of usage costs and unavailable open-source code.Source: Limitations · p. 9

Ethics Statement

CoRe may inherit bias and social risks from its PLM backbones, generated data, and downstream data environments.

  • CoRe's ethical risks include biased PLMs, socially stereotyped generated data, and problematic or private downstream data.Source: Ethics Statement · p. 9
  • Harmful PLMs can generate undesirable data that are fed back into language models, potentially deepening bias and unintended social impacts.The authors recommend recording generated sentences, monitoring content, and using human review.Source: Ethics Statement · p. 9

A Dataset Details

The experiments use several mathematical reasoning datasets, with GSM8K supplying human-written reasoning paths for fine-tuning and other datasets supporting zero-shot inference.

Dataset Details

  • GSM8K contains 8.8K human-written grade-school math problems split into 7.5K training and 1.3K test examples, with 2–8-step reasoning paths.Cooperative training and self-thinking use the GSM8K training set.Source: A Dataset Details · p. 12
  • Zero-shot inference uses ASDiv-A, SingleOP, SingleEq, and MultiArith, covering diverse, single-operation, single- or multi-step, and multi-step problems.Source: A Dataset Details · p. 12

Implementation

  • Generator and step verifier training runs for two epochs, while path verifier training runs for three epochs.The supplied implementation passages specify batch sizes, learning rates, schedulers, clipping, temperature, and random seed.Source: §B.1 · p. 12
  • MCTS inference uses 40 maximum search iterations, 20-token expansion steps, 300-token rollouts, and 400 total reasoning-path tokens.Source: §B.1 · p. 12

Dataset Details

  • Table 9 provides the dataset statistics referenced by the dataset-details description.Source: Table 9 · p. 12

B.2 Details of Training Verifiers

Verifier training begins by sampling solutions from the generator, while iterative self-thinking reuses prior weights and limits generated solutions to reduce computational cost.

  • The generator first samples 100 solutions per question before the two verifiers are fine-tuned on the resulting data.This procedure follows the verifier-training process described in Section 3.2 Step 1.3.Source: §B.2 · p. 12
  • Each self-thinking iteration initializes from the previous round’s weights to save computational costs.Source: §B.3 · p. 12
  • Cooperative inference is used to generate higher-quality training data, allowing the number of generated solutions M to be set to 50 for efficiency.The choice is motivated by reduced computational cost and time.Source: §B.3 · p. 12
  • Few-shot baseline prompts follow the settings reported by Kojima et al. (2022) and Wei et al. (2022c).The prompt configurations are listed in Table 10.Source: §B.4 · p. 12

C Extended Experiments

Extended experiments compare CoRe with Cobbe et al. (2021) using GPT-J and show stronger performance across datasets, while the paper scopes its study to arithmetic reasoning.

C Extended Experiments

D Future Work

  • The study measures CoRe’s ability to improve arithmetic reasoning and leaves commonsense, symbolic, and low-resource reasoning for future exploration.The authors describe arithmetic reasoning as the focus of this work and identify other reasoning tasks as future directions.Source: D Future Work · p. 13

Related papers

Loading 2210.16257v5…