PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning

Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, Ashwin Kalyan

arXiv:2209.14610v3cs.LGcs.AIcs.CLcs.CV

TL;DR

Existing mathematical reasoning benchmarks largely use text alone, leaving models’ ability to reason over heterogeneous tabular information uncertain. The paper introduces TABMWP and PROMPTPG, which learns in-context example selection for few-shot GPT-3 via policy gradient. PROMPTPG reaches 68.23% accuracy, improves over existing methods by up to 5.31%, and reduces prediction volatility compared with random selection.

  • Problem

    Existing math word-problem datasets focus mainly on text, leaving mathematical reasoning over heterogeneous tabular information insufficiently evaluated.

  • Method

    The paper introduces TABMWP and PROMPTPG, which uses policy gradient to learn dynamic in-context example selection for few-shot GPT-3.

  • Results

    68.23% overall accuracy on TABMWP lets PROMPTPG outperform previous methods by up to 5.31% and reduce prediction variance compared with random selection.

  • Takeaways & Limitations

    TABMWP evaluates multi-step mathematical reasoning over tabular contexts, while PROMPTPG provides a learned alternative to random or heuristic prompt-example selection.

  • Takeaways & Limitations

    Humans outperform all benchmarks consistently, with a 21.99% average accuracy advantage over PROMPTPG.

Abstract

from arXiv · show

Mathematical reasoning, a core ability of human intelligence, presents unique challenges for machines in abstract thinking and logical reasoning. Recent large pre-trained language models such as GPT-3 have achieved remarkable progress on mathematical reasoning tasks written in text form, such as math word problems (MWP). However, it is unknown if the models can handle more complex problems that involve math reasoning over heterogeneous information, such as tabular data. To fill the gap, we present Tabular Math Word Problems (TabMWP), a new dataset containing 38,431 open-domain grade-level problems that require mathematical reasoning on both textual and tabular data. Each question in TabMWP is aligned with a tabular context, which is presented as an image, semi-structured text, and a structured table. There are two types of questions: free-text and multi-choice, and each problem is annotated with gold solutions to reveal the multi-step reasoning process. We evaluate different pre-trained models on TabMWP, including the GPT-3 model in a few-shot setting. As earlier studies suggest, since few-shot GPT-3 relies on the selection of in-context examples, its performance is unstable and can degrade to near chance. The unstable issue is more severe when handling complex problems like TabMWP. To mitigate this, we further propose a novel approach, PromptPG, which utilizes policy gradient to learn to select in-context examples from a small amount of training data and then constructs the corresponding prompt for the test example. Experimental results show that our method outperforms the best baseline by 5.31% on the accuracy metric and reduces the prediction variance significantly compared to random selection, which verifies its effectiveness in selecting in-context examples.

1 INTRODUCTION

TABMWP introduces large-scale mathematical reasoning problems that combine textual questions with heterogeneous tabular contexts. PROMPTPG addresses unstable few-shot GPT-3 prompting by learning to select in-context examples with policy gradient, improving accuracy and reducing selection instability.

  • Dataset and challenge: Tabular math problems require selecting relevant cells and aligning heterogeneous information before performing multi-step numerical reasoning.This makes them more challenging than existing text-only math word-problem benchmarks.
  • Dataset and challenge: TABMWP contains 38,431 grade-level math word problems with tabular contexts represented as images, semi-structured text, and structured tables.The dataset includes free-text and multi-choice questions, with detailed solutions exposing multi-step reasoning.
  • Prompting challenge: Few-shot GPT-3 performance is highly sensitive to randomly selected in-context examples, with instability potentially worsened by diverse question types and table layouts.The baseline constructs prompts from randomly selected training examples and the test problem.
  • Dynamic prompting: PROMPTPG uses policy gradient to learn dynamically which candidate examples to select while interacting with the GPT-3 API.Its policy network uses fixed BERT representations followed by a learnable linear layer updated with policy-gradient rewards.
  • Results: 68.23% overall accuracy on TABMWP lets PROMPTPG surpass previous methods by up to 5.31% while significantly reducing prediction variance versus random selection.The approach learns prompt construction without manually designed heuristics.

2 THE TABMWP DATASET

TABMWP is a large-scale dataset of math word problems requiring reasoning over semi-structured tabular contexts and questions. It supports diverse answer formats, representations, and annotated multi-step solutions.

  • 2.1 TASK FORMULATION: TABMWP represents each problem as a semi-structured table context paired with an unstructured question, optionally including choices or a unit.Answers are numerical text for free-text questions or spans selected from multiple-choice options.
  • 2.2 DATA COLLECTION AND PREPROCESSING: Each problem includes tabular context in image, semi-structured text, and structured spreadsheet formats.The semi-structured representation separates rows with newlines and columns with vertical bars; the structured format supports retrieval and SQL-based execution.
  • 2.3 DATASET STATISTICS: 38,431 problems are partitioned into training, development, and test splits of 23,059, 7,686, and 7,686 examples.The split ratio is 6:2:2.
  • 2.3 DATASET STATISTICS: 74.7% of questions are free-text and 25.3% are multi-choice, with 28,876 distinct questions, 6,153 answers, and 35,442 solutions.Questions average 22.1 words, indicating substantial diversity in problem and solution forms.
  • 2.4 COMPARISON TO EXISTING DATASETS: TABMWP is described as the first open-domain dataset for math word problems over tabular context and includes natural-language annotations of multi-hop reasoning.The dataset differs from table QA resources because removing either mathematical reasoning or tabular context makes the problems unanswerable.

3 METHODS

The method learns to select in-context examples for GPT-3 instead of relying on unstable random or retrieval-based prompt selection. It uses policy-gradient rewards from answer correctness to improve prompt selection.

  • 3.1 FEW-SHOT GPT-3: Few-shot GPT-3 performance can vary substantially with the selected and permuted in-context examples, especially for TABMWP's heterogeneous problems.TABMWP combines unstructured questions with tabular information, making examples more distinct.
  • 3.2 DYNAMIC PROMPTING VIA POLICY GRADIENT: PROMPTPG learns a policy that selects K examples from a candidate pool for each training problem before prompting GPT-3.The selected examples and problem are combined as the input prompt used to generate an answer.
  • 3.2 DYNAMIC PROMPTING VIA POLICY GRADIENT: The reward is 1 for an answer aligned with the label and −1 otherwise, and the policy maximizes expected reward over generated answers.This reward is computed by evaluating GPT-3's generated answer against the ground truth.
  • 3.2 DYNAMIC PROMPTING VIA POLICY GRADIENT: The expected reward is estimated with Monte Carlo sampling and optimized using the REINFORCE policy-gradient algorithm.The batch size is N, and correct predictions increase the probability of selecting the same prompts while incorrect predictions reduce it.
  • 3.3 POLICY NETWORK: PROMPTPG represents problems and candidate examples with BERT encodings plus a learned linear layer while keeping BERT parameters fixed during training.The appended layer is intended to capture semantic and hidden logical similarity among math problems.

4 EXPERIMENTS

Experiments compare pretrained, fine-tuned, zero-shot, few-shot, and selection-based systems on TABMWP, then analyze input components, training scale, candidate-pool size, and selection strategies. PROMPTPG achieves the strongest reported performance, while results also reveal difficulty differences, indispensable inputs, and remaining human–model gaps.

  • Experimental Results: Few-shot-CoT GPT-3 is the strongest baseline, achieving 62.92% accuracy, while few-shot GPT-3 improves only 0.17% over zero-shot prompting.Zero-shot GPT-3 performs comparably to the best large fine-tuned UnifiedQA and TAPEX baselines.
  • Experimental Results: Models perform better on multi-choice than free-text questions, on lower-grade problems, and on extractive answers than Boolean or other text answers.Grades 7–8 are harder than grades 1–6, while Boolean and other answers require additional fact verification or language understanding.
  • Ablation Study: Removing either tabular context or question text sharply reduces accuracy, with T→A and Q→A averaging 6.10% and 7.00%, respectively.Using both tabular and textual inputs significantly beats heuristic guessing, and the complete input achieves the best performance.
  • Ablation Study: Accuracy rises with more training examples until around 160 examples, then declines with increasing variance; candidate-pool size similarly has a threshold beyond which performance falls.The authors attribute these patterns to inefficient exploitation of additional training data and the difficulty of learning in overly large search spaces.
  • Case Study: PROMPTPG selects more effective and stable prompts than random, heuristic, manual, and nearest-neighbor strategies.Manual selection has zero variance but improves only 1.7% over random selection, whereas PROMPTPG largely reduces instability and selects examples with multiple reasoning steps and similar mathematical abilities.

5 RELATED WORK

Prior MWP datasets largely use text alone, while Table QA datasets address tabular question answering with varying reasoning demands. TABMWP targets mathematical reasoning requiring both textual and tabular information.

  • Math Word Problems: MWP datasets formulate mathematical problems from natural-language descriptions, with prior examples including Math23K, MathQA, ASDiv, SVAMP, and Lila.
  • Math Word Problems: Existing MWP datasets primarily involve textual modality, whereas newer datasets add diagrams as visual context.
  • Table QA Datasets: TABMWP differs by requiring both mathematical reasoning and tabular context for its problems.
  • Methods: The paper evaluates GPT-3 in zero-shot and few-shot settings alongside prior approaches that generate and execute intermediate expressions.
  • Table QA Datasets: Table QA datasets span semi-structured Wikipedia tables, structured databases, and multi-hop reasoning over textual and tabular data.

6 CONCLUSION

The paper introduces TABMWP as a large-scale benchmark for mathematical reasoning in tabular contexts and proposes PROMPTPG for selecting few-shot GPT-3 demonstrations. PROMPTPG improves baseline accuracy and reduces volatility from random example selection.

  • Conclusion: TABMWP contains 38,431 open-domain problems with two question types, three answer types, and multi-step solution annotations.
  • Conclusion: PROMPTPG uses policy gradient to select in-context examples and construct prompts for few-shot GPT-3.
  • Conclusion: PROMPTPG outperforms existing strong baselines by 5.31% and reduces accuracy volatility compared with random selection.
  • Conclusion: The paper presents reinforcement learning for selecting in-context examples for few-shot GPT-3.

A.1 DATASET COLLECTION

TABMWP draws on expert-curated educational problems and applies human filtering and review to enforce its intended multi-hop mathematical reasoning requirements. The appendix also documents diversity in question, answer, and table formats.

  • Dataset Collection: The raw problems come from IXL, an online learning website hosting math problems curated by educational experts.
  • Dataset Collection: Human experts filter problems solvable without the table or through table lookup without numerical reasoning.
  • Dataset Collection: A final human review checks dataset quality and manually revises incorrect annotations.
  • Dataset Collection: The appendix includes tables documenting question-and-answer format diversity and three table formats in TABMWP.

A.2 HUMAN STUDY

The human evaluation uses Amazon Mechanical Turk to assess performance on both free-text and multi-choice TABMWP questions. Workers complete HITs containing exam and test questions under an approval-rate requirement.

  • Human Study: The human study evaluates answers to both free-text and multi-choice questions from the TABMWP test split.
  • Human Study: Each human intelligence task contains 5 exam questions and 15 test questions.
  • Human Study: Workers must have a HIT Approval Rate of 98% or higher and approval for 5,000 tasks.

A.3 THE PROMPTPG ALGORITHM

Algorithm 1 summarizes PROMPTPG’s policy-gradient procedure for learning how to select in-context examples.

  • Algorithm 1 presents the training pipeline for learning a policy that selects in-context examples.

A.4 IMPLEMENTATION DETAILS

The implementation evaluates PROMPTPG against heuristic, GPT-3, UnifiedQA, and TAPEX baselines using specified training and inference settings.

  • Baselines: Heuristic baselines randomly select options or extract numbers from the context and question before choosing a prediction.
  • PROMPTPG: PROMPTPG trains a policy network with REINFORCE by sampling candidate examples, evaluating GPT-3 answers against ground truth, and optimizing reward-weighted log probabilities.
  • Baselines: UnifiedQA and TAPEX are evaluated in pre-trained and fine-tuned settings as general and tabular question-answering baselines.
  • GPT-3 baselines: Zero-shot and few-shot GPT-3 baselines concatenate tabular context, questions, and options, with chain-of-thought variants generating solutions before answers.
  • Experimental settings: Experiments use PyTorch on two Nvidia RTX 3090 GPUs, with Adam optimization and reported settings for batch size, epochs, token limits, and learning rates.

A.5 MORE EXPERIMENTAL RESULTS

Additional experiments examine shot count, example-selection strategies, and representative successes and errors on free-text and multi-choice TabMWP questions.

  • Experimental results: Table 8 reports repeated test-split accuracy experiments comparing random selection with PROMPTPG across experimental settings.
  • Number of few-shot examples: 70.9% accuracy with two in-context shots makes PROMPTPG the highest-accuracy setting, with deviation comparable to random selection using more shots.
  • Number of few-shot examples: Increasing random few-shot-CoT examples from two to four reduces prediction variance and improves accuracy by 2.5%, while five examples provide no further benefit.
  • Selection strategies: PROMPTPG-selected examples share mathematical reasoning abilities with the test example, unlike nearest-neighbor examples selected mainly for semantic similarity.
  • Selection strategies: Randomly selected examples have limited relevance to the test example and can result in an incorrect prediction.
  • Correct predictions: Successful examples cover averaging, multi-item price aggregation, hierarchical tax reasoning, table lookup, multi-hop reasoning, and budget comparisons.
  • Wrong predictions: Observed errors involve incorrect table values, abstract stem-and-leaf interpretation, number ordering, unmatched time slots, long arithmetic chains, and answer-format extraction.
Loading 2209.14610v3…