PaperlayerView exact revision on arXiv

Source-linked AI summary

StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models

Jinghan Tan, Yuanzheng Wang, Lu Chen, Zijun Chen, Yuqian Wang, Maosong Sun

arXiv:2608.23475v1cs.AI

TL;DR

Direct few-shot ICL can be sensitive to example construction because models may not abstract stable task rules. The paper introduces StrategyBench to evaluate explicit strategy induction through strategy quality and downstream utility, finding that utility depends on task type, example configuration, and generator-executor matching. Experiments also show that supervised fine-tuning improves the stability and generalization of strategy generation.

  • Problem

    Direct ICL may rely on surface patterns rather than stable task rules, while systematic benchmarks for evaluating general task-level strategy induction remain limited.

  • Method

    StrategyBench selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and evaluates generated strategies using text quality and downstream application utility.

  • Results

    Explicit strategy utility differs across task types and depends on example configuration and generator-executor matching, while supervised fine-tuning improves strategy-generation stability and generalization.

  • Takeaways & Limitations

    Strategy induction is useful for task solving, but its benefits depend on the task and on how strategies are generated, formatted, and executed.

  • Takeaways & Limitations

    The benchmark mainly uses BIG-Bench and BBH, and experiments mainly evaluate Qwen3 models, limiting coverage of broader task distributions and model families.

Abstract

from arXiv · show

As large language models are increasingly used in data-scarce and evolving task scenarios, few-shot in-context learning (ICL) has become a key paradigm for task adaptation. However, direct ICL often uses a small set of examples without explicitly abstracting task rules, making it sensitive to example construction. In contrast, human learners often reduce such sensitivity by first summarizing task rules from examples and then applying them to new instances. To evaluate this ability, we propose StrategyBench, which selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility. We further analyze strategy induction from three perspectives: task variation, model configuration, and adaptation setting, covering category-wise differences, generator-executor choices, demonstration design, and SFT-based adaptation. Experiments show that explicit strategy utility differs substantially across task categories and depends on both strategy generation and execution conditions. The benchmark is released at: https://anonymous.4open.science/r/StrategyBench-D53C.

1 Introduction

Direct few-shot ICL can depend on surface patterns and example construction, motivating explicit task-level strategies. StrategyBench evaluates strategy quality and downstream utility while analyzing how task, model, and adaptation conditions affect strategy induction and application.

  • Motivation: Direct few-shot ICL may rely on surface patterns or shortcuts instead of stable task rules, limiting robustness when examples are scarce or varied.Limited context windows restrict the number and diversity of examples available to the model.
  • Motivation: Strategy-based ICL first abstracts explicit task rules from few-shot examples and then applies the resulting strategy to unseen questions.This makes the task rules explicit rather than relying only on direct prediction from demonstrations.
  • Benchmark: STRATEGYBENCH selects strategy-inducible BIG-Bench tasks and constructs reference strategies for evaluating explicit task-level strategy induction.The benchmark supports both evaluation and training with reference strategies.
  • Evaluation: The evaluation framework measures both strategy text quality and downstream application utility.Assessing strategies requires examining clarity and abstraction alongside reliable application and unseen-example performance.
  • Experimental analysis: Readable strategies do not always yield better execution, and utility depends on the interaction among the generator, executor, and format constraints.The experiments analyze model scale, few-shot examples, format constraints, supervised fine-tuning, and strategy executors.

2 Preliminaries

The paper formulates few-shot episodes around demonstrations and unseen examples, contrasting direct prediction with a two-stage strategy-based process. In the latter, a generator induces a task-level strategy that an executor uses to answer new queries.

  • Task and episode formulation: Each task contains problems sharing an input-output format and common underlying rules, with episodes including demonstrations and unseen examples.The formulation represents examples as question-answer pairs and evaluates models on unseen instances.
  • Direct ICL: Direct ICL predicts each unseen answer directly from the demonstrations and the new query.The task-solving model produces the answer without an explicit intermediate strategy.
  • Strategy-based ICL: Strategy-based ICL separates solving into strategy induction and strategy application.A generator first induces a task-level strategy from demonstrations, and an executor then uses it with the unseen query.

3 Benchmark Construction

STRATEGYBENCH is constructed from filtered BIG-Bench tasks, organized by dataset splits and ability-based categories, and paired with utility-filtered reference strategies.

  • Task filtering: The benchmark filters BIG-Bench tasks using model-assisted pre-filtering and human verification for objective scorability, format stability, rule inductibility, low external-knowledge dependence, and strategy application potential.
  • Dataset splitting: Filtered data are normalized into unified input and target fields, then split into training, in-distribution test, out-of-distribution test, and an additional BBH challenge set.
  • Task categorization: Retained tasks are grouped into Numerical, Logic, Language, Spatial, Procedural, and Induction categories using unsupervised clustering followed by human verification.
  • Reference strategy construction: Candidate strategies are generated for each task category and filtered by unseen-example execution against Direct ICL to form Seff, then by statistical significance to form Ssig.

4 Evaluation Metrics

The evaluation framework measures strategy content quality separately from application utility, which captures correctness, stability, and answer efficiency on unseen examples.

  • Framework: The framework evaluates strategy application utility and strategy content quality as complementary dimensions.
  • Strategy application utility: Application utility comprises correctness, stability, and answer efficiency on unseen examples.
  • Correctness: Correctness, measured by Corr, is the fraction of predictions matching the gold answer across three sampling runs for the same query.
  • Stability: Stability measures consistency regardless of correctness through sampling consistency across repeated runs and robustness consistency under few-shot example-order perturbations.
  • Efficiency: Answer efficiency rewards correct answers within token budgets and discounts correct answers requiring larger input or output token costs.
  • Strategy content quality: Content quality evaluates conciseness, which penalizes repeated semantic content, and format compliance, which checks adherence to the predefined strategy format.

5 Experimental Setup

Experiments compare direct, chain-of-thought, and least-to-most few-shot prompting with Qwen3 backbones under specified generator, executor, sampling, and fine-tuning settings.

  • Baselines: The study compares Direct ICL, CoT-prompted ICL, and Least-to-Most Prompting as example-based, implicit-reasoning, and explicit-intermediate-reasoning paradigms.
  • Models: Qwen3-1.7B, 4B, 8B, and 14B serve as backbone models, with Qwen3-8B as the default strategy generator and Qwen3-4B as the default executor.
  • Evaluation settings: Prompting-based inference uses k = 3 few-shot examples, while stability evaluation samples each query three times and applies M = 3 random example-order perturbations.
  • Adaptation: For fine-tuning, the study constructs SFT data from few-shot examples and reference strategies and applies LoRA for parameter-efficient training.

6 Results and Analysis

The results analyze explicit strategy induction across task categories, model configurations, demonstration sizes, prompt formats, and SFT. Explicit strategies show their strongest utility when generation and execution conditions are well matched, especially when combined with step-by-step reasoning.

  • RQ1: Task Categories: CoT+Ours achieves the best results on most task categories and metrics, with clear gains on Numerical, Logic, and Language tasks.The combination also improves output stability and answer efficiency on Stasamp, Starob, LenEffin, and LenEffout.
  • RQ1: Task Categories: Explicit strategies provide task-level rule constraints, while CoT strengthens instance-level reasoning, making the approaches complementary.Together, they use task-level rules and step-by-step reasoning to produce more stable gains across most task categories.
  • RQ1: Task Categories: Ours alone does not always outperform CoT or L2M, especially on Numerical and Procedural tasks.Explicit strategies function better as high-level constraints and cannot fully replace instance-level reasoning or multi-step execution.
  • RQ2: Model Scale: Qwen3-14B produces the best text quality with a fixed executor, whereas Qwen3-8B often yields stronger execution performance.With a fixed generator, Qwen3-4B achieves the best execution results under most settings, while scaling the executor to Qwen3-8B does not consistently improve correctness.
  • RQ2: Few-shot Number: Increasing the number of few-shot examples makes generated strategies more detailed and helps the model answer questions more effectively.Such strategies may become less concise or exceed length limits, suggesting that detailed strategies can be more useful than concise ones.
  • RQ2: Prompt Variant: The free-form prompt achieves higher Fmat., while the structured prompt performs better on Corr. and Stasamp.Format compliance does not necessarily imply stronger downstream performance because utility also depends on capturing task-relevant rules.
  • RQ3: SFT: Ours+SFT consistently improves Conc. and Fmat. over Ours on ID, OOD, and BBH, while also improving Stasamp and Starob.However, Ours+SFT does not improve Corr. and can slightly decrease it, indicating that greater stability and format compliance do not guarantee higher single-run correctness.
  • RQ3: SFT: SFT increases stability and controllability while preserving some exploratory capability, supporting its potential as an initialization for reinforcement learning.Stability improves but remains far from 100%.

7 Related Work

Related work covers few-shot ICL, benchmark-based and process-level evaluation, and concurrent strategy-based applications. Existing benchmarks primarily assess final answers, while this paper focuses on strategy induction and utility.

  • Few-shot In-context Learning: Few-shot ICL adapts large language models to new tasks using a small number of examples without parameter updates.Prior work finds ICL sensitive to example selection, semantic relevance, example order, context length, and example organization.
  • Benchmarks for In-context Learning: Existing benchmarks mainly evaluate final answer accuracy or task scores and provide limited evaluation of reasoning processes.Recent process-level studies address reasoning traces and step verification, but do not by themselves establish task-level strategy evaluation.
  • Concurrent Work: A concurrent anonymous study uses STRATEGYBENCH and an SFT model trained from its reference strategies to develop CoDI for OOD generalization.The concurrent work is described as complementary because this paper focuses on benchmark construction and evaluation.

8 Conclusion

STRATEGYBENCH evaluates explicit strategy induction through strategy quality and downstream utility, while experiments show utility depends on task and generation–execution conditions. The paper also identifies broader task coverage and finer failure analysis as future directions.

  • STRATEGYBENCH evaluates explicit strategies using metrics for strategy text quality and downstream application utility.
  • Explicit strategy utility depends on task type, example configuration, and the match between strategy generator and executor.
  • Supervised fine-tuning improves the stability and generalization of strategy generation.
  • Future work will extend the benchmark to real-world, interactive, and tool-use task sources.
  • Future work will distinguish strategy-generation failures from execution failures and improve strategy transferability and executability.

9 Limitations

The paper’s limitations concern restricted data and model coverage, incomplete attribution of strategy failures, and a narrow focus of fine-tuning experiments.

  • Methodology: STRATEGYBENCH is mainly built from BIG-Bench and BBH, limiting evaluation across broader task distributions.
  • Methodology: The evaluation does not fully distinguish errors caused by inaccurate strategies from failures to understand or apply them.
  • Experiments: The experiments mainly evaluate models from the Qwen3 family and do not cover more model families or closed-source models.
  • Experiments: Fine-tuning experiments focus on supervised fine-tuning and do not explore reinforcement learning, preference optimization, or multi-stage curriculum learning.

A Task Selection and Standardization Details

The benchmark filters BIG-Bench tasks for objective, stable, rule-inducible, low-knowledge-risk, and transferable settings, then standardizes inputs and targets for unified evaluation.

  • Task filtering: The construction removes tasks with dynamic evaluation, subjectivity, or unsuitable formats before strategy-induction selection.
  • Task filtering: Candidate tasks are scored on objective scorability, format stability, rule inductibility, external knowledge risk, and strategy transfer potential, followed by human verification.
  • Selection dimensions: Objective scorability measures whether evaluation criteria are clear and stable, assigning higher scores to exact-match or clearly multiple-choice tasks.
  • Selection dimensions: Format stability measures whether a task’s input-output structure remains fixed and clear within the task or subtask.
  • Selection dimensions: Rule inductibility measures whether stable rules, steps, or algorithms can be abstracted from a small number of examples.
  • Standardization: Normalization merges scattered task information into unified inputs and converts answers into unified targets, including option IDs where appropriate.

B Dataset Splitting and Sampling Details

STRATEGYBENCH separates training, in-distribution, out-of-distribution, and BBH evaluation settings, uses balanced k-cut sampling, and groups tasks into six ability-based categories.

  • Dataset splits: Training data supports demonstrations, reference-strategy generation, and development, while test data evaluates induction and application under different distributions.
  • Dataset splits: In-distribution tests use non-overlapping examples from training subtasks to assess strategy induction under consistent rules and formats.
  • Dataset splits: Out-of-distribution tests vary task source, input-output format, or required ability to examine transferable strategy induction.
  • Dataset splits: BBH provides an additional challenging test set for complex reasoning, symbolic manipulation, and multi-step problem solving.
  • Balanced sampling: Balanced k-cut sampling limits each collection to at most k examples, preventing large tasks from dominating and balancing task influence.
  • Task categories: Tasks are grouped by required abilities and solving processes into Numerical, Logic, Language, Spatial, Procedural, and Induction categories.
  • Reference strategies: Reference strategies are generated from few-shot examples and filtered according to downstream execution performance.

E Implementation Details

The implementation details specify strategy-generation, strategy-application, and supervised-fine-tuning settings, alongside prompt templates and related framework resources. They also document the SFT data format and the paper’s use of writing assistance.

  • Strategy generation settings: Strategy generation uses Prompt V1 by default, k = 3 few-shot examples, one candidate per bucket, temperature 0.2, and a 700-token maximum.The interface allows at most 6 retries with a 0.2-second interval between retries.
  • Strategy application settings: Strategy application feeds the induced strategy and a new question to an executor instructed to output only the final answer.Default application decoding uses temperature 0.7, a 256-token maximum, up to 3 retries, and concurrency of 8.
  • Supervised fine-tuning: SFT data pairs the strategy-generation instruction and few-shot QA block with the corresponding reference strategy as the supervised target.The experiments use LoRA-based parameter-efficient fine-tuning; the training model is Qwen3-8B.
  • Use of AI writing assistance: GPT-based writing assistance was limited to polishing author-written language and was not used for research ideas, results, analyses, citations, or conclusions.The authors state that all content was reviewed, revised, and verified by them.
  • Prompt templates: Prompt V1 is structured with explicit output fields, whereas Prompt V2 is free-form and emphasizes concise, executable task-level strategies.Prompt V1 is the default strategy-generation prompt in the main experiments.
  • Related framework: The paper distinguishes its benchmark-oriented contribution from CoDI, a concurrent downstream framework that separates strategy induction from answer execution.CoDI uses StrategyBench and an SFT model trained from reference strategies, while the paper focuses on benchmark construction, reference strategies, and evaluation.
Loading 2608.23475v1…