PaperlayerView exact revision on arXiv

Source-linked AI summary

Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoning

Xuetao Ma, Wenbin Jiang, Hua Huang

arXiv:2502.15401v2cs.CLcs.AI

TL;DR

The paper argues that problem-solving logic can capture intrinsic connections between examples and guide demonstration selection and ordering. It applies this logic in a curriculum ICL strategy, and reports improved average performance and efficiency over previous ICL methods across five datasets.

  • Problem

    The paper seeks a way to select and order ICL demonstrations that reflects intrinsic connections between examples.

    Source: §1 · p. 2

  • Method

    The method selects demonstrations using problem-solving logic, assesses difficulty by the number of steps, and orders examples from easy to hard.

    Source: §6 · p. 9

  • Results

    Across five datasets, the method improves average performance and efficiency over previous ICL methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents problem-solving logic as a criterion for selecting and ordering ICL demonstrations.

    Source: §1 · p. 2

    Hardware constraints limited experiments to 8B-scale LLMs, so the authors call for validation on larger models such as 70B.

    Source: Limitations · p. 9

Abstract

from arXiv · show

In-context learning (ICL) can significantly enhance the complex reasoning capabilities of large language models (LLMs), with the key lying in the selection and ordering of demonstration examples. Previous methods typically relied on simple features to measure the relevance between examples. We argue that these features are not sufficient to reflect the intrinsic connections between examples. In this study, we propose a curriculum ICL strategy guided by problem-solving logic. We select demonstration examples by analyzing the problem-solving logic and order them based on curriculum learning. Specifically, we constructed a problem-solving logic instruction set based on the BREAK dataset and fine-tuned a language model to analyze the problem-solving logic of examples. Subsequently, we selected appropriate demonstration examples based on problem-solving logic and assessed their difficulty according to the number of problem-solving steps. In accordance with the principles of curriculum learning, we ordered the examples from easy to hard to serve as contextual prompts. Experimental results on multiple benchmarks indicate that our method outperforms previous ICL approaches in terms of performance and efficiency, effectively enhancing the complex reasoning capabilities of LLMs. Our project will be released at https://github.com/maxuetao/CurriculumICL

1 Introduction

The paper argues that selecting and ordering demonstrations by problem-solving logic can better capture connections between examples than simple statistical features. It proposes a curriculum ICL strategy and reports improved performance and efficiency across five datasets.

1 Introduction

  • Because ICL can enhance problem-solving without training, it remains valuable when reducing costs or quickly applying models to new tasks.Source: §1 · p. 1
  • Similarity and perplexity measures are insufficient to capture intrinsic connections between examples from a problem-solving perspective.The paper frames demonstration selection and ordering as key to ICL, where examples are learned in sequence.Source: §1 · p. 2
  • The proposed curriculum ICL selects demonstrations with similar problem-solving logic and orders them from easy to hard by their number of steps.The authors use the number of problem-solving steps to assess difficulty.Source: §1 · p. 2
  • The authors construct a BREAK-based instruction set and fine-tune a language model to analyze input questions’ problem-solving logic.Source: §1 · p. 2
  • Across five datasets, the method improves average performance and efficiency over previous ICL methods.The authors report that it effectively enhances LLM reasoning abilities.Source: §1 · p. 2

2 Background

The background introduces in-context learning, QDMR-based problem-solving logic, and curriculum learning. Together, these concepts frame demonstrations as examples that can be selected for their logic and ordered by difficulty.

2.1 In-Context Learning

  • In-context learning uses a few question-and-answer examples to help LLMs learn new tasks, with queries formatted consistently with those examples.Source: §2.1 · p. 2

2.2 Problem-Solving Logic

  • QDMR decomposes complex questions into sub-questions represented by operations; this paper calls the ordered operation sequence problem-solving logic.The paper describes that sequence as encoding the required steps and their order.Source: §2.2 · p. 3

2.3 Curriculum Learning

  • Curriculum learning starts with simple examples and gradually increases difficulty, with difficulty measures depending on task characteristics.Source: §2.3 · p. 3

3 Problem-Solving Logic Guided Curriculum ICL

The method analyzes problem-solving logic to select demonstrations aligned with a query, then orders them by difficulty to build the in-context prompt. Its workflow includes fine-tuning a logic analyzer, matching candidate examples to query steps, and presenting selected examples from easy to hard.

3 Problem-Solving Logic Guided Curriculum ICL

  • The full workflow fine-tunes a logic analyzer, selects query-aligned demonstrations, orders them from easy to hard, and combines them with the query in the final prompt.The instruction set for fine-tuning is constructed from BREAK.Source: §3 · p. 3, Figure 3 · p. 4

3.1 Problem-Solving Logic Analysis

  • The model is fine-tuned on a BREAK-based instruction set whose problem input maps to problem-solving logic and its formal language.The formal language is intended to ensure the model correctly understands the problem-solving process.Source: §3.1 · p. 3

3.2 Curriculum ICL

  • Problem-solving logic provides the foundation for selecting and ordering demonstration examples in the curriculum ICL process.Source: §3.1 · p. 3, §3.2 · p. 4

3.2.1 Demonstration Example Selection

  • A candidate qualifies as a demonstration when its ordered operations exactly match the query’s first n operations, with n no greater than the query’s m operations.This prefix-match rule aligns the candidate's steps with the query and avoids mismatched or additional problem-solving steps.Source: §3.2.1 · p. 4

3.2.2 Demonstration Examples Ordering

  • Examples are ordered from easy to hard by increasing number of problem-solving steps before being combined with the query as the final prompt.The method treats more operations as more reasoning steps and thus greater difficulty.Source: §3.2.2 · p. 5

4 Experiments and Analysis

Experiments compare the method with ICL baselines across five reasoning datasets and three model configurations. Results report higher average accuracy, with analyses linking performance to example difficulty diversity, ordering, and demonstration count.

4.1 Experimental Setup

  • The evaluation validates on five datasets—AQuA, SVAMP, Gsm8k, CommonsenseQA, and StrategyQA—across arithmetic and commonsense reasoning, and evaluates the method on Llama3-8B, Llama3-70B, and Qwen2.5-7B.For fair comparison, baseline example counts follow CoT settings by benchmark, while the method selects no more examples than the baselines.Source: §4.1 · p. 5

4.2 Main Results and Analysis

  • Average accuracy improved by 2.24% for Llama3-8B, 1.48% for Llama3-70B, and 2.15% for Qwen2.5-7B compared with other ICL methods.The method achieved the best reported performance on four benchmarks for Llama3-8B, four for Llama3-70B, and three for Qwen2.5-7B.Source: Table 1 · p. 6, §4.2 · p. 6

4.2.1 Analysis of Example Selection and Ordering

  • The analysis compares four difficulty-sampling strategies and curriculum-based increasing order against random ordering.The sampling strategies prioritize simplicity, prioritize difficulty, select randomly, or prioritize diversity by sampling at most one example per difficulty level.Source: §4.2.1 · pp. 6–7, Table 2 · p. 7
  • Higher difficulty diversity correlated with higher performance, and ordering brought greater improvement as diversity increased.Prioritize diversity saw the largest ordering-related improvement; Figure 6 compares selection-strategy performance and ordering effects against diversity.Source: Figure 6 · p. 7, §4.2.1 · pp. 7–8
  • Prioritize diversity with curriculum ordering achieved 72.37% average accuracy, and problem-solving-logic strategies generally outperformed AL-ICL.Ordering had a particularly pronounced effect for prioritize diversity, but smaller effects for prioritize simplicity and prioritize difficulty.Source: Table 2 · p. 7, §4.2.1 · p. 7

4.2.2 Analysis of the Number of Examples

  • Prioritize diversity used an average of 2.74 demonstrations, while other strategies used more than six, and reduced time cost by 9% to 67%.The paper links fewer examples to shorter context and faster reasoning; it also argues that example quantity is not the only performance factor and cites data diversity as important.Source: §4.2.2 · p. 8

5 Related Work

Related work traces ICL from fixed demonstrations to methods for selecting and ordering examples. It also describes curriculum learning in LLM fine-tuning and in ICL, where demonstration difficulty can increase from easy to hard.

5.1 In-Context Learning

  • ICL research moved from manually designed CoT demonstrations to methods that select and order examples using clustering, similarity, retrieval, probe-set entropy, relevance, or length.The cited approaches include AutoCoT, PromptSO, trained retrieval, Lu et al., and KATE.Source: §5.1 · p. 8

5.2 Curriculum Learning in LLMs

  • Curriculum learning has been applied to LLM fine-tuning by progressing from easy to hard, including educational-stage and medical question-answering settings.Other work uses sequence length as a difficulty measure.Source: §5.2 · p. 8
  • ICCL applies curriculum learning in ICL by assessing demonstration difficulty with human- or LLM-driven metrics and gradually increasing difficulty.Source: §5.2 · p. 9

6 Conclusion

The conclusion presents an ICL strategy that selects examples by problem-solving logic and orders them from easy to hard using the number of problem-solving steps. Across multiple benchmarks, the method outperforms other ICL methods in average performance.

  • The conclusion reports that problem-solving-logic-guided selection and easy-to-hard ordering outperform other ICL methods in average performance across multiple benchmarks.Example difficulty is assessed by the number of problem-solving steps.Source: §6 · p. 9

Limitations

The evaluation was limited to 8B-scale LLMs and did not test curriculum learning with a large number of examples. The authors identify larger-model and many-example evaluation as future work.

Limitations

  • Experiments covered only 8B-scale LLMs, leaving the method's effectiveness on larger models such as 70B unvalidated.The authors cite hardware resource constraints as the reason.Source: Limitations · p. 9
  • Benchmark and hardware constraints prevented evaluating curriculum learning with a large number of examples.The authors identify evaluation with many examples as future work.Source: Limitations · p. 9

Potential Risks

The authors report no obvious risks associated with this work.

Potential Risks

A BREAK Dataset Description

BREAK represents questions as sequences of QDMR steps, using 13 operation types. The authors used BREAK as the basis for an instruction set to analyze problem-solving logic.

A BREAK Dataset Description

B Fine-Tuning Details

The method's logic-analysis model was fine-tuned with LoRA on Llama3-8B using the instruction set. The paper also provides prompt templates and benchmark-specific in-context examples.

B Fine-Tuning Details

  • LoRA fine-tuning used the problem-solving-logic instruction set on Llama3-8B.Training ran for four epochs with AdamW; the best model was selected by BLEU score.Source: B Fine-Tuning Details · p. 13

C Prompt Template

  • The paper provides prompt templates for fine-tuning and full in-context prompt examples for different benchmarks.Source: C Prompt Template · p. 13

D Supplementary Details

The supplementary details report the experimental setup and provide tables covering QDMR operators, BREAK-based instruction sets, and full in-context prompts for several datasets.

D Supplementary Details

  • Experiments used llama-factory with a 4096-token prompt, a 1024-token maximum answer, and temperature 0.01.The reported hardware was an Intel Xeon Platinum 8358 CPU and an NVIDIA Tesla A800 80G GPU; hyperparameters followed llama-factory's default configuration.Source: D Supplementary Details · p. 13
  • Tables 4–6 describe 13 QDMR operator types, their prevalence in BREAK, and the distribution of QDMR sub-question counts.Table 4 includes operator templates, signatures, and example questions.Source: Table 4 · p. 14, Table 5 · p. 14, Table 6 · p. 14
  • The cited material identifies Table 7 as examples and explanations of instruction sets based on the BREAK dataset, and Table 8 as fine-tuning prompts for problem-solving logic analysis.Source: Table 7 · p. 15, Table 8 · p. 15
  • Full in-context prompt examples are provided for AQuA, GSM8K, ComSenQA, StrategyQA, and SVAMP.Source: Table 9 · p. 16, Table 10 · p. 17, Table 11 · p. 18
Loading 2502.15401v2…