PaperlayerView exact revision on arXiv

Source-linked AI summary

SALA: Semantic-Aware Logical Alignment for Complex Reasoning in In-Context Learning

Zhao Ji, Wenqing Chen, Zhixuan Chu, Jianxing Yu, Jingping Liu, Shanhe Zhao, Zibin Zheng

arXiv:2609.02336v1cs.AIcs.CL

TL;DR

Complex-reasoning ICL needs demonstrations selected by compatible problem-solving logic, but surface similarity obscures that logic and fixed symbolic matching limits flexible comparison. SALA induces task-adaptive operations, embeds them semantically, and aligns operation sequences with DTW; experiments across four benchmarks and three LLMs show consistent gains over strong baselines.

  • Problem

    Surface-similarity retrieval can obscure problem-solving logic, while fixed operation spaces and exact matching limit generalization across task-specific and semantically equivalent reasoning patterns.

  • Method

    SALA induces task-adaptive reasoning operations from downstream data, represents problems as explicit operation sequences, and aligns them in semantic space with dynamic time warping.

  • Results

    Across four reasoning benchmarks and three LLMs, SALA consistently outperforms strong demonstration-selection baselines.

  • Takeaways & Limitations

    SALA combines explicit reasoning representations with flexible semantic matching for reasoning-oriented demonstration selection.

  • Takeaways & Limitations

    SALA's induced operations and parsed sequences may vary with the induction model and prompting strategy, while its linear representation may not suit more complex tasks requiring richer structures.

Abstract

from arXiv · show

Effective in-context learning (ICL) for complex reasoning relies on selecting the right demonstrations. Traditional retrieval methods based on surface similarity fail to capture the underlying problem-solving logic. Recent logic-based methods address this by matching predefined reasoning steps, but the rigid rules and exact-match criteria is improper to handle flexible or diverse reasoning processes. To address the problem, we propose SALA, a Semantic-Aware Logical Alignment framework. Instead of relying on a fixed inventory, SALA automatically learns task-specific reasoning operations. It then embeds these operations into a continuous semantic space and uses dynamic time warping (DTW) to align the reasoning sequences. This approach allows for soft, flexible matching of reasoning logic while remaining highly interpretable. Experiments across four reasoning benchmarks and three LLMs demonstrate that SALA outperforms existing demonstration selection methods. Further analysis confirms the roles of the operation induction and the logical semantic alignment.

1 Introduction

SALA addresses the limits of surface-similarity retrieval and rigid symbolic matching by representing reasoning explicitly while aligning it flexibly in semantic space. Across four reasoning benchmarks and three LLMs, it consistently improves demonstration selection over strong baselines.

  • Surface-similarity retrieval can obscure reasoning logic, while mismatched or noisy demonstrations may introduce reasoning biases and degrade performance.
  • Existing reasoning representations trade off flexibility and comparability: free-form paths can be noisy, whereas fixed symbolic operations limit adaptation and exact matching.
  • SALA induces task-adaptive reasoning operations from downstream data when predefined operations cannot fully express a question's logic.
  • SALA embeds operation descriptions in a continuous semantic space and uses dynamic time warping to align reasoning-operation sequences beyond exact symbolic correspondence.
  • SALA achieves better average performance than similarity-based, learning-based, and reasoning-aware selection baselines across four reasoning benchmarks and three LLMs.Ablations confirm contributions from task-adaptive operation construction and semantic sequence alignment.

2 Related Work

Demonstration selection has progressed from lexical and semantic retrieval toward reasoning-oriented representations, but existing approaches remain limited by noisy free-form reasoning or rigid symbolic matching. SALA combines adaptive operations with semantic DTW alignment to compare problem-solving logic flexibly.

  • Early retrieval methods use lexical overlap or sentence-level semantic similarity, later incorporating supportiveness, diversity, compositionality, or learned retrieval signals.
  • These methods improve useful-example retrieval, but their signals remain mainly based on input similarity or model feedback rather than the reasoning process itself.
  • Natural-language reasoning descriptions and generated paths preserve flexibility but can make retrieval noisy or unstable.
  • Structured approaches use reasoning patterns, latent skills, graphs, or semantic-inference balances to represent reasoning for exemplar retrieval.
  • PSL represents problems with predefined QDMR-style operation sequences and exact operation-level matching, but depends on a fixed operation space and rigid symbolic matching.
  • SALA extends symbolic representations with task-adaptive operations and DTW-based semantic alignment, enabling flexible comparison across diverse reasoning processes.

3 Methodology

SALA selects demonstrations by representing problem-solving logic as task-adaptive operation sequences and aligning those sequences semantically. Its methodology combines operation induction and deduplication with embedding-based dynamic time warping.

  • 3.1 Problem Formulation: SALA formulates demonstration selection as ranking candidates by the alignment between query and candidate problem-solving operation sequences.An LLM parses each question into an ordered sequence from the task-adaptive operation set, and similarity scores determine the selected demonstrations.
  • 3.2 Operation Construction: The task-adaptive operation set extends 13 predefined QDMR operations with operations induced from downstream training data.Candidate operations are induced only when the predefined set cannot cover a question’s reasoning logic.
  • 3.2 Operation Construction: SALA removes redundant induced operations through heuristic name-based deduplication followed by LLM-based functional duplication detection.The first stage normalizes names and applies containment rules; the second sequentially judges functional overlap against the evolving operation set.
  • 3.3 Embedding Library: Operation descriptions are encoded into normalized continuous vectors that form a reusable reasoning operation embedding library.A pretrained text embedding model encodes description texts, and the resulting operation-to-vector mappings are precomputed for retrieval.
  • 3.4 Semantic Alignment: SALA maps parsed query and demonstration operations to embedding sequences and uses DTW to obtain soft alignment across sequences of different lengths.The distance matrix compares operation vectors, while valid warping paths preserve sequence boundaries, monotonicity, and continuity.

4 Experiments

SALA is evaluated across four reasoning benchmarks, three LLMs, and multiple demonstration-selection baselines. Results show consistent performance gains, while ablations and analyses support both task-adaptive operations and semantic DTW alignment.

  • Experimental Setup: SALA is evaluated on four diverse reasoning benchmarks using three LLMs and comparisons with seven representative demonstration-selection baselines.The benchmarks include SVAMP, GSM8K, CommonsenseQA, and StrategyQA; baselines span random, retrieval-based, and reasoning-aware methods.
  • Main Results: 91.42% average performance makes SALA the best method across all four benchmarks on DeepSeek-V4-Pro.EPR and LMS3 are omitted for this API-only model because they require training or inaccessible internal hidden states.
  • Ablation Study: Removing DTW and using exact prefix matching lowers average accuracy by 0.6 points on Llama3-8B-Instruct and 1.8 points on Qwen2.5-7B-Instruct.Removing both components produces larger average drops of 2.1 and 2.9 points, respectively.
  • Operation Analysis: More than 68% of samples across all datasets require task-specific reasoning operations, exceeding 98% in GSM8K for both training and test sets.For the remaining datasets, the proportion generally falls between 70% and 80%.
  • Qualitative Case Analysis: Task-specific operations shorten reasoning sequences across all benchmarks, improving the matching of demonstrations with compatible reasoning patterns.Examples include reductions from 9 to 7 operations with Modulo and from 13 to 7 operations with Algebra.
  • Qualitative Case Analysis: DTW selects semantically aligned demonstrations when exact prefix matching favors incompatible or incomplete reasoning patterns.The cases include percentage demonstrations that capture add-insurance or tax computations despite differing operation sequences.

5 Conclusion and Future Work

The paper concludes that SALA combines task-adaptive operation construction with semantic DTW alignment for reasoning-oriented demonstration selection. Experiments across four benchmarks and three LLMs show consistent gains, while future work targets broader domains and richer reasoning structures.

  • Conclusion: SALA represents problem-solving logic with explicit reasoning-operation sequences while enabling flexible matching in semantic space.Its two components are task-adaptive operation construction and semantic DTW-based alignment.
  • Conclusion: Experiments across four reasoning benchmarks and three LLMs show that SALA consistently outperforms strong demonstration-selection baselines.The conclusion summarizes the framework's reported cross-benchmark and cross-model outcome.
  • Future Work: Future work will extend SALA to broader reasoning domains and explore richer reasoning structures beyond linear operation sequences.These directions target expansion of both task coverage and representation structure.

Limitations

SALA has two stated limitations: its LLM-generated operations and parsed sequences can vary with induction choices, and its current logic representation is linear.

  • LLM-based operation induction and question parsing may vary with the induction model and prompting strategy.The experiments use a separate LLM and fixed prompts to keep construction consistent.
  • SALA currently represents problem-solving logic as a linear operation sequence, while more complex tasks may require hierarchical or graph-based structures.The paper leaves extensions to these richer representations for future work.

A Dataset Statistics

This section documents the four benchmark datasets and their training and test split statistics, while the base operation inventory contains 13 predefined QDMR operations.

  • Table 3 summarizes the training and test splits of the four benchmark datasets used in the experiments.
  • SALA extends a base inventory of 13 predefined QDMR reasoning operations with task-specific operations.Table 4 lists the predefined operations, their core functions, example questions, and example sequences.

C Prompt Templates

The appendix gives English prompt templates for SALA’s operation induction, sequence analysis, deduplication, final ICL inference, evaluation, and hyperparameter settings.

  • C Prompt Templates: The prompt templates were used in English throughout the experiments.The appendix provides the full templates used at each SALA pipeline step.
  • C.1 Operation Induction Prompt: The operation-induction prompt asks an LLM to add reasoning operations not covered by the existing 13 and return “No new operation needed” when none are required.It also requests each new operation’s functionality, example question, decomposition, and operation sequence.
  • C.2 Sequence Analysis Prompt: The sequence-analysis prompt decomposes each input question into a reasoning operation sequence and corresponding operation name list using the extended operation set.An example applies select and successive filter operations to a flight question.
  • C.3 LLM Deduplication Prompt: The deduplication prompt flags a new operation when it overlaps an existing operation or can be decomposed into existing operations.The output is <1> for overlap or compositionality and <0> otherwise.
  • C.4 ICL Prompt Template: The final ICL template places retrieved demonstrations before the input question for reference during answering.
  • C.5 Evaluation Prompts: Evaluation prompts compare candidate and correct answers, using semantic consistency for CommonsenseQA, boolean conclusions for StrategyQA, and numerical consistency for GSM8K and SVAMP.The numeric criteria accept equivalent formats and results consistent after rounding.
  • C.5 Evaluation Prompts: The appendix also specifies numeric evaluation criteria and lists the key hyperparameters used in SALA’s experiments.Numeric outputs are compared for equivalent quantities, including percentage formats and rounded results.

E Induced Reasoning Operation Examples across Downstream Tasks

Table 6 presents representative reasoning operations induced by SALA for each downstream benchmark during adaptive operation-set construction.

  • Table 6 lists representative task-specific reasoning operations induced for each downstream benchmark.These examples come from SALA’s adaptive operation-set construction process.

F Case Study — Detailed Tables

The appendix presents detailed tables for four case studies summarized in Section 4.4. The cases separately examine task-specific operations and DTW semantic alignment using Llama3-8B-Instruct.

  • The appendix provides full detailed tables for four case studies summarized in Section 4.4.
  • Cases 1–2 isolate Module 1, which concerns task-specific operations.
  • Cases 3–4 isolate Module 2, which concerns DTW semantic alignment, and all experiments use Llama3-8B-Instruct.

F.1 Effect of Task-Specific Operations on Reasoning Sequence Compactness

Task-specific operations make reasoning sequences more compact, changing which demonstrations are retrieved and supporting correct answers. DTW semantic alignment further matches reasoning patterns flexibly, including cases where prefix matching selects logically inappropriate demonstrations.

  • Effect of Task-Specific Operations: Task-specific operations condense reasoning sequences that would otherwise require long chains of QDMR operations.This compactness affects demonstration retrieval and whether the model produces the correct answer.
  • DTW Semantic Alignment: The Module 2 isolation compares methods using only the 13 QDMR operations.
  • Effect of Task-Specific Operations: With Modulo, a compact sequence retrieves another remainder problem; without it, a longer chain retrieves a surface-similar proportion problem with different reasoning.
  • Effect of Task-Specific Operations: With Algebra, DEFINE–...–ALGEBRA matches equation-solving demonstrations, whereas QDMR-only decomposition attracts logically unrelated multi-item problems.
  • DTW Semantic Alignment: DTW selects a structurally aligned single-item percentage-markup demonstration despite a granularity mismatch that excludes it from prefix matching.
  • DTW Semantic Alignment: DTW matches a base value →percentage →total pattern, while prefix matching selects a valid-prefix demonstration that omits the percentage tax step.
Loading 2609.02336v1…