PaperlayerView exact revision on arXiv

Source-linked AI summary

Embedding Models for Stance-Aware Argument Retrieval

Angelo Sparacino, Francesca Toni, Adam Dejl

arXiv:2608.28283v1cs.CLcs.AI

TL;DR

Stance-aware argument retrieval requires dense embedding models to capture both a claim’s topic and the requested positive or negative stance, but existing models often favor topical overlap. The paper diagnoses this bias and contrastive-tuning overcorrection with word-ablation metrics, then uses balanced curricula and synthetic stance inversions to improve joint topic-and-stance retrieval. Results show that the data-centric approach can reduce topical collapse and improve retrieval, especially for Qwen3-Embedding-8B, while hybrid BM25 search adds topical coverage in specialized domains.

  • Problem

    Dense embedding models struggle to retrieve arguments that are both topically relevant and correctly aligned with an instructed supporting or attacking stance.

  • Method

    The paper combines word-ablation diagnostics with a two-stage data-centric pipeline using a balanced curriculum and synthetic stance inversions, plus hybrid sparse search.

  • Results

    Targeted fine-tuning improves stance-aware retrieval; for Qwen3-Embedding-8B, the proposed approach improves joint stance-topic encoding and overall retrieval performance.

  • Takeaways & Limitations

    Balanced training and stance-inverted data can reduce topical collapse and help sufficiently powerful embedding models learn deeper directional logic rather than lexical shortcuts.

  • Takeaways & Limitations

    The diagnostic metrics rely on input-output behavior during word ablation and may be sensitive to syntactic changes introduced by ablation.

Abstract

from arXiv · show

In computational argumentation, obtaining arguments that explicitly support or attack given claims is a critical precursor to downstream reasoning tasks. When these supporting and attacking arguments are to be retrieved using semantic search methods, they need to be assessed for topic-relevance to the claims of interest as well as for correctness of their (positive or negative) stance towards the claims. In this paper we explore how dense embedding models (hereafter, models), powering modern retrieval pipelines, can serve as the basis of semantic search incorporating this dual assessment. We show experimentally that existing models struggle with asymmetric reasoning, exhibiting a strong bias toward topical overlap while ignoring instructional stance. We also show that correcting this bias via contrastive training triggers a new failure mode where models over-correct, over-fixating on polarity keywords (e.g., "supports" or "refutes") at the expense of the semantic topic. We thus introduce diagnostic word-ablation metrics to quantify this phenomenon and propose a data-centric solution. By implementing a balanced argument curriculum alongside LLM-augmented, stance-inverted arguments, we force the (embedding) models to learn deeper directional logic rather than exploiting superficial lexical shortcuts. Our evaluation demonstrates that, for sufficiently powerful models, this approach can alleviate the observed overcorrection, achieving further improvements in stance-aware argument retrieval.

1. Introduction

The paper shows that dense embedding models often prioritize topical overlap over instructed stance in argument retrieval. It diagnoses this bias and proposes targeted interventions, including diagnostic metrics and refined fine-tuning, to improve stance-aware retrieval while avoiding topical collapse.

  • Problem: Dense embedding models often match arguments by topic while ignoring instructions to retrieve supporting or attacking stances.Figure 1 contrasts a base model retrieving topical hard negatives with a tuned model retrieving arguments having the requested opposing stance.
  • Failure mode: Contrastive tuning can cause topical collapse by shifting representational capacity from subject matter toward explicit polarity keywords.The paper identifies this as an overcorrection that reduces attention to the underlying topic.
  • Interventions: Targeted interventions combine a refined fine-tuning curriculum with hybrid search to improve stance-aware retrieval while mitigating topical collapse.The introduction presents these as the primary intervention directions.
  • Diagnostics: The paper introduces word-ablation metrics RIS/RCS and DI to quantify which input portions influence similarity and diagnose model focus.These metrics assess the effects of specific input text portions on model similarity through ablation.
  • Outcome: Experiments show that targeted fine-tuning can substantially improve the poor stance-aware retrieval performance of base embedding models.The code, datasets, and models are publicly available for reuse.

2. Related Work

Related work frames argument retrieval as semantic matching with an important representational bottleneck. Instruction-tuned embedding models condition retrieval on user intent, but empirical evidence shows that lexical overlap can still dominate stance-sensitive matching.

  • Argument Mining: Argument mining identifies arguments, their components, and relations such as attack or support; this paper focuses on retrieving arguments related to given claims.The paper uses embedding models rather than LLMs to perform this relation-based retrieval task.
  • Dense Retrieval and Foundation Models: Bi-encoder retrieval maps queries and documents into representations that enable semantic matching through embedding similarity.The supplied passage introduces this architecture as the basis of modern retrieval systems.
  • Dense Retrieval and Foundation Models: Static document embeddings create a representational bottleneck because one vector must compress all possible semantic interpretations.Instruction-tuned models address this by conditioning the embedding space on a natural-language instruction.
  • Training: Contrastive learning trains retrieval models by increasing query-positive similarity while decreasing similarity to negative documents.The passage identifies Multiple Negatives Ranking Loss as a standard contrastive methodology.
  • Lexical Bias and Instruction-Aware Retrieval: Hard negatives share high topical overlap with a query but violate its instruction, making lexical overlap a source of retrieval errors.Such negatives may contain as many or more query terms than true positives.

3. Task Formulation and Experimental Setup

Stance-aware argument retrieval encodes an instruction and claim into a query, then ranks corpus arguments by embedding similarity while distinguishing topical relevance from directional stance. The setup combines controlled datasets, contrastive triplets, retrieval metrics, and word-ablation diagnostics to assess both performance and model focus.

  • 3.1. Task Definition: Stance-aware retrieval seeks arguments matching both a claim’s topic and its requested supporting or attacking stance.Queries concatenate an instruction specifying stance with a claim specifying subject matter.
  • 3.1. Task Definition: Documents are categorized as positive, hard negative, semi-hard negative, or easy negative according to topic and stance matches.Hard negatives match the topic but violate stance; semi-hard negatives match stance but not topic.
  • 3.1. Task Definition: The central challenge is separating positive arguments from hard negatives because both share substantial lexical overlap with the claim.This overlap can make stance-violating arguments appear similar to the query.
  • 3.2. Datasets and Setup: Experiments enforce separate training, validation, and evaluation resources, using standardized supporting and attacking instructions to test generalization across phrasing.Contrastive tuning uses query-positive-negative triplets and batches with distinct claims to avoid false in-batch negatives.
  • 3.3. Evaluation Metrics: Precision@R and NDCG@10 evaluate retrieval quality, with Precision@R adapting to the number of positive documents and NDCG@10 discounting lower-ranked results.These metrics capture complementary aspects of ranking performance.
  • 3.3. Evaluation Metrics: Word ablation measures how removing query words changes cosine similarity, enabling RIS/RCS and DI diagnostics of instruction, claim, and stance sensitivity.RIS and RCS compare instruction and claim influence, while DI averages signed effects for stance keywords.
  • 3.3. Evaluation Metrics: For hard negatives, removing a stance word should typically increase similarity, corresponding to DI > 0.The diagnostic tests whether models represent the relational constraint rather than merely matching topical words.
  • 3.3. Evaluation Metrics: Figure 2 evaluates Average Precision@R on unseen instructions, showing that fine-tuning reduces hard-negative retrieval but increases semi-hard-negative retrieval.This pattern indicates possible topical collapse: stronger stance focus can come at the expense of claim-topic focus.

4. Baseline Model Evaluation

Base embedding models do not reliably perform stance-aware argument retrieval, while fine-tuning substantially improves stance discrimination. However, homogeneous hard-negative training can shift models toward stance at the expense of topical relevance, producing topical collapse.

  • 4.1. Baseline Performance: 0.95 Precision@R was achieved by fine-tuned Qwen3-Embedding-8B on held-out training data, reducing stance error from 39.7% to 2%.This was the strongest reported overall model result in the comparison.
  • 4.1. Baseline Performance: Fine-tuning increased BGE-Large’s held-out TFU Training Precision@R from 0.419 to 0.822 while reducing stance error to 8.8%.On GPT-5 validation data, semi-hard negatives constituted 27.1% of retrieved items.
  • 4.1. Baseline Performance: None of the evaluated base embedding models reliably perform stance-aware retrieval despite explicit stance instructions.Their failures reflect difficulty distinguishing topic-matched arguments with opposite stances.
  • 4.1. Baseline Performance: Qwen’s semi-hard-negative rate on GPT-5 validation more than doubled from 5.6% to 12.9% after fine-tuning.Instructor-XL showed a larger increase, from 7.5% to 22.6%.
  • 4.1. Baseline Performance: Homogeneous hard-negative contrastive optimization suppresses stance errors but can over-correct by neglecting claim topic.The resulting stance-topic imbalance is termed topical collapse.
  • 4.2. Topical Collapse: Word-ablation diagnostics show that fine-tuning increases sensitivity to stance logic while shifting representational focus away from topical content.For Qwen hard negatives, DI rose from 0.036 to 0.238 on TFU Training data, while RCS fell as RIS increased.

5. Targeted Interventions for Stance-Aware Retrieval

The paper addresses topical collapse caused by homogeneous hard-negative training through balanced curricula and stance-inverted synthetic negatives. These interventions aim to preserve topic sensitivity while reducing lexical shortcuts in stance-aware retrieval.

  • Motivation: The homogeneous hard-negative curriculum can trigger topical collapse by shifting representational capacity from subject matter toward polarity keywords.The authors hypothesize that optimization pressure to distinguish positive and hard-negative samples causes interference with topic representation.
  • Mixed Dataset: The mixed curriculum partitions negatives equally among hard, semi-hard, and easy examples while keeping the triplet count fixed at 31,800.Reintroducing semi-hard negatives is intended to force the model to preserve topical sensitivity.
  • Stance-Inverted Argument Augmentation: Stance-inverted hard negatives preserve near-identical sentence structure and substance while severing the causal link to the target claim.This design prevents the optimizer from exploiting spurious structural and lexical differences between positive and negative documents.
  • Validation: Synthetic argument pairs achieved Jaccard overlap of 0.5866, compared with 0.0938 for standard pairs.With NLTK lemmatisation, synthetic-pair overlap reached 0.5896; the authors describe this as a six-fold increase despite falling short of the 90% generation target.
  • Hybrid Retrieval: The hybrid pipeline fuses the tuned dense retriever with BM25 using Relative Score Fusion and conservative sparse weights λ∈[0.1, 0.15, 0.2].BM25 supplies lexical anchoring for rare, domain-specific entities, while the dense retriever handles stance constraints.

6. Results

The interventions produced architecture-dependent retrieval changes and improved domain-specific retrieval when dense and sparse methods were combined. Word-ablation analyses indicate that Qwen3-Embedding-8B regained topical sensitivity without losing asymmetric stance behavior.

  • 6.1. Intervention Evaluation: BGE-Large’s Mixed + Aug curriculum reduced semi-hard negatives from 20.7% to 8.5% but increased hard negatives to 23.2%.Its homogeneous curriculum reached Precision@R 0.627 on TFU Evaluation, while the intervention traded stance sensitivity for topical focus.
  • 6.1. Intervention Evaluation: Instructor-XL’s Mixed curriculum reduced semi-hard negatives from 10.1% to 5.1% but its stance error rate regressed to 31.4%.The model showed a strong dependency on dense hard-negative signals and did not robustly maintain relational logic after topic-sensitivity was restored.
  • 6.1. Intervention Evaluation: Hybrid fusion improved Qwen3-Embedding-8B’s ArgTumour Precision@R from 0.690 to 0.723 at λ=0.1.The sparse retriever provided a topical safety net for specialised domains, but increasing sparse weight inflated stance error across models.
  • 6.2. Word Ablation Validation: For Qwen3-Embedding-8B on AVeriTeC, mean RCS increased from 0.637 after homogeneous training to 0.661 with Mixed + Aug.The intervention flattened the collapse peak and shifted semantic focus partially back toward the claim.
  • 6.1. Intervention Evaluation: Across Figure 3, Mixed uniformly reduced semi-hard negatives, but BGE-Large and Instructor-XL showed stance regression while Qwen3-Embedding-8B slightly expanded positive retrieval.The chart compares homogeneous, Mixed, and Mixed + Aug conditions across ArgTumour, TFU Evaluation, and AVeriTeC.
  • 6.2. Word Ablation Validation: Mixed + Aug maintained an asymmetric stance boundary, with DI of 0.144 for hard negatives and −0.120 for positives.The base model’s hard-negative DI was 0.024, indicating that recovered topical sensitivity did not come at the reported cost of relational logic.
  • 6.2. Word Ablation Validation: Figure 4 shows base noun bias, homogeneous-training topical collapse, and progressive topic recovery under Mixed and Mixed + Aug for Qwen3-Embedding-8B.The distributions are Relative Claim Sensitivity results on AVeriTeC data.

7. Conclusion

The paper diagnoses a trade-off between topical relevance and directional stance in dense argument retrieval and proposes data-centric interventions to address it. The approach is especially effective for Qwen3-Embedding-8B, while BM25 adds topical protection in specialised domains.

  • Conclusion: Base embedding models show topic bias and disregard stance constraints, while homogeneous hard-negative fine-tuning can cause topical collapse.The paper frames both behaviors as failures to balance topical subject matter with directional stance.
  • Conclusion: The two-stage data-centric pipeline combines a balanced curriculum with synthetic stance inversions to improve joint encoding of stance and topical relevance.The balanced curriculum increases topical sensitivity, while stance inversions reduce residual lexical shortcuts.
  • Conclusion: The approach proved particularly effective for Qwen3-Embedding-8B and improved its overall retrieval performance.The conclusion reports this model-specific effectiveness without extending it to all evaluated architectures.
  • Conclusion: BM25 provides an additional topical safety net for dense models’ remaining blind spots in highly specialised domains.This is presented as a complementary role for sparse retrieval rather than a replacement for stance-aware dense encoding.
  • Future Work: Future work includes model-centric interventions, dynamic hybrid-search weighting, and applications to large-corpus argument mining or argumentative fact verification.The conclusion identifies these directions as areas for further exploration.

Declaration on Generative AI

The declaration states that generative AI tools supported instruction generation, code assistance, synthetic argument augmentation, and manuscript feedback. The authors reviewed and edited the outputs and retained responsibility for the work.

  • AI Use: Gemini 3 Pro generated standardised training and evaluation instructions and assisted with boilerplate PyTorch, Matplotlib, and LaTeX code.The declaration lists these uses as code and instruction support.
  • AI Use: Gemini 3.5 Flash generated synthetic stance-inverted arguments for dataset augmentation.This corresponds to the paper’s stance-inverted augmentation procedure.
  • AI Use: GPT-5.5 provided general feedback on the draft manuscript.The declaration distinguishes this use from data generation and coding assistance.
  • Author Responsibility: The authors reviewed and edited the generated outputs and took full responsibility for the resulting work.The supplied declaration explicitly assigns responsibility to the authors after tool use.

C. Experimental Details

The experiments use specified hardware, training settings, LoRA configurations, and datasets to evaluate stance-aware retrieval.

  • Experiments use L40S and A40 GPUs across two high-performance compute clusters.
  • All fine-tuning uses Multiple Negatives Ranking Loss for 2 epochs with batch size 8, gradient accumulation 2, and learning rate 2 × 10−5.
  • Training uses a 0.1 warmup ratio, FP16 or BF16 precision, default Transformer-library seeds, and dynamic claim shuffling.
  • LoRA uses α set to 2 × r with r = 64 and dropout 0.05, with target modules varying by model architecture.
  • Dataset statistics report claims, total supporting and attacking arguments, and average arguments per claim with minimum and maximum bounds.

D. Retrieval Metrics Tables

The retrieval tables report stance-aware performance across model families and datasets, using dense and hybrid configurations with mean and standard-deviation reporting.

  • Metrics include NDCG, Precision, Hard@R, Semi@R, and Easy@R, with negative categories describing retrieved hard, semi-hard, and easy negatives.
  • Results are reported as means and standard deviations across evaluation queries and unseen instructions.
  • The tables identify absolute best scores and best dense or hybrid subgroup scores using bold and underlining, respectively.
  • Hybrid Relative Score Fusion strategies operate exclusively over Mixed + Aug base models.
  • Retrieval results span BGE-Large, Instructor-XL, and Qwen3-Embedding-8B across TFU, ArgTumour, and AVeriTeC datasets.

E. Ablation Metrics Tables

Ablation tables and diagnostics measure instruction sensitivity, claim sensitivity, and directional stance effects across datasets and models.

  • The ablation metrics are Relative Instruction Sensitivity, Relative Claim Sensitivity, and Directional Impact for positive and hard-negative pairs.
  • These metrics are reported across TFU, ArgTumour, and AVeriTeC datasets, including validation and evaluation variants.
  • For Qwen3-Embedding-8B, the base model shows a strong noun bias, while Homogeneous training produces topical collapse toward the instruction.
  • After fine-tuning, all training paradigms establish asymmetric stance boundaries by bringing positive documents closer and repelling stance-violating hard negatives.
  • Mixed and Mixed + Augmentation preserve separated positive and hard-negative DI distributions on the out-of-domain AVeriTeC dataset.

G. Word-Ablation Heatmap

The word-ablation heatmap examines how tuning redistributes retrieval sensitivity between stance instructions and claim content, while related examples illustrate stance inversion and hybrid retrieval.

  • The heatmap removes words from instructions and claims, showing retrieval-score deltas across attacking and supporting evaluation documents.
  • In the base model, “disproving” has inconsistent and frequently minimal influence, indicating baseline instruction blindness.
  • In the tuned model, “disproving” becomes a salient anchor that penalises stance-violating documents and pulls correct documents closer.
  • This stance sensitivity coincides with topical nouns’ influence collapsing toward zero, reducing verification of the underlying subject matter.
  • Synthetic stance inversions preserve lexical content while changing relational verbs, polarity adjectives, or conjunctions to reverse argumentative polarity.
  • Hybrid retrieval combines the Mixed + Augmentation dense model with BM25 through Relative Score Fusion, using sparse retrieval as a topical safety net.
Loading 2608.28283v1…