PaperlayerView exact revision on arXiv

Source-linked AI summary

E-SENS: Exclusion-Sensitive Penalization for Negative-Constraint Retrieval

Yerang Kim, Jiyoon Myung, Joohyung Han

arXiv:2608.30130v3cs.IRcs.AI

TL;DR

Negative constraints are difficult for dense retrievers because excluded concepts remain salient in the query and can attract invalid evidence. E-SENS uses a training-free trap-query penalty during reranking, and on ExcluIR it reduces trap retrieval while preserving recall at moderate penalty strengths across four embedding models.

  • Problem

    Dense retrievers may return documents about explicitly excluded concepts because those concepts remain salient and semantically represented in the query.

  • Method

    E-SENS decomposes each negative query into target and trap intents, then subtracts trap-query similarity from the original-query retrieval score without retraining.

  • Results

    Across four embedding models, moderate β values preserve answer recall while reducing trap retrieval, producing a clear recall–violation trade-off.

  • Takeaways & Limitations

    Negative constraints can be treated as retrieval-time selection constraints, with β chosen along a frontier between recall preservation and stronger trap suppression.

  • Takeaways & Limitations

    E-SENS adds inference cost, assumes reliable decomposition and comparable normalized rewrite scores, and cannot distinguish incidental mentions from documents mainly about the excluded side.

Abstract

from arXiv · show

Retrieval-augmented language models can fail to respect negative constraints when the retriever supplies evidence about concepts the user explicitly excluded. Beyond explicit negation, queries may ask for answers that include one concept while excluding another, or for entities that belong to a category but differ from a closely related instance. Because the excluded concept still appears in the query text, dense retrievers may assign high similarity to documents about that concept even when the user asks to avoid it. We introduce E-SENS, a training-free reranking method for negation-sensitive retrieval. E-SENS extracts a compact trap query for the excluded side and subtracts trap-query similarity from the original-query retrieval score. On ExcluIR, E-SENS shows a clear recall-violation trade-off across four embedding models and reduces trap retrieval at recall-preserving settings.

1 Introduction

Negative-constraint retrieval must preserve relevant evidence while avoiding documents about explicitly excluded concepts. E-SENS addresses this problem with a training-free excluded-intent penalty and frames retrieval as a recall–violation trade-off.

  • Negative queries may seek one topic while excluding a related entity, concept, or instance.
  • Dense retrievers can rank excluded-side documents highly because excluded concepts remain salient in the query and are treated as semantic evidence.
  • E-SENS generates target and trap rewrites, then penalizes documents similar to the excluded intent without retraining the retriever.
  • The paper formulates exclusion-aware retrieval as balancing relevant-evidence recall against retrieval of constraint-violating evidence.
  • E-SENS analyzes when document-level trap penalties succeed or fail, including cases where valid answers contain incidental trap evidence or lie near trap documents.

2 Related Work

Prior work studies negation and exclusion in retrieval through benchmarks, neural-symbolic reranking, embedding optimization, and rank fusion. E-SENS instead applies a score-level penalty while keeping embeddings fixed.

  • NevIR, ExcluIR, BoolQuestions, and NegConstraint examine negated pairs, exclusion constraints, Boolean logic, and negative-constraint reranking.
  • Unlike embedding-level approaches such as DEO, E-SENS keeps retriever and document embeddings fixed and modifies scores at inference time.
  • E-SENS uses dense scores rather than anti-RRF because continuous scores provide a smoother recall–violation frontier.

3 E-SENS

E-SENS decomposes an exclusion query into target and trap intents, retains the original query as the main positive signal, and subtracts trap similarity during reranking. A target-minus-trap variant is evaluated separately.

  • E-SENS reranks exclusion queries by lowering documents similar to a trap rewrite while retaining fixed retriever scores.
  • 3.1 Query Decomposition: The decomposer produces a target query with the exclusion removed and a compact trap query for the excluded intent.
  • 3.2 Score Transformation: The main variant uses the original exclusion query as the positive signal and the trap rewrite as the penalty signal.
  • 3.2 Score Transformation: Target rewrites can remove contextual cues useful for recall, so target-minus-trap is treated as an appendix variant rather than the main score.
  • 3.2 Score Transformation: E-SENS can rerank a full corpus or candidate set by rescoring the same documents with original-query and trap-query similarities.

4 Experimental Setup

The evaluation tests E-SENS on ExcluIR using four embedding models, full-corpus retrieval, shared beta checkpoints, and recall and violation metrics across multiple cutoffs.

  • Dataset: The dataset contains 3,452 negative queries, each paired with a gold answer document and an excluded-side trap document in a 90,406-document corpus.
  • Retrievers and Decomposition: The study evaluates OpenAI text-embedding-3-small, OpenAI text-embedding-3-large, Qwen3-Embedding-0.6B, and Qwen3-Embedding-4B.
  • Retrievers and Decomposition: β is swept from 0 to 1 in increments of 0.01, while Table 1 reports shared checkpoints β ∈ {0.0, 0.1, 0.2, 0.3}.
  • Metrics: Recall@k measures answer-document retrieval, Violation@k measures trap-document retrieval, and average metrics summarize k ∈ {3, 5, 7, 9}.

5 Results

E-SENS produces a recall–violation trade-off across fixed penalty strengths: moderate β values reduce trap retrieval while preserving recall, whereas aggressive penalties reduce violations at greater recall cost.

  • At β = 0.10, average recall changes by only −0.0009 to +0.0014, while average violation drops by 0.074–0.106.
  • At β = 0.20, average violation drops by 0.193–0.217 with at most 0.0038 recall loss.
  • At β = 1.0, average violation drops to 0.030–0.036, but average recall falls to 0.779–0.835.
  • Appendix curves show the same recall–violation pattern across the complete sweep for both scoring variants.
  • The original-query baseline-minus-trap variant is retained because target-minus-trap lowers violation at β = 0 but has lower recall than the original-query baseline.

6 Failure Analysis

Failure analysis identifies document-level ambiguity as a recurring limitation: valid answers may mention the excluded entity incidentally, while trap and answer documents can be close embedding neighbors.

  • Manual inspection found incidental mentions of excluded entities and close embedding neighborhoods between answer and trap documents.
  • Document-level scoring cannot reliably distinguish a valid answer that briefly mentions the excluded side from a trap document primarily about it.

7 Conclusion

E-SENS converts exclusion into an inference-time score penalty by subtracting excluded-side similarity without retraining or changing the corpus. Across four embedding models, moderate penalties preserve recall while reducing trap retrieval.

  • E-SENS extracts a compact excluded-intent query and subtracts its similarity score as a training-free retrieval penalty.
  • The penalty applies without retraining the retriever or changing the corpus.
  • Across four embedding models, moderate β values preserve answer recall while reducing trap retrieval.
  • The recall–violation frontier provides a way to choose β, with lower values favoring recall and higher values more aggressively suppressing excluded-side documents.

Limitations

E-SENS has inference and modeling limitations: it relies on reliable query decomposition and comparable normalized scores, and document-level penalties cannot distinguish incidental mentions from genuine violations.

  • E-SENS adds inference cost through query decomposition, an additional trap-query score, and reranking.
  • The method assumes that decomposition isolates the excluded side and that normalized scores remain comparable across query rewrites.
  • Document-level penalties cannot distinguish valid answers that mention the excluded entity incidentally from documents mainly about it.
  • Because β is tuned on validation data without statistical uncertainty estimates, small recall differences require cautious interpretation.

A Experimental Details

The experimental setup uses a query-only decomposition prompt to produce target and trap retrieval queries for exclusion-aware retrieval. Experiments run on a single RTX 4090 system, with strict JSON output requirements for the decomposer.

  • Experimental Setup: GPT-4o mini generates qtarget and qtrap at temperature 0, using the specified recall-preserving query decomposition role.The experimental compute setup uses one RTX 4090 GPU, 16 vCPUs, 61 GB memory, and a 20 GB container disk.
  • Query Decomposition Prompt: The decomposer is instructed to produce qtarget for the positive information need and qtrap for the excluded, contrastive, or constraint-violating side.It must use only the original query and avoid gold documents, labels, annotations, and external knowledge.
  • Query Decomposition Prompt: qtarget preserves the main search intent while removing exclusion phrases, whereas qtrap is a compact, high-precision anchor for the unwanted side.The prompt favors explicitly excluded named entities or obvious counterparts and discourages broad anchors unless they are the exact unwanted category.
  • Query Decomposition Prompt: The prompt requires exactly two JSON fields, q_target and q_trap, and prohibits markdown, comments, explanations, or extra fields.The required output format is intended to make the decomposition directly consumable by the retrieval pipeline.

C Dataset Mapping and BoolQuestions-not Set

The evaluation maps positive and excluded retrieval intents to target and trap queries and documents across ExcluIR and BoolQuestions-not. BoolQuestions-not supplies a 323-example, 646-document test set, while the full beta sweep is used to expose trade-offs rather than select a test-set hyperparameter.

  • Dataset Mapping: ExcluIR maps the original query to q, the answer-side corpus index to dtarget, and the excluded-side corpus index to dtrap.Its paired construction also permits oracle-style target and trap query strings, but reported E-SENS queries are generated only from q.
  • Dataset Mapping: For other datasets, qtarget is the positive information need, qtrap is the compact excluded-side query, dtarget satisfies the positive need, and dtrap violates the constraint.Positive and negative passages are mapped to dtarget and dtrap when available; otherwise, query-only decomposition can generate qtarget and qtrap.
  • BoolQuestions-not Set: BoolQuestions-not contains 323 examples—195 from MS MARCO and 128 from Natural Questions—over a 646-document evaluation corpus.Each example provides a positive and negative passage treated as dtarget and dtrap, while GPT-4o mini generates the missing query fields.
  • Evaluation Design: The full beta results display the recall–violation trade-off across penalty strengths rather than implying that beta should be selected on the test set.This frames the sweep as characterization of the trade-off, not deployment-ready hyperparameter tuning.

D Additional Results

Additional results report beta checkpoints and full-sweep recall–violation curves across ExcluIR and BoolQuestions-not, using four embedding models and two scoring variants. The figures and tables organize results by dataset, retriever, scoring formula, beta, and retrieval cutoff.

  • ExcluIR: ExcluIR results cover text-embedding-3-small, text-embedding-3-large, Qwen/Qwen3-Embedding-0.6B, and Qwen/Qwen3-Embedding-4B.The baseline-minus-trap results are organized by average recall and average violation across these four retrievers.
  • BoolQuestions-not: BoolQuestions-not results use the same four embedding models and report baseline-minus-trap average recall and average violation curves.Figure 3 presents the dataset-level trade-off under the same model comparison structure.
  • Baseline-minus-trap: The baseline-minus-trap ExcluIR tables report recall and trap violation at k ∈ {3, 5, 7, 9} for 0.10 beta checkpoints.Tables 2–5 use S(d) = s(d, q) −βs(d, qtrap) for the four retrievers.
Loading 2608.30130v3…