$\mathrm{ECI}_{\mathrm{sem}}$: Semantic Residual Effective Contrastive Information for Evaluating Hard Negatives

Aarush Sinha; Rahul Seetharaman; Aman Bansal

Source-linked AI summary

$\mathrm{ECI}_{\mathrm{sem}}$: Semantic Residual Effective Contrastive Information for Evaluating Hard Negatives

Aarush Sinha, Rahul Seetharaman, Aman Bansal

arXiv:2603.20990v4Mar 22, 2026cs.IRcs.AI

TL;DR

Dense-retrieval hard-negative sources are usually selected only after expensive fine-tuning and evaluation, while hardness alone does not ensure useful supervision. ECIsem ranks candidate sources before training using frozen target-encoder embeddings and three validity gates. On MS MARCO, its in-family rankings recover aggregate BEIR transfer across three backbones, but constructed controls reveal an identifiability limit requiring audit rather than reweighting.

Problem
Hardness alone does not establish useful supervision, and source selection after fine-tuning and downstream evaluation is expensive and gives limited insight into source quality.
Method
ECIsem ranks candidate hard-negative sources with frozen target-encoder embeddings, weighting residual directions by target consistency, semantic locality, and lexical residuality.
Results
On MS MARCO, in-family ECIsem rankings recover aggregate BEIR transfer ordering across DistilBERT, E5-base, and Contriever.
Takeaways & Limitations
ECIsem supports training-free source screening for aggregate transfer, while the reported audit profile is needed because gate reweighting cannot resolve the constructed controls.
Takeaways & Limitations
ECIsem requires query-positive supervision and explicit candidate negatives, and frozen gate statistics cannot distinguish some false-negative-shaped sources safely while preserving transfer alignment.

Abstract

from arXiv · show

Hard-negative source selection for dense retrieval is usually decided only after fine-tuning and downstream evaluation. We propose ECIsem, a validity-weighted diagnostic that ranks candidate hard-negative sources using frozen target-encoder embeddings. ECIsem is training-free, not label-free: each scored example requires a query, a labeled positive, and an explicit candidate negative. Each negative is weighted by target consistency, semantic locality, and lexical residuality, and the source score aggregates weighted residual directions through a log-determinant. At the scale we study, the log-determinant operates in its linear regime, so the score reduces to the mean validity weight; we report this reduction as a finding, and gate ablations show that a single lexical scalar, mean lexical residuality, reproduces the transfer ordering in every block. On MS MARCO negative sources, in-family ECIsem recovers the aggregate BEIR transfer ordering across DistilBERT, E5-base, and Contriever. Constructed controls expose an identifiability limit: a plausible false negative is non-inverted, near the positive, and lexically novel, exactly the profile the gates reward, and across twelve weightings of the three gates and three backbones only one of thirty-six cells is safe, and it is not transfer aligned, a limit that extends to margin-style filters built from the same statistics. The same gate statistics separate the controls sharply when used as audit features rather than as rewards, so we pair the score with a reported audit profile instead of a reweighting.

1 Introduction

ECIsem addresses the gap between hard negatives and useful supervision by ranking candidate sources before fine-tuning with frozen target-encoder geometry. It combines validity gates with residual-direction aggregation, while experiments expose limits to what those statistics can identify.

Motivation: Hard-negative usefulness is not guaranteed by lexical overlap, dense similarity, or contrastive loss because candidates may be false negatives, duplicates, artifacts, or redundant examples.The proposed diagnostic therefore evaluates target consistency, semantic locality, and lexical residuality rather than hardness alone.
Contribution: ECIsem ranks hard-negative sources before fine-tuning using a frozen copy of the target encoder.Scores are computed in the representation geometry that will initialize training.
Contribution: For each query-positive-negative triplet, target consistency, semantic locality, and lexical residuality weight semantic residual directions aggregated through a log-determinant.The gates assess positive preference, document-space locality, and whether query-token overlap explains the match.
Findings: At the studied scale, the log-determinant operates in its linear regime and reduces to the mean validity weight; mean lexical residuality reproduces transfer ordering in every block.These are reported as empirical findings rather than as evidence that volume sensitivity is the operative mechanism.
Limitations: Constructed controls reveal an identifiability limit: gate-based rankings can reward plausible false negatives, motivating audit features alongside source screening.The controls are used to distinguish safety auditing from reweighting.

2 Related Work

Related work frames negative selection as a central design problem in dense retrieval, spanning multiple mining and synthesis strategies. It also motivates ECIsem through concerns about false negatives, synthetic relevance violations, and information-volume objectives.

Negative sampling: Dense-retrieval negative sampling includes random, static, dynamic, and synthetic sources, with contrastive supervision central to dual-encoder systems.Prior methods include BM25, dense mining, adaptive refresh or reweighting, and synthetic generation.
Negative selection: BM25 and dense mining define different notions of hardness, while other methods seek informative negatives that are unlikely to be false.False negatives can bias contrastive gradients, motivating regularization and filtering approaches.
Connections: Synthetic negatives may violate relevance assumptions without filtering, and log-determinant objectives connect to diversity, experimental design, active selection, and representation-collapse avoidance.ECIsem uses the log-determinant as a training-free diagnostic over weighted residual directions.

3 Methodology

The methodology constructs and scores explicit query-positive-negative records using frozen, normalized target-encoder embeddings. ECIsem applies three validity gates to residual directions and, in the studied regime, its log-determinant reduces to mean validity weight.

Candidate sources: Mining uses BM25, dense retrieval, and LLM generation to produce candidate negatives, generally retaining ten non-positive passages per query.Dense mining removes labeled positives before selecting by cosine similarity; LLM prompts request plausible, topical, diverse negatives that omit or contradict the information need.
Encoder: ECIsem uses a frozen target encoder with training-matched formatting, sequence length, normalization, and temperature.The method computes scores in the same representation geometry used for downstream fine-tuning.
Inputs: Each scored record contains a query, a labeled positive, and explicit hard negatives; files without explicit negatives are excluded.ECIsem therefore targets supervised or weakly supervised source selection rather than fully unlabeled corpora.
Aggregation: The score is a log-determinant over weighted residual directions, but all eigenvalues are far below one, making it approximately the mean validity weight.The paper verifies this linear-regime reduction numerically and notes that redundancy sensitivity is inactive at the studied scale.
Validity gates: The three gates measure target consistency, relative semantic locality, and lexical residuality to weight each negative’s semantic residual direction.They respectively favor negatives below the labeled positive, near the positive relative to the query, and not explained by query-token overlap.
Interpretation: The optimization connection is local: under first-order expansion, raw gradient energy favors low-consistency negatives, whereas ECIsem substitutes target-consistent, local, lexically residual weighting.This connection depends on local linearization assumptions.

4 Fine-Tuning

The fine-tuning pipeline trains retrieval backbones on mined and hybrid negative-source files, then evaluates transfer while comparing source rankings from ECIsem. The reported setup emphasizes matched temperature, large batches, single-seed training, and substantial but reduced screening compute.

Training data: Non-hybrid sources use ten negatives per record and yield approximately 4M flattened triplets, while hybrid sources use K=20 and yield approximately 8M triplets per source.The non-hybrid sources are BM25, Dense, and Qwen3-30B; hybrids combine them pairwise.
Training: Training uses CachedMultipleNegativesRankingLoss with query_to_doc sampling, no duplicates, batch size B=4096, and GradCache mini-batches capped at 2048.The training temperature is τ=0.05, matching ECIsem’s temperature.
Training: DistilBERT, E5-base, and Contriever are trained for one epoch with maximum sequence length 128, learning rate 2 × 10^-5, and warmup ratio 0.1.Mixed precision uses bfloat16 or fp16 when available, and results use a single seed.
Compute: ECIsem scoring took 8.5 GPU-hours versus 17.9 for fine-tuning and 38.9 for BEIR evaluation in one logged Contriever run.The ranking therefore required roughly 15% of the measured downstream compute budget.
Compute: Mean lexical residuality reproduces the transfer-aligned ranking without encoder cost, whereas the encoder computation supplies audit statistics in the target geometry.Per-source cost is dominated by encoder forward passes and is essentially linear in the number of scored negatives.

5 Results

In-family ECIsem rankings align with aggregate BEIR transfer across three backbones, while dataset-level winners can differ from the aggregate ordering.

Source rankings: Across DistilBERT, E5-base, and Contriever, LLM negatives receive the highest non-hybrid ECIsem scores, while Dense+LLM leads hybrid sources.The scores are computed with a frozen copy of the same backbone later used for fine-tuning.
Scope of results: Hybrid comparisons confound source composition with training-data volume because hybrid runs use K=20 versus K=10 and twice as many flattened triplets.The reported alignment claims are therefore stated within setting.
Transfer alignment: LLM mining gives the best non-hybrid mean nDCG@10 for DistilBERT (0.392), E5-base (0.479), and Contriever (0.434).The highest-scoring hybrid source also has the best mean downstream performance for each backbone.
Transfer alignment: All 18 within-block pairwise comparisons match the mean BEIR nDCG@10 ordering across six backbone and setting blocks.The replication consists of two source orderings—non-hybrid and hybrid—repeated across three backbones.
Gate-level diagnostics: LLM negatives have fewer target inversions, at 2.1% versus 16.8% for BM25 and 33.5% for dense, and lower query-token coverage than BM25.These statistics raise target consistency and lexical residuality in the ECIsem weighting.
Scope of results: Aggregate alignment does not imply per-dataset dominance: BM25 wins ArguAna and Touche on E5-base, while dense wins Quora and FiQA.Practitioners targeting a single dataset should evaluate on that dataset.

6 Training-Free Ablation Experiments

Training-free ablations show that ECIsem’s source ordering is driven by validity weights rather than raw hardness, while constructed controls reveal that the weighting is not a safe false-negative filter.

Baseline diagnostics: Across all three backbones, ECIsem ranks LLM highest among non-hybrid sources and Dense+LLM highest among hybrid sources, matching the strongest mean BEIR transfer pattern.MNRL and ambiguity instead favor dense-heavy sources, so the ECIsem ordering is not reducible to raw hardness, contrastive loss, or ambiguity rates.
Linear regime: 0.21% relative median 0.03%: log det(I_d+bI) and tr(bI) agree closely, with identical orderings in every block.Across gate ablations, the gap remains below 0.9%; the reduction follows from eigenvalues of order w/d≈10^-4.
Linear regime: 0 of 288 sweep rows deviate: source ordering remains invariant across the tested rescaling grid, including when ψ is removed.The validity weights shrink spectral radius by a median factor of 7.1, keeping the weighted score trace-like at admissible normalizations.
Gate ablations: Mean ψ alone reproduces the transfer ordering in every block, whereas dropping ψ breaks two block orderings and dropping ρ or η changes none.The trace form matches the full score exactly because it is numerically the mean validity weight.
Negative-source controls: Non-inverted, near-positive, and lexically novel negatives receive the same gate profile whether they are valuable hard negatives or plausible false negatives.The controls expose why each gate can reward corrupted or false-negative-shaped sources.
Negative-source controls: 1 of 36 backbone-weighting cells is safe, and it is not transfer aligned, showing that reweighting the three gates does not close the identifiability gap.The false-negative control causes 33 of 35 failures; small-sample verdicts also fail to replicate at 25,000 records.
Audit profile: Seven of eight controls are flagged and no real source is when ρ, η, ψ, and inversion rate are used as audit features calibrated on the six real sources.The paper therefore pairs ECIsem with an audit profile rather than treating reweighting as a safety mechanism.

7 Scope and Limitations

ECIsem is a screening diagnostic for curated hard-negative sources, not a replacement for downstream evaluation or a tool for unlabeled corpora. Its theoretical and empirical evidence is local, correlational, and limited by single-seed evaluation.

ECIsem requires query-positive supervision and explicit candidate negatives, so it cannot directly score a fully unlabeled corpus.It is training-free because scoring uses a frozen encoder rather than optimization or trained checkpoints.
ECIsem contains no hardness term and can score trivially easy or false-negative-shaped candidates highly.The diagnostic therefore presumes curated candidate hard negatives and should be read with its audit profile.
The theoretical loss-reduction connection is local and does not guarantee global optimization improvement or per-dataset generalization.The argument uses first-order expansion around initialization and frozen embedding residuals as a proxy for parameter-space gradient structure.
The empirical evidence is correlational and based on single-seed runs, without estimates of run-to-run variance.Some mean BEIR deltas are small, so the result is framed as rank alignment rather than prediction of score magnitudes.

8 Conclusion

The paper introduces ECIsem as a frozen-encoder, validity-weighted diagnostic for ranking hard-negative sources. It finds transfer-aligned source rankings while showing that safety cannot be achieved by reweighting the same gates, motivating audit features and downstream evaluation.

ECIsem ranks hard-negative sources using validity-weighted residual contrastive directions computed with a frozen target encoder.It is training-free but not label-free.
At the studied scale, the log-determinant reduces to mean validity weight, while mean lexical residuality reproduces transfer ordering in every block.These are reported empirical findings rather than claims that the volume objective remains operative.
On MS MARCO sources, in-family ECIsem recovers aggregate BEIR transfer ordering across DistilBERT, E5-base, and Contriever.
Constructed controls show an identifiability limit: no weighting is both safe on false-negative-shaped or degenerate sources and aligned with transfer.Safety is therefore handled through an audit using the same statistics as features rather than rewards.
ECIsem is a target-aligned diagnostic rather than an exact full-batch MNRL gradient estimator, and its structural motivation does not provide strict optimization or distribution-free generalization guarantees.

A.3 Semantic Residual Information

This section defines ECIsem as a log-determinant over weighted semantic residual directions and relates its gates to pairwise contrastive loss. The objective rewards nonredundant residual volume, but experiments operate in a linear regime where it reduces to mean weight, and its optimization interpretation remains local.

ECIsem aggregates weighted residual directions through a semantic residual information matrix and a normalized final diagnostic.The score is computed per explicit negative under a fixed per-negative budget.
In the experiments, the log-determinant operates in its linear regime and reduces to the mean weight rather than rewarding nonlinear residual volume.
The target-consistency gate 𝜌 is exactly the positive probability under the two-document temperature-scaled softmax used by MNRL.Thus −log 𝜌 is the corresponding two-document restriction of the MNRL loss.
For fixed total residual weight, spreading weight across non-collinear directions yields a larger log-determinant than concentrating it in one direction.Additional linearly independent directions contribute positive higher-order volume terms.
The connection to loss reduction and information volume holds only under local encoder linearization and additional idealized assumptions, not as a global guarantee.A rigorous parameter-space bound would require assumptions about Jacobians, gradients, label noise, and optimization trajectories.
The validity weight 𝜌𝜂𝜓 replaces raw gradient-energy weighting, trading preference for hard negatives for target-consistent, semantically local, lexically residual signal.Raw gradient energy scales with (1−𝜌)^2 and can favor inverted or mislabeled negatives.

B Robustness Checks

The passage identifies these checks as an expansion of Section 6.4.

These checks expand Section 6.4.
Section 6.4 is the section being expanded by these checks.
The checks are presented as an extension of Section 6.4.

B.1 Sample-Size Stability

ECIsem’s source rankings remain unchanged as seeded samples grow from 25% to 100%, indicating that the observed ordering is not an artifact of maximum sample size in this experiment.

ECIsem rankings are unchanged at 25%, 50%, 75%, and 100% of each seeded sample.The stability value of 1.000 denotes exact agreement with the full-sample rank order.

B.2 Tokenizer, IDF, and Failure-Case Sensitivity

Tokenizer and IDF choices modestly shift ECIsem scores while preserving source rankings, and failure buckets make the gate decisions and their potential conservatism auditable.

LLM > Dense > BM25 for non-hybrid sources, while Dense+LLM > BM25+LLM > BM25+Dense for hybrid sources under tokenizer and IDF variations.The largest tokenizer effects arise from changing lexical-matching granularity, while IDF-corpus effects are smaller for LLM and Dense+LLM than for BM25-heavy sources.
Failure buckets define inversions as 𝜌< 0.5, low locality as 𝜂≤0.25, and high lexical coverage as 𝐶≥0.50.Valid High-𝐶 and Valid Low-𝜂 combine these thresholds with additional gate conditions to identify auditable cases.
BM25-heavy sources have the highest high-coverage rates, while valid high-coverage and valid low-𝜂 buckets expose cases where gates may suppress plausible negatives.The buckets distinguish inversion, low locality, and lexical coverage, but do not prove that any individual negative is invalid.
Temperature values 0.03, 0.05, and 0.1 change absolute scale but not within-setting rank order, supporting 𝜏= 0.05 as a stable default.Smaller temperatures sharpen the gates, whereas larger temperatures smooth pairwise differences.