PaperlayerView exact revision on arXiv

Source-linked AI summary

Twin Worlds: Equivariance-Based Abstention for Evidence-Grounded Reasoning

Vy Nguyen, Ziqi Xu, Jeffrey Chan, Estrid He, Feng Xia, Renqiang Luo, Erik Cambria, Xiuzhen Zhang

arXiv:2608.28018v1cs.CLcs.AIcs.LG

TL;DR

Knowledge-intensive LLM reasoning can produce confident answers from memorised associations when evidence is insufficient, while existing abstention methods do not directly test grounding. Twin Worlds applies typed, structure-preserving entity substitutions and uses equivariance violations to decide when to abstain. Across four benchmarks and three backbones, it outperforms uncertainty- and sufficiency-based baselines while maintaining strong answer quality on answerable instances.

  • Problem

    LLMs may answer confidently from memorised associations despite insufficient or conflicting evidence, and existing uncertainty or sufficiency checks do not test whether generation is actually evidence-grounded.

  • Method

    Twin Worlds tests whether answer distributions transform equivariantly under typed, bijective entity substitutions that preserve relational structure and uses violations as an abstention signal.

  • Results

    Across four benchmarks and three model backbones, TW outperforms uncertainty- and sufficiency-based baselines while maintaining strong answer quality on answerable instances.

  • Takeaways & Limitations

    TW identifies when answers are not reliably grounded in provided evidence through structure-preserving transformations that reduce parametric priors.

  • Takeaways & Limitations

    TW is most effective for entity-grounded reasoning, while mathematical or procedural tasks and unreliable entity recognition are outside or beyond its straightforward scope.

Abstract

from arXiv · show

Knowledge-intensive reasoning requires Large Language Models (LLMs) to ground answers in provided evidence. When evidence is insufficient, it is desirable that models abstain rather than confidently generating unsupported answers. Existing abstention methods rely on uncertainty estimation or evidence sufficiency checks, but neither tests whether the reasoning process for generation, driven by the interaction of provided evidence and the model's internal memory parameters, is actually grounded in the evidence. A key contributing factor is that entity mentions in context activate memorised associations, causing models to generate plausible responses ungrounded in evidence. We propose Twin Worlds (TW), a framework for improving reliability in knowledge-intensive reasoning through equivariance-based abstention: unlike invariance, which requires outputs to remain unchanged, equivariance requires outputs to transform correspondingly under entity substitutions. A model grounded in the evidence should produce answers that shift consistently when entities are substituted while their relations are preserved. TW constructs multiple worlds via typed substitutions of the original input that preserve relational structure while reducing parametric priors, and uses equivariance violations as an abstention signal. Across four benchmarks and three model backbones, TW identifies when answers are not reliably grounded in the provided evidence and outperforms uncertainty- and sufficiency-based baselines.

1 Introduction

Twin Worlds reframes abstention as testing whether answers are structurally grounded in evidence rather than memorised associations. It uses structure-preserving entity substitutions and abstains when answer behaviour violates equivariance.

  • Motivation: LLMs may generate plausible answers from memorised associations when evidence is insufficient, misleading, or conflicting.This can produce confident answers unsupported by the provided passages.
  • Motivation: Existing methods ask whether the model is uncertain or whether the evidence is sufficient, but confidence and self-consistency can coexist with unsupported answers.The paper instead tests whether the generation process is grounded in the evidence.
  • Core idea: Equivariance requires evidence-grounded answers to transform consistently when entities are substituted while relational structure is preserved.The criterion tests structural grounding rather than merely whether outputs change.
  • Twin Worlds: Twin Worlds uses equivariance violations under structure-preserving substitutions as a training-free inference-time abstention signal.When back-mapped answers fail to agree across transformed worlds, the model can abstain.
  • Results: Across four benchmarks and three model backbones, TW outperforms baselines in abstention reliability and provides answer-grounding diagnostics without evidence-faithfulness annotations.The introduction presents this as the framework’s broad empirical result.

2 Related Work

Prior abstention methods estimate risk through uncertainty, context manipulation, agreement, or evidence sufficiency, while prior equivariance work focuses on architectures, training, or evaluation. Twin Worlds addresses the unexplored use of equivariance violations for inference-time abstention without modifying the model.

  • Abstention methods: Abstention methods estimate when to abstain using context manipulations, uncertainty, axiomatic constraints, contrastive decoding, or agreement.These approaches frame abstention as risk estimation during reasoning over evidence.
  • Abstention methods: Sufficiency-based methods evaluate whether retrieved evidence contains the information required to answer.This is a distinct line of work from uncertainty-based abstention.
  • Equivariance: Prior equivariance research establishes how predictions should transform under structured inputs, while NLP work links symmetry failures to spurious heuristics.Other approaches build equivariance into architectures or training, or use violations to study generalisation.
  • Research gap: Whether equivariance can improve LLM reliability at inference time without modifying the model remains unexplored in prior work.Twin Worlds addresses this gap by using equivariance violations as an abstention signal.

3 Methodology

Twin Worlds tests whether evidence-grounded answers transform consistently when entity identities are replaced while relational structure is preserved. It aggregates agreement across synthetic twin worlds and abstains when equivariance is low.

  • Framework overview: TW evaluates whether a language model's answer behaviour is equivariant under structure-preserving entity transformations of questions and evidence.The framework uses equivariance violations as an inference-time signal for answering or abstaining.
  • Structure-preserving transformations: Structure-preserving transformations preserve entity types, remain bijective and reversible, and change only entity identities while retaining relations.Applying a transformation consistently replaces entity mentions throughout the question and evidence.
  • Twin-world construction: TW delexicalises entity mentions into typed placeholders, then relexicalises them with one-to-one mappings to create twin worlds with reduced lexical cues.Synthetic replacement entities preserve relational structure while reducing cues that may activate parametric priors.
  • Equivariance criterion: Equivariance requires the transformed answer, mapped back through π−1, to agree with the original answer rather than remain unchanged.This back-mapping distinguishes equivariance from invariance-based tests that compare transformed and original outputs directly.
  • Abstention rule: TW averages agreement across k sampled twin worlds and abstains when the resulting equivariance score falls below a validation-tuned threshold.The rule signals that the answer is not reliably determined by the relational structure of the provided evidence, without identifying the failure source.

4 Experiments & Results

Across four benchmarks and three model backbones, Twin Worlds improves abstention reliability by testing whether answers transform consistently under structure-preserving entity substitutions.

  • Main Results: TW achieves the strongest overall performance on HotpotQA and MIRAGE across all three backbones, with highest F1 and Accuracy in all six dataset–backbone combinations.It also attains the highest RS in four of six cases while maintaining AR ≤0.029.
  • Main Results: TW improves answer quality without over-refusing on answerable questions, whereas RC-RAG and Context Perturbation abstain more frequently and underperform on F1 and RS.RC-RAG reaches AR values between 0.27 and 0.54, while Context Perturbation reaches up to 0.25.
  • Main Results: On FaithEval, TW achieves the best Accuracy and AR on LLaMA-4 and Mistral-Small and the second-best results on GPT-5.1.FaithEval contains only unanswerable questions, making abstention performance especially important.
  • Main Results: TW achieves the best F1 across all three FEVER backbones, indicating that equivariance-based abstention extends beyond question answering to fact verification.The method abstains when answers fail to track relational structure under entity substitutions.
  • Human Evaluation: Annotators rated structure preservation at 4.61 mean Likert score, with 99.1% of ratings ≥4, while TW reached 83.5% overall agreement with human abstention judgements.Agreement was 86% on answerable and 81% on unanswerable instances, exceeding 90% far from the decision boundary.
  • Probing and Ablations: Typed bijective substitutions best separated faithful from unfaithful evidence, synthetic substitutions reduced entity-associated activation, and k = 3 was the practical default.Removing back-mapping reduced RS by 0.067–0.076, while k = 6 offered negligible gains at doubled inference cost.

5 Conclusion

Twin Worlds probes evidence-grounded reasoning through equivariance: evidence-determined answers should transform consistently under structure-preserving entity substitutions, while violations signal abstention. Across four benchmarks and three backbones, TW outperforms uncertainty- and sufficiency-based baselines while maintaining strong answer quality on answerable instances.

  • Twin Worlds tests whether answers transform consistently under structure-preserving entity substitutions when they are determined by evidence.
  • Equivariance violations provide a training-free, inference-time signal for abstention rather than relying only on model uncertainty or evidence sufficiency.
  • Across four benchmarks and three backbones, TW outperforms uncertainty- and sufficiency-based baselines while maintaining strong answer quality on answerable instances.

Limitations

TW is most effective for entity-grounded reasoning with named-entity structure. Its current scope excludes tasks with limited entity structure, may depend on reliable entity recognition, and has been evaluated only on English benchmarks.

  • TW is most effective when questions, answers, and evidence are structured around named entities.
  • Tasks with limited entity structure, unreliable entity recognition, or multilingual representations may require extensions or language-specific substitution strategies.
  • Extending substitutions to numbers, spans, clauses, or domain-specific terms may broaden applicability to longer-form and open-ended generation.

Ethical considerations

The paper reports publicly available benchmarks without personal or sensitive information and describes human evaluation through Amazon Mechanical Turk. Its experimental pipeline uses fixed public datasets, controlled retrieval, synthetic entities, and batched inference.

  • All four benchmarks are publicly available and contain no personal or sensitive information.
  • Human evaluation is conducted through Amazon Mechanical Turk, assessing evidential sufficiency and structure preservation in factual question answering.
  • The evaluation uses four benchmarks, with 2,000 sampled instances for HotpotQA, MIRAGE, FaithEval, and FEVER.
  • FEVER retrieval uses BM25 over a Wikipedia dump dated 01 November 2025, identically for TW and all baselines to ensure fair comparison.
  • Synthetic entities are procedurally generated without attested referents and validated across all three backbones to remove residual factual associations.
  • The default inference setup batches one original input with three twin worlds for the evaluated backbones.

A.6 Human Evaluation Details

Human evaluation measures evidence sufficiency, transformed-passage structure preservation, and alignment between TW abstentions and majority judgments. Annotators also support semantic agreement checks and abstention-metric evaluation across task types.

  • Annotators judge whether evidence is sufficient without external knowledge and rate transformed-passage relational structure on a five-point Likert scale.
  • Mean structure-preservation rating is 4.61 (SD = 0.48), with 99.1% of ratings at least 4.
  • TW agrees with majority human judgments on 86% of answerable and 81% of unanswerable instances, yielding 83.5% overall agreement and Fleiss’ κ = 0.78.
  • For HotpotQA, MIRAGE, and FaithEval, an independent judge compares semantic equivalence between original and back-mapped twin-world answers, including abstention formulations.
  • For FEVER, outputs are normalized to yes, no, or neither, mapped to Supports, Refutes, or Not Enough Info for evaluation.
  • The evaluation reports F1, Accuracy, Reliability Score, and Abstention Rate, with a confusion matrix crossing answerability and model behavior.
  • Table 9 compares reliability and latency across twin-world counts, highlighting the default and noting negligible gains beyond k = 3.

B.1 Parameter Sensitivity Analysis

TW selects τ = 0.60 and k = 3 as defaults, balancing abstention conservatism, answer coverage, performance, and computational cost.

  • Threshold τ: τ = 0.60 maximises validation RS, with higher thresholds making TW more conservative and lower thresholds increasing answer coverage.The threshold controls how large an equivariance score must be to answer.
  • Number of Twin Worlds k: At k = 3, TW requires agreement in at least two of the three twin worlds.
  • Number of Twin Worlds k: Performance improves steadily from k = 1 to k = 3, changes only marginally thereafter, and computational cost continues increasing.The paper therefore selects k = 3 as the default.

B.2 Error Analysis

TW’s error analysis identifies entity-recognition failures and boundary cases as constraints on substitutions and decision alignment with human sufficiency judgments.

  • NER Failures: Missed entity spans occur in 4.3% of instances and type errors in 1.8% across the evaluation set.On FEVER, the corresponding rates are 6.1% and 2.4%.
  • NER Failures: Missed spans reduce substitution scope, while type errors can make substitutions less natural and weaken the resulting equivariance signal.
  • Boundary Cases: Boundary cases with s(q, C) ∈ {0.33, 0.67} account for 8.3% of the evaluation set around τ = 0.60.
  • Boundary Cases: Human agreement with TW is lowest for boundary cases, where equivariance decisions align less strongly with human sufficiency judgments.Practitioners may adjust τ according to domain-specific costs of answering and abstaining.

B.3 Computational Cost Analysis

TW uses four forward passes at k = 3, achieving competitive latency and higher RS than the compared multi-pass baselines while increasing token processing roughly fourfold.

  • Inference Cost: TW requires k + 1 forward passes per query: one original-input pass and three twin-world passes.The calls are independent and can be batched or parallelised.
  • Inference Cost: 3.8s mean wall-clock latency with four passes matches RC-RAG and is faster than Contrastive Decoding at 4.1s, while achieving higher RS.
  • Inference Cost: At an equal four-pass budget, Self-Consistency trails TW by 0.046 RS.
  • Inference Cost: At k = 3, TW requires approximately four times the token processing of single-pass inference because substitutions approximately preserve input length.

C.1 Case 1: Answerable Question with Faithful Evidence

The case studies show TW preserving an evidence-supported answer under typed entity substitutions, while abstaining when the evidence does not reliably determine the answer.

  • C.1 Case 1: Answerable Question with Faithful Evidence: The answer “dentist” remains unchanged across three typed entity substitutions because it is not itself an entity span.The substitutions transform names, locations, and organizations while preserving the evidence relation.
  • C.1 Case 1: Answerable Question with Faithful Evidence: 3(1 + 1 + 1) = 1.00 ≥ τ = 0.60, so TW answers “dentist” in the faithful-evidence case.
  • C.1 Case 1: Answerable Question with Faithful Evidence: Consistency across synthetic identities provides evidence that the answer is determined by the relational content of the provided evidence.Here equivariance appears as grounded invariance because the predicate remains stable while surrounding entities change.
  • C.2 Case 2: Unanswerable Question without Faithful Evidence: In the Los Angeles example, the passage mentions a mammoth fossil but does not provide sufficient evidence to answer what was found there as posed.
  • C.2 Case 2: Unanswerable Question without Faithful Evidence: 3(0 + 0 + 0) = 0.00 < τ = 0.60, so TW abstains after transformed outputs fail to agree with the original answer.
Loading 2608.28018v1…