PaperlayerView exact revision on arXiv

Source-linked AI summary

Meet, Compare, or Abstain: LatWeave for Deterministic Multi-Hop Question Answering on Knowledge Lattices

Yuze Ren, Shaoheng Fan, Tao Wang, Yabo Yan, Han Han

arXiv:2609.27225v1cs.CLcs.AIcs.IR

TL;DR

Probabilistic QA can produce answers that are not logically entailed by their evidence and lacks a structural way to refuse unsupported questions. LatWeave uses a knowledge lattice and deterministic operators, with LLMs limited to construction and planning. Its results characterize where this approach works well, where it degrades, and how it abstains under incomplete information.

  • Problem

    Probabilistic QA systems can generate answers not logically entailed by their evidence, and lack a structural mechanism to stop when supporting facts are absent.

    Source: §1 · p. 1

  • Method

    LatWeave organizes knowledge in a lattice and uses meet, compare, and abstain for deterministic execution, while LLMs handle construction and query planning.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    Results are near-lossless on complete or templated knowledge, degrade on open-text boundary tasks, and include IIRC abstain accuracy 0.971 with leak rate 0.029.

    Source: §1 · p. 2, §7.1 · p. 7, §7.2 · p. 7

  • Takeaways & Limitations

    The paper characterizes an operating envelope in which deterministic execution pays no performance penalty and answers can be replayed step by step.

    Source: §1 · p. 1, §9 · p. 8

    Static planning grounds only the first hop, and per-hop replanning would require LLM calls on the answer path, contrary to the paper’s design.

    Source: §7.3 · pp. 7–8

Abstract

from arXiv · show

Probabilistic question-answering systems -- whether large language models (LLMs) themselves, retrieval-augmented generation (RAG), or trained multi-hop retrievers -- conflate "what is known" and "how to reason" into a single probabilistic computation: hallucination cannot be eradicated, evidence chains cannot be audited, and the system answers even when it does not know. We present LatWeave, which organizes knowledge into a multidimensional knowledge lattice and compiles multi-hop QA into three deterministic operators -- meet (constraint intersection), compare (lattice-order comparison), and abstain (structural abstention); LLMs appear only on the construction side (one-shot extraction) and the query-planning side, while the answer-generation path is zero-LLM, zero-task-training, and auditable end to end -- so that question answering over Web-published knowledge becomes reproducible item by item. Rather than claiming across-the-board SOTA, we characterize the operating envelope of this paradigm on six public benchmarks: when knowledge is complete (MetaQA, 39,093 questions) meet chains are near-lossless over three hops (any-hit 0.9975, on par with fully supervised KBQA); on templated multi-hop home ground (2WikiMultihopQA held-out n=1,258) EM 0.865, well above published structure-augmented RAG reproductions; on open-text deep composition (MuSiQue) and extraction-coverage gaps (HotpotQA) we report degradation honestly and attribute it to causes outside the lattice-algebra layer; and when information is incomplete (IIRC) we achieve structural abstention with abstain accuracy 0.971 and leak rate 0.029. Within the operating envelope, deterministic execution pays no performance penalty, and every step on the answer path can be recomputed -- precisely the source of end-to-end auditability.

1 Introduction

Probabilistic QA systems can trace sources without establishing that answers are logically entailed, and they lack a structural way to refuse unsupported questions. LatWeave separates knowledge construction and query planning from deterministic, auditable answer execution.

1 Introduction

  • The paper identifies unsupported answers and unauditable reasoning as shared problems: source provenance does not show that an answer is logically entailed.When retrieval and the graph lack supporting facts, conventional systems have no structural mechanism to stop answering.Source: §1 · p. 1
  • LatWeave confines LLMs to building the lattice and translating queries; its answer path uses zero LLM calls and zero task training.Each answer is an auditable meet chain.Source: §1 · pp. 1–2
  • MetaQA any-hit 0.9975 and 2Wiki held-out EM 0.865 provide upper-bound evidence; IIRC abstain accuracy 0.971 and leak rate 0.029 support structural abstention.The authors also attribute degradation on MuSiQue, HotpotQA, and FRAMES to causes outside lattice algebra.Source: §1 · p. 2
  • LatWeave generalizes the group’s earlier use of lattice partial ordering for applicability gating into a multidimensional retrieval algebra.The authors identify this shift—from gating to retrieval space—as the paper’s core contribution.Source: §1 · p. 2

2 Related Work

Related work spans multi-hop retrieval, RAG and graph augmentation, KBQA, and selective prediction. LatWeave is positioned around the combination of deterministic answers, zero training, auditability, and structural abstention.

2 Related Work

  • Prior QA approaches address different needs: multi-hop recall, retrieval freshness, precise schema-based answers, or refusal based on confidence thresholds.The paper presents LatWeave as combining zero training, deterministic answers, mechanism-level auditability, and structural abstention.Source: §2 · p. 2

2.1 Multi-Hop Question Answering

  • Benchmarks span templated two-hop questions through deep composition and complete through incomplete information; mainstream solvers are probabilistic or training-dependent.The paper names HotpotQA [44], 2WikiMultihopQA [16], MuSiQue, IIRC [8], and FRAMES [23].Source: §2.1 · p. 2

2.2 Retrieval-Augmented and Graph-Augmented LLMs

  • RAG and graph-augmented methods strengthen retrieval and connect semantic chunks, but their answers remain LLM-generated.The authors state that hallucination and auditability problems therefore remain.Source: §2.2 · p. 2

2.3 KBQA and Semantic Parsing

  • KBQA offers verifiable deterministic inference but depends on a ready-made graph and trained parser; LatWeave instead builds a lattice from text with one-shot LLM extraction.Against LLM-Plan [34], LatWeave adds per-dimension constraint meets, compare semantics, structural abstention, and six-benchmark evaluation.Source: §2.3 · p. 2

2.4 Abstention, Selective Prediction, and Hallucination

  • Existing QA abstention generally uses probabilistic confidence thresholds whose risk–coverage trade-off drifts with the task; LatWeave abstains when a fact is absent from its lattice.The paper reports softmax-score rejection below 7.5% coverage at 1% risk and a learned selector at about 15.6% [40].Source: §2.4 · p. 2

2.5 Concurrent Work and Positioning

  • Concurrent systems differ from LatWeave on two poles: they use training or retain LLM calls on the answer path.CompactRAG reduces calls to two per question, but those calls still synthesize the final answer.Source: §2.5 · p. 2

3 The Knowledge Lattice Model

LatWeave models knowledge as a product lattice and treats meet, compare, and abstain as deterministic operations over it. The model also declares a capability boundary for questions requiring derived values.

3 The Knowledge Lattice Model

  • A product lattice gives LatWeave a retrieval space in which instances occupy multidimensional coordinates and queries select points satisfying constraints.The paper distinguishes this from FCA’s focus on shared attributes and knowledge graphs’ entity-relation structure.Source: §3 · pp. 2–3, §3.3 · p. 4
  • Figure 1 places extraction and query planning outside the deterministic answer path, whose operations form an auditable meet chain.Each lattice instance carries source_text.Source: Figure 1 · p. 3

3.1 Posets, Dimensions, and Product Lattices

  • A species × habitat instance is a point such as (whale, aquatic); “aquatic mammals” returns it, while conflicting “terrestrial whales” constraints trigger abstention.The example’s product-lattice slice also runs over the group’s oil and gas production corpus.Source: §3.1 · p. 3
  • Each dimension is a rooted directed acyclic graph whose values order from general to specific; instances bind to coordinates in the product lattice.Instances carry source_text for predicates, and missing components are read as the most general value, ⊤.Source: §3.1 · p. 3
  • The system uses distributivity to check lattice legitimacy, on-demand Dedekind–MacNeille completion for GLBs, FCA for cross-dimension references, and Dilworth width to assess independence.Dimensions with ambiguous sublattices do not go live.Source: §3.1 · p. 3

3.2 Three Deterministic Operators and the Capability Boundary

  • Meet intersects descendant sets per constrained dimension and extracts the greatest element; a dimension with no common lower bound is a conflict that triggers abstention.A hit requires descendant checks on every dimension.Source: §3.2 · p. 3, Figure 2 · p. 4
  • Compare unifies Boolean entity tests, order-based extrema, complete answer-set returns, and abstention when compared objects are outside the lattice.These are presented as instantiations of one lattice-order operator, not separate features.Source: §3.2 · p. 3
  • Questions deriving new values, such as arithmetic or derived counts, fall outside the implemented capability boundary and are excluded from in-scope results.The lattice relates instances through their existing values rather than computing new ones.Source: §3.2 · p. 4

4 LatWeave: System Design

LatWeave converts open text into a lattice, then uses a planner and deterministic three-state execution at query time. Its implementation section reports feasibility and latency figures at the 2Wiki scale.

4 LatWeave: System Design

  • The system separates construction from query-time execution, with LLM extraction and planning kept off the deterministic answer path.Section 4.3 reports implementation and efficiency notes.Source: §4 · p. 4

4.1 Construction: From Text to Lattice

  • Construction uses whole-document extraction for short texts and overlapping windows for long texts, then deduplicates facts and normalizes dimensions against a whitelist.Free naming split “educated at” into 1,017 dimension names during MuSiQue construction.Source: §4.1 · p. 4

4.2 Query Time: Planning and Deterministic Three-State Execution

  • At query time, an LLM planner compiles questions into typed plans, while deterministic execution returns a meet hit, compare result, or abstention.The planner is replaceable; its loss can be measured separately using an oracle-plan ceiling experiment.Source: §4.2 · pp. 4–5

4.3 Implementation and Efficiency

  • On 2Wiki’s 1.2M instances, retrieval runs below 18.7 ms at p50 and below 50 ms at p95; lexicon-scan construction takes 235 ms.Planning for 1,258 held-out questions replays from cache in about 1 s.Source: §4.3 · p. 5

5 Experimental Setup

The experiments use six benchmarks to test distinct roles, from complete knowledge and templated QA to extraction gaps, longer chains, incomplete information, and compounded stress. Construction is question-blind, configurations are frozen before held-out evaluation, and baselines and metrics distinguish answering, completeness, and refusal.

5.1 Datasets and Their Roles

  • MetaQA controls for complete knowledge, 2WikiMultihopQA is the main experiment, HotpotQA and MuSiQue probe boundary conditions, IIRC tests refusal, and FRAMES combines coverage and grounding stress.MetaQA tests whether the lattice loses information; HotpotQA probes fact-coverage gaps, while MuSiQue probes degradation as chains lengthen from 2 to 4 hops.Source: §5.1 · p. 5

5.2 Split Rules and Leakage Control

  • Text benchmarks use a stratified 90/10 dev/held-out split with seed 88; construction is question-blind, configurations are frozen on dev, and held-out evaluation runs once.MetaQA instead uses its official 39,093-question test split, with configuration frozen on official dev and 9-shot exemplars drawn only from train.Source: §5.2 · p. 5

5.3 Baselines and Condition Labeling

  • Baselines comprise a same-corpus, same-scoring zero-training block and a published-reference block, whose protocol differences preclude same-condition win/loss claims.The zero-training block includes keyword retrieval, Naive RAG, and LatWeave; published values are positioned as references only.Source: §5.3 · p. 5

5.4 Metrics

  • Evaluation measures correctness, completeness, and honest refusal, using normalized-string EM, token-overlap F1, and Faithfulness, dual measures, and leak rate.The paper defines F1 as max(F1raw, EM) and pins every LLM call to temperature 0.0 and seed 88.Source: §5.4 · p. 5

6 Main Results: Meet at Its Best

LatWeave performs strongly on templated or complete knowledge, while its results vary with task type and knowledge source. The 2Wiki held-out result and MetaQA control show the range: strong multi-hop performance and near-flat hop accuracy with complete KB knowledge, alongside explicit completeness and input-quality boundaries.

6.1 2WikiMultihopQA: The Main Battlefield

  • Bridge_comparison (0.975) and comparison (0.888) lead the 2Wiki per-type breakdown, matching the compare operator’s lattice-order selection and Boolean-comparison home ground.The comparison is performed without generation.Source: §6.1 · pp. 5–6
  • EM 0.865 on held-out 2WikiMultihopQA (n=1,258) exceeds same-protocol Naive RAG’s 0.198 by 337% and the published HippoRAG 2 value of 0.650 by 33%.The published-value comparison is a paradigm reference, not a same-condition comparison.Source: §6.1 · p. 6

6.2 MetaQA: The Representation-Independence Controlled Comparison

  • MetaQA any-hit is 0.9990 at 1 hop, 0.9983 at 2 hops, and 0.9955 at 3 hops, matching or nearly matching fully supervised results without training.The 2-hop result is within 0.07 percentage points of the best published value; supervised baselines require 96k–330k questions, while LatWeave reads the KB.Source: §6.2 · p. 6
  • MetaQA complete-answer recall is at least 0.996 in every answer-set-size bucket, while strict answer-set EM is 0.6714 and declines as answer sets grow.Over 99% of strict-EM misses still intersect the gold set; none are under-answering misses, and misses involve extra or wrong entities.Source: §6.2 · p. 6
  • For structured sources such as KGs, databases, or tables, the paper recommends mapping knowledge directly to a lattice; the lattice inherits missing edges and dirty triples from its input.The stated production corollary reports any-hit 0.9975, sub-50 ms retrieval, zero training, and zero LLM; fan-out value selection may still need deterministic post-processing or a human.Source: §6.2 · p. 6
  • Figure 3 contrasts near-flat MetaQA lattice any-hit across hops with same-corpus RAG falling to hits@1 0.364 at 3 hops, while MuSiQue lattice EM reaches zero on multi-hop composition.On MuSiQue, lattice EM is 0.048 / 0 / 0; RAG remains at 0.12–0.14 across hop counts.Source: Figure 3 · p. 7

7 Analysis: The Operating Envelope

LatWeave’s boundary failures are attributed to knowledge conversion and query parsing rather than lattice algebra. The section contrasts those open-text limits with near-lossless complete-knowledge performance and examines planner and build-layer constraints.

7.1 Failure Attribution: Four Boundary Classes

  • HotpotQA failures chiefly reflect missing answer edges, MuSiQue failures compound across hops, and IIRC adds derived-value or absent-subject answerability boundaries.On HotpotQA, 90.6% of residual broken chains lack the final-answer edge; on MuSiQue, 48.3% break at the first hop and EM is 0.0248.Source: §7.1 · p. 7

7.2 Hop Decay in Two Contrasts: Complete KB vs. Open Text

  • MuSiQue conditional meet accuracy falls from 0.37–0.44 at hop 1 to 0.09–0.22 at hop 2, approaching zero at hop 3.The analysis attributes the decline to compounding conditional success, with the first hop as the largest loss surface.Source: §7.2 · p. 7
  • On MuSiQue held-out, RAG reaches EM 0.1364 versus the lattice’s 0.0248; on HotpotQA, RAG reaches EM 0.4683 versus 0.1687.The lattice’s full-pool EM is about one third of the same-corpus generative RAG baseline, while resolved-EM is 0.4545 on HotpotQA and 0.938 on 2Wiki.Source: §7.2 · p. 7

7.3 Query-Grounding Levels and Loss Upper Bounds

  • On 2Wiki, oracle-plan resolved-EM is 0.938 versus 0.865 full-pool; on HotpotQA, it is 0.4545 versus 0.1687.The passage attributes only 9.4% of HotpotQA’s residual broken chains to the planner and reports a 4.9% FRAMES oracle ceiling versus EM 2.56%.Source: §7.3 · p. 7
  • Global-vocabulary and entity-menu query grounding both remain insufficient in the HotpotQA and FRAMES boundary experiments.The FRAMES planner restricts relations to those present for the seed entity, but static planning cannot access intermediate-entity menus before earlier hops execute.Source: §7.3 · p. 7
  • L3 replanning hit 0 of 15 FRAMES questions, while hop-2 oracle injection still failed to converge on 11 of 13 HotpotQA questions.L3 also requires one LLM call per hop, conflicting with the zero-LLM answer-path design.Source: §7.3 · p. 8
  • HotpotQA re-extraction raised first-hop connectivity to 78.9%, but only 21.9% of connected questions were answered correctly, yielding bucket EM 0 →17.3%.The estimated full-pool gain is only about +0.6pp, and the passage identifies extraction-mechanism changes as necessary for fundamental improvement.Source: §7.3 · p. 8

8 Abstention as a First-Class Capability

The IIRC evaluation separates answerable-question recall from abstention precision and tests a frozen held-out set. Value cleaning reduces leaks, while derived-value and absent-subject questions mark boundaries where abstention is appropriate.

8.1 IIRC: Abstention in Practice

  • The held-out IIRC set has 35 unanswerable questions: 21 no_hit, 13 sentinel abstains, and one value-match false-positive leak.The 93 answerable abstains comprise 71 no_hit cases and 22 sentinel refusals; held-out and tune-pool outcomes agree item by item.Source: §8.1 · p. 8, Table 8 · p. 8

8.2 Value Cleaning (Ablation)

  • On IIRC dev, value cleaning raised EMfull from 0.0156 to 0.0279 (+79%) and reduced leak rate from 0.052 to 0.017 (−67%).The ablation rewrites sentence-level dimension values into clean entities on the ingestion side.Source: §8.2 · p. 8

8.3 Two Honest Boundaries

  • IIRC value-boundary questions require derived quantities whose anchors exist in the lattice only ∼1.5% of the time, making abstention appropriate.For binary questions, compared entities are mostly not lattice subjects, so Boolean comparison cannot execute and degrades to abstention.Source: §8.3 · p. 8

9 Conclusion

LatWeave compiles multi-hop QA into deterministic meet, compare, and abstain operators, with LLMs confined to construction and query planning. Across six public benchmarks, deterministic execution pays no penalty within its operating envelope, and failure ends in honest abstention.

9 Conclusion

  • LatWeave uses meet, compare, and abstain for multi-hop QA, with LLMs confined to construction and query planning.The answer path is zero-LLM, zero-task-training, and auditable end to end.Source: §9 · p. 8
  • Across six public benchmarks, deterministic execution pays no performance penalty inside its operating envelope.The conclusion also reports that the method has been tested in an actual production environment.Source: §9 · p. 8
  • LatWeave is not meant to replace LLMs, but to move the answer path from a probabilistic black box to an auditable white box.Source: §9 · p. 8

A FRAMES: Controlled Single-Variable Ablation—Locating the Failing Stage

The FRAMES ablation locates its remaining failure in chain semantics rather than extraction coverage: adding evidence pages improves seed landing and chain walkability but leaves held-out EM unchanged. HotpotQA repair experiments likewise find that coverage gains have a ceiling, with many wrong answers caused by gold answers missing from the returned set.

B Evaluation Protocol Details

  • The FRAMES evaluation uses a frozen 90/10 split with seed 88, 742 development questions, and 82 held-out questions reserved for final testing.Construction is question-blind open extraction, with the LLM used only for query-side planning.Source: A FRAMES: Controlled Single-Variable… · p. 10

A FRAMES: Controlled Single-Variable Ablation—Locating the Failing Stage

  • Held-out EM stays 1/82 with or without evidence pages, while inclusion raises seed-entity resolution from 20.7% to 100% and chain-walkable questions from 15% to 87%, locating the remaining bottleneck in chain semantics.Source: A FRAMES: Controlled Single-Variable… · p. 10

C Prompt Templates

  • The supplementary material provides extraction, planner, and RAG-baseline prompts, including planner variants for known relations and entity menus.Source: C Prompt Templates · p. 10

D Construction Statistics

  • The reported construction scale includes a 2Wiki lattice of 1.2M instances and a full MuSiQue lattice of 40,566 instances across 1,741 dimensions.The passage also gives scale statistics for MetaQA, HotpotQA, IIRC, and the held-out-only MuSiQue sublattice.Source: D Construction Statistics · p. 10

E HotpotQA Re-extraction Repair, Measured

  • Re-extracting HotpotQA first-hop gaps yields only 16 genuine bridges among 150 questions, and the measured bridge-recovery and conversion rates imply an optimistic full-pool EM gain of about +0.6pp.Hop-2 conversion to a fully correct answer is 2/36 (5.6%), with a small sample.Source: E HotpotQA Re-extraction Repair,… · p. 10
  • Among 1,437 resolved-but-wrong questions, 77% have the gold answer outside the returned set; only about 11% have its content words fully inside.Questions with two or more returned candidates account for 9.3% of the pool, averaging 3.6 candidates.Source: E HotpotQA Re-extraction Repair,… · p. 11

F Value Cleaning Ablation (IIRC dev)

Value cleaning replaces sentence-level values in entity-typed dimensions with clean entities during ingestion, without changing query or lattice logic. On IIRC dev, this improves EM and precision while reducing leaks and raising abstain accuracy.

F Value Cleaning Ablation (IIRC dev)

G Additional Results Tables

The additional results tables collect approach positioning, benchmark ablations, failure attribution, and IIRC abstention analyses. They also report the full-system 2Wiki result and specify the materials available for independent recomputation, while noting that source code is not supplied.

G Additional Results Tables

  • The supplementary results include a 2Wiki ablation, MetaQA answer-set-size analysis, a cross-dataset failure matrix, and per-question IIRC abstention attribution.Figure 5 reports MuSiQue per-hop conditional meet accuracy on a tune pool of 2,175, grouped by true question depth.Source: G Additional Results Tables · p. 11, Table 10 · p. 11, Figure 5 · p. 11
  • Figure 4 depicts an LLM planner producing a typed plan, deterministic lattice execution returning a meet hit, compare verdict, or abstention, and an answer shipped with evidence text.Source: Figure 4 · p. 11
  • The positioning table marks LatWeave as the only approach family ticking all four columns; its detection criterion distinguishes engineering-level provenance from verifiable reasoning.The accompanying discussion contrasts training, LLM-in-the-loop, schema, and abstention properties across method families.Source: Table 9 · p. 11, G Additional Results Tables · p. 11
  • The full system after domain rebuild reports tune-pool EM 0.886 and held-out EM 0.865.Source: G Additional Results Tables · p. 11

H Auditability and Material Availability

  • Prompts, held-out split manifests, per-question evaluation details, and failure-attribution probes support independent recomputation, but source code is not included because of industrial deployment.Source: H Auditability and Material Availability · p. 12

Related papers

Loading 2609.27225v1…