Source-linked AI summary
ElementCheck: Complexity-Aware Long-Form Text Factuality Evaluation via Sentence Elements
Xinming Wang, Haoran Du, Yi Chen, Jian Xu, Hongming Yang, Han Hu, Yulong Chen, Cheng-Lin Liu, Xu-Yao Zhang
TL;DR
Long-form factuality evaluation struggles with noisy claim decomposition and fixed verification granularity. ElementCheck uses sentence elements and graph-based complexity to route direct or refined verification, and experiments show consistent improvements with stable, cost-aware evaluation across backbones. Its scope remains limited by backbone-dependent sentence acquisition, partial discourse coverage, noisy retrieval, and heuristic re-verification.
Problem
Long-form factuality evaluation suffers from noisy decomposition and fixed verification granularity, while fine-grained sentence-level resources for evaluating fact-checking pipelines are limited.
Method
ElementCheck extracts explicitly connected entity pairs, organizes them into element graphs, and routes sentences to direct verification or targeted element-level refinement based on graph complexity.
Results
ElementCheck consistently improves factuality verification across five backbone models and multiple benchmarks while maintaining a favorable balance between verification performance and computational cost.
Takeaways & Limitations
Complexity-aware verification preserves sentence context, allocates refinement effort to structurally difficult cases, and remains stable across different model backbones.
Takeaways & Limitations
The method depends on backbone capacity for sentence acquisition and only partially addresses discourse-level factuality, while retrieval remains noisy and re-verification uses heuristic subgraph reconstruction.
Abstract
from arXiv · showhide
Existing long-form factuality evaluation relies on the decompose-retrieve-verify pipeline. However, the pipeline suffers from noise from claim decomposition and fixed verification granularity, resulting in unreliable results. We propose ElementCheck, a complexity-aware framework that verifies long-form outputs via sentence elements. Instead of uniformly decomposing sentences into atomic sub-claims, ElementCheck extracts entity pairs that are explicitly linked through verifiable connections in the original sentence as elements, and organizes these into an element graph. The graph topology provides a structural signal for estimating sentence complexity, enabling direct verification for simple sentences and targeted element-level refinement and verification for complex ones. To support fine-grained evaluation, we construct a new benchmark FastFact-Sent by mapping isolated claims from FastFact-Bench back to their source sentences. Experiments on FastFact-Sent and two domain-specific benchmarks show ElementCheck consistently improves factuality verification across five backbone models while maintaining a favorable accuracy-cost trade-off. Further analyses demonstrate that complexity-aware verification reduces unnecessary re-verification and maintains stability across different backbones.
1 Introduction
ElementCheck addresses noise and inflexible verification granularity in long-form factuality evaluation with complexity-aware sentence-element verification. It also introduces FastFact-Sent for sentence-level evaluation and reports consistent gains across backbones and benchmarks.
- Motivation: Long-form outputs contain dense, interdependent factual claims whose errors are difficult to localize and verify.
- Limitations: Existing Decompose-Retrieve-Verify systems decompose text into atomic claims, retrieve evidence, and verify each claim, but decomposition can introduce over-decomposed or noisy claims.
- Limitations: Using one verification procedure for every claim can waste computation on simple claims or inadequately verify complex ones.
- ElementCheck: ElementCheck extracts explicitly connected entity pairs, builds an element graph, and adapts verification granularity using graph structure.
- Benchmark: FastFact-Sent maps human-annotated FastFact-Bench claims to source sentences, yielding 5,020 factually verified sentences across 380 model responses.
- Results: Across five backbone models and three benchmark settings, ElementCheck consistently outperforms baselines while balancing verification performance and computational cost.
2 Related Work
Related work establishes automated fact-checking as an evidence-based pipeline and motivates dedicated long-form evaluation because long-form generations contain many elements with mixed verifiability.
- Automated Fact-Checking: Automated fact-checking typically comprises claim detection and extraction, evidence retrieval, and verdict prediction.
- Automated Fact-Checking: Controlled benchmarks such as FEVER, HoVer, WikiFactCheck, and ChartCheck have extensively studied evidence-based fact-checking.
- Long-Form Factuality Evaluation: Existing approaches primarily target single-claim verification, whereas long-form generations rapidly increase the number of factual elements and the precision-efficiency trade-off.
- Long-Form Factuality Evaluation: Long-form factuality methods include atomic-claim checking, static relation graphs, external retrieval and execution tools, and search-augmented evaluation.
3 Method
ElementCheck represents check-worthy sentences through entity-linked factual elements and an element graph, then routes simple and complex sentences to different verification procedures. Complex cases receive element-level checking and targeted re-verification.
- Sentence Acquisition: ElementCheck uses the sentence as the initial claim rather than beginning with atomic claim extraction, preserving context and mitigating over-decomposition and semantic drift.
- Sentence Acquisition: A check-worthiness operator filters an input sentence sequence into factuality-checkable sentences, excluding intrinsically unverifiable content.
- Element Extraction: Elements are entity pairs explicitly connected by linguistically licensed relations that jointly express a verifiable fact, rather than merely co-occurring.
- Element Extraction: The element graph uses factual entities as nodes and explicit factual connections as edges while preserving predicate, scope, modality, and attribution in sentence context.
- Complexity-aware Verification: Graph diameter and connectivity estimate sentence complexity: large diameter indicates long-range dependencies, while disconnection indicates independent factual components.
- Complexity-aware Verification: For connected graphs, 73.0% have diameter no larger than 3, motivating the threshold δ = 3 for routing simple versus complex graphs.
- Complexity-aware Verification: Simple sentences receive direct ternary verification against retrieved evidence, while complex sentences receive local element verdicts and refinement of unresolved NEI regions.
- Complexity-aware Verification: ElementCheck reconstructs uncertain local subgraphs into contextualized claims, retrieves new evidence, and directly verifies only those refined claims.
4 Experiments Setup
The experiments compare ElementCheck with factuality-verification and claim-refinement baselines across three benchmarks, using coverage, balanced accuracy, and Overall as primary measures.
- Verification Procedure: Algorithm 1 takes a check-worthy sentence, its element graph, initial evidence, a retriever, and threshold δ, then returns a ternary factuality verdict.The algorithm routes structurally simple connected graphs to direct verification and otherwise performs element-level verification with refinement.
- Baselines: ElementCheck is benchmarked against SAFE, VeriScore, FastFact, VeriFastScore, DnDScore, and VeriFact.Claim-refinement baselines are evaluated on the same verifiable sentences for fair comparison.
- Datasets: FastFact-Sent remaps human-annotated FastFact-Bench claims to their originating response sentences for sentence-level evaluation.The construction retains 5,020 valid unique sentence alignments from 6,953 original claims.
- Datasets: The experiments use FastFact-Sent, AdjuvantBench, and CatalystBench, with FastFact-Sent aggregating responses from multiple domains.The aggregated sources include FactScore-Bio, Factcheck-Bench, ExpertQA, LongFact, and HelloBench.
- Metrics: The main task merges labels into Supported and Not Supported, while also reporting three-class balanced accuracy.Not Supported combines Refute and NEI instances.
- Metrics: Coverage measures effectively verified source sentences, BAcc measures balanced accuracy on covered items, and Overall equals Cov × BAcc.Class-specific recalls are computed within the same valid coverage scope.
5 Experiments Results
ElementCheck improves factuality verification across backbones and domains while balancing accuracy, coverage, and computational cost. Ablations indicate that adaptive refinement is most useful for structurally complex sentences and reduces unnecessary re-verification.
- Main Results: 65.9% Overall is achieved by ElementCheck under Qwen3-235B, the best average Overall score within each backbone group.The method maintains competitive sentence-level coverage and balanced accuracy.
- Main Results: ElementCheck remains robust across backbone variations and generalizes across general and specialized scientific domains with relatively low performance variance.Element graphs provide a stable intermediate structure and help align claims with evidence across diverse knowledge distributions.
- Efficiency: ElementCheck offers a favorable computational trade-off by reducing unnecessary calls while preserving sentence-level verification.FastFact is cheaper but verifies fewer units and yields weaker verification quality.
- Ablations: Expanding the evidence pool alone does not reliably improve verification, whereas adding Re-Verify to normal Top-10 retrieval improves balanced accuracy and NOSUP recall.Richer Jina evidence can introduce irrelevant or noisy information that interferes with verification.
- Ablations: Re-Verify provides little benefit for structurally simple sentences but is clearly more useful for structurally complex sentences.This supports reserving stronger verification for sentences with difficult entity relations or qualifications.
- Ablations: 58.6% to 21.4%: the Graph Router lowers the Re-Verify trigger rate under GPT-4o-mini without weakening balanced accuracy.Performance remains stable and slightly improves across tested backbones.
- Ablations: δ = 3 gives the best overall balance while ElementCheck remains stable across tested threshold settings.The selected threshold is consistent with the graph diameter distribution reported in Appendix C.5.
- Further Analysis: ElementCheck achieves a superior coverage–accuracy balance and remains stable across backbones, while extraction precision matters more than exhaustive element-pair recall.The element graph is also described as representation-efficient and scalable with moderate token overhead.
6 Conclusion
ElementCheck grounds long-form factuality evaluation in sentence-level fact elements, preserving sentence context while enabling complexity-aware verification. Across multiple benchmarks, it improves balanced accuracy and coverage–precision balance while remaining stable across model backbones.
- ElementCheck grounds long-form factuality evaluation in sentence-level fact elements, preserving sentence context while enabling complexity-aware verification.
- Experiments across multiple benchmarks show consistently improved balanced accuracy and a stronger balance between coverage and precision.
- ElementCheck remains stable across different model backbones, providing a more faithful and scalable basis for long-form factuality evaluation.
Limitations
ElementCheck has limitations involving backbone dependence, incomplete discourse-level factuality coverage, noisy retrieval, and heuristic subgraph reconstruction.
- Verifiable sentence acquisition depends on the underlying backbone model, so weaker models may miss check-worthy content before verification begins.
- The sentence-level formulation only partially addresses discourse-level factuality, including multi-paragraph evidence aggregation and cross-sentence temporal reasoning.
- Evidence retrieval remains noisy because longer document contexts may introduce irrelevant or conflicting information without improving verification accuracy.
- The current Re-Verify module reconstructs subgraph context with heuristic rules rather than more principled graph reasoning and evidence aggregation.
Ethical Considerations
The paper describes its benchmark sources, baseline pipelines, FastFact-Sent construction, and human validation procedures. Human validation supports reliability analysis rather than replacing automatic verification.
- Ethical considerations: Human validation served as a reliability check and error-diagnosis tool, not as an absolute ground-truth replacement for automatic verification.
- Datasets: FastFact-Bench contains 400 long-form QA generations across five source benchmarks and provides human annotations for claim extraction and verification outcomes.
- Dataset construction: FastFact-Sent maps atomic verifiable claims back to originating sentences through semantic retrieval ranking and model-based discrimination.
- Dataset construction: The constructed dataset retains 5,908 claims and 5,020 unique verifiable sentences from 6,953 raw claims.
- Human validation: Human validation assessed GPT-5.1’s sentence-claim alignment judgments on a stratified sample of 500 sentence-claim pairs.
- Human validation: Annotator agreement with GPT-5.1 was 97.4% for Annotator 1 and 95.2% for Annotator 2, with limited inference required by the entailment task.
A.4 Details of Metrics
The evaluation uses coverage-sensitive, class-balanced, and structural metrics to assess factuality, efficiency, fine-grained labels, and backbone stability. Results indicate that ElementCheck balances coverage and precision at moderate cost.
- Coverage: Coverage measures whether sentences containing factual elements are checked, using extracted claims or rendered verification verdicts for sentence-level methods.
- F1@K: F1@K balances verification accuracy and information coverage by taking the harmonic mean of Precision and Recall@K.
- F1@K: For ElementCheck, K is calibrated to the average number of verifiable sentences per response rather than the average number of atomic claims.
- Soft-Semantic F1: Soft-Semantic F1 measures structural consistency between element graphs using semantic precision and recall based on maximum cosine similarity.
- Cost and efficiency: ElementCheck maintains moderate cost while preserving sentence-level verification coverage across additional backbones.
- Fine-grained results: In the stricter 3-class setting, ElementCheck remains competitive across backbones and domains for distinguishing Support, Refute, and NEI.
- Backbone comparison: Generating or identifying more verifiable sentences alone does not guarantee stronger factuality evaluation, motivating joint reporting of coverage-sensitive and class-balanced metrics.
- Claim refinement comparison: ElementCheck remains close to claim-refinement baselines while using a simpler sentence-level pipeline with fewer decomposition and refinement steps.
B.5 Per-Class Error Analysis
ElementCheck’s gains are concentrated on difficult factuality decisions and remain comparatively stable across backbones and structured representations. Its graph-guided design also improves the accuracy–coverage–cost balance while reducing sensitivity to some extraction errors.
- Per-class gains: ElementCheck’s improvements primarily concentrate on Refute and NEI rather than Support.Fine-grained structural anchors are most valuable for identifying contradictions or evidence gaps.
- Error patterns: ElementCheck’s remaining errors are predominantly gold-Support sentences shifted to NEI.The paper characterizes this conservative abstention pattern as preferable because false support is more harmful than false abstention.
- Backbone stability: Element graphs remain semantically consistent across backbones, and changing the graph extractor causes only small performance variation under a fixed verifier.The results suggest verifier reasoning capacity contributes more to performance differences than instability in extracted graphs.
- Extraction perturbations: ElementCheck is relatively robust to pair-direction reversal and omission, moderately sensitive to entity replacement, and most sensitive to spurious pair addition.These perturbation results indicate that extraction precision matters more than recall for stable downstream verification.
- Representation and cost: Element Graph provides the most favorable trade-off among structured representations, improving verification over direct sentence checking without the larger overhead of SRL frames and triplet extraction.Graph routing also distinguishes sentences suitable for direct verification from those requiring stronger verification.
C.4 Scalability Analysis
The scalability analysis examines graph construction and verification cost as documents lengthen. Element graphs grow in a structured way, with bounded per-sentence complexity allowing many sentences to remain on the direct verification path.
- Analysis scope: Graph construction and verification cost are evaluated at separate levels as document length increases.The analysis measures graph growth during construction and verification cost after graph-guided routing.
- Graph growth: Graph size grows with document length, but longer responses split into more graph components rather than forming one monolithic graph.The per-sentence number of element pairs remains bounded.
- Verification routing: Many structurally simple sentences remain eligible for the direct verification path as responses become longer.This supports graph-guided routing without requiring verification costs to increase proportionally with response length.
- Complexity boundary: Most connected graphs have small diameters, while disconnected graphs reflect sentences containing multiple loosely coupled semantic units.The reported diameter distribution motivates δ = 3 as a boundary between structurally simple and complex sentences.
- Prompted processing: The extraction and verification prompts operate sentence by sentence, resolving coreference and verifying context-anchored entity pairs against evidence.The element prompt includes entity, temporal, spatial, causal, and other verifiable connections, while uncertain pairs may be skipped.
E Case Study
The case study illustrates ElementCheck on a long response about eating only vegetables for a week. It converts sentences into linked elements and uses evidence-dependent verification outcomes to distinguish supported claims from claims requiring re-verification.
- Case setup: The case study asks what may happen if someone eats only vegetables for a week and produces a detailed response covering benefits and challenges.The response discusses nutrient intake, digestion, energy, psychological effects, and longer-term considerations.
- Nutrient effects: The response describes vegetables as sources of vitamins, minerals, antioxidants, and fiber, while noting possible gaps in protein, healthy fats, and vitamins B12 and D.It also distinguishes short-term effects from longer-term risks of restricting other food groups.
- Digestive changes: A sudden increase in vegetable fiber may cause gas, bloating, or diarrhea, whereas cooking vegetables or increasing fiber gradually may mitigate discomfort.The same section presents fiber as a prebiotic that can support beneficial gut bacteria and bowel regularity over time.
- Conclusion: The case study states that a one-week vegetable-only diet is unlikely to cause severe harm but should not be extended long term without careful planning.It highlights possible short-term energy dips, cravings, and digestive adjustments alongside the need to avoid nutrient gaps.
- Element verification: A sentence linking vegetable variety, quantity, and preparation to short-term benefits and potential challenges is verified as supported across all extracted element pairs.Its element pairs connect the eating pattern to the week, outcomes, and conditioning factors.
- Re-verification: A sentence linking diverse vegetables to immune, skin, and digestive benefits initially requires re-verification because none of its extracted pairs receive support.The re-verification evidence specifically includes a digestion claim tied to high fiber content.