PaperlayerView exact revision on arXiv

Source-linked AI summary

UTS at CheckThat! 2026: Cite-Frame Engineering for Generated Fact-Checking Articles

Dima Galat, Marian-Andrei Rizoiu

arXiv:2608.24466v1cs.DLcs.CL

TL;DR

Generated fact-checking articles are difficult to optimize because the scorer rewards reference-entailing citation sentences and penalizes unconstrained LLM prose. UTS therefore combines deterministic drafting with per-citation validation, placing second of 11 teams with M4 = 0.484.

  • Problem

    Fact-checking article generation lacks drafting strategies aligned with a scorer that optimizes entailed citation sentences rather than articles as wholes.

  • Method

    The system uses a deterministic stub drafter plus one LLM-based shadow-validation call per evidence URL, with no LLM-generated body prose.

  • Results

    UTS placed 2nd of 11 teams on the official leaderboard with M4 = 0.484.

  • Takeaways & Limitations

    Scorer conservatism and judge-based anchor validation are the supported design rules, while citation precision/recall remains the principal performance gap.

  • Takeaways & Limitations

    Selective emission was not tested during the campaign, leaving its proposed role in closing the citation precision/recall gap unresolved.

Abstract

from arXiv · show

CheckThat! 2026 Task 3 asks systems to generate fact-checking articles, graded by an unweighted mean of four sub-metrics (M4). Our UTS submission placed 2nd of 11 teams (M4 = 0.484). The shipped system is a deterministic stub drafter wrapped by two single-lever interventions: a domain-attribution cite frame (HostCite) and a shadow-validated anchor picker (ShadowVal) that use Llama-3.2:1B only as a per-cite validator, never as a body-prose generator. The stack lifts M4 by +0.027 over the stub on the WatClaimCheck validation split, beats the field on entailment and coverage, and follows two design rules our ablation matrix made unambiguous. Scorer conservatism: credit only tokens the references entail - templates pay; LLM prose, reviewer names, and raw evidence all fail. Auxiliary anchor signals are miscalibrated against the Llama judge: every anchor proxy we tried (cross-encoder, length, lead position) picks anchors the judge rejects - gate on the judge itself. The remaining +0.062 gap to the winner sits on citation precision/recall (0.299 vs 0.671), consistent with a selective-emission policy that drops low-confidence cites.

1. Introduction

CheckThat! 2026 Task 3 rewards citation-sentence optimization under strict parsing and entailment checks, motivating a deterministic stub with two narrow interventions rather than LLM-generated prose. The system placed 2nd of 11 teams, while citation precision/recall remained the main gap to the winner.

  • Scorer mechanics: The scorer makes the citation sentence, rather than the article as a whole, the unit of optimisation.Citations are parsed with strict regular expressions, while entailment uses fixed three-sentence chunks and bidirectional RoBERTa-MNLI.
  • Baseline: Three LLM drafting configurations all lose to a deterministic templated stub on M4.The attempts used Qwen3:8B or Qwen3:30B-A3B for full-body drafting, plus Qwen3:30B-A3B for lede-only rewriting.
  • Interventions: HostCite and ShadowVal provide two narrow deterministic interventions that account for most of the headline lift.HostCite adds domain-attribution frames around citations, while ShadowVal validates anchors with the Llama-3.2:1B judge and escalates when necessary.
  • Design rules: The ablation matrix supports preserving chunk counts and emitting only tokens that the scorer’s sub-judges can verify against their references.This principle is expressed through MNLI-side chunk-count preservation and Llama-side cite-sentence entailability.
  • Remaining gap: 0.299 vs 0.671 citation P/R leaves a +0.062 M4 gap to the winner despite 2nd-place entailment and 1st-place coverage.The paper identifies selective emission of cites rejected by the per-cite judge as the highest-priority untested mechanism for the next cycle.

2. Task and Related Work

Task 3 requires systems to generate free-form fact-checking articles from claims, verdicts, and evidence, with citation-aware scoring across four sub-metrics. The work uses WatClaimCheck validation data and situates the task among prior automated verification and justification-generation research.

  • Task definition: Task 3 inputs contain a claim, claimant, original verdict label, and evidence URLs paired with extracted page text.The required output is a free-form article in which every cited statement is followed by a (source: URL) suffix.
  • Task definition: 1158 test items are evaluated using an unweighted average of four sub-metrics, including entailment and citation precision and recall.Entailment is computed with chunked bidirectional RoBERTa-MNLI against a held-out reference, while citation judgments are made per cite-sentence.
  • Validation data: 3372 WatClaimCheck validation items include human reference articles, with experiments using a 200-item stratified subsample for fast iteration.Human references for the test split are unavailable to participants, making test-side entailment unobservable until the post-competition reveal.
  • Prior baselines: The study compares against the organiser baseline on the official leaderboard and analyses the IFM_MBZUAI K2-V2-Instruct submission’s citation-format failures.The latter is treated as an external reference submission rather than the primary leaderboard comparison.
  • Related work: Related work spans automated claim verification, CheckThat! claim-verification subtasks, and generated justifications for fact-checking verdicts.Examples include extractive–abstractive explanations over PolitiFact, explainable public-health verdicts, and few-shot retrieval-augmented justifications on ExClaim.

3. Task Scoring Mechanics

The scorer’s four sub-metrics determine the system’s design choices: entailment rewards reference-style chunk matches, citation scoring verifies each cite against its evidence, and coverage measures evidence-URL inclusion. Across these metrics, unverifiable generated tokens reduce scores, making entailable templates and source-attribution frames central design constraints.

  • Scorer motivation: The scorer’s mechanics, rather than fact-checking generally, determine every design lever and ablation kill in the system.The section traces choices in §4 and ablations in §5.4 to the four sub-metrics.
  • Entailment: Entailment averages bidirectional roberta-large-mnli probabilities across three-sentence chunks, with A-side generation matching and B-side reference matching.The final entailment metric averages the precision-flavoured A-side and recall-flavoured B-side scores.
  • Citation scoring: Citation precision and recall use Llama-3.2:1B yes/no judgments on each cite sentence against its evidence text, with single-URL cites preventing precision dilution.Recall is the rate of yes verdicts across cite-sentences; precision is conditional and equals recall when every cite has one URL.
  • Coverage: Coverage measures the fraction of provided evidence URLs appearing in the article’s cited-URL union, serving as a prerequisite for top-tier placement rather than a differentiator.Coverage approaches 1.0 when each provided URL receives at least one cite, subject to parser-regex strictness.
  • Design rules: Unverifiable tokens lower sub-metrics: unmatched entailment chunks reduce A-side scores, while unsupported cite tokens trigger negative Llama judgments.A matching chunk can improve B-side entailment, but templated reviewer-style prose generally mismatches the reference distribution.
  • Design rules: −0.037 P/R followed adding “in a review by {reviewer_name}” to cites, whereas only source-attribution tokens entailable from the URL are permitted.Reviewer names, verdict labels, and dates are uniformly rejected by the Llama judge.

4. System

The system uses a deterministic stub drafter that emits citation-bearing fact-checking articles, with HostCite and ShadowVal as separate per-citation interventions. HostCite improves citation framing, while ShadowVal validates and selectively replaces anchors using Llama-3.2:1B.

  • Stub drafter: The deterministic stub emits a lede, one cite sentence per evidence URL, and a conclusion, using K=1 and parser-compatible semicolon-separated citations.No LLM calls generate body prose; one call per evidence URL is reserved for shadow validation.
  • HostCite: Each cite uses “According to {host}, {anchor} (source: {url}).”, adding domain attribution without reviewer names, verdicts, or dates.The wrapper produces one sentence in and one sentence out, preserving chunk-count invariance.
  • HostCite: +0.0199 M4: HostCite lifts the stub at full validation, with gains of +0.035 in precision/recall and +0.010 in entailment.Its paired-bootstrap 95% CI is [+0.0176, +0.0222], and its largest gains occur in the Mixture and Correct Attribution rating buckets.
  • ShadowVal: +0.0145 M4: ShadowVal lifts the stub by validating proposed anchors with Llama-3.2:1B and escalating through later non-boilerplate sentences when necessary.The gain comes entirely from precision/recall (+0.031), while entailment remains essentially flat; stacked with HostCite, it adds +0.0075 M4.

5. Experiments and Findings

UTS’s ShadowVal+HostCite system placed 2nd of 11 teams with M4 = 0.484, driven especially by entailment and coverage. Experiments show that cite-frame and judge-gated anchor interventions help, while LLM drafting and several anchor or reviewer-style refinements hurt or fail to improve results.

  • Validation results: M4 rose by +0.0274 over the K = 1 stub on the full validation split, with HostCite contributing +0.0199 and ShadowVal +0.0145.The combined lift was sub-additive, with stack-over-sum ≈0.80, because both interventions raised overlapping cite-sentence precision/recall signals.
  • Portfolio selection: ShadowVal+HostCite and StatesThat+ShadowVal were statistically tied on the visible test sub-score, requiring both systems to remain in the submission portfolio.The visible scores were 0.589 and 0.585 respectively, with a paired-bootstrap 95% CI of [−0.0009, +0.0085] for the lead-vs-alt difference.
  • Reviewer analysis: StatesThat+ShadowVal significantly outperformed ShadowVal+HostCite on AFP by Δ = −0.0054, while the systems were indistinguishable for every other reviewer.According to {host} matched PolitiFact and Snopes attribution patterns, whereas {host} states that matched AFP’s review-attribution prose better.
  • Ablations and failures: LLM-drafting ablations lost against the stub or lead, including −0.026 M4 for 8B full-body drafting, −0.070 versus the lead for 30B, and −0.011 for lede-only drafting.Anchor-selection refinements also lost against the lead, while reviewer-style lede injection lost −0.019 versus K = 1 and caused 22% of items to hit P = 0.
  • Metric decomposition: Entailment reached 0.341, ranked 2nd, while coverage reached 0.996, the leaderboard’s highest value.The entailment delta over the baseline was +0.043, and coverage was described as a precondition for top placement rather than a differentiator.

6. Discussion

The discussion attributes the remaining performance gap primarily to citation precision/recall and argues that selective emission, rather than further cite-content editing, is the next experimental axis. It recommends a per-cite confidence gate and identifies scorer-aware templating and calibration lessons for future cycles.

  • Performance gap: 0.299 vs 0.671 P/R outweighed +0.063 entailment and +0.433 coverage, leaving the 4-metric average 0.062 below Outsider’s.On the visible-scorer test set, the full stack raised P/R from 0.335 (K=1 stub) to 0.389, still below Outsider.
  • Selective emission: Selective emission is the proposed remaining axis: drop cites whose chosen anchors fail the per-cite faithfulness judge instead of emitting one cite per evidence URL.The cite-content levers plateaued at roughly +0.04 visible-scorer test P/R over K=1 and combined sub-additively.
  • Selective emission: The recommended refutation is a per-cite confidence gate that runs ShadowVal on the chosen anchor and drops the cite on a negative judgment.This treats cite entailability as allowing some candidates to be omitted rather than requiring every candidate cite to be emitted.
  • Scorer-aware design: Templated changes that enrich existing chunks outperform those that add or rewrite chunks, while short literal cites beat long evidence premises under the Llama-3.2:1B judge.Paraphrasing citations for readability risks failing the surface-pattern-driven judge.
  • Next cycle: The next cycle should test the confidence gate first, then build a rating-stratified v2 manifold before v1-tuned ablations to expose HostCite’s rating-mix sensitivity earlier.The discussion presents these as hindsight calibrations for shortening the cite-frame sweep.

7. Conclusion

UTS placed 2nd of 11 teams in CheckThat! 2026 Task 3, with M4 = 0.484 and a +0.062 gap to the winner concentrated on citation precision/recall. The conclusion recommends selective citation emission first, followed by constrained decoding for LLM-drafted bodies.

  • 7. Conclusion: UTS placed 2nd of 11 teams on the official CheckThat! 2026 Task 3 leaderboard with M4 = 0.484.The system used a deterministic stub generator wrapped by two single-lever interventions.
  • 7. Conclusion: +0.062 was the gap to the winner, entirely on citation precision/recall, 0.299 vs 0.671.Entailment ranked 2nd and coverage tied for 1st.
  • 7. Conclusion: Selective emission is the recommended next lever: a per-cite confidence gate that drops citations when the judge would refuse them.The conclusion hypothesizes that this mechanism underlies top leaderboard entries trading coverage for precision/recall.
  • 7. Conclusion: Constrained decoding for LLM-drafted bodies is identified as a second direction to try before abandoning that lever family.It is presented after selective emission as a next-cycle research direction.

A. Reproducibility

The system is reproducible from a pinned repository, scorer snapshot, model digest, and citation parser. Runtime and hardware are also documented, with GPU-host re-baselining required because CPU/MPS and CUDA runs shift M4.

  • Code and data: Reproduction requires source repository commit 2744d3f or a descendant, manifest scripts, and the clef2026-checkthat-lab-main snapshot from 2026-04-27.Manifest scripts cover reviewer-stratified, reviewer × rating-stratified, and full 3372-item constructions.
  • Models: The pinned Llama-3.2:1B blob digest and scorer regex prevent silent scoring changes from model-tag updates or altered citation parsing.At temperature=0, the pinned model is byte-deterministic on the reported hardware; unverified re-pulls can shift absolute scores by 0.005–0.01 M4.
  • Hardware and runtime: ∼50–90 min per 200-item validation pass, ∼14 h per full-validation pass, and ∼5 h for the 1158-item test split are reported for the GPU host.Development used an M-series MacBook Pro, while production used a local Ubuntu host with an RTX 5090 and 24 GB.
  • Hardware and runtime: CPU/MPS and CUDA Llama runs shift M4 by ∼0.01–0.02 absolute while preserving rankings, so comparisons must use one host and GPU-host re-baselining.The passage identifies single-host comparisons as the only safe unit before reporting deltas.
Loading 2608.24466v1…