PaperlayerView exact revision on arXiv

Source-linked AI summary

UTS at CheckThat! 2026: Cite-Frame Engineering for Generated Fact-Checking Articles

Dima Galat, Marian-Andrei Rizoiu

arXiv:2608.24466v1cs.DLcs.CL

TL;DR

Generated fact-checking articles are difficult to optimize because the scorer rewards reference-entailing citation sentences and penalizes unconstrained LLM prose. UTS therefore combines deterministic drafting with per-citation validation, placing second of 11 teams with M4 = 0.484.

  • Problem

    Fact-checking article generation lacks drafting strategies aligned with a scorer that optimizes entailed citation sentences rather than articles as wholes.

    Source: §1 · p. 1

  • Method

    The system uses a deterministic stub drafter plus one LLM-based shadow-validation call per evidence URL, with no LLM-generated body prose.

    Source: §4.1 · p. 4

  • Results

    UTS placed 2nd of 11 teams on the official leaderboard with M4 = 0.484.

    Source: §5.5 · p. 8

  • Takeaways & Limitations

    Scorer conservatism and judge-based anchor validation are the supported design rules, while citation precision/recall remains the principal performance gap.

    Source: §1 · p. 2, §7 · p. 11

    Selective emission was not tested during the campaign, leaving its proposed role in closing the citation precision/recall gap unresolved.

    Source: §6.1 · p. 10

Abstract

from arXiv · show

CheckThat! 2026 Task 3 asks systems to generate fact-checking articles, graded by an unweighted mean of four sub-metrics (M4). Our UTS submission placed 2nd of 11 teams (M4 = 0.484). The shipped system is a deterministic stub drafter wrapped by two single-lever interventions: a domain-attribution cite frame (HostCite) and a shadow-validated anchor picker (ShadowVal) that use Llama-3.2:1B only as a per-cite validator, never as a body-prose generator. The stack lifts M4 by +0.027 over the stub on the WatClaimCheck validation split, beats the field on entailment and coverage, and follows two design rules our ablation matrix made unambiguous. Scorer conservatism: credit only tokens the references entail - templates pay; LLM prose, reviewer names, and raw evidence all fail. Auxiliary anchor signals are miscalibrated against the Llama judge: every anchor proxy we tried (cross-encoder, length, lead position) picks anchors the judge rejects - gate on the judge itself. The remaining +0.062 gap to the winner sits on citation precision/recall (0.299 vs 0.671), consistent with a selective-emission policy that drops low-confidence cites.

1. Introduction

CheckThat! 2026 Task 3 rewards citation-sentence optimization under strict parsing and entailment checks, motivating a deterministic stub with two narrow interventions rather than LLM-generated prose. The system placed 2nd of 11 teams, while citation precision/recall remained the main gap to the winner.

Scorer mechanics

  • The scorer makes the citation sentence, rather than the article as a whole, the unit of optimisation.Citations are parsed with strict regular expressions, while entailment uses fixed three-sentence chunks and bidirectional RoBERTa-MNLI.Source: §1 · p. 1

Baseline

  • Three LLM drafting configurations all lose to a deterministic templated stub on M4.The attempts used Qwen3:8B or Qwen3:30B-A3B for full-body drafting, plus Qwen3:30B-A3B for lede-only rewriting.Source: §1 · p. 1

Interventions

  • HostCite and ShadowVal provide two narrow deterministic interventions that account for most of the headline lift.HostCite adds domain-attribution frames around citations, while ShadowVal validates anchors with the Llama-3.2:1B judge and escalates when necessary.Source: §1 · pp. 1–2

Design rules

  • The ablation matrix supports preserving chunk counts and emitting only tokens that the scorer’s sub-judges can verify against their references.This principle is expressed through MNLI-side chunk-count preservation and Llama-side cite-sentence entailability.Source: §1 · pp. 1–2

Remaining gap

  • 0.299 vs 0.671 citation P/R leaves a +0.062 M4 gap to the winner despite 2nd-place entailment and 1st-place coverage.The paper identifies selective emission of cites rejected by the per-cite judge as the highest-priority untested mechanism for the next cycle.Source: §1 · p. 2

2. Task and Related Work

Task 3 requires systems to generate free-form fact-checking articles from claims, verdicts, and evidence, with citation-aware scoring across four sub-metrics. The work uses WatClaimCheck validation data and situates the task among prior automated verification and justification-generation research.

Task definition

  • Task 3 inputs contain a claim, claimant, original verdict label, and evidence URLs paired with extracted page text.The required output is a free-form article in which every cited statement is followed by a (source: URL) suffix.Source: §2 · p. 2
  • 1158 test items are evaluated using an unweighted average of four sub-metrics, including entailment and citation precision and recall.Entailment is computed with chunked bidirectional RoBERTa-MNLI against a held-out reference, while citation judgments are made per cite-sentence.Source: §2 · p. 2

Validation data

  • 3372 WatClaimCheck validation items include human reference articles, with experiments using a 200-item stratified subsample for fast iteration.Human references for the test split are unavailable to participants, making test-side entailment unobservable until the post-competition reveal.Source: §2 · p. 2

Prior baselines

  • The study compares against the organiser baseline on the official leaderboard and analyses the IFM_MBZUAI K2-V2-Instruct submission’s citation-format failures.The latter is treated as an external reference submission rather than the primary leaderboard comparison.Source: §2 · p. 2

Related work

  • Related work spans automated claim verification, CheckThat! claim-verification subtasks, and generated justifications for fact-checking verdicts.Examples include extractive–abstractive explanations over PolitiFact, explainable public-health verdicts, and few-shot retrieval-augmented justifications on ExClaim.Source: §2 · pp. 2–3

3. Task Scoring Mechanics

The scorer’s four sub-metrics determine the system’s design choices: entailment rewards reference-style chunk matches, citation scoring verifies each cite against its evidence, and coverage measures evidence-URL inclusion. Across these metrics, unverifiable generated tokens reduce scores, making entailable templates and source-attribution frames central design constraints.

Scorer motivation

  • The scorer’s mechanics, rather than fact-checking generally, determine every design lever and ablation kill in the system.The section traces choices in §4 and ablations in §5.4 to the four sub-metrics.Source: §3 · p. 3

Entailment

  • Entailment averages bidirectional roberta-large-mnli probabilities across three-sentence chunks, with A-side generation matching and B-side reference matching.The final entailment metric averages the precision-flavoured A-side and recall-flavoured B-side scores.Source: §3.1 · p. 3

Citation scoring

  • Citation precision and recall use Llama-3.2:1B yes/no judgments on each cite sentence against its evidence text, with single-URL cites preventing precision dilution.Recall is the rate of yes verdicts across cite-sentences; precision is conditional and equals recall when every cite has one URL.Source: §3.2 · p. 3

Coverage

  • Coverage measures the fraction of provided evidence URLs appearing in the article’s cited-URL union, serving as a prerequisite for top-tier placement rather than a differentiator.Coverage approaches 1.0 when each provided URL receives at least one cite, subject to parser-regex strictness.Source: §3.3 · p. 3

Design rules

  • Unverifiable tokens lower sub-metrics: unmatched entailment chunks reduce A-side scores, while unsupported cite tokens trigger negative Llama judgments.A matching chunk can improve B-side entailment, but templated reviewer-style prose generally mismatches the reference distribution.Source: §3.4 · p. 3, §3.4.1 · p. 4
  • −0.037 P/R followed adding “in a review by {reviewer_name}” to cites, whereas only source-attribution tokens entailable from the URL are permitted.Reviewer names, verdict labels, and dates are uniformly rejected by the Llama judge.Source: §3.4.2 · p. 4

4. System

The system uses a deterministic stub drafter that emits citation-bearing fact-checking articles, with HostCite and ShadowVal as separate per-citation interventions. HostCite improves citation framing, while ShadowVal validates and selectively replaces anchors using Llama-3.2:1B.

Stub drafter

  • The deterministic stub emits a lede, one cite sentence per evidence URL, and a conclusion, using K=1 and parser-compatible semicolon-separated citations.No LLM calls generate body prose; one call per evidence URL is reserved for shadow validation.Source: §4.1 · p. 4, §4.2 · p. 4

HostCite

  • Each cite uses “According to {host}, {anchor} (source: {url}).”, adding domain attribution without reviewer names, verdicts, or dates.The wrapper produces one sentence in and one sentence out, preserving chunk-count invariance.Source: §4.3 · p. 4
  • +0.0199 M4: HostCite lifts the stub at full validation, with gains of +0.035 in precision/recall and +0.010 in entailment.Its paired-bootstrap 95% CI is [+0.0176, +0.0222], and its largest gains occur in the Mixture and Correct Attribution rating buckets.Source: §4.3 · p. 5

ShadowVal

  • +0.0145 M4: ShadowVal lifts the stub by validating proposed anchors with Llama-3.2:1B and escalating through later non-boilerplate sentences when necessary.The gain comes entirely from precision/recall (+0.031), while entailment remains essentially flat; stacked with HostCite, it adds +0.0075 M4.Source: §4.4 · p. 5

5. Experiments and Findings

UTS’s ShadowVal+HostCite system placed 2nd of 11 teams with M4 = 0.484, driven especially by entailment and coverage. Experiments show that cite-frame and judge-gated anchor interventions help, while LLM drafting and several anchor or reviewer-style refinements hurt or fail to improve results.

Validation results

  • M4 rose by +0.0274 over the K = 1 stub on the full validation split, with HostCite contributing +0.0199 and ShadowVal +0.0145.The combined lift was sub-additive, with stack-over-sum ≈0.80, because both interventions raised overlapping cite-sentence precision/recall signals.Source: §5.1 · p. 6

Portfolio selection

  • ShadowVal+HostCite and StatesThat+ShadowVal were statistically tied on the visible test sub-score, requiring both systems to remain in the submission portfolio.The visible scores were 0.589 and 0.585 respectively, with a paired-bootstrap 95% CI of [−0.0009, +0.0085] for the lead-vs-alt difference.Source: §5.2 · p. 7, §5.5 · p. 10

Reviewer analysis

  • StatesThat+ShadowVal significantly outperformed ShadowVal+HostCite on AFP by Δ = −0.0054, while the systems were indistinguishable for every other reviewer.According to {host} matched PolitiFact and Snopes attribution patterns, whereas {host} states that matched AFP’s review-attribution prose better.Source: §5.2 · p. 7, Table 5 · p. 8

Ablations and failures

  • LLM-drafting ablations lost against the stub or lead, including −0.026 M4 for 8B full-body drafting, −0.070 versus the lead for 30B, and −0.011 for lede-only drafting.Anchor-selection refinements also lost against the lead, while reviewer-style lede injection lost −0.019 versus K = 1 and caused 22% of items to hit P = 0.Source: §5.4 · p. 8

Metric decomposition

  • Entailment reached 0.341, ranked 2nd, while coverage reached 0.996, the leaderboard’s highest value.The entailment delta over the baseline was +0.043, and coverage was described as a precondition for top placement rather than a differentiator.Source: §5.5 · p. 8

6. Discussion

The discussion attributes the remaining performance gap primarily to citation precision/recall and argues that selective emission, rather than further cite-content editing, is the next experimental axis. It recommends a per-cite confidence gate and identifies scorer-aware templating and calibration lessons for future cycles.

Performance gap

  • 0.299 vs 0.671 P/R outweighed +0.063 entailment and +0.433 coverage, leaving the 4-metric average 0.062 below Outsider’s.On the visible-scorer test set, the full stack raised P/R from 0.335 (K=1 stub) to 0.389, still below Outsider.Source: §6.1 · p. 10

Selective emission

  • Selective emission is the proposed remaining axis: drop cites whose chosen anchors fail the per-cite faithfulness judge instead of emitting one cite per evidence URL.The cite-content levers plateaued at roughly +0.04 visible-scorer test P/R over K=1 and combined sub-additively.Source: §6.1 · p. 10
  • The recommended refutation is a per-cite confidence gate that runs ShadowVal on the chosen anchor and drops the cite on a negative judgment.This treats cite entailability as allowing some candidates to be omitted rather than requiring every candidate cite to be emitted.Source: §6.1 · p. 10

Scorer-aware design

  • Templated changes that enrich existing chunks outperform those that add or rewrite chunks, while short literal cites beat long evidence premises under the Llama-3.2:1B judge.Paraphrasing citations for readability risks failing the surface-pattern-driven judge.Source: §6.2 · pp. 10–11

Next cycle

  • The next cycle should test the confidence gate first, then build a rating-stratified v2 manifold before v1-tuned ablations to expose HostCite’s rating-mix sensitivity earlier.The discussion presents these as hindsight calibrations for shortening the cite-frame sweep.Source: §6.4 · p. 11

7. Conclusion

UTS placed 2nd of 11 teams in CheckThat! 2026 Task 3, with M4 = 0.484 and a +0.062 gap to the winner concentrated on citation precision/recall. The conclusion recommends selective citation emission first, followed by constrained decoding for LLM-drafted bodies.

7. Conclusion

  • UTS placed 2nd of 11 teams on the official CheckThat! 2026 Task 3 leaderboard with M4 = 0.484.The system used a deterministic stub generator wrapped by two single-lever interventions.Source: §7 · p. 11
  • +0.062 was the gap to the winner, entirely on citation precision/recall, 0.299 vs 0.671.Entailment ranked 2nd and coverage tied for 1st.Source: §7 · p. 11
  • Selective emission is the recommended next lever: a per-cite confidence gate that drops citations when the judge would refuse them.The conclusion hypothesizes that this mechanism underlies top leaderboard entries trading coverage for precision/recall.Source: §7 · pp. 11–12
  • Constrained decoding for LLM-drafted bodies is identified as a second direction to try before abandoning that lever family.It is presented after selective emission as a next-cycle research direction.Source: §7 · p. 12

A. Reproducibility

The system is reproducible from a pinned repository, scorer snapshot, model digest, and citation parser. Runtime and hardware are also documented, with GPU-host re-baselining required because CPU/MPS and CUDA runs shift M4.

Code and data

  • Reproduction requires source repository commit 2744d3f or a descendant, manifest scripts, and the clef2026-checkthat-lab-main snapshot from 2026-04-27.Manifest scripts cover reviewer-stratified, reviewer × rating-stratified, and full 3372-item constructions.Source: A. Reproducibility · p. 14

Models

  • The pinned Llama-3.2:1B blob digest and scorer regex prevent silent scoring changes from model-tag updates or altered citation parsing.At temperature=0, the pinned model is byte-deterministic on the reported hardware; unverified re-pulls can shift absolute scores by 0.005–0.01 M4.Source: A. Reproducibility · p. 14

Hardware and runtime

  • ∼50–90 min per 200-item validation pass, ∼14 h per full-validation pass, and ∼5 h for the 1158-item test split are reported for the GPU host.Development used an M-series MacBook Pro, while production used a local Ubuntu host with an RTX 5090 and 24 GB.Source: A. Reproducibility · p. 14
  • CPU/MPS and CUDA Llama runs shift M4 by ∼0.01–0.02 absolute while preserving rankings, so comparisons must use one host and GPU-host re-baselining.The passage identifies single-host comparisons as the only safe unit before reporting deltas.Source: A. Reproducibility · p. 14

Related papers

Loading 2608.24466v1…