Source-linked AI summary
Calibrating Small Language Models for Claim Check-Worthiness Detection
Pratuat Amatya, Venktesh Viswanathan, Vinay Setty
TL;DR
High-volume fact-checking needs cheaper claim check-worthiness detection because large LLM inference is costly and slow, while smaller models lose accuracy. The paper introduces NN-PPI, a pointwise residual-calibration layer using a small labeled set without retraining, and reports substantial gains across model scales plus improvements to a production fine-tuned model. The authors conclude that residual calibration can narrow the performance gap while retaining small-model deployment advantages.
Problem
Large LLMs are costly and latency-prohibitive for every incoming claim, while cheaper small models sacrifice accuracy and deployed systems need adaptation without retraining.
Method
NN-PPI extends PPI to pointwise inference by using semantically nearest labeled neighbors to apply residual corrections to frozen model predictions at inference time.
Results
NN-PPI improves weighted F1 across model scales, with the largest gains for smaller models, and improves production XLM-RoBERTa-Large by 11% over baseline and 7.03% over KNN on CLEF 2024.
Takeaways & Limitations
Residual calibration narrows the gap between small and larger models and is complementary to supervised fine-tuning.
Takeaways & Limitations
NN-PPI relies on a small distributionally representative calibration set, whose optimal selection remains challenging because semantic similarity may not always reflect distributional similarity.
Abstract
from arXiv · showhide
Assessing claim check-worthiness is an essential first step in automated fact-checking pipelines. This work is motivated by a real deployment challenge at an early-stage startup: running large language models (LLMs) over every incoming claim is cost- and latency-prohibitive, yet smaller models sacrifice accuracy. We propose NN-PPI, a pointwise extension of Prediction-Powered Inference (PPI) that calibrates model predictions at inference time as a lightweight post-hoc layer, without re-training the underlying model. NN-PPI achieves weighted F1 gains ranging from 12% to 33.80% depending on the size and performance of the baseline model, bringing SLMs on par with larger LLMs. Beyond few-shot SLMs, NN-PPI further improves a production-deployed fine-tuned model, demonstrating that residual calibration is complementary to supervised fine-tuning. By recovering LLM-level accuracy from models that are an order of magnitude cheaper to serve, it makes accurate check-worthiness detection substantially cheaper to operate at scale. Our code and data can be found at https://anonymous.4open.science/r/arr-claim-worthiness-F237.
1 Introduction
The paper addresses the cost, latency, and adaptivity constraints of claim check-worthiness detection by calibrating small-model predictions against human judgments without retraining. It proposes NN-PPI and evaluates its effects across model scales and baselines.
- Motivation: Production fact-checking services need inexpensive, low-latency claim filtering, but small models generally trail large LLMs in accuracy.The task filters high-volume streams before claims reach human fact-checkers.
- Motivation: Large language models can be unreliable, prompt-sensitive, and poorly aligned with human judgments of check-worthiness.These limitations motivate post-hoc calibration of their predicted outputs.
- Approach: NN-PPI calibrates LLM outputs using a small manually annotated set, extending PPI from population-level metrics to per-instance confidence intervals.The method is designed to align predictions more closely with human judgments without retraining the underlying model.
- Evaluation: The study compares uncalibrated few-shot LLMs, plain KNN label averaging, and NN-PPI across model scales, datasets, and neighbor sizes.The research questions cover calibration effects, baseline comparisons, and the impact of k.
2 Related Works
Claim check-worthiness detection is an early filtering task in automated fact-checking, studied through supervised, ranking, and few-shot or zero-shot language-model approaches. The paper situates NN-PPI within this landscape and presents its overview as a calibration method for model predictions.
- Task framing: Claim check-worthiness detection identifies which input claims should proceed to fact-checking in an automated pipeline.It is commonly framed as classification, with claim ranking as an alternative formulation.
- Prior approaches: Prior work includes classical supervised methods, fine-tuned transformers, and more recent few-shot or zero-shot large-language-model systems.Existing approaches span both classification and prioritization formulations.
- NN-PPI overview: Figure 1 depicts NN-PPI combining a frozen language-model score with a nearest-neighbor residual correction before thresholding the calibrated decision.The overview also includes a per-instance confidence interval.
3 Method
NN-PPI is a post-hoc, pointwise calibration layer that uses semantically similar labeled neighbors to correct frozen language-model scores at inference time. It produces calibrated decisions and uncertainty estimates without modifying the underlying model.
- Method: NN-PPI calibrates frozen LLM predictions at inference time using a small labeled calibration set, without changing the underlying model.It extends PPI to pointwise claim-level calibration.
- Decision rule: The calibrated score is thresholded at ϵ=0.5 to produce a binary check-worthiness prediction.The underlying language model first supplies a continuous score in [0, 1].
- Nearest-neighbor calibration: For each test claim, semantic K-nearest-neighbor retrieval selects labeled calibration examples and their prediction residuals.Neighbor sets are formed from the calibration set using semantic similarity, with k varied experimentally.
- Score correction: The calibrated score adds the average neighbor residual, Yj − ĉj, to the original model score ĉi.This correction targets local prediction bias estimated from semantically similar labeled instances.
- Uncertainty: Residual variance from the neighbors yields a per-instance confidence interval that quantifies uncertainty in each calibrated prediction.The approach therefore provides scores suitable for downstream decisions as well as uncertainty information.
4 Experimental Setup
The experiments evaluate few-shot SLMs, a production fine-tuned transformer, and large commercial models on ClaimBuster and English CLEF 2024 data. Calibration sets are class-balanced labeled subsets, and performance is measured with weighted and classwise F1.
- Datasets: Calibration sets are class-balanced subsets of training data selected with a fixed random seed for stable performance across neighbor sizes.Table 1 denotes these subsets as L and reports check-worthy and non-check-worthy test counts.
- Datasets: ClaimBuster uses 2012 debates for calibration and 2016 debates for testing, while CLEF 2024 provides English claims from a multilingual, multi-domain collection.The calibration sets contain 1,314 ClaimBuster examples and 2,406 CLEF 2024 examples.
- Models and prompting: The study generates few-shot confidence scores in [0, 1] using three model classes: SLMs, production XLM-RoBERTa-Large, and large commercial APIs.The SLM category includes models with at most 4B parameters.
- Evaluation: Each test claim retrieves k semantically nearest calibration examples, applies NN-PPI, thresholds at ϵ=0.5, and is evaluated with weighted and classwise F1.Sentence embeddings and cosine similarity support neighbor retrieval.
5 Results
NN-PPI improves weighted F1 across model scales, with the largest gains for smaller models and additional gains for a production fine-tuned model. Calibration performance generally favors smaller neighborhoods, while confidence intervals remain relative uncertainty indicators rather than frequentist guarantees.
- Model-scale results: NN-PPI improves weighted F1 across model scales, with the largest gains for smaller LLMs.Gemma 3 270M rises from 0.114 to 0.721 on ClaimBuster and from 0.179 to 0.827 on CLEF.
- Model-scale results: 33.80%: Gemma 3 4B improves on ClaimBuster from 0.568 to 0.760 weighted F1, while CLEF improves 20% from 0.688 to 0.827.
- Fine-tuned production model: 11%: NN-PPI improves the production fine-tuned XLM-RoBERTa-Large baseline on CLEF 2024, also exceeding KNN by 7.03%.The model ranks second only to frontier LLMs, supporting complementarity between residual calibration and supervised fine-tuning.
- Comparison with KNN: NN-PPI outperforms plain KNN in most conditions, but KNN slightly exceeds it for Gemma 3 1B on CLEF at k=10, 0.803 versus 0.761.The residual correction is most beneficial when model bias is systematic; KNN averaging may suffice for reasonably calibrated models.
- Error analysis: Class imbalance explains Gemma 3 4B’s lower baseline weighted F1 than Gemma 3 1B on ClaimBuster despite its larger size.Gemma 3 4B over-predicts the positive class at 65.7% versus a true rate of 26.5%, and NN-PPI corrects this bias.
- Neighbor-size analysis: Weighted and classwise F1 scores saturate as k increases from 3 to 5 and 10, likely because non-representative neighbors add residual noise and uncertainty.
- Failure modes: NN-PPI fixes over-triggered rhetorical claims and missed factual claims, but biased or topically unrelated neighborhoods can cause persistent errors and regressions.
- Confidence intervals: 64.8% overall CI coverage at k=3 versus 46.6% at k=10 on ClaimBuster shows that smaller neighborhoods yield better-calibrated intervals.Decision accuracy remains 70–90%, but the confidence intervals are better treated as relative uncertainty indicators than frequentist guarantees.
6 Conclusion
The paper proposes NN-PPI to calibrate claim check-worthiness predictions using a small human-annotated set and nearest-neighbor residual correction. It reports the largest gains for small language models and similar gains for smaller production-deployed transformer models, while identifying future adaptivity as an open direction.
- NN-PPI calibrates LLM predictions with a small human-annotated calibration set and nearest-neighbor residual correction.
- The largest performance gains occur for small language models, narrowing their gap with larger models.
- Smaller transformer models deployed in production also show similar gains from the proposed calibration approach.
- Future work will address adaptivity by updating the calibration set as claim-worthiness notions evolve without changing the underlying model.
7 Limitations
NN-PPI uses local, non-parametric residual correction to address input-dependent bias, but depends on a small representative calibration set whose selection remains challenging.
- NN-PPI targets input-dependent bias through labelled neighbours, unlike global calibrators that assume homogeneous miscalibration.Parametric methods learn a single score-to-probability mapping, whereas NN-PPI applies a local correction; conformal prediction does not relocate point estimates.
- Controlled comparison with Platt scaling, temperature scaling, isotonic regression, and conformal prediction is left for future work.The paper focuses on recovering LLM-level decisions from cheap models rather than conducting that comparison.
- The approach requires a small, distributionally representative calibration set, which may involve human annotations or repurposed training data.The authors describe this as minimal effort while retaining principled calibration.
- Selecting calibration samples that are distributionally similar to each test sample remains challenging.Semantic similarity works as a proxy in the experiments, but semantic relevance may not always imply distributional similarity.
8 Ethical Considerations and Risks
The study uses public, non-sensitive datasets but does not address fairness or demographic bias, which is outside its calibration-focused scope.
- The datasets come from public Creative Commons sources and contain no personally identifiable or sensitive information.The claims concern public data and domain-specific knowledge rather than private individuals.
- The approach does not currently mitigate fairness or bias across demographic attributes.The authors identify this as potentially relevant to politically or socially sensitive claims but beyond the paper’s focused contribution.
A Baseline Prediction Analysis: Gemma 3 1B vs. Gemma 3 4B
Gemma 3 4B systematically over-predicts positive claims, depressing non-check-worthy performance and weighted F1 despite its larger size; the analysis compares this with Gemma 3 1B and a supervised multilingual baseline.
- Baseline Prediction Analysis: 65.7% versus 26.5%: Gemma 3 4B’s predicted positive rate exceeds the true rate on ClaimBuster.On CLEF, the corresponding rates are 58.2% versus 34.0%.
- Baseline Prediction Analysis: Gemma 3 4B’s positive-class overprediction suppresses cls-0 recall and overall weighted F1.Its higher positive rate produces higher cls-1 recall but weaker performance on the negative class.
- Fine-Tuned Supervised Baseline: The supervised upper-bound baseline is a multilingual XLM-RoBERTa-Large model trained on 84,312 examples from public and in-house claim datasets.It uses a binary classification head, weighted positive-class loss, and training procedures intended to avoid test leakage.
- Model Generation Parameters: Gemma 3 models were served through Ollama, while API-based models used provider-default generation parameters.These settings are reported as the generation-parameter configurations for the evaluated models.
D NN-PPI calibration evaluation at model temperature = 0.1
NN-PPI’s gains remain consistent when evaluated at temperature 0.1. Lower temperature modestly benefits GPT-5.2’s baseline, while Gemma models’ structural biases remain largely unchanged.
- NN-PPI gains remain consistent at T=0.1 across GPT-5.2 and Gemma models.The table reports weighted F1 for Baseline, KNN, and NN-PPI at k = 3, 5, and 10.
- 0.843 to 0.853: GPT-5.2’s baseline weighted F1 improves at T=0.1.The gain is associated with Class 1 F1 increasing from 0.685 to 0.703 and Class 0 F1 from 0.901 to 0.907.
- 0.006: GPT-5.2’s NN-PPI Class 1 F1 difference between temperatures at k=3 is small.The values are 0.653 at T=0.1 versus 0.659 at the default temperature.
- Gemma 3 1B and Gemma 3 4B show negligible baseline movement across temperatures, with Gemma 3 4B’s weighted F1 fixed at 0.568.The authors interpret these stable biases as structural rather than sampling-induced, and report equally effective NN-PPI correction at either temperature.
E Ablation: Effect of labeled set size |L|
Calibration performance stabilizes at smaller labeled-set sizes for k=3 and k=5, while k=10 continues benefiting from larger calibration pools.
- Weighted F1 peaks and stabilizes around |L|=1,500–2,000 for k=3 and k=5.
- For k=10, weighted F1 continues growing beyond |L|=4,500 without reaching saturation.
- The authors use calibration sets of 1,314 examples for ClaimBuster and 2,406 for CLEF 2024, based on the observed stabilization range.