HerO at AVeriTeC: The Herd of Open Large Language Models for Verifying Real-World Claims

1 Introduction

• Earlier datasets faced context dependence, evidence insufficiency, and temporal leaks, while AVeriTeC addressed these issues with fine-grained crowdsourced annotations.Earlier work generated synthetic claims from Wikipedia or collected human-verified claims (Thorne et al., 2018; Aly et al., 2021; Wang, 2017; Augenstein et al., 2019).Source: §1 · p. 1
• HerO uses publicly available LLMs for evidence retrieval, question generation, and veracity prediction without proprietary models.The system was developed for the AVeriTeC shared task hosted by FEVER-24 (Schlichtkrull et al., 2024).Source: §1 · p. 1
• 2nd place and an AVeriTeC score of 0.57 put HerO behind the winning system, which used gpt-4o (Schlichtkrull et al., 2024).The authors describe this competitive result as suggesting the potential of open LLMs for verifying real-world claims.Source: §1 · p. 1

2 Related Work

• Major technology companies drove early LLM progress, but offered only limited API access; research groups pursued open LLMs to facilitate open research.The passage cites Brown et al. (2020), Thoppilan et al. (2022), and Achiam et al. (2023) for early success, and Zhang et al. (2022) and Le Scao et al. (2023) for early open models.Source: §2 · p. 1
• Initial open LLMs had unsatisfactory performance (Zhang et al., 2022; Le Scao et al., 2023).Source: §2 · p. 1
• Recent open LLMs are described as on par with closed models and as outperforming them in certain categories (Jiang et al., 2023; Dubey et al., 2024).Source: §2 · p. 1

3 Task Definition

• The task retrieves relevant information from web documents to verify a real-world claim.The system may generate verifying questions for collected evidence or choose not to.Source: §3 · pp. 1–2
• Veracity prediction assigns one of four verdicts: supported, refuted, not enough evidence, or conflicting evidence/cherry-picking.Source: §3 · p. 2
• Evaluation uses Hungarian METEOR for question quality (Q score) and question-answer-pair quality (Q+A score), plus overall accuracy (AVeriTeC score).Source: §3 · p. 2

4 Our System

• HerO uses open LLMs without proprietary models and uses the dataset-provided web documents as its knowledge store.The section introduces the system as the Herd of Open LLMs for verifying real-world claims (HerO).Source: §4 · p. 2

4.1 Evidence Retrieval

• HyDE-FC augments a retrieval query with hypothetical fact-checking documents generated by an instruction-following language model.The claim is used as an in-context sample; the resulting documents and claim inform a hybrid retrieval process.Source: §4.1 · p. 2
• BM25 retrieves the top 10,000 sentences using the claim and generated documents; dense retrieval reranks them to select the top 10 evidence candidates.The query vector averages embeddings of the claim and generated documents, with g denoting an embedding method; the described best model uses llama-3.1-70b and SFR-embedding-2, with N=8.Source: §4.1 · p. 3

4.2 Question Generation

• Question generation uses an instruction-following model for each evidence item, with the prompt adding the corresponding claim to BM25-retrieved labeled question-answer examples.Source: §4.2 · p. 3

4.3 Veracity Prediction

• Veracity prediction uses a fine-tuned llama-3.1-70b model that generates an explanation before predicting the label, with the claim and top 10 earlier question-answer pairs as in-context samples.Source: §4.3 · p. 3

5.1 Experimental Setups

• Experiments used training data for model training and development data for evaluation; A, Q, and Q+A scores assessed answers, questions, and question-answer pairs.HerO took approximately 6.6 hours to make 500 development-set predictions with two H100 GPUs.Source: §5.1 · p. 4

5.2 Experimental Results

• 0.2801 was the A score for HyDE-FC with llama-3.1-70b, 0.02 above the claim-only approach.For verbatim claim queries, SFR-embedding-2 reranking boosted A score by 0.073; HerO combines that reranker with HyDE-FC query augmentation.Source: Table 2 · p. 4, §5.2 · p. 5
• 0.4938 was llama-3-8b’s Q score with the claim as an additional in-context sample, 0.0728 above its counterpart.HerO uses llama-3-8b for question generation; among models using only retrieved sentences, llama-3.1-70b scored best.Source: Table 3 · p. 4, §5.2 · p. 5
• 0.578 was the highest AVeriTeC score, achieved by llama-3.1-70b with full fine-tuning for veracity prediction.Instruction fine-tuning boosted performance relative to in-context learning without parameter updates.Source: Table 4 · p. 5, §5.2 · p. 5

5.3 Test Set Results

• TUDA_MAI_0 scored 0.63, HerO 0.57, and CTU AIC 0.5 on AVeriTeC; HerO had the top Q and Q+A scores among the top three.HerO’s performance gap with the winning system was smaller for Q+A than for Q.Source: Table 5 · p. 5, §5.3 · p. 5

• HerO uses publicly available LLMs for evidence retrieval, question generation, and veracity prediction.Source: §6 · p. 5
• HerO achieved second place in the AVeriTeC shared task.Source: §6 · p. 5
• The authors released HerO’s code publicly for future research.Source: §6 · p. 5