PaperlayerView exact revision on arXiv

Source-linked AI summary

HARPO: Hallucination-Aware Reinforcement Learning for Faithful and Creative Language Generation

Tiezheng Yu, Yuxin Jiang, Jinpeng Li, Shuning Sun, Fei Mi, Haoli Bai, Lifeng Shang

arXiv:2610.03063v1cs.CL

TL;DR

LLMs can produce hallucinated content that undermines reliability, while optimizing faithfulness and creativity together remains challenging. HARPO combines hallucination-aware rewards, conditional writing-reward activation, and a creative-to-faithful data curriculum. It reports lower hallucination rates and higher writing scores than base models, alongside trade-offs across training strategies.

  • Problem

    Hallucinated outputs can be unfaithful to their sources and undermine LLM trustworthiness, while single-reward strategies and naive data mixing struggle to balance faithfulness and expressive quality.

    Source: §1 · p. 1

  • Method

    HARPO jointly optimizes faithfulness and creativity with hallucination-aware reward modeling, writing rewards gated by HA-GRM predictions, and a curriculum shifting from creative to hallucination-focused data.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    Over 85% lower HA-GRM-judged summarization hallucination rate versus base Qwen3-4B accompanies higher creative-writing scores.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported comparisons characterize trade-offs across training strategies, including lower hallucination rates alongside lower creative-writing scores for the curriculum relative to the reverse schedule.

    Source: §6 · p. 9

    SAM depends on HA-GRM predictions and does not guarantee factual correctness; reported generation hallucination-rate improvements also rely on this shared evaluator.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) are prone to generating hallucinated content, which compromises their reliability in knowledge-intensive tasks. To address this challenge without sacrificing creativity, we propose HARPO, a reinforcement learning framework designed to jointly optimize faithfulness and creativity. HARPO incorporates a Hallucination-Aware Generative Reward Model (HA-GRM), trained via verifiable feedback, to assess both faithfulness and writing quality. A Selective Activation Mechanism (SAM) activates writing rewards only for outputs judged hallucination-free by HA-GRM, while a data curriculum progressively shifts training from creative writing to hallucination-centric tasks. On RAGTruth, our Qwen3-4B-based HA-GRM achieves a response-level F1 score of 78.08%, compared with 66.37% for the supervised fine-tuning baseline. Experiments on Qwen2.5 and Qwen3 models from 1.7B to 8B parameters show improvements in both faithful generation and writing quality. On Qwen3-4B, HARPO reduces the HA-GRM-judged hallucination rate on MultiHopRAG from 3.29% to 1.02%, while increasing the Arena-Hard-v2.0 creative-writing score from 16.95% to 27.54%.

1 Introduction

The introduction frames hallucination mitigation and creative writing as competing objectives that existing single-reward strategies and naive data mixing struggle to balance. It presents HARPO as a framework for jointly optimizing faithfulness and creativity through hallucination-aware rewards, conditional writing-reward activation, and a shifting data curriculum.

1 Introduction

  • Hallucinated, source-unfaithful outputs undermine LLM trustworthiness and utility, while optimizing one capability can degrade performance in other dimensions.The introduction identifies single-reward strategies and naive data mixing as approaches that struggle to balance faithfulness and expressive quality.Source: §1 · p. 1
  • HA-GRM evaluates hallucinations and writing preferences, using response-level labels, hallucinated-span supervision, and pairwise creative-writing preferences.These signals support faithfulness evaluation and stylistic-quality evaluation.Source: §1 · pp. 1–2
  • SAM activates the writing reward only when HA-GRM predicts that a response is hallucination-free, making writing-reward assignment conditional on its judgment.This is presented as an alternative to fixed-weight scalarization, which is prone to reward conflicts.Source: §1 · p. 1
  • The data curriculum gradually shifts training from creative-writing instructions toward hallucination-focused scenarios to preserve broad linguistic proficiency while sharpening evidential rigor.The paper describes this shift as a way to mitigate catastrophic forgetting associated with single-objective alignment.Source: §1 · p. 2
  • Over 85% lower HA-GRM-judged summarization hallucination rate versus base Qwen3-4B accompanies higher creative-writing scores; evaluations span Qwen2.5 and Qwen3 models from 1.7B to 8B.The introduction also reports trade-offs against linear reward mixing and between curriculum schedules.Source: §1 · p. 2
  • HARPO combines hallucination-aware reward modeling, conditional reward aggregation, and a creative-to-faithful data curriculum to jointly optimize faithful generation and writing quality.The framework is introduced to address competing demands for precision and caution versus richness and novelty.Source: §1 · pp. 1–2

2 Methodology

HARPO combines HA-GRM’s hallucination assessment with a gated writing reward and a curriculum that shifts training from creative to faithful tasks. It trains the reward model on hallucination and writing-preference signals, then optimizes the policy with GRPO.

2 Methodology

  • HARPO integrates HA-GRM, a writing-reward gate, and a curriculum that progressively shifts training from creative to hallucination-sensitive tasks.The framework uses reinforcement learning to balance hallucination reduction and creative generation.Source: §2 · p. 2, §2.2 · p. 3

2.1 HA-GRM: Reward Model Training

  • HA-GRM detects hallucinations at response and span levels and evaluates writing preferences, using separately applied reward schemes for the two data types.Its training uses online RL with format validity and prediction accuracy as reward components.Source: §2.1 · p. 2

2.2 HARPO: Hallucination-Aware Reinforcement for Policy Optimization

  • GRPO updates the policy from groups of sampled rollouts using reward-based advantages, asymmetric clipping, and a KL penalty that keeps the policy near its reference model.The asymmetric clipping strategy is described as encouraging exploration while maintaining training stability.Source: §2.2 · p. 3
  • SAM activates the writing reward only when HA-GRM predicts no hallucinated spans; the gate reflects a prediction, not a guarantee of factual correctness.When spans are predicted, the hallucination reward penalizes their density; the writing reward evaluates helpfulness, relevance, conciseness, and novelty.Source: §2.2 · pp. 3–4
  • The cosine-annealed curriculum reduces the creative-writing data share from an initially high level to a 5% minimum at training’s end.The minimum creative-data proportion is retained to prevent forgetting; Figure 1 depicts the schedule’s shift from creative to faithful tasks.Source: Figure 1 · p. 3, §2.2 · p. 4

3 Experiments on HA-GRM

This section tests whether fine-grained reinforcement-learning feedback improves generative reward models, examining response- and span-level hallucination detection alongside writing-preference evaluation.

3.1 Experimental Setup

  • The experiments use RAGTruth summarization and QA subsets for hallucination detection, Auto-J preference data, and character-level span F1 alongside response-level metrics.The evaluated datasets are human-annotated.Source: §3.1 · pp. 4–5

3.2 Results and Analysis

  • 78.08% Overall response-level F1 for HA-GRM (Full) exceeds the base model’s 74.58% and SFT’s 66.37%.Table 1 reports the response-level detection results on RAGTruth.Source: Table 1 · p. 5, §3.2 · p. 5
  • Full has higher summarization F1 than SFT (69.47% vs. 66.10%), but SFT has higher QA F1 (70.48% vs. 67.75%).On D2T, excluded from training, Full reaches 83.94%, above SFT’s 65.09% but below the base model’s 85.41%.Source: §3.2 · p. 5
  • 50.38% Overall span-F1 for Full exceeds Response Only’s 44.43%, while SFT retains the highest QA span-F1 at 61.26%.Table 1 also reports response-level Overall F1 of 78.08% for Full versus 77.40% for Response Only.Source: Table 2 · p. 5, §3.2 · p. 5
  • 69.00% cross-task average for joint training exceeds hallucination-only training’s 67.86%, as Auto-J accuracy rises and RAGTruth response-level F1 falls.Auto-J accuracy changes from 57.31% to 59.91%, while RAGTruth F1 changes from 78.41% to 78.08%.Source: Table 3 · p. 5, §3.2 · pp. 5–6

4 Experiments on HARPO

This section tests whether HA-GRM feedback can mitigate hallucinations while incentivizing creative writing. It compares HARPO with base models and alternative training strategies, including curriculum schedules and response-length behavior.

4.1 Experimental Setup

  • Training combines RAGTRUTH summarization, HALUEVAL question answering, and creative-writing prompts; evaluation uses HHEM, MultiHopRAG, and Arena-Hard-v2.0 subsets.The HHEM and MultiHopRAG hallucination rates are judged by HA-GRM.Source: §4.1 · p. 6

4.2 Results and Analysis

  • Hallucination-Only reaches 0.72% HHEM hallucination rate but lowers Creative Writing from 16.95% to 9.74%, while Writing-Only raises it to 43.35% as HHEM rate rises to 24.31%.The reported single-objective results show gains on a target metric alongside degradation on another.Source: §4.2 · p. 6
  • 1.08% vs. 3.73% HHEM hallucination rate and 12.35% vs. 5.63% General score favor HARPO over Linear Mixture on Qwen3-4B.HARPO has a slightly higher MultiHopRAG hallucination rate (1.02% vs. 0.90%) and lower Creative Writing score (27.54% vs. 28.83%).Source: §4.2 · pp. 6–7
  • 5.05% to 1.56% HHEM hallucination rate and 4.68% to 6.78% Creative Writing score on Qwen2.5-7B show HARPO’s gains over the base model.Across every tested size, HARPO has lower HA-GRM-judged hallucination rates and higher Creative Writing and General scores than corresponding base models.Source: §4.2 · p. 6, Table 5 · p. 7
  • The proposed Creative → Faithful curriculum lowers HHEM hallucination rate from 4.09% to 1.08%, lowers MultiHopRAG rate from 1.17% to 1.02%, and raises Creative Writing from 23.65% to 27.54% versus Random Mixture.The reverse schedule reaches 34.30% Creative Writing, but has higher HHEM and MultiHopRAG rates than Creative → Faithful.Source: §4.2 · p. 7, Table 6 · p. 8
  • Response lengths rise initially, then peak and stabilize beyond step 60 as the curriculum shifts toward faithful tasks and SAM takes effect.The analysis contrasts HARPO’s trajectory with Writing-Only’s unbounded length increase and Hallucination-Only’s rapid convergence to shorter responses.Source: §4.2 · pp. 7–8, Figure 2 · p. 7

5 Related Work

Related work spans closed-domain hallucination detection, verifiable-reward training, and joint writing-quality and constraint optimization. HARPO combines hallucination-aware evaluation with gated writing rewards and a creative-to-faithful curriculum.

5 Related Work

6 Conclusion

HARPO jointly targets hallucination mitigation and writing quality through gated writing rewards and a curriculum that shifts from creative to faithful tasks. Reported results show gains relative to base models alongside trade-offs across training strategies.

6 Conclusion

  • HARPO reports lower HA-GRM-judged hallucination rates and higher writing scores than base models.The comparison with Linear Mixture lowers summarization hallucination and raises General scores, but has a slightly higher MultiHopRAG hallucination rate and lower Creative Writing score.Source: §6 · p. 9
  • SAM activates writing rewards only for responses HA-GRM judges hallucination-free, while the curriculum shifts training from creative to faithful tasks.These are the framework’s two stated optimization mechanisms.Source: §6 · p. 9
  • The reverse curriculum schedule has higher hallucination rates than the reported schedule, but a higher creative-writing score.The conclusion characterizes the outcomes as trade-offs across training strategies.Source: §6 · p. 9

Limitations

HARPO’s reported scope is constrained by annotation and computation costs, reliance on HA-GRM predictions, and unverified scaling and task transfer.

Limitations

  • High-quality span-level hallucination annotations are more labor-intensive and costly to acquire than standard binary labels.HA-GRM training relies on those annotations.Source: Limitations · p. 9
  • The generative reward process increases training computation compared with traditional discriminator-based reward models.Source: Limitations · p. 9
  • Effectiveness is demonstrated only up to 8B parameters in general instruction and RAG domains; scalability to 70B+ models and transfer to math or code tasks remain unverified.Source: Limitations · p. 9
  • SAM provides no guarantee of factual correctness because it depends on HA-GRM predictions, and those predictions also judge reported generation hallucination rates.Missed hallucinations can still receive writing rewards, while incorrectly flagged faithful responses have those rewards suppressed.Source: Limitations · p. 9

Ethics Statement

The ethics statement identifies dataset licensing and privacy considerations, alongside limits on what the experiments establish about safety and reliability.

Ethics Statement

  • The study uses public datasets with differing licenses and usage terms, and underlying source corpora and pretrained models have separate applicable terms.RAGTruth and HaluEval provide MIT licenses; Arena-Human-Preference-140k prompts use CC BY 4.0, while model outputs remain subject to provider terms, and Arena-Hard-Auto lists Apache 2.0.Source: Ethics Statement · p. 9
  • Public user conversations and preference annotations may contain sensitive information and reflect contributor biases; reuse should protect personal information and retain attribution and applicable usage restrictions.Source: Ethics Statement · p. 9
  • The experiments do not establish safety or reliability for high-stakes applications; deployment requires independent evaluation and appropriate human oversight.The passage also notes possible evaluator errors and preference optimization favoring styles represented in training data.Source: Ethics Statement · p. 9
Loading 2610.03063v1…