PaperlayerView exact revision on arXiv

Source-linked AI summary

CWF: A Collaborative Writing Framework for Personalized and Reliable Popular Science Writing

Ruibiao Fu, Di Tang, Yunlong Yang, Ran Wang, Sicheng Lu, Peixuan Wu, Xiaoyu Fan, Jiacheng Ma, HaoZhe Luo, Yang Xiao

arXiv:2609.06126v1cs.AI

TL;DR

Personalized popular science writing must adapt explanations to audiences without sacrificing factual accuracy, yet existing methods face costly aligned data requirements and weak verification in evidence-scarce settings. CWF introduces PSCB, DA-MoE, and graph-based multi-agent fact-checking to address these challenges. Experiments on PSCB show improved personalization and factual accuracy, including gains over the Base model.

  • Problem

    Personalized writing must adapt scientific explanations to audience cognitive levels while preserving factual accuracy, but existing methods require costly aligned data and struggle with reliable verification.

  • Method

    CWF combines PSCB with DA-MoE, which separates audience adaptation from domain knowledge, and multi-agent fact-checking that uses role-specific debate and graph-based confidence inference.

  • Results

    CWF improves average LLM-as-a-judge and human evaluation over the Base model by 15.2% and 8.1%, respectively, while CWF-R improves average factual accuracy by 28.7%.

  • Takeaways & Limitations

    CWF effectively improves both personalization and factual accuracy in popular science writing on PSCB.

  • Takeaways & Limitations

    The dataset remains expandable in domain balance and linguistic diversity, the framework focuses mainly on textual personalization, and multi-agent verification may increase token consumption.

Abstract

from arXiv · show

We introduce Personalized and Reliable Popular Science Writing, a novel task that requires adapting scientific explanations to audiences with different cognitive levels while preserving factual accuracy. However, improving personalization often introduces simplifications that increase the risk of hallucination and factual distortion. To address these challenges, we first construct a dataset of 39,134 entries and a reader-centric Personalized Science Communication Benchmark (PSCB) that jointly evaluates audience adaptation and factual accuracy. To reduce data and computational requirements while improving generalization across domains and audiences, we introduce DA-MoE, which explicitly decouples audience adaptation from domain knowledge through separate modeling. To enable robust verification and revision in evidence-scarce scenarios, a multi-agent fact-checking mechanism that augments limited evidence with role-specific agent debate and propagates confidence over a graph is proposed. Experiments on PSCB show that our approach achieves state-of-the-art performance. Our code is open-sourced at https://github.com/DPInnovationWorks/CWF.

1 Introduction

Popular science writing must adapt explanations to audiences while preserving factual accuracy, but existing approaches often separate these goals and require costly aligned data. CWF addresses both challenges with PSCB, DA-MoE, and multi-agent fact-checking.

  • Popular science writing must combine cognitive adaptation, writing style, and factual accuracy, yet existing methods typically treat these as separate objectives.
  • Audience adaptation can increase unsupported information and factual distortion, motivating PSCB’s joint evaluation of cognitive adaptation and factual accuracy.PSCB also introduces Cognitive Load and fine-grained dimensions beyond surface style.
  • CWF constructs a dataset of 39,134 entries without requiring joint alignment between scientific domains and target audiences.The design addresses the cost of collecting data that simultaneously contains accurate knowledge and audience-adaptive expressions.
  • DA-MoE separates audience-specific adaptation from domain-specific knowledge through pluggable experts, enabling lightweight adaptation to new domains.The architecture learns cognitive adaptation and knowledge from different sources.
  • CWF adds multi-agent fact-checking with graph-based reasoning to support perspective verification, generation-time intervention, and quantifiable confidence scores.The mechanism is designed for verification when direct evidence is sparse or conflicting.

2 Related Works

Prior work studies public-facing scientific writing, personalization, fact-checking, and evaluation, but these areas remain fragmented. Existing methods face challenges in long-form or heterogeneous-style generation, sparse evidence, proactive verification, and joint assessment of accessibility and factual accuracy.

  • Popular Science Writing: Lay summarization and academic-to-public paraphrasing use paired datasets and benchmarks such as PLOS, eLife, and VTechAGP.
  • Personalized and Controllable Writing: Few-shot prompting and supervised fine-tuning support personalized generation, but prompting struggles with long-form text and fine-tuning requires costly style-specific data.
  • Fact-checking in LLMs: LLM-based fact-checking with retrieval reduces reliance on human evaluation, yet sparse or conflicting evidence remains difficult to handle.
  • Fact-checking in LLMs: Claim decomposition and multi-agent debate provide multiple perspectives, but existing approaches remain post-hoc and static rather than proactively guiding generation.
  • Evaluation Benchmark: Existing benchmarks typically assess either readability and audience accessibility or factual accuracy, leaving joint evaluation across audiences insufficient.

3 Collaborative Writing Framework

CWF combines collaborative drafting, decoupled audience and domain modeling, and multi-agent fact-checking to produce personalized, coherent, and reliable science writing.

  • Framework overview: CWF coordinates drafting, personalization, and verification to balance audience adaptation with factual accuracy.Collaborative agents first plan and expand the article, while verification revises low-confidence or inconsistent claims.
  • DA-MoE: The framework separates audience-adaptation experts from knowledge-domain experts in two MoE modules.Experts specialize in audience-specific styles or scientific domains, while lightweight gating networks route inputs to appropriate experts.
  • DA-MoE: Training proceeds through expert specialization, gating-network learning, and AlphaNet-based balancing of style and knowledge contributions.The staged procedure fixes expert parameters before learning routing and then dynamically combines the two MoE outputs.
  • DA-MoE: Mean α decreases from children (0.686) to teenagers (0.262) and adults (0.183), indicating greater style reliance for younger readers.Here, larger α denotes greater reliance on the style network, whereas adults receive relatively more knowledge-oriented weighting.
  • Efficiency: Table 2 reports the highest average PSCB score for the method with substantially less training time and fewer training examples.The framework reuses existing style and knowledge experts for new domains, requiring only a small amount of single-style data to train the gating network.
  • Fact-checking: Fact-checking decomposes drafts into atomic claims and uses Leader, Trust, and Skeptic agents to debate evidence before graph-based confidence propagation.The debate can run for up to Rmax = 3 rounds, after which low-confidence or inconsistent claims are revised.

4 PSCB Benchmark

PSCB evaluates whether science writing adapts to readers while remaining reliable, combining personalization metrics, human questionnaires, and factuality-oriented consistency analysis.

  • Dataset: PSCB contains 39,134 entries spanning AI, Biology, and Medicine for child, teen, and adult reader groups.The dataset supports training across scientific domains and target audiences.
  • Personalization metrics: The benchmark scores Cognitive Load, Personalization Alignment, and Reader Attitude on 0–5 scales.These metrics assess cognitive fit, audience-tailored content and style, and reader engagement and credibility.
  • Evaluation protocol: PSCB combines dynamically weighted LLM-as-a-judge scoring with human questionnaires and consistency analysis between the two sources.Metric weights depend on the scientific domain and target persona, while the overall personalization score weights the three metrics equally.
  • Human evaluation: Human evaluation retained 30 participants per reader group, producing 540 questionnaires across six anonymized systems.Metric-level ordinal Krippendorff’s α was 0.781–0.803, and ICC(2, k) was 0.973–0.976.
  • Judge consistency: Dynamic dimension weighting achieved ρ=0.829 and 86.7% pairwise accuracy for Qwen-Plus, while seven judges reached ρ values of 0.829–0.943.MARD measures average rank difference, while MAE measures absolute score error; these metrics are reported alongside rank correlations and pairwise accuracy.
  • Factual accuracy: Factual accuracy is an article-level 0–5 score obtained by aggregating claim confidences and penalizing unsupported or incorrect scientific statements.PSCB represents claims in a reasoning graph with supporting and opposing evidence assigned confidence scores.

5 Experiment Results

CWF achieves the strongest personalization evaluations across children, teens, and adults, while its revision variant improves factual accuracy at a modest personalization cost. Ablations indicate that dynamic scoring, debate, graph propagation, and claim decomposition contribute to more reliable evaluation and verification.

  • Overall Performance: CWF achieves the strongest LLM-as-a-judge and human evaluation performance across children, teens, and adults.It also significantly outperforms Qwen2.5-14B for all three reader groups.
  • Overall Performance: CWF-R achieves the best factual accuracy, while its slightly lower personalization scores indicate a mild, audience-dependent trade-off.Stronger revision can increase information density and cognitive load, especially for younger audiences.
  • Overall Performance: CWF improves average LLM-as-a-judge and human evaluation over Base by 15.2% and 8.1%, respectively.CWF-R improves average factual accuracy by 28.7%.
  • Evaluation Analysis: Dynamic dimension weighting yields the best consistency between LLM-as-a-judge and human evaluation, outperforming equal weighting.Average weighting also lowers aggregate MAE compared with one-step scoring.
  • Fact-checking Ablation: Removing debate causes the largest fact-checking performance drop, while disabling graph propagation increases unsupported-claim confidence and variance.Claim decomposition produces modest but consistent gains in confidence margins and variance.
  • Fact-checking Ablation: Claim decomposition and multi-agent debate improve factuality by finding implicit reasoning evidence and checking retrieved evidence from supportive and skeptical perspectives.The reasoning graph can connect indirect clues for complex claims while filtering related but insufficient evidence.

6 Conclusion

CWF addresses personalized and reliable popular science writing through decoupled audience and domain modeling plus collaborative fact-checking. Experiments on PSCB show improvements in both personalization and factual accuracy.

  • Conclusion: CWF combines DA-MoE, which decouples audience adaptation from domain knowledge, with multi-agent debate and graph-based confidence inference for verification and revision.The framework targets personalized and reliable popular science writing.
  • Conclusion: Experiments on PSCB show that CWF improves both personalization and factual accuracy in popular science writing.This is the paper’s reported overall outcome.

Limitations

The evaluation examines judge robustness, debate-round effects, base-model generalization, and the remaining limits of CWF’s verification framework. Results support stable judgments and confidence separation, but additional rounds add little after three and CWF-R depends on informative signals.

  • Overall evaluation: CWF achieves the strongest LLM-as-a-judge and human evaluation performance across children, teens, and adults.The supplied passage reports this overall result across the three reader groups.
  • Judge robustness: CWF ranks first under six of seven judges and has the highest cross-judge average of 3.772.Qwen2.5-14B exceeds CWF only under DeepSeek-V4-Pro.
  • Judge–human alignment: Six judges rank CWF first, while DeepSeek-V4-Pro ranks it second, 0.031 below Qwen2.5-14B.Cross-judge ranking concordance is high, with macro-average Kendall’s W = 0.832.
  • Scoring protocols: Dynamic weighting lowers MAE from 0.837 to 0.417 while preserving 86.7% of human pairwise preferences.Both dynamic and average weighting induce the same system ranking, whereas one-step scoring preserves 73.3%.
  • Debate rounds: Three debate rounds increase the support-partial margin from 1.125 to 1.489, a 32.4% relative improvement.Further rounds add only 0.003 and 0.005 to the two margins while average variance rises from 0.482 to 0.527.
  • Base-model robustness: All tested base models preserve Support > Partial > Unsupport confidence ordering, while GPT-5.5 has the largest margins and lowest average variance of 0.458.GPT-5.5 reaches margins of 1.784 and 1.625; the framework therefore does not rely on one backbone.

A.7 Token Cost Analysis.

The token analysis compares SAFE, Multi-agent Debate, and CWF with three debate rounds. CWF uses fewer tokens than Multi-agent Debate, though SAFE remains the least expensive method.

  • Token usage: SAFE consumes the fewest tokens, with total usage of 15.19K tokens.The comparison covers prompt, response, and total token consumption.
  • Performance–cost trade-off: SAFE has lower factuality evaluation performance than the proposed CWF-based method despite its lower token cost.Multi-agent Debate has higher overhead because of extensive prompt construction and multi-agent interaction.
  • Token usage: CWF with 3 rounds consumes 53.32K tokens versus 89.54K for Multi-agent Debate, a 40.4% reduction.The reduction is computed from additive prompt and response token counts.
  • Token usage: CWF reduces prompt tokens from 85.00K to 50.36K, a 40.8% decrease, and response tokens from 4.54K to 2.97K.These figures compare CWF with three rounds against Multi-agent Debate.

B.1 Agent Configuration

CWF uses five specialized agents for summarization, trust assessment, skepticism, leadership, and rewriting. Debate agents retrieve evidence, exchange structured judgments, and feed revision suggestions to a rewriting model.

  • Agent roles: CWF uses five agents: Summarize, Trust, Skeptic, Leader, and Rewrite.The agents divide factual feedback, evidence assessment, debate integration, and paragraph revision.
  • Revision process: The Summarize Agent aggregates debate results into revision suggestions, and the Rewrite Agent revises the paragraph while preserving audience adaptation and style.The rewrite prompt also requires preserving correct content and avoiding new facts or viewpoints.
  • Agent roles: The Trust, Skeptic, and Leader agents use retrieval, respectively seeking support, challenging weak claims, and integrating debate results.Each retrieval-enabled agent has at most four retrieval actions per call.
  • Debate process: The debate runs for three rounds by default and stops early when all agents reach the same conclusion after a complete round.Otherwise, it continues until the maximum number of rounds is reached.
  • Debate process: Agents output opinions, factual stances, and cited evidence using five stance values from strong support to strong opposition.The stance set is {1, 0.2, 0, −0.2, −1}.

C.3 Human-Evaluation Agreement

Human-evaluation agreement is assessed with multiple reliability measures, participant screening, and bootstrap checks. Agreement remains substantial across dimensions and reader groups, supporting the stability of participant-group means.

  • Agreement results: Across dimensions, αK ranges from 0.741 to 0.824 and ICC(2, k) from 0.966 to 0.979.Extraneous Burden Control and Style Consistency are more subjective but remain reliable.
  • Robustness checks: In 5,000 bootstrap resamples, LLM–human Spearman correlation remains positive in 99.2% and CWF remains among the top two systems in 96.8%.These checks support the stability of participant-group means used as human evaluation scores.
  • Questionnaire design: The human evaluation uses a 0–5 integer response scale, with 0 meaning complete disagreement or non-applicability and 5 meaning complete agreement or applicability.Participants rate assigned science-popularization articles based on their reading experience.
  • Scoring procedure: Negative items are reverse-coded before dimension scores are computed, including Q1, Q5, Q15, Q18, Q23, Q28, Q32, Q36, and Q40.This enforces directional consistency in score aggregation.
  • Questionnaire design: The questionnaire maps Q1–Q12 to Cognitive Load, Q13–Q28 to Personalization Alignment, and Q29–Q40 to Reader Attitude.Dimension scores average the corresponding item ratings before dynamic weighting and metric aggregation.

D Metrics and Dimensions

The evaluation aggregates scientific domains and compares CWF, Qwen2.5-14B, and CWF-R across child, teen, and adult readers. Significant differences are observed across all reader groups, with CWF consistently outperforming the alternatives.

  • The evaluation aggregates the three scientific domains and retains only topics evaluated for all three systems.
  • Omnibus differences across CWF, Qwen2.5-14B, and CWF-R are significant for child, teen, and adult readers.
  • CWF outperforms Qwen2.5-14B across all reader groups, while CWF-R is consistently lower than CWF.
  • CWF-R exceeds Qwen2.5-14B for teen and adult readers but trails it for child readers, indicating an audience-dependent personalization–reliability trade-off.
  • Table 28 reports adjusted significance results without repeating method means presented in Table 4.

E.2 Fact-checking

The fact-checking analysis compares article-level factual accuracy between CWF-R and CWF on matched topics using paired tests. Revision significantly improves factual accuracy in seven of nine domain–reader conditions, although two child conditions do not reach conventional significance.

  • Paired differences compare CWF-R and CWF factual accuracy on the same evaluation topics within each domain–reader condition.
  • A positive t-statistic indicates higher factual accuracy after revision.
  • CWF-R achieves significantly higher factual accuracy in seven of nine domain–reader conditions at p < 0.05.
  • AI–Child and Medicine–Child show positive estimated differences, but neither reaches conventional significance, with p = 0.0782 and p = 0.0941.
  • Table 29 reports two-sided paired t-tests with unadjusted p-values and significance codes for comparing CWF-R with CWF.
Loading 2609.06126v1…