Source-linked AI summary
MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstracts
Weiyue Li, Ruizhi Qian, Yi Li, Yongce Li, Yunfan Long, Jiahui Cai, Yan Luo, Mengyu Wang
TL;DR
Biomedical conclusion generation lacks broad resources for testing whether models can infer author-written conclusions from structured evidence. MedConclusion supplies 5.7M PubMed abstracts with conclusions and journal metadata, then evaluates diverse LLMs across prompting and judging settings. The study finds distinct conclusion-versus-summary behavior, compressed scores among strong models, and sensitivity to judge identity.
Problem
Existing resources for testing inference of scientific conclusions from structured biomedical evidence remain limited in scope, metadata, and task-specific evaluation.
Method
MedConclusion pairs non-conclusion sections from 5.7M PubMed structured abstracts with author-written conclusions, adds journal metadata, and evaluates diverse LLMs with multiple prompts and hybrid metrics.
Results
Conclusion generation is behaviorally distinct from summary writing, strong models remain closely clustered under current automatic metrics, and absolute LLM-judge scores are sensitive to judge identity.
Takeaways & Limitations
MedConclusion provides a reusable data resource for studying scientific evidence-to-conclusion reasoning across biomedical domains and venue strata.
Abstract
from arXiv · showhide
Large language models (LLMs) are widely explored for reasoning-intensive research tasks, yet resources for testing whether they can infer scientific conclusions from structured biomedical evidence remain limited. We introduce $\textbf{MedConclusion}$, a large-scale dataset of $\textbf{5.7M}$ PubMed structured abstracts for biomedical conclusion generation. Each instance pairs the non-conclusion sections of an abstract with the original author-written conclusion, providing naturally occurring supervision for evidence-to-conclusion reasoning. MedConclusion also includes journal-level metadata such as biomedical category and SJR, enabling subgroup analysis across biomedical domains. As an initial study, we evaluate diverse LLMs under conclusion and summary prompting settings and score outputs with both reference-based metrics and LLM-as-a-judge. We find that conclusion writing is behaviorally distinct from summary writing, strong models remain closely clustered under current automatic metrics, and judge identity can substantially shift absolute scores. MedConclusion provides a reusable data resource for studying scientific evidence-to-conclusion reasoning. Our code and data are available at: https://github.com/Harvard-AI-and-Robotics-Lab/MedConclusion.
1 Introduction
The paper frames biomedical conclusion generation as inferring an author-written conclusion from structured evidence, while identifying gaps in existing resources and evaluation designs. MedConclusion addresses these gaps with a large, metadata-rich dataset and an initial empirical study.
- Structured abstracts support an evidence-to-conclusion task in which models infer a conclusion from Background, Methods, and Results without adding unprovided context.
- Existing datasets often target narrow study types or use conclusion reconstruction mainly for alignment or training, limiting broad biomedical reuse.
- Prior resources typically omit journal-level metadata such as biomedical categories and SJR, limiting analysis across biomedical subfields and venue strata.
- Adjacent biomedical benchmarks focus on question answering, medical reasoning, treatment-effect inference, or claim verification rather than deriving author-written conclusions.
- MedConclusion contains 5.7M PubMed structured abstracts paired with author-written conclusions and journal metadata, and supports analyses across biomedical domains.
- The initial study contrasts conclusion and summary prompting and evaluates outputs with reference metrics and LLM judges, including robustness across judge backbones.
2 Related work
Prior biomedical resources cover several reasoning and summarization tasks, but MedConclusion centers the evidence-to-conclusion reasoning step itself. It is positioned as a conclusion-generation benchmark rather than a question-answering, treatment-effect, or claim-verification resource.
- Biomedical resources include question answering, medical exam reasoning, treatment-effect inference, claim verification, and scientific claim generation.
- Structured abstracts and discourse-aware scientific summarization have been widely studied, but those resources do not specifically center conclusion inference.
- MedConclusion differs from adjacent resources by benchmarking the evidence-to-conclusion reasoning step itself.
- Figure 1 depicts the benchmark pairing non-conclusion abstract sections with author-written conclusions and evaluating generated outputs against those references.
3 Methodology
The methodology constructs MedConclusion from cleaned PubMed structured abstracts, adds journal metadata, and defines conclusion generation as predicting the removed conclusion from remaining sections. Diverse LLMs are tested under multiple prompts with hybrid automatic evaluation and judge-robustness analysis.
- Data collection: MedConclusion targets PubMed structured abstracts published between 2000 and 2025 and identifies candidate papers using PubMed’s hasstructuredabstract constraint.
- Data collection: The pipeline downloads PubMed XML records with Entrez Direct, parses metadata and structured abstract segments, and stores them in JSONL format.
- Data collection: 5,692,839 records remain after deduplication and rule-based cleaning requiring English text, core bibliographic fields, at least three abstract segments, and a conclusion section.
- Dataset metadata: Journal metadata covers 3,772 journals and 141 subject categories, with annual SJR values collected through 2024.
- Task formulation: With the conclusion removed, the model maps the concatenated remaining sections x to the original conclusion y⋆ and generates ˆy = fθ(x).
- Prompting: Four prompting modes vary the target between conclusion and summary and optionally impose sentence-count, word-count, and style constraints.
- Evaluation: Outputs are scored with five-dimensional LLM judging plus length, embedding, ROUGE, BLEU, and perplexity diagnostics.
- Experimental setup: Diverse LLMs are evaluated on a randomly sampled 30K subset, while GPT-5.4-mini and Gemini 3 Flash provide primary and secondary judge backbones.
4 Results
Results show that current evaluation only weakly separates strong models, while conclusion and summary prompting produce distinct metric patterns and judge identity shifts absolute scores.
- Overall performance: GPT-5.4 leads all five judge dimensions, but several models remain within a few points on most dimensions.This score compression suggests current reference-comparison evaluation separates strong models only weakly.
- Overall performance: DeepSeek-V3.2 achieves the best ROUGE-1/2/L scores and ties for best BLEU, whereas GPT-5.4 remains strongest on judge-based semantic and non-contradiction scores.Gemma-2-9B has the highest embedding similarity despite clearly lower judge scores than the best closed models.
- Overall performance: Lexical overlap, embedding similarity, fluency, and judge agreement capture different aspects of biomedical conclusion generation.These mismatches motivate the paper’s hybrid evaluation approach rather than reliance on traditional reference-based metrics alone.
- Prompting effects: Explicit style and length control improves numeric consistency to about 91 for GPT-5.4 and Gemini 3 Flash, while slightly reducing semantic and style similarity.The comparison concerns conclusion prompting under settings A and C.
- Judge robustness: Changing the judge from GPT-5.4-mini to Gemini 3 Flash raises semantic similarity, non-contradiction, and numeric consistency, while writing-style similarity changes modestly.Absolute score scales are highly judge-dependent, although GPT-5.4 remains the top generator across all five dimensions under both judges.
5 Analysis
Subgroup analyses show that conclusion-generation difficulty varies across journal prestige and biomedical categories, while lexical overlap alone does not capture overall quality.
- 5.1 SJR score: Most reference-based and judge-based metrics have small, statistically significant positive associations with SJR, while perplexity and non-contradiction show no significant trend.Numeric consistency has a small but significant negative correlation with SJR.
- 5.1 SJR score: The modest SJR effects indicate that venue prestige is a weak rather than dominant predictor of conclusion-generation difficulty.
- 5.2 Category: Semantic-similarity top categories perform strongly across writing style, numeric consistency, non-contradiction, and lexical-overlap metrics, unlike ROUGE-L top categories.ROUGE-L ranking produces lopsided profiles because high lexical overlap does not ensure strong judge-based dimensions.
- 5.2 Category: Gerontology ranks among the ROUGE-L top five but falls well below the semantic-similarity top five on writing style and numeric consistency.
- 5.2 Category: Lexical overlap is not a reliable proxy for overall conclusion quality, whereas semantic similarity aligns with stronger performance across reference-based and judge-based dimensions.The findings support using hybrid evaluation rather than relying on ROUGE or BLEU alone.
- 5.2 Category: Software, Computer Science Applications, and Applied Microbiology and Biotechnology appear among the bottom five under both ranking criteria, indicating consistently difficult categories.Software combines the lowest semantic similarity across 112 categories, 61.0, with among the highest numeric consistency, 96.4.
6 Conclusion
The paper introduces MedConclusion as a large-scale benchmark for biomedical conclusion generation and reports that task behavior, model rankings, and evaluation outcomes vary across prompting and analysis settings.
- MedConclusion contains 5.7M structured PubMed abstracts paired with author-written conclusions and journal-level metadata.
- The experiments find that conclusion generation is behaviorally distinct from summary writing and that strong LLMs remain closely clustered under current automatic metrics.
- LLM-as-a-judge absolute scores are sensitive to judge identity, while journal prestige, biomedical category, and lexical-overlap metrics reveal additional variation in task difficulty and quality.
A Example data
The appendix illustrates MedConclusion records and the prompt and judging materials used to distinguish model inputs, references, generation settings, and evaluation instructions.
- An example record includes PubMed bibliographic fields, journal identifiers, publication details, SJR, and biomedical subject categories.
- The structured abstract separates non-conclusion sections used as model input from the CONCLUSION section used as the ground-truth reference.
- Conclusion prompts distinguish constrained setting C, which enforces sentence and word counts and style matching, from unconstrained setting A.
- Summary prompts distinguish constrained setting D, which enforces style and length targets, from unconstrained setting B.
- The LLM judge compares generated and reference conclusions and scores multiple dimensions from 0 to 100 without explanations.
D Reference-based metrics
The paper evaluates generated conclusions with complementary reference-based metrics covering length, semantic proximity, lexical overlap, fluency, and reproducible aggregate diagnostics.
- Word-count ratio compares generated and reference word counts, with values below 1 indicating shorter generations and values above 1 indicating longer generations.
- Sentence-count ratio compares the number of generated and reference sentences to assess structural length control.
- Embedding cosine similarity measures semantic proximity between generated and author-written conclusions using sentence embeddings.Higher values indicate greater semantic proximity.
- ROUGE-1, ROUGE-2, ROUGE-L, and BLEU quantify lexical overlap or n-gram precision, so they reflect reference wording rather than scientific correctness directly.
- Perplexity under GPT-2 estimates fluency and distributional typicality, with lower values indicating text more probable under the external language model.
- The metrics are computed against paired author-written conclusions, aggregated by arithmetic mean, and interpreted jointly rather than as a complete standalone quality measure.
E Additional category analysis
The appendix provides representative structured-abstract examples spanning biomedical categories, with variation in study content, journal metadata, and reported evaluation scores.
- The examples span categories including Experimental and Cognitive Psychology, Endocrine and Autonomic Systems, Advanced and Specialized Nursing, Environmental Science, Emergency Nursing, and several lower-rated categories.
- The Applied Microbiology and Biotechnology example studies ohmic heating of Bacillus cereus spores in commercial tsuyu sauce across temperatures, durations, survival, and color change.
- The Applied Microbiology and Biotechnology example reports Numeric Consistency: 12, contrasting with Numeric Consistency: 100 in several other examples.
E.11 The conclusion–summary distinction holds across categories
Across biomedical categories, conclusion and summary prompting remain behaviorally distinct even when their semantic scores converge. Summary-mode outputs consistently diverge more in writing style and numeric consistency than in meaning.
- +8.3 mean writing-style gap and +21.6 mean numeric-consistency gap occur across all 112 categories.
- Across all 112 categories, writing style and numeric consistency gaps are always positive, with ranges of +3.8 to +13.7 and +11.3 to +41.3, respectively.
- Biotechnology is the only listed outlier where summary-mode outputs are semantically closer to the reference, with ∆= −4.0, while numeric consistency still drops by +27.2 points.
- Even categories with |∆| < 0.3 in semantic similarity retain writing-style gaps of +4.8 to +8.4 and numeric-consistency gaps of +14.3 to +23.1.
- The results identify the conclusion–summary distinction as a structural property of discourse functions rather than an artifact of category-level heterogeneity.
F MedConclusion dataset statistics
MedConclusion’s dataset statistics describe publication coverage, subject-category composition, and journal influence metadata. The corpus spans 3,772 journals and uses SJR to characterize venue strata.
- SJR accounts for citation volume and citing-journal prestige, uses an iterative PageRank-like algorithm, and down-weights self-citations.
- F.1 Abstracts’ year distribution: The publication-year distribution covers 2000–2025, with a median publication year of 2018.
- Medicine is the dominant subject category at 13.8% of abstracts, while SJR scores across 3,772 journals have median 0.77 and mean 0.98.