Source-linked AI summary
Large language models surpass human experts in predicting neuroscience results
Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin K. Nejad, Felipe Yáñez, Bati Yilmaz, Kangjoo Lee, Alexandra O. Cohen, Valentina Borghesani, Anton Pashkov, Daniele Marinazzo, Jonathan Nicholas, Alessandro Salatiello, Ilia Sucholutsky, Pasquale Minervini, Sepehr Razavi, Roberta Rocca, Elkhan Yusifov, Tereza Okalova, Nianlong Gu, Martin Ferianc, Mikail Khona, Kaustubh R. Patil, Pui-Shee Lee, Rui Mata, Nicholas E. Myers, Jennifer K Bizley, Sebastian Musslick, Isil Poyraz Bilgin, Guiomar Niso, Justin M. Ales, Michael Gaebler, N Apurva Ratan Murty, Leyla Loued-Khenissi, Anna Behler, Chloe M. Hall, Jessica Dafflon, Sherry Dongqi Bao, Bradley C. Love
TL;DR
The paper asks whether LLMs can predict experimental outcomes by synthesizing large scientific literatures that challenge human information-processing capacity. It introduces BrainBench, a forward-looking neuroscience benchmark, and finds that LLMs outperform human experts, with stronger performance after neuroscience-specific training. The authors also identify a limitation: models may have been exposed to the original abstracts.
Problem
It is an open question whether LLMs trained on scientific articles can predict experimental outcomes, a question motivated by the difficulty of keeping up with the scientific literature.
Method
BrainBench tests prediction by asking models and experts to select which of two neuroscience abstract versions contains the study’s actual results.
Results
LLMs outperformed human experts on BrainBench by a considerable margin across all neuroscience subfields, and their performance improved with additional neuroscience training.
Takeaways & Limitations
LLMs can identify likely neuroscience results and could form part of larger systems that assist researchers in determining scientific outcomes.
Takeaways & Limitations
A limitation is that LLMs may have been exposed to the original abstracts used in BrainBench.
Abstract
from arXiv · showhide
Scientific discoveries often hinge on synthesizing decades of research, a task that potentially outstrips human information processing capacities. Large language models (LLMs) offer a solution. LLMs trained on the vast scientific literature could potentially integrate noisy yet interrelated findings to forecast novel results better than human experts. To evaluate this possibility, we created BrainBench, a forward-looking benchmark for predicting neuroscience results. We find that LLMs surpass experts in predicting experimental outcomes. BrainGPT, an LLM we tuned on the neuroscience literature, performed better yet. Like human experts, when LLMs were confident in their predictions, they were more likely to be correct, which presages a future where humans and LLMs team together to make discoveries. Our approach is not neuroscience-specific and is transferable to other knowledge-intensive endeavors.
Introduction
Scientific literature is growing beyond human capacity to synthesize, while existing scientific benchmarks mainly test backward-looking factual recall rather than prediction. BrainBench addresses this gap by testing whether LLMs can predict neuroscience results, and LLMs surpassed human experts, with BrainGPT improving further.
- Motivation: The exponentially increasing scientific literature creates a superhuman challenge for researchers trying to keep up.Neuroscience is especially difficult because it spans many articles, noisy and potentially nonreplicating studies, multiple biological levels, and diverse analysis methods.
- Motivation: Potentially disruptive findings may go unnoticed because researchers must process and integrate an overwhelming volume of articles.LLMs can process substantially more scientific literature during training than individual human scientists.
- Benchmark gap: Most scientific benchmarks are backward-looking, testing retrieval and reasoning over accepted facts rather than prediction of novel outcomes.Existing question-answering benchmarks therefore do not directly evaluate forward-looking scientific prediction.
- BrainBench: BrainBench presents two coherent versions of a recent neuroscience abstract, requiring test-takers to select the original result from an altered version.The altered abstract significantly changes the study outcome while preserving overall coherence, making the task forward-looking.
- BrainBench: BrainBench was designed to quantify LLMs’ forward-looking ability and compare it directly with human experts.The benchmark evaluates whether models have learned patterns linking neuroscience methods and results.
- Results: LLMs surpassed human experts on BrainBench by a substantial margin, and additional neuroscience training increased that margin for BrainGPT.The paper frames this result as evidence that LLMs may support human scientists’ predictions and study designs.
Results
LLMs outperformed human experts on BrainBench across neuroscience subfields, and their confidence tracked accuracy. Their performance appears to depend on integrating information across abstracts rather than memorizing benchmark items, while neuroscience fine-tuning improved results further.
- 81.4% average LLM accuracy exceeded 63.4% for human experts on BrainBench.
- Base models outperformed chat and instruction-optimized counterparts, while smaller models performed comparably to larger models.The chat and instruction-optimized models performed worse than their base counterparts, with t(5) = 5.4, p = .002.
- LLMs performed better than human experts in every neuroscience subfield, although Behavioral/Cognitive cases were overrepresented.
- Restricting models to altered results sentences substantially reduced performance, supporting integration of background, methods, and results across the abstract.Performance also declined when sentences were randomly swapped within subfields, indicating only partial benefit from accurate but non-study-relevant context.
- Analyses found no indication that BrainBench was memorized, and all checks indicated that its items were novel for the LLMs.
- LLM confidence was positively associated with accuracy, as it was for human experts, indicating that more confident judgments were more likely to be correct.
Discussion
The paper introduces BrainBench to test whether LLMs can predict neuroscience outcomes, finding that LLMs outperform human experts and that BrainGPT performs better still. The results support calibrated, complementary human–LLM collaboration while highlighting constraints involving memorization, access to model weights, benchmark construction, and domain transfer.
- Benchmark and performance: BrainBench is a forward-looking benchmark that evaluates whether models can select between two possible neuroscience study results.The benchmark was created to quantify predictive ability and compare LLMs with human experts.
- Benchmark and performance: LLMs surpassed human experts on BrainBench by a considerable margin across all neuroscience subfields.The task required predicting which of two versions of a neuroscience abstract contained the actual results.
- Mechanism and memorization: Analyses found no indication that LLMs memorized BrainBench items, while a small model trained from scratch on published neuroscience literature still achieved superhuman performance.Membership tests and comparisons involving Galactica supported learning broad patterns rather than simple item recall.
- BrainGPT and updating: BrainGPT, created by re-orienting general-purpose LLMs with LoRA and neuroscience knowledge, further boosted BrainBench performance.The approach can be combined with retrieval of relevant, up-to-date scientific articles.
- Human–LLM collaboration: LLMs’ predictions were well calibrated: when confident, models were more likely to be correct, and their difficult items differed from those difficult for human experts.These properties support combining human and machine judgments rather than relying on either alone.
- Future use: Future systems could generate and rank multiple possible results to help scientists design experiments, although human experts may remain important for scientific explanations.The paper frames prediction as valuable but not sufficient by itself.
- Scope and limitations: The approach may transfer to other knowledge-intensive fields, but efficacy may depend on domain structure and could be lower in disciplines relying heavily on logical deduction.The authors also note that benchmark creation is labor intensive and that their analyses required access to LLM weights.
Data availability
The study’s human-participant data and intermediate simulation and analysis data are publicly available, along with model weights and training data.
- Human participant data are publicly available.
- Intermediate data generated through simulations and analyses are publicly available.
- Model weights and training data are available through Hugging Face.
Code availability
The computer code for model training, evaluation, data processing, and analyses is publicly available through the project’s GitHub repository.
- Code for model training is publicly available on GitHub.
- The repository includes code for evaluation and data processing.
- The repository also includes code for the analyses.
Materials and Methods
The study reports ethics approval from University College London and confirms informed consent, while noting that participant compensation was not applicable.
- The study protocol was approved by the Experimental Psychology Ethics Board at University College London.
- The reported ethics protocol number is EP/2017/011.
- The study confirms that informed consent was obtained from all human participants.
- Participant compensation was not applicable to the current study.
Dataset Creation
BrainBench test cases were built by modifying neuroscience abstracts so their results changed while their methods, background, coherence, and syntax remained intact. The benchmark combined human- and GPT-4-created cases across five neuroscience subfields and applied quality control.
- Dataset Creation: Test cases were sourced from Journal of Neuroscience abstracts and distributed across five neuroscience subfields.
- Dataset Creation: BrainBench comprised 200 human-crafted test cases and 100 additional GPT-4-generated test cases.
- Dataset Creation: The cases were subjected to extensive quality control by human experts and GPT-4.
- Dataset Creation: Each test case altered a published abstract’s result while keeping its methods and background unchanged.
- Dataset Creation: Edits were required to remain empirically different but logically coherent, syntactically consistent, and not trivially decodable from the abstract.
- Dataset Creation: GPT-4 was instructed to make scientifically meaningful edits that preserve the abstract’s overall meaning and address its significant findings.
Evaluations
BrainBench tests whether human experts and LLMs can identify altered neuroscience abstracts by comparing original and modified versions. LLM evaluation uses perplexity-based choices, accuracy, confidence calibration, cross-model difficulty agreement, contextual ablations, and memorization checks.
- BrainBench task: Human experts and LLMs selected which of two abstract versions contained the actual results, with one version altered.The benchmark included human-created and machine-created test cases.
- LLM evaluation: LLMs chose the abstract with lower perplexity, treating perplexity as the exponentiated average negative log-likelihood of its token sequence.The original was selected when its perplexity was lower; otherwise, the altered abstract was selected.
- Evaluation metrics: Accuracy was the primary metric, while confidence calibration compared accuracy across 20 bins ordered by the absolute perplexity difference between abstract versions.A well-calibrated model should achieve higher accuracy in higher-confidence bins.
- Cross-model analysis: Model agreement was assessed by correlating how different LLMs ranked test-case difficulty using Spearman correlation.Difficulty was based on the perplexity difference between incorrect and correct abstracts.
- Context and robustness: Context integration was tested by comparing performance on full abstracts with performance on individual result-containing sentences and result-swapped abstracts.Swapped abstracts retained the original number of results and were drawn within the same neuroscience subfield.
Fine-tuning on neuroscience corpora
The study built a neuroscience corpus from PubMed and PubMed Central and used it to adapt a general-purpose LLM with parameter-efficient fine-tuning. The resulting model was evaluated on BrainBench against its pre-fine-tuning performance.
- Fine-tuning approach: BrainGPT was created by enhancing general-purpose LLMs with neuroscience-specific expertise through low-rank adapters.LoRA adds trainable low-rank parameters without training the entire base model.
- Training setup: The fine-tuned model used Mistral-7B-v0.1 with LoRA adapters applied to self-attention and fully connected layers.The adapters comprised 629,145,600 trainable parameters, approximately 8% of the base model.
- Evaluation: Evaluation applied the same BrainBench procedure before and after fine-tuning and used a paired t-test on correct-option perplexity.The comparison tested whether fine-tuning changed model performance on the benchmark.
Relative perplexities evaluated at token and sentence levels
Token- and sentence-level analyses tracked how relative perplexity changed between original and altered abstracts. Correct classifications were generally associated with more stable evidence across the sequence, though models could revise initial judgments.
- Classification dynamics: Examples showed that models did not always identify the correct abstract at the first alternation and did not consistently retain the same preference.Relative perplexity could change between the altered and original abstracts over successive tokens.
- Token-level trajectories: More than half of abstracts classified correctly at the first result alternation remained correct at the end.20% of abstracts classified incorrectly at the first alternation were later classified correctly.
- Token-level trajectories: Correct and incorrect classifications differed in the number of sign flips in relative perplexity across tokens.Eventually correct abstracts showed fewer sign flips than abstracts eventually classified incorrectly.
- Sentence-level analysis: The same relative-perplexity patterns appeared when the analysis was performed at the sentence level.Sentence-level results were reported across multiple supplementary figures.
On the relationship between performance, model size and training data
BrainBench performance varied with model size and training-data quality rather than increasing uniformly with parameter count. Results suggest that both scale and corpus quality may contribute to strong neuroscience prediction performance.
- Scaling limits: Models around 7 billion parameters performed similarly, while larger models did not necessarily achieve higher accuracy.The authors posit that superhuman performance may reflect synergy between model size and training-set quality.
- Model size: GPT-2 large achieved 72% accuracy, reducing the gap to the much larger models tested.The authors interpret this pattern as suggesting that model size may support integration of deep knowledge.
- Model size: TinyLlama achieved 70.5% accuracy in its base version and 65.5% in its chat version, below the tested 7B models.The two reported TinyLlama variants performed differently despite their shared model family.
- Training-data quality: Phi-3 achieved 82.5% accuracy on BrainBench, which the authors attributed to its high-quality curated training data.This result motivated questions about scaling laws and diminishing returns from model scaling.
Tables
The supplementary tables document model dates, training-data inclusion, calibration analyses, LoRA fine-tuning sources, and author contributions.
- Table S.1 lists LLM training cutoffs and initial release dates.
- Table S.2 records whether data sources were included in AI model training sets.
- Table S.3 compares logistic fits linking model perplexity differences or human confidence with test-case correctness; positive correlations indicate calibration.
- Table S.4 identifies journals supplying abstracts and full articles from 2002–2022 for LoRA fine-tuning.
- Table S.5 breaks down author contributions by task, with earlier-listed authors contributing more for each task.
Figures
The figures describe BrainBench’s interface, model and expert performance, contextual dependence, difficulty and calibration analyses, memorization checks, and LoRA fine-tuning.
- The test interface asks participants to select the original abstract among colored alternatives, with one click selecting all options of that color.
- Models with at least 7 billion parameters perform similarly, instruct/chat variants perform worse, and neuroscience-trained GPT-2 models improve with size toward human performance.
- Removing or swapping background and methods context, especially eliminating context, reduces LLM performance, indicating reliance on information integrated across abstracts.
- LLMs’ difficulty rankings correlate more strongly with one another than human experts’ rankings, averaging 0.75 (±0.08) for LLMs versus 0.15 (±0.03) for human experts.
- LLMs outperform human experts across neuroscience subfields, while the test-case distribution roughly mirrors Journal of Neuroscience articles with Behavior/Cognitive overrepresented.
- Confidence tracks correctness for humans and LLMs; token-level examples show responses can remain stable or flip as altered abstracts are processed.