Source-linked AI summary
Class Imbalance and Batch Effects in LLM-Based Screening for Systematic Reviews
Gilberto Sussumu Hida, Danilo Monteiro Ribeiro, Clayton Suguio Hida
TL;DR
LLM-assisted screening still has limited evidence on how class imbalance, processing mode, and evaluation granularity shape decision behavior. This study uses a controlled experiment across five reviews and finds that batch processing changes behavior more than prevalence metadata, with effects varying across reviews and models. The findings support evaluating batch processing for decision effects as well as cost.
Problem
Evidence remains limited on how class imbalance, batch processing, screening metadata, and item-level decisions affect LLM behavior in systematic-review screening.
Method
A controlled factorial experiment evaluated screening metadata and individual versus batch processing across five reviews and two LLMs in binary study classification.
Results
Batch processing produced larger and more variable decision changes than textual metadata, while aggregate and item-level analyses did not always converge.
Takeaways & Limitations
Batch processing in LLM-assisted screening should be evaluated for its effects on decision behavior, not treated only as a cost-reduction strategy.
Takeaways & Limitations
The study covers five Software Engineering reviews, two models, one prompt formulation, and one batch arrangement and execution per condition.
Abstract
from arXiv · showhide
This study analyses LLMs in imbalanced binary classification, using study screening in systematic reviews as the application domain. An experiment was conducted in five reviews, comparing individual and batch processing, with and without prevalence metadata. The results indicate a limited influence of the prevalence metadata, with no evidence that it improves performance. In contrast, batch processing produced larger behavioral changes that varied according to the prevalence of the class. The aggregate and item-level analyses did not always coincide. Therefore, batch processing should be evaluated not only in terms of cost, but also in relation to its effects on decision-making behavior.
1. Introduction
This study examines how class imbalance, screening metadata, and individual versus batch processing affect LLM-based screening in systematic reviews. Using a 2×2 design across five reviews and two models, it finds limited effects from prevalence metadata but larger, more variable behavioral effects from batch processing.
- Motivation: Class imbalance is central because most screening processes include only a fraction of records, making traditional metrics misleading and prioritizing false-negative reduction.LLMs may process studies individually or in batches; batching reduces cost but can induce implicit comparisons between studies.
- Research gap: The study addresses whether expected prevalence and processing mode change LLMs’ propensity to include or exclude studies.Prior work often compared models and prompts under individual processing without isolating contextual metadata effects.
- Study design: The factorial design crossed prevalence metadata presence or absence with individual or batch processing across five reviews spanning 2.9% to 53.0% prevalence.The evaluation used Llama-3.3-70B-Instruct-Turbo and GPT-5-mini.
- Contributions: Prevalence metadata had a limited effect, whereas batch processing produced larger and more variable behavioral changes across reviews.The findings also indicate that how articles are presented to the model matters more than prompt-based calibration.
- Contributions: Aggregate and item-level analyses may diverge, complicating interpretation of LLM screening behavior.This observation is presented as an implication for responsible use of LLMs in screening.
2. Related Work
Prior work shows that LLM screening performance varies across reviews, prompts, calibration methods, batch contexts, and class-imbalance metrics. However, batch processing, contextual metadata, and item-level decision changes remain insufficiently examined, motivating this study’s controlled experiment.
- LLM screening and benchmarks: Variation between reviews may exceed variation between models in LLM-based screening benchmarks.SESR-Eval contains 34,528 labeled primary studies from 24 secondary reviews.
- Prompting and calibration: Instructions, examples, and thresholds can alter LLM screening performance, but contextual screening metadata has not been directly isolated.Prior work compared zero-shot, one-shot, and few-shot configurations and proposed calibration based on target recall.
- Batch processing: Batch prompting has mainly been studied for reducing token use, time, and cost while maintaining performance close to individual processing.Related work also indicates that expanded contexts can change behavior depending on information position.
- Class imbalance and metrics: Class imbalance complicates screening evaluation because Include is typically the minority class and accuracy-based metrics may be misleading.Prior work recommends robust agreement measures and metrics aligned with asymmetric false-negative costs, including the F2-score.
- Research gaps: Three gaps remain: batch processing is treated mainly as efficiency, prompts rarely use contextual metadata, and evaluation emphasizes aggregate rather than item-level decision changes.The study addresses these gaps through a controlled experimental design.
3. Methodology
The study used a controlled 2×2 experiment to test how inclusion-rate metadata and individual versus batch processing influence LLM binary screening decisions. It evaluated two models on five reviews spanning different class-prevalence levels, using aggregate and item-level comparisons.
- Task and data: The experiment classified candidate studies as Exclude or Include using titles, abstracts, and review-specific eligibility criteria.The unit of analysis was the candidate study, with reference and model decisions represented as binary labels.
- Experimental design: A 2×2 factorial design crossed metadata presence or absence with individual or batch processing, producing four experimental conditions.Metadata consisted of a sentence reporting the oracle estimated inclusion rate as class-imbalance context.
- Task and data: SESR-Eval supplied reference-labeled software-engineering reviews, and five reviews were selected to span prevalence levels from highly imbalanced to almost balanced.Reviews without both classes, few records, or sufficient eligibility criteria were excluded.
- Models and implementation: Llama-3.3-70B-Instruct-Turbo and gpt-5-mini-2025-08-07 were evaluated with the same prompts, conditions, and elements at temperature 0.Prompts requested JSON containing study identifiers and binary decisions; no invalid final decisions remained.
- Evaluation and comparisons: Evaluation used F2-score, false negative rate, Gwet’s AC1, Decision Flip Rate, net shift, McNemar’s exact test, and paired bootstrap confidence intervals.The analysis compared A↔B and C↔D for metadata effects, and A↔C and B↔D for batch-processing effects.
4. Results and Discussion
Metadata had limited and inconsistent effects on LLM screening decisions, whereas batch processing caused larger changes whose direction and magnitude varied with review prevalence and model. Aggregate performance effects did not always reflect substantial item-level decision changes, so batch processing should be assessed for behavioral effects as well as operational cost.
- Agreement and prevalence: Agreement varied more between reviews than conditions, and the more balanced rsl 4 review had lower agreement, indicating prevalence influenced results.rsl 0 had prevalence below 3%, whereas rsl 4 had 53%; GPT-5-mini showed higher and more stable Gwet’s AC1 than Llama-3.3-70B-Instruct-Turbo in almost all conditions.
- RQ1: Metadata effects: Metadata produced small, variable decision changes: Llama’s individual-processing DFR ranged from 0.9% to 2.3%, while GPT-5-mini’s ranged from 0.5% to 6.3%.For Llama, corresponding ∆F2-scores remained close to zero and 95% CIs crossed zero in all five reviews; GPT-5-mini showed greater amplitude in rsl 4.
- RQ1: Metadata effects: In batch processing, Llama’s C↔D DFR was 0.000 across five reviews, preserving 6,642 decisions, while GPT-5-mini’s DFR ranged from 0.9% to 9.3%.GPT-5-mini’s ∆F2 values were between −0.042 and +0.028, with 95% CIs crossing zero in all SLRs; the tested metadata formulation was not a stable calibration mechanism.
- RQ2: Batch effects: Batch processing caused larger changes than metadata: Llama’s A↔C DFR ranged from 5.6% to 26.7%, versus 1.3% to 14.0% for GPT-5-mini.Llama’s A↔C McNemar results were significant at p < 0.001 in all reviews, while GPT-5-mini results were significant in four of five SLRs.
- RQ2: Batch effects: Llama’s batch-effect ∆F2 varied by review, from gains of +0.096 and +0.043 to losses of −0.092 and −0.101.In rsl 1, F2 increased from 0.37 to 0.46 while 101 studies changed from Exclude to Include; the median A↔C ∆F2 was −0.034, and bootstrap CIs crossed zero in two of five reviews.
- RQ3–RQ4: Prevalence and model differences: Batch processing made Llama more inclusive in reviews with prevalence ≤20%, producing net shifts of +3.5 to +10.5 percentage points, but shifted −25.3 points in the 53%-prevalence review.This pattern is compatible with implicit normalization within batches, but the design does not identify the causal mechanism; Llama’s A↔C DFR also averaged 0.160 versus 0.057 for GPT-5-mini.
5. Conclusion
The study found that batch processing changed LLM screening behavior more than textual prevalence metadata, with effects varying by review and model profile. It recommends evaluating decision changes beyond global metrics while acknowledging limited generalizability and outlining broader future tests.
- Main findings: In a 2×2 factorial design across five reviews and two models, textual metadata had limited influence, whereas batch processing produced larger changes.The experiment examined title-and-abstract selection under individual and batch processing, with and without screening metadata.
- Main findings: Batch processing produced the largest effect, with substantially larger DFRs in A↔C and B↔D comparisons.Its effect varied by review and model profile: Llama tended to become more inclusive in more imbalanced reviews but more restrictive in the single almost balanced review.
- Evaluation: Aggregate metrics and item-level analyses did not always converge, so screening evaluation should combine global metrics, DFR, net shift, and paired article-level analysis.This combination was necessary to characterize how decisions changed between experimental conditions.
- Limitations: The findings are limited to five Software Engineering reviews, two models, one prompt formulation, and one batch arrangement and execution per condition.Binary inclusion/exclusion also simplifies screening, which may involve uncertainty, reviewer discussion, and additional contextual criteria.
- Future work: Future work will test other datasets, domains, model families, batch sizes, position effects, adaptive processing strategies, uncertainty estimation, and operational resource measures.Planned instrumentation includes token usage, monetary cost, and execution time, alongside cluster bootstrap resampling.
Generative AI use statement
ChatGPT and Claude supported linguistic revision, textual clarity, editorial refinement, and visual preparation of Figure 1. The authors prepared and reviewed all scientific content, analyses, interpretations, and conclusions.
- ChatGPT and Claude supported linguistic revision, textual clarity, editorial refinement, and visual preparation of Figure 1.
- The authors prepared and reviewed all scientific content, analyses, interpretations, and conclusions.