Source-linked AI summary
Higher Reasoning Effort Improves Reference Verifiability and Changes Reasoning Summaries in GPT-5.5
Melika Mobini, Andres Algaba, Vincent Ginis
TL;DR
Whether greater reasoning effort improves reference verifiability, and what visible reasoning summaries reveal about model behavior, remain open questions. In a paired study of GPT-5.5 generating references for 1,000 papers, high effort increased verified-reference rates and changed summary features, but feature scores did not consistently track verification within conditions.
Problem
How reasoning effort affects reference verifiability and whether observable reasoning behaviors relate to verification outcomes remain less understood, despite the importance of distinguishing reference existence, bibliographic accuracy, and relevance.
Source: §1 · pp. 1, 3Method
The study paired low- and high-effort GPT-5.5 generation of scholarly references for 1,000 papers, matched references to bibliographic records, and assessed predefined and model-discovered features in reasoning summaries.
Source: §1 · p. 3, §2.3 · p. 5Results
Mean paper-level verified-reference rate rose from 78.3% under low effort to 84.8% under high effort; summaries lengthened, predefined features scored higher, and feature scores did not consistently track verification within conditions.
Source: §3.2 · p. 7, §4 · p. 9Takeaways & Limitations
Reference quality and reasoning summaries should be evaluated separately, since describing caution or checking does not guarantee verifiable references.
Source: §4 · p. 11The study used one model and one prompt per paper and effort setting, with Q1 journal papers from 1999–2021 and uneven field coverage, limiting generalization.
Source: §4 · p. 11
Abstract
from arXiv · showhide
Large language models are increasingly being used in different stages of scientific knowledge production, including the suggestion of references. These models can operate at different levels of reasoning effort and provide summaries of their reasoning processes. This raises two questions: Does increasing reasoning effort improve the verifiability of generated references, and do the resulting reasoning summaries reveal how the model's behavior changes with greater effort? We examined summaries from GPT-5.5 as it generated scholarly references for 1,000 papers under paired low- and high-effort settings. Generated references were matched to external bibliographic records, and reasoning summaries were scored for four predefined features and explored for additional features discovered by GPT-5.5. High effort increased the mean paper-level resolver-verified reference rate from 78.3% to 84.8%, and produced longer reasoning summaries. All four predefined features received higher scores under high effort. The discovered features changed less uniformly: some appeared equally often in both conditions, whereas others were more common under high effort. Within each effort condition, neither feature set showed a consistent increase in verification rates with higher scores.
1 INTRODUCTION
The study asks whether greater reasoning effort improves reference verifiability and whether visible reasoning summaries reveal changes in model behavior, while recognizing that summaries may not capture the model’s full reasoning. It tests these questions with paired low- and high-effort reference generation and analysis of predefined and model-discovered summary features.
1 INTRODUCTION
- Generated-reference quality spans publication existence, bibliographic accuracy, and topical relevance, which prior work treats as distinct dimensions.A real publication match alone does not establish relevance or bibliographic accuracy.Source: §1 · p. 1
2.1 Citation generation and reference verification
- For 1,000 focal papers, GPT-5.5 generated 20 references per paper under paired low- and high-effort settings, and the study matched generated references to external bibliographic records.A reference qualified as verified-paper only when a candidate met title- and author-similarity thresholds and matched the normalized first-author surname; failure to match did not establish nonexistence.Source: §2.1 · p. 4
2.2 Reasoning summaries and predefined features
- Four predefined summary features were scored separately on a five-point scale, with scores of 3 or higher treated as feature presence.The rubric scored only behavior expressed in the saved summary; missing summaries received a score of 1, which did not establish absence from internal reasoning.Source: §2.2 · pp. 4–5
2.3 Model-discovered feature analysis
- GPT-5.5 separately discovered features in low- and high-effort batches, scored each paper against its batch’s features, and retained features matched across all five batches and both conditions.Scores of 3 or higher counted as feature presence.Source: §2.3 · p. 5
3 RESULTS
Higher reasoning effort increased resolver-verified reference rates and summary length, while changing the expressed reasoning features. Within either effort condition, feature scores did not show a consistent increase in verification rates.
3.1 Summary length and reference verification
- Median summary length increased from 74 words under low effort to 248 words under high effort, with mean lengths varying across fields.High-effort field means ranged from 199 words in Humanities to 390 words in Geography; saved-text length does not measure how much internal reasoning a field requires.Source: §3.1 · p. 5, Figure 2 · p. 6
- Mean paper-level verified-reference rate increased from 78.3% under low effort to 84.8% under high effort.Figure 2 reports paired observations across 1,000 focal papers; the mean paired increase was 6.53 percentage points (95% CI 5.78–7.28).Source: §3.1 · pp. 5, 7, Figure 2 · p. 6
3.2 Predefined-feature scores, citation outcomes, and summary length
- All four predefined features scored higher under high effort, but within-condition verification showed no consistent increase with feature score.Feature presence also varied with summary length, which was closely related to effort and could not be separated from it in these comparisons.Source: §3.2 · p. 7, Figure 3 · p. 8
3.3 Model-discovered feature prevalence, citation outcomes, and summary length
- Eleven matched model-discovered features shifted unevenly: seminal-source, temporal-selection, and relevance-note features were more prevalent under high effort, while formatting and list-organization features favored low effort.Within either condition, stronger feature scores did not consistently accompany higher verification rates.Source: §3.3 · p. 9
4 DISCUSSION
In this paired study, high effort improved reference verification while changing reasoning summaries, but feature scores did not consistently track verification within effort conditions. The authors caution that these findings do not establish whether the measured summary behaviors contributed to the improvement.
4 DISCUSSION
- Mean paper-level verified-reference rate rose from 78.3% under low effort to 84.8% under high effort, which also produced longer summaries.High effort also increased scores for all four predefined features; model-discovered features changed less uniformly.Source: §4 · p. 9
- High effort raised all four predefined-feature scores, while discovered features shifted unevenly; neither feature set showed consistent score–verification gains within effort conditions.Some discovered features appeared in similar numbers of summaries but were expressed more strongly under high effort.Source: §4 · p. 9
- Several predefined features and the number of discovered features increased across longer-summary bins, but high-effort summaries did not contain more features at every shared length.The findings do not establish how much summary length affects feature-score differences or whether longer summaries are linked to better verification.Source: §4 · p. 11
- The study used one model and one prompt per paper and effort setting, and its Q1 journal sample had uneven field coverage, limiting generalization.Feature scores covered whole summaries rather than individual references, and missing summaries were scored as 1, conflating missing information with feature absence.Source: §4 · p. 11
- Paper difficulty may confound within-condition associations, and the study cannot establish whether measured behaviors contributed to high-effort verification gains.The higher verification rate may also partly reflect a shift toward more highly cited or better indexed publications.Source: §4 · pp. 9, 11
- The authors recommend broader model, prompt, and field coverage, plus reference-level checking and tests of verification instructions or external retrieval tools.Future work should also examine how missing summaries and summary length affect results while accounting for field variation and the paired design.Source: §4 · p. 11
6 AUTHOR CONTRIBUTIONS
The authors report their individual contributions to the study.
6 AUTHOR CONTRIBUTIONS
- MM contributed to conceptualization, methodology, software, formal analysis, investigation, visualization, and the original draft; AA and VG contributed to conceptualization, methodology, resources, supervision, and review and editing.Source: §6 · p. 11
7 FUNDING
The paper acknowledges institutional, foundation, and fellowship support for AA and VG.
7 FUNDING
- AA acknowledges support from the VUB Research Council, the Francqui Foundation through a Francqui Start-Up Grant, and FWO fellowship Grant No.1286924N; VG acknowledges FWO Grants G032822N and G0K9322N.Source: §7 · p. 12
10 DATA AVAILABILITY STATEMENT
The authors provide the study code and archive the study data on Zenodo.
10 DATA AVAILABILITY STATEMENT
- The code covers reference generation, metadata extraction, reference resolution, feature discovery and scoring, and figure generation; the Zenodo archive includes prompts, responses, summaries, parsed and resolved references, and feature scores.The archive also includes focal-paper identifiers and retained candidate records with match scores; focal-paper metadata comes from Algaba et al. (2026b).Source: §10 · p. 12
S1 REFERENCE MATCHING AND CLASSIFICATION
The section defines resolver-based reference verification, separate metadata-agreement and temporal-consistency outcomes, and describes retrieval-source contributions and unresolved candidates.
S1.1 Candidate ranking and verification
- Candidates were ranked by title and author similarity; verification required thresholds of 0.90 and 0.50 plus matching normalized first-author surnames, while lower thresholds could yield a possible match.The resolver considered the three highest-ranked candidates, and failure to find an acceptable match does not establish that a publication does not exist.Source: S1.1 Candidate ranking and verification · p. 15
S1.2 Metadata agreement
- Metadata agreement was classified as correct, minor-error, or major-error, with insufficient metadata assigned when no candidate record or essential extracted citation fields were available.Year discrepancies over one year and conflicting identifiers counted as major errors; exact year matches were correct, while a one-year discrepancy or missing year could be minor absent a major error.Source: S1.2 Metadata agreement · p. 15
- Metadata-correct references rose from 61.47% (12,294) under low effort to 71.65% (14,320) under high effort.Minor errors fell from 5.03% to 3.85%, and major errors from 33.50% to 24.47%; these classifications measure agreement with selected candidate records independently of verification status.Source: S1.2 Metadata agreement · p. 15
S1.3 Temporal consistency
- Temporal consistency classified references dated in or before the focal paper’s year as valid and later-dated references as future references.The assessment used the year reported in the generated reference, so it measured consistency between reported years.Source: S1.3 Temporal consistency · p. 16
S1.4 Source contributions and candidate availability
- OpenAlex and Crossref provided most verified matches, while source-count comparisons depend on search order because retrieval stopped after a verified or possible match.Nearly all unresolved references had retrieved candidates, but none met the resolver’s matching criteria.Source: S1.4 Source contributions and… · p. 16
S2 PROMPTS USED IN THE STUDY
The supplement records the study’s LLM prompts and request settings. It documents reference generation, metadata extraction, feature scoring and discovery, alongside two figures describing reference years and resolver matches.
S2 PROMPTS USED IN THE STUDY
- The prompt supplement reproduces instructions for the study’s LLM stages; Table S1 lists model identifiers and request settings, and failed or canceled batches were resubmitted only for papers without a completed response.One response was retained per paper and condition.Source: S2 PROMPTS USED IN THE STUDY · p. 17, Table S1 · p. 17
S2.1 Citation-generation prompt
- GPT-5.5 was prompted to provide 20 relevant references using the focal paper’s metadata and abstract.The retained materials included the rendered prompt, raw API response, final answer, reasoning summary, and parsed references.Source: S2.1 Citation-generation prompt · p. 18
S2.2 Citation-metadata extraction prompt
- The extraction prompt asked GPT-4.1-mini to preserve each reference’s rank and extract bibliographic fields into a table without adding, omitting, or combining references.Requested fields included authors, title, year, venue, DOI, and arXiv ID.Source: S2.2 Citation-metadata extraction prompt · p. 18
S2.3 Predefined-feature scoring prompt
- The common scoring instructions asked judges to classify only what appeared in supplied citation-generation reasoning, using shared feature definitions and a rubric to assess one reasoning summary per request.The named labels included epistemic caution, temporal-constraint awareness, and reference-verifiability checking.Source: S2.3 Predefined-feature scoring prompt · p. 19
- GPT-5.5 returned four integer scores under a structured JSON schema, while Claude Opus 4.8 used the record_scores tool with the same substantive scoring instructions.The shared rubric described scores from absent or weak behavior through explicit and repeated verification.Source: S2.3 Predefined-feature scoring prompt · p. 19
S2.4 Model-discovered feature-discovery prompt
- GPT-5.5 discovered additional reasoning features separately by effort condition and 200-paper batch, naming each feature and providing an operational definition.Each batch’s corresponding reasoning summaries were supplied to the discovery stage.Source: S2.4 Model-discovered… · p. 20
S2.5 Model-discovered feature-scoring prompt
- Each paper was scored only on features discovered for its own effort condition and batch; the scoring prompt required frozen definitions and a 1-to-5 score for each feature.The rubric was unchanged in later batches, and scorers were told to judge only the single supplied reasoning summary, not reference correctness.Source: S2.5 Model-discovered feature-scoring… · pp. 20–21
- Figure S1 compares generated-reference years with focal-paper years by effort condition and shows year-offset distributions; Figure S2 plots title- and author-match scores against resolver classifications.Figure S1 marks later-dated references as positive offsets; Figure S2 notes that verified classification also requires normalized first-author surname agreement.Source: Figure S1 · p. 22, Figure S2 · p. 23
S3 FEATURE SCORES AT THE EXTREMES OF VERIFICATION
Table S2 compares mean predefined-feature scores for papers at the low and high extremes of verified-reference rates within each effort condition.
S3 FEATURE SCORES AT THE EXTREMES OF VERIFICATION
- Table S2 reports mean predefined-feature scores for papers with the lowest and highest verified-reference rates, compared within each reasoning-effort condition.The groups are defined by the 10th and 90th percentiles of paper-level verified-reference rate.Source: Table S2 · p. 24
- Because verified-reference rates are discrete, papers tied at either percentile threshold are included, so group sizes differ from 100.The rate advances in steps of about five percentage points.Source: Table S2 · p. 24
S4 MODEL-DEFINED REASONING FEATURE DEFINITIONS
Table S3 provides definitions for eleven model-defined reasoning features.
S4 MODEL-DEFINED REASONING FEATURE DEFINITIONS
- Table S3 lists definitions of eleven model-defined reasoning features.Source: Table S3 · p. 25
S5 ROBUSTNESS TO THE SCORING JUDGE
A second judge largely reproduced the feature-score patterns across reasoning-effort conditions. Both judges assigned higher scores to all four predefined features under high effort and found greater high-effort presence for four emergent features.
S5 ROBUSTNESS TO THE SCORING JUDGE
- Both GPT-5.5 and Claude Opus 4.8 assigned higher scores to all four predefined reasoning features under high effort.The comparison used valid scores from 994 low-effort and 995 high-effort observations; Figure S3A shows the judges’ mean scores on a 1–5 ordinal scale.Source: S5 ROBUSTNESS TO THE SCORING JUDGE · p. 26, Figure S3 · p. 26
- Both judges found greater high-effort presence for seminal sources, temporal selection, relevance notes, and bibliographic accuracy among the emergent features.Figure S3B defines feature presence as a score of at least 3; comparisons used 996 low-effort and 995 high-effort observations.Source: S5 ROBUSTNESS TO THE SCORING JUDGE · p. 26, Figure S3 · p. 26
- Claude assigned lower mean scores than GPT-5.5 except for temporal-constraint awareness under high effort.The rescoring used the same feature definitions and rubric as GPT-5.5.Source: S5 ROBUSTNESS TO THE SCORING JUDGE · p. 26