PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs Know a Good Hypothesis When They See One? Logit-Based Energy Scoring Outperforms Prompted LLM-as-Judge for Scientific Hypothesis Ranking

Swati Rajwal, Sanjay Das, Tirthankar Ghosal

arXiv:2608.17270v1cs.AI

TL;DR

Evaluating whether LLMs can distinguish correct scientific hypotheses from plausible distractors is difficult with judge-style and similarity-based methods. This paper tests logit-based intrinsic scoring and finds that likelihood-based open-weight scoring outperforms zero-shot listwise prompting in aggregate, while the benchmark remains limited to known published hypotheses.

  • Problem

    Evaluating generated scientific hypotheses is difficult because judge-style and semantic-similarity methods can favor conventional ideas, while expert review is costly and unscalable.

  • Method

    The study benchmarks model-intrinsic likelihood and raw-logit energy scores for identifying the correct hypothesis among sixteen candidates.

  • Results

    Likelihood-based scoring of open-weight models substantially outperforms zero-shot listwise prompting for identifying correct scientific hypotheses in aggregate.

  • Takeaways & Limitations

    The findings suggest that mechanistic confidence signals deserve more attention than prompting-based approaches for scientific hypothesis evaluation.

  • Takeaways & Limitations

    Because the benchmark uses published hypotheses with known ground truth, it tests identification of known hypotheses rather than genuinely open-ended scientific reasoning.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used for scientific hypothesis generation. However, evaluating generated hypotheses remains a challenge for trustworthy AI-enabled scientific workflows. Existing approaches often use LLMs as judges or rely on semantic similarity, which can favor familiar ideas over novel ones. We propose a logit-based energy scoring method that evaluates hypotheses using a language model's intrinsic confidence rather than comparative judgment. We benchmarked seven language models on 1,323 papers across 12 disciplines. Each paper was paired with its hypothesis and fifteen incorrect alternatives. Intrinsic scoring reached 33.0% Hit@1 pooled across both scorers, compared with 16.6% for prompted listwise ranking. The strongest configuration, a 1-billion-parameter model using logit-based energy scoring, reached 53.1%, though this was the maximum across 14 model-by-scorer combinations selected post hoc. Overall, intrinsic model confidence shows potential for scientific hypothesis evaluation. This study also motivates future research on confidence-based methods for trustworthy AI-enabled scientific discovery.

I. INTRODUCTION · II. BACKGROUND AND RELATED WORK

The paper frames reliable scientific-hypothesis evaluation as a prerequisite to trustworthy LLM-assisted discovery and proposes logit-based energy scoring as an intrinsic alternative to prompted judgment. Its reported results suggest that model likelihood can outperform prompted ranking, although energy scoring benefits only some models.

  • I. INTRODUCTION: LLMs increasingly support hypothesis generation, literature analysis, and autonomous scientific-discovery workflows.Examples include Google’s AI Co-Scientist and Sakana AI.
  • I. INTRODUCTION: A scientifically useful hypothesis should align with prior literature while explaining or predicting a phenomenon of interest.
  • I. INTRODUCTION: Before generating novel hypotheses, the paper asks whether models can recognize which candidate hypothesis domain experts actually pursued.This recognition task is presented as distinct from, and arguably prerequisite to, novel-hypothesis generation.
  • I. INTRODUCTION: Existing evaluation relies mainly on LLM-as-a-judge methods, semantic similarity metrics, or manual expert verification.Automated judges can exhibit position, verbosity, and semantic-alignment biases, while expert review is constrained.
  • I. INTRODUCTION: A model’s own likelihood over a candidate hypothesis is reported as a stronger correctness signal than prompting a proprietary model to rank candidates.Nearly every tested open-weight LLM beats the prompted baseline, but the better intrinsic scorer depends on the model.
  • I. INTRODUCTION: Logit-based Energy Scoring uses output-logit probabilities as a proxy for internal confidence to identify plausible hypotheses without verbalized judgment.The framework is benchmarked from 1B-parameter models through 20B systems and top intrinsic criteria.
  • I. INTRODUCTION: 53% is the best result among 14 model-by-scorer combinations, while raw target-logit energy provides the largest single gain but is not uniformly superior to likelihood scoring.The benefit of energy scoring is concentrated in a subset of models, motivating further characterization.
  • I. INTRODUCTION: The paper contributes a fully automatic logit-derived energy metric and a systematic comparison of open-source models with a state-of-the-art closed commercial model.

A. LLM-based Scientific Hypothesis Generation … A. Problem Setup: Confidence as a Plausibility Signal

The paper frames hypothesis evaluation as a plausibility-ranking problem: LLMs can generate fluent scientific candidates, but existing judging and similarity methods may not reliably identify the best hypothesis. It therefore evaluates candidates independently using token-level confidence and predictive surprisal under shared context, avoiding explicit comparative judgment.

  • A. LLM-based Scientific Hypothesis Generation: LLM systems increasingly participate in scientific discovery, including hypothesis generation, experimentation, coding, and paper drafting.Examples include Google’s AI co-scientist and Sakana AI’s AI Scientist, alongside systems for materials discovery, retrosynthesis, and theorem generation.
  • A. LLM-based Scientific Hypothesis Generation: Because LLMs reliably produce fluent, domain-appropriate candidate hypotheses, the literature treats generation as comparatively solved while evaluation remains harder.
  • B. Evaluation via LLM-as-Judge and Semantic Similarity: Current evaluation commonly uses prompted LLM-as-judge comparisons or embedding-based semantic similarity between candidate and reference hypotheses.LLM judges output scores, rankings, or preferences, whereas embedding methods compare representations using cosine similarity or related metrics.
  • C. Confidence and Likelihood as Evaluation Signals: Prior work uses token-level language-model probabilities for reranking, factuality estimation, and out-of-distribution or hallucination detection.Energy-based models provide an unnormalized compatibility signal, with lower energy indicating higher compatibility.
  • C. Confidence and Likelihood as Evaluation Signals: Raw pre-softmax logit magnitude complements normalized log-likelihood because vocabulary-wide normalization can compress or distort confidence signals.The proposed rationale is that logits may better distinguish a well-supported hypothesis continuation from one that is merely fluent.
  • III. METHODOLOGY: The method scores each hypothesis independently from per-token model confidence given the paper’s background survey and research question.A well-supported hypothesis should receive high probability, whereas a less plausible one should produce higher predictive surprise.
  • A. Problem Setup: Confidence as a Plausibility Signal: Each paper contains one gold hypothesis and fifteen plausible but incorrect distractors, which are scored under identical context and ranked by predictive surprisal.The lowest-surprisal candidate is selected, requiring one forward pass per candidate and no explicit comparative judgment step.

B. Prompt Construction and Hypothesis-Span Isolation

Each candidate is evaluated with a fixed prompt containing the paper’s background, research question, and candidate hypothesis. Scoring uses only the hypothesis-span token positions, excluding shared prefix predictions that would add noise without improving candidate discrimination.

  • Prompt template: A fixed prompt places the paper’s background and research question before each candidate hypothesis.The template uses the fields “Background,” “Research Question,” and “Hypothesis.”
  • Logit computation: The full prompt is tokenized and processed in one forward pass to produce next-token logits at every position.This provides token-level predictions for the complete prompt sequence.
  • Hypothesis-span isolation: Only tokens in the hypothesis span contribute to scoring because the background and research question are identical across all sixteen candidates.Including predictions from the shared prefix would add noise without aiding discrimination among candidates.
  • Hypothesis-span isolation: The hypothesis span is identified by tokenizing the fixed prefix through “Hypothesis:” and using its token length as the sequence offset.Positions at or beyond this offset define the hypothesis span H used for scoring.

C. Logit-Based Energy Scores · 1) Softmax Negative Log-Likelihood (NLL) Energy Score: · 2) Raw Target-Logit Energy Score:

The method derives two complementary token-level energies over each candidate hypothesis span: normalized softmax NLL and unnormalized raw target-logit energy. NLL measures calibrated likelihood, whereas raw-logit energy preserves sensitivity to output-logit scale and calibration.

  • C. Logit-Based Energy Scores: Two token-level scores—one normalized and one unnormalized—are aggregated over the hypothesis span into a scalar for each candidate.The hypothesis occupies the final token positions, and logits predict the observed hypothesis tokens.
  • 1) Softmax Negative Log-Likelihood (NLL) Energy Score:: Softmax NLL is the per-token cross-entropy between the model’s predictive distribution and the observed token.It is computed after softmax normalization.
  • 1) Softmax Negative Log-Likelihood (NLL) Energy Score:: Lower NLL indicates that the model assigns higher likelihood to the hypothesis given its context, serving as evidence of plausibility.Small per-token NLL occurs when normalized probability mass concentrates on the observed token.
  • 2) Raw Target-Logit Energy Score:: Raw target-logit energy negates the observed token’s raw logit before softmax normalization.It treats unnormalized log-probabilities as an energy function over the vocabulary.
  • 2) Raw Target-Logit Energy Score:: Unlike NLL, raw target-logit energy is not renormalized against the vocabulary and remains sensitive to logit scale and calibration.Hypotheses with identical softmax probabilities can receive different energies when their underlying logit distributions differ in sharpness.
  • 2) Raw Target-Logit Energy Score:: Candidates are ranked ascending by SNLL or SRaw, and the returned result is the gold hypothesis’s rank position.The algorithm computes both scores for each candidate before ranking the candidate set.
  • 2) Raw Target-Logit Energy Score:: For each candidate, the algorithm formats the prompt, identifies the hypothesis-token span, and applies teacher forcing in a single forward pass.The span begins at the offset after the prompt prefix and includes the candidate hypothesis tokens.
  • 2) Raw Target-Logit Energy Score:: The two criteria differ only in the partition term: NLL divides out the model’s total confidence mass, while raw energy retains it.The partition term measures total confidence mass at each position independently of the observed token.

3) Sequence-Level Aggregation:

Token-level scores are averaged across each hypothesis span to produce one scalar per candidate, with unnormalized sums reported as a length-sensitivity check. Candidates are ranked by ascending lower-is-better score, and performance is measured by the fraction of papers where the gold hypothesis ranks first.

  • Sequence-Level Aggregation:: Token-level scores are aggregated by averaging over each hypothesis span, yielding one scalar score per candidate hypothesis.The unnormalized sum over the hypothesis is also reported, while the mean is the primary score.
  • Sequence-Level Aggregation:: The mean score is primary, with unnormalized sums used to test sensitivity to hypothesis length.Both SNLL and SRaw are lower-is-better.
  • Sequence-Level Aggregation:: For each paper, all sixteen candidates are ranked in ascending score, and the gold hypothesis receives rank r_p.Both SNLL and SRaw use this lower-is-better ranking procedure.
  • Sequence-Level Aggregation:: The top-line metric is the fraction of papers where the gold hypothesis ranks first, r_p = 1.A model achieves this outcome when it reliably assigns the gold hypothesis the lowest energy.

D. Scoring Pipeline · E. Proprietary LLM-as-Judge

The scoring pipeline isolates each candidate’s hypothesis span and deterministically evaluates it in one forward pass, while the proprietary model is assessed through a zero-shot top-5 selection prompt because token-level logits are unavailable. The prompted baseline considers all 16 candidates but returns only the five judged most plausible.

  • D. Scoring Pipeline: The pipeline tokenizes the prefix separately from the full sequence to isolate each hypothesis span.Shared background and the research question therefore do not contribute to any candidate’s score.
  • D. Scoring Pipeline: A candidate requires one forward pass with no sampling or additional model calls, making scoring deterministic given the model weights.Evaluating a paper scales linearly with the number of candidates, which is 16.
  • E. Proprietary LLM-as-Judge: The proprietary instruction-tuned model is accessible only through an API, so token-level logits are unavailable for the logit-based pipeline.It is used as a comparison baseline instead.
  • E. Proprietary LLM-as-Judge: The baseline uses a single zero-shot prompt for each paper and asks the model to identify its top 5 candidates.Selecting a plausible subset is treated as simpler and more tractable than exhaustively ranking all 16 candidates.
  • E. Proprietary LLM-as-Judge: The prompt presents the background survey, research question, and an indexed list of all 16 candidate hypotheses.These components provide the context and candidate pool for comparative judgment.
  • E. Proprietary LLM-as-Judge: The instructions request ranking all 16 hypotheses from most to least plausible but require returning only a JSON object containing the top five indices.The required indices are integers from 0 to 15, and no extra text is allowed.

IV. EXPERIMENTAL SETUP · A. Dataset

The study uses ResearchBench to frame scientific hypothesis identification as a within-paper ranking task. The dataset contains 1,323 papers across 12 disciplines, each with one gold hypothesis among 16 candidates.

  • A. Dataset: ResearchBench comprises 1,323 papers spanning 12 disciplines.
  • A. Dataset: Each dataset instance corresponds to a single published paper.
  • A. Dataset: Every instance includes a background survey summarizing prior literature and study-motivating context.
  • A. Dataset: Every instance includes a research question posed by the paper.
  • A. Dataset: Each instance contains 16 candidate hypotheses, exactly one of which is the gold hypothesis.
  • A. Dataset: Hypothesis identification is formulated as within-paper ranking using the background survey and research question.
  • A. Dataset: Each LLM produces a complete ranking of the candidate hypotheses, and performance depends on placing the gold hypothesis above the alternatives.
  • A. Dataset: The evaluated-models table records knowledge cutoff dates, parameter counts, release dates, and license terms.

B. Compute Setup · C. Evaluation Metrics · V. RESULTS

The evaluation ranks each gold hypothesis within a sixteen-candidate pool across 1,323 papers. Results compare NLL and Raw energy scoring for open-weight models, alongside proprietary zero-shot prompted ranking.

  • B. Compute Setup: The study reports results for 8 language models spanning roughly one to twenty billion parameters, scored using both NLL and Raw energy criteria.
  • C. Evaluation Metrics: Each paper and language model yields the gold hypothesis rank within the full sixteen-candidate pool, with metrics computed across P = 1,323 papers.
  • C. Evaluation Metrics: Hit@k measures the fraction of papers whose gold hypothesis ranks at or above position k for k ∈{1, 2, 3, 5}.
  • C. Evaluation Metrics: Hit@1 identifies papers where the gold hypothesis is ranked as the single most plausible candidate.
  • C. Evaluation Metrics: Mean Reciprocal Rank averages the reciprocal of the gold hypothesis’s rank across papers.
  • V. RESULTS: Figure 3 compares Hit@k for NLL and energy-based scoring across open-weight models, with the proprietary model shown as a dashed zero-shot prompted-ranking baseline.
  • V. RESULTS: Figure 3 evaluates gold-hypothesis top-k ranking for k ∈{1, 2, 3, 5} across N = 1,323 papers using a shared y-axis scale.

A. Overall Hypothesis-Identification Performance · B. Per-Discipline Performance

Across 1,323 papers, likelihood-based scoring generally outperformed prompted GPT-5 ranking, with the strongest observed result from Llama 3.2 1B under Raw scoring. Across disciplines, likelihood-based methods were best in most cases, while apparent GPT-5 advantages were statistically indistinguishable.

  • A. Overall Hypothesis-Identification Performance: Raw energy substantially improved over NLL for Llama 3.2 1B, Llama 3.2 3B, and Mistral 7B, but degraded performance for Gemma 2 2B.GPT OSS 20B, Gemma 4 12B, and Phi-4 were roughly comparable under Raw and NLL.
  • A. Overall Hypothesis-Identification Performance: 0.5314 Hit@1 was the highest single result, achieved by Llama 3.2 1B under Raw scoring with a 95% interval of [0.504, 0.558].This maximum came from 14 model-by-scorer combinations selected after observing the results and should be interpreted as an upper bound.
  • A. Overall Hypothesis-Identification Performance: 0.3315 vs. 0.3276 Hit@1 and 0.4742 vs. 0.4622 MRR indicate indistinguishable Raw and NLL criteria overall.The differences fall within the ±0.026 margin of a single Hit@1 estimate at N = 1,323, while model-level variance is much larger.
  • A. Overall Hypothesis-Identification Performance: Likelihood-based scoring was generally stronger than zero-shot listwise prompting for identifying the true hypothesis across all papers and disciplines.Table II aggregates Hit@k and MRR across 1,323 papers; GPT-5 reports top-5 ranks and omits MRR.
  • A. Overall Hypothesis-Identification Performance: 0.1663 Hit@1 and 0.4172 Hit@5 under GPT-5 zero-shot listwise prompting fell below every NLL-scored model and every Raw-scored model except Gemma 2 2B.Explicitly ranking all 16 candidates in one pass was less reliable than measuring hypothesis probability with smaller open-weight models.
  • A. Overall Hypothesis-Identification Performance: Llama 3.2 1B and 3B produced the strongest results despite training data ending in December 2023, before the evaluated papers were published.Models with later cutoffs performed worse, although this pattern does not rule out memorization.
  • B. Per-Discipline Performance: Likelihood-based scoring yielded the best Hit@1 and Hit@5 in the large majority of disciplines.Table III compares pooled NLL and Raw results across seven open-weight models with GPT-5 alone.
  • B. Per-Discipline Performance: GPT-5 was nominally best in Physics and Material Science, but per-discipline samples of approximately 110 to 125 made those differences statistically indistinguishable.The authors present this pattern as a hypothesis for future work rather than a finding.

C. Pairwise Comparison of Correct & Competing Hypotheses … A. Mechanistic Signal May Outperform Explicit Prompting

Across pairwise and cross-model analyses, likelihood-based scoring generally identifies correct scientific hypotheses more reliably and consistently than prompted listwise ranking. The strongest result comes from a small open-weight model, highlighting that intrinsic confidence may outperform explicit judgment without scaling predictably with model size.

  • C. Pairwise Comparison of Correct & Competing Hypotheses: 69%–74%: likelihood-based scoring remains stable across models regardless of model scale.A random ordering would score 50% in the pairwise comparison setup.
  • C. Pairwise Comparison of Correct & Competing Hypotheses: 82.2%: Llama 3.2 1B using raw target-logit energy scoring correctly outranks competing hypotheses in pairwise comparisons.Llama 3.2 3B also reaches 79.1%, while Gemma 2 2B scores 45.8%, indistinguishable from chance.
  • D. Cross-Model Agreement: 12.6%: all seven open-weight models rank the gold hypothesis first for 167 of 1,323 papers under NLL scoring.Under Raw scoring, all seven agree for only 3 of 1,323 papers (0.2%), indicating greater cross-model consistency for NLL rankings.
  • D. Cross-Model Agreement: 41.7%: GPT-5 places the gold hypothesis in its explicit top five for 552 of 1,323 papers.Because candidates outside the top five are tied by construction, Hit@5 is the most directly interpretable statistic for prompted ranking.
  • VI. DISCUSSION: Likelihood-based scoring substantially outperforms zero-shot listwise prompting at identifying the correct hypothesis from sixteen candidates.The strongest single configuration is a one-billion-parameter open-weight model, rather than the largest or most capable model.
  • A. Mechanistic Signal May Outperform Explicit Prompting: Intrinsic NLL and Raw scores use each model’s next-token distribution over the hypothesis conditioned on the background and research question, without requesting an explicit ranking.This provides a mechanistic signal that can rival or exceed direct instruction of a much larger model.
  • A. Mechanistic Signal May Outperform Explicit Prompting: The study’s key novelty is comparing models against their own internal likelihoods rather than only comparing prompted model judgments.For choosing between a specific hypothesis and plausible alternatives, the authors argue that direct likelihoods may be preferable to asking models to explain uncertainty.
  • A. Mechanistic Signal May Outperform Explicit Prompting: The best performance by the smallest model does not establish that model size is unimportant, because training, data exposure, and probability calibration were not directly measured.The authors treat the factors explaining the result as an open question.

B. Limitations · C. Future Work · VII. CONCLUSION

The study is an initial rather than definitive evaluation, because its benchmark identifies known published hypotheses rather than testing truly open-ended scientific reasoning. Future work will evaluate unpublished hypotheses with expert rankings, while the conclusion argues that intrinsic likelihood-based scoring merits further attention.

  • B. Limitations: The study is an initial step, not a definitive evaluation of energy-based scoring, prompted ranking, or their relative strengths.The authors caution that several limitations should be considered when interpreting the results.
  • B. Limitations: The benchmark uses published hypotheses and candidates curated after publication, making the task closer to identifying a known hypothesis than evaluating open-ended scientific reasoning.This design limits how directly the benchmark represents genuinely novel scientific hypothesis evaluation.
  • C. Future Work: Future work will build a benchmark of open research questions paired with unpublished candidate hypotheses in collaboration with experts across scientific fields.Experts will independently rank the hypotheses based on scientific judgment.
  • C. Future Work: Expert rankings will enable comparison with energy-based and prompt-based methods, measurement of inter-annotator agreement, and derivation of additional evaluation measures.The planned benchmark is intended to evaluate ranking on genuinely novel scientific hypotheses.
  • C. Future Work: Open research questions require agreement-based evaluation, such as Kendall’s τ or Spearman’s ρ, rather than accuracy or Hit@k retrieval metrics.Because open research questions lack a single agreed-upon answer, the protocol should compare model and expert rankings.
  • VII. CONCLUSION: Intrinsic likelihood-based scoring, particularly an energy-based criterion, matches or exceeds prompting a much larger proprietary model for aggregate candidate-hypothesis ranking.The conclusion presents this as evidence that mechanistic scoring signals deserve more attention relative to prompting-based approaches.
  • VII. CONCLUSION: Per-discipline differences were too small at the available sample sizes to support paradigm-specific claims.The conclusion therefore frames the aggregate comparison as suggestive rather than definitive across disciplines.
Loading 2608.17270v1…