PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Information Is Worth Asking For: Decision-Informed Information Seeking with LLMs

Seunghan Lee, Jun Seo, Junhyeok Kang, Jaehoon Lee, Sangjun Han, Sungdong Yoo, Minjae Kim, Tae Yoon Lim, Dongwan Kang, Hwanil Choi, Soonyoung Lee, Wonbin Ahn

arXiv:2610.32181v1cs.AIcs.CLcs.LG

TL;DR

LLMs often acquire evidence whose decision value does not justify its cost and continue asking after evidence suffices. DIVA values checks by their expected effect on the decision net of cost, and DIVA-U accounts for uncertain check strengths. Across 20 LLMs and 12 environments, DIVA-U has lower first-decision regret than every LLM baseline.

  • Problem

    Existing question-asking and information-gain approaches do not value information by whether it can change a decision enough to justify its cost.

    Source: §1 · p. 1

  • Method

    DIVA uses an LLM-specified decision model to value imagined check outcomes net of cost and stop when no check is worthwhile; DIVA-U accounts for uncertainty in check strengths.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    Across 20 LLMs, DIVA-U has lower first-decision regret than every LLM baseline in all twelve environments, mainly by deciding when to stop.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The findings support using decision value, rather than information alone, as the goal of evidence-gathering LLM assistants.

    Source: §6 · p. 8

    DIVA assumes checks are independent given the state, which can over-count correlated evidence, and DIVA-U mitigates but does not remove limits from imperfect LLM knowledge of check informativeness.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used to support decisions under incomplete information, where deciding what to look up and when to stop matters as much as the final answer. However, LLMs fall into an information trap: They pay for checks, i.e., costly pieces of evidence, that cannot change their decision or are not worth their cost, and they keep asking when the evidence is already sufficient. To this end, we propose Decision-Informed Value-of-information Acquisition (DIVA), a simple yet effective procedure for LLM information seeking based on the decision value of evidence. DIVA 1) lets the LLM specify a small decision model, i.e., how likely a bad outcome is and how each check would shift this probability, through short numeric questions and 2) values each candidate check by how much its imagined outcomes improve the decision net of its cost, stopping when no check is worth its cost. Its variant, DIVA-U, further accounts for the LLM's uncertain estimates of how strongly each check shifts this probability. DIVA is training-free and applies to any LLM. Across four domains and 20 LLMs, DIVA-U has lower regret, i.e., value lost relative to the best possible decision, than every LLM baseline on real-world data, mainly by stopping once evidence suffices, and DIVA cuts the regret of the best LLM baseline by 75% in synthetic scenarios where every check's reliability is given. Code: github.com/seunghan96/diva.

1 Introduction

LLMs must decide both what evidence to acquire and when to stop, but often value information rather than its effect on the decision. DIVA uses a small LLM-specified decision model to value checks net of cost and stop when none is worthwhile.

1 Introduction

  • DIVA meets both desiderata—acquiring checks that can change decisions net of cost and stopping when no check is worth its cost—unlike prompted LLMs and information gain.Source: Figure 1 · p. 1
  • The paper identifies an information-seeking gap: prior approaches ask clarifying questions, identify missing information, or select by expected information gain, rather than value to the decision.A check is worth acquiring only when expected decision improvement exceeds its cost.Source: §1 · p. 1
  • LLMs pay for checks whose decision value does not cover their cost and often keep acquiring after evidence suffices, with most regret attributed to this information trap.On real credit and patient records, most LLM baselines have several times the regret of never acquiring evidence.Source: §1 · pp. 1–2
  • DIVA asks the LLM to specify a small decision model, imagines check outcomes, values expected decision improvement net of cost, and stops when no check is worthwhile.DIVA-U averages decision value over plausible check strengths when those strengths are uncertain.Source: §1 · p. 2
  • Across 20 LLMs, DIVA-U has the lowest regret in all 12 decision environments, with analyses of the information trap, cost and prompt robustness, and cross-domain transfer.The environments span four domains and include real-record and controlled synthetic settings.Source: §1 · pp. 2–3

2 Related Works

Related work spans question asking, information-gain selection, decision-aware acquisition, and LLMs as probabilistic-model specifiers. This paper evaluates the decision value of each check and when to stop.

2 Related Works

  • LLM information-seeking studies evaluate asking or identifying missing information across clinical, underspecified, game, and finance settings; this work instead evaluates each check’s decision value and when to stop.Examples include selective clarification with LLMs (Kuhn et al., 2023) and benchmarks for identifying missing information.Source: §2 · p. 3
  • Expected information gain guides Bayesian experimental design and recent LLM question-asking methods, while DIVA values a check by its effect on the decision.Related approaches include entropy search, value-of-information prompting, cost-aware stopping, and sequential diagnosis or active feature acquisition.Source: §2 · p. 3
  • Prior work uses LLMs to state priors and likelihoods for probabilistic pipelines, whereas this paper uses an elicited model to imagine check outcomes and decide what to ask and when to stop.Elicited probabilities are often imperfect (Xiong et al., 2024; Paruchuri et al., 2024).Source: §2 · p. 3
  • High-stakes decision studies commonly answer from available evidence, whereas this work studies which evidence to acquire before deciding using point-in-time filings.Point-in-time filings guard against look-ahead bias (Lee et al., 2026a).Source: §2 · p. 3

3 Methodology

DIVA lets an LLM choose what evidence to acquire and when to stop by evaluating how possible check outcomes affect the decision. It separates LLM-specified probabilities from Bayesian updating and decision valuation.

3.1 Problem Setup

  • A decision case has a binary adverse state, two possible actions, payoffs, and candidate checks with costs and state-conditional outcome reliabilities.A policy acquires a check or stops and acts; a check is worthwhile only when its expected decision improvement exceeds its cost.Source: §3.1 · pp. 3–4

3.2 Why Asking the LLM Directly Fails

  • Directly elicited belief updates are incoherent: across 20 LLMs, their correlation with exact Bayesian changes is at most 0.44, and the top check matches the best one barely more often than chance.The imagined beliefs violate the law of total probability by 6.3 percentage points on average.Source: §3.2 · p. 4

3.3 DIVA: Imagine, Value, Stop

  • DIVA elicits a small decision model, updates imagined check outcomes with Bayes’ rule, values each check net of cost, and stops when none has positive net value.The LLM specifies the model through short numeric questions; the remaining valuation and stopping steps repeat without further LLM calls.Source: §3.3 · p. 5
  • DIVA-DP replaces next-check valuation with exact dynamic programming, correctly valuing a cheap check whose value arises only in combination with later checks.The variant requires no extra parameter.Source: §3.3 · p. 5

3.4 DIVA-U: Uncertainty-Aware Valuation

  • DIVA-U treats check strength as uncertain and averages net decision value over plausible shifts, allowing checks valued at zero by DIVA to receive positive value.This averaging is deliberately optimistic rather than the exact decision value under uncertainty, so DIVA-U acquires more often than DIVA.Source: §3.4 · p. 5

4 Experiments

The experiments span real-record settings without stated check reliabilities and controlled synthetic settings where reliabilities are given. DIVA and DIVA-U outperform LLM baselines in first decisions and full episodes across the reported environments.

4.1 Experimental Settings

  • The study evaluates 20 open-weight LLMs against five baseline categories across knowledge-required environments with unstated check reliabilities and four controlled synthetic diagnostic domains with stated reliabilities and exact decision values.Regret is value lost relative to an optimal decision, normalized by loss l and multiplied by 100; episode regret compares a full acquire-then-decide episode with the optimal policy and is normalized by g+l in controlled environments.Source: §4.1 · p. 6

4.2 Main Results

  • 75%: DIVA reduces the best LLM baseline’s regret on average in controlled environments and beats each LLM’s best baseline in both environment types.Figure 5 compares regret for each LLM’s best baseline with DIVA, or DIVA-U in knowledge-required environments.Source: Figure 5 · p. 7, §4.2 · p. 7

4.3 End-to-End Episodes

  • About one sixth: DIVA has the regret of the best LLM baseline in controlled full episodes, beating every LLM baseline across domains and models.In knowledge-required episodes, DIVA-U has the lowest regret on each tabular record; every LLM baseline has higher average regret than never acquiring, and DIVA-U falls behind it only on SEC filings.Source: Table 4 · p. 7, §4.3 · p. 7

5 Analysis

The analyses attribute much LLM regret to information-seeking errors and show that cost-aware valuation, planning, and reliability knowledge affect performance in different settings.

Ablation study

  • 5.03 vs. 0.83: replacing DIVA’s decision value with cost-blind EIG erases most of the gain, while adding cost to EIG recovers it.Using ground-truth likelihoods lowers DIVA’s regret from 0.83 to 0.31, indicating that most remaining error comes from the LLM’s answers.Source: §5 · p. 7, Table 5 · p. 8

Anatomy of the information trap

  • Over 70% of regret for six of nine LLM baselines comes from the three trap errors in both settings, with uninformative checks the largest part.VoI-Ask avoids the trap but premature stopping causes over 80% of its regret; DIVA and its variants have the smallest regret in both settings.Source: Figure 6 · p. 8, §5 · p. 8

Myopic and planned acquisition

  • 0.65 vs. 1.07: without stated reliabilities, DIVA-U beats planning at the first decision but trails DIVA-DP and VoI-Ask over full episodes.Planning lowers DIVA’s regret in both environment types at the first decision and over full episodes, indicating that some remaining regret stems from myopic valuation.Source: Table 6 · p. 8, §5 · p. 8

What the LLM’s knowledge contributes

  • Without LLM input, DIVA-U has lower regret than the best LLM baseline and never acquiring in every column; LLM knowledge helps when reliabilities are stated.When reliabilities are not stated, LLM strengths do not lower regret versus one shared strength, limiting the gain from the LLM’s check knowledge.Source: Table 7 · p. 8, §5 · p. 8

Other analyses

  • DIVA-U beats DeLLMa and UoT for every LLM and environment, and with ±50% payoff and cost errors it beats every baseline in seven of eight knowledge-required environments.The comparisons include DeLLMa (Liu et al., 2025) and UoT.Source: §5 · p. 8

6 Conclusion

DIVA uses a small decision model to value evidence by its expected effect on decisions net of cost, with DIVA-U as its uncertainty-aware variant. Across 20 LLMs, DIVA-U has lower first-decision regret than every LLM baseline in all twelve environments, mainly by deciding when to stop.

6 Conclusion

  • DIVA asks the LLM for a small decision model and computes how much each check can improve the decision net of its cost.The paper also introduces DIVA-U as an uncertainty-aware variant.Source: §6 · p. 8
  • Across 20 LLMs, DIVA-U has lower first-decision regret than every LLM baseline in all twelve environments, mainly by deciding when to stop.Source: §6 · p. 8
  • The paper’s stated takeaway is to make decision value, rather than information, the goal of evidence-gathering LLM assistants.Source: §6 · p. 8

Limitations

The paper identifies assumptions and evaluation boundaries that constrain DIVA’s supported scope.

Limitations

  • DIVA assumes task specifications provide decision-maker payoffs; eliciting payoffs from LLMs is error-prone in the reported multi-state experiments.Source: Limitations · p. 9
  • DIVA assumes checks are conditionally independent given the state, which can over-count correlated evidence; DIVA-U mitigates but does not remove dependence on LLM knowledge.Source: Limitations · p. 9
  • Knowledge-required gains use a reference model fitted on historical records, inherit its errors, and do not reliably predict gains in realized utility.SEC payoffs place the threshold near the reference belief, favoring acquisition; full-episode gains also rely on the reference model’s belief for the final action.Source: Limitations · p. 9
  • DIVA-U is designed for the first decision and over-acquires in multi-step episodes because uncertainty about elicited strengths does not shrink with observations.Source: Limitations · p. 9
  • Evaluation covers open-weight LLMs up to 72B, binary or binned checks in English, and SEC filings from S&P 500 firms; free-text evidence remains unevaluated.Source: Limitations · p. 9

Ethical Considerations

The paper frames DIVA as decision support for evidence gathering, not autonomous decision-making, and places it alongside work on information seeking, information gain, and decision-theoretic test selection.

Ethical Considerations

  • DIVA is intended for research benchmarks, not validated deployment settings; credit and clinical records include demographic candidate checks, but the study does not evaluate decision fairness.The records are de-identified public data; SEC environments combine public filings with licensed stock prices.Source: Ethical Considerations · p. 9

A Further Related Works

B Environments and Data

The paper evaluates evidence-gathering policies in knowledge-required and controlled environments, then details DIVA’s elicitation and valuation procedure and its implementation.

B Environments and Data

  • Knowledge-required environments use credit and clinical records or SEC earnings releases; controlled scenarios state check reliabilities and include information traps and already-sufficient cases.Controlled scenarios test whether policies use stated numbers correctly.Source: B Environments and Data · pp. 14–15

B.1 Knowledge-Required Environments

  • Credit and clinical tasks use public datasets with binarized features, while SEC tasks use masked S&P 500 firm-quarter filings and market-adjusted return or volatility outcomes.The SEC reference model is trained on 2015–2022 filings and calibrated on 2023; test announcements are from 2025–2026.Source: §B.1 · p. 15

B.2 Controlled Environments

  • Controlled binary cases vary base rates and check reliability across six check types, while multi-state cases use three hidden factors, three actions, and checks that reveal factor outcomes.Some multi-state checks are informative but have zero decision value because their factor does not affect payoffs.Source: §B.2 · pp. 15–16

C.1 Checks, Costs, and Elicitation

  • Candidate checks form a closed menu, and acquiring a check incurs the stated cost; the policy may stop rather than inventing questions.DIVA reads the scenario’s check costs and payoff table rather than eliciting them.Source: §C.1 · p. 16
  • DIVA elicits a base rate and check risks through short numeric questions, then updates beliefs with Bayes’ rule after observed findings.For outcome-conditioned risks, DIVA asks three questions per check or known finding; elicitation calls are made once before acquisition.Source: §C.1 · p. 16

C.3 Pseudocode

  • The algorithm specifies a decision model once, repeatedly values remaining checks, acquires the highest-valued check, and stops when no check has positive net value.DIVA uses a point estimate of check strength; DIVA-U averages over uncertainty in that strength.Source: §C.2 · p. 18

C.2 Configuration

  • The selected configuration uses outcome-conditioned risks with shrinkage; DIVA-U averages decision value over plausible check strengths, but per-bin SEC risk questions often collapse to the base rate.The reported selection was made on development data; DIVA-U’s averaging can assign positive value to weak checks, with cost limiting acquisition.Source: §C.2 · pp. 18–19

C.4 Models and Hardware

  • The evaluated LLMs are served with vLLM on NVIDIA L40 or L40S GPUs, while reference models and dynamic-programming optima run on CPUs without LLMs.Greedy decoding is used; three large models use tensor parallelism across two GPUs.Source: §C.4 · p. 19

D.1 Evaluation Metrics

Across episode evaluations, DIVA-U outperforms LLM baselines, while analyses examine stopping, check selection, uncertainty, and transfer across environments. In multi-state settings, DIVA also has lower average regret than the compared policies.

D.1 Evaluation Metrics

  • DP regret compares decisions with dynamic-programming or myopic optima, and episode regret measures policy utility net of acquisition costs across the episode.The evaluation also reports over-acquisition and information-trap rates.Source: §D.1 · p. 19

D.2 Baseline Methods; D.3 Table Conventions

  • The baselines include direct prompting, information-gain methods, and program-aided approaches; reported regret is multiplied by 100, with lower values better.Tuned methods select cost trade-offs or thresholds on development scenarios, whereas DIVA and DIVA-U use a single configuration.Source: §D.2 · p. 20, §D.3 · p. 21

E.2 Full Episodes

  • DIVA-U beats every LLM baseline for all 20 LLMs, while DIVA-DP has lower episode regret than never acquiring in all eight knowledge-required environments and 119 of 144 LLM–environment cells.On SEC filings, DIVA-U’s fixed uncertainty can cause over-acquisition and leave it behind never acquiring.Source: Table 16 · p. 22, §E.2 · p. 23

E.1 Acquisition and Stopping Behavior

  • DIVA is the only policy with low over- and under-acquisition rates; LLM baselines and fixed-threshold EIG often acquire even when stopping is optimal.On cases where stopping is optimal, DIVA acquires in only a few cases; uninformative checks account for 43% of DIVA-U’s remaining tabular regret and 24% of DIVA’s.Source: §E.1 · pp. 21–22, Figure 8 · p. 22

E.3 What Drives the Gains

  • Shuffling elicited likelihoods or assigning all checks one strength retains much of DIVA’s tabular-record gains, though unshuffled elicited strengths improve results in 40 of 80 cells.With check directions set by design, generic DIVA-U beats DIVA-U in seven of eight environments; stated reliabilities instead make LLM strengths helpful.Source: §E.3 · p. 23
  • Fixed-threshold EIG has higher regret than DIVA in every knowledge-required environment, while tuned EIG can outperform DIVA on some targets; DIVA-U has the lowest average regret.Leave-one-source-out selection chooses λ = 0.5 and σ = 1 for every held-out source, with lower average regret than the λ = 0.75 configuration and DIVA without uncertainty.Source: §E.3 · pp. 23–24

E.4 Multi-State Environments

  • On held-out multi-state scenarios, DIVA has lower average regret than the direct policy, never acquiring, and Always ask; replacing decision value with EIG or entropy search increases regret.Eliciting payoffs from the LLM makes DIVA no better than the direct policy, indicating that the payoffs must also be correct.Source: §E.4 · p. 24

E.5 Other Metrics

The additional analyses assess realized utility, alternative reference models, stopping-rule contributions, and comparisons with added baselines.

E.5 Other Metrics

  • Fresh scenarios are drawn after design decisions were fixed, using unused test records for Taiwan and CDC and fresh scenario draws for German and Heart.SEC evaluation uses test scenarios beyond the first 1,000 of each target.Source: §E.5 · pp. 24–25, Table 26 · p. 26
  • On real-record replays, DIVA and DIVA-U stay close to never acquiring without a reliable gain, while the direct policy realizes far lower utility.The reference model’s DP policy also does not significantly beat never acquiring on three of four datasets in the original scenarios.Source: §E.5 · p. 24, Table 22 · p. 25
  • Under two alternative reference models, DIVA-U has lower regret than every baseline in all eight environments under one model and beats never acquiring in all but two LLM–dataset cells under the other.The reference models have lower Brier scores than a constant base-rate prediction; calibration errors are largest on the two volatility targets.Source: §E.5 · p. 24, Table 24 · p. 26

E.6 Further Analyses

  • Giving the direct policy DIVA-U’s stopping rule lowers regret more than giving it DIVA-U’s check choice, except on SEC filings, where both lower it equally.Across stopping rules, DIVA-U’s selected check has lower regret than the LLM’s choice and the largest-EIG check.Source: §E.6 · p. 25, Table 27 · p. 27

F.1 Additional Baselines

  • DIVA-U beats DeLLMa and UoT for every LLM in every environment, and beats most other added baselines in all twelve environments.The cost-aware EIG rule is an exception; DIVA-U has lower regret than it in seven of eight knowledge-required environments.Source: Table 28 · p. 27, §F.1 · pp. 27–28
  • The cost-aware EIG comparison indicates that cost-aware stopping on the elicited decision model carries much of DIVA’s gain.This pattern is consistent with the stopping-rule comparison in Table 27.Source: §E.6 · p. 25, §F.1 · p. 28

F.2 Further Controls

The further controls examine how DIVA and DIVA-U behave under altered payoffs, misspecified inputs, and uncertainty, alongside held-out results, statistical tests, prompt paraphrases, and a qualitative example.

F.2 Further Controls

  • With ±50% errors in payoffs and costs, DIVA-U still beats every LLM baseline in seven of eight knowledge-required environments, all but Vol-20d.DIVA’s and DIVA-U’s regret rises only moderately under the tested errors.Source: §F.2 · p. 28
  • On SEC payoffs drawn far from the reference belief, DIVA comes close to never acquiring, while DIVA-U keeps acquiring and beats it on both return targets.With uniform payoffs, stopping is DP-optimal in most scenarios; DIVA-U stops less often but beats never acquiring and Always ask on every target.Source: §F.2 · p. 28, Table 32 · p. 29
  • Across full episodes, DIVA-PV has the lowest average regret in the comparison, while DIVA-U’s first-decision gain comes from an upward correction that also causes over-acquisition.DIVA-PV averages likelihoods before computing decision value; its value is never larger than DIVA-U’s averaged decision value.Source: §F.2 · p. 28

F.3 Held-Out SEC Filings

  • On held-out SEC volatility targets, DIVA-U has the lowest DP regret of all policies, while the direct policy has the lowest myopic regret.Beyond the first 1,000 scenarios per target, DIVA-U has lower DP regret than DIVA on three targets and equal regret on Ret-5d.Source: §F.3 · p. 29, Table 31 · p. 29
  • On uninspected SEC filings, DIVA-U has lower regret than DIVA in 68 of 80 LLM–target cells and than never acquiring in 79.The tabular-record results re-score inspected decision models and are post hoc; the DIVA-DP hypotheses on test scenarios 300–999 hold for the 18 pre-registered LLMs.Source: §F.3 · p. 29

F.4 Statistical Tests

  • DIVA-U’s wins over every LLM baseline are significant for most LLMs on tabular records and for fewer LLMs on SEC targets.The statistical tests use paired DP-regret differences and 95% confidence intervals from a clustered bootstrap over scenario groups with 1,000 resamples.Source: §F.4 · p. 29, Table 33 · p. 29

F.5 Prompt Paraphrases

  • Across prompts, every baseline has more than four times DIVA’s regret under any prompt, and prompt wording does not cause the gap.Paraphrases change average regret little, while baselines vary more across prompts per LLM than DIVA.Source: §F.5 · p. 30, Table 34 · p. 30

G Qualitative Examples

  • In the controlled finance example, DIVA stops because no check can change the decision, while the direct policy requests the most informative check.DIVA’s stop is DP-optimal in this scenario.Source: G Qualitative Examples · p. 30, Figure 9 · p. 30

H Full Results

Per-LLM results show DIVA-U beats every LLM baseline on Taiwan, German, and Heart, with its few losses lying mostly on volatility targets.

H Full Results

Related papers

Loading 2610.32181v1…