PaperlayerView exact revision on arXiv

Source-linked AI summary

Right Numbers, Wrong Questions: Evaluating LLMs on Underspecified Financial Queries

Radha Yadav, Prasanna Devadiga, Peeyush Mishra, Apoorv Singh

arXiv:2610.38631v1cs.AIcs.CLq-fin.GN

TL;DR

Financial questions can leave material details about figures, product variants, or investors unstated, allowing an assistant to return a real number for a different reading. The paper studies these failures in Indian mutual funds using computed references and tests how models interpret ambiguous context. Models often followed labels, layout, or defaults, and evidence checks could still pass answers that silently chose another reading.

  • Problem

    Financial questions can omit what a figure measures, which product variant applies, or investor and rule details, leaving multiple defensible readings.

    Source: §1 · p. 1

  • Method

    The paper tests three underspecification challenges in Indian mutual funds, using computed references for each reading and additional tests of judging and prevention.

    Source: §1 · p. 1

  • Results

    Models often used ambiguously labelled figures, let column order determine which labelled plan came first, or filled missing investor facts with defaults; evidence checks could pass answers that chose another reading.

    Source: §1 · pp. 1–2, §8 · p. 8

  • Takeaways & Limitations

    Benchmarks should score whether models establish the reading, and judges should be tested without being given the intended reading.

    Source: §8 · p. 9

    The studies cover one jurisdiction and product family and do not estimate how often real users leave dimensions open.

    Source: §7 · p. 8

Abstract

from arXiv · show

Financial assistants answer questions that sound factual but depend on what the user leaves unsaid: what a data figure measures, which product variant is meant, and the investor's situation and the rules that apply. An assistant that ignores these can still answer with a real number from its data, one that answers a different question. We call this frame collapse. We study it in Indian mutual funds, where product variants and tax rules are formally defined, so the correct answer can be computed for every reading. One study tests each challenge. Given an ambiguously labelled figure, models used it, often under the user's label. Given every alternative labelled, they listed them all, but the layout decided which came first. Missing a fact about the investor, weaker agents assumed a default or an outdated tax rule; requiring the fact did not guarantee its use. Because such answers match their data, checks against the data pass them. An AI judge caught mislabelled figures only when told what each field means, and still passed answers that silently chose another reading. We show how these failures arise, not how often, and propose controls for financial assistants.

1 Introduction

Financial questions depend on unstated details about what a figure measures, which product variant applies, and the investor and governing rules. The studies show how models can answer under an unsettled interpretation, while the paper proposes a taxonomy, computed references, and controls for evaluating and preventing such failures.

1 Introduction

  • A real, traceable figure can answer a different question when the assistant commits to an interpretation without establishing or stating it; the paper calls this frame collapse.Because the figure matches the evidence, an evidence-support check can pass it.Source: §1 · p. 1
  • The paper studies these challenges in Indian mutual funds, where defined products and tax rules let researchers compute the correct answer for each reading.It contributes a taxonomy, three studies with computed references, evidence on judges and prevention checks, and proposed assistant controls.Source: §1 · pp. 1–2
  • When a field’s name did not specify its measure, models used its figure, often giving it the user’s label.Seven of eight models used the ambiguously named figure; with every candidate labelled, they rarely erred.Source: §1 · pp. 1–2
  • With both labelled plans shown, most current models listed both, but column order decided which plan came first.The two plans share a portfolio but have different fees and returns.Source: §1 · pp. 1–2
  • When the deciding investor fact was in a profile tool, agents sometimes assumed residency or used an outdated tax rule; requiring the fact did not always ensure its use.The newest models gathered the fact unprompted.Source: §1 · pp. 1–2

2 The Problem: What Must Be Settled Before Answering

A financial question has an appropriate answer only after its material dimensions are settled. The section classifies those dimensions and specifies how answers should handle unresolved readings.

2 The Problem: What Must Be Settled Before Answering

  • A question can have five defensible readings, each with a different correct figure, so the answer depends on which reading is intended.For example, the IDCW figure is an investor’s cash-flow return, not a fund return, and exceeds Growth even though the investor ends with less.Source: §2 · p. 2, Table 1 · p. 2
  • The taxonomy groups fourteen dimensions, including what is measured, how, against what, and in which context; it was compiled from regulatory definitions and practice, not real queries.A dimension is material when changing it moves the answer beyond the reporting tolerance.Source: §2 · p. 2
  • When a material dimension is unknown, answer using known values or state the basis and alternatives; ask only when no basis exists, and decline when the data cannot answer.For investor-specific dimensions, gather the facts, apply the rule, and use the version in force.Source: §2 · p. 2
  • The failure taxonomy distinguishes three forms of frame collapse from headline prioritisation, where both labelled figures appear but layout determines their order.The studies score these forms differently, so they do not produce one failure rate.Source: §2 · p. 3

3 Related Work

Prior work studies ambiguity, clarification, context sensitivity, and answer checking, but most financial benchmarks fully specify questions. This paper adds computed references for each reading in a regulated domain and tests where interpretations come from.

3 Related Work

  • Most financial benchmarks fully specify questions; this work instead uses computed references for each reading and scores the basis an answer states.Related work includes benchmarks on clarification and conditional or time-sensitive questions, plus FinInteract’s default-versus-intended readings.Source: §3 · p. 3
  • Prior studies examine position and format effects, tool use, and models’ acceptance of presuppositions; this paper’s context is true under another interpretation, not misleading.Models may also possess knowledge they do not use.Source: §3 · p. 3
  • Evidence-support checks and LLM judges can pass an answer supported by its evidence even when it reflects a collapsed frame.The paper tests what additional information a judge needs, drawing on field descriptions from text-to-SQL.Source: §3 · p. 3

4 Study Design

The study design varies what reaches the model’s context across three financial-query challenges, then reuses Study 1 answers to test judging and prevention. Correctness and references are measured with fixed code or a codebook, with stated limits on coding and samples.

4 Study Design

  • Each study changes one thing in the model’s context: Studies 1 and 2 leave out investor information, while Study 3 adds the investor and tax rule.Two additional tests reuse Study 1 answers to examine judging and prevention.Source: §4 · p. 3
  • Study 1 varies question wording and data availability across 22 questions, including a thin record with an ambiguously named field and fully labelled candidate records.It also rewrites questions to name the needed field and replicates the thin-data test with newer and smaller models.Source: §4 · p. 3
  • Study 2 tests eleven current models on real NAV records for ten funds, varying whether plan or investment mode is stated and counterbalancing Direct and Regular order.Plan order is varied in a table or tool search ranking.Source: §4 · p. 3
  • Study 3 uses two redemption questions tied to profile facts and compares mentions, demands for the fact, and supplied rules, with each arm run at least five times per case.The models were selected because they failed in exploratory runs.Source: §4 · pp. 3–4
  • The judge test scores 162 Study 1 thin-data answers with or without a data dictionary and intended reading; a data-match prevention prompt is tested on thin items with one open-weight model.The prevention prompt states what the question asks and what each field measures before answering.Source: §4 · p. 4
  • Correctness is decided by code or a fixed codebook, and Studies 2 and 3 use computed references from NAV records and the Income-tax Act, 2025.Study 1 uses one AI coder with several checks, including a human check on 50 answers; Study 2 rates count only extractor agreements, and Study 3 scores the figure rather than disclosure.Source: §4 · p. 4

5 Results

Across the three studies, models used ambiguously labelled figures, followed table order when listing alternatives, and sometimes filled missing investor facts with defaults. Checks and interventions exposed trade-offs: field meanings improved judging, but silent interpretations passed, and a data-match check reduced wrong answers while prompting many more questions.

5.1 Ambiguous data: one figure in context is adopted (Study 1)

  • 96 of 126 answers were wrong when models received a thin record whose figure was not the quantity asked for, versus 7 of 126 with labelled data.Only 1 of 162 thin-data answers asked the user anything; seven of eight models were wrong on 69–90% of the thin-data answers.Source: §5.1 · pp. 4–5

5.2 Product variants: set out, but in layout order (Study 2)

  • 130 of 145 answers showing both plans listed first the plan in the table’s first column; swapping columns flipped the leading plan in 61 of 84 matched pairs.The study did not test whether readers act on the first-listed figure.Source: §5.2 · p. 5, Figure 2 · p. 6

5.3 The investor and the rules: a missing fact is filled by a default (Study 3)

  • With residency removed, Gemini 3.1 Pro gave the resident figure in all 20 runs, while older-cutoff GPT-4.1 models applied a superseded capital-gains rule.By contrast, GPT-6.1 Sol and Gemini 3.8 Flash fetched the missing fact unprompted and answered both cases correctly in all 20 final planned runs.Source: §5.3 · pp. 5–6

5.5 Can a check see it?

  • With a neutral data dictionary, GPT-6.1 Sol identified 40 of 44 mislabels, versus 17 of 44 with data alone, but false alarms rose from 15 to 23 of 61 acceptable answers.Without the intended reading, it still identified 40 of 44 mislabels with the dictionary, while passing 42 of 52 silent readings of vague questions.Source: §5.5 · p. 6

5.4 A demanded fact is fetched, but not always used (Study 3)

  • 10 of 10 demands made Gemini 3.1 Pro fetch the decisive fact, versus 1 of 10 mentions, but fetching did not ensure correct use.GPT-4.1 fetched the fact in 10 of 10 checklist runs and was correct in none; supplying the current tax rule fixed the booked-gains case in 5 of 5 runs.Source: §5.4 · p. 6, Figure 3 · p. 7

5.6 Can it be prevented, and at what cost?

  • 38 to 9 of 53 wrong answers fell under a data-match check, while questions to users rose from 1 to 33 and plainly correct answers fell from 11 to 0.Cheaper changes showed no effect: rewriting questions to name the needed field left wrong answers at 131 versus 130 of 558 paired answers, and removing the instruction to use provided data did not reduce mislabels.Source: §5.6 · p. 7

6 Discussion

The studies suggest that model interpretations track what reaches their context, while defaults fill gaps; three manipulations were consistent with this account. The proposed controls aim to settle material dimensions before answering, but their tested parts were not tested together, and the evidence is bounded by model and jurisdiction.

6.1 What the studies show together

  • 64 of 66 precise answers were right with labelled data, column swaps flipped the leading plan in 61 of 84 pairs, and demands made a failing agent fetch the fact in 10 of 10 runs versus 1 of 10 for mentions.Together, the studies describe interpretations as tracking context, with defaults filling what context leaves open.Source: §6.1 · p. 7
  • The account is bounded by model differences: GPT-6.1 Sol still mislabelled 10 of 42 precise answers, and the studies do not show that context causes the cross-study pattern.When frontier agents choose their own context, the description predicts nothing in advance.Source: §6.1 · pp. 7–8

6.2 Controls for a financial assistant

  • The proposed principle is to settle material dimensions against the question, data, and investor records before answering, then label the interpretation used.The tested control components were not tested together; the controls were tested only in India.Source: §6.2 · p. 8

7 Limitations

Coding checks showed agreement on whether answers were wrong, while distinguishing silent readings from mislabels remained less certain. The reported evidence is also bounded by small samples and limited study scope, and does not estimate how often users leave dimensions open.

7 Limitations

  • Wrong-versus-not coding agreement ranged from kappa 0.70–0.95 across checks, with a finance-trained human coder at 0.90, but the form distinction rested on one coder and 50 answers.That other-family coder recoded 10 of 15 silent readings as mislabels, while the human kept all 15.Source: §7 · p. 8
  • The studies used one jurisdiction and product family, and did not measure how often real users leave each dimension open, so they do not estimate prevalence.Study 3 arms ran five times on two cases, and judge arms rested on 162 answers.Source: §7 · p. 8

8 Conclusion

Models can answer with a real, evidence-traceable figure while silently choosing an interpretation the user did not establish. The paper recommends evaluating whether models establish the reading, not merely whether the number is supported.

8 Conclusion

  • A support check can pass a real figure that answers a different question, while a judge caught most mislabels only when told what each field measures.Benchmarks should score whether the reading was established, and judges should be tested without being given the intended reading.Source: §8 · pp. 8–9

A Glossary of Indian Terms

B.1 Terms, tolerances and conventions

  • The appendix defines precise wording, mislabels, silent readings, and flags, and records coding departures that can raise Study 1’s wrong-answer counts.Study 1 counts naming the true measure without the alternative as wrong; its blind recode confirmed sampled wrong codes but could not establish how many were missed.Source: §B.1 · p. 10
  • Study-specific reporting tolerances govern scoring; in Study 2, the other plan’s return falls outside tolerance for all ten funds, narrowly for HDFC Flexi Cap Direct.Plan is scored from the stated label; changing the Study 2 tolerance to 0.25 points would change 28 of 1,591 specified-item scores.Source: §B.1 · pp. 10–11

B.2 Evidence summary

  • Figure 4 groups measurements by situation and marks failures, appropriate behaviour, and layout-following; comparisons are valid within, not across, blocks.Its blocks use different models and scoring rules.Source: §B.2 · p. 11

B.3 Cohorts and denominators

  • Study 1’s original arm generated 594 answers, of which 558 were included; Study 2’s silence rates use agreed answers that stated a figure.Study 3 reports distinct planned, check, and exploratory run cohorts.Source: §B.3 · p. 11

B.4 Coding and statistical methods

  • Study 1 used one AI coder with blind checks; Study 2 used two extractors, counted only their agreed answers, and used code-computed references for Studies 2 and 3.Study 2 model-cluster intervals are approximate and likely too narrow with 7–11 clusters; Study 3’s repeated-prompt tests show effects on the tested cases, not generalization.Source: §B.4 · p. 11

B.5 Study 1: replications, accurate records and further results

Study 1’s thin-data results show frequent wrong answers when a figure did not measure the requested quantity. Accurate records and explicit-field restatements did not establish a clear, confirmatory reduction in errors.

B.5 Study 1: replications, accurate records and further results

  • 69–90% of wrong-quantity answers from seven models were wrong, compared with 30% for GPT-6.1 Sol.Table 9 reports thin-data behaviour across models and the accurate-record condition.Source: §B.5 · p. 12
  • The accurate-record arm reduced precise mislabels from 101 to 67 of 126, but the planned test was not significant and the result is suggestive, not confirmatory.After a Holm correction, neither planned comparison remained significant; vague-question shifts partly reflect the codebook’s definition of wrong.Source: §B.5 · p. 12
  • Explicit-field restatement changed wrong answers only from 131 to 130 of 558 paired answers.On thin data, the count was 96 to 94 of 162; an effect of about 5 points could not be excluded.Source: §B.5 · p. 12
  • When models chose their own context, 44 of 65 answers fetching one plan series were silent on plan, versus 4 of 80 when both were fetched; the pooled contrast mostly reflected model composition.Within four models that fetched one plan in some answers and both in others, silence was 6 of 23 versus 1 of 43; an exploratory tool giving only lump-sum returns misled 35 of 72 SIP answers.Source: §B.5 · p. 12

B.6 Study 2: further results and coding exceptions

Further Study 2 results show uneven accuracy across models, sensitivity to column order, and frequent silent mode assumptions. The reported rates depend on specified items, extractor agreement, and how unworded layouts are coded.

B.6 Study 2: further results and coding exceptions

  • gpt-oss-120b was right on 58% with tools; Llama 4 Maverick was right on 69% with the table and 13% with tools.These are specified-item accuracy rates, not accuracy over the full 148-item inventory.Source: §B.6 · p. 12, Table 9 · p. 13
  • Counting extractor disagreements as silent gives upper bounds of 10% for Claude Opus 5.5 and 30% for GPT-6.1 Sol; treating five unworded Gemini layouts as flagged yields 135 of 144 silent agreed answers for five models.Nine of 149 silent answers concerned Mirae mode items where mode was immaterial, so they were not frame collapse.Source: §B.6 · p. 12
  • Among seven models that followed the first column, 49 of 50 pairs flipped with column order; Grok 4.7 called Regular “the standard plan” in 8 of 10 answers when it appeared first, and none when second.The Grok result is exploratory; the 49-pair subset was selected on the outcome.Source: §B.6 · p. 12
  • Of 81 agreed silent mode answers assuming a lump sum, all 72 on material items came from seven models; three models favoured Direct regardless of column order.Across all answers, including tool calls printed as text, the lump-sum assumption appeared in 83 of 85.Source: §B.6 · p. 13

B.7 Study 3: further results

Study 3 and the judge and data-match checks distinguish obtaining facts from applying them, while showing that judge results depend on the supplied reading. The appendix also reports a cost-sensitive data-match check and a layered control design.

B.7 Study 3: further results

  • Gemini 3.1 Pro defaulted to resident in all 15 runs without residency, while GPT-4.1 applied the 30% withholding only when the question explicitly asked for the after-tax amount.GPT-4.1 justified the rate with a superseded 36-month test; current-rule text and a tax tool each produced 5 of 5 correct answers against 0 of 5 with facts alone.Source: §B.7 · p. 13
  • Zero failures in 20 baseline runs yielded a 95% Wilson upper limit of 16% pooled and 28% per model, and says nothing about other cases.The pooled limit treats runs as independent and the two models as sharing one rate.Source: §B.7 · p. 13

B.8 Judges: all arms and tests

  • With the intended reading, Sol rated all 44 mislabels wrong; without it, the small judge passed 43 of 70 wrong vague answers.The dictionary arm outperformed data alone with the reading, while data alone passed 28 of 70 wrong vague answers against 4 without data in a post hoc comparison.Source: §B.8 · p. 13
  • A simpler percentage-presence check passed 31 of 35 misleading SIP answers in exploratory agent runs.On the judge’s data-alone comparison, 7 questions moved toward passing and 3 away overall; for vague answers, 6 moved toward passing and none away.Source: §B.8 · pp. 13–14

B.9 Data-match check

  • The data-match cost score beat doing nothing and always asking when an unnecessary question costs between half and about one wrong answer.At a half it tied always asking, 33 to 33; at one it beat doing nothing by one point, 46 to 47.Source: §B.9 · p. 14

B.10 Claims, evidence and status

  • Table 11 lists each main claim with its evidence, scoring, status, and scope.Source: §B.10 · p. 14

B.11 Controls in full

  • The proposed layered control loads rule inputs, supplies the rule in force and its legal basis, then checks data against the question.The appendix distinguishes platform-owned data contracts from model-side checks for third-party feeds.Source: §B.11 · p. 14

B.12 Further limitations

All three studies use Indian mutual funds, limiting conclusions about variation across markets. Appendix caveats also qualify individual tests and the evidence supporting proposed controls.

B.12 Further limitations

  • Because all three studies use Indian mutual funds, they cannot show how failures vary across markets; the taxonomy would also need additional market-specific values.The paper names real versus nominal returns and tax wrappers as examples.Source: §B.12 · p. 14
  • Appendix caveats include possible truncation in 53 Qwen answers, a repealed-Act citation by the tax tool, judges tested without the intended reading, and a stopped resolution test.The resolution-layer test stopped after 6 of 148 items because its pre-step assumed a default plan rather than the plan in the investor’s records.Source: §B.12 · p. 14
  • Evidence for the proposed controls is uneven: rows 5–8 are untested, while planned evidence for earlier rows is limited to specified controls and failing models.Row 3 has planned evidence only for a prompt-level check that over-asked and exploratory evidence for metadata.Source: Table 12 · p. 16

Related papers

Loading 2610.38631v1…