PaperlayerView exact revision on arXiv

Source-linked AI summary

RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?

Yuyang Dai, Yan Lin, Zhuohan Xie, Yuxia Wang

arXiv:2602.07096v2q-fin.STcs.AIq-fin.CP

TL;DR

Reliable financial reasoning requires models to recognize when missing premises make a question underdetermined rather than guessing an answer. REALFIN constructs bilingual paired questions with essential conditions removed and evaluates answering, condition detection, and rejection of unjustified options. Results show systematic degradation under missing conditions, while models differ in their tendencies to over-commit, guess, or fail to identify what is missing.

  • Problem

    Financial questions often omit assumptions needed for determinate conclusions, while existing evaluations largely test only fully specified questions.

  • Method

    REALFIN is a bilingual benchmark that removes logically necessary premises from exam-style questions while preserving coherence, then evaluates three task formulations.

  • Results

    Models show systematic performance degradation when conditions are missing or no options are correct, with general models tending to speculate and finance-specialized models struggling to identify missing premises.

  • Takeaways & Limitations

    Reliable financial models must answer when information is sufficient, identify missing conditions when it is not, and avoid unjustified commitment.

  • Takeaways & Limitations

    The study does not modify or train models, so it does not directly show how much fine-tuning can fix the observed failures; its multiple-choice setting may not generalize to free-form interaction.

Abstract

from arXiv · show

Reliable financial reasoning requires knowing not only how to answer, but also when an answer cannot be justified. In real financial practice, problems often rely on implicit assumptions that are taken for granted rather than stated explicitly, causing problems to appear solvable while lacking enough information for a definite answer. We introduce REALFIN, a bilingual benchmark that evaluates financial reasoning by systematically removing essential premises from exam-style questions while keeping them linguistically plausible. Based on this, we evaluate models under three formulations that test answering, recognizing missing information, and rejecting unjustified options, and find consistent performance drops when key conditions are absent. General-purpose models tend to over-commit and guess, while most finance-specialized models fail to clearly identify missing premises. These results highlight a critical gap in current evaluations and show that reliable financial models must know when a question should not be answered.

1 Introduction

REALFIN addresses a gap in financial evaluation: models must recognize when missing premises make a question underdetermined, not merely select answers to fully specified questions. It introduces a bilingual benchmark and task formulations for testing this capability.

  • Motivation: Reliable financial reasoning requires recognizing when missing assumptions, constraints, time horizons, or standards prevent a determinate conclusion.Under such conditions, committing to one interpretation is unjustified because different assumptions can yield different conclusions.
  • Research gap: Existing financial benchmarks largely assume every question is fully specified and has one correct answer.This design emphasizes answer selection while excluding cases where missing conditions invalidate answering or no listed option is correct.
  • Benchmark: REALFIN creates bilingual paired questions by removing logically necessary premises while preserving linguistic plausibility and realism.The revised questions remain coherent but underdetermined, so a concrete solution requires clarifying unstated assumptions.
  • Task design: The benchmark evaluates three formulations: Original full-condition questions, Revised condition-missing questions, and None-of-the-above questions.Together, these settings test answering, recognizing when clarification is needed, and rejecting unjustified options rather than relying on pattern recognition.
  • Contributions: 2,020 questions span English and Chinese full-condition and condition-missing variants.The dataset is designed to expose whether models over-commit and confidently answer questions that are not solvable from the stated information.
  • Contributions: 15 LLMs show systematic performance degradation when conditions are missing or no options are correct for open-source models.The evaluation includes commercial, finance-specialized, and reasoning-enhanced models.

2 Related Work

Prior financial benchmarks emphasize fully specified tasks such as extraction, prediction, and multi-step reasoning, while abstention research studies under-specified questions mostly outside professional finance. REALFIN targets this intersection by evaluating logical underdetermination in financial reasoning.

  • Financial benchmarks: Earlier financial benchmarks focus on extraction, sentiment, prediction, numerical reasoning, long-form reasoning, retrieval, and multi-step tasks.These evaluations generally treat the available evidence as local and fully specified.
  • Financial benchmarks: Financial benchmarks commonly assume that each problem admits a unique correct answer, limiting insight into behavior when critical conditions are missing.This assumption persists across newer reasoning-oriented and simulation-based evaluations.
  • Challenges in finance: Prior work reports that LLMs struggle with precise prediction, strict rule application, interpretability, control, uncertainty, temporal change, and incentive-aligned decisions.These findings motivate selecting challenging financial scenarios for benchmark construction.
  • Specialized models: Specialized financial models increasingly use domain-specific pretraining, instruction tailoring, reinforcement learning, and chain-of-thought reasoning.Despite these advances, they are generally assessed under closed-system standards that presume a valid answer exists.
  • Abstention: Research on unanswerable questions distinguishes safety refusals from clarification of under-specified or context-insufficient questions.REALFIN belongs to the clarification-oriented line of work.
  • Abstention: Models often overcommit by silently substituting missing conditions with learned defaults, while reliable agents should ask for missing information.Selective prediction formalizes the coverage-accuracy trade-off associated with abstention.
  • Research gap: REALFIN targets logical underdetermination caused by missing financial standards or valuation assumptions, which general-domain benchmarks do not capture.Its focus is professional financial reasoning rather than generic unanswerable-question behavior.

3 Dataset

The REALFIN dataset combines bilingual financial questions across cognitive types and topics, then creates realistic under-specified variants by removing premises needed for unique answers. Manual review and cross-validation preserve financial coherence and underdetermination.

  • Question types: The dataset covers six reasoning types, from financial concept explanation and simple calculation to complex calculation, judgment, application, and statistical methods.Question types reflect dominant reasoning demands and may overlap when a problem requires multiple capabilities.
  • Financial topics: English questions span nine financial sub-domains, while Chinese questions cover five institutionally grounded CPA categories.The categories include areas such as equity, fixed income, quantitative analysis, accounting, tax law, and wealth management.
  • Data sources: The full-condition dataset contains 1,178 questions drawn from public CFA-style and CPA-style instructional materials, excluding real examination content.The sources are released under permissive licenses for academic research.
  • Data selection: Selected questions emphasize conceptual understanding, multi-step reasoning, and information integration because these expose whether models understand structure and recognize insufficient information.Each question is manually reviewed and tagged by financial subdomain and question type.
  • Condition-missing data: Condition-missing questions are created by manually removing one or more assumptions required to determine a unique answer.The four removed condition types are macro assumptions, linking methods, constraint-defining information, and standard-selection cues.
  • Quality control: Two annotators independently revise and cross-validate each condition-missing question for contextual validity and lack of a unique answer.Disagreements are resolved through discussion to maintain task-design integrity.
  • Dataset statistics: 1,062 English examples and 959 Chinese cases are distributed across Original and Revised formulations, with Summary and Comprehensive Judgment the largest category.Simple Calculation also forms a substantial portion of the dataset.

4 Experiments

Across experiments, model performance varies by language, question type, topic, and model family, with missing-condition settings exposing distinct reasoning failures. General models often speculate, finance-specific models show severe dead zones, while some reasoning-focused models improve substantially when conditions are missing.

  • General-purpose models perform best overall, while financial models perform worst and reasoning-enhanced models fall between them.
  • 89.62% is Claude-Sonnet-3.5’s English CFA accuracy in the missing-condition setting, while Qwen3-Max leads Chinese CPA at 92.00%.
  • Financial models record repeated 0% dead zones in Chinese CPA knowledge application and complex calculation, while reasoning-focused models can still struggle to stop appropriately.
  • Information integration density correlates more strongly with question difficulty (r = 0.78) than question length (ρ = 0.43) or numerical frequency (ρ = 0.52).
  • General models improve on some English revised tasks but decline on Chinese tasks, where ambiguous residual wording can encourage unsupported interpretations.
  • Fin-R1-7B gains +65.57% on Chinese CPA under missing conditions, whereas earlier financial models remain stagnant or decline.
  • Models often guess instead of identifying missing premises, producing substantial Revised-to-NOTA accuracy drops and failing to explain which condition is absent.

5 Conclusions

Reliable financial LLMs must do more than answer accurately: they must detect missing conditions and abstain when no valid answer exists. Performance is inconsistent across these settings and languages, so answer accuracy alone is insufficient for assessing reliability.

  • Reliable financial reasoning requires answering correctly when information is sufficient, identifying missing conditions when it is not, and abstaining when no valid answer exists.

Limitations and Future Work

The study does not train or modify models, and its multiple-choice evaluation may not capture behavior in free-form or interactive settings. Future work proposes separate training stages for financial knowledge and missing-condition behavior, alongside broader coverage and interactive clarification.

  • Limitations: The study does not modify or train models, so it identifies failure sources without showing how much fine-tuning can correct them.
  • Limitations: Multiple-choice evaluation may produce behavior that differs from models answering free-form questions or interacting with users.
  • Future Work: Specialized supervised fine-tuning is proposed to teach professional knowledge and stable reasoning formats before logic-driven reinforcement learning controls behavior under missing conditions.
  • Future Work: The proposed training pipeline would reward identifying missing information and penalize guessing, invented assumptions, and reasoning beyond the question's support.
  • Future Work: Future dataset expansion will cover risk management, securities regulation, and quantitative finance beyond accounting and tax law.
  • Scope: The work targets safer financial-model use and risk assessment rather than autonomous financial decision-making.

A Dataset Curation

The dataset formalizes condition-missing financial questions by removing logically necessary premises while preserving coherence, then evaluates how models respond when conclusions are no longer uniquely justified.

  • Dataset construction: Condition-missing questions remove one or more necessary premises from full-condition questions while retaining a coherent interpretation.The remaining question must stay grammatically and semantically well-formed rather than becoming inconsistent.
  • Dataset construction: The construction also requires multiple internally consistent interpretations, ensuring that no single answer is justified by the available information.
  • Evaluation interpretation: Condition removal changes the task from computing a conclusion to recognizing that the information is insufficient for a uniquely justified answer.
  • Evaluation formulations: The benchmark distinguishes original questions, clarification-requiring revised questions, and NOTA questions that replace the correct option with “None of the above”.The NOTA formulation tests whether models infer that no provided answer is supported rather than relying on answer-pattern recognition.
  • Evaluation interpretation: Models may appear correct on revised questions by implicitly completing missing information with default assumptions that happen to match the retained reference answer.Thus, revised-question accuracy can reflect assumption alignment rather than logical support from the observed premises.
  • Additional analyses: Difficulty analyses compute class accuracy, rank question classes from hardest to easiest, and compare model rankings using pairwise Spearman correlation.The reported average correlation is ρ = 0.91 across M = 10 models.

B.3 Information Integration Metrics

The evaluation uses expert-annotated reasoning features, a cross-paradigm model suite, and structured bilingual outputs to examine financial reasoning under controlled conditions. Strict JSON improves auditability but imposes a measurable formatting cost.

  • Information-integration analysis: Experts annotate questions for interacting premises, cross-domain dependencies, and constraint reconciliation to characterize information-integration demands.Cross-domain dependencies and constraint reconciliation are each rated on 1–5 scales.
  • Model evaluation: The study compares general-purpose and financial models, with additional reasoning-enhanced models added to broaden coverage across families, scales, and training strategies.
  • Scope constraints: FinGPT-7B is evaluated only on English sub-benchmarks, while financial-specific models collapse on complex Chinese CPA calculations in both original and revised questions.
  • Evaluation protocol: The evaluation includes Original, Revised, and NOTA settings and requires bilingual JSON outputs containing reason, answer, and confidence fields.A cascaded parser recovers structured fields from standard JSON, tagged content, balanced braces, or field-level patterns.
  • Evaluation protocol: A 3% to 4% accuracy reduction occurs under strict JSON compared with free-form output for the five open-weight financial models.The structured protocol is retained so epistemic awareness can be audited through the reason field.

C.4 XuanYuan3-70B Configuration

XuanYuan3-70B is evaluated with its native Llama-3 chat template and standardized JSON prompts across English CFA and Chinese CPA tasks. Its results show strong task-specific gains but substantial calibration problems under NOTA masking.

  • Configuration: The evaluation prompt asks for strict JSON containing reason, answer, and confidence in both English CFA and Chinese CPA settings.
  • Configuration: XuanYuan3-70B uses the native Llama-3 chat template with explicit system, user, and assistant roles encoded at tokenizer level.
  • Results: 100.0% accuracy is reached for English CFA Complex Calculation under revised settings, while Chinese CPA Simple Calculation improves by more than 50.0%.
  • Results: Chinese CPA Knowledge Application rises from 0.00% to 66.67%, addressing the reported zero-shot performance collapse in that category.
  • Robustness: Under NOTA masking, confidence exceeds 90.0 while logical validity falls below 10%, indicating severe overconfidence despite frequent adversarial-marker detection.The model often selects NOTA through surface-level prompt-pattern matching rather than valid deduction.
  • Robustness: XuanYuan3-70B shows lower logical consistency than Fin-R1-7B despite having ten times as many parameters.The reported interpretation attributes the disparity to reasoning constraints from reinforcement learning rather than parameter scale alone.

C.5 Fin-R1-7B Configuration

Fin-R1-7B is evaluated with the Qwen2.5 chat template and a unified JSON protocol. Its reinforcement-learning-based reasoning yields strong bilingual accuracy and resilience when NOTA is the justified response.

  • Configuration: Fin-R1-7B uses the Qwen2.5 chat template through tokenizer.apply_chat_template(), while the study replaces the repository’s recommended chain-of-thought format with standardized JSON output.
  • Overall results: Fin-R1-7B reaches 83.65% overall accuracy on English CFA and 73.14% on Chinese CPA despite having 7 billion parameters.The reported performance closely approaches that of the larger XuanYuan3-70B.
  • Overall results: Chinese CPA accuracy rises from 7.57% to 73.14%, a ∆+65.57% improvement, with Economic Law reaching 91.89%.
  • Quantitative reasoning: Complex Calculation improves from 26.67% to 100.0%, while Simple Calculation gains more than 32%.The passage links these gains to structured chain-of-thought processes within the reasoning-model lineage.
  • Task coverage: The model reports 90.00% on CPA Conceptual Understanding and 93.62% on CFA Statistical Methods, with gains across all task categories.
  • Robustness: NOTA accuracy reaches 84.5% on CFA and 77.7% on CPA, with Fin-R1-7B rejecting plausible distractors rather than selecting the nearest numerical option.

C.6 CFGPT2-7B Configuration

CFGPT2-7B is evaluated as an InternLM-based Chinese financial assistant under original, revised, and adversarially masked conditions. Revision improves several benchmark scores, but masking exposes a large gap between answer selection and valid reasoning.

  • Configuration: CFGPT2-7B is a Chinese financial assistant built on the InternLM-7B base and evaluated with BF16 precision on one RTX 4090.
  • Detailed Results: The revised model reaches 49.71% in Chinese CPA, compared with 2.96% for the original model.
  • Detailed Results: The revised model achieves 100% accuracy in English Complex Calculation, while Simple Calculation improves by more than 38% and 44% across the two benchmarks.
  • Detailed Results: CFGPT2-7B performs strongly in CFA Corporate Finance at 90.00% and Portfolio Management at 100.00%, and reaches 80.00% in CPA Statistical Methods.
  • Robustness Analysis: Under adversarial masking, Answer Accuracy is 75.2% in CFA and 54.9% in CPA, while Reasoning Accuracy falls to approximately 10%.The results indicate that correct labels often arise without a corresponding valid derivation.
  • Robustness Analysis: Despite logical invalidity, CFGPT2-7B maintains an average confidence score above 81.0, producing authoritative but erroneous responses.

C.7 DISC-FinLLM-13B Configuration

DISC-FinLLM-13B is a Baichuan-based dialogue model evaluated on single-turn financial multiple-choice tasks, while FinGPT-7B uses a flat instruction format for English CFA. Both models show limited suitability for rigorous missing-premise reasoning.

  • Configuration: DISC-FinLLM-13B is primarily designed for multi-turn financial consulting rather than single-turn multiple-choice questions.
  • Detailed Results: DISC-FinLLM-13B has baseline accuracy of 13.11% in English CFA and 4.61% in Chinese CPA.
  • Detailed Results: Revision raises DISC-FinLLM-13B accuracy to 19.89% in CFA and 19.43% in CPA, while some CFA subjects regress.The passage attributes the regressions to possible interference between revised prompts and conversational priors.
  • Detailed Results: DISC-FinLLM-13B struggles with Statistical Methods and Quantitative Analysis, while CPA Conceptual Understanding rises from 4.20% to 30.00%.
  • Robustness Analysis: Under adversarial masking, DISC-FinLLM-13B shows diminished confidence and deteriorating reasoning accuracy, indicating reliance on conversational templates.
  • FinGPT Configuration: FinGPT’s Qwen-7B LoRA adapter interferes with zero-shot professional reasoning, so evaluation is limited to the English CFA sub-benchmark.
  • FinGPT Results: FinGPT-7B achieves only 2.19% original and 3.54% revised accuracy on English CFA, frequently producing sentiment markers or jumbled characters instead of logical steps.

C.9 GPT-OSS-20B and GPT-OSS-120B Configuration

GPT-OSS-20B and GPT-OSS-120B provide a fixed-paradigm scaling comparison, while DianJin-R1-32B tests whether finance-specialized reasoning gains persist at larger scale. High answer accuracy remains unreliable when reasoning accuracy collapses, especially on Chinese tasks.

  • Configuration: GPT-OSS-20B and GPT-OSS-120B are open-weight reasoning Mixture-of-Experts models compared under the same training paradigm and chat template.
  • Robustness Analysis: GPT-OSS-20B has 86.65% English and 67.43% Chinese Answer Accuracy, but Reasoning Accuracy falls to 12.26% and 4.00%.
  • Robustness Analysis: Scaling to 120B raises Answer Accuracy to 89.10% in English and 84.57% in Chinese, while Reasoning Accuracy reaches only 20.71% and declines to 1.14%.
  • Robustness Analysis: The answer–reasoning gap widens on Chinese from 63.4 to 83.4 percentage points and narrows only slightly on English from 74.4 to 68.4 points.
  • Robustness Analysis: Mean confidence rises to 95.66 on English and 91.46 on Chinese for GPT-OSS-120B, the highest values in the evaluation.This supports the paper’s distinction between answer accuracy and reliable reasoning.
  • Detailed Results: DianJin-R1-32B reaches 89.26% revised English CFA accuracy and 93.79% revised Chinese CPA accuracy.
  • Robustness Analysis: DianJin-R1-32B has 29.16% English and 5.71% Chinese Reasoning Accuracy under NOTA masking, despite Answer Accuracy of 88.28% and 77.71%.
  • Robustness Analysis: The model’s Chinese explanations rarely identify missing premises and instead commit to regulatory or accounting defaults.
Loading 2602.07096v2…