Source-linked AI summary
BudgetVerify: Budget-Tiered Verification for Financial QA
Janet Jenq, Hongda Shen
TL;DR
Financial QA requires precise numerical extraction, unit handling, and arithmetic, while verification needs vary across questions and generated answers. BudgetVerify routes each answer among three verification tiers using pre-verification information. Across six base models, it produces more efficient accuracy–cost frontiers than fixed verification policies, with a consistent qualitative frontier shape across models.
Problem
Financial QA requires precise extraction, unit handling, and arithmetic, while questions vary in difficulty and a fixed verification strategy may be inefficient or unsuccessful.
Source: §1 · pp. 1–2Method
BudgetVerify uses a lightweight router trained from offline correctness and token-cost outcomes to choose no verification, check-and-revise, or solve-first verification using pre-verification features.
Source: §3.2 · p. 5, §5 · p. 9Results
Across six commercial and open-weight base models, BudgetVerify consistently improves on uniform MID verification and often matches or approaches uniform HIGH verification with fewer tokens, while frontier shape is qualitatively consistent across models.
Source: §5 · p. 9Takeaways & Limitations
Selective verification yields initial accuracy gains followed by diminishing returns as verification becomes more expensive, while the frontier's absolute position varies by model.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Financial question answering often requires precise numerical extraction, unit handling, and arithmetic over tables and text, but applying expensive verification uniformly wastes test-time compute. We propose BudgetVerify, a budget-tiered generator-verifier framework that routes each generated answer to one of three verification tiers: no verification, lightweight check-and-revise, or higher-cost solve-first-then-compare verification. The router is trained from offline correctness and token-cost outcomes and, at test time, selects a verification tier using information available before verification, including the question, context statistics, the generated answer, and associated generator metadata. The selected tier either returns the generated answer directly or invokes the corresponding verifier. Across six commercial and open-weight base models, BudgetVerify consistently produces more efficient accuracy-cost Pareto frontiers than fixed verification policies by selectively allocating stronger verification only when it is useful. Although absolute performance varies across models, these efficiency gains and the resulting qualitative frontier shape are consistent across generator models.
1 Introduction
Financial QA combines precise extraction and numerical reasoning, while the value of verification varies with question difficulty and the generated answer. BudgetVerify allocates verification across tiers, and its accuracy–cost frontier has a consistent qualitative shape across base models despite model-dependent absolute performance.
1 Introduction
- Financial QA requires precise value extraction, unit and scale handling, and arithmetic, so small errors such as a wrong period or denominator can change the answer.Questions range from direct lookups to multi-step numerical reasoning, making a single fixed inference strategy potentially inefficient or unsuccessful.Source: §1 · p. 1
- Verification outcomes depend jointly on problem difficulty, generator capability, and verifier strength, so applying the strongest verifier uniformly can misallocate compute.Some answers may be cheaply verified, while others remain difficult even with stronger verification.Source: §1 · pp. 1–2
- BudgetVerify treats financial QA verification as budgeted routing among no verification, lightweight revision, and solve-first-then-compare verification.The goal is to improve aggregate accuracy under a fixed inference budget.Source: §1 · p. 2
- Across evaluated base models, the accuracy–cost frontier has a consistent qualitative shape, although its absolute position depends on the model.Selective verification yields early accuracy gains followed by diminishing returns as more expensive verification is applied.Source: §1 · pp. 2–3
2 Related Work
Prior work studies test-time compute, generator–verifier systems, and cost-aware routing. BudgetVerify instead allocates verification depth for a single generated financial answer.
2 Related Work
- Adaptive test-time compute allocation has been studied as an alternative to fixed-budget strategies, while BudgetVerify allocates verification compute rather than deciding how many samples to draw.The related work describes adaptive allocation as a constrained optimization problem and lightweight policies trained to imitate oracle budget decisions.Source: §2 · p. 3
- Generator–verifier systems use a generator to propose candidates and a verifier to score, filter, or refine them; BudgetVerify varies verification depth for one financial answer.Prior work also studies candidate scaling, verifier votes, process supervision, and generative verification.Source: §2 · p. 3
- Prior routers select among models, tools, or budgets, whereas BudgetVerify fixes the base model and routes each generated answer among verification actions.Its supervised router is trained from offline correctness and cost outcomes; lightweight checks may address some errors, while others require re-solving from source context.Source: §2 · pp. 3–4
3 Methodology
BudgetVerify formulates financial QA verification as a cost-sensitive choice among tiers for each generated answer. It trains a lightweight router from offline tier outcomes and restricts routing inputs to information available before verification.
3.1 Problem Formulation
- The router chooses a verification tier for a single generated answer to trade off final-answer correctness against inference cost.The tiers return the generator answer directly, apply lightweight check-and-revise, or use solve-first-then-compare verification; generator cost is included and is constant across tiers for a fixed example.Source: §3.1 · p. 4
3.2 BudgetVerify
- A generator produces a structured answer and metadata, then a lightweight router selects among no verification, MID, and HIGH.The answer can include a predicted operation, unit, formula, operands, evidence snippets, and confidence.Source: Figure 2 · p. 4, §3.2 · pp. 4–5
- The router is trained to imitate cost-aware tier choices derived from offline runs, using only pre-verification features rather than verifier outcomes or oracle labels.Features include question cues, context statistics, and generator-output properties; routing overhead is intended to remain small relative to verifier cost.Source: §3.2 · p. 5
4 Experimental Results
BudgetVerify is evaluated on FinQA across six commercial and open-weight base models, with routers trained to choose among fixed verification tiers using pre-verification information. Routed policies produce more efficient accuracy-cost frontiers than fixed policies, with consistent qualitative patterns across models and stronger performance from TabFM among the supervised routers.
4.1 Datasets and Experimental Configuration
- FinQA evaluation uses six base models, one generator sample per question, same-model verifiers, and official test accuracy and cost after train/validation-only router fitting.FinQA examples pair questions with financial tables, textual evidence, executable answers, and annotated reasoning programs; LOW returns the generator answer, MID checks and revises, and HIGH independently solves then compares.Source: §4.1 · pp. 5–6, Table 1 · p. 5
4.2 Main Results
- Across all six base models, routed policies improve over GENERATOR ONLY, generally outperform ALWAYS MID, and can approach ALWAYS HIGH accuracy at lower normalized token cost.In some settings, they reach or surpass ALWAYS HIGH; Figure 4 plots test accuracy against token cost normalized by the generator-only baseline, with each panel representing a separate model setting.Source: §4.2 · p. 7, Figure 4 · p. 8
4.2.1 Adaptive routing improves the accuracy-cost frontier across base models
- The frontier’s absolute position varies by model, but stronger generators start more accurate while weaker generators offer more opportunities for MID or HIGH to repair errors.Across commercial and open-weight models, adaptive verification provides better accuracy at a given token budget than applying one tier uniformly.Source: §4.2.1 · pp. 7–9
4.2.2 Comparison across router classifiers
- TabFM produces the strongest and most consistent supervised-router frontiers, while Decision Tree is weakest overall, especially in mid- and high-budget regimes.The reported gap suggests routing depends on nonlinear interactions among question cues, generator metadata, operation type, evidence coverage, and cost rather than only simple thresholds.Source: §4.2.2 · p. 9
5 Conclusions
BudgetVerify selects among verification tiers instead of applying one verifier uniformly, producing more efficient accuracy–cost Pareto frontiers. Across evaluated base models, the frontier shape is consistent: selective verification brings early accuracy gains, then diminishing returns as verification costs rise.
5 Conclusions
- BudgetVerify consistently improves on uniform MID verification and often matches or approaches uniform HIGH verification while using fewer tokens.Experiments cover FinQA and six commercial and open-weight base models.Source: §5 · p. 9
- Across every evaluated base model, selective verification yields substantial initial accuracy gains followed by diminishing returns as more expensive verification is applied.The frontier’s absolute position varies by model, while its qualitative shape is robust.Source: §5 · p. 9
- Easy or already-correct answers often benefit from being left unchanged, while more difficult numerical cases are more likely to benefit from stronger verification.The authors interpret the recurring frontier geometry as evidence that budget-aware verification is an inference-time allocation principle, not a property of one model family.Source: §5 · p. 9
A Appendix
The appendix feature dictionary specifies router inputs available before MID or HIGH verification, while excluding verifier outcomes and post-verification information. It lists answer and cross-risk features and includes templates for the MID and HIGH verifiers.
A Appendix
- Router features come from the question, lightweight context statistics, or the generator’s structured answer before MID or HIGH runs; verifier outcomes and post-verification outputs are excluded.The feature dictionary presents categories, feature names, and definitions.Source: Table 2 · p. 12, A Appendix · p. 13
- Generator-output features include the number of numeric strings in the answer, its unit, and the generator’s confidence level.Unit indicators cover number, percent, dollars, ratio, and unknown; confidence indicators cover high, medium, and low.Source: A Appendix · p. 13
- Cross-risk features flag multi-period numeric answers, calculation cues, missing formulas or operand sources, and unit or period mismatches.The listed signals also cover denominator-sensitive wording and signed-change wording.Source: A Appendix · p. 13
- The MID verifier uses lightweight, candidate-centered check-and-revise.Source: Figure 5 · p. 14
- The HIGH verifier independently solves from context, executes the extracted calculation with a deterministic calculator, checks operand grounding, and compares its answer with the candidate.That comparison determines whether the candidate is kept or revised.Source: Figure 6 · p. 15