Source-linked AI summary
E2A-Bench: Benchmarking Evidence-to-Action Reliability in Financial Chart Reasoning
Xiaoya Wang, Yutong Xu, Junjie Wang
TL;DR
Claim- and task-centric evaluations do not fully assess whether chart evidence remains traceable through rationales, confidence estimates, and final actions while preserving directional coverage. E2A-Bench evaluates financial chart reasoning with deterministic OHLCV evidence anchors and measures grounding, reasoning–action consistency, evidence-confidence calibration, and directional coverage. Low hallucination can reflect non-action; oracle-aided verification reduced unsupported claims while collapsing actionable coverage, and financial fine-tuning amplified BUY:SELL ratios by 4.21–4.68×.
Problem
Claim- and task-centric evaluations do not fully assess whether chart evidence remains traceable through rationales, confidence estimates, and final actions while preserving directional coverage.
Source: §1 · p. 1Method
E2A-Bench evaluates financial chart reasoning with deterministic OHLCV evidence anchors and measures grounding, reasoning–action consistency, evidence-confidence calibration, and directional coverage.
Source: §3.1 · p. 3, §3.4 · p. 6Results
Takeaways & Limitations
Financial VLM evaluation should trace the coverage–grounding frontier rather than rely on scalar hallucination scores, with NDR measuring evidence-to-action reliability rather than trading performance.
Source: §5 · p. 9The benchmark is a fixed-date HS300 snapshot and does not yet cover other market regimes, asset classes, or cross-market trading conventions.
Source: Limitations · pp. 9–10
Abstract
from arXiv · showhide
Can financial vision-language models (VLMs) turn chart evidence into reliable action recommendations? Existing hallucination evaluations are mostly claim-centric; they assess whether generated statements are supported, but not whether evidence remains traceable through rationale, confidence, and final action. We introduce E2A-Bench, a 969-query benchmark for financial chart reasoning, constructed from 323 HS300 constituents under three input modalities with deterministic OHLCV-derived evidence anchors. E2A-Bench evaluates grounding, reasoning-action consistency, evidence-confidence calibration, and directional coverage through UCR, RCI, ECI, and NDR, where NDR measures coverage-aware evidence-to-action reliability rather than realized trading performance. Evaluating 20 VLMs reveals three failures hidden by scalar hallucination scores: the lowest-UCR model ranks near the bottom by NDR due to only 6.4% directional coverage; oracle-aided verification reduces unsupported claims but can collapse coverage; and financial fine-tuning amplifies the BUY:SELL ratio by factors of 4.21 to 4.68 across strict base-fine-tuned pairs. These results show that financial VLM evaluation should trace the full evidence-to-action chain rather than rely on a single hallucination score. Code and data: https://github.com/wanng-ide/E2A-Bench
1 Introduction
E2A-Bench addresses gaps in claim- and task-centric evaluation by tracing verifiable chart evidence through claims, rationales, confidence, and actions while accounting for directional coverage. Its evaluation of 20 VLMs shows that scalar hallucination scores alone miss decision-reliability failures.
1 Introduction
- Claim-centric evaluations miss coverage, while task-centric evaluations miss whether evidence remains consistent through claims, rationales, confidence estimates, and final actions.A model can produce few unsupported claims yet offer little actionable value, or give plausible rationales with unsupported actions or confidence.Source: §1 · p. 1
- The proposed coverage–grounding frontier favors improving verifiable evidence grounding while preserving actionable coverage; the evaluation does not assess market-return prediction.The protocol can retain its evidence-to-action audit structure while replacing domain-specific anchors, claim patterns, and action schemas.Source: §1 · pp. 1–2
- E2A-Bench evaluates a 969-query benchmark across 323 HS300 constituents and three modalities, using OHLCV-derived evidence anchors and UCR, RCI, ECI, and NDR.The protocol assesses unsupported claims, reasoning–action inconsistency, overconfidence under weak evidence, and coverage-aware decision reliability.Source: §1 · p. 2
- Gemma-4-E2B-It had the lowest overall unsupported-claim rate (39.9%) but only 6.4% directional coverage; Qwen2.5-VL-3B-Instruct achieved the highest NDR (32.1%) with 82.5% coverage.These results illustrate how scalar hallucination scores can misrank decision reliability.Source: §1 · p. 2
- Oracle-aided verification reduced unsupported claims but yielded negative mean ∆NDR, while strict base–fine-tuned pairs amplified BUY:SELL ratios by 4.21–4.68×.The benchmark identifies verification-induced coverage collapse and BUY-side amplification after financial fine-tuning as distinct failure patterns.Source: §1 · p. 2
2 Related Work
Prior work improves chart and financial visual reasoning, and existing benchmarks assess chart or finance tasks and hallucinations. E2A-Bench shifts the evaluation unit to coverage-aware evidence-to-action reliability.
2 Related Work
- Table 1 compares representative benchmarks by supported input types, explicit evidence grounding, decision-oriented action evaluation, and coverage-aware scoring.The comparison situates these dimensions alongside existing claim- and task-centric evaluations.Source: Table 1 · p. 3, §2 · p. 3
- General-purpose, chart-oriented, and finance-oriented VLMs improve visual parsing and chart or financial-image reasoning, but do not establish that recommendations are evidence-grounded, coherent, calibrated, and actionable.Examples include Qwen2.5-VL (Bai et al., 2025), MatCha (Liu et al., 2023b), and FinLLaVA.Source: §2 · p. 3
- Existing benchmarks cover chart question answering, realistic and cross-modal chart reasoning, chart hallucination, and finance-specific QA and business scenarios, but are mostly claim- or task-centric.E2A-Bench is complementary, testing whether chart evidence remains traceable through claims, rationales, confidence estimates, and final actions.Source: §2 · p. 3
3 E2A-Bench: Evidence-to-Action
E2A-Bench evaluates financial chart recommendations as an evidence-to-action chain, using deterministic OHLCV evidence and coverage-aware scoring. Its design targets grounded, coherent, calibrated actions without rewarding systematic non-action.
3 E2A-Bench: Evidence-to-Action
- E2A-Bench evaluates chart-based recommendations for whether they are grounded, coherent, calibrated, and actionable.The benchmark treats the recommendation, its rationale, and confidence as the decision target rather than a chart description.Source: §3 · p. 3
3.1 Design Principles
- Coverage-aware chain integrity requires evidence to remain consistent through claims, rationale, confidence, and action while avoiding artificial gains from systematic non-action.The design uses deterministic OHLCV facts as evidence anchors rather than future returns or human investment judgments.Source: §3.1 · p. 3
3.2 Task Formulation
- Each query uses a preceding 30-trading-day OHLCV window formatted as serialized text, a candlestick chart with moving averages and volume bars, or both.The response includes technical analysis, rationale, an action, and a confidence score, without a ground-truth trading action.Source: §3.2 · pp. 3–4
3.3 Evidence Anchor Construction
- Ten fixed-rule OHLCV anchors verify technical claims and estimate evidence strength; they are evidence labels, not trading-action labels.The anchors cover moving averages, price–volume behavior, trend, volatility, range position, and short-term momentum.Source: Table 2 · p. 4, §3.3 · p. 4
3.4 Evaluation Protocol
- NDR combines directional coverage with directional-conditioned grounding, reasoning–action consistency, and confidence-calibration terms; non-directional outputs reduce coverage rather than errors.The multiplicative metric is not a measure of realized trading returns.Source: §3.4 · p. 6
3.5 Validation and Benchmark Statistics
- ρ = 0.466 (p = 0.009): NDR had the strongest correlation with holistic human reliability judgments in a blinded pilot, and findings stayed stable across 1,000 scorer perturbations.Professional annotations showed substantial inter-annotator agreement, and the scorer matched consensus labels with high reliability.Source: §3.5 · p. 6
4 Experiments and Findings
Across 20 VLMs, the leaderboard shows that low unsupported-claim rates do not guarantee reliable action: the strongest result combines high directional coverage with high NDR. Modality, financial fine-tuning, and model scale produce distinct reliability patterns, reinforcing the need to assess the full coverage–grounding frontier.
4.1 Setup
- The study evaluates 20 proprietary, open-weight, and financial VLMs across text-only, chart-only, and multimodal inputs in a trimodal leaderboard.The leaderboard aggregates performance over the three modalities.Source: §4.1 · pp. 6–7, Table 4 · p. 7
4.2 Overall Performance
- Qwen2.5-VL-3B-Instruct achieves the highest NDR (32.1%) with 82.5% coverage, while Gemma-4-E2B-It has the lowest UCRall (39.9%) but only 6.4% coverage and NDR = 5.1%.The leaderboard characterizes reliability by the coverage–grounding frontier rather than by a scalar hallucination score.Source: Table 4 · p. 7, §4.2 · p. 7
4.3.1 Input Modality Effects
- Chart-only input has the highest average baseline NDR across 20 models (19.2%), ahead of multimodal (15.6%) and text-only (14.5%), though the best modality varies by model.Qwen3.5-4B peaks with text-only input (37.9%); Gemma-4-31B and Qwen3.5-2B peak with chart-only input (29.4% and 24.1%).Source: Figure 4 · p. 8, §4.3.1 · p. 8
4.3.2 Fine-Tuning Redistributes Risk
- Financial fine-tuning yields inconsistent NDR changes (+1.5 pp for Amsi-fin-o1; −6.8 pp and −7.2 pp for PyFi-QwenVL-3B and -7B), but increases BUY:SELL ratios 4.21–4.68× across all three pairs.Across the pairs, 455 of 2,907 responses shifted to BUY while matched evidence claims decreased approximately 34–40%; the authors describe these as output shifts, not causal effects.Source: §4.3.2 · p. 8
4.3.4 Qualitative Illustration
- Gemma-4-E2B-It avoids verifiable claims and issues HOLD, while Qwen3.5-4B gives an evidence-backed SELL, illustrating why NDR accounts for directional coverage.Source: Figure 6 · p. 9, §4.3.4 · p. 9
4.3.3 Scaling Is Not Monotonic
- In chart-only Qwen3.5, NDR peaks at 2B (24.1), drops at 4B (12.6), recovers at 9B (18.1), then declines at 27B (13.1), while local diagnostics show non-monotonic trends.UCR increases from 57.0 to 71.0 between 2B and 27B; RCI and ECI follow non-monotonic, partially conflicting trends.Source: §4.3.3 · pp. 8–9, Figure 7 · p. 9
5 Conclusion
E2A-Bench traces chart evidence through claims, rationale, confidence, and action, revealing failures that scalar hallucination scores miss. Its conclusions are bounded by the benchmark’s market and detector scope, and its scores are not investment advice.
5 Conclusion
- Low hallucination can reflect non-action, oracle-aided verification can collapse actionable coverage, and financial fine-tuning can amplify BUY-side bias.NDR measures evidence-to-action reliability under the evaluation protocol, not market forecasting or realized trading performance.Source: §5 · p. 9
Limitations
- The fixed-date HS300 snapshot does not cover other market regimes, asset classes, or cross-market trading conventions.The authors propose broader regimes and instruments to test reliability under changing volatility, liquidity, and macro-market conditions.Source: Limitations · pp. 9–10
- Deterministic OHLCV anchors and pattern-based detectors support reproducibility but may miss rare paraphrases, nuanced technical claims, and uncovered reasoning patterns.Proposed extensions include broader multilingual patterns, stronger semantic parsers, and professional adjudication for ambiguous cases.Source: Limitations · p. 10
Ethical Considerations
- E2A-Bench is a research evaluation resource, not financial advice; a high NDR does not establish profitability, investment quality, or deployment readiness.Rankings should not be the sole basis for selecting or deploying financial systems, and downstream use must follow applicable regulations and market-data licensing requirements.Source: Ethical Considerations · p. 10
A Detailed Benchmark Specification
The detailed specification defines evidence labels and strong-signal rules using OHLCV anchors, while execution-record checks show that deduplication does not change the substantive comparisons.
A Detailed Benchmark Specification
- Evidence anchors are computed from 30-trading-day OHLCV windows and serve as evidence labels, not trading-action labels.The appendix expands the compact anchor summary with label spaces, deterministic computation rules, and strong-signal indicators.Source: A Detailed Benchmark Specification · p. 11
- Table 5 counts only directional or extreme anchor labels as strong signals; ECI fires when weak evidence receives a high-confidence directional action.Weak evidence is defined as S(F) = 0.Source: Table 5 · p. 12
A.1 Execution Records and Re-run Consistency
- Deduplication changes UCR, RCI, and ECI by at most 0.4 percentage points and NDR by at most 0.005, with model rankings unchanged.The consistency check compares aggregate metrics with and without paired chart-only baseline executions.Source: §A.1 · p. 12
A.2 Strong-Signal Rules for Evidence–Confidence Evaluation
Evidence–confidence evaluation distinguishes directional evidence from descriptive context, penalizing high-confidence directional actions when the chart evidence is weak.
A.2 Strong-Signal Rules for Evidence–Confidence Evaluation
- ECI penalizes high-confidence directional actions when evidence is weak, with weak evidence defined as S(F) = 0.Only directional or extreme anchor labels count toward the strong-signal count; descriptive anchors can verify claims without independently justifying high-confidence directional action.Source: §A.2 · p. 12
- MA alignment, volatility, and recent 5-day change are excluded from S(F) as descriptive or contextual signals, though they remain available for UCR claim verification.MA alignment may overlap with trend direction, while volatility is not directional by itself.Source: §A.2 · p. 12
- 63.6% of parsed actions are HOLD or UNCERTAIN, motivating coverage-aware scoring because low unsupported-claim rates can coincide with limited directional action.The reported distribution is over successfully parsed actions.Source: Figure 8 · p. 12
A.3 Benchmark Inventory and Output Distribution
The inventory describes E2A-Bench’s trimodal query and scoring coverage, response claim matching, final-action distribution, and ten OHLCV-derived evidence anchors.
A.3 Benchmark Inventory and Output Distribution
- 323 stocks across three modalities yield 969 queries and 19,380 completed response cells across 20 models, with up to 19 scoring slots per response.Each response has up to 16 claim-grounding judgments and three response-level judgments for reasoning–action consistency, evidence–confidence consistency, and directional coverage.Source: §A.3 · p. 12
- 89.2% of 19,380 responses contain at least one anchor-matched claim, averaging 2.80 matched claims per response (median 2); unmatched ontology expressions are not labeled unsupported.The remaining 10.8% contain no matched claim.Source: §A.3 · pp. 12–13
- Among successfully parsed actions, HOLD and UNCERTAIN account for 63.6%; this motivates coverage-aware scoring because low unsupported-claim rates can coexist with limited directional action.Figure 8 shows the final-action distribution over successfully parsed actions.Source: §A.3 · p. 13
A.4 Full Specification of Evidence Anchors
- The benchmark uses ten deterministic OHLCV-derived evidence anchors for claim verification and evidence-strength estimation.Table 7 specifies the anchors and whether they count toward the ECI strong-signal count; trend direction uses a stricter threshold for strong-signal counting than for label assignment.Source: §A.4 · p. 13, Table 7 · p. 13
A.5 Human Validation of the Automatic Scorer
The authors validate the automatic scorer against expert annotations and assess how its conclusions align with holistic human judgments. They also document the language setting and planned artifact release.
A.5 Human Validation of the Automatic Scorer
- Annotators showed substantial agreement, with AC1 scores of 0.71 for UCR, 0.74 for RCI, and 0.82 for ECI; all three scorer diagnostics exceeded 85% F1.The scorer was evaluated against human-consensus labels.Source: §A.5 · p. 14
- NDR had the strongest correlation with holistic human ratings (ρ = 0.466, p = 0.009), and the main model-level conclusions remained robust to scorer uncertainty.The holistic pilot covered 30 stratified responses; 1,000 Monte Carlo trials perturbed scorer outputs across 19,380 responses.Source: §A.5 · p. 14
A.6 Prompt Template and Language Setting
- Prompts and model responses use Simplified Chinese; English translations are provided for readability, while metrics are computed on original outputs.The framework can be adapted by replacing the prompt template, response parser, and claim-pattern detectors.Source: §A.6 · p. 14, Figure 9 · p. 14
A.7 License
- The authors plan to release code, evaluation scripts, prompt templates, benchmark metadata, and rendered charts under the MIT License.If raw market-data redistribution is restricted, they will release construction scripts and source specifications instead.Source: §A.7 · p. 14
B Scoring Implementation Details
The scoring details define three local diagnostics and aggregate them with directional coverage into NDR. Evaluations use deterministic generation settings when supported.
B Scoring Implementation Details
- UCR flags claims that contradict or assert signals absent from matched evidence anchors; responses without verifiable claims receive UCR = 0 but are penalized through NDR coverage.Each extracted technical claim is mapped to one deterministic evidence anchor.Source: B Scoring Implementation Details · pp. 14–15
- RCI flags only explicit contradictions between directional rationale polarity and the final BUY or SELL action.Neutral or unknown rationales and non-directional actions are not counted as violations.Source: B Scoring Implementation Details · p. 15
- ECI flags a high-confidence directional action when no strong verified chart signal is present.Weak evidence is S(F) = 0, and high confidence is c ≥4 on the five-point scale.Source: B Scoring Implementation Details · p. 15
- NDR uses multiplicative aggregation because reliable action requires directional coverage, grounding, reasoning–action consistency, and evidence–confidence calibration.The product reaches zero if any required component fails completely, while remaining sensitive to graded changes.Source: B Scoring Implementation Details · p. 15
C.1 Model Access and Generation Settings
- Evaluations used temperature 0 for open-weight and financially adapted models, and the most deterministic available setting for proprietary APIs.Open-weight and financially adapted models ran with vLLM on 8 NVIDIA L20 GPUs.Source: §C.1 · p. 15
C.2 Thresholded NDR Analysis
Threshold and aggregation analyses examine NDR’s behavior, the complementarity of its local diagnostics, and its distinction from claim-level factuality.
C.2 Thresholded NDR Analysis
- Under θ = 0 to 2, leading models retain broad NDR ordering; at θ ≥3, all models lose substantial coverage and NDR rapidly approaches zero.Increasing θ changes the coverage threshold over existing response logs; the directional-conditioned chain-reliability terms remain fixed.Source: §C.2 · pp. 15–16, Figure 10 · p. 16
C.3 Non-Redundancy of Local Diagnostics
- Pairwise correlations among UCRdir, RCI, and ECI are weak (ρ = 0.26, 0.12, and 0.18), indicating largely non-redundant reliability diagnostics.The three diagnostics capture factual grounding, reasoning–action consistency, and evidence–confidence calibration.Source: Figure 11 · p. 16, §C.3 · p. 16
C.4 Sensitivity to Alternative Aggregations
- The multiplicative aggregation retains sensitivity to all four components, unlike the arithmetic mean’s coverage-blind compensation, the minimum’s discarded improvements, or Pareto dominance’s weak ranking resolution.The comparison uses the same response logs and alternative aggregation rules.Source: §C.4 · p. 16, Table 10 · p. 17
C.5 Contrastive Case: Claim-Level Factuality vs. Evidence-Backed Action
- In a chart-only case, Qwen3.5-9B’s BUY remained covered, rationale-consistent, and calibrated despite one unsupported claim, while Gemma-4-E2B-It’s HOLD had zero UCR but no directional coverage.The two models received the same input: stock 601799.SH and a 30-day OHLCV window ending 2023-06-30.Source: §C.5 · p. 17
C.6 Qualitative Case: Imperfect Claims with Reliable Action
A qualitative case illustrates that NDR can recognize an evidence-backed action despite an unsupported claim, while distinguishing it from conservative non-action.
C.6 Qualitative Case: Imperfect Claims with Reliable Action
- Qwen3.5-4B makes one unsupported claim, yet its SELL recommendation remains covered, coherent with bearish rationale, and supported by multiple verified evidence anchors.The example demonstrates that NDR is not a perfection-based hallucination score.Source: §C.6 · p. 17, Figure 12 · p. 18
- Gemma-4-E2B-It avoids verifiable claims and outputs HOLD, producing zero observed UCR but zero NDR because it has no directional coverage.NDR penalizes conservative non-action while recognizing actionable recommendations with largely intact evidence-to-action chains.Source: §C.6 · p. 17
- The comparison separates unsupported-claim counts from whether a directional recommendation is evidence-backed.Qwen3.5-4B’s SELL remains supported across coverage, rationale coherence, and verified signals despite its unsupported claim.Source: §C.6 · p. 17, Figure 12 · p. 18