Source-linked AI summary
From Documents to Reasoning: A Validated Synthetic Data Pipeline and Semantic-Aware Fine-Tuning for Financial Numerical Reasoning
Lokendra Birla, Milind Savagaonkar, Visnu Srinivasan, Sowmya Rasipuram, Shubhashis Sengupta
TL;DR
Financial QA systems must reason over heterogeneous numerical documents, but performance consistency and strict answer matching remain limited. The paper proposes a validated synthetic-data pipeline, Expression Match Accuracy, and semantic-aware fine-tuning for smaller models. On ConvFinQA and generated data, the approach produced high-quality QA pairs and significant gains in financial numerical reasoning.
Problem
Financial QA involves heterogeneous documents and multi-step arithmetic, while few-shot performance is inconsistent and Exact Match mishandles equivalent expressions and formatting variations.
Method
The paper generates validated arithmetic-expression QA pairs and fine-tunes smaller language models with a loss combining cross-entropy, EMA, and semantic similarity.
Results
The approach produced high-quality question-answer pairs and significant gains on financial QA benchmarks, including ConvFinQA.
Takeaways & Limitations
Expression-based evaluation captures reasoning when arithmetic step formats vary, supporting the paper's synthetic-data and domain-adaptation strategy.
Takeaways & Limitations
The semantic similarity component is post-hoc and nondifferentiable, while the loss does not model operator-specific semantics such as commutativity or associativity.
Abstract
from arXiv · showhide
Financial question answering (QA) has emerged as a key benchmark for evaluating the performance of Large Language Models (LLMs) on domain-specific tasks involving complex data formats such as tables, charts, and rich textual narratives. While recent advancements have enabled models to reason across modalities and perform multi-step arithmetic operations, limitations remain in performance consistency, and evaluation reliability. In particular, standard evaluation metrics like Exact Match (EM) often fail to account for minor variations such as differences in units or formats, misleading performance assessments. In this work, we propose a comprehensive pipeline for improving financial QA systems through high-quality synthetic data generation and fine-tuning of smaller language models (SLMs) using Quantized Low-Rank Adaptation (QLoRA). Our pipeline includes aggressive data validation for synthetic question answer generation to ensure the relevance and correctness of synthetic question-answer pairs. We introduce a novel evaluation metric that matches answers computed from arithmetic expressions rather than ground-truth answers; providing a more accurate reflection of model reasoning capability. Furthermore, we propose a modified loss function that aligns predicted and reference expressions using semantic similarity, our novel evaluation metric and standard cross-entropy, resulting in improved performance. Experimental results on benchmark datasets, ConvFinQA demonstrate significant gains in QA accuracy after fine-tuning using synthetic dataset and proposed loss function.
1 Introduction
Financial QA requires models to reason over heterogeneous financial documents and perform precise multi-step arithmetic, while existing prompting and evaluation practices remain unreliable. The paper addresses these gaps with validated synthetic data, expression-based evaluation, and fine-tuning for smaller models.
- Financial QA combines tables, charts, textual context, multi-step reasoning, and precise arithmetic, making robust system development challenging.
- Manual construction of large financial QA datasets is costly because it requires skilled annotators to perform complex numerical reasoning.
- The pipeline generates arithmetic expressions rather than direct numerical answers because language models may fail to compute those answers reliably.
- EMA compares arithmetic expressions to handle mathematically equivalent answers and formatting differences that Exact Match can miss.
- Few-shot prompting produces poor and inconsistent performance on complex financial tasks, motivating domain adaptation with specialized training data.
- The proposed modified loss combines cross-entropy, EMA-based loss, and semantic similarity to align predicted and reference expressions during SLM fine-tuning.
2 Related Work
Prior financial QA research introduced diverse benchmarks and prompting or fine-tuning strategies, but evaluation commonly relies on strict exact or execution matching. This paper extends that work with expression-based matching and metric-aligned optimization for financial numerical reasoning.
- FinanceBench, FinQA, FinBen, and PIXIU broaden financial evaluation across multimodal reasoning, numerical arithmetic, diverse tasks, and instruction following.
- Existing benchmarks commonly use EM or Execution Accuracy, which can require exact agreement with a ground-truth answer or executable program.
- Chain-of-thought and Program of Thought prompting elicit intermediate reasoning, but few-shot outputs can depend strongly on examples and their ordering.
- Prior work also explored domain-specialized fine-tuning and teacher-student generation to reduce annotation needs and train smaller financial QA models.
- The paper introduces EMA and metric-driven loss optimization that aligns financial expression generation with semantic and arithmetic evaluation.
3 Proposed Methodology
The methodology combines expression-based numerical evaluation, validated synthetic data generation, and semantic-aware fine-tuning for financial QA. It generates and filters question-answer pairs from complex financial documents, then trains models with losses designed to preserve expression meaning.
- EMA Metric: Expression Match Accuracy (EMA) compares numerical answers computed from ground-truth and predicted arithmetic expressions instead of matching answer text.The expressions are evaluated before their final numerical answers are compared; an exact match receives 1, otherwise 0.
- Document Processing: The pipeline handles complex financial documents by converting PDFs to images, extracting tables and surrounding text, and aligning outputs from multiple extraction methods.It is designed for numeric, textual, and visual QA, with table alignment based on numerical, pattern, and sentence-level similarity.
- Question Validation: Synthetic questions are validated for well-formedness, financial relevance, relevance, coherence, and factual coverage before downstream use.Relevance, coherence, and factual coverage are scored on a 1-to-5 scale, and the validation process filters erroneous or low-quality questions.
- Question Validation: The generation pipeline includes arithmetic reasoning and independent checks of the question, relevant data points, derived approach, expression, and computed answer.The validation uses the pre-text, tabular data, post-text, generated question, arithmetic expression, and computed answer as its analysis context.
- Semantic-Aware Fine-Tuning: The semantic-aware loss combines token-level cross-entropy, semantic dissimilarity, and EMA-based loss to supervise expression generation.The approach targets token-order sensitivity and limited structural understanding while supporting semantically aligned outputs and tolerance to minor syntactic variation.
- Semantic-Aware Fine-Tuning: The loss has three stated limitations: post-hoc semantic similarity is nondifferentiable, operator-specific semantics are omitted, and invalid decoded expressions can propagate errors.These constraints limit end-to-end learning and robustness to operations such as commutativity or associativity.
4.1 Experimental Design
The experiments use FinQA and ConvFinQA to assess numerical reasoning over financial text and tables, including conversational context, while comparing open-source model configurations and QLoRA implementation details.
- Datasets: FinQA evaluates multi-step numerical reasoning over heterogeneous financial tables and text.
- Datasets: ConvFinQA extends financial reasoning to interdependent conversational questions that require cumulative context.
- Comparative setup: The study compares fine-tuned open-source models including Mistral-V0.3-7B, Llama3.1-8B, and Phi4-14B with models reported in prior work.
- Implementation: Experiments were conducted using two NVIDIA A100-SXM4 GPUs with 80 GB of memory each and 4-bit NF4 quantization.
4.2 Evaluation of Benchmarks with Expression Match Accuracy (EMA) Metric:
The ConvFinQA evaluation uses both Exact Match and Expression Match Accuracy, with development-set results showing improved accuracy under EMA.
- ConvFinQA contains 3,965 training samples and 542 development samples, and evaluation uses the development set because test labels are unavailable.
- Models show improved accuracy on ConvFinQA when evaluated with the EMA metric.
4.3 Comparative Analysis
The comparative analysis reports that the fine-tuned smaller models perform comparably to large language models and better than the smaller models from prior work, while the synthetic-data framework outperforms two existing frameworks.
- Model comparison: Fine-tuned smaller models perform comparably to large language models and better than the smaller models proposed in prior work.
- Model comparison: The experiments include Our-M-7B, Our-L-8B, and Our-P-14B, corresponding to fine-tuned Mistral-V0.3-7B, Llama3.1-8B, and Phi4-14B models.
4.4 Qualitative Assessment of the Synthetic Generated Question-answers
The synthetic-data assessment examines lexical diversity, semantic similarity, and numerical reasoning complexity, reporting that the framework generates more complex questions than reference datasets.
- Type-Token-Ratio (TTR): Type-Token-Ratio measures lexical diversity by dividing unique words by total words.
- Cosine Similarity Using TF-IDF: Cosine similarity compares generated questions with their source context using frequency-weighted TF-IDF vectors to approximate semantic similarity.
- Numerical reasoning complexity: The framework classifies numerical questions as direct retrieval, single-operation, or multiple-operation tasks.
- Numerical reasoning complexity: The framework generates more complex numerical reasoning questions than the reference datasets.
4.5 Fine-tuning Results
The experiments compare fine-tuning configurations using EM, EMA, standard cross-entropy, and semantic loss across several smaller language models. EMA outperformed EM, while semantic loss generally improved performance over cross-entropy but was less effective for Llama models.
- EMA outperformed EM in the fine-tuning experiments.Table 4 compares models evaluated with EM and EMA.
- Semantic Loss Function performed better than the cross-entropy function.The comparison is reported across the evaluated fine-tuning configurations.
- Semantic loss did not perform well for Llama models.The authors attribute this difference to Mistral and Phi being trained for mathematical problems, whereas Llama was trained for generic tasks.
- The experiments fine-tuned Mistral-v0.3-7B, Phi-4-14B, and Llama3.1-8B using QLoRA.
5 Conclusion and Future Work
The paper concludes that EMA, a validated synthetic-data pipeline, and a tailored loss function improve numerical reasoning in financial QA. Future work extends the approach to additional NLP tasks and explores RLAIF and broader cost-function applications.
- 5 Conclusion and Future Work: EMA captures model reasoning despite variations in arithmetic step formats.The metric is presented as a component of the synthetic data and fine-tuning strategy.
- 5 Conclusion and Future Work: Fine-tuning smaller language models with the custom loss function produced significant gains on financial QA benchmarks.The approach was validated on ConvFinQA and synthetic data generated by the proposed pipeline.
- 5 Conclusion and Future Work: The authors plan to extend the approach to summarization, sentiment analysis, and descriptive question answering.
- 5 Conclusion and Future Work: Future work also explores RLAIF and extending the cost function to improve answer relevance and coherence across other tasks.