PaperlayerView exact revision on arXiv

Source-linked AI summary

When Financial Fine-tuning Fails: A Three-Level Detectability Analysis of Numerical Hallucination in Domain-Adapted Language Models

Xiaodong Li, Peiwei Liu

arXiv:2609.04806v1cs.AI

TL;DR

This paper examines whether domain fine-tuning improves financial summarization reliability, where plausible numerical fabrications can carry high-stakes consequences. In controlled comparisons across three model variants, it finds that fine-tuning degrades numerical restraint across detectability levels, while numeracy supervision amplifies hallucination.

  • Problem

    Financial hallucinations can mislead investment decisions and evade traditional metrics when fabricated numbers remain plausible and professionally formatted.

    Source: §1.1 · p. 2, §2.1 · p. 6

  • Method

    The paper conducts controlled comparisons of Base, FT-A, and FT-A+B+C models to analyze numerical hallucination in financial summarization.

    Source: §7 · p. 46

  • Results

    5.4% L1 hallucination for Base versus 82.5% for FT-A and 90.8% for FT-A+B+C, with fine-tuning degrading restraint across all detectability levels.

    Source: §7 · p. 46

  • Takeaways & Limitations

    Financial summarization evaluation should cover all detectability levels, and deployment should include explicit grounding or abstention mechanisms.

    Source: §7 · p. 47

    The experiments use 7B-parameter models, and larger models may exhibit different hallucination dynamics.

    Source: §5.9 · p. 38

Abstract

from arXiv · show

Financial large language models are increasingly deployed for summarization of reports and disclosures, where numerical hallucination poses significant practical risks. While prior work often attributes such hallucination to insufficient numerical reasoning, this assumption has not been systematically tested under controlled fine-tuning settings. In this paper, we conduct a cost-effective, controlled study of numerical hallucination in financial summarization across three model variants: a base instruction-tuned model, a domain language-adapted model (FT-A), and a numeracy-enhanced domain model (FT-A+B+C). We introduce a three-level detectability taxonomy distinguishing between overt hallucination (currency-denominated fabrication), covert-explicit hallucination (professional-convention numbers), and covert-implicit hallucination (ungrounded quantitative claims). Our results reveal that domain fine-tuning substantially degrades numerical restraint at all detectability levels. While the Base model maintains near-zero hallucination rates (5.4\%), FT-A exhibits 82.5\% overt hallucination and FT-A+B+C reaches 98\%. Contrary to intuition, numeracy supervision amplifies rather than mitigates hallucination across all levels. We identify template injection---the insertion of memorized canonical values regardless of input content---as a primary hallucination mechanism in fine-tuned models. These findings demonstrate that numerical hallucination in financial summarization is driven by the degradation of numerical restraint through domain adaptation, not by insufficient numerical reasoning. We recommend that evaluation protocols assess hallucination across all detectability levels and that deployment practices include explicit mechanisms for grounding-aware generation or abstention.

1. Introduction

The paper asks whether financial domain fine-tuning improves summarization reliability, distinguishing numerical competence from restraint through a three-level hallucination taxonomy. Its findings preview substantial restraint degradation after fine-tuning, including when numeracy supervision is added.

1.1. Background and Motivation

  • Financial summarization is high-stakes because fabricated revenue, growth, or cash-flow figures can mislead investment decisions, violate regulations, and erode trust.The paper recommends multi-level assessment, grounding-condition routing, template-injection monitoring, and deployment decision frameworks.Source: §1.1 · p. 2, §1.3 · p. 4

1.2. Problem Statement

  • The paper frames the restraint gap as divergence between numerical competence and discipline: fine-tuning may improve computation while weakening abstention from ungrounded numbers.Numerical discipline is measured as the complement of L1 hallucination under weak-grounding S0 conditions.Source: §1.2 · p. 3

1.3. Contributions

  • The taxonomy distinguishes currency-denominated fabrications, professional-convention numbers, and ungrounded quantitative claims, exposing blind spots in narrower evaluations.It enables assessment across evaluation stringencies.Source: §1.3 · pp. 3–4
  • Template injection—memorized canonical values inserted regardless of input—appears in up to 37–50% of fine-tuned outputs across unrelated inputs.The paper presents this as a primary mechanism, with canonical financial narratives overriding evidence-based generation.Source: §1.3 · p. 4, §1.4 · p. 5

1.4. Key Findings Preview

  • 5.4% L1 hallucination for Base versus 82.5% for FT-A and 90.8% for FT-A+B+C; fine-tuning degrades restraint at all detectability levels.The Base model’s near-zero rate contrasts with both fine-tuned variants.Source: §1.3 · p. 4, §1.4 · p. 5
  • Numeracy supervision raises hallucination over FT-A by 7.9 points for L1, 11.6 for L2, and 9.5 for L3.FT-A+B+C improves arithmetic competence under constrained evaluation, yet its hallucination rates are higher at every detectability level.Source: §1.3 · p. 4, §1.4 · p. 5

2. Related Work

Prior work leaves limited evidence on numerical hallucination in domain-adapted financial generation, especially when fabrications remain plausible and professionally formatted. This paper situates that gap within hallucination, numerical grounding, domain adaptation, and evaluation design, then specifies controlled comparisons and detection procedures.

2.1. Hallucination in Neural Text Generation

  • Both fine-tuned models hallucinate substantially more than Base at all three detectability levels; FT-A+B+C exceeds FT-A on L1 and L2.Figure 1 compares overt, covert-explicit, and covert-implicit hallucination across the model variants.Source: Figure 1 · p. 6
  • Traditional metrics can miss plausible, stylistically appropriate financial fabrications, and strict hallucination metrics substantially underestimate risk in domain-adapted models.The paper motivates detecting errors beyond overt factual or malformed-numerical expressions.Source: §2.1 · p. 6

2.2. Numerical Reasoning and Numerical Grounding

  • Numerical reasoning benchmarks measure gains when arithmetic is directly required, but numerical restraint—knowing when grounding is insufficient—has received less attention.The paper reports that enhanced numerical competence alone does not ensure safer generation and may encourage ungrounded numerical output.Source: §2.2 · p. 7

2.3. Domain Adaptation for Financial Language Models

  • Prior financial adaptation work emphasizes fluency and task performance, while this study evaluates hallucination under fine-tuning and finds canonical narratives favored in open-ended generation.The paper’s analysis complements work on external grounding and task-appropriate financial safety evaluation.Source: §2.3 · pp. 7–8

2.4. Evaluation Sensitivity and Metric Design

  • Evaluation design can invert perceived safety when strict currency-based metrics are contrasted with semantic numerical evaluation.The paper argues that deployment-relevant evaluation should reflect real-world risks.Source: §2.4 · p. 8

3.2. Data Volume and Confound Considerations.

  • The study compares Base, FT-A, and FT-A+B+C, using financial language, QA, and numeracy data; its 3,500-sample versus 1,000-sample contrast creates a volume confound.The volume-matched FT-A×3.5 ablation uses Category A alone to separate content type from volume; its L1 rate is 64.6% versus 82.5% for FT-A, while FT-A+B+C reaches 90.8%.Source: §3.1 · p. 9, §3.2.4 · p. 10

3.3. Evaluation Dataset and Grounding Conditions

  • Evaluation covers 240 curated financial-summarization samples, stratified by numerical grounding (weak or strong) and source authenticity (160 synthetic, 80 real-world); the study also presents a three-level detectability-based taxonomy.Source: §3.3 · p. 11, §3.3.1 · p. 11, §3.3.2 · p. 12

3.5. Statistical Significance Testing

  • For L1, all pairwise model differences are significant after Bonferroni correction, including FT-A versus FT-A+B+C (46 discordant pairs, p = 0.0051).The paired comparisons use McNemar’s test on the same evaluation samples.Source: §3.5 · p. 14, Table 4 · p. 14

4. Experimental Results

Under strong output constraints, FT-C shows numerical competence that the Base model lacks, yet financial language fine-tuning degrades summarization reliability. The evaluation stratifies hallucination by detectability and input grounding to examine that gap.

4.1.1. Numerical Evaluation Results

  • 32% tolerance accuracy for FT-C contrasts with 0% exact-match and tolerance accuracy for the Base model across 50 financial calculations.Both models had a 100% parse rate, so the reported difference concerns accuracy rather than syntactic interpretability.Source: §4.1.1 · p. 16, Table 5 · p. 16

4.1.2. Implications

  • The constrained numerical results rule out arithmetic incompetence as the primary cause of downstream hallucination and show that numerical correctness alone is insufficient as a safety mechanism.The contrast suggests hallucination arises under weak or ambiguous grounding rather than an inability to compute.Source: §4.1.2 · p. 16

4.2. Effects of Financial Domain Language Fine-tuning (FT-A)

  • FT-A summarization quality falls from 4.7 to 1.6 on a 5-point scale, primarily because it fabricates revenue, cash-flow, or growth figures absent from the input.By contrast, FT-A improves structured extraction formatting and coverage, with more structured and comprehensive risk-factor outputs.Source: §4.2 · p. 17, Table 6 · p. 17
  • Financial-corpus narrative priors can independently induce numerical hallucination in FT-A, even without numeracy supervision.The model shifts from evidence-preserving compression toward generation of a canonical financial narrative.Source: §4.2 · p. 17

4.3.1. Input Grounding Conditions

  • The evaluation uses a three-level detectability taxonomy and separates weak-grounding S0 inputs from strong-grounding S1 inputs in a 3 × 2 analysis.S0 inputs lack absolute numerical quantities; S1 inputs contain explicit absolute values for verification.Source: §4.3 · p. 17, §4.3.1 · p. 18

4.4.1. Overall Hallucination Rates

  • The experimental analysis compares hallucination rates across detectability levels, grounding conditions, and model variants, using 240 evaluation samples.The supplied results text flags the Base model's elevated L3 rate as largely faithful paraphrase that triggers the semantic matcher, with actual ungrounded L3 hallucination near zero.Source: §4.4 · p. 18, §4.4.1 · p. 18

limitations.

Fine-tuning is associated with pervasive numerical fabrication across detectability levels, grounding conditions, and model variants. The reported mechanism is template injection: memorized financial values appear regardless of input content.

limitations.

  • 82.5% L1 hallucination for FT-A and 90.8% for FT-A+B+C contrast with 5.4% for the Base model.The findings characterize domain adaptation as destroying the Base model's numerical restraint.Source: limitations · p. 19
  • FT-A+B+C exceeds FT-A by 7.9 percentage points for L1, 11.6 for L2, and 9.5 for L3 hallucination.The paper reports that enhanced numerical competence does not translate to numerical discipline.Source: limitations · p. 19

4.4.2. Impact of Grounding Conditions

  • Under weak-grounding S0, L1 hallucination is 96.7% for FT-A and 90.0% for FT-A+B+C, while the Base model maintains 0%.With strong S1 grounding, FT-A's L1 rate is 58.8–90.0% and FT-A+B+C's is 85.0–95.0%, depending on source.Source: §4.4.2 · pp. 19, 21
  • Real-world 10-K excerpts trigger substantially higher L2 and L3 hallucination than synthetic inputs for both fine-tuned models.The paper suggests authentic financial discourse activates stronger canonical narrative priors.Source: §4.4.2 · p. 21

4.4.3. Aggregated View by Grounding Condition

  • Across grounding conditions, the Base model remains near-zero in hallucination while FT-A and FT-A+B+C show pervasive fabrication.L1 and L2 are detected independently, so currency expressions and their bare magnitude components can trigger both levels in the same output.Source: Figure 2 · p. 19, §4.4.3 · p. 22

4.5. Template Injection: A Primary Correlate of Hallucination

  • Fine-tuned models inject memorized canonical values regardless of input content, with distinct template libraries across FT-A and FT-A+B+C.The example replaces the input's 142% growth with template values of 11–14% and includes fabricated absolute monetary and semantic claims, while the Base model preserves the percentage without numerical fabrication.Source: §4.5 · p. 22, §4.5.1 · p. 22, §4.5.2 · pp. 22, 25

4.6. The Restraint Gap: Numerical Competence vs. Numerical Discipline

  • Numerical competence and numerical discipline diverge: FT-C reaches 32% tolerance accuracy on calculations, while fine-tuned models show 90–100% L1 hallucination under weak grounding.The paper describes this loss of restraint as a restraint gap; the Base model maintains near-zero L1 hallucination under S0.Source: §4.6 · p. 25

4.7. Sensitivity Analysis

  • Sensitivity analysis compares hallucination across data sources and input characteristics.The supplied passage introduces analyses of source and input characteristics but does not state their outcomes here.Source: §4.7 · p. 25, §4.7.1 · p. 25

1. Base model shows elevated rates on real-world S1 inputs.

The Base model's elevated L1 rate on real-world S1 inputs is attributed to format-matching artifacts, not genuine hallucination.

1. Base model shows elevated rates on real-world S1 inputs.

  • 35.0% L1 hallucination for the Base model on real-world S1 excerpts is attributed to format-matching artifacts.The rate reflects faithfully reproducing grounded source values in currency-denominated form.Source: §1 · pp. 25–26
  • Manual inspection confirms these flagged Base outputs do not constitute genuine hallucination.Source: §1 · pp. 26–27
  • Currency expressions such as “$X million” can trigger L1 detection even when the reproduced numerical values are grounded in the source.The passage identifies this as a limitation of purely automatic L1 detection on authentic financial text.Source: §1 · p. 27

2. FT-A shows substantially higher hallucination on real-world inputs.

Sensitivity analyses find elevated hallucination across input sources and lengths, with real-world text amplifying FT-A's L1 rate and template patterns closely associated with hallucination. A separate FT-A×3.5 failure mode is repetitive degeneration.

4.7.3. Template Injection Correlation

  • Among FT-A outputs, 84.3% of the 236 containing template patterns show L1 hallucination, versus 0% of the four without detected templates.The paper cautions that the no-template group is small; for FT-A+B+C, 48.5% of 33 outputs without detected templates still show L1 hallucination, and 97.6% of 207 with templates do.Source: §1 · p. 28, §2 · p. 28, Table 13 · p. 28

4.7.4. Qualitative Observation: Repetition Degeneration in FT-A×3.5.

  • FT-A×3.5 shows repetitive degeneration on 100% of synthetic inputs (27/27), but it is largely absent from real-world inputs.The repeated content is predominantly grounded; this differs from FT-A and FT-A+B+C template injection, which produces plausible but ungrounded values. The paper attributes the pattern to output-format overfitting and notes repetition penalties can mitigate it at inference time.Source: §4.7.4 · pp. 28–29

1. Hallucination is not an artifact of input characteristics.

  • FT-A and FT-A+B+C maintain elevated L1 hallucination rates of 58.8–100% across input lengths and data sources.The paper concludes the restraint gap is not an artifact of those input characteristics.Source: §1 · p. 29

2. Real-world inputs amplify hallucination in models with partial restraint.

  • FT-A's S1 L1 rate rises from 58.8% on synthetic inputs to 90.0% on real-world 10-K excerpts, while FT-A+B+C records 95.0% and 85.0%, respectively.The paper notes synthetic-only evaluation may underestimate real-world risk for models with partial restraint; FT-A+B+C is near-saturated on both sources.Source: §2 · pp. 27, 29, §3 · p. 27

3. Template injection is strongly associated with hallucination.

Template presence is strongly associated with L1 hallucination in fine-tuned models, and hallucination is higher across detectability levels with numeracy supervision. Real-world inputs amplify hallucination relative to synthetic evaluation.

3. Template injection is strongly associated with hallucination.

  • For FT-A, L1 hallucination occurred in 84.3% of outputs with detected templates and 0.0% without them; the latter group contained only four outputs.For FT-A+B+C, outputs without detected templates still had 48.5% L1 hallucination (n=33).Source: §3 · pp. 29–30
  • Hallucination was amplified on real-world inputs, indicating that synthetic-only evaluation substantially underestimates real-world risk, especially for FT-A.The findings support evaluating models on authentic domain data.Source: §3 · p. 30

1. Fine-tuning substantially degrades numerical restraint at all detectability levels.

  • 82.5% for FT-A and 90.8% for FT-A+B+C are their L1 hallucination rates, while the Base rate is 5.4%.These rates contradict the assumption that currency-formatted fabrication is rare in domain-adapted models.Source: §1 · p. 30

2. Numeracy supervision amplifies rather than mitigates hallucination.

  • +7.9pp L1, +11.2pp L2, and +9.5pp L3 are the increases for FT-A+B+C over FT-A.The results state that improved arithmetic competence does not translate to improved numerical discipline.Source: §2 · p. 30

3. Weak numerical grounding (S0) represents a catastrophic vulnerability.

  • Under S0 conditions, fine-tuned models show 90–100% L1 hallucination, while the Base model shows 0.8%.S0 denotes inputs with weak numerical grounding in the surrounding findings.Source: §3 · p. 30

4. Template injection is a primary correlate of hallucination.

  • Template presence is associated with L1 hallucination in 84.3% of FT-A outputs and 97.6% of FT-A+B+C outputs.The reported examples include memorized values inserted regardless of input content.Source: §4 · p. 30

5. The Base model demonstrates that restraint is achievable.

The Base model maintains near-zero hallucination, showing that numerical discipline is achievable rather than inherently beyond model capability. The paper attributes its degradation to domain fine-tuning.

5. The Base model demonstrates that restraint is achievable.

  • Near-zero hallucination in the Base model indicates that numerical discipline is not inherently beyond model capability.The paper identifies domain fine-tuning as the source of its degradation.Source: §5 · pp. 30–31

5. Discussion

The discussion argues that domain fine-tuning degrades numerical restraint and that memorized templates help explain fabricated numbers. It connects this pattern to evaluation and deployment choices, while noting limitations of the analysis.

5.1. The degradation of Numerical Restraint

  • Domain fine-tuning degrades numerical restraint across detectability levels; the Base model’s near-zero hallucination contrasts with fine-tuned models’ severe rates.The paper frames the problem as lost restraint rather than insufficient numerical reasoning.Source: §5.1 · p. 31, §5.10 · p. 38

5.2. Template Injection as a Hallucination Mechanism

  • Fine-tuned models insert memorized canonical values regardless of context, with recurring phrases and model-specific templates linked to fabricated numerical statements.The paper says template injection helps explain prevalent overt hallucination.Source: §5.2 · pp. 31–32

5.3. The Summarization Prior Shift

  • Fine-tuned models favor canonical financial narratives, generating revenue, growth, or cash-flow content even when the source lacks it.This narrative prior is associated with stronger performance on structured extraction than on complete-narrative summarization.Source: §5.3 · p. 32

5.4. How Numeracy Supervision Relates to Amplified Hallucination

  • FT-A+B+C has higher hallucination than FT-A across all three levels despite improved arithmetic competence; a volume-matched ablation attributes the amplification to numeracy supervision rather than data volume.The ablation reports 64.6% L1 for FT-A×3.5 versus 82.5% for FT-A, while FT-A+B+C reaches 90.8%.Source: §5.4 · pp. 32–33

5.5. The Restraint Gap

  • Numerical competence and discipline are distinct: models can improve at arithmetic while becoming less reliable at summarization; Figure 5 depicts the restraint gap.Numeracy training improves fluency without adding signals for when numerical generation should be avoided.Source: §5.4 · pp. 32–33, §5.5 · p. 33, Figure 5 · p. 34

5.6. Three distinct failure modes

  • The discussion distinguishes template injection, repetition-induced omission, and a format-matching detection artefact as separate failure modes.Low hallucination rates for FT-A×3.5 can reflect omission rather than restraint.Source: §5.6 · pp. 34–35

5.7. Implications for Evaluation Methodology

  • L1-only and L1+L2 evaluation miss some covert or implicit hallucination, while full L1+L2+L3 coverage requires semantic understanding; the authors recommend reporting each level.This makes fully automated comprehensive evaluation challenging.Source: §5.7 · p. 35

5.8. Practical Deployment Recommendations

  • The discussion recommends multi-level and S0 stress testing, template audits and filters, grounding-aware safeguards, and hybrid or retrieval-augmented architectures.It also notes detection-method limitations, model-scale and training-data scope, and the focus on financial summarization; the summary calls for grounding-aware generation or abstention.Source: §5.8.1 · p. 36, §5.8.2 · p. 36, §5.8.3 · p. 37

6. Limitations and Future Work

The paper limits its conclusions to a particular detection approach, model scale, training setup, and financial-summarization task. It outlines future work on improved detection, restraint-aware training, mitigation, and broader validation.

6.1. Detection Methodology Limitations

  • L2/L3 detection relies on patterns and heuristics that can miss subtle or coincidental cases, motivating more principled evaluation methods.The paper identifies entailment models and hybrid human-machine evaluation as possible directions.Source: §6.1 · p. 39

6.2. Template Injection Mechanism

  • Template injection is identified as a primary correlate, but its mechanism remains descriptive rather than explanatory.Future work should investigate how templates are learned, triggered, and suppressed.Source: §6.2 · p. 40

6.3. Training Objectives for Numerical Restraint

  • The study uses supervised fine-tuning objectives that model what to generate, not whether numerical generation should occur.The authors propose restraint rewards, contrastive abstention examples, and confidence-calibrated generation as research directions.Source: §6.3 · pp. 40–41

6.4. Mitigation Strategies

  • The cost-effective fine-tuning setup limits exploration of mitigation strategies; proposed options include hybrid routing, retrieval grounding, contrastive decoding, constrained decoding, and post-generation verification.The paper presents these strategies as promising directions, not validated interventions in this study.Source: §6.4 · pp. 41–43

6.5. Model Scale and Architecture

  • Experiments use Mistral-7B-Instruct-v0.2, so generalization to larger models remains uncertain despite the practical relevance of 7B-scale findings.Larger models could show greater restraint or produce more sophisticated fabrications; systematic cross-scale analysis remains future work.Source: §6.5 · pp. 43–44

6.6. Domain Generalization

  • The study focuses on financial summarization; proposed extensions to healthcare, legal, and scientific reporting are possibilities to test, not established findings.The paper suggests applying the L1/L2/L3 framework in those domains to assess generality.Source: §6.6 · pp. 44–45

6.7. Real-World Deployment Validation

  • The curated dataset contains 240 samples, and the authors call for larger-scale deployment studies to test findings under higher volume, more diverse, or unusual inputs.Future validation could incorporate user feedback on output reliability and downstream decision quality.Source: §6.7 · p. 45

6.8. Summary

  • The paper concludes that reliable domain-adapted summarization requires discipline alongside competence, with future work spanning restraint-aware training, detection, mitigation, and evaluation.It leaves this research agenda for subsequent work.Source: §6.8 · pp. 45–46

7. Conclusion

The study finds that fine-tuning degrades numerical restraint, numeracy supervision further amplifies hallucination, and template injection is a primary mechanism. It argues for grounding or abstention mechanisms and documents code availability.

7. Conclusion

  • Base records 5.4% L1, 3.8% L2, and 17.9% L3* hallucination; FT-A records 82.5% L1, and FT-A+B+C records 90.8% L1.The conclusion characterizes fine-tuning as associated with pervasive fabrication across all evaluation criteria.Source: §7 · p. 46
  • Fine-tuned models insert memorized canonical values regardless of input content, making template injection a primary hallucination mechanism.The conclusion suggests domain adaptation strengthens narrative priors that compete with evidence-based generation.Source: §7 · p. 46
  • The paper distinguishes numerical competence from numerical discipline and introduces a three-level taxonomy to analyze evaluation blind spots.The conclusion describes competence and discipline as orthogonal capabilities.Source: §7 · p. 46
  • The authors recommend explicit grounding or abstention mechanisms for fine-tuned financial summarization, particularly under weak-grounding S0 conditions.They state that hallucination rates approach 100% under S0 and call for training, decoding, and evaluation advances that reward abstention.Source: §7 · p. 47

Data and Code Availability

  • The study’s code is publicly available, while raw financial documents are not redistributed because of licensing constraints.Synthetic-data scripts, evaluation protocols, and experimental configurations are provided for reproducibility.Source: Data and Code Availability · p. 47

Appendix A. Training Detail

The appendices detail a controlled fine-tuning setup and evaluation corpus, including synthetic financial tasks and real-world 10-K excerpts. Models use deterministic inference settings, and dataset sizes are reported for training and evaluation.

Appendix A. Training Detail

  • The training-detail appendix documents model configurations, fine-tuning procedures, and training settings used in the experiments.The experiments use Mistral-7B-Instruct-v0.2 with QLoRA and supervised fine-tuning, without reinforcement learning or preference optimization.Source: Appendix A · p. 51, Appendix A.1 · p. 52

Appendix A.1. Base Model and Fine-Tuning Method

  • Mistral-7B-Instruct-v0.2 is adapted with QLoRA, selected for resource-efficient generation-task performance while preserving base-model capacity at inference.The study uses supervised fine-tuning alone to attribute behavioral changes to data composition rather than optimization complexity.Source: Appendix A.1 · p. 52

Appendix A.2. Fine-Tuning Configurations

  • FT-A uses financial language tasks, while FT-A+B+C mixes language, financial knowledge QA, and numeracy tasks; all variants train for one epoch.FT-C is trained exclusively on numeracy-focused financial calculation tasks under strong output constraints.Source: Appendix A.2 · p. 52

Appendix A.3. Training Hyperparameters

  • Training uses 4-bit NF4 quantization with FP16 computation and runs on a single NVIDIA RTX A3000 GPU with 12GB memory.The appendix reports that all runs complete within several hours.Source: Appendix A.3 · p. 53

Appendix A.4. Inference Settings

  • Inference uses greedy decoding to produce deterministic outputs and avoid stochastic variation during evaluation.The maximum number of new tokens is task-dependent, typically 128–256.Source: Appendix A.4 · p. 53

Appendix B.1. Overview of Data Categories

  • Training data cover synthetic financial language tasks, conceptual financial QA, and calculation-focused tasks with known ground-truth values.The categories target domain-specific language and structure, financial concepts, and calculations such as ratios, growth rates, and margins.Source: Appendix B.1 · pp. 53–54

Appendix B.2. Real-World Financial Text Evaluation Set

  • An evaluation-only set of 80 real-world 10-K excerpts tests generalization to authentic disclosures under controlled numerical-grounding conditions.The excerpts are evenly split between S0 and S1 and are not used for training.Source: Appendix B.2 · p. 54

Appendix B.3. Dataset Sizes

  • Training uses 1,000 Category A, 500 Category B, and 2,000 Category C samples; evaluation uses 240 summaries: 160 synthetic and 80 real-world 10-K.Each evaluation source is split evenly between S0 and S1.Source: Appendix B.3 · p. 54

Appendix C. Supplementary Quantitative Results

Supplementary results break down hallucination by data source and grounding, report statistical comparisons, and examine representative individual samples. They show persistent hallucination in fine-tuned models, including when inputs provide numerical grounding.

Appendix C.1. Full Hallucination Rate Breakdown

  • Under S0_Synth, fine-tuned models show 100% L1 hallucination, while Base shows 0%.The authors call this the clearest demonstration of the restraint gap.Source: Appendix C.1 · p. 55, Table C.16 · p. 55
  • For S0_Real, fine-tuned models show L3 rates of 10.0% for FT-A and 17.5% for FT-A+B+C, while both have 0% L2 hallucination.The revised L2 protocol targets absolute magnitude expressions and excludes percentage-based quantities.Source: Appendix C.1 · pp. 55–56, Table C.16 · p. 55
  • On S1_Synth, FT-A hallucinates in 58.8% of samples and FT-A+B+C in 95.0%; on S1_Real, their L1 rates are 90.0% and 85.0%, respectively.Base’s 35.0% S1_Real rate is attributed to format-matching artifacts rather than genuine hallucination.Source: Appendix C.1 · pp. 55–56, Table C.16 · p. 55

Appendix C.2. Hallucination Rate Confidence Intervals

  • The FT-A versus FT-A+B+C L1 difference is statistically significant despite marginally overlapping confidence intervals (Fisher’s exact test, p = 0.015).The Base L1 upper bound (10.1%) does not overlap FT-A’s lower bound (77.6%; p < 0.001).Source: Table C.17 · p. 56, Appendix C.2 · p. 57
  • FT-A’s L1 rate falls from 96.7% under S0 to 69.2% under S1, while FT-A+B+C remains near-identical at 90.0% and 91.7%.The authors describe partial grounding protection for FT-A but not FT-A+B+C; Base’s S0 upper bound is 4.6%.Source: Appendix C.2 · p. 57, Table C.18 · p. 57

Appendix C.3. Sample-Level Results

  • Across sample types, Base shows no hallucination in synthetic samples, while both fine-tuned models consistently show L1 hallucination and template injection.When L1 occurs, L2 and L3 often co-occur, particularly in real-world samples.Source: Appendix C.3 · p. 58, Table C.19 · p. 58

Appendix C.4. Statistical Significance Tests

Pairwise tests find significant L1 differences between models. The appended detection analysis describes the L3 procedure and its limitations, alongside human agreement and automatic-detector error rates for all three levels.

Appendix C.4. Statistical Significance Tests

  • All pairwise L1 differences are significant: Base comparisons reach p < 0.001, and FT-A versus FT-A+B+C yields p = 0.015 after Bonferroni correction.The adjusted threshold is approximately 0.017 across three comparisons.Source: Appendix C.4 · p. 59, Table C.20 · p. 59

Appendix D.1. Overview

  • L3 detection targets ungrounded quantitative claims implying numerical relationships without explicit values, using semantic patterns and input-grounding verification.Unlike L1 and L2, L3 requires semantic pattern matching combined with grounding checks.Source: Appendix D.1 · p. 59

Appendix D.2. Semantic Pattern List

  • The L3 candidate-pattern list covers financial states such as revenue and income changes, stable debt or cash flow, margin changes, and performance claims.Patterns include terms such as “increased,” “remained stable,” and “outperformed.”Source: Appendix D.2 · p. 60

Appendix D.3. Grounding Verification

  • A matched L3 phrase is not flagged when its core state word appears in the source input, allowing faithful paraphrases to remain unflagged.The grounding check uses case-insensitive exact matching of the extracted verb or adjective.Source: Appendix D.3 · p. 60

Appendix D.4. Application by Grounding Condition

  • Under S0, any ungrounded quantitative state claim is L3; under S1, L3 is flagged only when L1 or L2 hallucination also occurs.The S1 rule treats implicit claims co-occurring with explicit fabrication as part of the same hallucination episode.Source: Appendix D.4 · p. 60

Appendix D.5. Limitations

  • L3 rates are conservative lower bounds because the pattern list is incomplete and grounding verification relies on lexical rather than semantic matching.The authors recommend human verification of a random L3 sample for downstream use.Source: Appendix D.5 · p. 61

Appendix D.6. Inter-Annotator Agreement and Detection Reliability

  • Human agreement is almost perfect for L1 (κ = 0.883), substantial for L2 (κ = 0.657), and moderate for L3 (κ = 0.451).The authors recommend treating L3 as a conservative lower bound and using human verification in downstream applications.Source: Appendix D.6 · pp. 61–62, Table D.21 · p. 61

Appendix D.6.1. False Positive and False Negative Analysis

  • The detector’s L1 recall is 98.9%, L2 precision is 95.1%, and L3 precision and recall are 67.9% and 54.3%, respectively.L2 false negatives concentrate in S0; L3 errors reflect implicit-claim subjectivity and Base-model paraphrase artifacts.Source: Appendix D.6.1 · pp. 62–63, Table D.22 · p. 62
Loading 2609.04806v1…