PaperlayerView exact revision on arXiv

Source-linked AI summary

FinRegQA-EU: Corruption-Based Preference Data for Grounded EU Financial Regulatory Question Answering

Aulia Kharis Rakhmasari, Fan Yu, Alexander Hoyle, Elliot Ash

arXiv:2609.36856v1cs.CE

TL;DR

European financial regulation lacks a dedicated benchmark despite extensive official EBA and ESMA Q&A material. The paper builds an evaluation and preference-training pipeline from those sources, finding that SFT’s high judge scores conflict with its low citation F1, while DPO and GRPO offer a better trade-off among fine-tuned models.

  • Problem

    Existing evaluations focus largely on textbook-style financial QA and numerical problem solving, and no comparable resource isolates European financial regulation.

    Source: §1 · pp. 1–2

  • Method

    The paper builds an EBA- and ESMA-grounded benchmark and preference-training pipeline using pointwise scores from three judges and targeted regulatory corruptions.

    Source: §1 · p. 2, §4.1 · p. 7

  • Results

    SFT achieves the highest overall judge score but the lowest citation F1, while DPO and GRPO offer a better trade-off and the highest citation F1 among fine-tuned models.

    Source: §5.3 · p. 12, §6 · p. 13

  • Takeaways & Limitations

    The released benchmark and rule-based citation metric support citation-aware evaluation of models proposed for compliance use.

    Source: §6 · p. 13

    The fine-tuned policy is Llama 3.2, selected to match the candidate-answer generator’s model family.

    Source: §4 · p. 7

Abstract

from arXiv · show

Large Language Models (LLMs) struggle with region-specific factual knowledge, particularly in financial regulation. While benchmarks such as CFinBench, provide broad coverage of financial knowledge in other regions, no comparable resource exists for European financial regulation. We close this gap by proposing an end-to-end pipeline for evaluating and improving LLMs on European financial regulatory question answering. Our dataset is grounded in the official Q&A corpora of the European Banking Authority (EBA) and the European Securities and Markets Authority (ESMA). We evaluate candidate answers with a pointwise LLM-as-a-Judge protocol using three judges from distinct model families, retain only unanimously judged pairs, and sharpen the rejected side through a taxonomy of regulatory failure modes : law swaps, article swaps, hallucinated citations, and hallucinated text. By fine-tuning on the resulting preference pairs exposes a divergence at the core of our findings: supervised fine-tuning attains the highest judge score of any method yet got the lowest rule-based Citation F1. SFT is producing longer answers with three times as many citations, most of them unsupported. DPO and GRPO offer the better trade-off with more concise answers and the highest citation F1 among fine-tuned models. Standard LLM-as-a-Judge evaluation rewards citation density as evidence of grounding and cannot detect when those citations are fabricated. Thus, we detect a blind spot that matters wherever answers must be verifiable. We release the benchmark and evaluation stack at https://github.com/auliakharis/FinRegQA-EU.

1 Introduction

Existing financial and legal benchmarks do not isolate European financial regulation, despite authoritative EU regulatory Q&A being publicly available. The paper introduces an EBA- and ESMA-grounded benchmark and pipeline for evaluating and improving EU regulatory question answering.

1 Introduction

  • Existing financial benchmarks emphasize extraction and arithmetic, while generic legal benchmarks do not isolate financial regulation.CNFinBench covers financial regulatory compliance but is built around China’s regulatory regime.Source: §1 · pp. 1–2
  • The paper fills the missing European financial-regulation benchmark with a pipeline grounded in EBA and ESMA Q&A.It evaluates and improves LLMs on EU financial regulatory question answering.Source: §1 · p. 2
  • The benchmark scores accuracy, completeness, topic coherence, and citation quality, and uses regulatory failure modes to construct preference-learning data.The comparison evaluates SFT, DPO, Dr. DPO, and GRPO using judge scores and rule-based citation F1.Source: §1 · p. 2

2 Related Work

Prior work leaves EU financial regulation without a dedicated benchmark, while LLM-as-a-Judge approaches and preference-optimization methods offer relevant evaluation and training foundations. The paper motivates pointwise judging and model fine-tuning in light of known judge biases and citation-grounding failures.

Legal and financial QA benchmarks

  • CNFinBench [Ding et al.] evaluates Chinese financial regulatory compliance, while generic legal benchmarks do not isolate financial regulation and no equivalent EU resource exists.Financial benchmarks largely focus on numerical reasoning and extraction [Chen et al., a,b, Islam et al., Wang et al., a].Source: §2 · p. 2

LLM-as-a-Judge

  • Pointwise LLM judging avoids position bias in pairwise comparisons and yields dimension-level scalar scores for constructing preference pairs.Pointwise judgments are also more vulnerable to adversarial responses [Raina et al.].Source: §2 · p. 2

Judge agreement

  • Krippendorff’s α measures judge disagreement against chance-expected disagreement and accommodates multiple raters and missing verdicts.The text identifies it as appropriate for the paper’s three-judge ensemble.Source: §2 · p. 2

Citation grounding and attribution

  • Prior work reports that retrieval does not reliably ensure citations support generated claims, motivating the paper’s focus on model fine-tuning for citation-grounded output.The cited reports include commercial generative search engines and frontier models.Source: §2 · p. 3

Preference optimisation

  • DPO trains directly on preferred and rejected responses without a separate reward model, but learns little from rejected answers the model never produces.The cited discussion identifies this as a consequence of the gradient pushing probability away.Source: §2 · p. 3

3 Datasets

The dataset draws authoritative regulatory question-answer material from the EBA and ESMA, then structures it for evaluation and preference-pair construction. Candidate answers are judged, corrupted in targeted ways, and retained only when the quality gap is sufficiently distinguishable.

Sources

  • The dataset uses official EBA and ESMA Q&A repositories, whose regulator-issued answers address questions about specific EU financial-regulatory provisions.The EBA covers banking, payments, AML/CFT, and related legislation; ESMA’s Q&A concerns securities markets and consistent application of the EU Single Rulebook.Source: §3 · p. 5

Schema and Preprocessing

  • Entries are parsed into six fields, including legal act, topic, question, and ground truth, with legal acts normalized to official identifiers when available.The topic label is retained as published by the regulator.Source: §3 · p. 5

Dataset composition

  • Each source question is answered by two answerer models and scored independently by three judges.Table 2 reports judgments per answerer–judge pair and notes fewer judgments from GLM-4.7-Flash because of truncation in thinking mode.Source: Table 1 · p. 5, Table 2 · p. 5

Preference-Pair Construction

  • Two open-weight 70B answerers generate candidates that three judges score on four dimensions, combined into a weighted quality score favoring accuracy and citation quality.A candidate becomes preferred when it meets the preferred-score threshold; four targeted corruptions create corresponding less-preferred answers.Source: §3 · pp. 5–6
  • The corruption operations swap articles or laws, substitute hallucinated citations, or inject fabricated text into an otherwise correct answer.Pairs are retained only when the score gap exceeds a separability threshold, excluding subtle corruptions and borderline preferred candidates.Source: §3 · p. 6

4 Methods

The methods combine pointwise judging, corruption-augmented preference fine-tuning, and evaluation with both judges and rule-based Citation F1. Preference pairs are built off-policy from regulatory answers and teacher-model outputs.

4.2 Fine-tuning

  • The pipeline builds off-policy pairs from regulatory answers and teacher outputs, corrupts rejected responses, and trains SFT, DPO, Dr. DPO, and GRPO.DPO is also trained separately for each corruption strategy to isolate its contribution.Source: §4 · p. 7, §4.2 · p. 8

4.1 Pointwise Prompting with LLMs

  • Three open-weight judges independently score each answer on four 1–5 rubric dimensions, and their dimension scores are averaged.The dimensions are Accuracy, Completeness, Topic Coherence, and Citation Quality.Source: §4.1 · p. 7
  • Of 2,134 questions, 858 (40.2%) achieved unanimous judge agreement and 359 also met the strong-preference criterion; 100 pairs were held out for evaluation.The 359 qualifying pairs form the DPO training set.Source: §4.1 · p. 8

4.3 Evaluation Strategy

  • Evaluation uses a held-out test of 100 EU regulatory questions and reports judge scores, pairwise win rates, and rule-based citation metrics.The test questions are disjoint from training and validation data.Source: §4.3 · pp. 8–9

4.3.1 Evaluation Metrics: Citation F1

  • Citation F1 compares generated citations with references extracted from official answers, at both instrument and article granularity.The rule-based metric is deterministic, inexpensive, and directly discriminates fabricated citations from correct ones.Source: §4.3.1 · p. 9

5 Results

Across the three phases, candidate answers were often on-topic but less accurate, and unanimous-judge preferences emphasized accuracy and completeness. Fine-tuning exposed a divergence: SFT led on judge scores but had the lowest citation F1, while DPO and GRPO offered a better citation trade-off.

5.1 Phase 1: Candidate Answer Quality and Judge Reliability

  • Topic coherence scored 4.12–4.57, while accuracy scored 2.13–2.40 across both answerers and regulator subsets; Apertus-70B also trailed Llama-3.3-70B by 0.2–0.5 points per rubric dimension.The authors suggest the matched-scale gap may reflect differences in instruction-following post-training rather than model capacity.Source: §5.1 · p. 9
  • Judge agreement was strongest for accuracy (α 0.68–0.78) and completeness (0.70–0.74), but Qwen–GLM agreement on citation quality fell to 0.014.The authors cite this disagreement as motivation for complementing judge assessments with deterministic citation scoring.Source: §5.1 · p. 10, Table 6 · p. 10

5.2 Phase 2: Preference Pair Construction

  • Accuracy had the widest chosen–rejected score gap (1.998 points), followed by completeness (1.461), while topic coherence had the narrowest (0.928).The preference signal therefore emphasized correctness and completeness more than whether answers stayed on topic.Source: Table 7 · p. 11, §5.2 · p. 11
  • The unanimous-judge filter and targeted corruptions produced 359 preference pairs with a verifiable quality gap for Phase 2 training.Corruptions included law or article swaps and fabricated citations or text.Source: §5.2 · p. 11

5.3 Phase 3: Fine-Tuning and Evaluation

  • SFT gained the most overall judge score (+0.322), but its citation F1 was lowest (0.181); DPO and GRPO had the highest citation F1 among fine-tuned models (0.229).SFT also produced longer answers and more citations, which judges scored highly for citation quality despite the lower rule-based citation F1.Source: §5.3 · pp. 11–12, Table 10 · p. 12
  • In the per-corruption DPO ablation, law swap scored 2.027 overall and achieved citation F1 of 0.251; all four variants improved overall scores slightly over the 1.959 baseline.Win rates were close across corruption types, at 0.43–0.45.Source: Table 11 · p. 12, §5.3 · p. 12

6 Conclusion

The paper builds preference data from EBA and ESMA Q&A answers using three judges and four corruption strategies. Fine-tuning exposes a gap between judge scores and citation grounding, with DPO and GRPO offering a better trade-off than SFT.

6 Conclusion

  • The pipeline retains unanimously judged pairs and augments rejected answers with law swaps, article swaps, hallucinated citations, and hallucinated text.Source: §6 · p. 12
  • Answerer models score consistently high on topic coherence but struggle with accuracy and completeness.The conclusion characterizes staying on topic as easier than being correct.Source: §6 · pp. 12–13
  • SFT achieves the highest overall judge score but the lowest citation F1; DPO and GRPO are more concise and have the highest citation F1 among fine-tuned models.SFT answers carry nearly three times as many citations, most unsupported.Source: §6 · p. 13
  • In the per-corruption ablation, law swap yields the highest overall score and citation F1 of the four strategies.Source: §6 · p. 13
  • The authors release the benchmark and evaluation stack, including a rule-based citation metric, for citation-aware evaluation of models proposed for compliance use.Source: §6 · p. 13

A.1 Dataset Generation Result Statistics

The appendix reports pairwise inter-judge agreement and answerer-model score distributions, but the supplied captions do not state numerical results.

A.1 Dataset Generation Result Statistics

  • Table 12 reports pairwise inter-judge agreement using linear- and quadratic-weighted Cohen’s κ and Krippendorff’s α.Source: Table 12 · p. 17
  • The table identifies κlin and κquad as linear- and quadratic-weighted Cohen’s κ, respectively, and α as Krippendorff’s alpha.Source: Table 12 · p. 17
  • Figure 3 presents score distributions across answerer models; its caption gives no distribution values or comparison outcome.Source: Figure 3 · p. 17

B Prompts

Appendix B specifies prompts for generating regulatory answers and judging them against official EBA/ESMA answers, including scoring criteria and response instructions.

B Prompts

  • Appendix B provides the pipeline prompts, with curly-braced placeholders filled at inference time from evaluation-instance fields.Source: B Prompts · p. 18

B.1 Answer Generation Prompt

  • The answer-generation prompt conditions an EBA/ESMA-style response on the legal act, topic, subject matter, and background.Source: §B.1 · p. 18
  • The generation prompt requires answers to follow the identified legal act and relevant secondary instruments, addressing the specific subject matter rather than a broader question.Source: §B.1 · p. 18
  • It asks for specific legal citations after substantive claims and an explicit no-provision marker instead of fabricated citations when a provision cannot be identified.Source: §B.1 · p. 18
  • Generated answers are specified as 100–400 words of official-style prose, with instructions to avoid padding and repetition.Source: §B.1 · p. 18

B.2 Judge Prompt

  • The judge prompt compares a candidate with the official answer on accuracy, completeness, topic coherence, and citation quality, returning a structured JSON score.Source: §B.2 · p. 18
  • The rubric defines five-point scales for factual accuracy, coverage of key points, topic alignment, and citation specificity and correctness.Source: §B.2 · p. 19
  • Judge instructions require independent dimension scores, treat unsupported specific claims as accuracy violations, and distinguish specific legal citations from generic references.Source: §B.2 · pp. 19–20

C Examples of SFT vs GRPO

A held-out PSD2 example contrasts SFT and GRPO answers, while the judges rate the SFT answer higher on citation quality.

C Examples of SFT vs GRPO

  • The example asks whether PSD2 Article 97(1)(b) applies to pay-by-invoice when the payer’s funds are covered by a payment service provider’s credit line.Source: C Examples of SFT vs GRPO · p. 21
  • On the held-out question, SFT gives a longer, heavily cited answer that inverts the rule, while GRPO answers directly and correctly.Source: Figure 4 · p. 21
  • Despite the incorrect SFT answer, judges score it higher on citation quality than the GRPO answer.Source: Figure 4 · p. 21

D Corruption Strategies: Worked Example

The worked example applies five corruption strategies to one EBA Q&A item, keeping the regulator’s answer as preferred and corrupting only the rejected answer.

D Corruption Strategies: Worked Example

  • The five strategies target law swaps, article swaps, fabricated citations, unsupported regulatory text, and combinations of these failure modes.Table 13 summarizes which failure mode each strategy targets.Source: D Corruption Strategies: Worked Example · p. 22, Table 13 · p. 22
  • In a law-swap error, the article number remains unchanged while the cited instrument is replaced by a topically adjacent but incorrect regulation or directive, targeting plausible-article recall under the wrong instrument.Source: D Corruption Strategies: Worked Example · p. 22
  • An article swap preserves the cited instrument while replacing its article identifier with a different article in that same instrument, targeting citations to the correct law but wrong provision.Source: D Corruption Strategies: Worked Example · p. 22
  • Hallucinated citations append a fabricated regulation and article to an otherwise fluent, non-committal answer.The example adds a nonexistent regulation and Article 731.Source: D Corruption Strategies: Worked Example · p. 22
  • Hallucinated text inserts a plausible-sounding but unsupported regulatory claim without adding a fabricated citation.The example claims competent authorities may exempt branches with fewer than 50 employees.Source: D Corruption Strategies: Worked Example · p. 22
  • The combination strategy applies a random subset of the four individual strategies jointly, allowing multiple failure modes to co-occur.The reference answer remains the preferred side, while only the rejected side is corrupted.Source: D Corruption Strategies: Worked Example · p. 22

Related papers

Loading 2609.36856v1…