PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Regulatory Compliance Question Answering in Financial Services with Domain-Adapted Retrieval-Augmented Generation

Tobias Deußer, Abhishek Pillai, Aurelio F. Bariviera, Dhananjay Bhardwaj, Lorenz Sparrenberg, David Berghaus, Christian Bauckhage, Rafet Sifa

arXiv:2609.30009v1cs.CLcs.AI

TL;DR

Financial firms need compliance answers grounded in frequently changing rulebooks, but confidentiality often limits deployment to compact on-premise models. This paper adapts retrieval and generation around compact models; on ObliQA, retrieval improves directly measured Recall@10 and RAFT-LoRA improves RePASs, while transfer and grounding remain unresolved.

  • Problem

    Financial firms need accurate answers grounded in dense, frequently amended rulebooks, while confidentiality often confines deployment to compact on-premise models.

    Source: §1 · pp. 1–2

  • Method

    The paper domain-adapts a LegalBERT retriever through entailment and contrastive tuning with BM25 fusion, and compares prompted compact generators with RAFT-LoRA adaptation.

    Source: §1 · p. 2

  • Results

    On ObliQA, staged retrieval raises Recall@10 from 0.256 to 0.774, while RAFT-LoRA raises RePASs for every stably adapted model, most for the weakest.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that deployments across regulatory regimes should plan for regime-specific RAFT data, while retrieval remains important for transparency and traceability.

    Source: §5 · pp. 11–12

    RePASs alone cannot validate grounding: a closed-book model cited nothing and misstated obligations yet scored within 0.011 RePASs of the full pipeline.

    Source: §4.6 · p. 11, §5 · p. 12

Abstract

from arXiv · show

Financial institutions operate under dense, frequently amended rulebooks, and answering a compliance question correctly requires not only fluency but verifiable grounding in the authoritative text. Large language models are attractive for this task, yet the models that firms can realistically deploy on-premise are compact ones, and compact models hallucinate obligations. We study whether a carefully domain-adapted retrieval-augmented generation pipeline closes that gap. Our retriever is built in three stages on top of LegalBERT: entailment tuning that recasts question--passage matching as premise--hypothesis reconstruction, contrastive tuning with in-batch negatives, and score-level fusion with BM25. Our generator is a compact model (2B--12B parameters) served under 4-bit quantization, either prompted or adapted with retrieval-aware fine-tuning (RAFT) through LoRA. On ObliQA, a question-answering benchmark built from the Abu Dhabi Global Market rulebooks, the staged retriever raises Recall@10 from 0.256 to 0.774 and outperforms BM25 (0.678) and E5-large-v2 (0.758), the strongest general-purpose dense encoder we tested. RAFT-LoRA then improves the composite RePASs answer-quality score for every model we could adapt, with the largest gain on the weakest one. However, the adapted models do not transfer to Australian case-law questions, and a closed-book model that receives no passages at all scores within 0.011 RePASs of the full pipeline while producing answers that cite nothing and misstate obligations. The retrieval gain is therefore measured directly, the generation gain is a gain in RePASs rather than demonstrated grounding, and grounding itself requires an evaluation protocol that RePASs does not provide.

1 Introduction

Regulatory QA requires answers grounded in scattered, frequently changing rules, but institutions face practical constraints on using large hosted models. The paper evaluates a domain-adapted retrieval and generation pipeline, finding strong retrieval gains and improved RePASs scores while identifying transfer and grounding limitations.

Motivation

  • Rules can require reconciling facts scattered across chapters, while confidential questions can force firms to run quantized models of roughly 12B parameters or fewer on their own hardware.The authors argue improvements must therefore come from the pipeline rather than model scale.Source: §1 · pp. 1–2

Contributions

  • Recall@10 rises from 0.256 to 0.774, exceeding BM25 (0.678) and E5-large-v2 (0.758) with a three-stage LegalBERT retriever.The stages are entailment tuning, contrastive tuning, and score-level fusion with BM25.Source: §1 · p. 2
  • The authors rebuild ObliQA splits at document level because its published train, validation, and test splits share source documents.They quantify the resulting dataset-size cost.Source: §1 · p. 2
  • RAFT-LoRA improves composite RePASs scores for all three compact generators adapted stably, with the largest gain on the weakest model.Prompting effects vary by model and are often negative.Source: §1 · p. 2

Negative results

  • ADGM-trained adapters do not transfer to Australian case-law questions, and a passage-free model scores within 0.011 RePASs of the full pipeline despite ungrounded answers.The paper treats the closed-book result as evidence that RePASs alone does not measure grounding.Source: §1 · p. 2

2 Related Work

Prior work supplies datasets, retrieval techniques, and parameter-efficient adaptation methods for legal and regulatory NLP. This paper combines established components in a controlled financial-regulatory QA evaluation that varies retrieval and generation and includes transfer and closed-book tests.

2.1 Regulatory and Legal Question Answering

  • Regulatory QA often requires multi-passage aggregation because conditions and exceptions governing an obligation may appear elsewhere in the text.ObliQA pairs questions with ADGM provisions and introduced the RePASs obligation-aware answer metric.Source: §2.1 · p. 2

2.2 Sparse, Dense, and Hybrid Retrieval

  • BM25 handles fixed terminology but not paraphrase, while dense retrieval supports paraphrase; entailment, contrastive training, and sparse–dense fusion address complementary retrieval limitations.The paper sequences these three approaches rather than choosing among them.Source: §2.2 · pp. 2–3

2.3 Retrieval-Augmented Generation

  • RAG conditions generation on retrieved evidence, while RAFT trains generators to use retrieved context and abstain when oracle evidence is withheld.The authors adopt RAFT to target hallucinations from irrelevant retrieved passages.Source: §2.3 · p. 3

2.4 Parameter-Efficient Fine-Tuning

  • LoRA and QLoRA reduce adaptation costs; QLoRA quantizes the frozen base to 4-bit, enabling the paper’s RAFT experiments on a single GPU.The approach is framed as matching the on-premise setting described in the introduction.Source: §2.4 · p. 3

2.5 Positioning of this work

  • The components are established; the paper’s contribution is their composition under an on-premise budget and a controlled evaluation of both pipeline halves.The evaluation includes retrieval ablations, five compact generators, cross-genre transfer, and a closed-book control.Source: §2.5 · p. 3

3 Methodology

The system separates retrieval from generation so their contributions can be evaluated independently. To avoid passage-level train–test leakage in ObliQA, the authors rebuild document-disjoint splits and audit them for residual overlap.

3.1 Task and Pipeline

  • The pipeline maps a question to top-k regulatory passages, then generates an answer entailed by those passages and covering applicable obligations.Retrieval and generation are trained and evaluated separately; Figure 1 depicts the shared top-k context.Source: §3.1 · p. 3, Figure 1 · p. 4

3.2 Leakage-Free Data Construction

  • ObliQA’s published splits overlap in source documents, allowing training questions to expose passages that answer evaluation questions.The authors exclude this leakage by re-splitting at document level.Source: §3.2 · pp. 3–4
  • The 40 ADGM documents are assigned 80%, 15%, and 5% to train, validation, and test; multi-passage questions crossing splits are discarded, removing 989 question–passage pairs.All reported numbers use these new splits and are not directly comparable to results on the original ObliQA splits.Source: §3.2 · p. 4
  • The overlap audit found no exact or 85% fuzzy matches; 139 pairs flagged at 0.85 embedding cosine similarity were retained as distinct obligations phrased alike.For Australian LegalQA, the authors grouped questions by legal citation and split 80/20, retaining inspected near-duplicates as distinct questions.Source: §3.2 · p. 4

3.3 Stage 1: Entailment Tuning

The first retrieval stage adapts LegalBERT from legal-language knowledge toward query–passage matching by training it to reconstruct a claim from a supporting passage.

3.3 Stage 1: Entailment Tuning

  • LegalBERT knows legal vocabulary but its pretraining does not teach query–passage geometry, so the authors tune it on passage support for a claim.This reframes retrieval as deciding whether a passage supports a question-derived claim.Source: §3.3 · p. 4
  • Questions are rewritten as declarative claims, with the gold passage as premise and the claim as hypothesis.A rule-based mapping covers six question types—when, why, who, where, does, and how—with a generic fallback.Source: §3.3 · p. 5
  • With masking probability β = 0.8, the model reconstructs masked hypothesis tokens using the passage context, then the tuned encoder becomes a mean-pooled bi-encoder.The high masking rate discourages copying and encourages use of the premise.Source: §3.3 · p. 5

3.4 Stage 2: Contrastive Tuning

Contrastive tuning directly optimizes query–passage embeddings after entailment tuning, separating relevant from irrelevant passages in the embedding space. It uses in-batch negatives rather than mined hard negatives.

3.4 Stage 2: Contrastive Tuning

  • Stage 2 optimizes query–passage embeddings with multiple-negatives ranking loss over in-batch negatives.The loss uses cosine similarity and a temperature.Source: §3.4 · p. 5
  • The query: and passage: prefixes are retained at inference because mismatched prefixes degrade retrieval performance.Training used batch size 16, with 2 ObliQA epochs and 3 Open Australian LegalQA epochs selected using validation Recall@10.Source: §3.4 · p. 5
  • The batch supplies N −1 negatives per query at no additional cost, and the authors report more stable training than with mined negatives.They avoided mined negatives because semantically adjacent passages can also be valid answers, and development runs showed lower Recall@10 with mining.Source: §3.4 · p. 5

3.5 Stage 3: Hybrid Fusion and Context Assembly

The hybrid stage combines normalized dense and BM25 scores, then filters the ranked passages before they reach the generator. This balances semantic retrieval with exact terminology while limiting weak context.

3.5 Stage 3: Hybrid Fusion and Context Assembly

  • Dense retrieval under-weights exact identifiers, so the system linearly fuses per-query-maximum-normalized dense and BM25 scores.The fusion uses α = 0.8, keeping the tuned dense retriever dominant while allowing BM25 to break ties on exact terminology.Source: §3.5 · p. 5
  • The top 10 passages are filtered by a 0.7 score threshold and a 0.2 consecutive-score drop, while retaining at least one passage.A drop greater than 0.2 truncates the remaining ranked tail.Source: §3.5 · p. 5
  • Filtering weakly relevant passages limits obligations the generator is expected to cover under RePASs.The metric evaluates obligation coverage against the supplied context, so retrieval errors can penalize the generator.Source: §3.5 · p. 5

3.6 Generation with Prompting

The prompted generator is a frozen instruction-tuned model conditioned on the question and retrieved passages. It is tested with zero-shot and two few-shot strategies.

3.6 Generation with Prompting

  • The first generation variant conditions a frozen instruction-tuned model on the question and retrieved passages.The shared instruction asks models to synthesize every obligation in the supplied passages into one coherent answer.Source: §3.6 · pp. 5–6
  • The prompting strategies are zero-shot, few-shot with three worked examples, and few-shot with chain-of-thought instructions.One worked example demonstrates the correct response when the context does not answer the question; all strategies emit only the final answer.Source: §3.6 · p. 6
  • Decoding is greedy with a 512-token budget.Source: §3.6 · p. 6

3.7 Generation with RAFT-LoRA

RAFT trains generators on contexts assembled by the hybrid retriever, with teacher-generated targets, then adapts the base model using QLoRA. The authors characterize the recipe as efficient but do not establish behavior independent of its teacher or training contexts.

3.7 Generation with RAFT-LoRA

  • RAFT contexts contain the oracle passage plus distractors for 70% of questions and distractors only for the remaining 30%.Passage order is shuffled to suppress positional bias.Source: §3.7 · p. 6
  • GPT-4o-mini produces targets that model exhaustive obligation coverage in supplied passages and explicit abstention when obligations are absent.The teacher is prompted to use only supplied documents and state when they do not settle the question.Source: §3.7 · p. 6
  • RAFT sets contain 150–200 instances per corpus, under 1% of available training questions, and the authors do not claim teacher- or context-independent behavior.They present the recipe as one teacher pass and a few hundred optimization steps on a single GPU.Source: §3.7 · p. 6
  • QLoRA freezes the 4-bit NF4 base model and adds rank-16 LoRA adapters to query and value projections.Loss is computed only on reason and answer spans, masking instruction, question, and context tokens.Source: §3.7 · p. 6

3.8 Evaluation

Evaluation measures retrieval against annotated gold passages and scores generated answers against retrieved context, while separating RePASs components because they can move in opposite directions.

3.8 Evaluation

  • Retrieval is evaluated with Recall@10, MRR@10, and nDCG@10 against annotated gold passages.Source: §3.8 · p. 6
  • Each retriever stage starts from the previous one, and the hybrid adds BM25 with α = 0.8.Table 3 reports the staged retriever ablation; Figure 2 plots it.Source: Table 3 · p. 7
  • Entailment tuning is decisive on Australian case law, while contrastive tuning contributes more on the ADGM rulebooks.The ordering of stages is identical on both corpora, but their relative contributions differ.Source: Figure 2 · p. 7
  • RePASs scores answers against retrieved context, combining entailment, contradiction, and obligation coverage, whose terms can move in opposite directions.Obligation coverage checks whether answer sentences entail context sentences classified as stating obligations, using a 0.7 threshold.Source: §3.8 · pp. 6–7

4 Experiments

The experiments evaluate a compact-model RAG pipeline on ObliQA and Australian legal questions, testing retrieval, generation, cross-domain transfer, and whether measured answer quality depends on retrieval. The strongest retrieval results exceed standard baselines, while transfer and closed-book controls expose limits in adaptation and RePASs as a measure of grounding.

4.1 Setup

  • Generation uses five compact instruction-tuned models in 4-bit NF4 quantization; evaluation uses 150 fixed validation questions per configuration, while retrieval is exhaustive.The two-document ObliQA test split is held out and untouched.Source: §4.1 · p. 7

4.2 Retrieval

  • On ObliQA’s full validation split (3,733 question–passage pairs; 13,705-passage corpus), the hybrid scores 0.774 Recall@10 and 0.638 nDCG@10, above every baseline; E5-large-v2’s MRR@10 is 0.595 versus 0.594.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 Generation on ObliQA

  • Few-shot prompting is model-dependent on ObliQA: Qwen improves, but Teuken, Gemma-2, and DeepSeek-R1 degrade; chain-of-thought helps only Gemma-3 and DeepSeek-R1.For Teuken and Gemma-2, demonstrations sharply reduce obligation coverage without a compensating reduction in contradiction.Source: §4.3 · p. 8
  • RAFT-LoRA raises RePASs for all three models adapted stably, improving obligation coverage and reducing contradiction, though Teuken's entailment score falls.Gemma-2-2B benefits most; two of five generators could not be adapted reliably, and the unadapted DeepSeek-R1-Distill has the strongest single RePASs score.Source: §4.3 · pp. 8–9, Table 5 · p. 9

4.4 Cross-Domain Transfer

  • On Australian case law, zero-shot is the best prompted strategy for Qwen, Teuken, Gemma-2, and Gemma-3; few-shot and chain-of-thought degrade all four.DeepSeek-R1 instead prefers chain-of-thought, with 0.613 RePASs through obligation coverage.Source: §4.4 · p. 9
  • ObliQA-trained adapters mostly fail to transfer to Australian case law: only Teuken improves, while Qwen and Gemma-2 lose RePASs.The paper attributes the contrast to differences between short normative provisions and extended judicial reasoning; it recommends regime-specific RAFT data.Source: §4.4 · pp. 9–10, Figure 3 · p. 10, Table 6 · p. 10

4.5 Is Retrieval Doing the Work?

  • Closed-book Qwen scores 0.657 versus 0.668 with retrieval, and Teuken 0.662 versus 0.673, despite producing answers that cite nothing and misstate obligations.RePASs can understate the retrieval gap because generic answers have little to contradict; the authors say it should not rank systems alone.Source: §4.5 · p. 11, Table 7 · p. 11

4.6 Limitations

  • Generation scores use 150 validation questions per configuration, while closed-book controls use fewer; the re-split removes 989 multi-passage pairs and leaves grounding insufficiently measured.The two English public-source corpora do not test internal policies, tabular disclosures, or multilingual rulebooks.Source: §4.6 · p. 11

5 Conclusion

Domain-adapted retrieval substantially closes the retrieval gap for compact regulatory QA, while RAFT-LoRA improves in-domain RePASs scores without demonstrating grounding. The results also show that adapter benefits do not transfer reliably to Australian case law and that RePASs alone cannot validate compliance QA.

5 Conclusion

  • Recall@10 moves from one query in four to three in four after staged adaptation; RAFT-LoRA raises RePASs for every adaptable generator, most for the smallest.The paper characterizes the retrieval improvement as directly measured, but the generation improvement as improved in-domain answer behaviour rather than improved grounding.Source: §5 · p. 11
  • ADGM-trained adapters lose most of their benefit on Australian case law, so multi-regime deployments should plan regime-specific RAFT data rather than reuse adapters.The paper concludes that the adapter gains are in-domain only.Source: §5 · pp. 11–12
  • A passage-free model scores within 0.011 RePASs of the full pipeline despite citing nothing and misstating obligations, so RePASs alone cannot validate compliance QA.The conclusion calls for evaluation protocols that measure transparency and traceability.Source: §5 · p. 12

Related papers

Loading 2609.30009v1…