Source-linked AI summary
Synthesizing Behaviorally-Grounded Reasoning Chains: A Data-Generation Framework for Personal Finance LLMs
Akhil Theerthala
TL;DR
Financial advice systems face challenges in delivering robust, personalized guidance, and agentic approaches can carry substantial deployment costs. This paper builds behaviorally grounded supervision data and fine-tunes an 8B model; in held-out evaluation and a blind LLM-jury study, it performs on par with or sometimes exceeds larger baselines.
Problem
No model excels across all financial task categories, and robust performance often requires large, expensive models, constraining their practicality for financial advice.
Source: §1 · pp. 1–2Method
Results
Across held-out evaluation and a blind LLM-jury study, the 8B model is on par with or sometimes exceeds 27–32B baselines; it outperforms Gemma3-27B by 5% on BLEURT.
Source: §6 · p. 8Takeaways & Limitations
The authors’ data-centric framework offers a path toward smaller, more trustworthy personal-finance models within the scope of their evaluations.
Source: §6 · p. 8The primary weakness is factual hallucination, especially for jurisdiction-specific regulations and tax details, with errors more frequent in regulation-heavy domains.
Source: §4.3 · p. 7
Abstract
from arXiv · showhide
Personalized financial advice requires consideration of user goals, constraints, risk tolerance, and jurisdiction. Prior LLM work has focused on support systems for investors and financial planners. Simultaneously, numerous recent studies examine broader personal finance tasks, including budgeting, debt management, retirement, and estate planning, through agentic pipelines that incur high maintenance costs, yielding less than 25% of their expected financial returns. In this study, we introduce a novel and reproducible framework that integrates relevant financial context with behavioral finance studies to construct supervision data for end-to-end advisors. Using this framework, we create a 19k sample reasoning dataset and conduct a comprehensive fine-tuning of the Qwen-3-8B model on the dataset. Through a held-out test split and a blind LLM-jury study, we demonstrate that through careful data curation and behavioral integration, our 8B model achieves performance comparable to significantly larger baselines (14-32B parameters) across factual accuracy, fluency, and personalization metrics while incurring 80% lower costs than the larger counterparts.
1 Introduction
Personalized financial advice is difficult to deliver reliably: broad agentic systems face deployment and cost constraints, while users’ financial biases vary. This work proposes training data that integrates financial context and behavioral reasoning to support personalized advice.
1 Introduction
- Agentic financial advisors face technical, security, maintenance, and inference-cost constraints; one cited study found these factors kept systems below 25% of anticipated returns.Deployment performance can also deteriorate within months as real-world conditions change, and personalization is limited by the context available to an agent.Source: §1 · p. 2
- The framework integrates financial, behavioral, and psychological knowledge into training data to support personalized guidance, core financial reasoning, and recognition and mitigation of user-side biases.It makes inference of the user’s psychological state a standalone reasoning phase, motivated in part by findings that advisor persona influences trust and engagement.Source: §1 · p. 2
- The framework retrieves relevant financial context through a dedicated reasoning stage, since the authors say general advice often does not require real-time financial knowledge.They contrast this with agentic systems whose knowledge sources may require manual curation.Source: §1 · p. 2
- Users’ financial decision-making biases vary with age, experience, and location, yet many financial agents do not directly address them in their advice.The authors incorporate these biases into the reasoning chain so final responses can acknowledge and address them.Source: §1 · p. 2
- LLM juries validate each chain-of-thought stage; the resulting 19k-sample dataset is used to fine-tune Qwen-3-8B and compare it with similarly sized models.The paper also frames the data-centric framework as a possible backbone policy for agentic workflows, with evaluation of thinner planning chains and lower orchestration cost deferred to future work.Source: §1 · p. 2
2 Related Works
Prior work spans financial-task benchmarks and increasingly complex agentic systems, but the paper argues that these approaches leave gaps in holistic advice, deployment practicality, and behaviorally grounded training data.
2 Related Works
- Earlier automated financial advice used classic methods such as collaborative filtering and case-based reasoning for domains including loan and insurance policy recommendation.LLM research later expanded the range of capabilities and challenges.Source: §2 · pp. 2–3
- Benchmarks of general-purpose LLMs report improving accuracy on specific financial topics, but also persistent difficulty with complex queries and holistic financial advice.The paper characterizes off-the-shelf models as insufficient for nuanced holistic advice.Source: §2 · p. 3
- Agentic workflows address multi-step planning with specialized agents, but can entail architectural complexity and maintenance costs; confident-sounding agents may also earn trust despite lower-quality advice (Takayanagi et al., 2025a).The cited example of agentic crews combines roles such as data scientists and compliance checkers for financial modeling and risk management (Okpala et al., 2025).Source: §2 · p. 3
- The authors argue for a data-centric alternative because LLMs can amplify financial biases, and financial fine-tuning may exacerbate them; their framework constructs behaviorally grounded supervision data rather than scaling capacity or live data ingestion.The related-work discussion cites anchoring and overconfidence, product bias (Zhi et al., 2025), and advice that increases portfolio risk; it contrasts its dataset methodology with FinGPT-style real-time market-data updates.Source: §2 · p. 3
3 Dataset construction
The dataset combines real-world personal-finance questions with modular reasoning and response-generation stages. Its pipeline analyzes questions, retrieves financial and behavioral context, identifies psychological cues, formulates response instructions, and validates generated stages with LLM juries.
3.1 Data Collection and Processing
- 405k unique questions were filtered and 19k representative queries sampled across eight personal-finance categories; the full corpus remains available for scaling.Questions came from Reddit scenarios spanning topics such as debt consolidation, retirement planning, tax optimization, and insurance; the source data was publicly archived before June 2023 and de-identified.Source: §3.1 · pp. 3–4
3.2 Generation methodology
- The generation methodology separates chain-of-thought generation from response generation and uses four modular reasoning phases that can be independently rubric-checked and stitched into a coherent chain.Figure 1 depicts the four phases feeding final response generation, with LLM-jury validation at each phase.Source: §3.2 · p. 4, Figure 1 · p. 5
3.2.1 Query Analysis
- Query analysis decomposes potentially inconsistent or redundant user input into the primary conflict, principal people involved, and essential financial facts for the later reasoning stages.This is intended to keep subsequent processing aligned with the user’s inquiry.Source: §3.2.1 · p. 4
3.2.2 Context Analysis
- Context analysis uses modular retrieval to assemble relevant financial and behavioral evidence, then condenses retrieved material before passing it with the query to downstream reasoning.The self-curated financial and behavioral corpora contain approximately 600k and 300k tokens, respectively; retrieval, reranking, and condensation use a top-25, top-15 workflow.Source: §3.2.2 · pp. 4–5
3.2.3 Psychological Cue identification
- Psychological cue identification extracts sentiment, primary emotions, and certainty from the query, then generalizes communicative intent to guide the tone of the final response.The framework uses an LLM-based cue-identification process at scale.Source: §3.2.3 · p. 5
3.2.4 Response Formulation
- Response formulation consolidates information from the preceding phases into instructions that the response-generation stage must follow..Source: §3.2.4 · p. 5
3.3 Response generation
- Response generation uses the optimized stages’ financial context to formulate a conclusive answer in a tone suitable for the user..Source: §3.3 · p. 5
3.4 Data Validation
- LLM juries assess and rank generations for each phase, selecting the highest-ranked response for subsequent generation jobs.The authors use Gemini-2.0-Flash and o4-mini in a three-shot evaluation framework.Source: §3.4 · pp. 5–6
4 Evaluation
Held-out reference-based and blind-judge evaluations find the fine-tuned 8B model competitive with substantially larger systems, alongside recurring factual-grounding weaknesses and lower operating costs. Results also show strong structure and personalization, with limitations concentrated in regulation-heavy advice.
4.1 Quantitative Evaluation
- The 8B model achieves semantic accuracy comparable to Gemma3-27B/12B and Mistral-24B, and exceeds them by approximately 3–5% in human-likeness and fluency.These metrics were assessed against ground truths on a 500-query held-out dataset.Source: §4.1 · p. 6
4.2 Qualitative Evaluation
- Blind judges rank anonymized responses for financial accuracy, plausibility, and relevance; overall agreement is τ = 0.64 and ρ = 0.79.The authors interpret the higher ρ than τ as indicating mostly local swaps rather than wholesale ranking changes.Source: §4.2 · p. 7, Table 3 · p. 7
4.3 Qualitative Analysis and Error Patterns
- Inaccurate answers often co-occur with degraded reasoning, but relevance can remain high despite factual errors or poor reasoning.The authors identify hallucinated jurisdiction-specific regulations and tax details as the primary weakness, especially in taxes and insurance.Source: §4.3 · p. 7
4.2 Qualitative Evaluation
- On 504 unseen subreddit queries, the 8B system outperforms other sub-14B models and approaches 27B–32B leaders; overall, it achieves parity with models 2–3 times its size and cuts operational costs at least 80% versus baselines over 12B parameters.Source: Figure 2 · p. 7, §4.2 · p. 7, §4.4 · p. 8
4.3 Qualitative Analysis and Error Patterns
- Responses are typically structured with clear headers and sequential steps, extract user-specific details, and acknowledge emotional context before offering tailored advice.The paper reports that this emotional framing enhances perceived helpfulness.Source: §4.3 · p. 7
- The paper suggests targeted regulatory retrieval and calculation verification as the highest-marginal-improvement areas, while describing current utility as practical for non-critical advisory scenarios.This follows the authors’ characterization of factual grounding as the key bottleneck.Source: §4.3 · p. 7
4.4 Cost Analysis
- At $0.8 per hour and 34.15 seconds average inference time, the model is presented as enabling responsive advisory without prohibitive infrastructure requirements.Table 4 compares hosting, latency, and total operating expenses across models.Source: §4.4 · p. 8
5 Future Works
Future work will test paths to regional scaling, deployment in a thin agentic stack, and preference-based refinement with rule-based compliance safeguards.
5 Future Works
- Global scaling may use a US-optimized pipeline ported systematically or, if regional signals can be detected precisely, a shared backbone with lightweight regional experts.The proposed regional signals include currency symbols, policy terminology, and spelling conventions.Source: §5 · p. 8
- The planned thin agentic stack would let the core model resolve queries internally and invoke regulatory databases or fact-checking APIs only to resolve uncertainty.The authors plan to measure cost-latency trade-offs across regions.Source: §5 · p. 8
- The authors plan to test DPO/IPO preference optimization and rule-based layers for regulatory fidelity, bias mitigation, and tone consistency.Success would be evaluated through targeted safety, compliance-adherence, and user-trust measures.Source: §5 · p. 8
6 Conclusion
The conclusion presents a data-centric framework for an 8B finance model that matches or sometimes exceeds 27B–32B baselines, while noting unresolved scope and evaluation limits.
6 Conclusion
- The 8B model achieves semantic fidelity and human-likeness on par with, and sometimes exceeding, 27B–32B baselines in held-out evaluations and blind LLM judging.On the 500-query test, it outperforms Gemma3-27B by 5% on BLEURT and is competitive on BERTScore, with only a 2% difference.Source: §6 · p. 8
- The conclusion attributes the gains to explicit psychological cues, retrieval-augmented grounding, and a thin agentic execution layer.It says the modular design can support incremental extension, such as regional experts with minimal retraining.Source: §6 · p. 8
- Geographic scope, behavioral depth, and privacy safeguards remain limitations, and a precise cost-latency audit is deferred to future work.The paper frames the model as a cost-aware backbone for standalone personal-finance assistants and a potential alternative to monolithic cloud deployments.Source: §6 · p. 8
Limitations
The stated limitations concern query-source diversity, dataset coverage, shallow psychological analysis, and a scope restricted to core NLP tasks.
Limitations
- Queries come solely from Reddit, potentially missing other demographics and query formats; the 19k-sample dataset is described as proof-of-concept rather than broad scenario coverage.The authors call for more diverse data sources and a larger, more varied corpus.Source: Limitations · p. 8
- Psychological analysis derives basic sentiment from phrases rather than richer indicators such as risk tolerance or financial stress.The paper suggests specialized surveys or transfer learning from clinical datasets as possible routes to improved indicators.Source: Limitations · p. 8
- The framework excludes tasks beyond core natural language processing, including multimodal processing and reasoning capabilities.The authors identify these as areas for future work.Source: Limitations · p. 8
Data & Code Availability
The paper releases its dataset, fine-tuned model, and code publicly under the Apache 2.0 license.
Data & Code Availability
- The dataset, model, and code are publicly available under the Apache 2.0 license.The paper states that the data was de-identified, with personally identifiable information removed from Reddit sources.Source: Data & Code Availability · p. 9
- The released model is a fine-tuned Qwen-3-8B model.The paper provides its Hugging Face location.Source: Data & Code Availability · p. 9
- The release includes a 19k-sample reasoning dataset.The paper provides its Hugging Face location.Source: Data & Code Availability · p. 9
Ethical Considerations
The paper describes de-identification procedures and limits the system to educational use, with safety filters against harmful guidance.
Ethical Considerations
- Reddit posts are de-identified by removing usernames, links, metadata, and personally identifiable information, then lightly rephrased to retain the financial situation.No raw identifiers are stored or released.Source: Ethical Considerations · p. 9
- The system is for educational use only, not fiduciary or personalized financial advice.Prompts and filters forbid unsafe guidance such as evasion or guaranteed returns.Source: Ethical Considerations · p. 9
- Evaluation uses multiple LLM judges, with inter-judge agreement reporting and judge-swap checks intended to limit model-family bias.Source: Ethical Considerations · p. 9
Appendices
The appendices describe phased, reusable prompts for generation and explicit criteria for blind LLM-judge rankings. They detail query analysis, financial and psychological context analysis, and response-evaluation procedures.
A.1.1 Overarching principles
- The generation-stage prompts use a consistent, swappable structure across tasks, with a distinct persona suited to each phase.The stated purpose of role-playing is to prime relevant knowledge, tone, and task-specific behavior.Source: §A.1.1 · p. 12
A.1.2 Individual Phases
- Generation prompts classify each query into one personal-finance category by primary intent, with an option for Not_Applicable.Categories include scope and examples to guide classification.Source: §A.1.2 · p. 12
2. Query Analysis:
- Query analysis distills noisy or redundant inputs into the primary conflict, principal players, and essential financial facts.The appendix frames this as an initial step for handling inconsistent or obscured information.Source: §2 · p. 12
3. Context Analysis:
- The reasoning prompt requests an internal chain-of-thought Analysis Block and stakeholder impacts for each approach before the final response.This scenario-based analysis is described as covering financial and emotional consequences for relevant parties.Source: §3 · pp. 12–13
- Psychological analysis requires sentiment, emotion, or intent conclusions to cite specific words or phrases from the query.It is separate from the financial reasoning step.Source: §3 · p. 13
- The context-analysis prompts separate emotional state from objective financial analysis so the final response can synthesize both.The appendix also describes consolidating prior information into a rubric and highlighting key facts linked to parts of the query.Source: §3 · pp. 12–13
A.2 Prompt Guidelines for Evaluation through LLM-as-a-Judge
- The evaluation uses multi-shot listwise rankings of anonymized responses, converts ranks to Borda points, and averages scores across judges and queries.Higher aggregated scores are better.Source: §A.2 · p. 13
- Evaluation prompts define separate criteria for financial accuracy, plausibility, and relevance, including penalties for excessive verbosity or hard-to-follow reasoning.Relevance requires addressing query components, figures, constraints, and details without irrelevant additions.Source: §A.2 · p. 14
B Modular RAG for Context Analysis
The section describes a two-corpus context pipeline and a multi-criterion LLM-jury evaluation of the resulting personal-finance model. The 8B model is competitive on plausibility and relevance, mid-pack on absolute accuracy, and most parameter-efficient across the three criteria.
B Modular RAG for Context Analysis
- The context-analysis phase combines behavioral and financial corpora in one context pack: behavioral cues guide advice framing, while financial chunks ground its content.The corpora draw on peer-reviewed and practitioner psychology sources and practical personal-finance sources; retrieval re-ranks candidates from both indexes before LLM synthesis and handoff.Source: B Modular RAG for Context Analysis · pp. 14–15
- The context pipeline’s coverage and freshness depend on public-source snapshots; blogs may introduce style bias, and synthesis may favor well-structured sources.The described mitigations include preserving attributions, tracking snapshot timestamps, and preferring higher-priority sources when conflicts arise.Source: B Modular RAG for Context Analysis · p. 15
C.1 Score Definitions and Rationale
- Evaluation separates financial correctness, reasoning quality, and task alignment so fluent but unsafe answers are not rewarded by a single blended score.Accuracy ignores style and partial coverage; plausibility focuses on reasoning structure, while relevance checks coverage of the query and its constraints.Source: §C.1 · p. 15
C.2 Borda Points
- Borda scoring converts listwise ranks into points, uses every rank position, and treats successive rank gaps as uniform.The section also notes that scores can change when candidates are added or removed and that the method assumes equal intervals between ranks.Source: §C.2 · pp. 15–16
C.3 LLM-Jury Protocol
- The jury uses heterogeneous judges, multi-shot rubrics, shuffled anonymized lists, and averaged Borda scores; zero-shot judging can exhibit position, same-family, and prompt or leniency bias.Scores are averaged across repeats for each judge, then across judges and queries for each criterion and category.Source: §C.3 · p. 16
C.4 Overall Category Scores (Accuracy, Plausibility, Relevance)
- The proposed 8B is mid-pack on accuracy, exceeds Mistral-Small-24B on plausibility, and ranks 4/8 on relevance.The largest baselines lead across criteria, while the 8B remains ahead of several smaller baselines on accuracy and the remaining baselines on relevance.Source: §C.4 · p. 16, Figure 3 · p. 17, Figure 4 · p. 17
C.5 Parameter Efficiency: Category-wise Borda per Billion Parameters
- The proposed 8B has the highest Borda-per-parameter across Accuracy, Plausibility, and Relevance, though this efficiency measure does not equal absolute quality.The ratio omits runtime constants and training cost, so it should be considered alongside absolute scores and deployment budgets.Source: §C.5 · p. 17
C.6 Qualitative Category-wise Evaluations
- Subdomain results show relative strengths in Budgeting, Employment, and Planning, with wider relevance gaps in Auto, Housing, and Credit and accuracy gaps in Housing, Insurance, Taxes, and Credit.On plausibility, the 8B clusters near leaders across most subdomains, with stronger showings in Debt and Planning and lower margins in Taxes and Retirement.Source: §C.6.1 · p. 18, §C.6.2 · p. 18, §C.6.3 · p. 18
C.7 Overall Summary, Limitations, and Next Steps
The 8B model trails larger baselines on absolute scores but leads on Borda points per parameter, with its largest gaps in regulation- and table-heavy areas. The section recommends targeted grounding and evaluation improvements while describing case-study trade-offs between scale, detail, and resource use.
C.7 Overall Summary, Limitations, and Next Steps
- The 8B model is just behind the leading cluster on Relevance and Plausibility, mid-pack on Accuracy, and highest in Borda-per-parameter across all three metrics.Everyday tasks are steadier, while gaps widen in regulation- and table-heavy areas.Source: §C.7 · p. 19
- The next steps prioritize compact, versioned rule tables, lightweight calculators and unit tests, contrastive edge-case supervision, diversified judge checks, and multi-turn evaluation.The suggested grounding targets regulation-intensive domains such as taxes, insurance, and credit.Source: §C.7 · p. 19
- The proposed 8B model tops the cohort in mean raw Borda points per billion parameters, while larger models lead overall on absolute category accuracy.The accuracy gaps are smaller in everyday planning and larger in Housing, Insurance, and Taxes.Source: Figure 8 · p. 19, Figure 10 · p. 21
D Training Details
- Training used 15.6K samples and 2.6K validation samples for four epochs, taking three hours on one A100 GPU.The optimal validation loss occurred at step 101.Source: D Training Details · p. 19
E Sample Model Responses
- The appendix presents three comparative case studies using real-world queries, abridged model responses, and judge rankings based on accuracy, relevance, and plausibility.Each case includes query context and a rationale for the final ranking.Source: E Sample Model Responses · pp. 19, 23
Query (Abridged)
- The sample queries cover allocating idle cash, deciding whether to phase a Roth conversion, and choosing where to keep $25,000 that may be needed soon.Their supplied contexts include business reinvestment, conversion taxation and bracket timing, and liquidity-versus-yield considerations.Source: Query (Abridged) · pp. 23–24, Context Titles (Summary) · p. 23
Analysis
- Baseline-L ranks above Ours–8B, which ranks above Baseline-S in all three cases; the 8B responses are practical and user-aligned but less detailed in the examples.The Roth-conversion example identifies a material Form 8606 filing error, while the savings example notes that Ours–8B gives less comparison of alternatives.Source: Analysis · pp. 23–24
Conclusion
- The case studies show Baseline-L’s advantage in detailed multi-step reasoning and factual integrity, while the 8B model offers a balance of quality and resource economy at one quarter of its size.The passage identifies factual integrity as an area where the 8B model faltered in Case C2.Source: Conclusion · p. 24