PaperlayerView exact revision on arXiv

Source-linked AI summary

FinDialogLens: Event Extraction over Multi-Party Dialogue for Missed-Trade Identification in Financial Chatrooms

Chin-Lun Fu, Hong Ni, Behrouz Madahian

arXiv:2610.02455v1cs.CLcs.AI

TL;DR

Financial chatrooms contain concurrent RFQs whose final prices and trade outcomes appear in downstream messages, making manual recovery of missed-trade records infeasible at production scale. FinDialogLens uses compact in-domain classifiers for message metadata, segments chat into per-event RFQ windows, and applies an LLM-powered Trade Engine to fill event roles. FinDialogLens achieves 94.3% accuracy on final price and trade outcome, outperforming full-chatroom CoT prompting; fine-tuned open-source LLMs as small as 3B parameters achieve comparable performance.

  • Problem

    Financial chatrooms contain concurrent RFQs whose final prices and trade outcomes appear in downstream messages, making manual recovery of missed-trade records infeasible at production scale.

    Source: §1 · p. 1, §3.1 · p. 2

  • Method

    FinDialogLens uses compact in-domain classifiers for message metadata, segments chat into per-event RFQ windows, and applies an LLM-powered Trade Engine to fill event roles.

    Source: §1 · p. 1, §3.2 · p. 3, §5 · p. 7

  • Results

    FinDialogLens achieves 94.3% accuracy on final price and trade outcome, outperforming full-chatroom CoT prompting; fine-tuned open-source LLMs as small as 3B parameters achieve comparable performance.

    Source: §5 · p. 7

  • Takeaways & Limitations

    Routing 85% fewer final-price cases to the LLM recovers half the accuracy gap to FinDialogLens (GPT-4o) and saves over $300 per day at 70,000 RFQs per day.

    Source: §1 · p. 2, §5 · p. 7

    Reliable message-level metadata models require labeled in-domain data, and their utility may decrease when annotations are scarce or communication styles and product conventions shift.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Multi-party financial chatrooms are vital for sales-and-trading professionals, but their complexity makes manual recovery of missed trades infeasible: each Request for Quote (RFQ) is an event whose final price and trade outcome appear many messages after the RFQ-trigger message (the inquiry message), interleaved with concurrent RFQs from other participants. We cast this as event extraction (EE) over multi-party dialogue and present FinDialogLens, a hybrid LLM pipeline in which compact fine-tuned classifiers act as inference-time scaffolds: they detect RFQ-triggers and price/trade outcome metadata, an RFQ-Level Module segments per-event RFQ windows, and a Trade Engine fills argument roles. With GPT-4o, FinDialogLens reaches 92.1% and 94.3% accuracy on final price and trade outcome, respectively, outperforming full-chatroom CoT prompting methods; fine-tuned open-source LLMs with as few as 3B parameters achieve comparable performance with modest in-domain data. To make the LLM-based solution practical at scale, a difficulty-aware router balances cost and accuracy by allocating RFQs between a low-cost rule-based engine and the higher-performing LLM-powered Trade Engine, cutting LLM calls by 85% on final price while recovering half of the accuracy gap to FinDialogLens (GPT-4o), saving over $300/day at our 70,000-RFQ/day scale.

1 Introduction

Financial chatrooms interleave concurrent RFQ negotiations, making missed-trade records difficult to recover manually at scale. FinDialogLens frames recovery as event extraction and combines per-message metadata, event-specific windows, and LLM role filling; a difficulty-aware router reduces LLM use while recovering accuracy.

1 Introduction

  • RFQ negotiations are asynchronous and noisy: quotes and outcomes can be separated from the inquiry by minutes and mixed with parallel negotiations and irrelevant messages.At production volumes of tens of thousands of RFQs per day, manual recovery of missed-trade records is infeasible.Source: §1 · p. 1
  • The paper casts each RFQ as an event triggered by an inquiry, with final price and trade outcome drawn from downstream messages interleaved with other RFQs.The authors contrast this setting with existing event-extraction recipes targeting sentence-level text or coherent single-author documents.Source: §1 · p. 1
  • FinDialogLens uses compact fine-tuned classifiers to produce RFQ-trigger, price, and trade-outcome metadata, then segments chat into per-event windows for LLM role filling.Its difficulty-aware router assigns easier windows to a rule-based engine and harder ones to the LLM-powered Trade Engine.Source: §1 · pp. 1–2
  • 92.1% accuracy for final price and 94.3% accuracy for trade outcome; FinDialogLens also substantially improves GPT-4o over full-chatroom CoT prompting.The pipeline combines RFQ windowing with compact specialized classifiers as inference-time scaffolds.Source: §1 · p. 2
  • At 70,000 RFQs per day, routing about 15% of RFQs to the LLM cuts final-price LLM calls by 85%, recovers half the accuracy gap to FinDialogLens (GPT-4o), and saves over $300 per day.The router is supervised by per-event correctness of a rule-based weak engine.Source: §1 · p. 2

2 Related Work

Prior event extraction ranges from sentence-level methods to document-level argument linking and generative approaches, while financial NLP has focused mainly on news and filings. The paper distinguishes its application by the structural demands of conversational financial data.

2 Related Work

  • Event extraction has progressed from sentence-level ACE formulations to document-level argument linking and generative or instruction-tuned methods.The cited examples include Doddington et al. (2004), Ebner et al. (2020), and Lu et al. (2021).Source: §2 · p. 2
  • Table 1 compares existing event-extraction method families with this application along structural axes.Source: Table 1 · p. 2
  • Financial NLP has focused primarily on news and filings, while conversational financial data has received little attention.Dialogue disentanglement separates multi-party threads but targets thread separation rather than event extraction.Source: §2 · p. 2

3 Methodology

FinDialogLens frames RFQs as trigger-anchored events with arguments scattered across chat, then uses message-level cues, per-event windows, and an LLM-based Trade Engine to recover them.

3.1 Problem Setting

  • Each RFQ is a trigger-anchored event whose final price and trade outcome may appear later amid concurrent RFQs.Source: §3.1 · pp. 2–3

3.2 Message-Level Module

  • Three fine-tuned RoBERTa-base models detect RFQ triggers, classify trade-outcome cues, and extract price entities as message-level metadata.Source: §3.2 · p. 3

3.3 RFQ-Level Module

  • The RFQ-Level Module bounds each detected trigger’s context at a 20-minute gap or 25-message cap before passing its per-event window onward.Source: §3.3 · p. 3

3.4 Trade Engine

  • The LLM-powered Trade Engine predicts final price and trade outcome from each metadata-augmented RFQ window.The chosen window contains the gold final price and trade outcome for over 96% of RFQ events.Source: §3.4 · pp. 3–4

3.5 Difficulty-Aware Router

  • A router estimates weak-engine reliability from labeled RFQ windows, sending lower-ambiguity cases to rules and reserving the LLM for harder cases.A threshold controls the cost–accuracy trade-off.Source: §3.5 · p. 4

4 Experiments

The experiments evaluate FinDialogLens on time-held-out financial conversations and test metadata scaffolding, per-event windowing, open-source models, and difficulty-aware routing.

4.1.1 Data Overview

  • The test set contains 8,970 messages and 966 RFQs from a contiguous June–July 2024 period held out from training and evaluation.The annotated data covers 47,362 messages and 8,343 RFQs across two years of financial conversations.Source: §4.1.1 · p. 4

4.1.2 Baselines

  • Baselines include a Rule-based Engine using message metadata and full-chatroom GPT-4o CoT prompting with zero-shot or few-shot examples, with or without metadata.FinDialogLens instead segments chatrooms into RFQ-specific windows before role filling.Source: §4.1.2 · pp. 4–5, Table 2 · p. 5, §4.2.1 · p. 6

4.1.3 Evaluation Protocol and Metrics

  • Evaluation aligns predicted and reference events by trigger message, scores final price by exact match, and treats trade outcome as three-way Hit, Miss, or Unknown classification.The pipeline runs end-to-end without gold triggers; router efficiency is measured with Performance Gap Recovery and Call-Performance Thresholds.Source: §4.1.3 · p. 5

4.2.1 Effect of Message-Level Metadata and Per-Event Windowing

  • Adding metadata raises few-shot CoT final price accuracy from 64.4% to 79.2% and trade outcome accuracy from 70.5% to 87.0%; zero-shot gains are 13.0 and 20.4 points.With metadata, the Rule-Based Engine reaches 77.1% final price and 90.3% trade outcome accuracy; the three message-level classifiers achieve F1 scores of 0.97, 0.95, and 0.93, respectively.Source: §4.2 · p. 5, §4.2.1 · p. 6
  • Final price rises from 79.2% to 92.1% and trade outcome from 87.0% to 94.3% with FinDialogLens (GPT-4o) versus the strongest metadata-augmented CoT baseline.These are gains of 12.9 and 7.3 points, respectively.Source: §4.2.1 · p. 6

4.2.2 Open-source vs. Proprietary LLMs

  • With the full training set, all four open-source LLMs outperform GPT-4o CoT baselines; Flan-T5-XL reaches 91.7% final price and 92.8% trade outcome accuracy.Accuracy improves rapidly as training data grows and plateaus after about 1,000 examples; models as small as 3B parameters approach proprietary-model performance with modest in-domain supervision.Source: §4.2.2 · pp. 6–7, Figure 3 · p. 6

4.2.3 Difficulty-Aware Router

  • Routing 14.9% of RFQ windows to the LLM for final price extraction recovers 50% of the accuracy gap and saves over $300/day at more than 70,000 RFQs/day.Fine-tuned routers approach full-LLM performance at a 40% call fraction; for trade outcome, Flan-T5-XL reaches 50% and 80% gap recovery at call fractions of 25.1% and 32.8%.Source: §4.2.3 · p. 7, Table 4 · p. 7, Figure 4 · p. 7

5 Conclusion

FinDialogLens applies a hybrid LLM pipeline to missed-trade identification in multi-party financial dialogue. Its reported results include strong GPT-4o accuracy, comparable performance from small fine-tuned open-source models, and lower-cost routing.

5 Conclusion

  • FinDialogLens combines compact message-level classifiers, per-event RFQ windows, and an LLM-powered Trade Engine to extract final price and trade outcome for each RFQ.With GPT-4o, it achieves 92.1% final price accuracy and 94.3% trade outcome accuracy, outperforming full-chatroom CoT prompting.Source: §5 · p. 7
  • Fine-tuned open-source LLMs with as few as 3B parameters achieve performance comparable to proprietary models with modest in-domain data.A difficulty-aware router cuts final-price LLM calls by 85%, recovers half the accuracy gap, and saves over $300/day at a 70,000-RFQ/day scale.Source: §5 · p. 7

Limitations

FinDialogLens has portability and reproducibility constraints: some components need adaptation across settings, and its source conversations cannot be publicly released.

Limitations

  • Compact metadata models require labeled in-domain data, and their utility may decrease when annotations are scarce or communication styles and product conventions shift.The pipeline depends on these models for RFQ triggers, price mentions, and trade outcome cues.Source: Limitations · p. 8
  • Fixed temporal and message-count thresholds for RFQ windows, and a binary rule-based-versus-LLM router, are tailored to the reported setting.Other deployments may need adaptive boundaries or routing across multiple model tiers, latency constraints, or risk policies.Source: Limitations · p. 8
  • Client and regulatory confidentiality prevents public release of the conversations, limiting direct external replication on the original corpus.The authors provide guidelines, statistics, model and training details, and evaluation protocols to support reproduction on comparable proprietary datasets.Source: Limitations · p. 8

Ethics Statement

The study frames FinDialogLens as a tool to support financial professionals while complementing human judgment, and reports privacy safeguards for its internally governed data.

Ethics Statement

  • FinDialogLens is designed to help financial professionals identify missed trading opportunities and improve decision-making while complementing, not replacing, human expertise and judgment.Source: Ethics Statement · p. 8
  • Experiments used anonymized, internally governed datasets; user identities were replaced with generic identifiers, and no personal or client-identifying information was collected, stored, or used.Source: Ethics Statement · p. 8
  • Access to data and model outputs is restricted to authorized personnel, and data handling follows relevant regulatory and organizational standards.Source: Ethics Statement · p. 8

Disclaimer

The paper is informational, disclaims warranties and liability, and is not investment research, advice, or a solicitation.

Disclaimer

  • The document disclaims representations and warranties about the information’s completeness, accuracy, or reliability, and disclaims liability.Source: Disclaimer · p. 8
  • It is not intended as investment research or investment advice, nor as a recommendation, offer, or solicitation concerning a financial instrument or service.Source: Disclaimer · p. 8
  • The document states that it should not be used to evaluate a transaction and does not constitute a solicitation where such solicitation would be unlawful.Source: Disclaimer · p. 8

A Data Privacy, Annotation Guidelines, and Statistics

This appendix describes the data, annotation and evaluation setup, alongside implementation details for the message-level models, Trade Engine, router, and GPT-4o evaluations.

A Data Privacy, Annotation Guidelines, and Statistics

  • 38.1% of test RFQs (368 of 966) occur in windows with two or more gold RFQ triggers, the interleaved cases categorized as hard.Two professional annotators independently labeled message-level triggers, outcomes, and prices, and RFQ-level final prices and outcomes; disagreements and cases unresolved from chat alone were excluded.Source: A Data Privacy, Annotation Guidelines,… · pp. 10–11, Table 5 · p. 11

B Training Details of FinDialogLens

B.1 Message-Level Module

  • Three fine-tuned RoBERTa-base models generate message-level metadata: RFQ triggers, trade-outcome cues, and extracted price values.The resulting metadata is stored per message for use by later modules.Source: §B.1 · p. 11

B.2 Trade Engine

  • The Trade Engine uses GPT-4o or one of four fine-tuned open-source LLMs, with a unified prompt template.The GPT-4o API version is gpt-4o-2024-05-13.Source: §B.2 · p. 11

B.3 Difficulty-Aware Router

  • The difficulty-aware router has encoder-only, decoder-only, and encoder-decoder classifier implementations; the decoder-based variants predict whether the rule engine is correct.The encoder-only classifier uses a classification layer on the <s> token embedding and cross-entropy loss.Source: §B.3 · p. 12

5. Optimizer: AdamW

  • The supplied optimizer subsection names AdamW for Mistral-7B-Instruct and Flan-T5-XL and points to the prompt template in Figure 10.Source: §5 · p. 12

C Multi-Run Evaluation for GPT-4o-Based Methods

  • GPT-4o-based settings use temperature-0 decoding and are run five times; Table 8 reports mean and standard deviation because API nondeterminism can cause small run-to-run variation.Source: C Multi-Run Evaluation for… · p. 12

D Error Analysis

The error analysis finds that closely timed RFQs are a common source of edge cases. In one example, the system linked a price from a second RFQ to the target RFQ.

D Error Analysis

  • Most test-set edge cases occurred when multiple RFQs were requested within a short time, when the LLM could mismatch responses and RFQs.The example involves two clients requesting different products shortly apart and receiving prompt replies.Source: D Error Analysis · pp. 12–13
  • The LLM correctly identifies the target RFQ as a missed trade based on message 16, but incorrectly links the second RFQ’s “75 76” price to it; the correct final price is NONE.Source: D Error Analysis · p. 13
  • The system correctly identified the target RFQ as a missed trade from message 16 but incorrectly linked the second RFQ’s price to it.The two outputs therefore differed: the trade outcome was correct, but the final price was not.Source: D Error Analysis · p. 13

E Ablation Study on Metadata Error Propagation

The ablation suggests metadata errors propagate only to a limited extent, while routing results show that fine-tuned routers can direct RFQs by rule-engine suitability and reduce processing costs and latency.

E Ablation Study on Metadata Error Propagation

  • Final price accuracy changes from 92.1% to 91.9% with gold metadata, while trade outcome accuracy changes from 94.3% to 95.9% and trade outcome F1 from 94.5% to 95.7%.The comparison is between predicted message-level metadata and an oracle setting with gold labels.Source: E Ablation Study on Metadata Error… · p. 13, Table 9 · p. 14

F.1 Routing Quality

  • Rule-engine final price accuracy ranges from 95% to 100% in the top three router-logit deciles, versus 14.6% in the lowest decile for the RoBERTa-base router.Figure 6 groups RFQ windows into deciles by router-logit ranking.Source: §F.1 · p. 14

F.2 Cost and Latency

  • In the routing experiments, processing over 70,000 RFQs per day with the RoBERTa-base router yielded daily savings exceeding $300 while maintaining approximately 85% accuracy for final price extraction.Source: §F.2 · p. 14
  • Under CPT(50%), RoBERTa-base reduces total LLM API latency for final-price extraction to 217.4 seconds, compared with 717.8 seconds for the Random router.Source: §F.2 · p. 14
Loading 2610.02455v1…