Source-linked AI summary
FinLifeBench: Exhaustive Life-Event History and Financial-State Reconstruction from Longitudinal Banking Dialogue
Hangyeul Lee, Juyoung Oh, Jaeyong Ko, Sunmin Kim, Jaeik Park, Hyunkyu Kim, Jungmin Son, Pilsung Kang
TL;DR
Repeated banking dialogue can reveal life changes that require assistants to reconstruct complete, traceable histories and current financial states, a requirement largely absent from existing benchmarks. FinLifeBench evaluates both reconstructions over shared cumulative dialogue and finds complementary failures: event histories become incomplete with depth, while financial records retain outdated values and the two tasks are only weakly associated.
Problem
Existing benchmarks largely test question answering, retrieval, request understanding, bounded tracking, or targeted memory rather than exhaustive reconstruction of grounded life-event histories and complete financial states.
Method
FinLifeBench uses deterministic persona-conditioned trajectories and cumulative banking dialogue to evaluate first-establishing anchors for every life event and complete 34-path financial states at consecutive checkpoints.
Results
Across eleven LLMs, event histories became incomplete with longer dialogue while financial reconstruction retained superseded or potentially outdated values as current, and task performance was only weakly associated.
Takeaways & Limitations
Longitudinal assistants need separate completeness, evidence-localization, and temporal-validity checks before reconstructed outputs can serve as customer records.
Takeaways & Limitations
The synthetic benchmark targets coverage rather than real-world distributions, ambiguity, or stakes, and uses static prefixes rather than evaluating memory-management policy.
Abstract
from arXiv · showhide
Repeated banking interactions require assistants to maintain complete, current, and traceable customer records as life changes emerge incidentally in routine requests. Existing benchmarks emphasize question answering, bounded episodes, or targeted recall rather than exhaustive longitudinal reconstruction. We introduce FinLifeBench, which evaluates two tasks over the same cumulative dialogue: reconstructing every life-event instance with its first-establishing session and reconstructing a complete 34-path financial state at consecutive checkpoints. The benchmark contains 6,000 eight-turn Korean banking sessions from 20 independent synthetic trajectories, with deterministic, exhaustive gold for 24 event types and 34 state paths and consensus quality assurance. Across eleven LLMs under a full-context condition, event-anchor recall falls from 0.591 at 15 sessions to 0.445 at 300. Errors are driven primarily by omitted events rather than poor anchor localization, while financial-state reconstruction frequently treats superseded or potentially outdated information as current; the best GCA@15 reaches 0.470. Performance on the two reconstruction tasks is only weakly associated. These results show that models can localize evidence for recovered events while still failing to maintain complete and temporally valid longitudinal records.
1 Introduction
FinLifeBench addresses the untested need to reconstruct complete, evidence-grounded life-event histories and temporally valid financial states from longitudinal banking dialogue. Across eleven LLMs, event histories became incomplete with longer dialogue while financial reconstruction retained outdated information, and the two tasks were only weakly associated.
- Existing benchmarks largely test question answering, retrieval, request understanding, bounded tracking, or targeted memory rather than schema-complete reconstruction of life-event histories and financial states.
- FinLifeBench evaluates every life-event instance with its first-establishing session and all 34 financial-state paths at consecutive checkpoints.
- Across eleven LLMs, life-event histories increasingly omitted established events as dialogue grew, despite usually identifying the correct anchor for recovered events.
- Financial-state reconstruction frequently treated superseded or potentially outdated information as current, while performance on the two tasks was only weakly associated.
- The benchmark measures completeness, provenance, and temporal validity rather than reducing longitudinal reconstruction to a single accuracy notion.
2 Related Work
FinLifeBench extends long-horizon memory and state-tracking benchmarks by requiring exhaustive reconstruction directly from static dialogue prefixes. It jointly evaluates grounded event histories and complete checkpoint-wise financial states, exposing coverage and synchronization requirements that prior systems do not directly test.
- Prior long-horizon benchmarks cover recall, temporal reasoning, incremental updating, evolving profiles, interference, and forgetting, but differ in which state information they inspect.
- HorizonBench records triggering events for preference changes, whereas DynamicMem keeps state-evolution traces implicit and scores profile completion at checkpoints.
- MEMPROBE recovers a hidden 31-dimensional user state from an agent’s memory artifact, while FinLifeBench reconstructs from dialogue itself at 20 consecutive checkpoints.
- FinLifeBench gives models identical static dialogue prefixes and evaluates exhaustive reconstruction, requiring explicit coverage and state-synchronization mechanisms from adapted memory systems.
3 FinLifeBench Dataset
FinLifeBench is a deterministic synthetic benchmark built from persona-conditioned trajectories, Korean banking dialogues, and checkpoint financial histories. Its controlled construction supplies exhaustive event and state annotations while testing discrimination among anchors, routine activity, follow-ups, stale values, and cancelled plans.
- 3 FinLifeBench Dataset: FinLifeBench contains 20 persona-conditioned trajectories with 300 sessions, 20 life-event instances, and a common 34-path financial-state schema, totaling 6,000 sessions.
- 3 FinLifeBench Dataset: Personas are normalized into typed demographic, household, employment, housing, financial, and dialogue-style fields that instantiate the initial financial state.
- 3 FinLifeBench Dataset: The benchmark separately inventories life events that may alter customer needs and state paths whose values or validity may change, with a many-to-many relationship between them.
- 3 FinLifeBench Dataset: A finite-state transition graph enforces event precedence, minimum intervals, recurrence limits, and cooldown periods for internally consistent trajectories.
- 3 FinLifeBench Dataset: Each 15-session window contains exactly one first-establishing anchor session for one event instance, with remaining sessions drawn from controlled non-anchor categories.
- 3 FinLifeBench Dataset: Eight-turn Korean banking conversations place event evidence incidentally within banking tasks and undergo deterministic schema, grounding, safety, semantic, and contract validation.
4 Benchmark Tasks
FinLifeBench evaluates exhaustive reconstruction of grounded life-event histories and complete financial states at consecutive checkpoints. The tasks require models to preserve event provenance and represent the temporal validity of all 34 financial-state paths.
- Task 1: Grounded Life-Event History Reconstruction: Task 1 reconstructs the cumulative life-event history as event–anchor pairs at each 15-session checkpoint.The model receives all sessions through the checkpoint and the life-event ontology, without being told the gold history size or one-event-per-window construction rule.
- Task 2: Complete Financial-State Reconstruction: Task 2 reconstructs the complete financial state across all 34 paths at the same checkpoints.Each path includes a normalized value, validity status, and supporting model-visible session identifiers.
- Task 2: Complete Financial-State Reconstruction: Checkpoint state records require evidence identifiers for paths differing from S000, while unchanged paths require empty evidence lists.Unknown and not_applicable paths must still be included, and every checkpoint is evaluated through a fresh request.
- Task 2: Complete Financial-State Reconstruction: Gold financial states distinguish current, historical, stale, unknown, and not_applicable validity statuses.Current is the latest supported value; historical is superseded, stale is plausibly invalidated without replacement, unknown lacks establishing evidence, and not_applicable is excluded by the current configuration.
5 Experiments
The experiments evaluate eleven LLMs on standardized full-context requests using metrics that separately assess reconstruction completeness, provenance, temporal validity, schema compliance, and cross-task relationships. The analysis emphasizes whether models retain complete histories and apply state changes correctly as dialogue accumulates.
- Experimental Setup: Eleven LLMs each produced 400 outputs per task, yielding 8,800 predictions under the full-context condition.Every model received the complete dialogue history available at each checkpoint.
- Experimental Setup: All models received identical sessions, taxonomies or schemas, and output contracts through fresh requests without future sessions or earlier predictions.Fallbacks and output repair were disabled, while tokenizer-dependent context lengths ranged from 64k to 125k tokens.
- Task 2 Metrics: ESM equals 1 only when all 34 predicted value–status pairs match gold at a checkpoint.CSA instead measures cell-level snapshot correctness.
- Evaluation Analysis: The evaluation reports task-specific schema validity, history completeness and anchor localization, evidence and temporal-validity diagnostics, and cross-task analysis.Cross-task attribution compares exact event–anchor recovery with correct values and statuses on currently attributed state paths; uncertainty uses shared trajectory-cluster bootstrap resampling.
- Aggregation and Uncertainty: Confidence intervals use 10,000 shared trajectory-cluster percentile-bootstrap resamples with checkpoints nested within trajectories.Rank intervals recompute model-level EA-F1, GCA@15, and correlations for the fixed eleven-model set.
6 Results
As dialogue depth increases, models recover a shrinking share of life-event history while usually anchoring recovered events correctly. Financial-state reconstruction also struggles with changes and temporal validity, often missing updates or corrupting stable state.
- Life-event history: 0.591 to 0.445: model-macro event–anchor recall falls between checkpoints 15 and 300, while precision rises from 0.573 to 0.762.Mean per-output EA-F1 also falls from 0.579 to 0.532, and underprediction grows from 28.2% to 98.2%.
- Life-event history: 0.462 pooled pair recall contrasts with 0.866 conditional anchor accuracy, showing that omitted events dominate wrong-anchor assignments.Among 46,200 gold instances, 21,574 are missing outright versus 3,306 assigned the wrong anchor.
- Financial-state reconstruction: 0.470: Claude Opus 4.8 achieves the highest GCA@15, while models correctly reconstruct 59.8% of changed transitions.Among unchanged transitions, 74.0% are preserved, but 11.6% receive spurious updates and 13.9% remain previously wrong.
- Financial-state reconstruction: 18.9% of changed transitions remain unchanged, whereas 20.5% are updated incorrectly, indicating both missed required changes and incorrect updates.Because unchanged cases outnumber changed cases by roughly 12:1, pooled errors are dominated by spurious updates despite misses being conditionally more frequent.
- Financial-state reconstruction: 79.7% value accuracy and 85.4% status accuracy coexist with only 6.2% historical recall and 10.4% stale recall.Gold historical and stale states are frequently predicted as current, exposing temporal-validity failures masked by pooled accuracy.
7 Analysis
The two tasks fail in different ways: event reconstruction is primarily limited by coverage, whereas financial-state reconstruction has stronger value recovery but weak lifecycle tracking. Their outcomes are only weakly associated, even when outputs are schema-valid.
- 7.1 Omission and Status Collapse Dominate Mis-grounding: 0.462 pair recall versus 0.866 conditional anchor accuracy: Task 1 is bottlenecked by event coverage rather than localization of recovered evidence.Task 1 misses exceed unmatched predictions across models, with missed:spurious ratios from 1.6:1 to 28.0:1.
- 7.1 Omission and Status Collapse Dominate Mis-grounding: 0.797 value recovery contrasts with 0.062 historical recall and 0.104 stale recall in Task 2.Pooled state-transition errors are dominated by spurious updates rather than missed changes.
- 7.2 The Two Reconstruction Objectives Are Only Weakly Associated: 23.6% of matched observations are correct on both tasks, while 33.2% are wrong on both, 24.1% are correct only on Task 1, and 19.1% only on Task 2.These outcomes show that exact event grounding neither guarantees correct attributed state paths nor is necessary for recovery from later evidence.
- 7.2 The Two Reconstruction Objectives Are Only Weakly Associated: Spearman ρ=0.291 and Kendall τ_b=0.164 indicate weak association between EA-F1 and GCA@15 across the eleven-model set.Mean absolute rank displacement between tasks is 3.1 of 11 positions.
- 7.1 Omission and Status Collapse Dominate Mis-grounding: 0.470: GCA@15 never exceeds this value despite schema validity ranging from 0.963 to 1.000 on Task 2 and 0.978 to 1.000 on Task 1.Perfectly schema-valid GPT-OSS 120B reaches only 0.249 GCA@15 and 0.124 EA-F1.
8 Limitations
The benchmark’s synthetic, static-prefix, Korean setting constrains what its results measure and how broadly they should be interpreted.
- Synthetic personas and event rates target coverage rather than representative real-world distributions, ambiguity, or stakes.The benchmark may therefore not reproduce real customer conditions.
- Static prefixes evaluate long-context reconstruction, not systems’ memory-management policies.This excludes the effects of systems selecting, consolidating, or overwriting evidence themselves.
- The Korean-only dialogue means cross-model differences partly reflect Korean language proficiency.
- One event per 15-session window may reveal gold event cardinality to benchmark-aware systems.
- Evidence is scored separately from value and status, comparisons are observational, and no downstream decision is evaluated.
9 Conclusion
FinLifeBench addresses an untested need for exhaustive, checkpoint-wise reconstruction of grounded life-event histories and complete financial states. Its results show complementary failures across the two tasks, so reconstructed records require completeness and validity checks before use.
- FinLifeBench separates completeness, provenance, and temporal validity for exhaustive longitudinal reconstruction.
- Life-event histories become incomplete as dialogue deepens, although recovered events are usually anchored to the correct session.
- Financial-state reconstruction misses required updates, overwrites stable cells, and reports superseded or plausibly invalidated values as current.
- The two tasks show no robust rank association across models.
- Accurately grounded outputs still require completeness and validity checks before serving as customer records.
Ethics and Privacy Statement
The benchmark uses fictional personas and synthetic conversations, while the measured capability is privacy-sensitive and requires safeguards for profile updates and consequential actions.
- No real customer records are involved because the benchmark uses fictional personas and synthetic conversations.
- Systems inferring life changes from incidental cues should obtain consent before profile updates.
- Consequential actions should require confirmation, and event rates must not be used for profiling, eligibility, or pricing.