Source-linked AI summary
DelistBench: Evaluating Search-Enabled LLMs for Auditable Corporate-Event Database Completion
Xuan Yao, Li Shuping, Dai Yang, Zhou Yi, Ke-Wei Huang
TL;DR
Financial institutions need independent ways to detect incomplete or misclassified corporate-event records in vendor databases. This paper introduces Search-to-Record and DELISTBENCH, then evaluates closed-book and web-enabled systems for cutoff-aware delisting reconstruction. Web access improves timing and status accuracy, while economy systems approach the strongest system at much lower API cost and risk-based triage supports targeted review.
Problem
Financial institutions need independent public-source audits to detect missing, stale, and misclassified corporate-event records in vendor databases.
Method
The paper defines Search-to-Record, builds the 1,200-record DELISTBENCH delisting benchmark, and compares five models in paired closed-book and web-enabled conditions.
Results
Web access sharply improves event timing and enables economy systems to approach stronger-system accuracy at low API cost.
Takeaways & Limitations
Deployment should calibrate triage to local prevalence and market mix, preserve positive-event recall, and route positive or ambiguous cases to targeted review.
Takeaways & Limitations
The balanced benchmark does not reflect natural event prevalence, risk estimates are not reweighted to institutional distributions, and historical reconstruction uses the current web rather than a sealed historical index.
Abstract
from arXiv · showhide
Financial institutions need an independent way to detect missing, stale, and misclassified corporate-event records in vendor databases. We introduce Search-to-Record, a database-assurance task in which search-enabled large language models reconstruct institution-defined event records from public sources for a known security universe and historical cutoff, and DelistBench, a 1,200-record benchmark for security-level delisting announcements. We evaluate five models in paired closed-book and web-enabled conditions. Web access raises announcement-date accuracy within seven days by 34.0 to 48.0 percentage points and event-status accuracy by approximately 2.8 to 21.7 points; the best system achieves 81.5% overall joint accuracy within seven days. Economy web systems achieve 75.9-78.3% overall joint accuracy within seven days at 4.5-6.6% of the API cost of the most expensive web system. Risk-based triage identifies low-error subsets, although the highest-coverage operating point still sends 27.3% of the balanced test set to review. The evaluation identifies web retrieval as the main source of timing gains and shows that low-cost systems can approach the best system's accuracy. Together, Search-to-Record, DelistBench, and the evaluation provide concrete deployment guidance: calibrate triage to local event prevalence and market mix, preserve positive-event recall, and route positive and ambiguous cases to targeted review.
1 Introduction
Financial institutions need independent public-source audits to identify incomplete, inaccurate, or untimely corporate-event records across a known security universe. The paper frames this as Search-to-Record and evaluates it with a delisting benchmark across closed-book and web-enabled systems.
- Corporate-event databases can be incomplete, inaccurate, untimely, or inconsistent with an institution’s definitions, especially across small markets, local-language disclosures, cross-listed securities, and differing exchange procedures.
- Public-source auditing requires identifying the correct listing, formal event stage, announcement date, reason, and evidence trail while avoiding cutoff violations and unsupported citations.
- Search-to-Record reconstructs institution-defined event records independently of vendor feeds for a known target universe and cutoff.
- DELISTBENCH contains 1,200 records with balanced positive, later-event, and still-listed strata, public evidence, and issuer-family-disjoint splits.
- The evaluation compares five model families in paired closed-book and web-enabled conditions across accuracy, evidence, cost, latency, and risk-based acceptance.
2 Related Work
Prior financial event-extraction work generally assumes supplied documents, while search-grounded language-model research studies browser use or changing facts. Search-to-Record instead fixes the target identity, event definition, and historical cutoff while retrieving from the open web.
- Financial event-extraction resources typically extract events, triggers, arguments, or reasons from supplied documents.
- Its distinctive output is a database record that includes retrieval and listing resolution rather than only an event mention.
- Search-grounded language-model research established browser-assisted answering, interleaved tool use, and retrieval for changing facts.
- Financial RAG and deep-research benchmarks generally use bounded document collections or study corporate analysis and forecasting.
- Search-to-Record uses the open web while fixing the target identity, event definition, and historical cutoff.
3 Methodology
The methodology defines a cutoff-aware, security-level delisting reconstruction task and evaluates it on a balanced benchmark with hierarchical correctness, evidence, cost, latency, and review measures. It also controls model comparisons and calibrates risk-based routing for acceptance or review.
- 3.1 Search-to-Record Task: The task input identifies the issuer, security, venue, country, optional stable identifier, and historical cutoff.
- 3.1 Search-to-Record Task: The output records cutoff status, formal announcement date, normalized reason, claim-level evidence, and uncertainty; date and reason are null unless status is yes.
- 3.1 Search-to-Record Task: Public reconstruction supports independent checks for missing, stale, misdated, misreasoned, or wrongly assigned vendor events, as well as confirmed no-event cases.
- 3.1 Search-to-Record Task: A qualifying delisting requires a formal public decision or action for the specified security and venue, with the canonical date set to the first qualifying disclosure rather than an effective-removal date.
- 3.2 DelistBench: DELISTBENCH contains 1,200 evenly divided records: positive completions, future-announcement negatives, and active still-listed controls.
- 3.3 Evaluation Protocol and Experimental Setup: Joint correctness requires all-record decision accuracy and, for positives, a date within tolerance plus the correct broad reason; evidence, cost, and latency are separate.
- 3.3 Evaluation Protocol and Experimental Setup: All ten systems use closed and web conditions with fixed within-model settings, while cross-model comparisons represent complete systems rather than equal-compute architectures.
- 3.3 Evaluation Protocol and Experimental Setup: Risk routing fits logistic regression on a 300-record calibration split and selects the largest threshold whose accepted-set error is at most 5%, then applies it unchanged to test data.
4 Results
Web access improves event-status and record-completion performance, but positive-event timing, reasons, listing scope, and cutoff discipline remain major error sources. Risk-based routing finds low-error accepted subsets, while evidence quality and review requirements constrain full automation.
- Record Reconstruction Performance: Every web-enabled system has higher event-status accuracy and macro-F1 than its closed-book counterpart, with Gemini 3.6 Flash web leading the reported status metrics.Web access reduces C-FPR for every model, although some category-specific changes are less uniform.
- Record Reconstruction Performance: 34.0–48.0 percentage points: web access increases A-date-±7 accuracy, while A-joint-±7 completion rises 24.8–35.3 percentage points.Gemini 3.6 Flash web still exactly dates only 46.0% of A records and jointly recovers status, date, and broad reason for 45.2%.
- Record Reconstruction Performance: Gemini 3.6 Flash achieves the highest A-joint-±7 completion while costing less than half as much per 1,000 records as GPT-5.6 Luna.DeepSeek V4 Flash web is 3.0 pp below GPT-5.6 Luna in A-joint-±7 completion at 6.6% of the cost, while Gemini 3.5 Flash-Lite is 3.8 pp below at 4.5%.
- In-Depth Analysis: 60.4% of failures are date-only or date-plus-reason errors, while only 14 outputs across the ten system runs are analysis-invalid.The main bottleneck is identifying the formal event stage, binding it to the target listing, and preserving the cutoff rather than producing valid JSON.
- In-Depth Analysis: 52.8% of evidence packages for yes predictions passed the protocol, compared with 88.3% for no predictions.The asymmetric protocol requires supporting evidence for affirmative predictions but does not fail negative predictions solely for lacking submitted evidence.
- Risk-Based Acceptance and Review Routing: Gemini 3.6 Flash accepts 72.7% of test records with 3.7% error, while DeepSeek V4 Pro accepts 65.7% with 2.5% error.GPT-5.6 Luna reaches 6.8% test error despite the 5% calibration target, so calibration is not a test-set guarantee.
5 Deployment and Discussion
Deployment is framed as risk-adaptive verification rather than unattended record generation: calibrate review to local prevalence and market mix while protecting positive-event recall. Web systems improve status performance, but date and reason errors require targeted escalation and analyst review.
- Deployment Calibration: 27.3% minimum review rate on the balanced test set is not a production estimate.Lower positive-event prevalence may reduce overall review load, but does not eliminate high Category A error rates.
- Deployment Calibration: A production policy should constrain both review load and positive-event recall using the same unified risk score and threshold.The recall constraint prevents low review rates from being achieved by overlooking positive events.
- Operational Implications: Risk-adaptive routing clears low-risk records, escalates medium-risk cases for additional verification, and reserves unresolved high-risk cases for human review.Escalation options include official-source retrieval, timeline checks, an independent verifier, or a stronger model.
- Operational Implications: 97.3% decision accuracy but 46.0% exact date accuracy on Category A makes the system closer to candidate detection than unattended complete-row generation.Separating status risk from conditional date/reason risk is proposed as a deployment design.
- Operational Implications: Web-enabled systems improve status macro-F1 and Category A recall in every displayed country, while web joint-±7 completion ranges from 27.3% to 55.8%.Country results are descriptive heterogeneity diagnostics, not causal country effects or stable rankings.
- Operational Implications: The intended use is an independent quality-assurance layer that flags missing, stale, or definition-misaligned records and concentrates analyst attention.The relevant economic comparison is total cost per corrected or newly recovered record, including human review and the incumbent vendor baseline.
6 Limitations and Ethical Considerations
The evaluation has important scope and reproducibility boundaries: balanced benchmark results are not institution-level prevalence estimates, and several labels and subgroup interpretations are constrained. Evidence auditing is only partially independently validated, while production automation requires provenance and human safeguards.
- Scope and Data: The balanced benchmark does not reflect natural event prevalence, and reported risk estimates are not reweighted to an institutional distribution.The benchmark therefore does not directly estimate production review rates.
- Scope and Data: Category C means no qualifying action was identified through 6 August 2026, not that none existed.Gold dates are the first qualifying disclosures identified during construction, not necessarily the earliest-ever disclosures.
- Scope and Data: Country denotes issuer geography rather than listing market, and the absence of Vietnam-listed records limits conclusions about the motivating low-resource-market setting.The country field is distinct from exchange country.
- Evidence and Reproducibility: Two nonoverlapping 100-package human-validation samples achieved 78.0% exact agreement with Codex, while all 750 packages were not independently double-coded.Reproducibility requires archiving prompts, access dates, configurations, and raw outputs because web pages, indexes, prices, and model aliases may change.
- Ethical Considerations: Production use should preserve provenance, prevent silent overwrites, monitor subgroup recall and calibration drift, and retain human review for positive, conflicting, or high-impact records.Automation may propagate plausible errors into downstream research, surveillance, or client systems.
7 Conclusion
Search-to-Record reframes search-enabled LLM evaluation as auditing corporate-event records over a known security universe. DelistBench shows strong timing gains from web access and low-cost systems approaching stronger-system accuracy, while deployment still requires better positive completion, market-aware calibration, risk-triggered escalation, and a budgeted human-review tail.
- Conclusion: Web access sharply improves event timing, while economy systems approach stronger-system accuracy at low API cost.Remaining errors concentrate in positive-record dates, reasons, listing scope, and cutoff discipline.
- Conclusion: Balanced-benchmark coverage is not a production review estimate, and its empirical target is not a guarantee.The proposed path combines stronger positive completion, market-aware calibration, risk-triggered machine escalation, and a small human-review tail.