Source-linked AI summary
DealsFlow: An Evidence-Gated Agentic Harness for Venture Capital Deal Triage
Ming Ze Tang, David Cutting, Boyd Carson, Vasiliki Carson
TL;DR
Unchecked LLM scoring of startup decks risks score inflation, run-to-run variation, and rationales that are hard to audit. DealsFlow separates LLM ratings from code-based score constraints and human approval; on mature investments its scores correlate with outcomes, but hindsight may inflate the result.
Problem
Top-of-funnel startup triage is repetitive and time-consuming, while unchecked LLM scores can mirror deck framing, vary across runs, and provide hard-to-audit rationales.
Source: §1 · p. 1Method
DealsFlow is a human-in-the-loop harness in which the LLM reads, cites evidence, and rates categories while code combines and constrains scores and humans retain approval.
Source: §1 · p. 1, §8 · p. 10Results
On 142 mature investments, the full pipeline’s score correlated with total-value multiples (ρ = 0.45) and identified write-offs better than chance (AUC0 = 0.71).
Source: §7 · p. 9Takeaways & Limitations
In 20 inbound applications, analyst time fell by 58% even after verification and correction, while 84% of cited claims were fully supported.
Source: §8 · p. 11Retrieval was not date-restricted and the model’s knowledge cutoff postdates most outcomes, so hindsight cannot be excluded as an explanation for part or all of the correlation.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Large language models (LLMs) can read startup pitch decks, but letting them emit investment scores directly invites score inflation, run-to-run variance and rationales that are hard to audit. We describe DealsFlow, a human-in-the-loop agentic harness deployed at a UK early-stage fund. The LLM reads the deck, cites evidence and rates each category; code combines the ratings, caps them and only ever lowers them. Three mechanisms implement this: a stage-aware corroboration ceiling (78, 70 or 60 out of 100) on factual categories without external evidence; a sceptic call that may lower but never raise a score; and a pre-fetch pattern that retrieves and indexes all evidence before a single generation pass. We evaluate on $N=243$ SEIS/EIS investments with recorded outcomes. Because recent investments have unresolved outcomes, we use the 142 that are at least three years old. There, the full pipeline's score has Spearman $\rho=0.45$ (95% CI 0.32 to 0.57) with the total-value multiple and AUC 0.71 for identifying write-offs, against $\rho=0.38$ and AUC 0.68 for an unconstrained LLM. The gain is small and comes mostly from the sceptic call; we did not test whether its lower-only rule matters. Investments that later exceeded $2\times$ had 64% of factual categories corroborated, against 38% for write-offs (omnibus Kruskal-Wallis test over five outcome groups, $p=0.031$), and our retrieval was not date-restricted, so these figures may overstate screening skill. In a study of 20 inbound applications, analyst time per application fell from 44.7 to 18.9 minutes including verification and correction, and 84% of 240 cited claims were fully supported by their sources, but 70% of reports needed at least one correction.
1 Introduction
DealsFlow separates LLM reading and rating from code-based score combination and constraints, while preserving explicit human sign-off. The paper introduces evidence-gated scoring and evaluates its outcome associations while accounting for limitations such as hindsight in retrieved evidence.
1 Introduction
- Top-of-funnel triage is repetitive and time-consuming, and the founder’s optimistic framing can have the most influence during this stage.The described work includes reading decks and checking company registers, founders, competitors, and markets.Source: §1 · p. 1
- Unchecked LLM scores are difficult to defend because models can mirror a deck’s framing, vary across runs, and give rationales that may not reflect what drove their outputs.The paper therefore asks whether harness-generated scores carry information about real outcomes after accounting for evaluation pitfalls.Source: §1 · p. 1
- The design keeps the LLM reading and rating while code combines, caps, and only lowers scores; investment approvals require explicit human sign-off.The paper’s mechanisms include a stage-aware corroboration ceiling, a lower-only sceptic pass, and pre-fetching evidence before generation for audit.Source: §1 · pp. 1–2
- Because retrieval was not restricted to the decision date, later success-related web evidence may inflate corroboration and risk scoring outcomes rather than applications.The paper states that it measures this risk rather than assuming it away.Source: §1 · pp. 1–2
- The evaluation uses a leakage audit, corroboration-only baseline, maturity restriction, guardrail ablations, and analyst assessments across 243 holdings, with 142 mature holdings as the primary set.The analysis also reports rank correlation and AUC with bootstrap intervals, alongside analyst time and factual-support quality.Source: §1 · p. 2
- On 142 mature investments, the full pipeline’s score was rank-correlated with total-value multiples, with a modest but measurable margin over an unconstrained LLM.The authors do not claim this establishes better screening than analysts or that the improvement survives without hindsight in retrieved evidence.Source: §1 · p. 2
2 Related Work
Prior work covers data-driven startup prediction, pitch-deck analysis, LLM failure modes, and evaluation leakage, but often lacks realised fund-level outcomes or staged human supervision. DealsFlow’s pre-fetch pattern trades run-time tool flexibility for templated, logged evidence retrieval.
Machine learning for VC screening
- Public startup-screening datasets rarely contain realised fund-level returns or write-offs, while screening is usually framed as single-pass classification rather than staged human-supervised work.The paper’s one-fund dataset addresses the first limitation but does not remove selection concerns.Source: §2 · p. 2
LLM failure modes relevant to scoring
- Sycophancy, judging biases, investment-analysis preferences, nondeterminism, and potentially unfaithful explanations motivate keeping the model out of score arithmetic and storing evidence.The paper connects this design choice with arguments for inherently interpretable models in high-stakes settings [27].Source: §2 · p. 2
Look-ahead and survivorship bias in financial LLM evaluation
Tool use and retrieval
Regulatory context
- The paper describes logging and approval controls as support for firms evidencing their own systems and accountability, not as making them compliant.The passage says compliance is a property of firm processes, not software architecture.Source: §2 · p. 2
3 System Design
DealsFlow runs in production as a staged, human-approved triage harness; its design combines evidence gathering, capped scoring, a lower-only sceptic pass, and audit-oriented controls. The section also describes additional deployed components that were not evaluated.
3.1 Extraction and corroboration
- Evidence is gathered from a company-register snapshot and templated web searches, then category-level support is labelled external, registry, deck_only, or none.Only external and registry labels count as corroboration; the label is a model judgment rather than a code-level claim-to-source match.Source: §3.1 · p. 3
3.2 Deterministic composite and the corroboration ceiling
- The code computes a weighted composite from six LLM-scored categories, with founders weighted highest and the weights fixed before portfolio evaluation.The rubric weights were agreed with the investment committee and were not tuned on the evaluation portfolio.Source: §3.2 · p. 3
- Uncorroborated factual categories are capped at 78 for early-stage, 70 for unknown-stage, or 60 for growth-stage decks; founders and vision are exempt.The stage-scaled ceiling is intended to avoid making the composite a proxy for public-data availability; scores already below the ceiling are unchanged.Source: §3.2 · pp. 3–4
3.3 Asymmetric sceptic pass
- A second, sceptic call can lower category scores only, with each reduction limited to 20 points and requiring a specific contradiction or implausibility.Missing corroboration alone is not a reason for a second deduction because the ceiling already addresses it.Source: §3.3 · p. 4
3.4 Declared-dependency pre-fetch
- Declared dependencies are retrieved and indexed before a single generation call, but pre-fetch does not make evidence identical across screens or make the LLM deterministic.Search backends rotate, pages change, and results are cached for 24 hours; run-to-run variance was not measured.Source: §3.4 · p. 4
3.5 Sequential gating
- Funds configure sequential stages so inexpensive checks run first and the costly parallel evaluation runs only for applications that pass triage.Intake fields use semantic roles so analytics can work across funds with different form designs.Source: §3.5 · p. 4
3.6 Components not evaluated in this paper
- The unevaluated stage-3 component runs one evidence-prefetched agent per criterion in parallel and synthesizes p10/p50/p90 survival estimates.The paper explicitly makes no effectiveness claims about components outside the experiments.Source: §3.6 · p. 4
- The unevaluated portfolio-memory branch compares applications with past deals and can apply a bounded historical-outcome adjustment or raise a divergence flag.The shift is capped at ±0.35, and the outcome index is not a probability; quarterly updates can also generate lessons that remain inactive until supported by three distinct outcomes.Source: §3.6 · p. 4
4 Governance Design
Governance design combines retained screening records, mandatory human approval for investment progression, and safeguards for untrusted inputs. The described controls support auditability but do not themselves establish regulatory compliance.
4 Governance Design
- Screening records retain the structured profile, scores before and after guardrails, source list, and model identifier, but omit full page content and the triage prompt.Records are append-only by application design and isolated per tenant; the paper says they lack cryptographic tamper-evidence.Source: §4 · p. 5
- The system may reject below a configured threshold but cannot approve an application; investment progression requires explicit human action.Overrides require a written justification stored with user identity and timestamp.Source: §4 · p. 5
- The controls are intended to help evidence existing FCA framework obligations, but the authors do not claim they make a firm compliant.Source: §4 · p. 5
- Decks and web pages are treated as untrusted inputs, with delimiter wrapping, instruction-pattern stripping, and model-reported injection flags; these defenses were not evaluated.Similarity search, lesson retrieval, and base rates are confined to one tenant’s data.Source: §4 · p. 5
5 Evaluation Design
The evaluation uses mature, previously funded SEIS/EIS investments to examine outcome ranking, guardrail comparisons, evidence leakage, and analyst workflow measures. It defines the dataset’s limits and specifies the baselines, metrics, and review protocols.
5.1 Portfolio dataset
- 243 SEIS/EIS investments form the portfolio dataset; the primary evaluation uses 142 investments at least three years old, all of which entered at seed stage.The mature set includes 43 write-offs and 22 investments above 2×.Source: §5.1 · p. 5
- Because every portfolio deal was approved, the evaluation measures ranking among funded deals and cannot measure false negatives among rejected applications.Rejected applications have no observed outcomes in this dataset.Source: §5.1 · p. 5
5.2 Research questions
- Five questions cover outcome-linked corroboration, score discrimination, guardrail contributions, analyst time, and factual support and omissions in generated reports.The research questions also ask how corroboration alone compares with the composite score.Source: §5.2 · p. 5
5.3 Leakage channels
- Retrieval is not restricted to information available by the investment-committee date, so the study does not claim point-in-time isolation.Current Companies House data, live web search, and model knowledge may carry post-decision information; the submitted deck is archived from before the decision date.Source: §5.3 · pp. 5–6
5.4 Configurations and baselines
- Comparisons include unconstrained, ceiling-only, full, and sceptic-only LLM configurations, plus corroboration-only, vintage, structured-field, and initial-cheque baselines.The structured-field logistic regression uses sector, stage, and initial cheque size; the corroboration-only baseline uses the share of factual categories labelled external or registry.Source: §5.4 · p. 6
5.5 Metrics and statistics
- Spearman’s ρ and two AUC metrics use bootstrap intervals; a Kruskal–Wallis test reports H(4) = 10.65, p = 0.031 for per-deal corroboration fractions across outcome groups.AUC0 measures write-off discrimination and AUC2 discrimination of deals above 2×; intervals use 10,000 bootstrap resamples, and configuration differences use paired bootstraps.Source: §5.5 · pp. 6–7, Table 4 · p. 7
5.6 Analyst time and factual-support protocol
- A counterbalanced study assigns 20 separate inbound applications to manual and DealsFlow triage, timing analyst work through completed assessments.Assisted time includes reading, source verification, corrections, and completion; paired timing intervals use application-level bootstrap resampling, with no significance claims.Source: §5.6 · p. 7
- Two reviewers independently assess 240 source-cited claims across 20 reports, reconcile disagreements, and record source support, missed corroboration, omissions, and corrections.The four ceiling-eligible categories are also checked for source-confirmed labels and potentially missed corroboration.Source: §5.6 · p. 7
6 Results
On 142 mature deals, the full pipeline modestly outperformed the unconstrained LLM, while evidence availability remained an unresolved source of hindsight risk. In a 20-application study, analyst time was shorter, but reports still often needed correction.
6.1 RQ1: Is the signal evidence availability?
- Deals later exceeding 2× had 63.6% of factual categories corroborated, versus 37.8% for write-offs (Kruskal–Wallis p=0.031); undated retrieval cannot distinguish hindsight from genuine early traction.The corroboration-only baseline reached ρ=0.22 [0.05, 0.37] and AUC0=0.61 [0.50, 0.70], below the full pipeline. The stage-aware ceiling changed composites for 48.6% of deals by 1.23 points on average, yet accounted for little overall discrimination.Source: §6.1 · p. 7
6.2 RQ2–RQ3: Discrimination and ablation (142 mature deals)
- Sector, stage and initial-cheque records ranked mature outcomes at ρ=0.27; initial cheque alone showed no clear write-off separation (AUC0=0.51).Table 5 reports results on the 142 mature deals and includes portfolio-record baselines.Source: §6.2 · pp. 7–8, Table 5 · p. 8
- The full pipeline improved on the unconstrained LLM across all three metrics, with paired intervals excluding zero, but gains were small and mostly present in the sceptic-only configuration.The ablation does not show that the sceptic’s lower-only rule is responsible for the gain.Source: §6.2 · p. 8
- Mean composite scores fell from 77.2 for investments above 2× to 62.7 for write-offs, but heavy overlap means they should not be read as a fine-grained ranking.The medians of the impaired/loss and written-off groups were nearly equal, at 65.0 and 65.2.Source: §6.2 · p. 8, Table 7 · p. 9
- Neither guardrail raised scores; the ceiling changed fewer than half of deals by about one point on average, while the sceptic changed almost all by about three points.The study did not examine the ceiling’s effect on rankings within particular groups.Source: §6.2 · p. 8, Table 8 · p. 9
6.3 RQ4: Analyst time
- DealsFlow reduced mean analyst time from 44.69 to 18.85 minutes per application, a paired saving of 25.84 minutes (95% CI 22.99–28.86), including verification and correction.The comparison involved 20 applications and was not a controlled trial.Source: §6.3 · p. 8
6.4 RQ5: Factual-support quality
- Reviewers found 84.2% of cited claims fully supported and 93.3% at least partly supported, but 14 of 20 reports needed correction.They also found nine material issues omitted from initial reports; severity was not rated.Source: §6.4 · p. 8
- Corroboration labels were imperfect in both directions, and the study set no advance quality threshold or comparison of final assessment quality between workflows.Six of 48 external or registry labels lacked confirmation from a cited source; seven of 32 deck_only or none labels could have been corroborated.Source: §6.4 · p. 9
7 Discussion and Limitations
On 142 mature investments, the full pipeline’s scores were associated with outcomes and modestly improved on an unconstrained LLM, mostly through the sceptic pass. The authors qualify these findings with leakage, selection, unresolved outcomes, and limits of the analyst study and evaluation scope.
Results
- On 142 mature investments, the full pipeline’s score had ρ=0.45 with total-value multiples and AUC0=0.71 for write-offs, modestly improving over an unconstrained LLM.The improvement came mostly from the sceptic pass; the corroboration ceiling added little by itself.Source: §7 · p. 9
Leakage
- Because retrieval was not date-restricted and the model’s knowledge cutoff postdated most outcomes, hindsight may explain part or all of the correlation.Web search lacked a date filter, while current Companies House status and officers could reveal later dissolution; prospective evaluation is the cleanest remedy.Source: §7 · p. 9
Analyst study
- For 20 inbound applications, analyst time fell from 44.7 to 18.9 minutes per application, 84.2% of cited claims were fully supported, and 70% of reports needed correction.Two reviewers were not independent of the developers or blind to system labels, and workflow was confounded with analyst; these measures address usability and accuracy, not investment decisions.Source: §7 · pp. 9–10
Selection
- All 243 deals were funded, so the evaluation cannot observe rejected applications’ outcomes or assess false negatives.The results measure ranking within an already-screened population.Source: §7 · p. 9
Censored outcomes
- Even among the 142 mature holdings, 26 remained held at cost, and active-holding marks were manager valuations rather than realised returns.Results could shift as the portfolio matures.Source: §7 · p. 9
Corroboration ceiling
- The ceiling caps deck-only factual categories at 78 for recognised early-stage companies or 70 otherwise, so scores partly encode public-evidence availability.In deployment, such companies are routed to an analyst.Source: §7 · p. 9
Sceptic mechanism
- The sceptic’s gain could reflect its lower-only rule, the extra review, or combining two noisy calls; these mechanisms were not separated.The comparison lacked an equally resourced reviewer permitted to revise scores in either direction and was not matched on call budget.Source: §7 · p. 10
Scope and reproducibility
- Scores were evaluated in a single run, results come from one fund and model, several components were unevaluated, and proprietary data and code prevent independent reruns.The rubric may not transfer to other funds, and the stability of an individual deal’s score is unknown.Source: §7 · p. 10
8 Conclusion
DealsFlow separates LLM rating from code-controlled combination and score reductions, with sources and guardrail changes recorded for audit. On mature portfolio holdings it showed modest outcome correlations, while leakage may inflate screening skill and analyst reports still required correction.
Conclusion
- The LLM reads, cites evidence, and rates categories; code combines and caps ratings and only lowers them, recording sources and scores before and after each guardrail.The system does not make investment decisions; approvals require explicit human sign-off.Source: §8 · pp. 10–11
- On 142 mature investments, the full pipeline’s score correlated with total-value multiples at ρ=0.45 (95% CI [0.32, 0.57]) and improved modestly over an unconstrained LLM, mostly through the sceptic pass.Write-offs were corroborated less often than later successes, and retrieval was not date-restricted, so results may overstate screening skill.Source: §8 · p. 11
- In 20 inbound applications, analyst time fell by 58% after verification and correction, 84% of cited claims were fully supported, and 70% of reports still needed correction.The authors highlight leakage diagnostics as potentially more useful than the system itself for evaluating historical financial-data screening.Source: §8 · p. 11
Ethics and Privacy Statement
The study used aggregated portfolio and application data and an EU-region Azure API deployment. The authors also identify public-evidence disadvantage and hosted-model confidentiality as risks, while disclosing fund-related interests and generative-AI copyediting.
Data and privacy
- Portfolio records and application files came from Sapphire Capital Partners; reported figures are aggregated, no company, founder, or investor is identifiable, and investor-level records were not used.Model inference ran through the Microsoft Azure API in an EU region; prompts and outputs may be retained for up to 30 days for abuse monitoring under default terms.Source: Ethics and Privacy Statement · p. 11
Analyst participation
- The two participating analysts worked with the fund manager and timed workflows and rated reports as part of their normal work.No personal data about the analysts is reported.Source: Ethics and Privacy Statement · p. 11
Societal risks
- Rewarding public evidence may disadvantage founders with small public footprints, and disparate impact was not tested.The stage-aware ceiling softens but does not remove this risk; automated rejections could exclude founders without human deck review, so sampling and review are recommended.Source: Ethics and Privacy Statement · p. 11
Disclosures
- The codebase is jointly owned by Queen’s University Belfast and Sapphire Capital Partners, and some authors work for Sapphire, which uses DealsFlow and supplied the portfolio data.Generative AI tools were used for copyediting and formatting; the authors state that system design, experiments, analysis, and conclusions are their own.Source: Ethics and Privacy Statement · p. 11
A Models and Configuration
The appendix identifies a table of experimental settings.
A Models and Configuration
- Table 12 lists the settings used in the experiments.The supplied passage gives the table’s scope but not its settings.Source: Table 12 · p. 12
B Score Bands and Category Definitions
The scoring prompt applies shared score bands across categories, while Table 14 maps the evaluation to Kong et al.’s [18] structural validity checklist.
B Score Bands and Category Definitions
- The scoring prompt uses one set of bands across all categories: 75 and above is exceptional or clearly strong; 55–74 is credible with open questions; 35–54 is weak; below 35 is poor or incoherent.There are no per-category anchors; Table 13 gives each category’s definition and what counts as corroboration.Source: B Score Bands and Category Definitions · p. 12
C Structural Validity Checklist
- Table 14 maps the evaluation onto Kong et al.’s [18] structural validity checklist.Source: C Structural Validity Checklist · p. 12
D Reviewer Agreement on Claim Support
Two reviewers agreed on 223 of 240 initial claim-support ratings, with unweighted Cohen’s κ of 0.751.
D Reviewer Agreement on Claim Support
- 92.9% agreement and unweighted Cohen’s κ=0.751 across 240 initial claim ratings; Table 15 lays out Reviewer A’s rows against Reviewer B’s columns.Source: D Reviewer Agreement on Claim Support · p. 12, Table 15 · p. 13