PaperlayerView exact revision on arXiv

Source-linked AI summary

Better Deck or Different Judge? Evaluating Agentic Harness Gains in Corporate and Investment Banking

Ludovic Gibert, Matis Despujols, Andre-Louis Rochet

arXiv:2609.39958v1cs.AI

TL;DR

Because judges guide harness changes and assess their results, it is unclear whether higher scores reflect better decks or changes in grading. The retrospective study evaluates 17 development deliverables with eight judges from six model families, comparing harness versions, direct-generation baselines, and repeated assessments. On development cases, the complete harness scores consistently higher than direct generation by its 27B model, with cleaned text-only gains of 20.4 to 33.6 points against the short prompt.

  • Problem

    Because judges guide harness changes and assess their results, it is unclear whether higher scores reflect better decks or changes in grading.

    Source: §1 · p. 1

  • Method

    The retrospective study evaluates 17 development deliverables with eight judges from six model families, comparing harness versions, direct-generation baselines, and repeated assessments.

    Source: §1 · p. 1, §4 · p. 2

  • Results

    On development cases, the complete harness scores consistently higher than direct generation by its 27B model, with cleaned text-only gains of 20.4 to 33.6 points against the short prompt.

    Source: §8 · p. 8

  • Takeaways & Limitations

    Judges agree on broad development progress but less on final-deck rankings, and score shifts on unchanged decks make small improvements difficult to distinguish from judge variability.

    Source: §8 · p. 8

    Direct-generation baselines are single samples without comparable adaptation budgets or reserved test tasks, and text-only comparisons leave length differences and possible within-session contrast effects.

    Source: §6 · p. 8

Abstract

from arXiv · show

Corporate and investment banking teams use presentations to support credit decisions and advise clients on financing and transactions. Producing these decks requires reconciling financial data, tracing sources and turning analysis into a recommendation. We retrospectively study the development of an agentic harness combining a 27B language model, financial calculations, narrative templates and validation checks. LLM judges guide engineering changes and assess the resulting decks, raising the question of whether higher scores reflect better documents or changes in grading. In shared-session text-only grading with template markers removed, five judges score the complete system 20.4 to 33.6 points out of 95 above the same model generating directly from a short prompt. Every judge scores the system higher on all seventeen development deliverables. Margins against direct Opus generation from a short prompt range from -4.7 to +0.8 points. Judges agree on broad progress across development rounds but agree less on final-deck rankings than on pooled scores. Repeated grading also shifts scores on unchanged decks, making small improvements difficult to distinguish from judge variability.

1 Introduction

CIB presentations turn financial analysis into traceable recommendations for credit and transaction decisions, so automating them requires more than drafting prose. The study examines whether judge-guided harness development improves decks and how reliably judges measure that improvement.

1 Introduction

  • CIB presentations must reconcile figures across slides, trace sources, and support recommendations for credit, financing, and transactions.These requirements make document production more than prose drafting.Source: §1 · p. 1
  • Judges can provide rubric-based scores, verdicts, and defect lists, but the same judge may score an unchanged deck differently across passes or configurations.Prior work reports position, verbosity, and self-preference biases, alongside repeated-rating inconsistency.Source: §1 · p. 1
  • The retrospective study tracks 17 deliverables over eight rounds and uses eight judges from six model families to assess repeatability, progress agreement, rankings, and harness gains.The panel compares development versions, direct-generation baselines, and two text-only passes, with template markers removed in the second.Source: §1 · p. 1

2 Related work

Prior work studies LLM-judge accuracy, bias, and repeatability, alongside financial and slide-generation benchmarks. This study focuses on repeated and cross-judge assessments of complete CIB decks during harness development.

2 Related work

  • LLM judges have been studied for agreement with human ratings, positional and length biases, self-preference, and task-dependent agreement.Examples include Zheng et al. (2023), Liu et al. (2023), Wang et al. (2024), and Bavaresco et al. (2025).Source: §2 · p. 1
  • Repeated-rating studies examine reliability and score variation, while Li (2026) distinguishes system drift from judge drift using human anchors and sequential inference.This retrospective study measures repeated scores without human anchors or a drift-detection guarantee.Source: §2 · pp. 1–2
  • The analysis uses established agreement statistics, detectable-change estimates, kappa, paired differences, and rank tests.The cited methods include Shrout and Fleiss (1979), Weir (2005), Cohen (1960, 1968), and Wilcoxon (1945).Source: §2 · p. 2
  • Complete CIB decks are compared across judges and repeated assessments, extending financial and slide-generation work that evaluates other tasks or outputs.Prior examples include financial reasoning and spreadsheet benchmarks, document summarization, agentic slide generation, and broader PowerPoint evaluation.Source: §2 · p. 2

3 The generation harness

The harness combines a language model with financial calculations, templates, playbooks, and validation to produce CIB presentations. Its evaluation starts from collected inputs and does not include collection services or the banker-supervisor review loop.

3 The generation harness

  • The system combines an on-premises Qwen3.8-27B model with financial calculations, playbooks, and output checks in a staged pipeline for debt-capital-markets and coverage pitches.The evaluated contrast separates model roles from compiled prose and begins with collected inputs and a mandate.Source: §3 · p. 2
  • A deterministic engine builds a typed fact registry containing values, units, periods, and provenance classes, records sources and formulas where applicable, and represents unavailable bank-held information as named gaps.Collection services are part of the wider application but are outside the evaluated contrast.Source: §3 · p. 2
  • Playbooks set page plans, while narrative production combines compiled registry clauses, prose templates, and conditional model selection or writing; template pages can bypass the writer.After generation, validation rejects unsupported numeric literals; it is not constrained decoding.Source: §3 · p. 2
  • Historical templates contain long literal passages in every evaluated final deck, but incomplete code, input, and generation-log snapshots prevent quantifying the model-written share of the prose.Later logs with fewer writer calls cannot establish that share retrospectively.Source: §3 · p. 2
  • Post-render checks cover model configuration, registry consistency, narrative contradictions, and layout defects; the application’s supervisor and banker review loop is outside the evaluation.Source: §3 · p. 2

4 Setting and protocol

The protocol evaluates 17 CIB deliverables across development rounds, judge configurations, and direct-generation baselines. It uses repeated assessments and text-only comparisons to examine judge variability and harness scores.

4 Setting and protocol

  • The 17 deliverables span coverage documents, credit reviews, financing pitches, and M&A advisory, using public information about listed or rated issuers.Between rounds, coding agents changed templates, the engine, and prompts, mostly to address defects in prior verdicts and partly based on practitioner feedback.Source: §4 · p. 2
  • The rubric scores eight criteria out of 100, then records quotations, a verdict, and five worst defects; judges see rendered slides and extracted text without authorship or round information.Criteria include reader orientation, synthesis, recommendation, financial depth, consistency, auditability, storyline, bank rationale, and visual quality.Source: §4 · p. 3
  • Judges included Opus, Sonnet, Haiku, DeepSeek, GLM, GPT-6 Luna Pro, Gemini, and Qwen, with controlled and unreconstructable configurations across campaigns.Cross-family judges assessed rounds 1 and 7, often twice in original and mixed sessions; configuration checks showed score changes on unchanged decks.Source: §4 · p. 3
  • Direct-generation baselines used collected inputs and a short prompt or a stronger rubric-and-conventions prompt; the latter 27B decks also differed in serving and reasoning effort.The harness contrast omits its fact engine, playbooks, compiled narration, and gates; the stronger-prompt decks were mixed in sessions with harness decks of the same deliverables.Source: §4 · pp. 3–4
  • The second text-only pass removes template markers and visual scoring, but wording, structure, and length still differ; judges compare three versions per deliverable in randomized order.The first pass retained identifying template material and is summarized separately.Source: §4 · p. 4
  • Repeated passes on identical files support score-repeatability analysis, while detectable-change and deck-level tests rely on independence assumptions treated as exploratory.The mean detectable-change estimate is 1.96 sd(d)/√n under independent errors; session-level sensitivities have limited resolution with four sessions. Table 1 lists the deliverables and notes that each judge session grades three to five decks.Source: Table 1 · p. 3, §4 · p. 4

5 Results

The harness receives substantially higher text-only scores than direct generation from the same 27B model, while repeated grading and judge differences make some score changes difficult to interpret. Judges agree on progress across rounds but less on final-deck rankings.

5.2 Shifts cluster by judge session

  • For unchanged round-7 decks, campaign shifts imply an exploratory 3.2-point noise scale versus 1.9 under independent deck errors; the 3.2-point estimate is not a calibrated threshold.The deck-level sign-flip test gave p = 0.063, but its session-level counterpart gave p = 0.375.Source: §5.2 · p. 4

5.3 An offset that did not replicate

  • Sonnet’s round-5 development pass scored 11.8 points above Opus, but its panel rerun averaged 73.8, 13.0 below the earlier pass; the model name alone did not reproduce judge level.The cause was unidentified; settings, tool access, and served model version may have differed.Source: §5.3 · pp. 4–5

5.1 Two passes of the same judge

  • Two Opus passes on identical decks agreed at ICC 0.77 in round 7 and 0.78 in round 8; the per-deck noise scale was 7.9 points (6.4 to 10.4).Including the systematic shift raises the estimate to 8.4 points; estimates depend on the stated independence and distributional assumptions.Source: §5.1 · p. 4, Table 2 · p. 5

5.5 Observed system comparisons

  • In mixed sessions, harness decks led the stronger-prompt 27B baseline by 10.6 to 20.4 points; against direct Opus, total differences ranged from −2.1 to +4.3 with short prompts and −3.4 to +1.6 with strong prompts.Against direct Opus with the short prompt, six content gaps were close to zero; length-related bias remains possible, and the second text-only pass cannot isolate template removal.Source: Table 3 · p. 5, §5.5 · pp. 6–7

5.4 Eight judges from six model families

  • Every judge saw round-1-to-7 gains of 5.6 points (Haiku) to 13.9 (DeepSeek), while median rank correlation fell from 0.63 in round 1 to 0.30 in round 7.Round-7 scores averaged 75.5 with Sonnet and 93.4 with Qwen; the convenience-selected panel’s judge and deck variance shares do not generalize to other judges. Per-deck detectable change was 13.5 to 16.1 points for the four judges measured that way.Source: §5.4 · pp. 5–6, Figure 5 · p. 7

5.5 Observed system comparisons

  • 20.4 to 33.6 points out of 95: every judge scored harness text above direct 27B text on all 17 deliverables; against direct Opus text, margins ranged from −4.7 to +0.8.These shared-session text-only gaps may include contrast effects, and a non-significant difference does not establish equivalence.Source: §5.5 · p. 7

5.6 Progress across development rounds

  • The Opus mean rose from 63.3 in round 1 to 76.8 in round 4, but single-pass trajectories cannot establish whether later reconciliation work improved professional quality.Rounds 2 to 4 raised almost every deck; round 8 also added a controls workbook to the judge’s input.Source: §5.6 · p. 7

5.7 Verdicts and arithmetic

  • Opus never returned “send as is” in 153 development records, its verdict tracked the score (ROC area 0.89), and it made no addition errors in 357 complete records.The other reported addition-error counts varied across judges and configurations.Source: §5.7 · p. 7

6 Threats to validity

The evaluation compares judges without a banker-rated reference set, and the harness comparison bundles multiple components. Small samples, session effects, judge adaptation, and incomplete controls constrain what the observed scores establish.

6 Threats to validity

  • Without banker-rated reference decks, judge agreement cannot establish accuracy or professional acceptability.The convenience panel includes related model families, and the residual combines judge-deck interaction with measurement error.Source: §6 · p. 7
  • Seventeen deliverables across four sessions and two replication pairs limit inference; tests are exploratory and unadjusted for multiple comparisons.Deck-level intervals ignore session dependence, while session-level sensitivities have little resolution; more randomized passes and sessions are needed for stable variance estimates.Source: §6 · p. 8
  • The harness comparison combines calculations, playbooks, compiled prose, model roles, validation, and presentation, with no matched ablation isolating their contributions.Historical runs show operational differences between writing modes but cannot attribute final quality gains; exact historical input equality and a complete provenance chain are unverified.Source: §6 · p. 8
  • Final harness decks received iterative, partly judge-driven engineering, while direct-generation baselines were single samples without comparable adaptation budgets or reserved test tasks.Serving and reasoning effort also confound the stronger 27B prompt comparison; separate-session comparisons mix system differences with session offsets.Source: §6 · p. 8
  • Text-only passes only partially control presentation and template markers, retain length differences, and may introduce within-session contrast effects.Development can adapt to its judge; gains across other model families suggest they are not specific to Opus, but shared preferences remain possible, and the passes are not identical-condition replications.Source: §6 · p. 8

7 Reproducibility

The authors make selected research materials available on request, while keeping the decks and several underlying artifacts private.

7 Reproducibility

  • The development and panel tables contain 187 and 1,479 records, respectively; anonymised scores, scripts, and an exclusion register are available from the authors on request.The decks, verdict texts, generator inputs, and parser remain private because they contain bank and issuer names.Source: §7 · p. 8
  • Archived API expenditure was 8.2 US dollars for judging and strong-prompt 27B generation, excluding local compute, agent sessions, and development labour.Source: §7 · p. 8
  • The decks, verdict texts, generator inputs, and parser are not public because they contain bank and issuer names.Source: §7 · p. 8

8 Conclusion

The conclusion reports consistently higher development-case scores for the complete harness than for direct generation by its 27B model, while cautioning that judge and session changes can move scores. The original pipeline separates data collection, calculation, drafting, verification, and review into staged steps.

8 Conclusion

  • On the development cases, the complete harness receives consistently higher scores than direct generation by its 27B model.Judges agree on broad progress but less on final-deck rankings, and judge or session changes can shift scores on unchanged decks.Source: §8 · p. 8

A The original pipeline

  • The original pipeline targets debt-capital-markets and coverage pitches, with each stage using a separate model session and versioned inputs and outputs.Source: A The original pipeline · p. 10
  • A first collection fills and checks a manifest; a second targets gaps and conflicts, producing a self-contained data block and a data contract that limits later guessing.A calculation engine derives arithmetic from that block and records each number with a key and audit log.Source: A The original pipeline · p. 10
  • Three audience-specific documents are drafted from one reconciliation file, then edited without recomputing; a script blocks failures in cited-number checks.A fresh-context adversarial review judges scope, storyline, and stress cases; failures are corrected at their root step before documents are regenerated and reviewed again.Source: A The original pipeline · p. 10
  • A defect register carries findings across runs, while periodic reviews turn recurring error classes into rules, preferably implemented in code.The design also calls for mandatory calculations, blocking verification, structured output, and seeded defects to test checks before production.Source: A The original pipeline · p. 10

B Harness gap by criterion and on text alone

The appendix reports an earlier text-only comparison and lays out protocols for grading and reading judge scores. It also specifies judge configurations and the rubric and verdict instructions used to assess decks.

B Harness gap by criterion and on text alone

  • The first text-only pass gave the harness a mean advantage of 17.5 to 36.4 points over direct 27B generation and −4.1 to +1.9 over direct Opus; each judge preferred it to the 27B on 16 or 17 deliverables.The authors identify these results as secondary to the cleaned pass; Table 6 breaks down criterion advantages and Table 7 covers the cleaned text-only comparison.Source: B Harness gap by criterion and on text… · p. 10

C Protocol for grading with an LLM judge

  • For judge grading, randomize deck assignments across sessions, treat same-session decks as clusters, estimate variability with uncertainty, and compare within a fixed judge configuration.Anchor judge changes on shared decks; repeat grading, and do not treat a small change as conclusive from one pass.Source: C Protocol for grading with an LLM judge · p. 11

5. Read the continuous score rather than a ship-

  • Read the continuous score alongside the ship-or-not verdict, record judge-call configuration, and use text-only passes with a length check when presentation differs.The protocol also recommends checking a subset of decks with human experts before trusting absolute score levels.Source: §5 · p. 11

D Judge configurations

  • The judge-configuration table records judge identity, reasoning effort, sampling settings, and whether slide images were omitted.Source: Table 8 · p. 11

E Judge instruction

  • The rubric weights consistency and auditability at 20 points, four criteria at 15 each, title storyline at 10, and two criteria at 5 each.Judges provide criterion scores with quotations, a total, a verdict, and five serious defects; a send-as-is verdict is barred by unverifiable figures, contradictions, machine-sounding wording, or noticeable layout defects.Source: E Judge instruction · p. 11

F Verdicts against scores

A 70-point threshold classifies most development Opus judgments, but verdict categories overlap substantially. The section also points readers to paired-score and judge-agreement analyses.

F Verdicts against scores

  • 86% of development Opus judgments fall into a class under the 70-point threshold, while the verdict classes overlap from 56 to 79.Figure 6 shows the distribution of development Opus judgments by verdict.Source: F Verdicts against scores · p. 12, Figure 6 · p. 12

G Paired scores on identical decks

H Judge agreement

  • Figure 8 projects dependability when averaging judges, while round-specific rank agreement is shown in Figure 5.The projection is described as conditional on the round-7 variance decomposition.Source: H Judge agreement · p. 12

I Evaluation inventory and archives

The evaluation inventory distinguishes recoverable scores from repeated measurements and documents what was excluded from the archive. The statistical estimates use defined scoring records, while their dependence and projection assumptions constrain interpretation.

I Evaluation inventory and archives

  • 1,666 evaluation records include 1,661 recoverable totals; Table 9 separates evaluation phases, and missing totals are not imputed.These records are repeated measurements of the same deliverables, not independent documents; parsed scores were checked against raw responses.Source: I Evaluation inventory and archives · p. 12, Table 9 · p. 12
  • Figure 7 plots deck-by-deck scores from two passes on identical decks for the Opus rounds 7 and 8 and the Opus/Sonnet round-5 pair.Open points indicate changed verdicts between passes.Source: Figure 7 · p. 12
  • The archive audit excluded pilot scores, earlier verdicts, and calibration reports whose deck sets, formats, grids, or admissibility rules differed from the main study.The pilot exclusion rationale was not fully recorded contemporaneously, limiting retrospective selection checks.Source: I Evaluation inventory and archives · pp. 12–13
  • An earlier free-field versus compiled-assertions case records 121 versus 41 model calls and approximately 191 versus 85 seconds, but does not establish a component effect on final-deck quality.The prose and thesis differed, the claimed common registry lacked a preserved snapshot, and later tuning was not an independent replication.Source: I Evaluation inventory and archives · p. 13

J Statistical estimands

  • Fixed-group repetitions estimate within-context variability, while reassigned sessions and configuration comparisons vary additional factors; their detectable-change estimates are not interchangeable.The two-way model uses deck and judge effects, with the residual combining interaction and error; negative component estimates are truncated to zero, and session and model-family effects are not fitted.Source: J Statistical estimands · p. 13
  • The pooled Opus estimate reuses 17 deliverables across 34 differences, and its bootstrap and chi-square intervals omit repeated-document and session dependence.Judge-averaging projections assume uncorrelated judge and residual contributions and are conditional on the convenience panel; extrapolation beyond it is unvalidated.Source: J Statistical estimands · p. 13, Figure 8 · p. 13

Related papers

Loading 2609.39958v1…