PaperlayerView exact revision on arXiv

Source-linked AI summary

FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents

Hoyoung Lee, Suyeol Yun, Jack Haverty, Yunju Cho, Meesong Kim, Daekyung Park, Sumin Kim, Jihoon Kwon, Jasmine Jia Geng, Andrew Chin, Yin Luo, Edward Tong, Yu Yu, Zach Golkhou, Minkyu Kim, Igor Halperin, Young Cha, Alejandro Lopez-Lira, Chanyeol Choi, Yongjae Lee

arXiv:2609.35744v2cs.AIq-fin.CP

TL;DR

Finance research evaluation needs cutoff-specific expected values and rubrics that reflect institutional standards, but expert-authored benchmarks are costly to update and fixed per item. FinAutoRubric uses reusable guidance and query-specific research, review, and validation; on three finance benchmarks, it tracks expert scoring as closely as the strongest evaluated generator while stating expected values for more criteria, and its rubrics also agree with human grading and are preferred by analysts.

  • Problem

    Expert-authored finance benchmarks are costly to extend and fix one cutoff and standard per item, while institutions may need different standards and expected values grounded as of a cutoff.

    Source: §1 · p. 2

  • Method

    FinAutoRubric uses reusable expert guidance and a Task Bank to direct agents and code in researching, reviewing, and validating query-specific rubrics.

    Source: §1 · p. 3

  • Results

    On three finance benchmarks, FinAutoRubric tracks expert scoring as closely as the strongest evaluated generator while stating expected values for more expert criteria; its scores agree with human grading, and analysts prefer its rubrics in a blind review.

    Source: §4.1 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    The 100-query released benchmark spans 78 primary tasks and eight asset classes, enabling evaluation beyond the predominantly public-equity focus of existing public sets.

    Source: §1 · p. 3

    Automated review checks source grounding and internal consistency but does not certify that an ambiguous query was interpreted as its author intended.

    Source: §4.4 · p. 10

Abstract

from arXiv · show

Evaluating finance research agents requires rubrics that reflect expert standards and fix the values correct as of an information cutoff. Expert-reviewed finance benchmarks rely on fixed, per-item rubrics, which are costly to extend and cannot encode each institution's own standard. In FinAutoRubric, experts specify reusable evaluation guidance, while agents and code carry out query-specific rubric generation, review, and validation. This expert guidance governs every agent, as prompts and as rules that code enforces, and a Task Bank of reusable criteria carries it across tasks. In long-horizon loops that follow the expert guidance, a writer agent researches every expected value and a reviewer agent verifies it, and failures escalate to a human. On three expert-authored finance benchmarks, its rubrics track expert scoring as closely as the strongest evaluated generator while stating the expert rubric's expected value for more criteria, their scores agree with human grading, and in-house analysts prefer them in a blind review. The released 100-query FinAutoRubric Benchmark, built from in-house analysts' key questions across 78 tasks and eight asset classes, shows that rubrics from an earlier model generation still leave headroom for a later one.

1 INTRODUCTION

Finance research agents need item-specific rubrics that verify expected values at an information cutoff and reflect an institution’s standard, but expert-authored benchmarks are costly to update and fixed to one context. FinAutoRubric uses reusable expert guidance and query-specific research and review; on existing benchmarks, its rubrics track expert scoring while providing expected values for more criteria, and its released benchmark broadens task and asset-class coverage.

1 INTRODUCTION

  • Expert-authored finance rubrics are costly to update, fixed to one cutoff and perspective, and cannot encode each institution’s own standard.Extending benchmarks to new asset classes also requires authoring new items.Source: §1 · p. 2
  • Existing rubric-generation methods do not research expected values as of a cutoff before responses exist or apply an institution’s rules as coded checks.Finance evaluation needs both cutoff-specific values and institution-specific standards.Source: §1 · p. 2
  • FinAutoRubric uses reusable expert guidance and a Task Bank to govern query-specific rubric research, review, and validation, leaving the process inspectable and revisable by institutions.Writers research values from permitted sources or scripts; reviewers recheck them, and code validates outputs and escalates failures.Source: §1 · p. 3
  • On three finance benchmarks, FinAutoRubric tracks expert scoring as closely as the strongest evaluated generator while stating expected values for more expert criteria.Its scores also agree with human grading, and in-house analysts prefer its rubrics in a blind review.Source: §1 · p. 3
  • The released FinAutoRubric Benchmark contains 100 analyst-derived queries across 78 primary tasks and eight asset classes.It extends evaluation beyond the predominantly public-equity focus of existing public benchmarks.Source: §1 · p. 3

2 THE FINAUTORUBRIC FRAMEWORK

FinAutoRubric separates reusable expert guidance and task criteria from each query’s context and expected values. Its three stages build a Task Bank, contextualize queries, and generate, review, and validate rubrics under expert instructions and coded rules.

2 THE FINAUTORUBRIC FRAMEWORK

  • Expert guidance governs every agent through prompt instructions and coded orchestrator rules that reject noncompliant outputs and request human intervention.The framework builds a Task Bank, contextualizes queries, then uses writer–reviewer loops to research expected values and produce rubrics.Source: §2 · p. 3

2.1 BANK CONSTRUCTION (STAGE 1)

  • The Task Bank holds reusable, value-free criteria for tasks, drafted by models and screened by experts before being frozen at 200 tasks.Analysts’ queries are converted into task signatures, clustered by analytical operation, and used to draft criteria tagged to principles.Source: §2.1 · p. 4

2.2 CONTEXTUALIZATION (STAGE 2)

  • Stage 2 separates a query into a value-free task signature and verbatim instance-specific bindings, then selects a bank task only on a complete, unique fit.Otherwise, task-level criteria are written from the signature and principles; the cutoff is recorded as an inclusive date.Source: §2.2 · pp. 4–5

2.3 RUBRIC GENERATION (STAGE 3)

  • Before research, the orchestrator divides 100 points equally among query obligations and freezes that budget so retrieved values cannot shape initial weighting.After research, the composition agent may shift points when minor obligations dilute the query’s central one.Source: §2.3 · p. 5
  • The writer grounds expected values in cited sources or executed scripts, while a different-family reviewer reopens sources and re-derives values, calculations, and score roles.A validator checks each submission against its task contract, and revisions are reviewed again up to a bound.Source: §2.3 · p. 5
  • A reviewer-family composition agent can edit the whole rubric, but edits must pass coded gates, preserve a 100-point total, and be confirmed in a fresh session.If a run fails acceptance, the orchestrator blocks it and requests human intervention; stage records let institutions trace how guidance shaped the rubric.Source: §2.3 · p. 5

3 EXPERIMENTAL SETUP

The experiments compare generated rubrics with expert rubrics by scoring shared responses with fixed judges, using public finance benchmarks, matched generator baselines, and human-graded responses. Metrics capture score agreement and expected-value coverage.

3 EXPERIMENTAL SETUP

  • The evaluation scores the same responses under each rubric source with a fixed three-judge ensemble, using expert rubrics as the reference.A response includes the agent’s full trace of messages, tool calls, and tool results, followed by its final answer; generators do not see responses.Source: §3 · p. 6

3.1 DATASETS

  • The study uses all 50 public BigFinanceBench queries and all 27 public Finance Agent Benchmark v2 queries, plus a 48-query FrontierFinance sample.ProfBench supplies ten finance tasks with expert-graded model responses for human-grading comparisons.Source: §3.1 · p. 6

3.2 BASELINES

  • Baselines include one-call generation, checklist and scenario-based methods, and retrieved-web-criteria aggregation, with comparisons run in matched settings.Methods requiring trained rubric models or RL rollouts, or using each response to form part of its checklist, are excluded.Source: §3.2 · p. 6

3.3 MODELS AND ENVIRONMENT

  • Writers and reviewers use different model families and harnesses, while isolated sandboxes, access blocks, and trace audits guard against benchmark-data leakage.The audit checks traces and files because web search on provider servers bypasses the network block.Source: §3.3 · p. 6
  • Eight solvers provide BigFinanceBench traces; the other benchmark solvers use the same tool-enabled harness, and three judges decide criterion pass/fail and coverage by majority vote.The judges are Muse Spark 1.3, GLM-5.3-Flash, and DeepSeek V4.1-Flash.Source: §3.3 · p. 6

3.4 METRICS

  • Kendall’s τb and MAE measure response-ranking agreement and score-level differences between generated and expert rubrics.τb is averaged across eligible queries; MAE is calculated over query–response pairs in percentage points.Source: §3.4 · pp. 6–7
  • Coverage counts the share of expert criteria matched by generated criteria with both the same demand and expected value.Criteria count equally, and the judges determine matches.Source: §3.4 · p. 7, Table 1 · p. 7

4 RESULTS

Across three finance benchmarks, FinAutoRubric closely tracks expert scoring while stating expected values for more criteria than baselines. Human evaluations and component analyses examine rubric preferences, scoring agreement, and how the generation loops shape rubric quality.

4.1 COMPARISON WITH EXPERT-AUTHORED RUBRICS

  • FinAutoRubric tracks expert scoring as closely as the strongest generator, leads on BigFinanceBench, and states expert values for more criteria than any baseline.It matches expert values for 60.8% of BigFinanceBench, 40.7% of FrontierFinance, and 80.8% of Finance Agent Benchmark criteria, versus Direct-Generate’s 32.5%, 32.9%, and 46.4%.Source: §4.1 · p. 7

4.2 HUMAN EVALUATION

  • In blind review, all three analysts significantly preferred FinAutoRubric, whose rubric-conditioned scores also achieved the highest pairwise human-grading accuracy, within noise.The blind comparison asked analysts to choose between anonymized rubrics; pairwise accuracy was measured on ProfBench Finance with 95% bootstrap confidence intervals.Source: §4.2 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

4.3 CONTRIBUTION OF EACH COMPONENT

  • Removing the Task Bank lowers agreement and coverage while barely changing MAE, with its largest gains in driver-based projection and valuation-basis comparison.Financial-metric retrieval, stipulated recalculation, and investor-return tasks do not improve, consistent with the bank helping where quality depends on reusable analytical checks.Source: §4.3 · p. 8
  • Removing contextualization produces the largest agreement drop and raises MAE, while changing coverage little; this ablation also removes validator and composition review.The end-to-end comparison counts 11 of 50 runs that fail after two human recovery attempts as zero.Source: §4.3 · pp. 8–9
  • Removing both loops’ reviews lowers coverage and raises MAE more than removing any other component; the loops rebalance support points while keeping outcomes the majority.Every point change carried a material finding and fresh-session confirmation.Source: §4.3 · p. 9, Figure 6 · p. 9

4.4 THE FINAUTORUBRIC BENCHMARK

  • GPT-6 Astra leads overall and in every dimension on the FinAutoRubric Benchmark, yet remains below full marks despite being a later generation than the rubric writer.The benchmark evaluation scores six solvers using point-weighted pass share; automated review does not certify that an ambiguous query was interpreted as its author intended.Source: §4.4 · pp. 9–10, Table 3 · p. 9

5 CONCLUSION

FinAutoRubric uses reusable expert guidance and a Task Bank to guide query-specific rubric generation, review, and validation. Its framework uses typed contracts, coded checks, constrained review stages, and explicit acceptance gates.

5 CONCLUSION

  • Reusable expert guidance and Task Bank criteria govern query-specific rubric generation, review, and validation in long-horizon loops that escalate failures to a human.The approach lets institutions build benchmarks as their tasks and information change, but takes more time and cost than single-call or short-pipeline baselines.Source: §5 · p. 10

A FRAMEWORK DETAILS

  • Each model stage writes files against typed contracts, with one same-session repair turn for contract violations and up to two further writer turns for validator findings.A configuration lock records hashes for the bank, principles, prompts, and validator.Source: A FRAMEWORK DETAILS · p. 14
  • Factorization binds query spans to task obligations before bank matching, while the context stage carries forward the cutoff, bindings, routing plan, and selected criteria.Schema alignment records whether each obligation fits the task-level criteria or conflicts with them.Source: A FRAMEWORK DETAILS · p. 14
  • A pure validator checks targets, coverage, evidence links, dependencies, convention and population closure, cutoff dates, and contamination before review.The writer’s score blocks are bound to a frozen point budget, and each target’s evidence and resolution status are recorded.Source: A FRAMEWORK DETAILS · p. 14
  • Content reviewers verify evidence and coverage, while bounded revisions add research only under stated conditions and reviewers cannot edit the candidate.Revision stages preserve frozen request, obligation, and score-block identifiers; changes to identifiers trigger a full review.Source: A FRAMEWORK DETAILS · p. 15
  • Composition review may revise rows and points, but gates require 100 total points, an outcome majority, material findings for edits, and fresh-session confirmation.A run is accepted only after validation, content review, composition confirmation or unchanged approval, and clearance of all blockers.Source: A FRAMEWORK DETAILS · p. 15

B RUN CONFIGURATION

The run configuration standardizes model and judging conditions, isolates roles, and applies shared comparison rules. It also specifies solver budgets and how failed ablation runs enter the reported metrics.

B RUN CONFIGURATION

  • The writer is GPT-5.6 Sol and content review and composition use Claude Opus 5; the offline bank contains 200 tasks and 1,224 criteria.Bank construction uses query factorization, embeddings, UMAP and HDBSCAN clustering, followed by counterbalanced authoring and review and an expert gate.Source: B RUN CONFIGURATION · p. 15
  • Each role runs in an isolated container, with 3,600 seconds for the writer and a shared 3,000-second reviewer budget per query.Known benchmark, dataset, and code-hosting hosts are blocked, and run manifests lock inputs and configuration artifacts.Source: B RUN CONFIGURATION · pp. 15–16
  • Generators share the author model and reasoning effort, with common access, cutoff, and judging rules where applicable; FrontierFinance uses 48 deterministically selected queries.The selection excludes one query whose expert rubric contains 339 criteria against a median of 21; no query was selected or removed based on a method’s score.Source: B RUN CONFIGURATION · p. 16
  • Six solvers answer the FinAutoRubric Benchmark with one response per query and a 50-step budget; none generates rubrics for this comparison.Runs that reach the budget receive one closing turn with only the final-answer tool; Gemini 3.8 Flash reaches the limit on 11 queries.Source: B RUN CONFIGURATION · p. 16
  • Three judges at medium effort provide majority decisions for criterion pass/fail and coverage in Tables 1 and 4, while Table 3 uses mean scores from three judges outside each solver’s family.ProfBench Figure 5 uses the same three-judge majority rule across all six rubric sources.Source: B RUN CONFIGURATION · p. 16
  • Only the contextualization ablation has failed runs: 11 of 50 after two recovery attempts, counted as zero coverage and zero MAE performance.Failed queries contribute τb = 0 and remain in the denominator; the contextualization row averages τb over 47 queries after excluding one with constant expert scores.Source: B RUN CONFIGURATION · p. 16

C UNCERTAINTY

The section distinguishes marginal uncertainty intervals from paired comparisons and reports where FinAutoRubric’s differences from baselines are supported.

C UNCERTAINTY

  • Paired bootstrap intervals resample the same queries for FinAutoRubric and each baseline, directly estimating uncertainty in their differences.The intervals use 10,000 replicates.Source: C UNCERTAINTY · p. 17
  • Marginal intervals describe uncertainty for individual cells, and overlap between them does not establish that two rows are indistinguishable.Source: C UNCERTAINTY · p. 17
  • On BigFinanceBench, FinAutoRubric has lower MAE than every baseline and higher τb than four of five; differences from the strongest baseline are within noise on the other two benchmarks.Against TICK on BigFinanceBench, the τb interval just reaches zero.Source: C UNCERTAINTY · p. 17

D BLIND EXPERT REVIEW

The blind review finds an overall preference for FinAutoRubric, while analyst rationales expose trade-offs in scope, weighting, and calculation granularity. The section also describes the expert-authored principle library that supplies reusable guidance for task criteria.

D BLIND EXPERT REVIEW

  • All three analysts prefer FinAutoRubric overall, but preferences are not significant on every benchmark considered separately.All three give the same verdict on 23 of 45 queries, each favoring FinAutoRubric; at least two agree on 44.Source: D BLIND EXPERT REVIEW · p. 17, Table 6 · p. 19
  • Analysts value close adherence to the query’s requested scope and greater weight on final answers, while another values step-by-step grading that localizes calculation errors.Expert rows sometimes cover unrequested facts; a flat list of steps can award half credit to an answer that stops halfway.Source: D BLIND EXPERT REVIEW · p. 18
  • FinAutoRubric’s five reported weaknesses concern intermediate-step credit, too few calculation steps, concentrated point allocations, a tolerance band judged too wide, and unexplained fractional points.On Q06, Q07, and Q17, analyst A says the 1% tolerance can accept an incorrect 365-day fiscal-year basis.Source: D BLIND EXPERT REVIEW · p. 19
  • Three of 45 expert rubrics contain errors; analysts identifying the two independently verified errors and the later-discovered Tesla inconsistency preferred FinAutoRubric on those queries.The two verified errors were confirmed against the issuer’s Form 10-K.Source: D BLIND EXPERT REVIEW · p. 20

E PRINCIPLE LIBRARY

  • The principle library is expert-authored guidance from which every task-level criterion is written, and revising it and rebuilding the bank changes subsequently generated rubrics.Its preamble limits activated principles to requirements whose violation materially reduces factual reliability or decision usefulness.Source: E PRINCIPLE LIBRARY · p. 21

F BANK CONSTRUCTION

The Task Bank is built by converting analyst queries into reusable task schemas, then reviewing and refining their criteria. Its examples show criteria tailored to both credit analysis and operating-unit-economics calculations.

F BANK CONSTRUCTION

  • The frozen bank contains 200 tasks and 1,224 criteria, with 3–11 criteria per task and a mean of 6.1.Analyst queries are factorized into task signatures, embedded and clustered, then schemas are authored, reviewed, and refined by publication reviewers and domain experts.Source: F BANK CONSTRUCTION · p. 21, Table 8 · p. 22
  • Task schemas are reviewed across model families, repaired so every criterion applies to every query of its task, and screened by publication reviewers and domain experts.The orchestrator returns criteria written as conditional clauses for repair.Source: F BANK CONSTRUCTION · p. 21
  • The operating-unit-economics schema requires requested measures, traceable inputs, aligned entity and period bases, reproducible calculations, and reconciled results.Source: F BANK CONSTRUCTION · p. 21
  • The credit-report task schema covers issuer and instrument identity, debt capacity, liquidity, valuation, sourced evidence, as-of dating, and a supported credit conclusion.Source: F BANK CONSTRUCTION · p. 22

F.1 WITHIN-TASK COMPARISON

Within recurring BigFinanceBench tasks, rubrics using the Task Bank agree more with expert scores in six of nine tasks, though gains do not extend to all calculation-intensive tasks. The remaining passages trace prompts and a Texas Roadhouse query through factorization, budget allocation, and contextualization.

F.1 WITHIN-TASK COMPARISON

  • Full has higher mean agreement than w/o Task Bank in six of nine tasks, with largest gains in driver-based projection and valuation-basis comparison.Retrieval and stipulated recalculation show no gain, and investor-return tasks decline; the table does not establish which bank criteria caused score differences.Source: §F.1 · p. 23, Table 9 · p. 23

G PROMPTS

  • The released prompts cover offline Stage 1 and online Stages 2–3, with the orchestrator supplying each agent its stage inputs.Prompts are Markdown instruction files, with offline inputs appended as JSON and online inputs read from workspace files.Source: G PROMPTS · p. 24, Table 10 · p. 24

H ONE QUERY END TO END

  • The end-to-end example follows a Texas Roadhouse query asking for the simple average of Bubba’s 33 capex share over five reported quarters, with a stated information cutoff.The writer side runs five turns and the reviewer side three fresh sessions.Source: H ONE QUERY END TO END · p. 25

H.1 FACTORIZATION

  • Factorization separates the requested operation from instance-specific bindings and records two possible meanings of averaging a percentage without choosing between them.The population is fixed as the five most recent quarters reported by January 9, 2026.Source: §H.1 · p. 25

H.2 POINT BUDGET

  • With one obligation, the orchestrator assigns the full 100-point budget to the final-answer role and its single core-outcome target.Source: §H.2 · p. 27

H.3 CONTEXTUALIZATION

  • Contextualization matches the query signature to the bank’s Capital Expenditure Retrieval and Direct Derivation task before evidence gathering, aligning the obligation with its six criteria.The first context file failed a contract check because it named the schema origin task_bank instead of bank, then was corrected.Source: §H.3 · p. 27

H.4 WRITER DRAFT AND VALIDATION

The writer grounded a Texas Roadhouse rubric in filings available before the query cutoff, resolving the averaging convention and validating the draft through repair.

H.4 WRITER DRAFT AND VALIDATION

  • The writer used five pre-cutoff filings to record 13 facts, 9 derived values, and five population members for the Texas Roadhouse query.The fourth quarter of 2024 was derived as fiscal 2024 less the first 39 weeks; the arithmetic mean was selected over the aggregate ratio.Source: §H.4 · p. 28, Table 11 · p. 28
  • The accepted target was 11.8 percent, compared after half-up rounding to one decimal place, with a single numeric core-outcome row.The first sandbox validation returned 34 findings, the second returned 2, and the writer repaired both empty-rationale findings before passing.Source: §H.4 · p. 28

H.5 CONTENT REVIEW

A separate content reviewer verified the Texas Roadhouse evidence and target without requesting content changes; composition review then replaced a single all-or-nothing outcome row with weighted support rows.

H.5 CONTENT REVIEW

  • The different-family reviewer reopened all five filings, recomputed the mean as 11.768121 percent, and confirmed the window ended at Q3 2025.Q4 results were not yet reported by the 2026-01-09 cutoff; the aggregate route gave 11.4 percent, and no source postdated the cutoff.Source: §H.5 · pp. 28–29, Table 12 · p. 29

H.6 COMPOSITION REVIEW AND ACCEPTANCE

  • The proposal's 100-point outcome-only row failed a material check, so composition review added support rows for the window, averaging method, and derived fourth quarter.The outcome row was reduced to 60 points and each support row received 20; the accepted rubric passed all eight confirmation checks.Source: §H.6 · p. 29, Table 13 · p. 30, Table 14 · p. 30

I GENERATED AND EXPERT RUBRICS SIDE BY SIDE

Across three expert-authored benchmarks, the side-by-side examples show generated rubrics allocating more weight to requested outcomes and avoiding specific mismatches in expert rubrics; analysts preferred the generated rubrics in two reported blind-review cases.

BigFinanceBench

  • For QVC, the expert rubric expects ratios based on both $880M and $888M, while FinAutoRubric assigns 60 of 100 points to the two requested ratios.The expert's $880M denominators produce 7.52x and 5.45x, conflicting with its final expected values of 7.45x and 5.40x; all three blind-review analysts preferred the generated rubric for this query.Source: Table 15 · p. 31, I GENERATED AND EXPERT RUBRICS SIDE BY… · p. 31

FrontierFinance

Finance Agent Benchmark

  • For Finance Agent Benchmark, the expert rubric gives the four requested outputs 4 of 14 equal shares, while FinAutoRubric assigns them 80 of 100 points.FinAutoRubric also assigns 20 points to twelve reconciliation lines, including checks that they tie to operating income and Adjusted EBITDA.Source: I GENERATED AND EXPERT RUBRICS SIDE BY… · pp. 32, 34, Table 17 · p. 34

J SENSITIVITY TO CONTEXT VIOLATIONS

On the matched set of context-violating answer pairs, Full rubrics lowered scores in fewer pairs but by a larger mean amount than rubrics without contextualization; a Chipotle interpretation mismatch and historical pipeline differences constrain attribution.

J SENSITIVITY TO CONTEXT VIOLATIONS

  • The evaluation paired 32 existing answers with 50 synthetic variants, scoring 82 distinct final answers under fixed Full rubrics with three models.Original answers were not certified correct, and five pairs had a zero-scored original under Full.Source: J SENSITIVITY TO CONTEXT VIOLATIONS · p. 35
  • On 41 matched pairs, Full lowered scores in 34 (82.9%) by a mean 55.0 points, while w/o contextualization lowered scores in 40 (97.6%) by a mean 43.0 points.Nine pairs without an approved w/o contextualization rubric were excluded; differences also reflect criterion content and weights.Source: J SENSITIVITY TO CONTEXT VIOLATIONS · p. 35
  • The Full reversal was a Chipotle scope mismatch: the accepted rubric applied an increase to all year-end stores rather than new openings.The year-end-scope variant scored 55 versus the original's zero under Full; the w/o contextualization version using new openings scored 45.5 versus zero. The historical arm also omitted the typed validator and composition review, precluding attribution to contextualization alone.Source: J SENSITIVITY TO CONTEXT VIOLATIONS · p. 35
  • Recovered Full and w/o contextualization judgments used a 32,768-token output cap and excluded DeepSeek's Krea provider route.The initial Full run had no explicit DeepSeek output cap; the recovered calls also used a process-enforced 300-second timeout.Source: J SENSITIVITY TO CONTEXT VIOLATIONS · p. 35

Related papers

Loading 2609.35744v2…