Split the Labor: Separating Evidence Interpretation from Decision Aggregation

Zhelun Wu

Source-linked AI summary

Split the Labor: Separating Evidence Interpretation from Decision Aggregation

Zhelun Wu

arXiv:2608.14509v1Aug 14, 2026cs.AIcs.CLcs.LG

TL;DR

The paper addresses how to separate evidence interpretation from decision aggregation in multi-source systems. It proposes an explicit evidence interface and calibrated aggregation, showing analytically that count-scale drift affects vote rules and reporting a large but confounded margin in a second instantiation.

Problem
Multi-source decision-support systems concatenate heterogeneous evidence, losing provenance, allowing uninformative sources to dominate, degrading retrieval, and forcing conclusions without support.
Method
The paper separates per-source interpretation from aggregation through a four-field tuple carrying hypotheses, reliability buckets, rationales, and provenance.
Results
Calibrated log-likelihood-ratio pooling removes count-scale drift analytically, while the second instantiation reports a large margin on an uncontrolled comparison.
Takeaways & Limitations
Evidence organization is an architectural decision: explicit interfaces make reliability estimable, provenance structural, failures localizable, and abstention expressible.
Takeaways & Limitations
Instantiation I evaluates only 33 cases after filtering out instances labeled not determinable, while the aggregation repair is not empirically tested.

Abstract

from arXiv · show

Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt. This conflates two operations with different requirements. Interpreting a source rewards capacity and context. Combining interpretations rewards fixed arithmetic, comparability across instances, and the option to return nothing. Once separated, the design problem becomes the interface between them. We propose a four-field evidence tuple (hypothesis, reliability bucket, rationale, provenance) and show that fixing it determines both halves. The separation also reveals a failure mode in how such systems combine, which we call count-scale drift. Thresholding a sum of unnormalized weights is exactly posterior thresholding, but at an operating point that slides with the number of sources consulted. The slide grows with reader reliability. When source reliabilities differ, the vote rule and the posterior order instances differently, and no threshold reconciles them. Pooling calibrated log-likelihood ratios addresses both problems. The fix is arithmetic rather than architectural, and applies to a class of rules beyond language models: score-summing triage engines, diagnostic panels scored by counting positives, and additive multi-signal detectors. We then instantiate the principle twice on one longitudinal corpus, once after outcomes resolve and once before. The same partition helps in both, at different granularities: over reading in the first, over learning capacity in the second. There, a small sequence encoder on an easy auxiliary objective plus a tree ensemble carrying the censored survival loss reaches 0.921 AUPRC against 0.805 for a hand-crafted baseline. We separate what transfers from what must be re-estimated per domain, and state five predictions that would falsify the framework, three negative results, and which comparisons remain confounded.

1 Introduction

The paper argues that evidence interpretation and decision aggregation should be separated by a fixed, auditable evidence-tuple interface. This exposes count-scale drift in unnormalized vote rules, motivates calibrated log-likelihood-ratio pooling, and supports a falsifiable framework whose empirical and analytical claims remain distinct.

Motivation: A monolithic reader conflates nuanced source interpretation with aggregation requiring fixed arithmetic, comparability, calibrated strength, and a reject option.The resulting interface fails to preserve provenance, account for heterogeneous evidence quality, or prevent uninformative sources from dominating.
Design principle: The proposed principle reads each source in isolation under a causal context constraint, emits a fixed evidence tuple, and combines tuples with an explicit auditable rule.The contract makes reliability declarative, provenance structural, and weighting an estimation problem rather than a tuning problem.
Novelty and method: The contribution is the composition of established mechanisms: the partition, the contract at its interface, the admissible combination rules, and the diagnosis of unsuitable aggregation.The paper also specifies observable reliability buckets, dependence discounts for shared upstream origins, and data-based estimation of sensitivity and false-positive rates.
Combination rule: Count-scale drift makes thresholding unnormalized weight sums equivalent to posterior thresholding at an operating point that rises linearly with source count.The bias favors over-assertion on evidence-rich instances and worsens for more reliable readers; calibrated log-likelihood-ratio pooling removes both defects under stated conditions.
Scope and evaluation: The same partition is tested on one corpus after outcomes resolve and before they do, while the framework separates domain-neutral components from quantities requiring domain-specific re-estimation.The paper states five falsifiable predictions, three negative results, and an explicit confound register.
Interpretive boundary: Empirical claims concern a running system using the criticized weighted-vote combination step, whereas log-likelihood-ratio pooling is derived rather than deployed.Thus the empirical evidence bears on partitioning and reading strategy, while the analytical results bear on the combination rule.

2 Related work

The paper situates its evidence-aggregation design among selective prediction, unreliable-source aggregation, reasoning decomposition, retrieval, attribution, and survival-analysis work. It distinguishes its structural provenance output, supervised confidence-bucket estimation, and discrete-time tabular-learning approach from these precedents.

Selective prediction: Selective prediction motivates the paper’s emission threshold through the coverage/risk trade-off, while conformal prediction is presented as an alternative thresholding framework.The paper reports no risk–coverage curve and defers its reasons for not adopting conformal prediction to Section 9.
Aggregating unreliable sources: The paper’s confidence-bucket error rates coarsen Dawid–Skene by replacing annotator identity with reader-confidence buckets and estimating parameters supervised.This reduces parameter count at the observed label frequencies, trades label efficiency for stability, and contrasts with an unsupervised baseline.
Decomposition and attribution: Unlike prior decomposition methods that majority-vote over samples from one context, the paper aggregates independent evidence sources and makes provenance a structural output.RAG usually concatenates supplied sources, while attribution and consistency-based metrics assess groundedness after generation.
Survival analysis and tabular learning: The paper uses discrete-time survival analysis and combines competing-risk, neural, and tree-based precedents, with boosted trees remaining competitive for moderate-scale tabular data.Its tree-based survival-loss result is framed as consistent with prior evidence that trees remain competitive on tabular data.

3 The evidence interface

The evidence interface separates source reading from aggregation through a four-field tuple, allowing interchangeable readers while requiring a shared calibration scale. Its design depends on locality, reliability heterogeneity, dependence, and outcome resolution, with causal reading preserving traceability but inducing dependence.

3 The evidence interface: Each reading returns a hypothesis, ordinal reliability bucket, rationale, and provenance, making commitments explicit and enabling outcome-based weighting.Buckets are ordinal because elicited model confidence is poorly calibrated, so they are built from observable properties of the reading operation.
3 The evidence interface: Any reader that emits the tuple can compose, including language, telemetry, vision, and human systems, provided they share a calibration scale rather than an architecture.This shared scale is the mechanism supporting transfer claims across reader types.
3 The evidence interface: Four properties fix most design choices: evidence locality, reliability heterogeneity, dependence structure, and resolution regime.The two studies differ only in whether outcomes have resolved, making them one experiment on the principle rather than unrelated applications.
3 The evidence interface: Partitioning is worthwhile when source counts vary, reliability is heterogeneous, provenance is obligatory, and abstention is valuable despite added estimation and maintenance costs.Partitioning also forfeits joint reasoning across sources.
3 The evidence interface: Sources are read alone with bounded causal context, preserving traceability and failure isolation while inducing dependence that requires discounting shared information during aggregation.Future sources are excluded because live deployment cannot access them; full-context reading is only an oracle reference.

4 Combining evidence: a defect and its repair

The deployed evidence-summing rule suffers from count-scale drift and heterogeneous misweighting, so no threshold can generally make it equivalent to posterior ordering. Calibrated log-likelihood ratios repair both defects arithmetically, while requiring care about retrieval, bucket assignment, and competing-label evidence.

Uniform versus heterogeneous weights: Uniform corroboration bonuses add no decision power: they are equivalent to thresholding vote count, leaving λ unidentifiable from decisions alone.The bonus matters only with heterogeneous weights, where it trades weight against count without a principled exchange rate.
Count-scale drift: Two instances with two asserting sources receive the same vote score but posterior probabilities near 0.39 and 0.001 when consulted-source counts are three and six.With α = 0.9, β = 0.2, and prior 0.2, their posterior log-odds are −0.46 and −6.69, respectively.
Heterogeneous misweighting: In the heterogeneous case, vote and posterior orderings agree for every instance only when each bucket weight is proportional to its per-bucket log-odds gain.Otherwise, the two orderings cross, so no posterior threshold reproduces the vote rule.
Scope and repair: The defect applies beyond language models to additive triage and detection rules, and the remedy is to sum calibrated log-likelihood ratios before thresholding the posterior.The result requires variable numbers of consulted channels with informative non-firing; unretrieved passages in retrieval pipelines are absent rather than silent, so drift need not appear.
Reader-channel modeling: For single-label readers, competing-label assertions are stronger evidence against a target than silence, motivating a three-outcome emission model rather than independent per-label rate pairs.A full confusion matrix costs O(|Y|2) parameters per bucket, which the available label frequencies do not support.
Posterior pooling: Pooling independently asserting sources makes corroboration automatic and yields probabilities comparable across source counts and bucket compositions, eliminating separate bonus and mixing coefficients.This posterior interpretation is exact only when bucket assignment is ancillary; otherwise, the pooled score is a lower-variance approximation that can conservatively understate evidence.

5 Experimental setup

The experiments vary source-reading strategy or model architecture while holding the other components fixed, using a pinned, low-temperature language-model reader and instance-level bootstrap evaluation. The proprietary corpus prevents external replication, and no experiment directly compares aggregation rules.

Reader and context: The reader is a hosted instruction-tuned language model run at temperature 0.1 with a 4,000-token output cap and one sample per source.The setup uses no self-consistency, and the prompt is pinned to a dated version identifier.
Reproducibility: External replication of the numerical results is impossible because the corpus is proprietary operational data and cannot be released.The paper states that prompts, model settings, construction rules, code, seeds, fold assignments, and metric definitions can otherwise be released.
Experimental factors: The two instantiations vary one axis at a time: source-reading strategy in Instantiation I and model architecture or features in Instantiation II.Instantiation I keeps the deployed weighted vote fixed; Instantiation II keeps the reader fixed.
Experimental factors: No experiment varies the combination rule, so the study does not compare Section 4’s aggregation proposals with the deployed vote.The paper explicitly draws no such comparison.
Uncertainty estimation: Bootstrap confidence intervals resample instances rather than rows, and paired configuration comparisons use the same instance resamples.This avoids understating variance when forecasting test sets contain correlated snapshots per instance.

6 Instantiation I: the resolved regime

In the resolved regime, evaluation is a small, filtered descriptive case report rather than a statistically established benchmark. Independent causal reading is favored for precision and provenance, while its apparent advantage over oracle context remains confounded and requires confirmation.

Evaluation design: Filtering out “not determinable” conditions precision on determinable cases and prevents this evaluation from measuring abstention’s selective-prediction benefit.A complete risk–coverage evaluation on the unfiltered pool remains future work.
Interpretation limits: Table 2 is an internal design comparison, not positioning against published methods, because no external baseline or requested comparator was run.The table also lacks inter-annotator agreement, instance-level bootstrap intervals, and source-count distribution, reinforcing its descriptive status.
Reading strategies: Independent causal reading is the most precise configuration, while prior fallback raises macro F1 from 0.416 to 0.466 and keeps precision above one-shot.On instance F1, the recommended configuration and one-shot concatenation are identical at 0.614; the study’s case for independent reading also rests on per-source provenance.
Reading strategies: The oracle loses to causal context on instance F1 (0.534 vs. 0.587), but the comparison is confounded by both future-evidence availability and total context length.No arm separates hindsight contamination from context dilution; a length-matched control varying only extra-context informativeness is proposed.
Reading strategies: Causal context is at least as good as oracle context on every Table 2 metric: precision 0.824 vs. 0.816, recall 0.459 vs. 0.439, instance F1 0.587 vs. 0.534, and macro F1 0.416 vs. 0.392.The supported claim is absence of a causal-context penalty, not a causal-context gain.

7 Instantiation II: the unresolved regime

In the unresolved regime, the partition shifts from reading-versus-combining to representation learning-versus-censored decision modeling. The hybrid reaches 0.921 AUPRC, but interpretation is limited by population differences, semantic leakage, unresolved feature attribution, and temporal calibration failure.

Partition: The unresolved setting shifts the separation from reading versus combining to representation learning versus censored timin.Outcomes have not occurred, so labels and empirical priors are unavailable, and future sources do not exist.
Evaluation: Ranking uses AUPRC at each horizon as the only primary metric, while MAE in days is descriptive and no time-dependent concordance or IPCW Brier score is computed.Metrics use 14,524 resolved plus 2,701 censored-known instances, whereas the length-stratified table uses 4,874; figures across these populations are not compared.
Leakage and attribution: 0.300 AUPRC is added by hand-crafted features, but the increment’s attribution is unidentified because two categoricals are semantically exposed to outcome information.The exposed fields are pipeline stage and the reader’s dominant_reason label; no feature-dropping ablation or fixed-lead-time evaluation was run.
Model results: 0.921 AUPRC is the headline hybrid result, compared with 0.805 for the 18-feature baseline and 0.833 for cadence-only sequence encoding.Combining cadence encoding with the hand-crafted block adds 0.088; the comparison between rows 2 and 5 is suggestive rather than clean because feature counts differ.
Source-length analysis: 0.145 AUPRC separates the one-to-two-source bin from the thirteen-or-more bin, but longer histories and later-stage instances remain confounded.AUPRC rises from 0.795 to 0.940, while descriptive close-time error falls from roughly 59 to 39 days; the data cannot distinguish greater sequence capacity from trajectory maturity.
Robustness: 0.009 AUPRC was lost after isotonic calibration across a temporal shift, so scores are consumed as rankings pending shift-aware calibration.No weight setting improved AUPRC after down-weighting synthetic midpoint rows, and pooled models beat per-segment models despite per-segment AUPRC ranging 0.723–0.969.

8 Transfer

The evidence contract, pooling arithmetic, and drift diagnostic transfer across domains, but reader reliabilities, blocking rules, and thresholds require domain-specific re-estimation or specification. These domains especially benefit because provenance is required and no answer is a legitimate output.

Cross-domain instantiation: Table 5 applies the contract across five external domains that aggregate partially reliable signals while requiring justified conclusions.Several use fixed-cut score summation, which corresponds to a drifting posterior threshold under Corollary 1.
What transfers: The contract, pooling arithmetic, and drift diagnostic transfer without re-derivation across domains.The contract is a schema; Equation (9) is Bayes’ rule under stated independence; and the diagnostic uses logged channel counts, firings, and outcomes.
What must be re-estimated: Reader reliability rates, blocking rules, and thresholds do not transfer unchanged and must be re-estimated or specified for each domain.Reliabilities depend on the reader and corpus, blocking depends on shared upstream origins in the data pipeline, and thresholds reflect local costs of wrong assertions versus no answer.
Why transfer matters: Provenance is especially valuable in these domains because each must show the clinical, regulatory, or evidentiary basis of its conclusions.This makes provenance a structural output rather than merely a post-hoc explanation.
Why transfer matters: No answer is legitimate, and sometimes preferred, in these domains, matching the calibrated-evidence rule’s ability to abstain.The passage contrasts this with a generative reader’s unreliable production of no answer.

9 Discussion

The paper’s two instantiations support separation at different granularities, but share an architecture rather than a demonstrated unified method. This partition exposes distinct uncertainties, enables modular improvement, and leaves conformal prediction and pooled-posterior integration as limitations or tests.

The two instantiations apply the same separation principle at different granularities: per-source interpretation plus aggregation, and representation learning plus censored timing estimation.The forecasting result is described as a large margin on an uncontrolled comparison.
The studies are adjacent contributions sharing a corpus and reader, because forecasting uses categorical reader features but omits LLR pooling, abstention, and dependence discount.A demonstrated unification would require supplying the pooled posterior ˆp(y) and its abstention flag to the forecasting model.
The architecture distinguishes source uncertainty, represented by a low reliability bucket, from aggregate uncertainty, represented by a posterior in the abstention band.It can abstain at either level and report which.
Conformal prediction remains a limitation because exchangeability fails across the temporal split and conformal sets do not provide per-source attribution.Time-series conformal variants are identified as the natural next step.
The partition supports independent improvement: readers can be replaced without changing arithmetic, and arithmetic can be corrected without rereading sources.This modularity is presented as a lifecycle advantage over monolithic systems.

10 Falsifiable predictions

This section presents five falsifiable predictions, each paired with a discriminating experiment feasible using existing data or one additional training run. The most consequential untested prediction is that a pooled posterior with abstention information will improve forecasting over raw extracted categories.

10 Falsifiable predictions: Five falsifiable claims are paired with experiments that can use already collected data or one additional training run.The framework treats testability as a criterion of usefulness.
10 Falsifiable predictions: Count-scale drift should be practically material: vote-rule precision should decline with source count relative to per-bin prevalence, unlike posterior thresholding.Evaluation should stratify instances by source count and compare per-bin precision with per-bin prevalence; a flat profile would make the proposition practically irrelevant at the observed n.
10 Falsifiable predictions: The most consequential untested prediction is that forecasting with the pooled posterior and its abstention flag will outperform forecasting with raw extracted categories.Testing this would turn the two instantiations into one system rather than two applications of one principle.

11 Limitations

The evaluation has limited coverage and unresolved confounding: Instantiation I excludes cases where abstention is correct, while Instantiation II does not identify the credit assigned to reader-extracted content. Additional interpretation limits arise from conditional metrics, resolution-based sampling, finite calibration, correlated reader errors, and reliance on one domain.

Evaluation coverage: Instantiation I reduced 1,000 labelled instances to 162 eligible cases and 33 evaluation instances by removing “not determinable” labels, excluding cases where abstention is correct.At 33 instances, no ordering among Table 2 rows is statistically supported.
Feature confounding: Instantiation II cannot identify whether pipeline stage and the reader’s reason label predict outcomes or encode them, confounding the reported +0.300 increment and operating points.Row 5 reaches 0.833 without either exposed feature, while row 2 reaches 0.805 with both, but the rows also differ in architecture and feature count.
Metric interpretation: Threshold precision and recall are conditional on resolved instances, whereas ranking metrics use the scored evaluation population; restricting tests to pre-cutoff resolutions also over-represents fast-resolving cases.These choices make threshold metrics unlike deployed precision over all scored instances and motivate AUPRC as the primary metric.
Method and domain limits: Calibrated pooling cannot correct label-correlated reader errors, and finite calibration makes rare-label log-ratios unstable as β →0 while clipping biases evidence weaker.The reported mechanisms also rest on one corpus in one domain.

12 Broader impacts

The deployment raises performance-management and privacy concerns because it records employee-related risk flags and attributes deal outcomes to named individuals. Its safeguards preserve human control and provenance, while its data practices limit exposure but still require auditing of potentially judgmental labels.

Performance-management context: The deployment operates within a performance-management context by recording identifiable employees’ work and surfacing risk flags on their deals.This applies whether or not the system is formally used for performance management.
Human oversight and provenance: Written values never overwrite human entries, carry machine-generated provenance flags, and link assertions to supporting sources for contestability.These safeguards are described as two properties already enforced, alongside the system’s source-linking requirement.
Data governance and labeling: The corpus is processed under its recording-consent and retention policy, exposes no participant identifiers beyond transcript content, and releases no personal data.Because explanations of lost deals attribute causes to named individuals’ conduct, label sets should also be audited for implicit performance judgments.

13 Conclusion

The paper argues that evidence organization is an architectural decision centered on a cut and contract, not merely prompt construction. A four-field evidence tuple makes reliability estimable, provenance structural, failures localizable, and abstention expressible, while designed interfaces support inspectable conclusions with calibrated evidence and no-conclusion options.

Architectural interface: A four-field evidence tuple at the architectural cut makes reliability estimable, provenance structural, failures localizable, and abstention expressible.The cut and its contract determine both halves of the system.
Combination failure: The paper identifies count-scale drift as a failure mode in how such systems usually combine evidence.The supplied passage introduces this failure mode as part of the conclusion’s architectural argument.
Transfer and re-estimation: What transfers across settings is the cut, contract, pooling arithmetic, and a diagnostic computable on logs that most such systems already keep.The constants do not transfer and must be identified separately.
Inspectable conclusions: Inspectability requires conclusions accompanied by supporting evidence, calibrated strength, and the standing option of no conclusion.The passage frames this as an interface property rather than a prompting property.