Harnessing The Collective Wisdom: Fusion Learning Using Decision Sequences From Diverse Sources

• Binary-only reporting, cross-source heterogeneity, dependence, and incompatible study designs complicate coherent meta-analysis.Existing tools often require continuous summaries such as p-values, while some studies report only rejection lists.Source: §1 · p. 1
• IRT is proposed as a general framework for fusing multiple statistical testing decisions from diverse studies.The framework uses study-specific binary decisions, FDR levels, and tested hypotheses.Source: §1 · p. 1
• IRT constructs evidence indices from binary decisions, then fuses them into a single measure of evidence for each hypothesis.Source: §1 · pp. 1–2
• IRT guarantees overall FDR control when individual studies control their designated FDR levels, including under arbitrary dependence among fused evidence indices.Source: §1 · p. 2
• The framework can combine p-value reports with rejection-only reports, remains model-free, and applies across different underlying multiple-testing algorithms.Source: §1 · p. 2

2.1 Notations and problem setup

• IRT targets meta-analysis with multiple studies whose available outputs are decisions, FDR levels, and tested-hypothesis sets rather than original test statistics or p-values.The goal is to produce joint rejections across all unique hypotheses while controlling overall FDR at a user-specified level.Source: §2.1 · pp. 2–3

2.2 The IRT framework

• The framework has three stages: construct evidence from binary decisions, aggregate evidence across studies, and generate final discoveries with FDR control.For each study, the decision sequence is converted into evidence against null hypotheses before cross-study aggregation.Source: §2.1 · p. 2, §2.2 · p. 3
• For a study controlling FDR at αj, IRT’s evidence indices form compound e-values, with rejected hypotheses sharing study evidence and larger or more conservative studies receiving greater weight.Theorem 1 establishes the compound e-value property for the study-level evidence indices.Source: §2.2 · p. 4
• IRT aggregates study evidence by averaging, assigning e-values of 1 when a hypothesis is absent from a study; the resulting collection remains compound e-values.Theorem 2 supplies the cross-study validity guarantee, while simple averaging has only heuristic justification in this specific compound-e-value setting.Source: §2.2 · pp. 4–5
• The e-BH route controls FDR under arbitrary dependence, but cannot reject when the target α is smaller than every contributing study’s αj.This boundary reflects the inability to create stronger evidence from binary decisions alone without additional information or assumptions.Source: §2.2 · p. 5
• When an independent study supplies full p-values, ep-BH can combine them with aggregated e-values and may yield more discoveries than either p-values or e-values analyzed separately.The e-values act as unnormalized, data-driven weights for re-weighting the independent study’s p-values before BH.Source: §2.2 · p. 5

• The hybrid procedure combines Cauchy-pooled p-values from reporting studies with IRT-generated aggregated e-values from binary-decision studies before applying ep-BH.The block-diagonal covariance structure makes the pooled p-values and aggregated e-values independent for FDR control.Source: §3 · p. 6
• Cauchy + IRT uniformly outperformed Cauchy d1 across all correlation levels while all integrative methods controlled FDR at 0.5%.The comparison used 2 p-value studies, 3 binary-decision studies, and 2,000 Monte Carlo repetitions.Source: §3 · p. 6, Figure 1 · p. 6
• Conservative p-value imputation had the least power, whereas using p-values from all studies produced the highest power overall.At high dependence, pooling only the p-value studies could be less powerful than a single study.Source: §3 · p. 6
• Cauchy + IRT was more powerful than Cauchy + Imputation, Cauchy d1, and the Single Study across all p-value-study correlations while controlling FDR at 1%.This example used asymmetric dependence: correlated p-value studies and independent binary-decision studies.Source: §3 · pp. 6–7, Figure 2 · p. 7

• IRT fuses binary decision sequences from diverse studies to conduct integrative inference on a common parameter of interest.For dependent meta-analysis, it accommodates settings where some studies report p-values and others report only FDR-controlled rejections.Source: §4 · p. 7
• Extending IRT to partial-conjunction hypotheses requires a powerful aggregation scheme for ranking composite null hypotheses.The target is testing whether at least u out of d studies reject a hypothesis.Source: §4 · p. 7
• The current framework does not handle settings where some studies reveal only ranks of top differentially expressed genes.Extending IRT to mixed data types across dependent studies is identified as future research.Source: §4 · p. 7

• The supplement presents IRT* as an alternative evidence-aggregation scheme and gives proofs of the paper’s theoretical results.Source: Supplementary material · p. 11
• Additional analyses connect IRT evidence indices to aggregation and derandomization procedures and establish asymptotic FDR control when some studies control FDR asymptotically.Source: Supplementary material · p. 11
• The supplement extends IRT to alternative forms of Type I error control and adds numerical studies plus a real-data application.Source: Supplementary material · p. 11

• When α is smaller than some study-specific levels, IRT may fail to recover discoveries identified by lower-weight studies and may have no power.This issue arises when pooling aims for higher reliability and α is often below the minimum study-specific FDR level.Source: A Impact of α on the power of IRT · p. 11
• With two studies rejecting only the same hypothesis, IRT requires α ≥ 2α1α2/(α1 + α2), which is at least min(α1, α2).Equality with min(α1, α2) holds when α1 = α2; the three-study setting shows a similar pattern.Source: A Impact of α on the power of IRT · p. 11, Figure 3 · p. 12
• The threshold behavior reflects IRT’s need to preserve FDR validity under arbitrary or worst-case inter-study dependence.Identical data and methods across studies may provide no fresh information beyond an individual study.Source: A Impact of α on the power of IRT · p. 11

B IRT*: more powerful evidence aggregation via multiplication

• IRT* multiplies scaled study-specific e-values under conditional independence, producing valid e-values for each null hypothesis.Theorem 3 requires the Lemma 1 conditions and independence of summary statistics across studies for each testing problem.Source: B IRT∗: more powerful evidence… · pp. 12–13
• IRT* averages products over all k-element study subsets, preserving the e-value property while reducing sensitivity to one study failing to reject.The averaging construction uses the fact that averages of e-values remain e-values.Source: B IRT∗: more powerful evidence… · p. 13
• Under exchangeability, symmetric decision rules, and independent summary statistics, IRT* retains valid FDR control and is relatively robust to exchangeability violations.The stated assumptions support validity, while numerical experiments assess violations and report relative robustness to exchangeability.Source: B IRT∗: more powerful evidence… · p. 13
• The scaling by πj calibrates evidence because discoveries from denser signal settings may require less stringent thresholds.The paper presents this scaling as both a theoretical requirement and an automatic calibration mechanism.Source: B IRT∗: more powerful evidence… · p. 13
• In the two-study example, IRT* can reject H0i when α < min(α1, α2), whereas IRT requires α to remain in the gray region to reject H0i.IRT* rejects H0i at FDR level α if α ≥ 8α1α2/(α1 + α2 + m).Source: B IRT∗: more powerful evidence… · p. 14
• The hybrid IRT_H uses IRT-derived evidence for data-sharing studies and evidence from studies satisfying Theorem 3 conditions, with e-BH guaranteeing FDR control at level α.In the example, IRT_H can reject H0i when α < min(α1, α2), whereas IRT has no power unless α > min(α1, α2).Source: B IRT∗: more powerful evidence… · p. 15

C.3 Proof of Lemma 1

• Under a symmetric decision rule and partial exchangeability, null-study evidences have equal expectations across i, and each satisfies E[eij] ≤mj/|H0j|.Source: §C.3 · p. 16

D.1 Connections to existing aggregation and derandomization procedures

• The construction in Equation (3) is asymptotically equivalent to e-value formulations used by Ren and Barber (2024) and Li and Zhang (2023).The equivalence is stated for large-scale inference as the number of discoveries grows.Source: §D.1 · pp. 16–18
• The generic connection covers ranking, FDP estimation, and thresholding procedures whose resulting e-values can be represented through the IRT evidence construction.The supplement describes this correspondence for several aggregation and derandomization methods.Source: §D.1 · p. 16

D.2 Asymptotic FDR control

• If each study controls FDR only asymptotically, IRT correspondingly controls overall FDR asymptotically as the per-study hypothesis counts grow.Proposition 1 assumes FDR(δj) ≤ αj + op(1) and concludes asymptotic FDR control for IRT.Source: §D.2 · p. 18

E.1 IRT for k-Family-wise error rate (k-FWER) control

• For studies controlling k-FWER, IRT constructs evidence indices using the rejection indicator and a denominator depending on αj, the rejection count, and k.The proposed index uses constants c1 and c2 satisfying the stated condition.Source: §E.1 · p. 19
• If study j satisfies Equation (15), the evidence indices in Equation (16) are compound e-values; Equation (4) gives the aggregation scheme, and when Theorem 3’s conditions hold, Equation (5) represents aggregated evidence indices.The recommended constants are c1 = (2kj − 1)/kj and c2 = (2kj − 1)/(kj − 1).Source: §E.1 · pp. 19–20
• When k = 1, the construction reduces to the FDR evidence index because FWER control implies FDR control.The paper fixes c1 = 1 in this special case.Source: §E.1 · p. 20
• Applying e-BH to the aggregated evidence rejects H0i when eagg_i ≥ m/(αk), controlling k-FWER at level α.This is the stated final rejection rule for the k-FWER extension.Source: §E.1 · p. 20

E.2 IRT for Per-family error rate (PFER) control

• PFER-controlled studies yield compound e-values that can be aggregated by IRT and processed with e-BH.The same product aggregation remains available when Theorem 3’s conditions hold.Source: §E.2 · pp. 20–21
• The PFER extension rejects H0i when eagg_i ≥ m/k and controls PFER at level k.The threshold is the PFER analogue of the k-FWER rejection rule.Source: §E.2 · p. 20
• IRT can combine studies controlling k-FWER, PFER, and FDR by constructing the corresponding evidence indices and aggregating them jointly.The paper identifies IRT*, and the hybrid IRT_H, as applicable when additional dependence or side-information structure is available.Source: §E.2 · pp. 20–21

F.1 d1 studies report p-values while d2 studies report {Dj}j∈[d2]

• In Example 3, Fisher + IRT had the highest power across the evaluated dependence settings, while Cauchy + Imputed and single-study inference were among the least powerful.The comparison used average FDP and ETP over 2000 Monte Carlo repetitions.Source: §F.1 · p. 21, Figure 5 · p. 21
• In the independent setting of Example 4, Fisher dominated competing methods in power and Fisher + IRT was the next-best method.The example varied π0 across 0.5, 0.8, and 0.95; the supplied table caption identifies FDP and ETP comparisons.Source: §F.1 · pp. 21–22, Table 1 · p. 22

Scenario 1 (Independent studies)

• Under independent studies, all methods control FDR, while Fisher has the highest power, followed by IRT*; IRT H outperforms IRT, which has no power here.IRT* also exceeds Fisher + Imputed, while its theoretical conditions explain its advantage over IRT H.Source: §F.2 · pp. 22–23

Scenario 2 (Correlated studies I)

• When studies are correlated, IRT H is the most powerful method with provable FDR control, whereas Fisher and IRT* lose their guarantees.Fisher fails to control FDR at larger correlations, while IRT* appears to control it without theoretical support.Source: §F.2 · pp. 23–24

Scenario 4 (Correlated studies and dependent test statistics)

• With dependent studies and test statistics, IRT provably controls FDR and is substantially more powerful than competing methods lacking theoretical guarantees.This scenario varies the number of studies from 5 to 10 and uses correlations across studies and test statistics.Source: §F.2 · p. 24

Scenario 5 (Varying m_j and α_j)

• As study-overlap heterogeneity increases, IRT* and IRT H become more powerful than IRT, with higher study weights on rejections improving power.The comparison varies η from 0.1 to 0.5 while assigning smaller FDR levels, and hence larger weights, to studies testing more hypotheses.Source: §F.2 · p. 25

Scenario 6 (conservative p-values: I)

• For conservative p-values under independence, power decreases as conservativeness increases; Fisher remains most powerful, while IRT* and Cauchy have similar power profiles.Under correlated studies, Fisher fails FDR control except at the largest conservativeness level, while IRT is more powerful than Cauchy + Imputed when conservativeness is small.Source: §F.2 · pp. 25–26

Scenario 7 (conservative p-values: II)

• With t-based conservative p-values, all methods gain power as degrees of freedom increase; under correlation, IRT outperforms Cauchy + Imputed for ν > 10 while Fisher never controls FDR.IRT* and Cauchy have similar power in the independent setting, whereas IRT* is excluded from the correlated comparison because Theorem 3 does not apply.Source: §F.2 · p. 27

Interpretation

• Near-zero FDP for IRT methods is explained by independent studies requiring simultaneous false rejections, whereas stronger dependence raises FDP and can undermine guarantees.Power can remain high when signals are detected independently by several studies, producing overlap among true discoveries.Source: §F.2 · pp. 27–28

Scenario 1

• When study-specific FDR levels are misspecified, IRT* and IRT H fail FDR control for large α*, but remain relatively robust when α* is small.The evidence indices cease to be compound e-values when α* exceeds the reported 0.01 level.Source: §F.3 · p. 28

Scenario 2

• When cross-study dependence increases, both IRT* and IRT H fail FDR control, whereas small dependence levels preserve relative robustness.The same experiment also finds little material impact from increasing the within-study dependence parameter ρ1, indicating greater robustness to exchangeability violations.Source: §F.3 · pp. 28–29

Scenario 3

• Under AR(1) dependence, IRT* and IRT H continue to control FDR for negative cross-study dependence but fail when test statistics exhibit almost perfect positive dependence.The dependence structure is imposed both across studies for each hypothesis and across hypotheses within each study.Source: §F.3 · p. 29

Study and data construction

• The analysis used eight microarray studies containing 23,367 unique genes, with each gene profiled by a study-specific number n_i of datasets.Nearly 30% of genes were analyzed by one study, whereas approximately 18% were analyzed by all eight.Source: G Real data illustration · p. 30, Figure 16 · p. 31

Evidence construction and aggregation

• IRT converted each study’s expression data into BH-controlled binary decisions, then constructed and aggregated evidence indices before e-BH thresholding at α = 0.1.This procedure supports ranking when studies cannot share p-values or details of their multiple-testing procedures.Source: G Real data illustration · pp. 30, 32

Study heterogeneity

• Study-specific evidence magnitudes were heterogeneous: study 5 produced the largest evidence for rejected hypotheses, whereas study 8 produced the smallest non-zero evidence.The heatmap also showed that studies analyzed different genes and had limited or uneven rejection overlap.Source: G Real data illustration · pp. 30–31, Table 2 · p. 30, Table 3 · p. 31

Ranking and thresholding using IRT and P2E

• IRT ranked highly supported genes even when analyzed by only two or three studies, while P2E’s top 25 genes all had n_i ≥7.IRT rankings therefore reflected the magnitude of study-specific evidence in addition to the number of studies analyzing each gene.Source: Figure 17 · p. 32, G Real data illustration · p. 33

Comparison with P2E

• P2E can produce substantially larger aggregated evidence than IRT when one study rejects a null with an astronomically small p-value.The P2E construction relies on transformed p-values and, in the described setting, independence assumptions for aggregation; IRT’s e-BH thresholding permits arbitrary dependence among aggregated evidences.Source: G Real data illustration · pp. 31–33

Ranking and thresholding using IRT* and P2E*

• IRT* ranked genes with limited study coverage and rejected a higher percentage of hypotheses with n_i ≤6 than P2E*.Among the top 25 genes, IRT* included three with n_i ≤6, compared with two for P2E*.Source: G Real data illustration · pp. 33–34, Table 5 · p. 33