Identification of and correction for publication bias

Isaiah Andrews; Maximilian Kasy

Source-linked AI summary

Identification of and correction for publication bias

Isaiah Andrews, Maximilian Kasy

arXiv:1711.10527v1Nov 28, 2017econ.EM

TL;DR

Selective publication can bias empirical estimates when significance or other result features affect publication. This paper identifies result-dependent publication probabilities and develops corrected inference; in replication studies, results significant at the 5% level were over 30 times more likely to be published than insignificant results.

Problem
Because publication may favor significant, belief-confirming, or surprising results, published estimates can be biased, motivating ways to characterize result-dependent publication.
Source: §1 · p. 2
Method
The paper nonparametrically identifies conditional publication probabilities and derives bias-corrected estimators and confidence sets when those probabilities are known.
Source: §1 · p. 2
Results
Over 30 times more likely to be published were results significant at the 5% level than insignificant results in replication studies; the meta-study approach yielded similar conclusions.
Source: Applications · p. 3
Takeaways & Limitations
The corrections are intended to let readers draw valid inferences from published research while treating the publication rule as given.
Source: Lemma 5 · p. 24
The conditional-independence assumption may fail when publication also depends on other findings in the manuscript or on unreported results.
Source: §3.1.3 · p. 14

Abstract

from arXiv · show

Some empirical results are more likely to be published than others. Such selective publication leads to biased estimates and distorted inference. This paper proposes two approaches for identifying the conditional probability of publication as a function of a study's results, the first based on systematic replication studies and the second based on meta-studies. For known conditional publication probabilities, we propose median-unbiased estimators and associated confidence sets that correct for selective publication. We apply our methods to recent large-scale replication studies in experimental economics and psychology, and to meta-studies of the effects of minimum wages and de-worming programs.

1 Introduction

The paper develops nonparametric methods to identify result-dependent publication probabilities and correct statistical inference, motivated by selective publication that distorts published estimates and confidence sets.

1 Introduction

The paper provides, to the authors’ knowledge, the first nonparametric identification results for conditional publication probabilities as a function of study results.Once publication probabilities are known, the paper derives bias-corrected estimators and confidence sets.Source: §1 · p. 2
Replication data identify publication probabilities from asymmetries in initial and replication estimates, assuming publication depends only on the initial estimate.Absent selectivity, the joint distribution of those estimates is symmetric.Source: §1 · p. 2
Meta-study data identify publication probabilities from deviations in the estimate distributions across study variances, under an independence assumption; both approaches identify probabilities only up to scale.Absent selectivity, high-variance estimates should be a noisier version of low-variance estimates, and multiplying publication probabilities by a constant does not change published-estimate distributions or bias and size distortions.Source: §1 · pp. 2–3
For known selectivity, the paper proposes median-unbiased estimators and valid confidence sets for scalar study parameters, and Bonferroni-corrected intervals when selectivity must be estimated.The supplement also derives optimal quantile-unbiased estimators and results on Bayesian inference.Source: §1 · p. 3

Applications

Over 30 times: results significant at the 5% level were more likely to be published than insignificant results in Camerer et al. (2016) and Open Science Collaboration (2015) replication data.Meta-study estimates using originally published results reached similar conclusions.Source: Applications · p. 3
About 3 times: negative significant minimum-wage employment effects were more likely to be published than insignificant results, while estimates for positive versus negative significant effects were noisy.These estimates use the meta-study by Wolfson and Belman (2015).Source: Applications · p. 3
In the mass-deworming literature, results without a significant child-body-weight impact appeared more likely to enter the meta-study, but large standard errors left no selectivity detectable.The estimates use the meta-study by Croke et al. (2016), and the null hypothesis of no selectivity could not be rejected.Source: Applications · p. 3

2 Setting

The section models publication as selective sampling from a population of studies with potentially different true effects, then explains how this selection shapes observed results and inference. An example illustrates the bias, while extensions address study covariates, alternative observability, and result manipulation.

2 Setting

Latent studies draw true effects from µ and results from a known conditional distribution, publish with result-dependent probability, and are observed only when published.The distribution µ can be degenerate when all studies share the same true effect.Source: §2 · p. 5
The truncated likelihood reweights results by p(x) and, when comparing candidate true effects, normalizes by the expected publication probability at each effect.The paper uses this likelihood as the basis for identification and inference in later sections.Source: §2 · p. 6
Conditioning on study covariates W can allow publication decisions and true-effect distributions to differ across study characteristics; applications condition on journal and initial-circulation year.The theoretical discussion suppresses this additional conditioning for simplicity.Source: §2 · p. 6

2.1 An illustrative example

Under a policy publishing |Z|>1.96 results with probability 1 versus .1 otherwise, published estimates overstate effects; conventional 95% intervals under-cover for small Θ and over-cover for somewhat larger Θ.The policy makes significant results ten times more likely to be published.Source: Figure 1 · p. 7, §2.1 · p. 7

2.2 Alternative data generating processes

When unpublished studies’ existence or results are observed, the truncated likelihood remains valid as a limited-information conditional likelihood, though that extra information may offer further insight.The alternatives include knowing of unpublished studies without their results and observing results from unpublished working papers, as in Franco et al. (2014).Source: §2.2 · p. 8
Assuming a known result distribution limits result inflation, but selecting significant analyses from many independent analyses fits the model, and broader manipulation can be represented by letting publication depend on both results and true effects.The discussion cites Brodeur et al. (2016) on result inflation and Simonsohn et al. (2014) on p-hacking.Source: §2.2 · p. 8

3 Identifying selection

This section examines how publication selectivity can be identified and why meta-regression coefficients may mislead in its presence. It presents replication-based asymmetry and meta-study variation in standard errors, showing how published test-statistic distributions can identify publication probabilities up to scale and the distribution of true effects under the stated assumptions.

3 Identifying selection

Replication studies and meta-studies each identify p(·) up to scale under their respective assumptions; both also identify the distribution of true effects under the stated conditions.Replication-based identification of the effect distribution additionally assumes normally distributed estimates; the meta-study approach assumes independence between latent effects and standard errors.Source: §3 · p. 9

3.1.1 The symmetric baseline case

With conditionally independent, identically distributed original and replication estimates and selection only on the original, asymmetry in published pairs identifies p(·) up to scale.The latent joint distribution is symmetric, so deviations from symmetry in the observed pairs reveal selection.Source: §3.1.1 · pp. 10–12

3.1.2 Generalizations and practical complications

With known normal sampling variances, identification extends to differently sized replications; sign-normalized estimates also identify p(·) when p is symmetric.The unequal-variance result uses deconvolution and identifies µ; the sign-normalized result identifies the distribution of |Θ*| under additional symmetry conditions.Source: §3.1.2 · pp. 13–14, Corollary 1 · p. 14, Corollary 2 · p. 14

3.1.3 Selection depending on Θ∗given X∗

When publication depends on both the estimate and true effect, fZ|Θ is identified, but p(z, θ) is identified only up to a normalization for each θ.This accommodates dependence on other findings, such as alternative specifications and robustness checks, that are not reported in the replication dataset.Source: §3.1.3 · pp. 14–15, Theorem 2 · p. 15

3.2 Meta-studies

With independent latent effects and standard errors and an open interval of standard-error support, meta-studies identify p(·) up to scale and the distribution µ.The identifying variation is that, absent selection, estimates at larger standard errors follow a noise-convolved version of the distribution at smaller standard errors.Source: §3.2 · pp. 16–17

Corollary 3

Under symmetric publication probabilities, sign-normalized meta-study estimates identify p(·) up to scale and the distribution of |Θ*|; selection may also depend on σ through reweighting.The generalized model is observationally equivalent to one that reweights the distribution of σ* by q(·)/E[q(σ*)].Source: Corollary 3 · p. 18, §3.2 · p. 18

3.3 Relation to approaches in the literature

Under the meta-study assumptions, meta-regressions provide a valid test of no selectivity but are difficult to interpret under the alternative.The section also reviews significance-test replication and bunching or discontinuities in p-value and z-statistic distributions as approaches to detecting selectivity.Source: §3.3 · pp. 19–20
0.025 is the same-direction significance replication probability when all true effects are zero, while it approaches one for sufficiently large effects.Thus, the replication rate discussed for Open Science Collaboration (2015) and Camerer et al. (2016) offers little indication of publication bias without restrictions on true effects, unless the rate is below 0.025.Source: §3.3 · pp. 19–20

Lemma 4

Meta-regression tests based on β0 = 0 and γ1 = 0 can detect some publication bias, but these coefficients are generally biased estimates of the average true effect when selection is present.β1 and γ0 recover the latent-study average absent publication bias; the γ1 test may test a zero true effect under no heterogeneity, but has no power against some forms of selectivity. Coefficients are not interpretable with sign-normalized estimates, as in two applications.Source: Lemma 4 · p. 20
Distributions of published p-values and z-statistics have been used to study research-process distortions, including skewness analyses and compilations of test results.Absent selectivity, the model equates the published z-statistic density with the latent density; under a normal sampling model and latent-effect distribution, this imposes testable smoothness restrictions.Source: Lemma 4 · p. 21
Discontinuities in the published z-statistic density, including at critical values such as z = 1.96, indicate corresponding discontinuities in publication probabilities; step-function probabilities can be identified up to scale.The argument uses the smoothness restrictions on the latent z-statistic density implied by the model.Source: Lemma 4 · p. 21
Spikes in the z-statistic distribution indicate selectivity or inflation, while Brodeur et al. (2016) test non-monotonicities that monotone publication probabilities alone cannot explain.Their approach focuses on potential researcher inflation of test results and uses parametrically estimated latent z-statistic distributions.Source: Lemma 4 · pp. 21–22

4 Corrected inference

This section develops frequentist inference conditional on a study’s true effect, correcting for the reweighting of published results. With known publication probabilities, it constructs median-unbiased estimators and confidence sets; it also gives a way to obtain valid confidence sets when publication probabilities are estimated.

Theorem 4

When the published-results cdf is continuous and strictly decreasing in the effect, cdf inversion yields a quantile-unbiased estimator; under the theorem’s additional conditions, it exists uniquely and is continuous and strictly increasing.For normal latent results, the assumptions hold when publication probabilities are strictly positive and almost everywhere continuous.Source: §4 · p. 22, Theorem 4 · p. 23

Lemma 5

The median-unbiased estimator and equal-tailed confidence interval correct the bias and coverage distortions induced by selective publication.In the normal model, the interval has exactly correct coverage.Source: Lemma 5 · p. 23
In the illustration, the corrected estimator is below the usual estimator for small positive results, while the corrected interval is narrower for small results, wider for moderate results, and essentially the same for X ≥5.Figure 4 uses a normal model publishing significant-at-5% results with probability one and insignificant results with probability 10%.Source: Lemma 5 · p. 24, Figure 4 · p. 24
The corrections are intended to support valid inference under a given publication rule, not to recommend changing publication standards.The appropriate correction would itself change under a different rule; standards can instead be chosen on other grounds, such as social welfare or researcher incentives.Source: Lemma 5 · p. 24
With an estimated publication probability, median-unbiased estimation is challenging, but a confidence interval using a consistent, asymptotically normal estimator of its parameters has at least 1 − α coverage in large samples if the model is correctly specified.The stated construction applies for 0 < δ < α.Source: Lemma 5 · p. 25

5 Applications

The applications estimate publication selectivity in experimental economics, psychology, and minimum-wage and deworming research using replication studies or meta-studies, then use the estimates to correct inference. The deworming results are uncertain and specification-sensitive, while the minimum-wage application relies on a conditional-independence assumption whose failure would misspecify the model.

5 Applications

Replication-based identification assumes publication depends on original estimates, not replication estimates; meta-study identification assumes studies with different sample sizes share the same estimand.The authors say the replication assumption is plausible by design in their settings; the meta-study assumption cannot be guaranteed by design, but its plausibility is supported by estimates similar to those from replications.Source: §5 · p. 26

5.1 Economics laboratory experiments

For the economics replication data, selecting the most important significant result in each paper for replication means publication probability also reflects selection for replication.The authors interpret their corrected estimates and confidence intervals as guidance for the headline results of published studies.Source: §5.1 · pp. 27–28
In the economics replication data, significant results were more than 30 times as likely to be published as insignificant results; meta-study estimates were similar but suggested somewhat less selection.Using the replication estimate βp = .029, adjustment increased the number of confidence sets including zero from 2 to 12 of 18 studies, even without Bonferroni corrections.Source: §5.1 · pp. 29–30, Table 1 · p. 29, Figure 6 · p. 31

5.2 Psychology laboratory experiments

The psychology analysis uses 73 studies with z-statistics or close analogs; restricting to 52 studies with at least 30 denominator degrees of freedom gave quite similar results.Restricting to replications using protocols approved by original authors also produced roughly similar estimates, though estimated selection was smaller.Source: §5.2 · pp. 32, 35
In psychology, results significant at 5% were over 100 times more likely to be published than results insignificant at 10%, while meta-study estimates indicated somewhat less selection.Adjustment reduced the number of the 73 original 95% confidence sets excluding zero from 62 to 28; estimates tracked replications less well for studies with larger original z-statistics.Source: §5.2 · pp. 33–35, Table 2 · p. 33, Figure 8 · p. 34

5.3 Effect of minimum wage on employment

For minimum-wage studies, estimates insignificant at 5% were about 30% as likely to be published as significant estimates indicating negative employment effects.The point estimate suggests significant positive-effect results may be less likely to be published, but is noisy; the study cannot reject selection based only on significance rather than sign. Meta-regressions also suggested selection favoring negative effects but did not reject no selection.Source: §5.3 · pp. 36–37, Table 3 · p. 37
The estimated mean effect across latent minimum-wage studies is a small but statistically significant negative employment effect, about half the naive average that ignores selectivity.The naive average is .041 with standard error 0.011.Source: §5.3 · p. 37
The minimum-wage meta-analysis includes multiple estimates per study, complicating inference and identification; standard errors are clustered by study because estimate–standard-error pairs may be dependent within studies.Restricting to one estimate per study would be arbitrary because it is difficult to identify a single main estimate.Source: §5.3 · pp. 37–38
For the minimum-wage application, conditional independence is justified if publication decisions are made estimate by estimate or within-study estimands are independent; violations misspecify the model.Source: §5.3 · p. 38

5.4 Deworming meta-study

The deworming analysis uses 22 estimates from 20 studies, focuses on intent-to-treat child-weight effects at the longest follow-up, and clusters by study.Source: §5.4 · p. 38
With only 22 estimates, the normalized-estimate histogram’s apparent density jump at zero suggests selection toward positive estimates, but warrants caution.The histogram is the left panel of Figure 10.Source: §5.4 · p. 38, Figure 10 · p. 39
Neither meta-regression rejects no selection: the slope for X on σ is −0.296 (SE 0.917), and the intercept for Z on 1/σ is 0.481 (SE 0.889).Source: §5.4 · p. 39
Baseline deworming estimates suggest insignificant results were more likely to be included than significant ones, but the difference and the estimated positive average effect are not statistically significant.The authors caution against strong interpretation given large standard errors and limited robustness to changes in the publication-probability specification.Source: Table 4 · p. 40, §5.4 · p. 40
Alternative specifications allowing asymmetric publication probabilities suggest selection against negative deworming estimates.Source: §5.4 · p. 40

6 Conclusion

The paper identifies publication selectivity, develops corrections for known selection patterns, and applies them across several literatures. Its conclusion offers guidance for researchers and readers while noting that optimal publication rules may depend on results.

6 Conclusion

The paper contributes nonparametric identification of conditional publication probabilities, bias-corrected estimators and confidence sets, and applications documenting variation in selectivity.These are the paper’s three stated contributions.Source: §6 · p. 40

Implications for empirical research

Researchers can use the methods to assess selectivity in a literature’s meta-analysis, correct estimates and inference, and implement flexible selection models with the authors’ code.The code is available on the authors’ webpages.Source: §6 · pp. 40–41
When publication rises above the 5% significance threshold, estimates near zero or with z-statistics above 4 may be taken at face value, while estimates around z = 2 should be adjusted downwards.This advice assumes publication otherwise does not depend on findings.Source: §6 · p. 41
Researchers should not apply corrected critical values as publication standards, because doing so would trigger an arms race of selectivity and invalidate those cutoffs.Source: §6 · p. 41

Optimal publication rules

In a stylized setting where editors share policymakers’ welfare goal and policymakers have limited reading capacity, optimal publication favors surprising findings rather than result-blind selection.The authors identify a general theory of optimal publication as an important topic for future research.Source: §6 · p. 41

Supplement to the paper … B Interpretation of meta-regression coefficients

The supplement collects proofs and additional results for “Identification of and correction for publication bias,” including a simple example of how meta-regression coefficients behave. Its remaining sections describe empirical likelihoods, selection models, further inference results, and related topics.

B Interpretation of meta-regression coefficients

Section B examines the behavior of meta-regression coefficients, discussed in the main text’s Section 3.3, in a simple example.The appendix introduction identifies this as Section B’s focus.Source: Supplement to the paper · p. 47

A Proofs 48

Section A collects proofs for results stated in the main text.Source: Supplement to the paper · p. 47

Appendix Contents

Sections C and D discuss the likelihoods used in the empirical applications and a simple model for selection on both Z* and latent V*.Source: Supplement to the paper · p. 47
Sections E and F provide empirical-application details and additional results.Source: Supplement to the paper · p. 47
Sections G and H report moment-estimator results nonparametric in μ and corrected inference plots for psychology, minimum-wage, and deworming applications.Source: Supplement to the paper · p. 47
Sections I, J, and K extend inference to multivariate normal settings and discuss Bayesian inference under selective publication and optimal selection in a stylized model.Source: Supplement to the paper · p. 47

C Likelihood and parametric specifications … A Proofs

The proofs establish identification of publication-selection and effect distributions under several model setups. Theorem 3’s differential-equation argument also yields overidentifying restrictions.

A Proofs

Theorem 1 identifies p(x) up to scale by comparing marginal densities wherever the joint density has support.The support assumptions imply that for a supported (a, b), the marginal density is positive at (a, c) for every c in A.Source: A Proofs · p. 49
Deconvolution identifies fΘ|Z,σ, from which p(·) is identified; knowledge of p(·) up to scale permits recovery of fZ∗ in the argument establishing identification of µ.Source: A Proofs · pp. 49–50
Corollaries 2 and 3 use sign symmetrization to apply earlier identification results and recover the distribution of |Θ*|.Corollary 2 applies Corollary 1 to transformed data; Corollary 3 applies Theorem 3.Source: A Proofs · pp. 50, 55, 56
Theorem 2 identifies fΘ|Z by deconvolution and then obtains the desired result using the directly identified fZ and Bayes’ rule.The proof relies on the setup’s conditional-independence assumptions.Source: A Proofs · pp. 50–51
Theorem 3 identifies h, and thereby μ and p, through differential equations that impose overidentifying restrictions on finitely many free parameters.The proof eliminates derivatives to determine constants and then pins down the remaining solution parameters.Source: A Proofs · pp. 51, 54
Under a normal model for Θ*, evaluating the resulting equation at different z values identifies its mean and variance parameters.The proof notes that this normality assumption simplifies the identification argument.Source: A Proofs · p. 55
Theorem 4 establishes a unique, strictly monotone and continuous threshold ˆθα(x), with its coverage property following from conditional uniformity.Continuity of FX|Θ in x makes FX|Θ(X|θ) uniform conditional on Θ=θ.Source: A Proofs · p. 56
Lemma 5 shows that FX|Θ(x|θ) is strictly decreasing in θ and tends to zero as θ tends to infinity.The proof adapts Lee et al. (2016) for monotonicity and uses a tail bound for the limit.Source: A Proofs · pp. 56–57

B Interpretation of meta-regression coefficients … F.1 Additional results for economics laboratory experiments

The appendix explains when meta-regression coefficients do not correct for selection, details estimation and data construction, and reports additional maximum-likelihood specifications.

B Interpretation of meta-regression coefficients

Meta-regressions can test for no selectivity under the assumptions, but their slope generally cannot be interpreted as a selection-corrected estimate of the average latent effect.With strict selection on significant positive effects, the conditional expectation is nonlinear and the best linear predictor’s slope can differ from the average effect.Source: B Interpretation of meta-regression… · p. 58

C.1 Systematic replications

In systematic replications, step-function publication probabilities are estimated by likelihood, and the model identifies relative publication probabilities and the distribution of absolute latent effects.The relative probabilities compare latent studies in each z-statistic bracket with those in the highest bracket; estimation normalizes the highest-bracket probability to one.Source: §C.1 · p. 59

C.2 Meta-studies

With sign-normalized meta-study data, meta-regressions do not provide a valid test of no selectivity, because nonzero slopes or intercepts can arise even when publication is nonselective.The authors therefore do not report meta-regression results for their sign-normalized applications.Source: §C.2 · pp. 60–61

D Latent selection model

The latent selection specification allows publication to depend on both the observed estimate and the latent effect, using an unobserved independent estimate to model selection.It serves as a specification check against the baseline model, which assumes publication depends only on the observed estimate.Source: D Latent selection model · p. 61

E.1 Details for economics laboratory experiments

For economics laboratory experiments, the authors derive z-statistics from p-values and standard errors from Fisher-transformed effect sizes, recomputing p-values censored below .001.They normalize all z-statistic signs to be positive because signs are arbitrary and not comparable across studies.Source: E Details on data and variable… · p. 62, §E.1 · p. 63

E.2 Details for psychology laboratory experiments

For psychology laboratory experiments, the authors derive z-statistics from reported p-values, accounting for one-sided tests, and construct standard errors from Fisher-transformed effect sizes.The inputs come from the Open Science Collaboration’s RPPdataConverted spreadsheet.Source: §E.2 · p. 63

F Additional maximum likelihood results

The appendix reports additional maximum-likelihood specifications intended to complement the main-text results.Source: F Additional maximum likelihood results · p. 64

F.1 Additional results for economics laboratory experiments

In both replication and meta-study specifications, the QJE is estimated to publish more insignificant results, but estimates are noisy and the hypothesis βp,2 = 0 cannot be rejected.The authors therefore conclude that the same publication rule is used at both journals.Source: §F.1 · p. 64

F.2 Additional results for psychology laboratory experiments

Three psychology-data robustness checks broadly support the baseline selection estimates, while journal-specific publication rules show no significant differences. Restricting to author-endorsed replications suggests somewhat less selection, but estimates remain broadly similar.

Denominator degrees of freedom

For the 52 original studies with denominator degrees of freedom of at least 30, results were broadly similar to those from the full data.Screening used original-study degrees of freedom because replication-study degrees of freedom depend on initial-study results and could introduce additional selection.Source: §F.2 · p. 65, Table 6 · p. 66

Approved replications

Among 51 author-endorsed replications, estimates suggest somewhat less selection than baseline but remain broadly similar to other estimates.This pattern is consistent with a higher replication rate for approved replications.Source: §F.2 · p. 66, Table 7 · p. 66

Publication rule varies by journal

No significant journal differences were found, and joint tests did not reject a common publication rule across journals.The results were noisier than baseline; joint-test p-values were .78 for the replication specification and .84 for the meta-study specification.Source: §F.2 · pp. 66–67

F.3 Additional results for minimum wage meta-study … G.2 Empirical Applications

The supplementary analyses find little evidence that minimum-wage publication rules changed over time, while deworming selection estimates vary sharply across specifications and warrant caution. Moment-based methods relax assumptions about the distribution of true effects; applications often give less precise but broadly robust findings.

F.3 Additional results for minimum wage meta-study

In the published-only minimum-wage sample, estimates based on 705 estimates from 31 studies are broadly similar to those from the full sample.Standard errors are clustered at the study level.Source: §F.3 · p. 68, Table 9 · p. 68
The estimates are consistent with the baseline model’s assumption that publication rules remain constant over time; a joint test that βp,2, βp,4, and βp,6 equal zero yields a p-value of 0.7.Source: §F.3 · p. 69

F.4 Additional results for deworming meta-study

The flexible deworming specification suggests strong selectivity against negative estimates, especially significant negatives, but only one negative significant estimate makes large-sample approximations suspect.The flexible estimates differ substantially from those in the main text.Source: §F.4 · p. 70, Table 11 · p. 70
Positive effects were estimated to be ten times more likely to be published than negative effects in the restricted deworming model, but specification search makes this result uncertain.The model allows publication probability to vary by the sign of the estimate, not statistical significance; the authors caution that conventional asymptotic approximations may be unreliable.Source: §F.4 · p. 70, Table 12 · p. 71

G Moment-based estimation results

Moment-based estimators leave the distribution of true effects nonparametric by specifying only the publication-probability function.The moments are motivated by the paper’s identification arguments.Source: G Moment-based estimation results · p. 71

G.2 Empirical Applications

Across applications, moment-based methods often yield less precise conclusions, while the main findings remain robust to dropping parametric specifications for the true-effect distribution.The applications include replication and meta-study settings.Source: G Moment-based estimation results · p. 71

G.1.1 Replication Moments

Replication moments use symmetry absent selective publication and inverse-probability weighting to identify publication probabilities from observable moments.The symmetry implication holds for any distribution of true effects; for unequal variances, the authors describe adjusting the moments by adding noise to a common level.Source: §G.1.1 · pp. 71–72

G.1.2 Metastudy Moments

Meta-study moments use the implication that noisier studies’ effects are a noised-up version of less noisy studies’ effects, then weight pairs inversely by publication probability.For estimation, moments are averaged over pairs with σ_j > σ_j′, and asymptotic distributions use U-statistic results.Source: §G.1.2 · pp. 73–74

G.2.1 Economics laboratory experiments … I.2 Sufficient statistic for nuisance parameter

The applications show evidence of publication selection, but its strength and identification vary across settings. The paper also extends its inference approach to multivariate selection, illustrating how conditioning can remove nuisance parameters.

G.2.1 Economics laboratory experiments

Baseline replication moments for economics lab experiments yield a robust confidence set roughly consistent with the main-text estimate, while alternative moments cover the full parameter space; meta-study moments show strong selection for significant results.The alternative moments are less informative than the baseline moments.Source: §G.2.1 · p. 75, Table 13 · p. 75, Table 14 · p. 76

G.2.2 Psychology laboratory experiments

Psychology lab results weakly identify βp2: replication confidence sets admit only small βp,1 values, while meta-study sets allow wide values but exclude cases where both parameters are large.With only selection on 5% significance, the alternative moments still yield informative results.Source: §G.2.2 · p. 76, Figure 11 · p. 77, Table 15 · p. 77

G.2.3 Effect of minimum wage on employment

The minimum-wage results favor significant negative employment effects; apparent selection for significant positive effects is imprecise and consistent with selection on significance alone (joint-test p-value .86).The table reports meta-study selection estimates and standard errors for publication probabilities relative to studies finding a positive significant effect.Source: §G.2.3 · p. 78, Table 16 · p. 78

G.2.4 Deworming meta-study

For the deworming meta-study, the robust confidence set is unbounded above, indicating weak identification and making the point estimate likely unreliable.The point estimate also differs considerably from the baseline specification.Source: §G.2.4 · pp. 78–79, Table 17 · p. 78

H Bias corrections based on applications

The paper plots median-unbiased estimates and corrected confidence sets for applications using economics and psychology replications, minimum-wage estimates, and deworming data.The corrections are shown in Figures 12–15.Source: H Bias corrections based on applications · p. 79

I Inference when selection depends on multiple variables

The inference extension allows publication decisions to depend on a normally distributed vector of estimates and targets Γ = v′Θ while treating the remaining elements of Θ as nuisance parameters.For inference on an individual element of Θ, v can be its standard basis vector; the paper illustrates the approach with difference-in-differences selection on significance and a parallel-trends test.Source: I Inference when selection depends on… · p. 79, Assumption 1 · pp. 79, 81

I.1 Illustrative example: difference in differences

In the difference-in-differences example, parallel trends requires no state-time-specific effects beyond the policy change; selective publication creates substantial bias in the conventional estimator and coverage distortions.With two periods the assumption is untestable, but adding a third period makes it testable.Source: §I.1 · pp. 81–83, Figure 16 · p. 83, Figure 17 · p. 83

I.2 Sufficient statistic for nuisance parameter

Conditioning on W makes the conditional distribution of G depend on the parameter of interest γ but not the nuisance parameter ω, enabling inference on γ alone.The transformation from X to (G, W) is one-to-one, and the nuisance-parameter elimination also holds for the conditional distribution of published G given W.Source: §I.2 · pp. 83–84, Lemma 6 · p. 84

I.3 Optimal quantile-unbiased estimates

The section defines a conditional-quantile estimator and establishes its quantile-unbiasedness and strong optimality under regularity assumptions. In the difference-in-differences example, both the estimate and the test depend on the parallel-trends pretest result as well as the treatment-effect estimate.

I.3 Optimal quantile-unbiased estimates

The estimator is the value γ for which the observed G lies at the α quantile of its conditional distribution given W.The theorem presents this as the quantile-unbiased estimator in the setting considered.Source: §I.3 · pp. 84–85

Theorem 5

Under the stated regularity assumptions, the estimator is exactly α-quantile-unbiased and uniformly most concentrated among level-α estimators, with a natural equal-tailed confidence interval.Its conditional probability of being no greater than γ is α for every γ and ω; its expected loss is no greater than that of any other level-α quantile-unbiased estimator for losses minimized at γ and increasing away from it.Source: Theorem 5 · pp. 85–86
In the difference-in-differences example, the corrected estimate and the 5% test both depend on G and L, so the parallel-trends pretest result affects both.The estimate is compared with the conventional estimator, and the test uses an equal-tailed confidence interval for γ.Source: Theorem 5 · p. 86, Figure 18 · p. 86, Figure 19 · p. 86

J Bayesian inference · K Optimal selection for publication in a simple model

With published results, Bayesian inference depends on whether studies have unrelated or common parameters. A simple policy model can justify selective publication, but its optimal rule still creates publication bias that requires corrected inference.

J Bayesian inference

The two prior classes represent distinct latent-study parameters that are iid across studies versus a single common parameter perfectly dependent across studies; either permits an unrestricted marginal prior.Source: J Bayesian inference · p. 87
For a published observation X, the prior class matters: unrelated- and common-parameter priors yield different posteriors, although observing an unselected X* gives the same posterior under either class.Source: J Bayesian inference · p. 87
Under unrelated-parameter priors, the posterior after observing published X is the same as without selection, and the publication probability p(·) has no effect on inference.Source: J Bayesian inference · p. 88
Under common-parameter priors, the posterior for published X updates the marginal prior using the truncated likelihood, so the selection adjustment remains relevant.Source: J Bayesian inference · pp. 88–89

K Optimal selection for publication in a simple model

Selective publication may be justified when journals face limits on studies published and read, and readers use results to inform policy decisions.Source: K Optimal selection for publication in… · p. 89
When publication costs c, the journal’s optimal rule is to publish studies with expected treatment effects above c, equivalent under monotonicity to positive, one-sided results exceeding critical value xc.The threshold satisfies E[Θ*|X*=xc]=c, so this model rationalizes selective publication as one-sided testing.Source: K Optimal selection for publication in… · p. 90
Optimal critical values can vary with study precision, implementation costs, and population size; optimal selectivity nevertheless causes publication bias, so inference must account for it.Source: K Optimal selection for publication in… · p. 90