PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimating Psychological Networks and their Accuracy: A Tutorial Paper

Sacha Epskamp, Denny Borsboom, Eiko I. Fried

arXiv:1604.08462v4stat.APstat.ME

TL;DR

Psychological network inferences may be inaccurate or unstable under sampling variation, but few methods assess this problem. The paper develops bootstrap-based accuracy and stability checks, introduces two statistical methods, and finds substantial instability in an empirical PTSD symptom network.

  • Problem

    Few studies had assessed how sampling variation affects the accuracy of estimated psychological networks and the stability of centrality-based inferences.

  • Method

    The paper uses bootstrap confidence intervals, subset-based centrality stability analyses, difference tests, and the bootnet R package for generalized network estimation.

  • Results

    In a 17-node network of 359 women with PTSD, most edge intervals were large, node strength was moderately stable, and betweenness and closeness were not stable.

  • Takeaways & Limitations

    Centrality interpretations in psychological networks should be treated cautiously because sampling variation can make apparent differences uninterpretable.

  • Takeaways & Limitations

    The paper focuses on cross-sectional network models, while variability assessment for longitudinal and multilevel models remains beyond scope and unimplemented in bootnet.

Abstract

from arXiv · show

The usage of psychological networks that conceptualize psychological behavior as a complex interplay of psychological and other components has gained increasing popularity in various fields of psychology. While prior publications have tackled the topics of estimating and interpreting such networks, little work has been conducted to check how accurate (i.e., prone to sampling variation) networks are estimated, and how stable (i.e., interpretation remains similar with less observations) inferences from the network structure (such as centrality indices) are. In this tutorial paper, we aim to introduce the reader to this field and tackle the problem of accuracy under sampling variation. We first introduce the current state-of-the-art of network estimation. Second, we provide a rationale why researchers should investigate the accuracy of psychological networks. Third, we describe how bootstrap routines can be used to (A) assess the accuracy of estimated network connections, (B) investigate the stability of centrality indices, and (C) test whether network connections and centrality estimates for different variables differ from each other. We introduce two novel statistical methods: for (B) the correlation stability coefficient, and for (C) the bootstrapped difference test for edge-weights and centrality indices. We conducted and present simulation studies to assess the performance of both methods. Finally, we developed the free R-package bootnet that allows for estimating psychological networks in a generalized framework in addition to the proposed bootstrap methods. We showcase bootnet in a tutorial, accompanied by R syntax, in which we analyze a dataset of 359 women with posttraumatic stress disorder available online.

Introduction

Psychological network research models behavior as interacting components and combines statistical network estimation with graph-theoretic analysis. This paper motivates evaluating sampling accuracy and introduces a methodology and software for assessing uncertainty in network structures and inferences.

  • Psychological networks represent observed variables as nodes connected by edges that portray potential interactions among psychological components.
  • Network analyses typically estimate a statistical model and then analyze its weighted structure to infer properties such as the most central nodes.
  • Accuracy analysis is important because sampling variation can make estimated network structures and substantive conclusions unreliable, yet few analyses had addressed accuracy.
  • The paper proposes assessing edge-weight confidence intervals, centrality stability under fewer observations, and differences between edge-weights and centrality indices.
  • The tutorial provides interpretation guidance and aims to help researchers judge the accuracy and certainty of network results and whether conclusions are defensible.

Estimating Psychological Networks

Psychological networks are commonly estimated with pairwise Markov Random Fields, using Ising models for binary data and Gaussian graphical models for multivariate normal data. Because parameter counts grow rapidly with network size and often exceed available observations, LASSO regularization is used to estimate sparse networks.

  • Network estimation: Pairwise Markov Random Fields represent variables as nodes connected by undirected edges, and the described methodology can also apply to other network models.The paper focuses on PMRFs while noting that the methodology is more general.
  • Network estimation: Binary data require the Ising model, whereas multivariate normal data use Gaussian graphical models whose edges represent partial correlation coefficients.GGMs use an estimated covariance matrix as input, while the network itself is based on its inverse covariance matrix.
  • Small samples: LASSO regularization addresses limited sample sizes by penalizing the total absolute parameter values and encouraging sparse network estimates.The penalty limits the sum of absolute parameter values, shrinking estimates and setting some associations to zero.
  • Implementation: IsingFit implements EBIC-selected LASSO estimation for Ising models, while glasso and qgraph implement graphical LASSO estimation for regularized GGMs.qgraph combines glasso with EBIC model selection.

Network Accuracy

This section addresses the limited research on the accuracy of psychological network inferences. It outlines methods for evaluating edge-weight accuracy and centrality-index stability.

  • Psychological network inference commonly assesses edge strengths and node centrality.
  • Little research has investigated how accurate these network inferences are.
  • The section outlines methods for assessing edge-weight accuracy and centrality-index stability.

Edge-weight Accuracy

Edge-weight accuracy can be assessed with bootstrap confidence intervals that quantify sampling variability, although their precision depends on the number of bootstrap samples. Bootstrapped intervals should not be used uncritically to test edge significance, and wide intervals limit interpretation of edge strength.

  • Bootstrap confidence intervals: Bootstrap confidence intervals estimate edge-weight variability by repeatedly estimating the model and approximating the statistic’s sampling distribution.A 95% confidence interval is intended to contain the true parameter value in 95% of cases.
  • Bootstrap confidence intervals: At least 2/α bootstrap samples are required for a confidence interval at level α, while more samples are recommended for greater consistency.With NB samples, the maximum interval has α = 2/NB and equals the range of bootstrap estimates.
  • Bootstrap procedures: Non-parametric bootstrapping is broadly applicable, whereas parametric bootstrapping requires a specified data model such as the multivariate normal distribution for GGMs.The paper distinguishes continuous multivariate-normal parametric samples from model-specific sampling procedures.
  • Interpretation and limitations: Bootstrapped confidence intervals should not be used straightforwardly to test whether an edge differs from zero, especially for LASSO-regularized edges.Although observing whether zero lies in the interval behaved as a valid null-hypothesis test in unreported simulations, its utility for LASSO edges is questionable.
  • Interpretation and limitations: Wide edge-weight confidence intervals make edge strength difficult to interpret, but edge presence and sign remain interpretable under LASSO model selection.The LASSO rarely retains an edge whose sign could plausibly be either positive or negative.

Centrality Stability

Centrality stability is difficult to assess with confidence intervals, so the paper focuses on case-dropping bootstrap methods. It introduces the CS-coefficient to quantify how much data can be removed while preserving a target correlation with the original centrality indices.

  • Centrality Stability: The authors found that constructing confidence intervals for centrality indices is far from trivial.This contrasts with the more straightforward construction of bootstrapped confidence intervals for edge weights.
  • Centrality Stability: The analysis investigates stability under case-dropping, whereas node-dropping is harder to interpret because dropping 50% of nodes produces entirely different network structures.The described methods can also be applied to node-dropping, but the paper does not investigate it here.
  • Centrality Stability: The CS-coefficient is the maximum proportion of cases that can be dropped while retaining, with 95% probability, a correlation of at least 0.7 between original and subset-network centrality indices.The target correlation of 0.7 can be changed according to the stability level of interest.

Testing for Significant Differences

Bootstrapped differences between edge weights or centrality indices can test whether two network quantities differ, but results require caution because the test has limited power and multiple-testing corrections are impractical. The method uses bootstrap confidence intervals rather than approximate p-values for LASSO-regularized parameters.

  • Testing for Significant Differences: Researchers can compare bootstrap differences to test whether one edge weight or centrality index is significantly larger than another.The procedure contrasts bootstrap values for two edges or two centrality estimates.
  • Testing for Significant Differences: The edge-weight bootstrapped difference test has Type I error rates close to the nominal significance level, though it is slightly conservative at low sample sizes.Its performance was evaluated in simulations for LASSO-regularized partial-correlation networks based on polychoric correlations.
  • Testing for Significant Differences: Non-rejection does not establish that two values are equal, and the test’s slightly lower power requires cautious interpretation.A null-hypothesis test provides evidence for a difference when significant, but not evidence that the null hypothesis is true when nonsignificant.
  • Testing for Significant Differences: Approximate p-values cannot be obtained because LASSO-regularized parameter distributions are far from normal, so null-hypothesis tests rely on bootstrapped confidence intervals.This limitation is especially relevant for extreme significance levels used in multiple-testing corrections.
  • Testing for Significant Differences: With 1,000 bootstrap samples, the smallest attainable two-sided significance level is α = 0.002, making extremely stringent corrections computationally infeasible.The widest confidence interval spans the two most extreme bootstrap values, corresponding to α = 2/NB.
  • Testing for Significant Differences: Reducing Type I error through extreme corrections increases Type II error and can leave the test with very low statistical power.At α = 0.000003, significant differences would likely not be detected even if that significance level were testable.

Summary

The paper summarizes bootstrap methods for assessing network-estimation accuracy, testing differences in edge weights and centrality indices, and evaluating centrality stability. In the simulation, no centrality indices significantly differed, while all three centrality stability estimates were below stable thresholds.

  • Bootstrap methods: Non-parametric bootstrap assesses sampling variability in edge weights and tests whether edge weights and centrality indices differ significantly.It resamples rows from the data with replacement and applies the bootstrapped difference test.
  • Bootstrap methods: Case-dropping subset bootstrap assesses centrality-index stability and how well the order of centralities is recovered with fewer observations.It drops rows from the data to evaluate stability under reduced sample sizes.
  • Simulation results: None of the centrality indices significantly differed from one another in the simulation.This matched the result that the true centralities did not differ.
  • Simulation results: CS(cor = 0.7) = 0.08 for node strength, CS(cor = 0.7) = 0.05 for closeness, and CS(cor = 0.7) = 0.05 for betweenness were far below stable thresholds.The reported estimates indicate that all three centrality indices were considered unstable.

Tutorial

The tutorial demonstrates how bootnet estimates flexible psychological network models and evaluates their edge-weight accuracy and centrality stability using a PTSD dataset of 359 women. In the example network, strongest edges are distinguishable, but centrality-order interpretations require caution because only node strength shows limited stability.

  • Bootnet implementation: bootnet estimates diverse network models through configurable R chains spanning data preparation, model estimation, and network selection.Default sets include Gaussian graphical model and Ising-model procedures.
  • Network estimation: The estimated 17-symptom PTSD network contains 78 non-zero edges out of 136 possible edges, with several especially strong symptom connections.Strong connections include jumpiness–alertness, feeling distant–loss of interest, and trauma reminders–upsetting thoughts/images.
  • Centrality indices: Node 17 has the highest strength and betweenness, while Node 3 has the highest closeness, but centrality differences cannot be interpreted without stability information.The example uses centrality indices displayed as standardized z-scores.
  • Edge-weight accuracy: Bootstrapped confidence intervals are sizable, but trauma reminders–upsetting thoughts/images, jumpiness–alertness, and feeling distant–loss of interest remain the three strongest reliably distinguishable edges.Their bootstrapped confidence intervals do not overlap with those of any other edges.
  • Centrality stability: Betweenness and closeness each have CS(cor = 0.7) = 0.05, whereas node strength has CS(cor = 0.7) = 0.44 and remains below the 0.5 stability cutoff.Thus, node-strength ordering is interpretable only with care, while betweenness and closeness ordering is not.

Simulation Studies

Three simulation studies evaluated the correlation-stability coefficient and bootstrapped difference tests for edge weights and centrality indices. They used 10-node networks and varied network structure, sample size, and bootstrap procedures to assess method performance.

  • Simulation design: Three simulation studies assessed the CS-coefficient, bootstrapped edge-weight difference test, and bootstrapped centrality-index difference test.All simulations used networks of 10 nodes, converting partial-correlation matrices into four-level ordinal multivariate-normal data.
  • CS-coefficient: The CS-coefficient study simulated 10-node chain networks with 50% negative edges and edge weights of 0.25 or −0.25.Edges were randomly rewired with probabilities 0, 0.1, 0.5, or 1 to create networks with differing centrality patterns.
  • Edge-weight bootstrapped difference test: The edge-weight study used 10-node chain networks with all edge weights set to 0.3 across sample sizes of 100, 250, 500, 1,000, 2,500, and 5,000.Each condition was replicated 1,000 times, producing 6,000 total simulated datasets.
  • Centrality bootstrapped difference test: The centrality study repeated the first simulation design across 24,000 datasets and tested all node pairs using 1,000 bootstrap samples and α = 0.05.Performance was evaluated by computing the rate of rejecting the null hypothesis that centralities were equal.

Discussion

The paper recommends bootstrap-based checks of network accuracy, centrality stability, and differences between edges or centralities. In the PTSD example, most edge estimates were imprecise, and only strength centrality showed moderate stability.

  • The paper reviews psychometric network modeling and motivates assessing estimated psychological networks’ susceptibility to sampling variation.
  • The proposed workflow assesses edge-weight accuracy, centrality stability after data subsetting, and differences between edge-weights or centralities.Bootstrapping procedures can perform all three checks.
  • In a 17-node network of 359 women with subthreshold PTSD, most bootstrapped edge-weight confidence intervals were large, indicating susceptibility to sampling variation.
  • Only node strength centrality was moderately stable when participants were dropped, whereas betweenness and closeness centrality were not stable.

Limitations and Future Directions

The paper identifies unresolved questions about power analysis, unbiased centrality estimation, and confidence intervals, while noting that longitudinal and multilevel accuracy analyses remain beyond its scope. Increasing sample size improves network accuracy and centrality stability, but stable sample-size requirements remain largely unanswered.

  • Power-analysis in psychological networks: Increasing sample size improves network-estimation accuracy, centrality-difference detection, and stability of centrality ordering.The paper leaves the number of observations needed for a reasonably stable network as an important unanswered power-analysis question.
  • Power-analysis in psychological networks: Power analysis for psychological networks remains largely unanswered, although parametric bootstrap can evaluate expected edge-weight and centrality-index accuracy across sample sizes.This approach requires a reasonable prior guess of network structure, which is currently difficult because the field is still young.
  • Future directions: Future work should determine whether unbiased centrality estimators and true 95% confidence intervals can be constructed for partial-correlation networks.The example showed that sampling variation can make centrality indices highly unstable and their estimated sampling distributions severely biased.
  • Related research questions: The paper addresses accuracy only for cross-sectional network models; longitudinal and multilevel variability assessment is more complicated, beyond scope, and not implemented in bootnet.The authors refer readers to Bringmann and colleagues (2015) for a longitudinal multilevel demonstration of confidence-interval estimation.

Conclusion

The paper concludes that network accuracy remains a major blind spot in psychological network analysis, making its assessment important for establishing network analysis as a full-fledged methodology. The bootnet framework supports accuracy testing and offers additional functionality for checking accuracy and stability.

  • Conclusion: bootnet provides a framework for network estimation and the accuracy tests proposed in this paper.
  • Conclusion: bootnet also supports further accuracy and stability checks through parametric and node-dropping bootstraps and plots of centrality indices under different subsetting levels.These capabilities were beyond the scope of the paper.
  • Conclusion: Network accuracy remains a blind spot in psychological network analysis, and addressing it is essential for establishing network analysis as a full-fledged methodology.The authors identify only one prior paper examining network accuracy, which used an earlier version of bootnet.
Loading 1604.08462v4…