PaperlayerView exact revision on arXiv

Source-linked AI summary

Statistical Methods and Computing for Big Data

Chun Wang, Ming-Hui Chen, Elizabeth Schifano, Jing Wu, Jun Yan

arXiv:1502.07989v2stat.COmath.STstat.TH

TL;DR

Big data exceed standard software capacity, and computational statisticians have been under-recognized in big-data discussions. This article reviews scalable statistical methods and R software for memory and computing barriers, with an airline-delay logistic-regression case study. In that case study, RRE used 8% of the fitting time of the other two approaches.

  • Problem

    Big-data volume and velocity can exceed one computer's memory or make computation too slow, while statisticians' role in big-data discussions has been under-recognized.

    Source: §1 · pp. 1–2

  • Method

    The article reviews subsampling, divide-and-conquer, and online-updating methods, alongside R tools for big-data analysis.

    Source: §1 · p. 2, §2.3 · p. 7

  • Results

    RRE used 8% of the fitting time of the other two approaches in the 12GB airline-delay logistic-regression case study.

    Source: §6 · p. 19

  • Takeaways & Limitations

    The article describes progress in extending standard statistical analysis to big data, especially through the open-source R community.

    Source: §7 · p. 21

    Analyses not already implemented for big-data structures require an external-memory algorithm version, which may not be straightforward to implement.

    Source: §7 · p. 21

Abstract

from arXiv · show

Big data are data on a massive scale in terms of volume, intensity, and complexity that exceed the capacity of standard software tools. They present opportunities as well as challenges to statisticians. The role of computational statisticians in scientific discovery from big data analyses has been under-recognized even by peer statisticians. This article reviews recent methodological and software developments in statistics that address the big data challenges. Methodologies are grouped into three classes: subsampling-based, divide and conquer, and sequential updating for stream data. Software review focuses on the open source R and R packages, covering recent tools that help break the barriers of computer memory and computing power. Some of the tools are illustrated in a case study with a logistic regression for the chance of airline delay.

1 Introduction

The introduction frames big data as a statistical and computational challenge, with data scale exceeding standard tools and statisticians under-recognized in data-science discussions. The article reviews scalable statistical methods and R software focused on memory and computing barriers.

1 Introduction

  • The introduction highlights concerns that statisticians are under-recognized in big-data discussions and that mainstream academic statisticians risk being left behind.The cited calls urge statisticians to work with researchers in other fields on real scientific problems.Source: §1 · pp. 1–2
  • Big data exceed standard software capacity through their volume, velocity, and variety, raising computational and algorithmic challenges.High volume and velocity may bring noise accumulation, spurious correlation, and incidental homogeneity.Source: §1 · p. 2
  • The article focuses on fitting standard statistical models when high volume or velocity makes data too large for one computer or analysis too slow.The two barriers are insufficient computer memory and computing tasks that take too long.Source: §1 · p. 2
  • It reviews subsampling, divide-and-conquer, and online-updating methods, alongside R and R packages for big-data analysis.Commercial tools are also briefly covered, with RRE included in a case study and benchmark.Source: §1 · pp. 2–3

2 Statistical Methods

The article groups recent big-data statistical methodologies into resampling-based, divide-and-conquer, and online-updating approaches.

2 Statistical Methods

  • Resampling-based methods are one of the three categories used to organize recent big-data methodologies.The article uses a dataset with too many observations for standard routines such as logistic regression as its motivating setup.Source: §2 · p. 3
  • Divide-and-conquer methods are a second category in the article's organization of big-data methodologies.The motivating data consist of independent, identically distributed observations.Source: §2 · p. 3
  • Online updating is the third category in the article's organization of big-data methodologies.The motivating sample is too large for standard routines such as logistic regression.Source: §2 · p. 3

2.1 Subsampling-Based Methods

The section surveys subsampling methods that reduce big-data computation by using smaller samples, weighted approximations, or selective likelihood evaluation. It covers bootstrap, leveraging, mean-log-likelihood, and MCMC approaches, with some methods constrained by memory.

2.1.1 Bags of Little Bootstrap

  • Bags of Little Bootstrap combines bootstrap estimates from multiple smaller subsamples to obtain overall point estimates and quality measures.Each subsample of size m supports bootstrap samples of size n; the suggested m is n^γ for γ ∈ [0.5, 1], and consistency and high-order correctness are established.Source: §2.1.1 · p. 3

2.1.2 Leveraging

  • Leveraging uses a weighted subsample as a surrogate for the full data, assigning higher sampling probabilities to influential observations.It can enable visualization when the full sample cannot be visualized and can support computation without high-performance computing.Source: §2.1.2 · p. 4

2.1.3 Mean Log-likelihood

  • Mean-log-likelihood estimation approximates full-data quantities with subsample averages and updates estimates stochastically using subsamples of size m.Its estimator is established as consistent and asymptotically normal under mild conditions; simulated convergence rate was almost independent of full sample size n.Source: §2.1.3 · p. 4

2.1.4 Subsampling-Based MCMC

  • Bootstrap Metropolis–Hastings approximates likelihood ratios from bootstrap subsamples, while Firefly Monte Carlo can simulate the exact posterior by querying a subset of likelihoods.Firefly Monte Carlo's computational gain depends on a tight likelihood lower bound and inexpensive auxiliary-variable simulation.Source: §2.1.4 · p. 4
  • Firefly Monte Carlo still requires the whole dataset in memory, limiting the size of data it can handle.Pseudo-marginal Metropolis–Hastings instead uses an unbiased likelihood estimator for the acceptance step, with later work using an efficient robust difference estimator.Source: §2.1.4 · p. 5

2.2 Divide and Conquer

Divide-and-conquer methods partition data, process blocks separately, and aggregate block-level results. The section describes estimating-equation aggregation, variable-selection methods, and parallel MCMC, along with computational conditions and limitations.

2.2 Divide and Conquer

  • Divide-and-conquer algorithms partition data into blocks, process each block separately, and aggregate the block solutions into a full-data solution.Block processing may be parallelized.Source: §2.2 · p. 5

2.2.1 Aggregated Estimating Equations

  • For nonlinear estimating equations, linearization at each block solution enables weighted aggregation; under stated conditions, the estimator has the same limit as the full-data estimator.The block weights are the estimating-function slope matrices; the condition includes K = O(n^γ), with γ ∈ (0, 1).Source: §2.2.1 · p. 5

2.2.2 Majority Voting

  • Majority voting combines blockwise penalized-regression selections, yielding a model-selection-consistent estimator asymptotically equivalent to the full-data penalized estimator under regularity conditions.The setup allows p bounded or increasing slowly, with pn not faster than o(e^nk), and model size increasing at o(nk).Source: §2.2.2 · pp. 5–6

2.2.3 Screening with Ultrahigh Dimension

  • Split-and-merge variable selection partitions covariates, runs Bayesian selection on lower-dimensional subsets, then selects again on aggregated variables.The method targets ultrahigh-dimensional settings with Pn ≫ n and uses joint information among predictors rather than only marginal information.Source: §2.2.3 · p. 6

2.2.4 Parallel MCMC

  • Parallel MCMC combines independent subset chains using density products or weighted sample averages to approximate the full-data posterior.Consensus Monte Carlo may be biased for skewed or multimodal posteriors; moving beyond embarrassingly parallel MCMC remains challenging because of storage and communication overheads.Source: §2.2.4 · pp. 6–7

2.3 Online Updating for Stream Data

For streaming data, sequential updating supports inference without storing historical data. The authors extend these methods with updated estimating procedures, diagnostics, and a new estimator that performs better in finite-sample simulations.

2.3 Online Updating for Stream Data

  • Stream algorithms update as data arrive without access to historical data, unlike divide-and-conquer methods that process subsets in parallel.The approach also addresses rank deficiencies and uniqueness of combined or cumulative estimators using a generalized inverse.Source: §2.3 · p. 7
  • Divide-and-conquer variance estimates support inference about regression parameters from previously developed divide-and-conquer point estimates.These estimates apply to linear models and estimating-equation settings.Source: §2.3 · p. 7
  • Because historical data are unavailable, standard residual-based diagnostics cannot be used; the authors instead propose outlier tests based on predictive residuals.The predictive values use the cumulative coefficient estimate from the previous accumulation point.Source: §2.3 · p. 7
  • CUEE is asymptotically consistent and less biased in finite-sample simulations than Lin and Xi’s cumulatively estimated version.It updates regression coefficients and their standard errors using information from previous data.Source: §2.3 · p. 8

3 Criterion-Based Variable Selection with Online Updating

This section introduces criterion-based variable selection for online updates and evaluates AIC, BIC, and DIC in simulated streaming regression settings. The results distinguish the criteria by sample size and covariate dependence.

3 Criterion-Based Variable Selection with Online Updating

  • The authors introduce criterion-based variable selection for online updating, which they say had not previously been considered.They identify limited access to or storage of historical data as a reason the problem warrants investigation.Source: §3 · p. 8
  • The method updates AIC, BIC, and DIC using cumulative estimates as data blocks arrive.The sequential setting requires storing and updating cumulative estimates rather than the historical data.Source: §3 · pp. 8–9
  • DIC is updated under a joint conjugate prior: model coefficients are normal conditional on variance, which has an inverse-gamma prior.The prior specifies a mean vector and precision matrix for the coefficients and shape and scale parameters for the variance.Source: §3 · p. 9
  • Simulations compared AIC, BIC, and DIC across independent or AR(1)-correlated covariates, four coefficient patterns, and cumulative block counts k=2, 25, and 100.Each block contained 100 observations, and 10,000 datasets were generated per scenario.Source: §3 · pp. 9, 11
  • BIC performs extremely well with many blocks and beats AIC and DIC for independent covariates; AIC and DIC are more reliable for small, highly dependent samples.AIC and DIC perform similarly, and BIC is generally preferable for large samples, especially when covariates are not highly correlated.Source: Table 1 · p. 10, §3 · p. 11

4 Open Source R and R Packages

R tools address two big-data bottlenecks: data that exceed RAM and computations that take too long. The section surveys external-memory data structures and calculations, code acceleration, and parallel and distributed computing options.

4.1 Breaking the Memory Barrier

  • External-memory algorithms keep data on disk and process it in RAM one chunk at a time, updating or saving results until the dataset is exhausted.Iterative algorithms may need to reread the data; implementing each function requires both data management and numerical calculation.Source: §4.1 · p. 11

4.1.1 Data Management

  • bigmemory uses memory-mapped numeric matrices with shared-memory support, while ff maps file pages and supports standard and storage-efficient atomic types.Both provide R-like structures for data larger than RAM; ff also provides ffdf and CSV import/export filters.Source: §4.1.1 · p. 12

4.1.2 Numerical Calculation

  • For chunked calculations, biglm fits linear models with incremental QR decomposition and uses p^2 memory; bigglm applies IRLS across data chunks for GLMs.speedglm instead accumulates X'X and X'y; iterative methods such as IRLS require repeated passes, supported by resettable data functions.Source: §4.1.2 · p. 13

4.2.1 Speeding Up

  • R code can be sped up through byte-code compilation, compiled C/C++ interfaces such as Rcpp, and profiling tools that help locate bottlenecks.The paper reports byte-code compilation can boost speed by a factor of 2 to 5.Source: §4.2.1 · p. 14

4.2.2 Scaling Up

  • R's parallel and foreach tools support independent jobs, including shared-memory processing of out-of-memory matrices with bigmemory.For distributed memory, pbdR offers MPI-based data parallelism and distributed dense-matrix computing while retaining familiar R syntax.Source: §4.2.2 · p. 15
  • Hadoop distributes storage and MapReduce work, while Spark processes data in memory and can spill to disk when datasets do not fit.Spark can work with Hadoop storage; the paper reports it could be 100 times faster for some applications when data fit in memory.Source: §4.2.2 · p. 16
  • MPI supports distributed and shared-memory systems, whereas OpenMP targets shared memory; GPU acceleration can outperform multicore CPUs, especially on larger samples.The paper notes that parallel debugging and learning GPU tools may be challenging or prohibitive.Source: §4.2.2 · p. 17

5 Commercial Statistical Software

The section reviews commercial statistical software for big-data analysis, describing how RRE, SAS, SPSS, and MATLAB address memory and computing constraints.

Revolution R Enterprise

  • RRE uses XDF storage and external-memory processing, with functions such as rxGlm operating on chunks and exploiting multicores or clusters.The same code is described as working across workstations, servers, clusters, Hadoop, and cloud environments; its single-workstation version was free for academic use at the time.Source: §5 · p. 18

SAS

  • SAS High Performance Analytics addresses massive datasets through grid computing, in-database processing, in-memory analytics, and Hadoop connections.Its product range includes statistics, econometrics, optimization, forecasting, data mining, and text mining.Source: §5 · p. 18

Other commercial software

  • SPSS supports analytics on Hadoop-based systems, while MATLAB combines file-backed data access with multicore, GPU, parallel, cloud, and Hadoop computing.MATLAB's memory tools include memory-mapped variables, disk variables, and datastores for data that do not fit in RAM.Source: §5 · p. 18

6 A Case Study

The case study fits a logistic regression to airline flight data using bigmemory, ff, and RRE. RRE fit the model fastest under the reported settings, while all three methods produced identical regression results.

Data and setup

  • The analysis used 120,748,239 complete observations derived from approximately 12 million flights recorded over 1987–2008.The preprocessed compressed dataset was approximately 1.7 GB and 12 GB when uncompressed.Source: §6 · p. 19

Timing comparison

  • RRE used only 8% of the fitting time required by the other two methods under the reported settings.The comparison used 500,000-observation chunks on an eight-core machine; the authors attribute the advantage to implicit use of all cores and efficient storage and retrieval.Source: §6 · p. 19

Logistic regression results

  • All three methods returned identical logistic-regression results for late arrival, summarized in Table 3.Later departure hours and longer distances were associated with higher delay likelihood, while night and weekend flights were less likely to be delayed.Source: Table 3 · p. 20, §6 · p. 20

Parallel computing illustration

  • For 14 independent daily-quantile jobs, bigmemory and ff performed similarly, but communication costs made core-count speedups less ideal as cores increased.Observed time reductions were closer to the ideal expectation when fewer cores were used.Source: Table 4 · p. 20, §6 · p. 20

Timing comparison

  • bigmemory and ff had similar fitting performance, but bigmemory read the data faster and transformed variables in about one-fifth of ff's time.The XDF data version was about one-tenth the size of the external files saved by bigmemory or ff.Source: §6 · p. 20

7 Discussion

The discussion frames the article as a snapshot of progress in statistical analysis beyond a single computer's capacity. It highlights open-source R tools while emphasizing that implementation and broader statistical work remain challenging.

Progress and limitations

  • bigmemory and ff make it possible in principle to implement statistical analyses on massive data, but models not already implemented require external-memory versions that may be difficult to develop.The paper names survival analysis, generalized estimating equations, and mixed-effects models as examples of methods that may need implementation.Source: §7 · p. 21
  • Hadoop readily extends algorithms that need no repeated data passes, but the paper characterizes such analyses as mostly descriptive.The discussion mentions visualization as an important exploratory tool and points to adapting a bin-summarize-smooth framework for large data.Source: §7 · p. 21

Statisticians and big data

  • The discussion calls for statisticians to engage with real-world problems, combining domain knowledge, computing skills, and statistical thinking.Examples include structuring poorly defined problems and developing methods for data such as images or networks.Source: §7 · p. 21

Supplementary Materials

The supplementary materials provide four R scripts, their outputs, and a descriptive README. Three scripts cover logistic regression implementations, while the fourth covers parallel computing.

Supplementary Materials

Related papers

Loading 1502.07989v2…