Source-linked AI summary
Second-generation PLINK: rising to the challenge of larger and richer datasets
Christopher C. Chang, Carson C. Chow, Laurent C. A. M. Tellier, Shashaank Vattikuti, Shaun M. Purcell, James J. Lee
TL;DR
Existing PLINK tools faced growing demands for faster, more scalable analyses and data representations supporting probabilities, phase, and multiallelic variants. The paper develops PLINK 1.9 and plans PLINK 2.0, reporting substantial speed and scalability improvements while addressing richer genomic data formats.
Problem
Growing genomic datasets require faster, more scalable analyses, while PLINK 1 binary cannot represent probabilistic calls, phase, or multiallelic variants.
Method
The paper develops PLINK 1.9 through algorithmic and performance improvements and proposes PLINK 2.0 with richer data representations and extended functions.
Results
PLINK 1.9 speedups frequently exceed two and sometimes three orders of magnitude, while processing a 960-marker block in less than twice the time PLINK 1.07 needs for one marker.
Takeaways & Limitations
The second-generation PLINK versions improve performance and scaling for large genetic datasets while expanding compatibility with probabilistic, phased, and multiallelic data.
Takeaways & Limitations
PLINK is not intended for structural-variation analyses and remains less expressive than VCF for those data.
Abstract
from arXiv · showhide
PLINK 1 is a widely used open-source C/C++ toolset for genome-wide association studies (GWAS) and research in population genetics. However, the steady accumulation of data from imputation and whole-genome sequencing studies has exposed a strong need for even faster and more scalable implementations of key functions. In addition, GWAS and population-genetic data now frequently contain probabilistic calls, phase information, and/or multiallelic variants, none of which can be represented by PLINK 1's primary data format. To address these issues, we are developing a second-generation codebase for PLINK. The first major release from this codebase, PLINK 1.9, introduces extensive use of bit-level parallelism, O(sqrt(n))-time/constant-space Hardy-Weinberg equilibrium and Fisher's exact tests, and many other algorithmic improvements. In combination, these changes accelerate most operations by 1-4 orders of magnitude, and allow the program to handle datasets too large to fit in RAM. This will be followed by PLINK 2.0, which will introduce (a) a new data format capable of efficiently representing probabilities, phase, and multiallelic variants, and (b) extensions of many functions to account for the new types of information. The second-generation versions of PLINK will offer dramatic improvements in performance and compatibility. For the first time, users without access to high-end computing resources can perform several essential analyses of the feature-rich and very large genetic datasets coming into use.
Figures
The figures and tables document PLINK 1.9’s computational methods and benchmark targets, while detailed examples show bit-parallel identity-by-state computation and rapid population counting. The figures also illustrate numerical behavior in Fisher’s exact testing and recombination classification.
- Statistical and recombination figures: Figure 1 plots log-frequencies of 2x2 contingency tables to show where chi-square and Fisher’s exact tests differ for a low-MAF variant.The plotted tables have top row sum 1000, left column sum 40000, and grand total 100000.
- Statistical and recombination figures: Figure 2 plots 101 D’ log-likelihoods for a 1000 Genomes variant pair to classify historical recombination using a 90% confidence interval.An upper confidence-interval endpoint below 0.90 provides strong evidence for recombination.
- Identity-by-state and software popcount: PLINK 1.9 computes identity-by-state by transposing genotype data, XORing encoded calls, masking missing values, and counting set bits.The main IBS computation has O(mn^2) time complexity, while transposition takes O(mn).
- Identity-by-state and software popcount: Software popcount uses SSE2 or basic word operations to implement a bitslice algorithm on batches of genotype data.The SSE2 implementation applies the same idea to batches of fifteen or thirty 128-bit blocks.
- Identity-by-state and software popcount: The bitslice example reduces 192 one-bit values through two-, four-, and eight-bit partial sums before totaling them.The final addition, 14 + 11 + 14 + 11 = 50, matches the number of set bits in the original 192 values.