Source-linked AI summary
Twelve years of SAMtools and BCFtools
Petr Danecek, James K. Bonfield, Jennifer Liddle, John Marshall, Valeriu Ohan, Martin O Pollard, Andrew Whitwham, Thomas Keane, Shane A. McCarthy, Robert M. Davies, Heng Li
TL;DR
SAMtools and BCFtools are widely used to process and analyse high-throughput sequencing data. This paper describes their development and features, reporting over a million installations via Bioconda and GitHub releases.
Problem
SAMtools and BCFtools are widely used programs for processing and analysing high-throughput sequencing data.
Source: Discussion · p. 4Method
The paper describes SAMtools and BCFtools capabilities for processing alignment data and manipulating and analysing variant files.
Source: SAMtools · p. 2, BCFtools · p. 3Results
Over a million installations via Bioconda and GitHub releases and more than 900 support and feature requests resolved on GitHub are reported for the packages.
Source: Discussion · p. 4Takeaways & Limitations
The packages form a unique collection of tools useful for processing and analysing sequencing data.
Source: Discussion · p. 4Some BCFtools commands handle only haploid and diploid organisms, and support for large 64-bit genomes is currently partial.
Source: Discussion · p. 4
Abstract
from arXiv · showhide
Background SAMtools and BCFtools are widely used programs for processing and analysing high-throughput sequencing data. Findings The first version appeared online twelve years ago and has been maintained and further developed ever since, with many new features and improvements added over the years. The SAMtools and BCFtools packages represent a unique collection of tools that have been used in numerous other software projects and countless genomic pipelines. Conclusion Both SAMtools and BCFtools are freely available on GitHub under the permissive MIT licence, free for both non-commercial and commercial use. Both packages have been installed over a million times via Bioconda. The source code and documentation are available from http://www.htslib.org.
Findings
SAMtools and BCFtools have continued to develop into broad toolkits for alignment and variant processing, with BCFtools bringing both stages of variant calling into one package.
SAMtools
- SAMtools has received over 2,200 commits and 52 releases since its initial release, with version 1.11 released in September 2020.Source: SAMtools · p. 2
- SAMtools provides commands for alignment conversion, filtering, sorting, indexing, manipulation, statistics, format export, visualization, and amplicon-based sequencing.The view command can convert formats, filter files, and extract sequence ranges; markdup flags duplicate reads, while mpileup can support downstream processing such as variant calling.Source: SAMtools · p. 2
- By release 1.0, SAMtools could automatically detect and directly read and write SAM, BAM, and CRAM, after CRAM support was introduced.Earlier releases supported SAM and BAM; transitioning to HTSlib removed input-format selection and BAM-specific command restrictions.Source: SAMtools · p. 2
- SAMtools added threading from version 0.1.19 for sorting and BAM writing, and later extended it to reading and writing formats across most commands.Since version 1.10, files can also be indexed as they are written, removing a separate indexing stage and reread in pipelines.Source: SAMtools · p. 2
BCFtools
- BCFtools calling first builds position-wise read pileups and genotype likelihoods, then calls genotypes using Hardy-Weinberg equilibrium and estimated or user-supplied allele frequencies.Likelihoods account for mapping and base qualities and local misalignment; moving likelihood generation into BCFtools in release 1.4 (2016) made both steps part of one package and prevented errors from incompatible versions.Source: BCFtools · p. 3
- BCFtools had more than 2,300 commits and 29 releases since 2012, and by September 2020 comprised 21 commands and 38 plugins.It also had more than 230 documented command-line switches and options.Source: BCFtools · p. 3
- BCFtools converts between VCF and BCF and supports variant indexing, sorting, normalization, annotation, subsetting, filtering, comparison, merging, concatenation, and querying.For performance-critical pipelines, the passage recommends streaming uncompressed BCF with -Ou rather than costly text conversion and compression or decompression.Source: BCFtools · p. 3
- BCFtools includes biallelic and multiallelic SNP and short-indel callers, plus algorithms for homozygosity runs, copy-number variants, chromosome aberrations, consensus sequences, identity checks, and statistics.Its plugins include tools for VEP annotation extraction, trio de novo variant analysis, and gVCF manipulation.Source: BCFtools · pp. 3–4
Discussion
SAMtools and BCFtools are widely adopted, fast, low-memory tools whose continued development addresses both growing data scale and diverse genomic use cases. The discussion highlights their performance and adoption alongside current limitations and planned improvements.
Discussion
- Over a million installations via Bioconda and GitHub releases and more than 900 resolved GitHub requests reflect broad use and active development.The tools support processing and analysis of sequencing data.Source: Discussion · p. 4
- Two orders of magnitude faster than VEP, bcftools csq uses only a fraction of its memory for haplotype-aware functional consequence prediction.Source: Discussion · p. 4
- 4GB-per-site limits can be exceeded by highly polymorphic sites in large cohorts; threaded processing is planned, and a linear-scaling VCF annotation extension [23] is implemented in BCFtools.Annotations such as FORMAT/PL scale quadratically, and large files require parallel processing on computing clusters.Source: Discussion · p. 4
- At least 50 feature requests are registered on GitHub for investigation, while these tools have been used to analyze sequencing data from vertebrates, non-vertebrates, pathogens, plants, and viruses.Future development includes addressing limits for haploid and diploid organisms, partial support for large 64-bit genomes, VCF allele-encoding ambiguities, and result visualization.Source: Discussion · p. 4
Additional Files
The additional files list command tables for SAMtools and BCFtools.
- Table S1 lists SAMtools commands.Source: Additional Files · p. 5
- Table S2 lists BCFtools commands.Source: Additional Files · p. 5
Additional material for Twelve years of SAMtools and BCFtools
The supplementary material inventories SAMtools and BCFtools commands, documenting their availability and, for SAMtools, command history and naming conventions.
SAMtools commands
- The SAMtools command list records initial commit dates and the releases in which commands became available; coverage was contributed by Florian Breitwieser, and renamed commands retain their former names as aliases.The old names are no longer documented, but remain accepted when used.Source: Additional material for Twelve years… · p. 9, Table S1 · p. 9
- SAMtools command entries cover alignment sorting and merging, SAM/BAM/CRAM conversion, tag recalculation, BED-region depth, duplicate marking, and amplicon clipping and statistics.The entries include markdup, ampliconclip, and ampliconstats.Source: Additional material for Twelve years… · p. 9
BCFtools commands
- The BCFtools command and plugin list records initial commit dates and versions and marks contributions by David Laehnemann, Nicola Asuni, and Giulio Genovese.Plugins are prefixed with “+”; the contributor marks are *, +, and #, respectively.Source: Additional material for Twelve years… · p. 10, Table S2 · p. 10