Source-linked AI summary

Twelve years of SAMtools and BCFtools

Petr Danecek, James K. Bonfield, Jennifer Liddle, John Marshall, Valeriu Ohan, Martin O Pollard, Andrew Whitwham, Thomas Keane, Shane A. McCarthy, Robert M. Davies, Heng Li

arXiv:2012.10295v2q-bio.GN

TL;DR

SAMtools and BCFtools are widely used to process and analyse high-throughput sequencing data. This paper describes their development and features, reporting over a million installations via Bioconda and GitHub releases.

  • Problem

    SAMtools and BCFtools are widely used programs for processing and analysing high-throughput sequencing data.

    Source: Discussion · p. 4
  • Method

    The paper describes SAMtools and BCFtools capabilities for processing alignment data and manipulating and analysing variant files.

    Source: SAMtools · p. 2, BCFtools · p. 3
  • Results

    Over a million installations via Bioconda and GitHub releases and more than 900 support and feature requests resolved on GitHub are reported for the packages.

    Source: Discussion · p. 4
  • Takeaways & Limitations

    The packages form a unique collection of tools useful for processing and analysing sequencing data.

    Source: Discussion · p. 4

    Some BCFtools commands handle only haploid and diploid organisms, and support for large 64-bit genomes is currently partial.

    Source: Discussion · p. 4

Abstract

from arXiv · show

Background SAMtools and BCFtools are widely used programs for processing and analysing high-throughput sequencing data. Findings The first version appeared online twelve years ago and has been maintained and further developed ever since, with many new features and improvements added over the years. The SAMtools and BCFtools packages represent a unique collection of tools that have been used in numerous other software projects and countless genomic pipelines. Conclusion Both SAMtools and BCFtools are freely available on GitHub under the permissive MIT licence, free for both non-commercial and commercial use. Both packages have been installed over a million times via Bioconda. The source code and documentation are available from http://www.htslib.org.

Findings

SAMtools and BCFtools have continued to develop into broad toolkits for alignment and variant processing, with BCFtools bringing both stages of variant calling into one package.

SAMtools

  • SAMtools has received over 2,200 commits and 52 releases since its initial release, with version 1.11 released in September 2020.Source: SAMtools · p. 2
  • SAMtools provides commands for alignment conversion, filtering, sorting, indexing, manipulation, statistics, format export, visualization, and amplicon-based sequencing.The view command can convert formats, filter files, and extract sequence ranges; markdup flags duplicate reads, while mpileup can support downstream processing such as variant calling.Source: SAMtools · p. 2
  • By release 1.0, SAMtools could automatically detect and directly read and write SAM, BAM, and CRAM, after CRAM support was introduced.Earlier releases supported SAM and BAM; transitioning to HTSlib removed input-format selection and BAM-specific command restrictions.Source: SAMtools · p. 2
  • SAMtools added threading from version 0.1.19 for sorting and BAM writing, and later extended it to reading and writing formats across most commands.Since version 1.10, files can also be indexed as they are written, removing a separate indexing stage and reread in pipelines.Source: SAMtools · p. 2

BCFtools

  • BCFtools calling first builds position-wise read pileups and genotype likelihoods, then calls genotypes using Hardy-Weinberg equilibrium and estimated or user-supplied allele frequencies.Likelihoods account for mapping and base qualities and local misalignment; moving likelihood generation into BCFtools in release 1.4 (2016) made both steps part of one package and prevented errors from incompatible versions.Source: BCFtools · p. 3
  • BCFtools had more than 2,300 commits and 29 releases since 2012, and by September 2020 comprised 21 commands and 38 plugins.It also had more than 230 documented command-line switches and options.Source: BCFtools · p. 3
  • BCFtools converts between VCF and BCF and supports variant indexing, sorting, normalization, annotation, subsetting, filtering, comparison, merging, concatenation, and querying.For performance-critical pipelines, the passage recommends streaming uncompressed BCF with -Ou rather than costly text conversion and compression or decompression.Source: BCFtools · p. 3
  • BCFtools includes biallelic and multiallelic SNP and short-indel callers, plus algorithms for homozygosity runs, copy-number variants, chromosome aberrations, consensus sequences, identity checks, and statistics.Its plugins include tools for VEP annotation extraction, trio de novo variant analysis, and gVCF manipulation.Source: BCFtools · pp. 3–4

Discussion

SAMtools and BCFtools are widely adopted, fast, low-memory tools whose continued development addresses both growing data scale and diverse genomic use cases. The discussion highlights their performance and adoption alongside current limitations and planned improvements.

Discussion

  • Over a million installations via Bioconda and GitHub releases and more than 900 resolved GitHub requests reflect broad use and active development.The tools support processing and analysis of sequencing data.Source: Discussion · p. 4
  • Two orders of magnitude faster than VEP, bcftools csq uses only a fraction of its memory for haplotype-aware functional consequence prediction.Source: Discussion · p. 4
  • 4GB-per-site limits can be exceeded by highly polymorphic sites in large cohorts; threaded processing is planned, and a linear-scaling VCF annotation extension [23] is implemented in BCFtools.Annotations such as FORMAT/PL scale quadratically, and large files require parallel processing on computing clusters.Source: Discussion · p. 4
  • At least 50 feature requests are registered on GitHub for investigation, while these tools have been used to analyze sequencing data from vertebrates, non-vertebrates, pathogens, plants, and viruses.Future development includes addressing limits for haploid and diploid organisms, partial support for large 64-bit genomes, VCF allele-encoding ambiguities, and result visualization.Source: Discussion · p. 4

Additional Files

The additional files list command tables for SAMtools and BCFtools.

Additional material for Twelve years of SAMtools and BCFtools

The supplementary material inventories SAMtools and BCFtools commands, documenting their availability and, for SAMtools, command history and naming conventions.

SAMtools commands

  • The SAMtools command list records initial commit dates and the releases in which commands became available; coverage was contributed by Florian Breitwieser, and renamed commands retain their former names as aliases.The old names are no longer documented, but remain accepted when used.Source: Additional material for Twelve years… · p. 9, Table S1 · p. 9
  • SAMtools command entries cover alignment sorting and merging, SAM/BAM/CRAM conversion, tag recalculation, BED-region depth, duplicate marking, and amplicon clipping and statistics.The entries include markdup, ampliconclip, and ampliconstats.Source: Additional material for Twelve years… · p. 9

BCFtools commands

Loading 2012.10295v2…