Source-linked AI summary
Minerva and minepy: a C engine for the MINE suite and its R, Python and MATLAB wrappers
Davide Albanese, Michele Filosi, Roberto Visintainer, Samantha Riccadonna, Giuseppe Jurman, Cesare Furlanello
TL;DR
The paper addresses the limited scalability and integration of MINE.jar for large datasets. It implements the MINE algorithms in ANSI C with R, Python, MATLAB, Octave, and C++ interfaces, and reports reduced memory use with R parallelization. The suite was consistent with MINE.jar on benchmark datasets and was tested on transcriptomics data.
Problem
MINE.jar’s applicability to large datasets is limited by memory requirements and a lack of programming interfaces and native parallelization.
Method
The authors implement the MINE algorithms in ANSI C and provide R, Python, MATLAB, Octave, and C++ interfaces.
Results
The suite matched MINE.jar closely on Spellman and microbiome benchmarks and was evaluated on microarray and RNA-seq datasets.
Takeaways & Limitations
The C engine and wrappers support lower-memory MINE analyses and integration into computational workflows.
Abstract
from arXiv · showhide
We introduce a novel implementation in ANSI C of the MINE family of algorithms for computing maximal information-based measures of dependence between two variables in large datasets, with the aim of a low memory footprint and ease of integration within bioinformatics pipelines. We provide the libraries minerva (with the R interface) and minepy for Python, MATLAB, Octave and C++. The C solution reduces the large memory requirement of the original Java implementation, has good upscaling properties, and offers a native parallelization for the R interface. Low memory requirements are demonstrated on the MINE benchmarks as well as on large (n=1340) microarray and Illumina GAII RNA-seq transcriptomics datasets. Availability and Implementation: Source code and binaries are freely available for download under GPL3 licence at http://minepy.sourceforge.net for minepy and through the CRAN repository http://cran.r-project.org for the R package minerva. All software is multiplatform (MS Windows, Linux and OSX).
Summary:
MINE applicability to large datasets is limited by the Java implementation’s memory requirements and lack of programming interfaces or native parallelization.
- MINE.jar’s scalability is constrained by high memory requirements and limited programming interfaces.The lack of native parallelization is also identified as a hurdle for systematic application in computational biology.
- Native parallelization would provide a significant benefit for broader MINE application.
1 INTRODUCTION
MINE provides dependence measures for exploring varied two-variable relationships in high-dimensional datasets, including complex biological data.
- MINE computes four dependence measures—MIC, MAS, MEV, and MCN—for pairs of variables.The suite is designed for fast exploration in many-dimensional datasets.
- The measures are intended to capture relationships of different nature while treating similar noise levels comparably across relationship types.
2 THE MINE C ENGINE AND ITS WRAPPERS
The paper delivers an ANSI C MINE engine with R, Python, MATLAB, Octave, and C++ interfaces, then evaluates consistency and performance on benchmark and transcriptomics datasets.
- The MINE C engine and its wrappers: The ANSI C engine exposes dataset, configuration, and normalized mutual-information score structures through a core scoring function.
- The MINE C engine and its wrappers: R minerva and Python minepy wrap the C implementation, while minerva additionally supports native multicore parallelization.The R interface accepts the number of cores as a parameter.
- Performance comparison: The suite was tested for consistency with MINE.jar on Spellman and microbiome datasets and for performance using RAM and CPU comparisons.Spellman testing covered 4,381 transcripts and 23 timepoints; microbiome testing examined 77 ranked association pairs.
- Performance comparison: The authors additionally evaluated minerva and minepy on human brain microarray and human lung-cancer RNA-seq datasets.
- Performance comparison: Table 1 reports elapsed process time and resident-set memory for one-versus-all analyses by minerva and minepy.
1 IMPLEMENTATION DETAILS
The implementation resolves ambiguities in the MINE pseudocode, adds entropy-related optimizations, and documents implementation choices that can affect numerical results and performance.
- Implementation choices: The core implementation was rewritten from the published pseudocode in ANSI C because the original Java source was unavailable.The authors note that ambiguities in the pseudocode required explicit implementation choices.
- Implementation choices: The implementation uses the pseudocode’s xy ≤ B bound for the characteristic matrix and MCN computation, despite the formal definition using xy < B.
- Implementation choices: MCN uses ϵ = 0 and base-2 logarithms, following settings inferred for MINE.jar.
- Implementation choices: When equipartitioning yields fewer than y clumps, normalization uses min{log x, log ŷ} to avoid underestimating normalized mutual information.
- Partitioning: GetClumpsPartition and GetSuperclumpsPartition are explicitly implemented to preserve equal x-values within clumps and enforce a clump bound.
- Partitioning: MINE.jar can return k̂ + 1 superclumps, and differing partition implementations may produce minor MIC discrepancies.
- Performance optimizations: Two precalculated entropy structures accelerate OptimizeXAxis without changing the final mutual-information matrix or MINE statistics.The optimizations precompute grid-cell counts and the entropy matrix H(⟨c_s, c_t⟩, Q).
2 COMPARISON WITH MINE.JAR
The C-based MINE wrappers reproduce MINE.jar results while substantially reducing memory use and supporting parallel computation. Performance scales with dataset size, core count, and the α parameter, while larger c values reduce observed MIC differences.
- Consistency: MIC values for all 4381 Spellman transcripts matched MINE.jar to five-significant-digit precision under α=0.67.
- Consistency: 44 of 77 microbiome association MIC values were identical, with 73 differences below 0.01 and a maximum difference of 0.014.
- Memory: MINE.jar used 5.1 GB for 1500 variables and 7.5 GB for 2000 variables, then stopped after about 2.7 ×10^6 comparisons.
- Memory: The C++ interface used less than 2 MB for 4382 variables, while minepy used less memory than MINE.jar on the 675-sample microbiome dataset.
- Parallelization: Parallel minerva reduced Spellman computation from about 5000 seconds with one core to 1500 seconds with eight cores for 4382 features.
- Scaling: A speedup of about 70 was achieved with 100 cores for 4382 features, while α=0.6 required less than 15% of the time needed for α=0.7.
3 TRANSCRIPTOMIC DATASETS
The experiments use transcriptomic datasets spanning human brain microarray and lung cancer RNA-seq data, alongside Spellman and Microbiome benchmark comparisons. Figures describe MIC agreement and memory-usage evaluations across these datasets.
- Human brain transcriptome: The human brain transcriptome dataset contains 1340 samples from 57 postmortem brains and represents 17,565 mainly protein-coding genes.It is the gene-level version of exon-array dataset GSE25219.
- Lung cancer RNA-seq: The NSCLC RNA-seq dataset includes 16 lung adenocarcinoma samples, plus one duplicate quality-control sample, with raw counts for 22,316 genes.The cohort contrasts tumors with mutant and non-mutant KRAS.
- Benchmark comparisons: Figure 4 compares minepy and MINE.jar MIC values for time versus 4381 other Spellman variables using α=0.67 and c=15.Both implementations use the same parameters.
- Benchmark comparisons: Figure 5 compares MINE.jar and minepy memory usage for six statistics on variable 1 versus all Microbiome variables, with dataset sizes reported in megabytes.The evaluated statistics include four MINE measures, Pearson correlation, and the nonlinearity index.
4 HARDWARE/SOFTWARE CONFIGURATIONS
The experiments used heterogeneous workstations, laptops, and a high-performance Linux cluster, with tables defining memory and timing measurements for multiple interfaces. A figure examines elapsed time as feature counts increase.
- Hardware: The high-performance experiments ran on a Linux cluster with more than 700 cores and 200 TB of disk space.Reserved nodes used quad-core Intel Xeon processors and 2 GB RAM.
- Measurements: Table 3 reports resident-set memory in megabytes for MINE.jar, minepy, minerva, MATLAB, Octave, and C++.Resident set size is the non-swapped physical memory used by the task.
- Measurements: Table 4 compares elapsed real time in seconds for the same interfaces on all Spellman feature pairs.Figure 6 plots elapsed time against increasing feature counts on the CDC15 Spellman yeast dataset using a logarithmic scale.
5 EXAMPLES
The examples demonstrate MINE analyses through the command-line application and Python, R, and MATLAB interfaces. They cover master-variable, pairwise, and all-versus-all workflows, while noting that some large analyses can be slow on standard workstations.
- Interface examples: The libmine-based examples use datasets distributed with minepy and minerva, including command-line, Python, R, MATLAB, and related interfaces.The examples are based on minepy 0.3.5 and minerva 1.1.
- Spellman: The Spellman example computes MINE statistics between time and every other variable with α=0.67 and c=15.The command-line and Python examples use the same parameter configuration.
- Execution considerations: Computational time may be very high on a standard workstation for the Spellman example.The warning appears alongside instructions for installing and launching minepy.
- Microbiome: The Microbiome examples support all-versus-all analysis, a master-variable comparison, and direct pairwise computation using α=0.551 and c=10.The command-line options identify a master variable or a pair of variable indices.
5.3 Baseball dataset
The examples extend MINE analysis to an all-versus-all Baseball dataset workflow and a C++ program that constructs synthetic nonlinear data. The interfaces expose the four MINE statistics for inspection.
- Baseball dataset: The Baseball example computes all-versus-all MINE statistics with α=0.7 and c=15 using the mine application.The output is written to an MLB2008 results file.
- Interfaces: The examples include minerva, Python, MATLAB, and C++ interface installation or usage instructions.The C++ example is compiled against cppmine.cpp, core.c, and mine.c.
- C++ example: The C++ example constructs 1001-point arrays for x and y and initializes MINE with α=0.6 and c=15.The code includes exception handling and falls back to those default parameters when initialization fails.
- C++ example: The C++ program prints MIC, MAS, MEV, and MCN after computing the MINE statistics.The example then deletes the MINE object and allocated arrays.
5.5 Other examples
The section points readers to online documentation for minerva and minepy, including API and application examples. The illustrated minepy workflow constructs sample x and y arrays before computing a score.
- Online documentation provides API and mine application examples for minepy, including version-specific examples for v0.3.5.
- A sample x array is constructed as evenly spaced values from 0 to 1.
- The corresponding y array is generated as sin(10 * pi * x) + x.
- The example computes a score from the constructed x and y arrays using mine->compute_score(x, y, n).
- For minerva, updated documentation and examples are available online.