Source-linked AI summary
Expanding tidy data principles to facilitate missing data exploration, visualization and assessment of imputations
Nicholas J Tierney, Dianne H Cook
TL;DR
Missing-data research has emphasized inference and imputation, while workflows for handling and exploring missingness remain difficult to integrate. This paper extends tidy-data principles with a nabular structure and operations for connected missing-data workflows, illustrating the approach by identifying missingness patterns and variables for removal or imputation.
Problem
Missing-data tools and visualizations often produce incompatible outputs or limited representations, making missingness difficult to explore and integrate into analysis workflows.
Method
The paper introduces nabular data and tidy operations for handling, exploring, visualizing, imputing, tracking, and flagging missing values.
Results
The application reveals missingness clusters, removes two variables with more than 50% missingness, and selects predictors of missingness for the imputation model.
Takeaways & Limitations
Standard outputs, arguments, and behavior connect missing-data methods with existing imputation, visualization, and modelling workflows.
Takeaways & Limitations
The approach assumes wide tidy tabular data, while large datasets and special missing-value codes may require further development.
Abstract
from arXiv · showhide
Despite the large body of research on missing value distributions and imputation, there is comparatively little literature with a focus on how to make it easy to handle, explore, and impute missing values in data. This paper addresses this gap. The new methodology builds upon tidy data principles, with the goal of integrating missing value handling as a key part of data analysis workflows. We define a new data structure, and a suite of new operations. Together, these provide a connected framework for handling, exploring, and imputing missing values. These methods are available in the R package `naniar`.
1. Introduction
Tidy data and graphics principles guide analysis workflows but provide little guidance for handling, exploring, and imputing missing values. Existing tools and methods leave analysts to determine missingness structures and translate disconnected approaches into reproducible pipelines.
- Motivation: Tidy data and grammar-of-graphics principles do not address missing data, while software commonly drops missing values from plots.ggplot2 provides a warning, but the missing values are not shown.
- Motivation: Valid analysis requires understanding missingness structures before deletion or imputation, but the missing-data mechanism is typically difficult to determine.Decision-tree and latent-group methods can reveal patterns, but are not definitive.
- Motivation: Existing graphics approaches incorporate missing values into plots, but these ideas need translation into tidy data tools for integrated analysis workflows.Examples include assigning values below the minimum or treating missingness as an equal category.
- Contribution: The paper develops a connected framework for handling, exploring, and imputing missing values within data-analysis workflows.The paper presents the framework after reviewing tidy data, missingness representations, and existing software.
2. Background
Tidy tools organize data and analysis around consistent structures, but missing-data tools often violate these principles through incompatible outputs and limited visualization mappings. The paper motivates extending tidy data concepts to make missingness more interoperable and explorable.
- Tidy data: Tidy data assigns each variable a column, each observation a row, and each value a cell.These rules define an analysis-ready data format.
- Tidy tools: Tidy tools use consistent tidy input and output, allowing operations to compose into maintainable analysis sequences.Messy outputs require transformation between steps and can disrupt workflow.
- Tidy tools: The tidyverse spans data manipulation, visualization, and modelling through packages designed to share principles in design and behavior.The paper focuses on R because tidy data principles are comprehensively implemented there.
- Missing data tools: Missing-data tools commonly accept tidy input but produce difficult-to-integrate output, while their visualizations do not map data features to graphic aesthetics.These properties limit workflow integration and expressive exploration.
- Missing data tools: The proposed direction translates and expands missing-data graphics methods into tidy data and tidy tools for more effective, extensible exploration.The goal is an extensible framework rather than software alone.
- Missingness representation: Missingness is conventionally represented by a binary matrix B, with 1 for missing y_ij and 0 for observed y_ij.The matrix supports formal categories such as MCAR, MAR, and MNAR.
3. Existing Software
Existing missing-data software supports imputation and exploration, but inconsistent outputs, inaccessible visualization data, and limited workflow integration make tools difficult to combine and extend. Visualization behavior can also conceal missingness, especially for continuous variables.
- Imputation software: Existing imputation packages support diverse single- and multiple-imputation methods for continuous, count, categorical, and semi-continuous data.Examples include VIM, mice, Amelia, norm, and mi.
- Imputation software: Different imputation methods commonly use incompatible output structures and interfaces, complicating combinations across software.simputation improves integration with unified syntax and dataframe output but does not identify imputed values.
- Exploration software: Missing-data packages focus more on inference and exploring imputed values than on exploring relationships and patterns in missingness.Exploration tools also commonly produce outputs that do not work with other tools.
- Exploration software: The missingness matrix can link missing and imputed values to data for display, while MANET explicitly treats missingness as a categorical value.Related systems targeted multivariate numeric or categorical data and are not consistently maintained.
- Exploration software: MissingDataGUI enables rapid numerical and visual exploration, but GUI insights are not recorded, which can disrupt reproducible workflows.The stated trade-off is that unrecorded insights may invite mistakes.
- Exploration software: VIM offers visualizations for observed, imputed, and missing values, but its graphics do not map variables to aesthetics and their source data cannot be accessed.These limitations hinder extension and further exploration.
- Visualization behavior: ggplot2 omits missing continuous values with a warning but creates an NA category when missingness occurs in a discrete mapped variable.Thus, missingness visibility depends on the variable type and plot mapping.
4. Tidy framework for missings
The paper extends tidy data principles to missing-data work through a connected representation that makes missingness explicit, coordinated, and trackable alongside observed values.
- The framework adds data structures and intuitive tidyverse-inspired features for exploring, visualizing, and imputing missing data.The design covers data structures, operations, graphics, and summaries.
- Data structure: Four additions extend the shadow matrix: missing-value labels, special missing values, coordinated variable names, and connectedness with the original data.Special labels can encode causes such as instrument error or indicate imputations.
- Data structure: A shadow matrix records missingness with explicit NA and !NA labels, avoiding ambiguity associated with binary 0 and 1 indicators.The labels make the matrix meaning transparent for analysis and plotting.
- Nabular data: Nabular data binds original and shadow columns so corresponding observations remain together during visualization, summaries, and imputation tracking.The coordinated _NA variables let analysts inspect a variable alongside its missingness state.
- Nabular data: Nabular data preserves special missingness and identifies imputations more effectively than a binary logical matrix.A logical matrix cannot easily encode special missingness or capture which values were imputed after imputation.
4.3. Missing data operations
The missing-data workflow is organized as tidyverse-style verbs that scan, replace, summarize, augment, impute, track, and flag missingness within the data pipeline.
- Naniar expresses common missing-data operations as tidyverse-style verbs for scanning, replacing, adding summaries, shadowing, imputing, tracking, and flagging.These operations connect missingness exploration and handling through a common workflow.
- scan and replace: The scan operation searches variables for conventional missing codes such as N/A, MISSING, and -99, returning their occurrence counts.Common numeric and character missing-value lists can support these checks.
- scan and replace: Values identified as missing can be replaced with NA using replace_with_na(), including scoped variants for selected groups of variables.The scoped forms are _all, _if, and _at.
- add: Add operations append missingness summaries such as counts, proportions, clusters, or any-missing indicators to the data.These summaries can support analyses of variables associated with missingness structures.
- shadow and flag: Nabular data combines original columns with shadow columns to support visualization, summaries, and imputation tracking.The example shows original variables paired with x_NA, y_NA, and z_NA shadow variables.
- shadow and flag: Special missingness can be encoded by recoding flagged values into shadow labels such as NA_broken_sensor.The example recodes x == -99 with the suffix broken_sensor.
impute: Imputing values
Naniar supports exploratory imputation and evaluation by combining existing imputation methods with nabular tracking, visual comparison, and summaries of imputed versus observed values.
- Imputing values: Naniar supplies exploratory imputation helpers including impute_below, impute_mean, and impute_median, while leaving substantive imputation to existing methods.The exploratory helpers are intended to reveal missingness structure and impute_below uses jitter to reduce overplotting.
- Tracking imputations: Without nabular data, impute functions do not track imputed-value locations; the nabular workflow resolves this challenge.The tracking limitation applies to the direct impute_ approach described before nabular data is used.
- Tracking imputations: Imputation tracking starts with nabular data, applies imputation, and uses shadow variables such as _NA to identify formerly missing values.label_shadow adds labels that distinguish missing-derived values from observed values.
- Visual evaluation: Figure 4 colors imputed values green and observed data orange in scatterplot and density views of ozone and solar radiation.The imputed values are similar to observed values but slightly trended toward the mean.
- Visual evaluation: Imputed and complete-case values can be compared by grouping on any_missing and summarizing statistics such as minimum, mean, median, and maximum.Table 2 reports similar means and medians but very different minimums and maximums for imputed versus non-imputed values.
5. Graphics
The graphics workflow systematically explores missingness from broad overviews to univariate and multivariate plots, using nabular data to retain and display missingness.
- Overviews: Overview plots show the amount and distribution of missingness across variables and cases, ordering observations by missingness.For airquality, two variables contain missings, about one third of observations have one missing value, and very few have two.
- Overviews: Heatmaps arrange the shadow matrix so variables form columns and observations form rows, with clustering revealing missingness structure.The ggplot2 foundation makes these visualizations customizable.
- Overviews: Upset plots display overlapping missingness sets, combining intersection counts with the sizes of combinations and missingness in each variable.They scale better than Venn diagrams when more variables are involved.
- Univariate: Nabular data supports univariate views by imputing missing values below the observed range or splitting distributions according to another variable’s missingness.These approaches show both the magnitude of missingness and differences in distributions associated with missingness.
- Multivariate: Missing values can be displayed in plot margins for two-dimensional graphics and in parallel coordinates for higher-dimensional data.In oceanbuoys, humidity was missing at low air and sea temperatures and for one year and one location.
6. Numerical summaries
The numerical-summary workflow provides consistent, composable measures and tables for describing missingness in variables and cases, including grouped and run-based summaries.
- Single-number summaries: naniar supplies consistently named functions for overall, variable-level, and case-level counts, proportions, and percentages of missingness.The complements of these summaries report completeness.
- Tables: Variable and case summaries can guide data handling decisions such as dropping highly incomplete variables or selecting predictors for imputation.Frequency tables additionally count how often variables or cases contain zero, one, two, or more missing values.
- Tables: miss_case_summary and miss_case_table implement tabulations of missingness across cases.These outputs support consistent numerical summaries within the naniar workflow.
- Patterns: miss_var_span and miss_var_run identify repeated spans, runs, or streaks of missing values in variables.Rows can also be ordered by the number of missings to place the most incomplete cases first.
- Grouped summaries: naniar summaries work with dplyr group_by and the pipe operator, enabling grouped exploration such as missingness by month.In airquality, ozone has more missings in June than May.
7. Application
The Melbourne housing case study uses the workflow to identify missingness structure, select variables for imputation, compare imputations, and assess their effects on price modeling.
- Case study: The 27,247-property, 21-variable housing dataset contains substantial missingness, motivating exploratory analysis before predicting Melbourne housing prices.The study covers properties recorded from January 28, 2016 to March 17, 2018.
- Exploring patterns of missingness: Variables and cases exhibit multiple missingness groups: building area and year built exceed 50% missingness, while most cases have more than 5% missing values.These patterns led to removing building area and year built from subsequent analysis.
- Exploring patterns of missingness: Upset plots, clustering, and tabulations reveal recurring intersections and three variable-level groups plus six case-level missingness patterns.Two prominent upset-plot patterns involve two and five variables missing together.
- Exploring missingness patterns for imputation: CART modeling identifies rooms, price, suburb, council area, distance, and region name as important predictors of missingness clusters for the imputation model.The workflow first clusters missing values, then predicts the clusters using remaining dataset variables.
- Imputation and assessment: Linear-regression and K-nearest-neighbor imputations were compared for log house-price modeling, with linear-model imputations closer to the mean than KNN values.Complete cases underestimated the impact of rooms on log price, while partial residual plots showed little variation among models.
- Summary: naniar and visdat implement the paper’s methods as tidy tools balancing automation and control for efficient, readable, and fluent missing-value analysis.Each tool has a clear intent, such as summarizing, plotting, generating, or augmenting data.
8. Discussion
The paper presents tidy-data-based methods for exploring, visualizing, and imputing missing data, with standardized interfaces and outputs. The approach supports flexible extensions and integration with existing missingness analyses, imputation evaluation, visualization, and modelling.
- The methods provide standard outputs, function arguments, and behavior for exploring, visualizing, and imputing missing data.
- Consistent workflows integrate missing-value handling with existing imputation methodology, visualization, and modelling.
- Nabular structures are deliberately simple, allowing analysts to choose visualizations and potentially support interactive, linked, or animated graphics.
- Spatial, time-series, network, and longitudinal data require conversion to wide tidy format, while large data may need special handling.
- The methodology can be combined with multivariate missingness methods and testing frameworks for evaluating imputed values.
- The approach supports movement from raw data to model data in a fluid workflow.
Journal of Statistical Software http://www.jstatsoft.org/
The supplied passages contain publication and access metadata rather than substantive discussion.
- The supplied metadata includes the journal website http://www.foastat.org/.
- The issue metadata lists placeholder volume, issue, date, submission, acceptance, and DOI fields.