Source-linked AI summary
Shape Outlier Detection and Visualization for Functional Data: the Outliergram
Ana Arribas-Gil, Juan Romo
TL;DR
Shape outliers can be hidden among curves even when their levels are not atypical. The paper uses modified band depth and modified epigraph index to visualize and detect shape outliers; its discussion reports better specificity for the rule in simulations.
Problem
Shape outliers may be hard to identify because curves can differ in shape while remaining near the sample rather than far away in Euclidean distance.
Source: §1 · p. 2Method
The paper represents functional observations using modified band depth and modified epigraph index and proposes an outliergram and shape-outlier rule.
Source: §6 · p. 20Results
The discussion reports that the rule exhibits better specificity in simulation results.
Source: §6 · p. 20Takeaways & Limitations
Once distances to the parabola are obtained, the rule adds no extra computational cost.
Source: §6 · p. 20The liergram is not resistant when too many trajectories are outlying.
Source: §4 · p. 16
Abstract
from arXiv · showhide
We propose a new method to visualize and detect shape outliers in samples of curves. In functional data analysis we observe curves defined over a given real interval and shape outliers are those curves that exhibit a different shape from the rest of the sample. Whereas magnitude outliers, that is, curves that exhibit atypically high or low values at some points or across the whole interval, are in general easy to identify, shape outliers are often masked among the rest of the curves and thus difficult to detect. In this article we exploit the relation between two depths for functional data to help visualizing curves in terms of shape and to develop an algorithm for shape outlier detection. We illustrate the use of the visualization tool, the outliergram, through several examples and asses the performance of the algorithm on a simulation study. We apply them to the detection of outliers in a children growth dataset in which the girls sample is contaminated with boys curves and viceversa.
1 Introduction
Functional outliers may be atypical in magnitude or shape, and shape outliers can remain hidden among curves with otherwise ordinary levels. The paper combines two functional depth measures to visualize shape variation and propose a shape-outlier detection algorithm.
1 Introduction
- Shape outliers can be difficult to see because they may have average levels yet differ in pattern, unlike curves with atypically high or low values.Atypical observations can affect statistical analysis, motivating an outlier-detection step in preliminary analysis.Source: §1 · p. 2
- The method combines modified band depth and modified epigraph index, whose distinct curve orderings reveal information about shape variation.Source: §1 · pp. 2–3
- Existing functional outlier methods can detect different kinds of outlying trajectories, but their mechanisms do not always make the outlier designation easy to interpret.Source: §1 · p. 3
- The paper proposes both a shape-outlier detection algorithm and a visualization tool for understanding shape variation and helping correct the algorithm’s output.Source: §1 · p. 3
2 Modified Band Depth and Modified Epigraph In-
Modified band depth and modified epigraph index capture different aspects of curve centrality and shape. Their relationship is bounded generally and becomes a perfect parabola for non-crossing curves, motivating a depth-plane view of shape outlyingness.
2 Modified Band Depth and Modified Epigraph In-
- MBD averages the proportion of time a curve spends inside bands formed by pairs of sample curves, while MEI is the mean proportion of time it lies below sample curves.Source: dex · p. 4
- A curve with MEI near 0.5 but low MBD lies in few bands despite being central in level, indicating a shape different from the other curves.Source: dex · pp. 4–5
- When curves do not cross, their (MEI, MBD) points define a perfect parabola; in general, the points need not follow that exact relation.For non-crossing curves the relationship is an equality, while the general relationship gives an upper bound.Source: dex · p. 7
- In their MBD-versus-MEI representation, girls 3 and 32 lie far from the other points and exhibit different growth patterns; both ranked among the sample’s highest during childhood, especially girl 3.They stopped growing earlier than the rest of the girls and ended in the lower height quartile at age 18.Source: dex · pp. 7–8
3 Shape outlier detection algorithm and outliergram
The outliergram plots modified epigraph index against modified band depth with a shifted parabola to show shape variation and an outlying boundary. The detection rule uses distances from the parabola, then checks extreme curves after shifting them toward the sample center.
3 Shape outlier detection algorithm and outliergram
- The rule uses the univariate boxplot cutoff on distances to the parabola; the paper notes it may detect more outliers than there are.Source: §3 · p. 8
- The initial rule flags curves whose vertical distance below the parabola exceeds the third quartile plus 1.5 times the interquartile range of those distances.The algorithm computes the depths, parabola values, and distances before applying this univariate boxplot rule.Source: §3 · p. 8
- The outliergram plots (MEI, MBD) points with the parabola shifted down by Qd3 + 1.5IQRd, displaying shape variation and the boundary between outlying and non-outlying curves.Source: §3 · p. 8
- The outliergram is not designed to detect magnitude outliers, though some curves lying below or above most of the sample for much of the interval appear in plot corners.The authors suggest combining the shape rule with the functional boxplot algorithm for magnitude-outlier detection.Source: §3 · p. 9
- Curves lying above or below most sample curves may have low MBD regardless of shape, so the algorithm shifts such extreme curves toward the sample center and retests them.A shifted curve is classified as a shape outlier if its new depth point falls in the previously determined outlying region.Source: §3 · p. 9
4 Simulation study
The simulation study evaluates the outliergram against other functional and multivariate outlier-detection procedures across three contaminated-curve models, examining detection performance and the adjusted version’s sensitivity and false-positive rate.
4 Simulation study
- The study compares eight existing techniques with the standard and adjusted outliergram versions, for ten methods in total.The adjusted version calibrates its boundary factor using simulated uncontaminated data, selecting the factor whose false detection rate is closest to 0.007 and applying the rule d_i ≥ F × Q1_d.Source: §4 · pp. 10, 12, 13
- Three models pair main curve-generating processes with contamination processes that differ in shape.The study specifies separate main and contamination models for each of the three cases.Source: §4 · pp. 13–14
- For each of three models, evaluations used sample sizes n = 100 and 200, contamination rates c = 0, 0.05, 0.1, 0.15, and 0.2, and 400 simulations per combination.Curves were sampled at 50 equidistant points.Source: §4 · p. 14
- For procedures requiring an outlying-region coverage probability or trimming proportion, both were set to the true contamination rate c.Procedures 6 and 7 used 200 bootstrap samples and modal depth; procedures 2 and 10 used 200 sampled datasets.Source: §4 · p. 14
- For model 1, Robust Mahalanobis Distance performs best, with pc and pf very high and low, respectively, across sample sizes and contamination rates; ISE performs very well on models 2–3 and slightly worse on model 1.Source: §4 · p. 15
- Depth-based trimming usually beats weighting (with similar performance at c = 0.05) and resists rising contamination except in model 1, while projection-based detection has low sensitivity and high false positives.Source: §4 · p. 15
- The outliergram achieves very high detection rates across all three models, especially at c = 0.05 and 0.1, but sensitivity falls as contamination increases in model 1.Source: §4 · p. 16
- The outliergram is not resistant to heavy contamination: too many outlying trajectories lower modified band depth across the sample and spread the (MEI, MBD) representation, hindering outlier detection.Source: §4 · p. 16
- The adjusted outliergram reduces false detections to near the nominal 0.007 level in uncontaminated samples, at the cost of slightly reduced sensitivity, especially in model 1.The standard outliergram has high false detection rates particularly when there are none or few outliers; these rates decrease as contamination increases.Source: §4 · p. 16
5 Applications
The applications compare the outliergram with other methods on Berkeley growth curves and Australian male mortality curves. They show differing outlier classifications across methods and, for mortality data, between raw and smoothed analyses.
5 Applications
- The applications use two real datasets to compare the outliergram with existing methods and illustrate its use and interpretation.The analyses cover girls’ and boys’ Berkeley growth curves and Australian male mortality rates, including raw and smoothed curves for the latter.Source: §5 · p. 17, §5.1 · p. 17, §5.2 · p. 19
5.1 Berkeley growth data
- In the girls sample, the tallest girl’s curve is detected by all methods, while curves 3 and 32 are also flagged by the outliergram.When shifted toward the sample center, the tallest girl’s curve is classified as a shape outlier because its slopes across growth phases exceed those of the other girls.Source: §5.1 · p. 18
- For the boys sample, five methods detect no outliers and the remaining methods show no consensus; the outliergram flags curves 9 and 28, but the adjusted outliergram does not consider them atypical observations.Source: §5.1 · p. 18
5.2 Male mortality rates in Australia 1901-2003
- For Australian male mortality curves, only 5 of 10 methods flag the 1919 curve in raw data, and 5 of 10 do so for smoothed data, with some changes.The outliergram representation places the curve apart from the others after shifting it toward the sample center, and both standard and adjusted versions detect it.Source: §5.2 · p. 19
- Across the mortality analysis, methods tend to detect more outliers in raw than smoothed data, while the adjusted outliergram gives the same output for both.The (MEI, MBD) representation is described as quite stable across the two samples.Source: §5.2 · pp. 19–20
6 Discussion
The outliergram maps functional curves by two depth indices to visualize shape variability and potential shape outliers. The discussion presents distance-based detection rules, ways to inspect isolated curves, and a combined approach for shape and magnitude outliers.
6 Discussion
- The outliergram represents curves in the MBD–MEI plane to assess shape variability and identify potential shape outliers.It uses modified band depth and modified epigraph index.Source: §6 · p. 20
- Curves closer to the parabola are more similar and smooth, while noise and crossings disperse points below it; the largest distances mark shape outliers.The authors suggest smoothing noisy datasets before applying the outliergram to enhance shape differences and similarities.Source: §6 · p. 20
- A simple boxplot-based rule classifies curves by distance from the parabola without extra computational cost once distances are obtained, but may overdetect outliers.The authors also suggest visual inspection of outliergram points to understand the sample and classification rule.Source: §6 · p. 20
- The data-dependent boundary rule has better specificity and equivalent sensitivity than the simple rule, but higher computational cost.The boundary is chosen to control the false detection rate; alternative shift rules may suit different datasets.Source: §6 · pp. 20–21
- Curves above or below the majority are difficult to assess for atypical shape without comparison curves, so the authors propose shifting them toward the sample center.The specified shift rule may be extended to address the particular nature of different datasets.Source: §6 · pp. 20–21
- Combining the outliergram with the functional boxplot can account for both shape and magnitude outliers.The authors provide R functions that combine the methods.Source: §6 · p. 21
7 Supplementary Material
The supplementary material provides R functions and scripts for using the outliergram and replicating the paper’s analyses. Although the functions can combine shape and magnitude detection, the reported simulation and real-data results consider shape outliers only.
7 Supplementary Material
- The supplementary material includes R functions for the shape-outlier algorithm and visualization, including an adjusted version and functional boxplot functionality.The functional boxplot functionality allows shape and magnitude outlier detection to be combined.Source: §7 · p. 21
- The reported simulation and two real-data analyses consider shape outliers only, despite the functions supporting combined shape and magnitude detection.Source: §7 · p. 21
- Scripts are provided for the example, simulation study, growth and mortality analyses, and computing-time evaluation.The example script replicates Figure 3, and the files are collected in a zip file.Source: §7 · p. 21