Scatteract: Automated extraction of data from scatter plots

1 Introduction

• The system detects tick marks, tick values, and points, then maps their pixel locations into chart coordinates to produce a table of data points.The figure distinguishes detected tick values, tick marks, and points with red, green, and blue boxes, respectively.Source: §1 · pp. 1–2, Figure 1 · p. 2
• To our knowledge, Scatteract is the first system to automatically extract scatter-plot data in chart coordinates; the closest prior approach reports only point-detection results.Available semi-automatic tools require users to define the coordinate system, click points, or provide axis and data metadata.Source: §1 · p. 2
• Earlier component-localization systems did not convert pixel coordinates to chart coordinates, while prior bar- and pie-chart extraction systems relied heavily on OCR accuracy.Chart-data recovery addresses the need to make visualized values available in numeric form for further analysis or processing.Source: §1 · pp. 1, 3
• Scatteract combines deep learning for identifying scatter-plot components with OCR and a pixel-to-chart mapping method designed to be robust to OCR errors.The authors also generate training plots with known ground truth, allowing plot aesthetics to be changed by modifying the generated distribution rather than redesigning heuristics.Source: §1 · p. 3

2 Datasets

• Scatteract’s object detector requires substantial annotated training data; the authors generated 25,000 training images and 600 test images and also collected 50 web plots.Procedural generation provides known bounding-box labels without manual annotation.Source: §2 · pp. 3–4

2.1 Procedurally generated scatter plots

• Procedural generation randomizes plot aesthetics and data, producing varied plots with ground-truth labels, but manually excludes unrealistic plots from the test set.Some generated plots have overlapping tick values or unreadable tick-value fonts; such plots remain in training.Source: §2.1 · p. 4

2.2 Scatter plots from the web

• The web test set contains 50 human-generated scatter plots downloaded from a Google image search; plots had to show tick values without units.Taking units into account was possible but postponed for simplicity.Source: §2.2 · p. 4

3 Methodology

• The pipeline detects tick marks, tick values, and points, applies OCR to tick-value boxes, pairs values with nearby ticks, assigns ticks to axes, and maps point pixels into chart coordinates.Separate object-detection models locate the three object types; point images are resized to 1440×1440 to help resolve dense points.Source: §3 · p. 5, §3.1 · p. 5

3.1 Object detection

• 87.1% average precision for points, 99.1% for tick values, and 93.0% for tick marks on the procedural test set; tick and label errors can be especially costly for coordinate conversion.Fig. 2 plots recall–precision curves for object detection.Source: §3.1 · pp. 5–6, Figure 2 · p. 6

3.2 Optical character recognition

• 91.2% tick-value OCR accuracy with preprocessing, versus 63.8% without it.Preprocessing converts images to grayscale, resizes them to 130 pixels high, and heuristically rotates labels to horizontal alignment; it fails for labels rotated exactly ±90° or upside-down.Source: §3.2 · pp. 6–7

3.3 Axis splitting

• Tick values are paired with their closest tick marks, then two one-dimensional DBSCAN clusterings use tick-coordinate projections to separate X-axis from Y-axis ticks.The axis assignment relies on ticks from one axis clustering together in the projection while those from the other axis are spread out.Source: §3.3 · pp. 7–8

3.4 RANSAC regression

• Assuming linear scales, the system fits affine pixel-to-chart mappings with RANSAC, which the authors found more robust than other regressions when OCR errors create outliers.RANSAC selects a model based on inliers and their residuals; Fig. 4 illustrates an OCR-missed minus sign that throws off other regression techniques.Source: §3.4 · pp. 8–9, Figure 4 · p. 8

4.1 Performance analysis

• 89% of procedurally generated plots meet the paper’s success criterion; average precision is 88% and average recall is 87%.The criterion is an F1 score above 80%; the F1 distribution is described as separating into high- and low-scoring plots.Source: §4.1 · p. 9, Figure 5 · p. 10
• RANSAC achieved a 5.4% absolute performance boost over Theil-Sen, the next-best method; extraction takes 3.5 seconds per plot.The comparison is among alternative versions of the system; Table 1 presents test results on those variations.Source: §4.1 · p. 10, Table 1 · p. 10
• 78% of web plots were judged successful by visual inspection, which was less systematic because ground truth was unavailable.The authors attribute the lower rate to occasional plot features absent from the training data, which make object detection more challenging.Source: §4.1 · p. 10

4.2 Error analysis

• Reported failures include ambiguous Y-axis tick placement, missed minus signs, rotated Y tick values, and shamrock markers split into three points.The examples respectively involve tick detection, OCR, rotation correction, and object detection.Source: §4.2 · p. 11

• 89% of procedurally generated plots and 78% of web plots were successfully extracted by the end-to-end system.The authors report that web-set performance is somewhat worse than performance on the procedurally generated set.Source: §5 · p. 11
• Scatteract combines deep-learning detection of points, tick marks, and tick values with OCR preprocessing and RANSAC regression to map pixels into chart coordinates.Its training data can be procedurally generated with known ground truth, avoiding manual labeling.Source: §5 · p. 11
• Generalization to new chart aesthetics depends on generating training data with those aesthetics; improving generalization is left for future work.The authors also identify deeper extraction and extension to other chart types as future directions.Source: §5 · p. 11