Source-linked AI summary
Auctus: A Dataset Search Engine for Data Augmentation
Sonia Castelo, Rémi Rampin, Aécio Santos, Aline Bessa, Fernando Chirigati, Juliana Freire
TL;DR
Finding relevant datasets is difficult when data are spread across repositories and discovery relies on incomplete metadata. Auctus profiles and indexes dataset contents to support richer search and integration queries, then lets users inspect and augment results. In a bicycle-usage use case, adding compatible weather data was followed by an R^2 score of approximately 0.56.
Problem
Finding relevant data is difficult because datasets are dispersed across repositories and existing systems often search only published metadata, which may be incomplete.
Source: §1 · p. 1Method
Auctus profiles dataset contents and builds indices for keyword, spatial, temporal, and data-integration queries, with an interface for exploring and materializing augmentations.
Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 3Results
In a bicycle-usage case study, adding compatible weather data, including daily rainfall, was followed by an R^2 score of approximately 0.56.
Source: §3 · p. 4Takeaways & Limitations
The use cases illustrate Auctus supporting dataset augmentation for machine-learning models and analytics.
Source: §3 · pp. 3–4
Abstract
from arXiv · showhide
The large volumes of structured data currently available, from Web tables to open-data portals and enterprise data, open up new opportunities for progress in answering many important scientific, societal, and business questions. However, finding relevant data is difficult. While search engines have addressed this problem for Web documents, there are many new challenges involved in supporting the discovery of structured data. We demonstrate how the Auctus dataset search engine addresses some of these challenges. We describe the system architecture and how users can explore datasets through a rich set of queries. We also present case studies which show how Auctus supports data augmentation to improve machine learning models as well as to enrich analytics.
1 INTRODUCTION
Auctus addresses the difficulty of finding relevant datasets when data are distributed across repositories and search is limited to incomplete metadata. It combines richer queries over profiled data with tools for exploring and augmenting results.
The Need for Data Discovery
- Existing discovery systems often restrict users to keyword searches over published metadata, which may be incomplete or inconsistent with the data.Datasets are also distributed across many sites and repositories.Source: §1 · p. 1, Figure 1 · p. 1
Our Approach
- Auctus supports keyword, spatial, temporal, data-integration, and combined-constraint queries using information extracted from dataset contents.Its profiler detects data types and spatial and temporal columns, and generates summaries used to build query indices.Source: §1 · p. 1
- Auctus is used in production and supports research groups in the DARPA D3M program [7] [15] [20].The demonstration lets visitors query more than 19,000 datasets.Source: §1 · p. 1
- Auctus presents result snippets and a distinctive interface combining Web-search conventions with visual metaphors from analytics tools such as Tableau.Users can explore returned tables and materialize joins or unions as CSV or D3M files, or connect through a REST API.Source: §1 · p. 2
2 THE AUCTUS SYSTEM
Auctus profiles ingested datasets and indexes their summaries to support discovery and integration queries. Its interface lets users search, inspect, and augment datasets; the system is containerized and available through web and programmatic APIs.
2.1 Auctus Architecture
- Auctus profiles datasets to infer attribute types and compute type-dependent statistics, then stores summaries rather than full datasets.Fine-grained ranges generated with k-means help estimate attribute intersections for join feasibility; provenance supports retrieving datasets for augmentation.Source: §2.1 · p. 2
- Auctus indexes numerical and temporal summaries as ranges, spatial summaries as geo-shapes, and categorical attributes with Lazo set-overlap search using MinHash and LSH.Source: §2.1 · p. 2
- Join searches rank datasets by intersecting attribute summaries, while union searches use matching data types and attribute-name similarity.Results are filtered by the query and ranked; union matches may cover only a subset of the input dataset’s attributes.Source: §2.1 · p. 2
2.2 Auctus User Interface
- Users can search with keywords and filters, including date ranges, geographic areas, data sources, and inferred attribute types.For data integration, users select or upload an input dataset and can request and customize augmentation from result snippets.Source: §2.2 · pp. 2–3
- Result snippets lead to dataset details with descriptions, sources, attribute types, content summaries, samples, and column statistics.For joins, users choose matching columns and can select result columns to include in the augmented table; union results let them select matched column pairs.Source: §2.2 · p. 3
2.3 Scalability and APIs
- The containerized system can run discovery plugins in parallel and scale profiling and query containers with load; it is accessible through a Web UI, Python, and REST APIs.Auctus has indexed over 19,000 datasets.Source: §2.3 · p. 3
3 USE CASES
Auctus is demonstrated through bicycle-usage prediction and conflict forecasting use cases, with visitors able to interact with the system. The examples show dataset augmentation for both model improvement and conflict-data analysis.
3 USE CASES
- Figure 4 shows a data-integration query and augmentation interface, Data Summary views, and an upload page.Visitors can interact with Auctus through its interface and see the use cases.Source: §3 · p. 3, Figure 4 · p. 4
Bicycle Usage Prediction
- R^2 rises from close to 0.25 to approximately 0.56 after the bicycle-trip data is augmented with weather information, including daily rainfall.The initial model uses daily temperatures as its feature; the augmented data covers more months and adds weather information.Source: §3 · pp. 3–4
Conflict Forecasting
- For conflict forecasting, Auctus finds SCAD and joins its event data to an Africa grid dataset, aggregating events and fatalities at selected spatial and temporal levels.The augmented dataset can be used to explore whether destabilizing events at time t predict state-based violence at time t+1.Source: §3 · p. 4