PaperlayerView exact revision on arXiv

Source-linked AI summary

Accelerating Human-in-the-loop Machine Learning: Challenges and Opportunities

Doris Xin, Litian Ma, Jialin Liu, Stephen Macke, Shuchen Song, Aditya Parameswaran

arXiv:1804.05892v1cs.DB

TL;DR

ML development depends on repeated workflow experimentation, while existing systems leave opportunities to support iteration. The paper proposes tracking workflows and intermediate results to support human-in-the-loop development, and Helix reports runtime reductions on iterative workflows.

  • Problem

    ML applications require repeated trial-and-error workflow changes, and existing systems often rerun workflows from scratch rather than reusing intermediate results.

    Source: §1 · p. 1

  • Method

    The paper envisions a human-in-the-loop ML system that tracks workflow changes and intermediate results to support iterative development.

    Source: §1 · p. 1, §2.1 · p. 2

  • Results

    Helix reduces cumulative runtime by 60% in one workflow and by an order of magnitude on the classification task compared with both KeystoneML and DeepDive.

    Source: §3.2 · p. 4

  • Takeaways & Limitations

    The paper identifies research problems for efficient end-to-end ML systems focused on iterative, human-in-the-loop workflow development and presents Helix as an initial attempt.

    Source: §4 · p. 4

    Changes to external libraries may be harder to detect, though the authors expect them to be infrequent.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

Development of machine learning (ML) workflows is a tedious process of iterative experimentation: developers repeatedly make changes to workflows until the desired accuracy is attained. We describe our vision for a "human-in-the-loop" ML system that accelerates this process: by intelligently tracking changes and intermediate results over time, such a system can enable rapid iteration, quick responsive feedback, introspection and debugging, and background execution and automation. We finally describe Helix, our preliminary attempt at such a system that has already led to speedups of up to 10x on typical iterative workflows against competing systems.

1 INTRODUCTION

ML application development requires repeated trial-and-error workflow changes guided by model results. The paper envisions a human-in-the-loop system to address shortcomings in existing ML systems and presents Helix as an initial attempt.

1 INTRODUCTION

  • ML development involves many iterations of trial-and-error, with developers modifying workflows in response to model results.Changes can include modifying features, data sources, models, regularization, or evaluation metrics.Source: §1 · p. 1
  • Even small workflow changes can require minutes to hours because developers rerun workflows from scratch instead of reusing intermediate results.Source: §1 · p. 1
  • Developers struggle to explain how changes affect accuracy and to debug performance across an ML workflow.Source: §1 · p. 1
  • Traditional ML systems do not use developers’ think-time for background exploration or provide rapid approximate feedback before computation finishes.Source: §1 · p. 1
  • The envisioned human-in-the-loop system targets these deficiencies, including missing suggestions for subsequent workflow changes, while aiming to shorten time to deployable models.The system also aims to support declarative workflow specification for novice users while benefiting experts.Source: §1 · p. 1
  • Helix is presented as a preliminary declarative, general-purpose system for accelerating iterative ML application development, with preliminary results on multiple applications.Source: §1 · p. 1

2 OPTIMIZING HUMAN-IN-THE-LOOP ML

The section sets out prerequisites for human-in-the-loop ML systems and research directions for making workflow iteration more efficient. Its approach relies on declarative workflow specifications and a DAG representation, while tracking changes across iterations.

2.1 Prerequisites: Usability and Model

  • HILML systems should generalize across applications, workflow choices, and expertise levels, using declarative or semi-declarative specifications while allowing imperative code.The workflow is represented as a DAG whose nodes are operator outputs and whose edges encode input-output relationships.Source: §2.1 · p. 2
  • Workflow-item identity across iterations depends on unchanged operator code and identical parents, while source-item identity depends on unchanged underlying data.Source: §2.1 · p. 2
  • Tracking what changed across iterations helps a system understand how a workflow evolved, though external-library changes may be harder to detect.Source: §2.1 · p. 2

2.2 HILML Research Challenges

  • Intermediate reuse requires selecting what to materialize and reuse; materializing everything may cost more than its future savings, and decisions depend on predictions about future iterations and modifications.Reuse selection also requires estimating operator runtime and output size during compile time.Source: §2.2 · p. 2
  • Introspection should relate workflow changes to accuracy and runtime, compare versions logically, and help find versions pertinent to a performance gap.The paper frames finding pertinent versions as path finding in workflow-version space, with distance reflecting logical changes.Source: §2.2 · p. 2
  • Systems should test promising workflow modifications during developer think-time and optimize end-to-end across operators, requiring a framework-independent abstraction of data flow.Source: §2.2 · p. 2
  • Approximate execution can trade redundant precision for speed when developers need performance trends, while novice users may benefit from suggestions about operator changes and expected outcomes.Recommendations should account for workflow and data characteristics, user skills, system settings, and previous results, while avoiding trapping developers in a local optimum.Source: §2.2 · p. 3

3 HELIX FOR HUMAN-IN-THE-LOOP ML

Helix is a first attempt at a declarative human-in-the-loop ML system that uses workflow-DAG optimizations to address iterative development challenges. Its preliminary results show reduced cumulative runtime on information-extraction and classification tasks.

3 HELIX FOR HUMAN-IN-THE-LOOP ML

  • Helix is a declarative, general-purpose first attempt that partially addresses the first HILML research challenge using a workflow DAG for end-to-end and cross-iteration optimizations.Source: §3 · p. 3

3.1 System Architecture

  • Helix separates runtime materialization of intermediates for later reuse from compile-time loading and pruning to speed the current iteration.The system’s stated objective is to minimize end-to-end execution time across iterations.Source: §3.1 · p. 3, Figure 2 · p. 3

3.1.1 Programming Interface.

  • Helix’s declarative Scala-embedded DSL specifies whole workflows with composable operators and supports user-defined functions and varied preprocessing and learning tasks.The paper reports using it across social science, information extraction, computer vision, and natural-science workflows.Source: §3.1.1 · p. 3

3.1.2 Compilation.

  • The compile-time DAG optimizer detects changed operators, prunes dead operators, and optimally assigns compute, load, or prune states via a polynomial-time Max-Flow reduction.Source: §3.1.2 · pp. 3–4

3.1.3 Execution Engine.

  • Under storage limits, Helix makes online materialization decisions with an approximate cost model, since even simplified future-reuse selection is NP-hard.Plans run using Spark, while the paper describes Helix’s core algorithms as independent of the data-processing platform.Source: §3.1.3 · p. 4

3.2 Performance Gains

  • On information-extraction (IE), Helix reduces cumulative runtime by 60% versus DeepDive; on classification, it achieves an order-of-magnitude reduction versus both DeepDive and KeystoneML.Source: §3.2 · p. 4, Figure 4 · p. 4

4 CONCLUSIONS

The paper concludes with a vision for efficient end-to-end ML systems that support iterative, human-in-the-loop workflow development, and introduces Helix as an initial attempt to address some of its challenges.

  • The paper presents a vision for efficient end-to-end ML systems focused on iterative, human-in-the-loop workflow development and introduces Helix as an initial attempt to address some identified problems.Source: §4 · p. 4
Loading 1804.05892v1…