PaperlayerView exact revision on arXiv

Source-linked AI summary

ModelHub: Towards Unified Data and Lifecycle Management for Deep Learning

Hui Miao, Ang Li, Larry S. Davis, Amol Deshpande

arXiv:1611.06224v1cs.DBcs.CV

TL;DR

Deep learning lifecycle tasks such as understanding models and exploring variants remain cumbersome because current systems largely focus on building and training. ModelHub combines declarative model exploration, version management, parameter archival, and progressive evaluation. Its evaluation is reported to be useful on real models and scalable on synthetic models.

  • Problem

    Deep learning modelers face difficult model understanding and empirical variant search, while lifecycle data-management tasks remain largely outside training-focused systems.

    Source: §I · p. 1

  • Method

    ModelHub combines DLV model versioning, an SQL-inspired DQL for model exploration, and PAS archival storage with progressive parameter evaluation.

    Source: §I · p. 2, 2) Model Enumeration Queries · p. 5

  • Results

    The evaluation shows the proposed techniques are useful on real models and scale well on synthetic models.

    Source: §I · p. 2

  • Takeaways & Limitations

    ModelHub presents an integrated approach to managing model exploration, versioned artifacts, and parameter storage across the modeling lifecycle.

    Source: §I · p. 2, §VII · p. 12

Abstract

from arXiv · show

Deep learning has improved state-of-the-art results in many important fields, and has been the subject of much research in recent years, leading to the development of several systems for facilitating deep learning. Current systems, however, mainly focus on model building and training phases, while the issues of data management, model sharing, and lifecycle management are largely ignored. Deep learning modeling lifecycle generates a rich set of data artifacts, such as learned parameters and training logs, and comprises of several frequently conducted tasks, e.g., to understand the model behaviors and to try out new models. Dealing with such artifacts and tasks is cumbersome and largely left to the users. This paper describes our vision and implementation of a data and lifecycle management system for deep learning. First, we generalize model exploration and model enumeration queries from commonly conducted tasks by deep learning modelers, and propose a high-level domain specific language (DSL), inspired by SQL, to raise the abstraction level and accelerate the modeling process. To manage the data artifacts, especially the large amount of checkpointed float parameters, we design a novel model versioning system (dlv), and a read-optimized parameter archival storage system (PAS) that minimizes storage footprint and accelerates query workloads without losing accuracy. PAS archives versioned models using deltas in a multi-resolution fashion by separately storing the less significant bits, and features a novel progressive query (inference) evaluation algorithm. Third, we show that archiving versioned models using deltas poses a new dataset versioning problem and we develop efficient algorithms for solving it. We conduct extensive experiments over several real datasets from computer vision domain to show the efficiency of the proposed techniques.

I. Introduction

Deep learning makes model development an empirical, iterative search, while existing training-focused systems leave lifecycle artifacts, model comparison, versioning, and sharing largely to users. ModelHub proposes declarative exploration and version management, parameter archival, and model sharing; the background describes the DNN structures and artifacts that make these tasks substantial.

II. Background

  • DNNs learn hierarchical features from data through layered transformations, and modelers vary network architecture and optimization hyperparameters during development.The background characterizes architecture changes and hyperparameter selection as two orthogonal model-adjustment steps.Source: §I · p. 1, §II · p. 3

I. Introduction

  • Modelers repeatedly adjust architectures and hyperparameters through heuristic train-evaluate loops, producing many variants and costly checkpointed snapshots.The loop continues until prediction accuracy stops improving, while the reasons models work remain difficult to understand.Source: §I · p. 1
  • Existing systems focus on model building and training, leaving users to track model differences, automate repetitive adjustments, manage versions and parameters, and compare results.The paper also notes that sharing and reuse are difficult given model sizes, specialized tools, and user-generated scripts.Source: §I · p. 2
  • ModelHub combines DLV version management, DQL model-adjustment and tuning abstractions, and PAS archival storage with progressive parameter evaluation.Its broader design also includes model sharing, while the paper frames its contributions as a comprehensive lifecycle system and algorithms for versioned parameter storage.Source: §I · p. 2

II. Background

  • DNN lifecycles produce weights, hyperparameters, logs, periodic snapshots, and provenance artifacts; large parameter sets and long training times make storage and reproduction consequential.Weights are stored in PAS as float matrices, while intermediate training data is retained only when training must be resumed.Source: §II · p. 3

III. ModelHub System Overview

ModelHub divides functionality between local lifecycle tooling and remote model sharing. Locally, DLV manages model artifacts and versions, while DQL supports model derivation and external learning tools remain responsible for training and testing.

III. ModelHub System Overview

  • ModelHub has local functionality integrated with popular DNN systems and remote functionality for sharing models and versions; the paper primarily focuses on the local side.The architecture includes local use on a machine or cluster and remote sharing among user groups.Source: §III · p. 4
  • DLV is a specialized command-line VCS for querying lifecycle artifacts and model relationships, while DQL helps modelers derive new models.The model-learning module wraps external DNN systems to extract and reproduce modeling artifacts; the hosted service supports publishing, discovering, and reusing models.Source: §III · p. 4

A. Data Model

ModelHub represents DNNs as layer-level graphs and tracks their versions and associated artifacts in a repository. Its query facilities support model inspection and comparison, as well as enumerating new models through DQL.

A. Data Model

  • ModelHub models a DNN as a directed acyclic graph of layer-level operators, prioritizing lifecycle productivity over training and testing implementation efficiency.A model version includes a network definition, weights, extracted metadata, and associated files.Source: A. Data Model · p. 4
  • A repository version records the network, weights, training and testing metadata, and associated files, and receives a human-readable name.The implementation also records autogenerated version IDs, metadata provenance, and parent-version lineage.Source: A. Data Model · p. 4

B. Query Facilities

  • Model exploration queries help users understand models, inspect lineages, compare models, and evaluate models on different data or settings.The facilities include describing model metadata and activations, comparing aligned model components, and evaluating models for accuracy.Source: 1) Model Exploration Queries · p. 5
  • DQL supports model enumeration by selecting, slicing, constructing, and mutating networks, then varying hyperparameters and retaining models by stopping rules.The evaluate workflow separates network enumeration from hyperparameter tuning and can keep top-performing models or apply accuracy thresholds.Source: 2) Model Enumeration Queries · pp. 5–6
  • DQL combines SQL-style repository queries with model-graph selectors and one-hop traversal through node relationships.Its selector can filter model nodes, while prev and next support traversal and SPJ operations can combine relational and graph conditions.Source: 2) Model Enumeration Queries · p. 5

C. ModelHub Implementation

ModelHub maintains repository data across multiple backends and provides a service for archiving, collaboration, and model sharing.

C. ModelHub Implementation

  • The local implementation distributes repository data across multiple backends, uses Git for arbitrary file diffs, and decomposes queries across backends.Queries are chained after being sent to their respective backends.Source: C. ModelHub Implementation · p. 6
  • ModelHub hosts each standalone repository as a whole in a service for archiving, collaboration, or sharing.Modelers can publish repositories and discover or reuse remote models through search and pull.Source: C. ModelHub Implementation · p. 6
  • The authors envision that accessible, understandable lifecycle versions can facilitate a collaborative learning environment.Source: C. ModelHub Implementation · p. 6

IV. Parameter archival storage (PAS)

PAS targets compact storage of learned parameters while preserving query performance. Its design accounts for checkpoint and version relationships, group retrieval needs, parameter similarity, and differences in snapshot access and precision requirements.

IV. Parameter archival storage (PAS)

  • PAS stores learned parameters compactly while targeting query performance, and its design covers matrix storage, delta construction, version-graph planning, and approximate evaluation.Source: §IV · p. 6

A. Weight Parameters & Query Type of Interests

  • A model version contains checkpoint snapshots, and each snapshot contains intermediate data and trained parameters; PAS stores only the parameters because intermediate data is useful only for resuming training.The trained parameters are treated as collections of float matrices.Source: A. Weight Parameters & Query Type of… · p. 6
  • Many PAS operations retrieve all weight matrices in a snapshot together, even though the matrices are stored independently.The target operations include testing, fine-tuning, parameter and result comparisons, and model exploration.Source: A. Weight Parameters & Query Type of… · p. 6
  • Fine-tuned models can have similar parameters, while latest or best-accuracy snapshots are used most often and some queries tolerate low-precision weights.Checkpoint snapshots also serve limited purposes such as debugging and comparisons.Source: A. Weight Parameters & Query Type of… · pp. 6–7

B. Parameters As Segmented Float Matrices

PAS offers several float representations and segments float matrices into bytes to address compression and storage needs. The segmentation supports remote storage of low-order bytes and queries using high-order bytes when small errors are acceptable.

B. Parameters As Segmented Float Matrices

  • Float-point, fixed-point, and quantized representations let users trade storage efficiency against lossiness for different snapshot uses.PAS supports IEEE single and half precision, fixed-point encoding with dropped tail positions, and quantization with k bits where k ≤ 8.Source: B. Parameters As Segmented Float… · p. 7
  • Bytewise segmentation addresses the high entropy that makes float values difficult to compress by separating high- and low-order mantissa bytes.The first chunk contains eight high-order bits, and subsequent chunks contain one byte each; high-order bits have lower entropy and compress effectively with schemes such as zlib.Source: B. Parameters As Segmented Float… · p. 7
  • Segmentation allows low-order bytes to be offloaded remotely and queries to read only high-order bytes in exchange for tolerating small errors.The paper states that comparison and exploration queries tolerate such errors, and that evaluation queries can also be made tolerant.Source: B. Parameters As Segmented Float… · p. 7
  • Delta encoding can exploit similar parameters in checkpoint snapshots and fine-tuned versions by storing some matrices as differences rather than in full.The proposed delta operators are arithmetic subtraction and bitwise XOR.Source: B. Parameters As Segmented Float… · p. 7

C. Optimal Parameter Archival Storage

PAS models versioned parameter storage as a tradeoff between storage footprint and retrieval cost, extending prior formulations to handle groups of artifacts retrieved together. It minimizes storage subject to snapshot recreation budgets using retrieval-scheme-specific algorithms.

C. Optimal Parameter Archival Storage

  • A matrix storage graph represents parameter matrices as vertices and direct storage or delta-recovery options as edges weighted by storage and recreation costs.A storage plan is a connected subgraph from which PAS can recreate any matrix along a path from the empty matrix.Source: C. Optimal Parameter Archival Storage · pp. 7–8
  • PAS supports all three schemes during query execution, but storage optimization supports only independent and parallel schemes because reusable-scheme Steiner-tree costs are intractable.Reusable retrieval may require multiple large matrices in memory simultaneously.Source: C. Optimal Parameter Archival Storage · p. 8
  • PAS can select among directed delta options with different storage and recreation costs and make those choices at byte-segment granularity.Segments can be treated as separate matrices that must be retrieved together in some cases, but not others.Source: C. Optimal Parameter Archival Storage · p. 8
  • The objective is to minimize total storage while meeting each snapshot’s recreation-cost budget under independent, parallel, or reusable retrieval schemes.Independent retrieval sums matrix path costs, parallel retrieval uses the longest shortest path, and reusable retrieval accounts for shared paths.Source: C. Optimal Parameter Archival Storage · p. 8
  • PAS formulates storage minimization under snapshot recreation budgets, extending prior work to handle co-usage constraints on groups of simultaneously retrieved artifacts.Treating a whole snapshot as one artifact can force suboptimal delta choices, while splitting its retrieval budget across matrices can increase storage utilization.Source: C. Optimal Parameter Archival Storage · pp. 7–8
  • The optimal storage problem is NP-hard for all retrieval schemes; its optimum is a spanning tree for independent and parallel retrieval, but reusable retrieval can yield a subgraph.Reusable retrieval may share intermediate nodes across snapshot Steiner trees.Source: C. Optimal Parameter Archival Storage · pp. 8–9
  • PAS-MT refines an MST by edge swaps, while PAS-PT grows a priority-based tree; both target independent and parallel recreation schemes.PAS-PT checks affected group constraints before adding a vertex and adjusts the tree heuristically when constraints remain unsatisfied.Source: C. Optimal Parameter Archival Storage · pp. 9–10

D. Model Evaluation Scheme in PAS

PAS progressively evaluates models from segmented parameters, carrying output ranges to determine whether a prediction is certain. If those ranges cannot establish the result, it retrieves lower-order segments and retries.

D. Model Evaluation Scheme in PAS

  • Progressive evaluation applies the DNN forward while accessing lower-order parameter segments only when needed, and guarantees error-free predictions for arbitrary data points.If the result is not determined, the evaluation retrieves lower-order segments and is re-performed.Source: D. Model Evaluation Scheme in PAS · p. 10
  • The scheme tests whether the prediction remains the same across the parameter ranges available from the retrieved higher-order bytes.If every value within those ranges gives the same prediction, lower-order bytes need not be accessed.Source: D. Model Evaluation Scheme in PAS · p. 10
  • The formal problem asks whether the highest-scoring class can be determined when each learned weight is known only within a range.The prediction is the output dimension with the highest value; the stated setting assumes the prediction function returns relative class strengths.Source: D. Model Evaluation Scheme in PAS · p. 10
  • A class is certain when its minimum output exceeds every other class’s maximum output.The evaluation propagates output perturbation bounds layer by layer, accounting for uncertain weights and previous-layer outputs.Source: D. Model Evaluation Scheme in PAS · p. 10

V. Evaluation Study

The evaluation uses a Caffe-connected ModelHub prototype and examines PAS design choices, archival performance, and progressive query evaluation.

Evaluation setup

  • The prototype interfaces with Caffe through a wrapper that extracts training logs and reads and writes parameters; a custom Caffe layer supports progressive queries.The dlv command-line suite uses Git internally, with SQLite and PAS backends; PAS is implemented in C++.Source: §V · p. 11
  • Experiments use zlib compression and a Linux machine with an eight-core AMD processor, 16 GB memory, and an NVIDIA GPU.The default zlib compression level is 6; the paper specifies Ubuntu Linux 16.04 and an NVIDIA GTX 370 GPU.Source: §V · p. 11

Evaluation aims

  • The evaluation examines design decisions, PAS configurations, and the performance of archival and progressive query techniques.Source: §V · p. 11

A. Dataset Description

The study combines shared Caffe models with a generated repository of model versions to examine PAS under real-world and synthetic conditions.

Real World Dataset

  • The real-world collection includes LeNet-5, AlexNet, VGG-16, CNN-S/M/F, and the fine-tuned VGG-Salient model.LeNet-5 has 431k parameters; AlexNet and VGG-16 are tested on ILSVRC-2012, and CNN-S/M/F are retrained from scratch.Source: A. Dataset Description · p. 11

Dataset coverage

  • The real-world collection includes similar VGG-like models retrained from scratch and a fine-tuned VGG model; the synthetic task models face prediction and fine-tuning a trained VGG.In the synthetic task, prediction labels change from 1000 to 100, so the last layer changes.Source: A. Dataset Description · p. 11

Synthetic Datasets

  • The synthetic dataset is generated by an automatic modeler enumerating models and hyperparameters for a face-recognition task and VGG fine-tuning.The resulting repository contains 54 model versions, each with 10 snapshots; each snapshot has 16 parametric layers and 1.96 × 10^10 floats.Source: A. Dataset Description · p. 11

B. Evaluation Results

Across the reported tests, PAS offers substantial compression with limited accuracy loss, while archival and progressive-query results depend on the storage setting and retrieved precision.

Float Representation & Accuracy

  • A factor of 20 or so in compression is achievable without significant accuracy loss across three real-world models.The evaluation compares average compression ratio with average accuracy drop across float encoding schemes.Source: B. Evaluation Results · p. 11

Delta Encoding & Compression Ratio Gain

  • Delta encoding is not uniformly beneficial: materialized parameters often win for similar architectures, while deltas work better for fine-tuning and nearby snapshots.For fine-tuning and nearby snapshots, arithmetic subtraction consistently outperforms bitwise XOR.Source: B. Evaluation Results · pp. 11–12

Optimal Parameter Archival Storage

  • PAS-PT and PAS-MT consistently outperform LAST on the synthetic optimal archival problem.The authors attribute this to treating co-usage constraints as a group when selecting storage plans.Source: B. Evaluation Results · p. 12

Progressive Query Evaluation

  • Only a very small percentage of predictions require full-precision lower-order bytes; using fewer high-order bytes raises potential error rates.The evaluation varies the share of data retrieved at two bytes or one byte per float on LeNet, AlexNet, and VGG16.Source: B. Evaluation Results · p. 12

VI. Related Work

Prior deep-learning systems largely emphasize training, leaving data and lifecycle management challenges unaddressed. ModelHub distinguishes itself through first-class versioning and parameter archiving for lifecycle queries.

VI. Related Work

  • Training-focused deep-learning systems largely leave data and lifecycle management challenges unaddressed, while ModelHub treats versioning as a first-class construct.Source: §VI · p. 12
  • ModelHub’s archival approach is orthogonal to lossy DNN compression, and simplified models may further increase its compression gains.The cited compression methods may require heavy retraining or expensive computations, making them too heavyweight for DLV’s interactive setting.Source: §VI · p. 12
  • Unlike low-precision methods for accelerating training and testing, ModelHub uses segmented floats to archive parameters and answer lifecycle modeling queries.Source: §VI · p. 12
  • ModelHub applies basic perturbation analysis but focuses on progressive query answering over segmented-float storage, unlike stability studies centered on statistical measures.Source: §VI · p. 12

VII. Conclusion and Future Work

ModelHub aims to simplify model exploration and tracking while compactly storing models without compromising query or retrieval performance. Its DSL, version-management tool, parameter archive, and algorithms address these goals; early-user experience and experiments provide supporting evidence.

VII. Conclusion and Future Work

  • ModelHub targets easier model exploration, reduced metadata-tracking burden, and compact storage of models and snapshots without compromising query or retrieval performance.Source: §VII · p. 12
  • A command-line version-management tool and DSL address exploration and metadata tracking, with anecdotal early-user experience suggesting they simplify exploration tasks.Source: §VII · p. 12
  • The read-optimized parameter archive and storage-optimization and progressive-query algorithms support learned-weight storage; experiments on real-world and synthetic models verify design decisions and demonstrate advantages.Source: §VII · p. 12
Loading 1611.06224v1…