PaperlayerView exact revision on arXiv

Source-linked AI summary

NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System

Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, Michael D. Ernst

arXiv:1802.08979v2cs.CLcs.SE

TL;DR

NL2Bash addresses the challenge of mapping English descriptions to Bash commands, where open-vocabulary constants and Bash-specific argument formatting complicate semantic parsing. The paper constructs a large expert-described corpus and evaluates Seq2Seq, CopyNet, and Tellina across token granularities. CopyNet performs best at sub-token granularity, while the results show substantial room for future work, especially for rare utilities and flags.

  • Problem

    NL2Bash must generate open-vocabulary Bash constants and correctly reformat command arguments despite unseen tokens and idiomatic syntax rules.

  • Method

    The paper constructs a corpus of web-scraped Bash commands with expert descriptions and evaluates Seq2Seq, CopyNet, and Tellina across token granularities.

  • Results

    CopyNet at sub-token granularity significantly outperforms Tellina, while ST-CopyNet achieves the highest full command accuracy and competitive command template accuracy.

  • Takeaways & Limitations

    NL2Bash establishes a large benchmark for practical English-to-Bash semantic parsing and demonstrates significant room for future work.

  • Takeaways & Limitations

    Rare utilities and flags are the leading error source because gathering sufficient training data for all of them is expensive.

Abstract

from arXiv · show

We present new data and semantic parsing methods for the problem of mapping English sentences to Bash commands (NL2Bash). Our long-term goal is to enable any user to perform operations such as file manipulation, search, and application-specific scripting by simply stating their goals in English. We take a first step in this domain, by providing a new dataset of challenging but commonly used Bash commands and expert-written English descriptions, along with baseline methods to establish performance levels on this task.

1. Introduction

NL2Bash frames natural-language control of operating systems as a semantic-parsing problem, aiming to let users express computer tasks in English. The paper introduces a corpus of Bash commands with expert descriptions and evaluates baseline parsing models.

  • Natural-language programming offers broad accessibility for repetitive tasks such as file manipulation, search, and application-specific scripting.
  • The paper studies mapping English sentences to Bash commands as a first step toward natural-language operating-system control.
  • NL2Bash provides over 9,000 English-command pairs covering more than 100 Bash utilities, collected from practical command sources and expert descriptions.
  • The evaluation compares Seq2seq, CopyNet, and Tellina as baseline semantic-parsing approaches for NL2Bash.
  • At sub-token granularity, CopyNet significantly outperforms Tellina with less pre-processing and post-processing.

2. Domain: Linux Shell Commands

The domain represents Bash commands through utilities, option flags, and arguments, while restricting the corpus to selected command structures. The dataset excludes syntax requiring contextual interpretation, including redirection, variable assignment, and compound statements.

  • A shell command has three basic components: a utility, option flags, and arguments.Examples include find or grep utilities, -name or -i flags, and file or text arguments.
  • The dataset targets 135 useful Bash utilities from a broader ecosystem of over 250 utilities.The paper notes that less common utilities had fewer available examples and required greater annotator expertise.
  • Table 1 presents example natural-language descriptions paired with their corresponding shell commands.
  • The corpus includes single commands, logical connectives, and nested commands such as pipelines and command substitutions.
  • Table 2 summarizes which Bash command syntax structures are in scope and out of scope for the dataset.
  • I/O redirection, variable assignment, and compound statements are excluded because they require contextual interpretation.

3. Corpus Construction

The NL2Bash corpus pairs Bash commands with expert-written English descriptions, then filters, cleans, and splits them to support evaluation of natural-language-to-command generalization. The resulting dataset covers a broad but deliberately bounded command domain with substantial lexical diversity, many-to-many mappings, and a long-tailed utility distribution.

  • Corpus collection and cleaning: The corpus contains 12,609 collected text–command pairs, of which 9,305 remained after filtering.The released corpus includes both filtered and full data, along with cleaning scripts.
  • Corpus statistics: Descriptions and commands are short on average, at 11.7 words and 7.7 tokens respectively, while both median frequency measures are 1.The low medians reflect many open-vocabulary constants appearing only once.
  • Corpus statistics: Natural-language descriptions and Bash commands form a many-to-many mapping because equivalent commands and alternative descriptions coexist.This structure affects both evaluation and modeling choices.
  • Corpus statistics: The utility distribution is long-tailed: find appears 6,268 times, xargs 1,047 times, and the 52 least common utilities 984 times collectively.The disproportionate frequency of find partly reflects an initial collection stage focused on commands containing that utility.
  • Data split: Data are split by normalized-description clusters at a 10:1:1 train/dev/test ratio, with overlapping commands moved into training to prevent trivial memorization.This split is intended to evaluate generalization beyond descriptions or commands seen during training.

4. Evaluation Methodology

Evaluation uses manual majority judgments of ranked translations, distinguishing exact full commands from command templates. The paper also notes that Bash-command equivalence is difficult to verify automatically and leaves execution-based evaluation for future work.

  • Manual Evaluation: Three shell-scripting freelancers independently judged each system’s top-3 translations, with majority voting producing the final evaluation.Test pairs sharing normalized natural-language descriptions were grouped as one test instance.
  • Accuracy Metrics: Top-k full-command accuracy measures whether a correct full command appears at rank k or above in the model output.This metric evaluates correctness including command arguments.
  • Accuracy Metrics: Top-k command-template accuracy measures whether a correct template appears at rank k or above, ignoring incorrect arguments.Template judgments are distinguished from full-command judgments in the inter-annotator agreement analysis.
  • Evaluation Challenges: Bash-command equivalence is undecidable in general, so the paper leaves virtual-environment execution comparison for future work.Prior work also used formal verification for regular expressions and fuzzy metrics such as BLEU, but BLEU does not effectively measure formal-language semantic similarity.

5. System Design Challenges

The Bash domain creates challenges through broad application coverage, open-vocabulary and reformatted constants, flexible command composition, and idiomatic syntax. These properties complicate both semantic parsing and evaluation.

  • Rich Domain: Bash spans file management, text processing, network control, and advanced operating-system functionality, making its semantic-parsing domain unusually broad.The paper contrasts this breadth with prior work focused on a single application domain.
  • Out-of-Vocabulary Constants: Open-vocabulary constants such as paths, file properties, and time expressions are unseen during training but must be generated and idiomatically reformatted.These constants are especially problematic because they often correspond directly to command arguments.
  • Language Flexibility: Large flag inventories and compositional commands create multiple correct solutions for one task, complicating training and evaluation.The same language description may therefore map to more than one valid Bash command.
  • Idiomatic Syntax: Bash’s shallow grammar, pattern-matched options, and command-specific syntax make syntax-tree-based parsing approaches difficult to apply.Examples include pipeline and code-block structures and idiomatic remote-path syntax.

6. Baseline System Performance

The paper establishes neural and stage-wise baselines for NL2Bash and compares token granularities, copying, and end-to-end versus heuristic approaches. Results show strong full-command performance from sub-token copying, while structure accuracy and data sparsity remain important challenges.

  • Baseline Systems: Seq2Seq, CopyNet, and Tellina provide baseline systems spanning neural translation, copying, and stage-wise heuristic processing.Seq2Seq and CopyNet translate sequences end to end, while Tellina abstracts constants, translates templates, and fills arguments with alignment and reformatting heuristics.
  • Token Granularity: Token-level models generally achieve higher command structure accuracy, whereas character- and sub-token models achieve higher full command accuracy.Character and sub-token systems must learn token composition over longer sequences; character-level Seq2Seq can retain competitive full-command accuracy despite significantly lower structure accuracy.
  • Copying: Copying significantly improves full command accuracy for token- and sub-token models, making ST-CopyNet the strongest full-command system with competitive template accuracy.Character-level copying provides only a slight improvement, while token-level copying slightly reduces command template accuracy, possibly because source constants and command arguments require reformatting.
  • End-To-End vs. Pipline: ST-CopyNet outperforms Tellina on full command accuracy, especially on Acc3T, while showing only a mild decrease in structure accuracy.The comparison is presented as evidence that learned string-level transformations can outperform manually written heuristics when sufficient data is available.
  • Error Analysis: Rare utilities and flags are the top error cause for both evaluated models because the Bash domain contains many items that are expensive to cover with training data.Other errors involve RNN translation failures, enumerated constants, long or complex descriptions, preprocessing, and Tellina-specific stage-wise mistakes.
  • Future Work: Future work should explore shallow command structures, semi-supervised learning, and external resources to address sequential decoding and training-data sparsity.The paper specifically suggests separate recurrent networks for template translation and argument filling, alongside unlabeled Bash commands or Linux man pages.

7. Comparison to Existing Datasets

The paper compares NL2Bash with natural-language-to-code datasets across language, size, difficulty, and collection methodology. NL2Bash is distinguished by practical Bash snippets, expert-written descriptions, and broad diversity, while automatically collected datasets offer greater scale and diversity.

  • Comparison Dimensions: The comparison considers programming language, dataset size, difficulty statistics, and collection methodology.Difficulty measures include unique natural-language words, unique program tokens, average text length, and average code length.
  • Collection Methodology: NL2Bash is the largest manually constructed dataset using practical code snippets and expert-written natural language.The dataset comparison includes short-code translation datasets spanning DSLs and general-purpose languages such as Java, Python, C#, and Bash.
  • Dataset Diversity: NL2Bash has 7,790 unique words and 6,234 unique command tokens, making it more diverse than other manually constructed datasets.The paper reports these statistics in its comparison of collection methodologies and dataset diversity.
  • Alternative Collection Strategies: Automatically scraped or extracted datasets generally surpass manually gathered datasets in size and diversity, despite the difficulty of obtaining good natural-language/code alignments.The paper also discusses synthetic parallel data produced with synchronous grammars and paraphrased by crowd workers.

8. Conclusions

The paper introduces NL2Bash as a large dataset and baseline framework for mapping English sentences to Bash commands. Experiments show competitive existing-model performance while leaving substantial room for future work on this challenging semantic parsing task.

  • Conclusions: NL2Bash contributes a large new dataset and baseline methods for mapping English sentences to Bash commands.The conclusion frames the work as an initial study of this semantic parsing problem.
  • Conclusions: Existing models achieve competitive performance, but the task retains significant room for future research.The conclusion characterizes NL2Bash as a challenging semantic parsing problem.

Appendices

The corpus has uneven utility and flag coverage: some utilities are sparsely represented, and training data includes only a subset of available flags.

  • A1. Distribution of Less Frequent Utilities: Only 38 instances made dig the most frequent among the 52 least frequent utilities, while 7 utilities appeared 5 times or fewer.The paper links this sparsity to insufficient examples for learning all usages of low-frequency utilities.
  • A2. Flag Coverage: Most utilities have fewer than half of their total flags represented in the training set.Total flag counts were manually extracted from GNU man pages, giving a lower-bound estimate.
  • A2. Flag Coverage: Short flags are common in practice even when utilities provide more readable long-form replacements, reducing apparent training coverage.The paper suggests normalizing commands to short flags and restoring readable forms deterministically for users.
  • A2. Flag Coverage: Many utility flags remain absent from the corpus, motivating zero-shot learning as future work.The missing-flag problem persists beyond coverage issues caused by equivalent short and long flag forms.

B Data Quality

Manual checks found substantial but imperfect data quality, while automatic metrics only partially align with human judgments and can lack discrimination.

  • B Data Quality: 15 of 100 sampled training pairs contained errors, corresponding to approximately 85% annotation accuracy.Errors involved unclear directory handling, copied constants, misunderstood flags or reserved tokens, and syntactic mistakes.
  • B Data Quality: 13 of the 15 wrong pairs involved an annotator misinterpreting only one command token.The authors characterize overall annotator performance as high despite the corpus’s large domain.
  • C Automatic Evaluation Results: TM measures close-vocabulary token overlap for command structure, while BLEU approximates full-command accuracy across top-k candidates.TMk and BLEUk take the maximum respective score among the top-k generated candidates.
  • C Automatic Evaluation Results: Automatic metrics identify the systems with the highest full-command and structure accuracy but do not agree with manual evaluation in every case.Character-based models, for example, receive the second-best BLEU score despite disagreement with manual rankings.
  • C Automatic Evaluation Results: TM is not sufficiently discriminative because several systems receive similar scores.This limits its ability to distinguish baseline systems reliably.
Loading 1802.08979v2…