PaperlayerView exact revision on arXiv

Source-linked AI summary

DynaCI: Generating GitHub Actions Workflows via Dynamic Analysis

Javid Ditty, Rahul Ghanshyam Chavan, Nimmi Weeraddana, Foyzul Hassan

arXiv:2610.29890v1cs.SE

TL;DR

Existing workflow-generation tools require partial workflows or detailed specifications, while configuring GitHub Actions is challenging and demands domain knowledge. DynaCI traces local commands, dependencies, and execution order, applies heuristics to build an initial workflow, and iteratively refines it using LLM-based RAG. DynaCI achieved a BLEU score of 52.81% and an F1 score of 62.06%, outperforming the state-of-the-art by more than 21.13% and 20.96%, respectively.

  • Problem

    Existing workflow-generation tools require partial workflows or detailed specifications, while configuring GitHub Actions is challenging and demands domain knowledge.

    Source: §I · p. 1

  • Method

    DynaCI traces local commands, dependencies, and execution order, applies heuristics to build an initial workflow, and iteratively refines it using LLM-based RAG.

    Source: §I · p. 2

  • Results

    DynaCI achieved a BLEU score of 52.81% and an F1 score of 62.06%, outperforming the state-of-the-art by more than 21.13% and 20.96%, respectively.

    Source: §I · p. 2

  • Takeaways & Limitations

    DynaCI relies on developers’ local environments rather than substantial GitHub Actions domain knowledge, reducing knowledge barriers and effort requirements.

    Source: §VIII · p. 10

    The evaluation covers 50 Python projects, and DynaCI currently supports only Linux-based operating systems.

    Source: §VI · p. 10

Abstract

from arXiv · show

Continuous Integration (CI) is an essential practice that automatically validates code changes through builds and tests. However, configuring CI workflows remains challenging, as modern software often features complex architectures, numerous dependencies, and support for diverse execution environments. Most CI platforms also require workflows to be written in their particular domain-specific languages. As a result, CI workflows are often error-prone and hard to maintain, especially for those unfamiliar with CI semantics. Prior work on automated CI workflow generation require developers to create specifications that describe their intended workflows using natural language, partial workflows, etc. However, in practice, developers may struggle to articulate requirements and may be unaware of available CI features. To address these limitations, we propose DynaCI, a novel approach that uses runtime information from developers' local environments to generate CI workflows. DynaCI (i) traces the developers' local environment to collect runtime information, (ii) constructs an initial workflow by applying runtime-informed heuristics, and (iii) iteratively refines the workflow via LLM-based Retrieval Augmented Generation (RAG). We evaluate DynaCI on 50 popular Python projects that use GitHub Actions, a popular CI platform, and compare it against the state-of-the-art CI workflow generation approaches: GH-WCOM and LLM4GW. DynaCI outperforms these existing approaches by more than 21.13% in BLEU score and 20.96% in F1 score on average. It also generates workflows without missing functionality in 56% of cases, while requiring fewer manual modifications to repair such exceptions. These results demonstrate that leveraging runtime information substantially improves LLM-based CI workflow generation, and offer a promising direction for broader DevOps configuration automation.

I. INTRODUCTION

DynaCI uses developers’ local runtime traces to build and refine GitHub Actions workflows, addressing the input burden of prior generators. On 50 popular Python projects, it outperformed existing tools on workflow similarity and required fewer modifications.

I. INTRODUCTION

  • GH-WCOM requires developers to write most of a workflow, while LLM4GW requires detailed workflow-, job-, and step-level specifications.The introduction presents these input requirements as demanding substantial GitHub Actions domain knowledge.Source: §I · p. 1
  • 52.81% BLEU and 62.06% F1; DynaCI outperformed state-of-the-art tools by more than 21.13% and 20.96%, respectively, across 50 popular Python projects.The BLEU and F1 scores are DynaCI’s reported scores; the respective deltas are its comparisons against the state-of-the-art.Source: §I · p. 2
  • DynaCI incurs a median overhead of 5.55 minutes and 653,755 tokens per workflow.The paper measures overhead in elapsed time and LLM token usage per workflow.Source: §I · p. 2
  • 44% of DynaCI workflows lacked intended functionality, while every workflow from state-of-the-art tools needed modification.The manual analysis assessed whether workflows required modification to match intended functionality.Source: §I · p. 2
  • DynaCI is presented as the first approach, to the authors’ knowledge, to combine runtime analysis with LLMs for GitHub Actions workflow synthesis.It traces executed commands, dependencies, and execution order, applies heuristics to assemble an initial workflow, then iteratively refines it using LLM-based RAG.Source: §I · p. 2
  • The ablation study finds that DynaCI’s hybrid approach contributes substantially beyond its heuristics or the LLM alone.The introduction reports this as an ablation finding.Source: §I · p. 2

II. BACKGROUND AND MOTIVATION

GitHub Actions workflows offer extensive configuration, making them difficult to write without supporting tools. Existing generation scenarios still require developers to provide workflow fragments or instructions, while DynaCI uses local execution traces to generate and refine a workflow.

A. GitHub Actions Workflows

  • GHA workflows use YAML to define event-triggered jobs, shared machines, steps, and reusable actions, alongside many settings that make them difficult to write without support.Jobs contain ordered steps and can configure environment variables, execution limits, and configuration matrices.Source: A. GitHub Actions Workflows · p. 2

B. Motivating Example

  • GH-WCOM completes partial workflows and LLM4GW uses natural-language instructions, but both scenarios still require developers to supply workflow-related input.DynaCI instead traces local execution and needs no additional workflow components or natural-language instructions.Source: B. Motivating Example · pp. 2–3

III. APPROACH

  • DynaCI’s approach executes and traces commands, applies heuristics, and refines the workflow with LLMs to produce a workflow that reproduces the developer’s local environment.The described design targets Python environments and could be generalized to languages such as Java.Source: §III · p. 3

A. Execute Commands

DynaCI starts from build and test commands developers run locally, where project-specific resources are available. It executes those commands twice to collect runtime traces that inform workflow synthesis.

A. Execute Commands

  • Local build and test runs can access project-specific test files, packages, and environment variables, and help detect faults before code is pushed.Project documentation and configuration commonly record the commands and dependencies needed for these activities.Source: A. Execute Commands · p. 3

B. Trace Commands

  • DynaCI takes developer commands as input and executes them twice: strace captures command durations, file accesses, and package installations, while ltrace captures environment-variable accesses.The two runs are separated because multiple ptrace tools cannot attach to the same process simultaneously.Source: B. Trace Commands · p. 3

1) First Execution:

  • In the first execution, four probes partition commands and trace their durations, file accesses, and package dependencies for later workflow-step construction.The Command probe partitions command sequences into executable units; Table I gives an example of the extracted traces.Source: 1) First Execution · pp. 3–4

2) Second Execution:

DynaCI traces environment-variable accesses in a second execution and uses local runtime information to select workflow components with local counterparts or a stated need for workflow execution.

2) Second Execution:

  • The second execution traces environment-variable accesses using shell and Python probes, including instrumentation of Python’s os.environ interface.The paper notes that a similar runtime-interface probe could be implemented for Java.Source: 2) Second Execution · p. 4

C. Apply Heuristics

1) Filter Candidate Workflow Components:

  • For environment components, DynaCI includes only locally observable hard-coded variables; for execution components, it retains command execution but excludes OS or shell configuration components as rare in the dataset.For Worker components, it retains job and step ordering and timeout controls, but not CI-specific job-trigger conditions.Source: 1) Filter Candidate Workflow Components · p. 4

2) Apply Non-Parametric Heuristics:

DynaCI uses runtime-derived rules to assemble workflow settings and steps, then supplements them with empirically tuned parameters for components that runtime traces cannot determine.

2) Apply Non-Parametric Heuristics:

  • DynaCI’s non-parametric heuristics use observed runtime behavior without tunable parameters to generate a preliminary workflow.They initialize job environment variables, configuration matrices, execution machines, steps, and reusable setup actions from local observations.Source: 2) Apply Non-Parametric Heuristics · pp. 4–5
  • DynaCI’s current tracing tools restrict generated workflows to Ubuntu runners, which account for 64.6% of GHA workflow jobs in general.The matrix uses the OS and Python version observed in runtime traces.Source: 2) Apply Non-Parametric Heuristics · p. 5
  • For workflow steps, DynaCI installs missing apt and pip packages, groups developer commands, and uses multi-criteria decision making to balance command-duration spread against step count.Command order is preserved and each candidate step group contains at least one command.Source: 2) Apply Non-Parametric Heuristics · p. 5

3) Apply Parametric Heuristics:

  • Parametric heuristics use thresholds to decide whether to include components and multipliers to set numeric values based on observed job duration.For example, the timeout limit is included above 319 seconds and set to 13 times observed job duration; concurrency control is included above 138 seconds.Source: 3) Apply Parametric Heuristics · p. 5
  • DynaCI derives empirical heuristic parameters from 7,157 successful job runs in its Heuristic-Tuning Dataset, filtered from a dataset of 2,595,399 GHA workflows.The workflow-component thresholds and multipliers are based on average durations after outliers are removed.Source: 3) Apply Parametric Heuristics · p. 5
  • When a job exceeds 66 seconds, DynaCI sets matrix fail-fast to false so other jobs can run despite a failure; otherwise, the default true behavior remains.The initial workflow is intended for later refinement via LLMs.Source: 3) Apply Parametric Heuristics · pp. 5–6

D. Refine with LLMs

DynaCI starts with a permissive, runtime-informed workflow, then uses LLM refinement to remove unnecessary or problematic components and correct syntax. RAG supplies relevant local and runner runtime context for job-level refinement.

D. Refine with LLMs

  • The initial workflow aims to include components, but this permissiveness can make it complicated and error-prone; LLM refinement removes unnecessary or problematic components and patches syntax errors.Source: D. Refine with LLMs · p. 6

1) Extract Runtime Environment from GitHub Actions:

  • DynaCI gathers runner environment information to help the LLM remove redundant local environment variables and package installations.Runner information is collected manually by running apt, pip, and env commands on the target runner.Source: 1) Extract Runtime Environment from… · p. 6

2) Embed Runtime Information:

  • Because runtime information can exceed 1 GB, DynaCI chunks and embeds it, then retrieves only relevant chunks for the LLM.This RAG approach limits the information sent to the LLM.Source: 2) Embed Runtime Information · p. 6

3) Refine at Job-Level:

  • DynaCI refines jobs separately, retrieving runtime chunks with cosine similarity ≥60% of their run blocks and prompting the LLM to remove unneeded components.The prompt includes target GitHub Actions environment details and warns the LLM against extra changes, following preliminary experiments in which it tended to hallucinate beyond the intended scope.Source: 3) Refine at Job-Level · p. 6

4) Merge Refined Jobs and Refine at Workflow-Level:

DynaCI merges separately refined jobs into a workflow and validates it with actionlint, regenerating after errors up to five iterations. The evaluation uses 50 high-star Python project workflows, reconstructed local environments, and several baseline approaches.

4) Merge Refined Jobs and Refine at Workflow-Level:

IV. EXPERIMENTAL SETUP

  • The evaluation compares DynaCI, GH-WCOM, and LLM4GW against target workflows from 50 high-star projects, measuring effectiveness, efficiency, and modification needs.The projects were held aside to avoid data leakage from heuristic tuning.Source: §IV · p. 6

A. Local Environment Configuration

  • Researchers reconstructed local environments from developer-facing documentation and configurations, adding setup commands when needed to make them executable and representative.90.8% of commands in the sample involved generic commands such as tox.Source: A. Local Environment Configuration · p. 6

B. Workflow File Generation Using DynaCI

C. Workflow Generation Using Baseline Approaches

  • The baselines include commands-only LLM generation, DynaCI heuristics without refinement, GH-WCOM completion, and LLM4GW using comparable prompt P1.P1 was selected because the use case assumes developers cannot articulate target workflow details.Source: C. Workflow Generation Using Baseline… · p. 7

D. Performance Metric Computation

The evaluation measures workflow similarity, generation efficiency, and the manual work needed to match target execution. Its measures include BLEU and F1 scores, time and token use, and repair rate and changed lines of code.

1) RQ1. Effectiveness:

  • BLEU compares generated and target workflows at the token level, while F1 balances correctness and completeness of matching workflow components.F1 is computed from true positives, false positives, and false negatives, with component values required to match.Source: 1) RQ1. Effectiveness · p. 7

2) RQ2. Efficiency:

  • Tool efficiency is characterized by workflow-generation time and, for LLM-based tools, token consumption.Source: 2) RQ2. Efficiency · p. 7

3) RQ3. Modification:

  • For modification needs, authors execute generated workflows and revise them until execution logs match targets, recording changed lines and repair categories.Source: 3) RQ3. Modification · p. 7
  • Repair rate measures the share of generated workflows missing functionality, while Delta LoC totals lines inserted, modified, or deleted to add it.Source: 3) RQ3. Modification · p. 7

E. Hardware / Software Requirements

DynaCI outperforms the baselines on average BLEU and F1 scores. Its refinement balances removing extraneous components against the risk of losing needed ones, with strengths varying by workflow component.

E. Hardware / Software Requirements

  • The implementation used GPT-4o-mini for generation and ADA-002 for embeddings, though neither model is intrinsic to the approach.The Dockerized environments ran on Ubuntu 24.04 with an Intel i5-10600K CPU, Radeon RX 5700 XT GPU, and 16 GB RAM.Source: E. Hardware / Software Requirements · pp. 7–8

A. RQ1. How effective is DynaCI at generating CI workflows?

  • Refinement reduces false positives by removing extraneous components, but risks removing important components and increasing false negatives.The reported F1-score differences between DynaCI and baselines in Table II are statistically significant (p < 0.05).Source: A. RQ1. How effective is DynaCI at… · p. 8
  • DynaCI and Heuristics-Only generally perform better on environment-variable, error-handling, and command-execution components, while package-installation steps can count as false positives.Source: A. RQ1. How effective is DynaCI at… · p. 8
  • +21.13% higher average BLEU score and +20.96% higher average F1 score versus the state-of-the-art approaches.DynaCI’s average F1 score is 62.06%.Source: Table II · p. 8, A. RQ1. How effective is DynaCI at… · p. 8
  • DynaCI and Heuristics-Only outperform other tools on fail-fast and environment-variable components, although runtime-derived variables include false positives.Source: A. RQ1. How effective is DynaCI at… · p. 8
  • DynaCI, Heuristics-Only, and GH-WCOM outperform LLM4GW and Commands-Only on execution-configuration components, where missing matrix variants lead to inconsistencies and false negatives.Source: A. RQ1. How effective is DynaCI at… · p. 8

B. RQ2. How time and cost efficient is DynaCI?

DynaCI’s workflows often avoid manual repair, but runtime-informed generation carries time and token costs. The repair analysis also identifies fewer required changes than for baseline workflows and describes recurring sources of repair.

B. RQ2. How time and cost efficient is DynaCI?

  • DynaCI’s average workflow-generation time was 333.66 seconds, compared with 187.45 seconds for Heuristics-Only; the other tools had negligible durations.Trace Commands, which monitors the local environment twice, was the longest step.Source: B. RQ2. How time and cost efficient is… · p. 8
  • DynaCI used 653,755 tokens per workflow on average, versus 868 for Commands-Only and 199 for LLM4GW, largely because runtime-information chunks account for most usage.The authors describe this cost as modest because most repositories use only a few CI workflows and DynaCI’s performance gains outweigh the trade-off.Source: B. RQ2. How time and cost efficient is… · p. 9, Table V · p. 9

C. RQ3. How much modification is needed for CI workflows generated by DynaCI?

  • DynaCI required 52 total changes and a median of 4, matching Heuristics-Only’s median; LLM refinement removes many unsupported installations introduced by heuristics.Heuristics-Only may attempt to install locally observed apt packages unavailable in the GHA environment.Source: C. RQ3. How much modification is… · p. 9
  • GH-WCOM and LLM4GW required the most changes (592), with insertions dominating (340) because of missing functionality and incompatible component configurations.The baselines often used Python versions unsupported by projects.Source: C. RQ3. How much modification is… · p. 9
  • 28 of 50 DynaCI workflows (56%) required no manual repair, while GH-WCOM, LLM4GW, and Commands-Only required repairs in every case to add missing functionality.Heuristics-Only required repairs in 39 cases.Source: C. RQ3. How much modification is… · p. 9
  • Repair cases include missing developer-executed commands and apt packages unavailable in GHA; DynaCI refinement mitigates unresolved-package issues but may not eliminate them.Developers may still need to remove problematic packages or configure required repositories.Source: C. RQ3. How much modification is… · p. 9
  • Other repair causes include incompatible Python versions, environment variables that behave differently in GHA, and missing uv setup in GH-WCOM and LLM4GW.The cited repairs include specifying a compatible Python version, removing problematic variables, or adding uv setup to failing jobs.Source: C. RQ3. How much modification is… · pp. 9–10

VI. THREATS TO VALIDITY

The evaluation’s validity is bounded by its project and platform scope, its textual similarity measures, and the process used to quantify repairs.

Internal Validity

  • Repair counts may vary with reviewers’ chosen fixes; two authors independently assessed 25 projects, reached consensus, and the first author applied the criteria to all 50.Source: §VI · p. 10

External Validity

  • The study covers 50 Python projects using GitHub Actions, runs only on Linux, and requires workflow commands that can run outside CI, limiting coverage of CI-specific features.The authors note that results may not fully capture features such as unofficial reusable actions.Source: §VI · p. 10

Construct Validity

  • BLEU and F1 measure textual similarity but do not fully capture practical correctness, so the authors also manually assessed required repairs.The combined evaluation addresses textual and practical aspects of generated workflows.Source: §VI · p. 10

VII. RELATED WORK

DynaCI uses local runtime information to generate GHA workflows, unlike prior approaches that complete existing workflows or require GHA domain knowledge. The conclusion describes potential uses for reducing developer effort, troubleshooting generated workflows, and extending divide-and-conquer generation to other DevOps configurations.

VII. RELATED WORK

  • Unlike GH-WCOM’s workflow-gap completion and LLM4GW’s domain-knowledge-dependent prompting, DynaCI generates workflows from scratch using local runtime information.The authors say this reduces developer effort by reusing information from existing local testing environments.Source: §VII · p. 10
  • DynaCI extends runtime-information research to CI workflow synthesis; prior GHA runtime-log approaches diagnose and repair failures but do not generate workflows from scratch.The authors describe DynaCI as the first runtime-informed approach to CI workflow synthesis, to the best of their knowledge.Source: §VII · p. 10

VIII. CONCLUSION AND IMPLICATIONS

  • Developers may use DynaCI to translate local environments into effective CI workflows with little cost and modification, reducing GHA knowledge barriers.The conclusion connects this implication to more effective workflows, fewer required modifications, and low production cost.Source: §VIII · p. 10
  • Tool builders could create recommendation systems to troubleshoot generated-workflow problems, including missing functionality, using DynaCI’s repair categories.The paper notes that DynaCI needs fewer modifications than state-of-the-art tools, but can still produce problems developers must correct.Source: §VIII · pp. 10–11
  • Researchers may extend DynaCI’s divide-and-conquer approach from CI workflows to other DevOps configurations, such as Docker and Terraform.The approach targets individual workflow components and integrates them through a refinement process.Source: §VIII · p. 11
Loading 2610.29890v1…