Source-linked AI summary
FLOWSTEP: A Framework for Learning to Predict GitHub Actions Workflow STEPs
Jiangnan Huang, Bin Lin, Yutaro Kashiwa
TL;DR
GHA workflow recommendation is challenging because Actions and Scripts require different forms of prediction, a distinction existing approaches often overlook. FLOWSTEP separates those tasks using step-type prediction, contrastive learning, and LLM generation; it improves raw-data workflow-step accuracy and reports substantial human–LLM judge agreement.
Problem
Existing GHA workflow recommenders often treat reusable Actions and project-specific Scripts as homogeneous targets, despite their different prediction requirements.
Source: §I · p. 1Method
FLOWSTEP predicts whether a workflow step is an Action or Script, then applies contrastive-learning-based Action prediction or LLM-based Script generation.
Source: §I · p. 1, §III · p. 3Results
Raw-data workflow-step accuracy improves from 38.23% to 51.09%, and DeepSeek-V3.2-Exp achieves Krippendorff’s alpha of 0.701 in zero-shot human alignment.
Source: §I · p. 2, B. Result Analysis · p. 8Takeaways & Limitations
The study finds substantial alignment between human annotations and LLM evaluations of generated workflow Scripts.
Source: B. Result Analysis · p. 8, §VIII · p. 10The Action–Script distinction can be simplistic because some functionalities can be implemented either way.
Source: §VI · p. 10
Abstract
from arXiv · showhide
Continuous Integration and Delivery (CI/CD) are fundamental to modern software development, yet configuring pipelines such as GitHub Actions (GHA) workflows remains complex and error-prone. Recent work has explored Transformer-based models for GHA workflow recommendation, but existing approaches typically model workflow steps as a single homogeneous generation task. This formulation overlooks a key distinction: reusable Actions are structured, ecosystem-level components, whereas project-specific Scripts encode local build, test, and deployment semantics. Consequently, current models struggle to capture the different forms of knowledge required for accurate workflow completion. In this work, we present FLOWSTEP, a framework for GHA workflow step completion that explicitly distinguishes between Action and Script generation. FLOWSTEP combines large language models (LLMs) with contrastive learning to better capture semantic relationships among workflow contexts, Actions, and Scripts. Evaluated on established benchmarks, FLOWSTEP improves end-to-end workflow step prediction accuracy from 38.23% to 51.09%. Beyond prediction accuracy, we further study whether LLM-as-a-judge can provide a more semantic evaluation of generated workflow steps. By comparing LLM-based assessments with human annotations, we find that the best-performing model configuration reaches substantial agreement with human ground truth (Krippendorff's alpha = 0.701), suggesting that LLM-as-a-judge can serve as a scalable complement to manual evaluation.
I. INTRODUCTION
GHA workflow authoring remains challenging, and existing recommendation models often treat reusable Actions and project-specific Scripts as homogeneous targets. FLOWSTEP separates their prediction strategies, while the introduction also situates workflow steps within GHA’s YAML-based job structure.
I. INTRODUCTION
- About 60% of 90 survey participants reported difficulty automating workflows using GHA.Workflow authoring also combines reusable Actions, free-form scripting, documentation, and project-specific details such as paths and URLs.Source: §I · p. 1
- Existing approaches treat workflow steps as homogeneous targets, although Actions require ecosystem identifier selection and Scripts require context-specific command generation.The paper reports that this mismatch contributes to prediction errors and reduces robustness across heterogeneous steps.Source: §I · p. 1
- FLOWSTEP predicts whether the next step is an Action or Script, then uses RoBERTa with contrastive learning for Actions and an LLM for Scripts.Its evaluation uses raw and abstracted variants of the GHA workflow corpus processed in prior work,.Source: §I · p. 1
- 51.09% versus 38.23% accuracy (raw data) is the reported improvement for FLOWSTEP’s step-type-aware framework over the existing result.The framework differentiates reusable Actions from project-specific Scripts.Source: §I · pp. 1–2
A. GitHub Actions Workflow
- A GHA workflow is a YAML configuration file whose jobs contain steps that invoke shell commands with run or reusable Actions with uses.Jobs may run concurrently or sequentially; steps are the smallest executable workflow elements.Source: A. GitHub Actions Workflow · p. 2
B. Transformers and Contrastive Learning
The section introduces Transformers, contrastive learning, and LLM prompting as relevant approaches, then frames GHA workflows as configuration artifacts that prior generation work has only partly captured. It also describes LLM-as-a-Judge as a developing evaluation paradigm with known challenges.
B. Transformers and Contrastive Learning
- Contrastive learning brings representations of positive pairs closer and separates negative pairs, and the paper uses it to fine-tune RoBERTa for workflow-step representations.RoBERTa is described as a BERT refinement using dynamic masking.Source: B. Transformers and Contrastive Learning · p. 2
C. LLMs and the “Judge" Paradigm
- Prior GHA workflow-generation attempts using fine-tuned T5 and LLMs only partially capture workflow semantics, with limited accuracy and generalization.The text characterizes configuration artifacts as underexplored relative to general-purpose programming languages.Source: C. LLMs and the “Judge" Paradigm · p. 2
- LLM-as-a-Judge uses language models as automatic evaluators, and prior work describes it as scalable and cost-effective while noting bias, inconsistency, and over-alignment challenges.Studies in software engineering have explored its use for code and software artifacts, with calibration proposed as relevant.Source: C. LLMs and the “Judge" Paradigm · p. 2
III. FLOWSTEP
FLOWSTEP represents workflow completion as predicting a step from preceding context and its descriptive name, with the target typed as an Action or Script. Its pipeline routes each type to a component suited to its output space.
III. FLOWSTEP
- Each dataset instance pairs preceding workflow context and the target step’s descriptive name with the next step and its Action-or-Script type.The paper derives separate Action and Script datasets from this representation.Source: §III · p. 3
- FLOWSTEP’s pipeline first predicts step type, then uses contrastive-learning RoBERTa for Action selection or an LLM for Script generation.The architecture is presented as a unified pipeline in which each subtask is handled by a model aligned with its output space.Source: §III · p. 3, Figure 1 · p. 3
A. Step-Type Predictor
- In the inspected GH-WCOM errors, 33.3% (53/159) were mismatches between an expected Action and a predicted Script, or vice versa.This analysis motivates distinguishing the two step types.Source: A. Step-Type Predictor · p. 3
- A RoBERTa step-type predictor classifies workflow context as Action or Script, and routes it to the corresponding downstream predictor or generator.The binary labels are 1 for Action and 0 for Script.Source: A. Step-Type Predictor · p. 3
B. Action Predictor with Contrastive Learning
- The Action Predictor contrastively aligns contexts using the same Action and separates contexts using different Actions, then trains a classification head to map representations to Action identifiers.The encoder is contrastively trained first; it is then frozen while the classification head is trained on DAction.Source: B. Action Predictor with Contrastive… · p. 3
C. LLM-based Script Generator
FLOWSTEP uses prompted LLMs to generate Script steps and selects its final generator through calibration-set comparisons. Claude-Sonnet-4.5 with one-shot prompting performs best in that selection process.
C. LLM-based Script Generator
- The study evaluates five LLMs from different providers and capability tiers, using zero-shot and one-shot prompts to test the effect of a contextual example.The prompts require shell-command Script steps without explanations or commentary.Source: C. LLM-based Script Generator · p. 4
- The comparison is limited to zero-shot and one-shot prompting; broader few-shot and advanced prompting strategies are left for future work.Generation temperature is fixed at zero, and each model generates one candidate per instance.Source: C. LLM-based Script Generator · p. 4
- A T5 model trained only on DScript serves as a Script-generation baseline and offline alternative to the prior GH-WCOM setup.Unlike GH-WCOM, this T5 model is not trained on the full dataset D.Source: C. LLM-based Script Generator · p. 4
- Claude-Sonnet-4.5 with one-shot prompting performs best on the 1,000-instance calibration set and is selected as FLOWSTEP’s Script Generator.Generated steps were compared with reference steps using accuracy, BLEU-4, and ROUGE-L.Source: C. LLM-based Script Generator · p. 4
A. Research Questions
The section frames two evaluation questions—overall recommendation accuracy and model-configuration effects—then specifies dataset construction and evaluation metrics.
A. Research Questions
- RQ1 tests whether FLOWSTEP recommendations match developer-created steps against GH-WCOM, while RQ2 examines how model configurations affect performance.RQ2 isolates the Action/Script separation and component contributions through error analysis and ablation studies.Source: A. Research Questions · p. 4
B. Data Preparation
- The data comprise repository-level train, validation, and test splits, with broad, Action-specific, and Script-specific datasets constructed using task-specific filters.The source workflows are deduplicated and filtered; Action labels must appear more than five times in training, and Script instances require a non-empty run field. Raw and abstracted variants are used, with project-specific tokens replaced by placeholders in the abstracted data.Source: B. Data Preparation · pp. 4–5
C. Evaluation Metrics
- Overall workflow completion is scored by exact match, while component evaluations use task-specific accuracy, Top-K accuracy, BLEU-4, and ROUGE-L.The Script Generator is also assessed on accuracy, BLEU, and ROUGE against a GH-WCOM baseline on the calibration test set.Source: C. Evaluation Metrics · p. 5
- The evaluation studies step-type errors and whether a dedicated predictor reduces them, then ablates Action Predictor and Script Generator components while holding the Step-Type Predictor fixed.Action Predictor variants remove contrastive learning or the classification head; the no-classification-head variant recommends Actions by cosine similarity to training-set mean embeddings.Source: C. Evaluation Metrics · p. 5
D. Experimental Setting
The experimental setting specifies training configurations and a fixed random seed for reproducibility.
D. Experimental Setting
- The Step-Type Predictor and Action Predictor use RoBERTa, train for 10 epochs with batch size 32, and use Adam at learning rate 2e−5.The Script Generator uses temperature 0 for deterministic LLM outputs; T5-small trains for 70 epochs with batch size 8, and the random seed is 123456.Source: D. Experimental Setting · p. 5
1) RQ1: How effective is FLOWSTEP in generating accurate recommendations for GitHub Actions workflows?:
FLOWSTEP improves end-to-end workflow-step prediction by separating Action and Script tasks, with gains across both datasets. Its component studies show that explicit type prediction, contrastive learning, and stronger Script-generation models contribute distinct performance patterns.
1) RQ1: How effective is FLOWSTEP in generating accurate recommendations for GitHub Actions workflows?:
- 51.05% versus 39.33% on abstracted data and 51.09% versus 38.23% on raw data are FLOWSTEP and GH-WCOM end-to-end accuracies, respectively.Table III reports accuracy across both Action and Script records.Source: 1) RQ1: How effective is FLOWSTEP in… · p. 5, Table III · p. 6
- 92.9%/91.96% type accuracy on abstracted/raw data exceeds GH-WCOM’s 82.86%/82.28%, while Action Top-1 reaches 76.28%/76.58% versus 61.71%/61.09%.For Scripts, FLOWSTEP reaches 24.72%/24.48% accuracy versus GH-WCOM’s 15.0%/13.4%, although Script generation remains challenging.Source: 1) RQ1: How effective is FLOWSTEP in… · p. 6
2) RQ2: What is the impact of different model configurations within the FLOWSTEP framework on its final performance?:
- Step-type mismatches account for 28.25%/28.69% of GH-WCOM errors but 14.51%/16.45% of FLOWSTEP errors on abstracted/raw data.Those FLOWSTEP mismatch totals are 7.1%/8.04% of all evaluated instances, compared with 17.14%/17.72% for GH-WCOM.Source: 2) RQ2: What is the impact of… · p. 6
- Without contrastive learning, Action Top-1 is 21.51%/23.37%; contrastive learning raises it to 80.05%/83.57%, and the classification head adds 7.18%/4.51%.The full model’s Top-5 accuracy is 94.63%/94.17%.Source: 2) RQ2: What is the impact of… · pp. 6–7, Table V · p. 7
- Under zero-shot prompting, Claude-Sonnet-4.5 leads most Script-generation metrics, while GPT-3.5-turbo reaches only 7.2%/11.5% accuracy.The comparison uses abstracted/raw calibration data and includes accuracy, BLEU, and ROUGE.Source: 2) RQ2: What is the impact of… · p. 7, Table VI · p. 7
- One-shot prompting raises Claude-Sonnet-4.5 accuracy to 27.5%/29.6%, but its benefit is not uniform across models.T5-small offers an offline alternative at 18.8%/14.7% accuracy without API access, illustrating a quality–deployment-cost tradeoff.Source: 2) RQ2: What is the impact of… · pp. 7–8, Table VII · p. 7
V. LLM-AS-A-JUDGE
Exact-match metrics can underrate generated Scripts when distinct commands achieve the same functionality. The paper therefore examines whether LLM-as-a-Judge can assess Script correctness more semantically.
V. LLM-AS-A-JUDGE
- The motivating concern is that exact string mismatches need not indicate different Script functionality.Source: §V · p. 8
- Exact-match accuracy is a conservative lower bound because Scripts can differ in form while achieving the same functionality.For example, “pip install mythx-cli” may be functionally equivalent to “pip3 install mythx-cli” in some environments.Source: §V · p. 8
- The paper investigates whether LLM-as-a-Judge can evaluate generated Scripts more effectively than commonly used metrics.Source: §V · p. 8
A. Study Design
The study asks whether LLMs can judge generated Script correctness, using the same five models tested for generation. It evaluates sampled incorrect outputs against human annotations and establishes consensus labels for comparison.
1) Research Question:
- The study asks whether LLM-as-a-Judge can more effectively evaluate automatically generated Script steps.Source: 1) Research Question · p. 8
2) Model Selection:
- The five Script-generation models—Claude-Sonnet-4.5, Claude-Haiku-4.5, DeepSeek-V3.2-Exp, GPT-4.1, and GPT-3.5-turbo—also serve as judges.Source: 2) Model Selection · p. 8
3) Evaluation Datasets:
- The evaluation samples 350 non-overlapping incorrect raw-data Scripts proportionally across models and compares generated steps with human judgments.Two annotators achieved α = 0.82; a third author adjudicated the 18 cases without consensus.Source: 3) Evaluation Datasets · p. 8
- Human labels distinguish functionally correct, almost correct, and incorrect generated steps.The categories are defined by equivalent purpose or result, need for a minor modification, or differing purpose, behavior, or missing key functionality.Source: 3) Evaluation Datasets · p. 8
4) Data Collection:
The authors compared LLM judges with human annotations using structured prompts and assessed agreement, correlation with human judgments, and evaluation cost. DeepSeek-V3.2-Exp had the highest zero-shot agreement, while LLM judges showed mixed one-shot effects and cost $0.07–$1.41 for 350 samples.
4) Data Collection:
- The judges received workflow context, ground-truth steps, and generated steps in structured prompts, and were instructed to return only a decision number.The prompts followed the human annotation guidelines, covered zero-shot and one-shot settings, and the first author manually verified all judgments.Source: 4) Data Collection · p. 8
B. Result Analysis
- α = 0.701 was the highest zero-shot agreement with human judgments, while GPT-4.1 reached α = 0.687 in one-shot; human–human agreement was α = 0.82.One-shot prompting improved agreement for Claude-Sonnet-4.5 and GPT-4.1 but reduced it for the other judges; even the best LLM remained about 0.12 below human–human agreement.Source: B. Result Analysis · pp. 8–9, Table VIII · p. 9
- ρ = 0.703 was the highest zero-shot correlation with human judgments; in one-shot, GPT-4.1 reached ρ = 0.695, while ROUGE reached ρ = 0.692.All 350 samples were incorrect script snippets, so accuracy was always zero and not meaningful for this comparison; unlike lexical-overlap metrics, LLM judges also assess functional correctness.Source: B. Result Analysis · p. 9, Table VIII · p. 9
- Judging all 350 samples cost $0.07–$1.41 depending on the model, supporting LLM judges as an affordable first-pass evaluation mechanism.The authors state that human judgment remains necessary for calibration and adjudication.Source: B. Result Analysis · p. 9
C. Discussion
LLM judges most often disagreed with humans on borderline functional-equivalence cases, and different models showed distinct strictness tendencies. The authors identify implications for semantic evaluation and threats related to validity and study scope.
C. Discussion
- Most disagreements occurred for Functionally Almost Correct scripts, which judges often misclassified as correct or incorrect.DeepSeek’s predictions were more balanced, while Claude models and GPT-4.1 tended to downgrade correct scripts and GPT-3.5 rarely predicted correct samples.Source: C. Discussion · p. 9
D. Implication
- LLM judges can evaluate script intent and logic beyond character-level overlap, but remain less reliable than human experts.The authors present them as a scalable, cost-effective evaluation method, while retaining human annotation as the reliability gold standard.Source: D. Implication · p. 9
VI. THREATS TO VALIDITY
- Data preprocessing and training configurations may introduce bias, although the authors fixed random seeds and used consistent data splits.For manual annotation, two annotators labeled samples independently, disagreements went to a third annotator, and inter-rater agreement was measured.Source: §VI · p. 10
- The study’s external validity is limited because its dataset covers open-source GitHub projects, not necessarily private or industrial workflows.LLM evaluation may also behave differently with changes in prompt design or model versions.Source: §VI · p. 10
- The binary Action–Script distinction can be simplistic because some functionality can be implemented either way.The authors identify this as a construct-validity limitation for future work.Source: §VI · p. 10
VII. RELATED WORK
Prior work establishes the practical importance of GitHub Actions and explores workflow prediction and LLM evaluation, but leaves gaps in step-type modeling and semantic correctness assessment. FLOWSTEP addresses these gaps with differentiated Action and Script handling; its conclusion also identifies repository-specific context as unexploited.
A. GitHub Actions and CI/CD Automation
- Studies of GitHub Actions cover Action reuse, resource consumption, vulnerabilities, workflow failures, and security risks, motivating reliable workflow authoring support.GitHub Actions is described as a dominant CI/CD service on GitHub.Source: A. GitHub Actions and CI/CD Automation · p. 10
B. Learning-based Program and Workflow Generation
- Prior learning-based approaches include CIGAR’s contrastive Action recommendation, GH-WCOM’s T5 workflow-step completion, and LLM-based workflow generation studies.The passage notes that Gavel addresses Action categorization from README files to improve Marketplace visibility.Source: B. Learning-based Program and Workflow… · p. 10
C. LLMs for Software Engineering Tasks
- LLM-as-a-Judge studies in software engineering report promise alongside imperfect alignment with human judgment [37].Related LLM research also spans code generation, completion, summarization, bug localization, test generation, and repair –[51].Source: C. LLMs for Software Engineering Tasks · p. 10
D. Distinction and Contribution of FLOWSTEP
- Prior work leaves gaps because GH-WCOM treats steps as monolithic outputs, CIGAR focuses only on Actions, and LLM workflow studies give limited attention to contextual correctness and semantic evaluation.FLOWSTEP responds by modeling step type, predicting reusable Actions, generating project-specific Scripts, and evaluating generated scripts with human-calibrated LLM judges.Source: D. Distinction and Contribution of… · p. 10
VIII. CONCLUSIONS
- FLOWSTEP combines step-type prediction, contrastive learning for Action selection, and LLM-based Script generation, and the authors report substantial alignment between LLM evaluations and human annotations.The framework relies only on workflow specifications, not repository-specific information; future work plans repository exploration and security analysis.Source: §VIII · p. 10