PaperlayerView exact revision on arXiv

Source-linked AI summary

Specifying and Maintaining Agentic Workflows: An Empirical Study of GitHub Agentic Workflows

Jasem Khelifi, Issam Oukhay, Ali Ouni, Mohammed Sayagh, Mohamed Aymen Saied

arXiv:2609.27263v1cs.SE

TL;DR

How developers maintain gh-aw Markdown files that combine agent instructions with execution configuration, and which operational requirements and safeguards they encode, remains insufficiently understood. The study analyzes gh-aw workflow structure, maintenance histories, and instruction content, developing a taxonomy and assessing LLM labeling against resolved human labels. Among 348 files with at least 120 days of observed activity, 78.2% received updates in the fourth month; instruction bodies accounted for most changed lines.

  • Problem

    How developers maintain gh-aw Markdown files that combine agent instructions with execution configuration, and which operational requirements and safeguards they encode, remains insufficiently understood.

    Source: §I · p. 2

  • Method

    The study analyzes gh-aw workflow structure, maintenance histories, and instruction content, developing a taxonomy and assessing LLM labeling against resolved human labels.

    Source: §I · p. 2, A. Novelty statement · p. 2

  • Results

    Among 348 files with at least 120 days of observed activity, 78.2% received updates in the fourth month; instruction bodies accounted for most changed lines.

    Source: §VIII · p. 15

  • Takeaways & Limitations

    Workflow instructions merit explicit attention during review, and the findings identify safety instructions and resource budgets as areas for closer review without showing that corresponding runtime protections are absent.

    Source: §VIII · p. 15

    Flesch Reading Ease scores for complete Markdown bodies are descriptive formula outputs, not validated measures of human or agent comprehension.

    Source: §VI · p. 14

Abstract

from arXiv · show

Agentic workflows shift software development from prompting AI agents for individual tasks to defining recurring work that agents execute automatically. GitHub Agentic Workflows (gh-aw) enables this approach through Markdown files that combine natural-language instructions with configuration and compile into executable GitHub Actions workflows. Unlike conventional workflows that primarily prescribe scripted operations, these files delegate tasks requiring interpretation to AI agents. They also couple agent instructions with execution triggers, making those instructions operational specifications for repeated repository activities. However, how developers structure and maintain these specifications, and which execution requirements and safeguards they express, remains insufficiently understood. In this paper, we examine the structure, evolution, and instruction content of gh-aw Markdown files to inform how practitioners define and maintain agent-run work. We analyze 1,248 files from 276 repositories, 20,841 commit-file events, and 288 resolved instruction-label sets from a sample of 294 files. Our results show that workflow instructions extend beyond short prompts, with a median of 556.5 words and code blocks in 62.1% of files. Among files with at least 120 days of observed activity, 78.2% still receive updates in month 4, while size-normalized churn decreases after the first month. Tasks, outputs, constraints, and process instructions each appear in over 93% of labeled workflows, yet only 9.4% explicitly address prompt-injection defense. LLM classification achieves an F1 score of 0.818 and Cohen's Kappa of 0.715 against the resolved human labels. These findings suggest that developers should account for the evolution of workflows copied or referenced across repositories and consider adding prompt-injection defenses, resource budgets, and evidencecredibility checks where applicable.

I. INTRODUCTION

Agentic workflows make natural-language instructions persistent specifications for recurring repository tasks, but how developers maintain and structure these combined instruction-and-configuration files remains insufficiently understood. This study examines gh-aw workflow structure, evolution, and instruction content, including a taxonomy and an assessment of LLM labeling.

I. INTRODUCTION

  • The study addresses limited evidence on maintenance over time, edits to configuration versus instructions, and the operational requirements and safeguards developers express.It analyzes 1,248 files from 276 repositories and manually examines a sample of 294 files.Source: §I · p. 2

• RQ3.What agentic instructions do developers set in GitHub Agentic Workflows?

  • Among files with at least 120 days of observed activity, 78.2% receive updates in month 4; LLM classification achieves F1 0.818 and Cohen’s κ 0.715 against resolved human labels, indicating substantial agreement.Source: • RQ3.What agentic instructions do… · p. 2

A. Novelty statement

  • The paper develops a ten-category, 42-subcategory taxonomy describing tasks, outputs, execution steps, constraints, and safeguards in gh-aw instructions.The taxonomy includes definitions and examples intended to support workflow review.Source: A. Novelty statement · p. 2
  • The study characterizes workflow size, structure, and composition, and tracks maintenance and reuse across projects while distinguishing frontmatter changes from instruction-body changes.Its maintenance analysis follows changes over four months while accounting for workflow size and repository activity.Source: A. Novelty statement · p. 2

B. Open Science

gh-aw combines operational configuration and agent-facing instructions in Markdown, which is compiled into GitHub Actions workflows. A motivating example illustrates recurring workflows exchanging repository artifacts and undergoing reviewed instruction updates; the study also provides replication materials.

II. BACKGROUND

  • A gh-aw source file combines YAML frontmatter for triggers, permissions, model or engine choices, and tool access with a Markdown body describing the agent’s task, context, steps, and expected output.The compiler turns the source into a generated Actions workflow.Source: §II · p. 2

A. Motivating Example

  • In dotnet/runtime, three scheduled workflows investigate CI failures, attempt repairs, and assess outputs and feedback to propose instruction updates.They exchange information through issues, pull requests, and run records; proposed instruction changes affect later runs after review and integration.Source: A. Motivating Example · pp. 2–3
  • A failed platform-specific cryptography-test change led to reviewed guidance directing such cases to domain owners, refining which work agents handle versus defer to expertise.The proposal was preserved as an issue after an automated push failed, then implemented in a reviewed, merged pull request.Source: A. Motivating Example · p. 3

B. Open Science

  • The example motivates studying maintained instructions and configuration, but does not establish how common these practices are across repositories.A replication package provides the dataset, scripts, labeling material, and experiments.Source: B. Open Science · p. 2, A. Motivating Example · p. 3

III. STUDY DESIGN

The study selects gh-aw projects, collects Markdown sources and histories, cleans the corpus, and develops an instruction taxonomy. The resulting dataset contains 1,248 files from 276 repositories and 20,841 file-level changes.

A. Project Selection

  • The study identifies generated gh-aw workflows through GitHub code search, splitting oversized result sets into non-overlapping file-size buckets to retrieve results beyond the per-query limit.The final buckets each returned fewer than 1,000 matches, after which results were deduplicated by repository and path.Source: A. Project Selection · p. 3
  • Retaining repositories with at least 10 stars yielded 1,435 generated workflows across 365 repositories; source mapping and deduplication produced 1,295 Markdown files hosted by 281 repositories.At least one workflow in 108 of the 365 projects referenced Markdown from another repository.Source: A. Project Selection · p. 4, B. Data Collection · p. 4

B. Data Collection

  • The researchers collected commit histories and snapshots for Markdown sources, recording file-level additions, deletions, status, commit metadata, and dates.The histories include pre-deletion snapshots for deleted files.Source: B. Data Collection · p. 4

C. Data Cleaning

  • After excluding 29 empty files and 18 confidently classified non-English files, the corpus contains 1,248 files from 276 repositories and 20,841 file-level changes.Those changes occur across 7,446 distinct commits touching retained files.Source: C. Data Cleaning · p. 4

D. Instruction Taxonomy

The study manually codes workflow instructions and characterizes their size, organization, and readability. The files use structured headings, but their median Flesch Reading Ease score falls in the conventionally difficult range, with interpretation qualified by mixed content.

D. Instruction Taxonomy

  • The authors code a representative sample of 294 files using pinned snapshots and Markdown bodies only, assigning multiple labels when a file contains different instruction types.The sample-size calculation uses 95% confidence, a 5% margin of error, and a conservative population proportion of 0.5.Source: D. Instruction Taxonomy · p. 4
  • The thematic analysis calibrates two authors on 30 files, assigns descriptive labels, reviews overlapping concepts, and groups labels into themes.For example, required sequences, decision branches, and within-run iteration are grouped under Process.Source: D. Instruction Taxonomy · p. 4

A. RQ1: What are the characteristics of GitHub Agentic Workflows?

  • Workflow files have a median of 9 headings, and H2 and H3 headings account for 85.5% of all headings.Heading counts range from zero to 73 per file; the per-file medians are four H2 and three H3 headings.Source: A. RQ1: What are the characteristics… · p. 5
  • At file level, step, issue, and phase appear in headings in 31.3%, 28.4%, and 11.7% of files, respectively; step headings specify actions, issue headings cover issue handling and reporting, and phase headings divide work into numbered stages.Source: A. RQ1: What are the characteristics… · p. 5
  • 38.8 is the median Flesch Reading Ease score, placing gh-aw Markdown bodies in the metric’s conventional difficult range.The comparison with prose-focused studies is not a direct ranking because these measurements include complete Markdown bodies.Source: A. RQ1: What are the characteristics… · p. 5
  • Code blocks occur in 62.1% of files, template expressions in 60.4%, and pictographic symbols in 47.4%, potentially affecting FRE inputs.The scores alone do not establish that developers have little need to read or review these files.Source: A. RQ1: What are the characteristics… · p. 5

B. RQ2: How are GitHub Agentic Workflows maintained over time?

This section examines how GitHub Agentic Workflows are maintained over time and what their instructions specify. It covers changes in project-level collections and the prevalence of instruction categories, including the uncommon use of explicit prompt-injection defenses.

B. RQ2: How are GitHub Agentic Workflows maintained over time?

  • 395 median project-level Markdown lines in month 1 rose to 429, 430, and 467 in months 2–4.Growth was significant from month 1 to 2 and month 3 to 4 (both Holm-adjusted p < .001), but not from month 2 to 3 (p = .511); median file size stayed near 200 lines, and the different panels do not establish the source of project-level growth.Source: B. RQ2: How are GitHub Agentic… · p. 6, Figure 6 · p. 7, Figure 5 · p. 7
  • 31.2 to 5.6 changed lines per 100 lines was the median project-level churn drop from month 1 to 2 (adjusted p = .011).Later consecutive months showed no significant churn differences, while the share of repository commits touching workflows also did not differ significantly between consecutive months (all adjusted p ≥ .561); the early churn drop therefore does not establish a decline independent of overall project activity.Source: B. RQ2: How are GitHub Agentic… · p. 6, Figure 6 · p. 7
  • 74.5% of project-level maintenance changes touched frontmatter at the median, versus 50.0% touching body text.Across pooled changes, 44.8% affected only frontmatter, 37.8% only the body, and 17.3% both; frontmatter-only changes rose from 36.7% in month 1 to 71.7% in month 4. Both-region changes count in both shares, and pooled percentages weight changes rather than projects.Source: B. RQ2: How are GitHub Agentic… · p. 6, Figure 7 · p. 9

C. RQ3: What agentic instructions do developers set in GitHub Agentic Workflows?

  • Tasks (97.6%), outputs (95.8%), constraints (93.8%), and processes (93.4%) appear in most labeled workflows, while prompt-injection defense appears in 9.4%.The taxonomy also reports execution isolation in 13.9%, information confidentiality in 7.3%, and vulnerability handling in 6.3% of the 288 resolved-label files.Source: C. RQ3: What agentic instructions do… · pp. 7–8
  • Task instructions define agent work, output instructions specify delivery, and constraints set action limits or quality standards.Frequent subcategories include Evidence Investigation (51.7%), Control Signal (52.1%), Scope Boundary (76.7%), and Quality Standard (74.7%); resource budgets (38.2%) and evidence grounding (36.1%) are also described.Source: C. RQ3: What agentic instructions do… · pp. 7, 9
  • Process instructions cover ordered steps (Required Sequence, 88.2%), conditional actions (Decision Branch, 87.2%), iteration, cross-run memory, and delegation.Within-Run Iteration appears in 35.8%, Cross-Run Memory in 18.1%, and Work Delegation in 9.0% of labeled files.Source: C. RQ3: What agentic instructions do… · pp. 9–10
  • Resource instructions appear in 89.9% of labeled files, commonly naming GitHub interfaces (60.4%) or local workspaces (57.6%).Local Workspace had the lowest pre-resolution agreement: 61.81% agreement and κ = 0.238.Source: C. RQ3: What agentic instructions do… · p. 10
  • Agent-resource instructions appear in 21.5% of workflows, development-toolchain instructions in 18.8%, and web retrieval instructions in 14.2%.These direct agents to use resources such as subagents or reusable skills, development tools such as Gradle or Playwright, and external search or documentation.Source: C. RQ3: What agentic instructions do… · p. 10
  • Context appears in 84.7% of workflows: repository context in 70.1%, domain context and execution environment in 34.0% each, and historical context in 8.3%.These categories supply repository or task background, subject-matter rules, runtime details, and information from earlier runs or decisions.Source: C. RQ3: What agentic instructions do… · pp. 10–11
  • Evaluation instructions appear in 70.1% of workflows, including rule conformance in 47.2%, risk assessment in 29.9%, execution checks in 25.7%, and evidence-credibility checks in 11.1%.Examples include checking outputs against protocols, classifying change severity, running tests before and after edits, and assessing source authority and recency.Source: C. RQ3: What agentic instructions do… · p. 11
  • Examples appear in 68.1% of workflows: output templates in 61.1%, technical snippets in 30.2%, and worked cases in 19.4%.Templates specify report formats, technical snippets show sample commands or code, and worked cases distinguish acceptable, unacceptable, or borderline outcomes.Source: C. RQ3: What agentic instructions do… · p. 11
  • Communication instructions occur in 63.9% of workflows, including role assignment in 46.9%, automation disclosure in 25.0%, and audience voice in 20.1%.They define the expertise expected, require automated authorship or limitations to be disclosed, or specify tone and audience.Source: C. RQ3: What agentic instructions do… · p. 11
  • Safety instructions appear in 25.0% of workflows; prompt-injection defense appears in 9.4%, execution isolation in 13.9%, confidentiality in 7.3%, and vulnerability handling in 6.3%.Prompt-injection defenses treat issue, comment, code, or linked content as untrusted data rather than instructions; isolation can restrict commands and network access.Source: C. RQ3: What agentic instructions do… · p. 12

D. RQ4: Can LLMs assign instruction labels in GitHub Agentic Workflows?

The study evaluates LLMs as multi-label classifiers for gh-aw instructions against resolved human labels. One-shot Gemini records the highest observed F1, while its disagreements show that instruction categories can be interpreted beyond explicit text.

Motivation

  • The study treats automatic labeling as multi-label classification because each Markdown body may receive zero, one, or several category-subcategory labels.The motivation is to assess whether LLMs can apply the manually developed taxonomy as gh-aw repositories grow.Source: D. RQ4: Can LLMs assign instruction… · p. 12

Approach

  • The evaluation compares three models with resolved labels on 288 files, using a shared prompt and zero-shot, one-shot, three-shot, and five-shot settings.Each request contained one Markdown body without YAML frontmatter; examples excluded the target and identical-body copies.Source: D. RQ4: Can LLMs assign instruction… · p. 12

Results

  • Gemini has the highest F1 at every shot setting, followed by GLM and GPT-OSS, but more examples do not benefit every model equally.GLM peaks at three shots (0.810), while GPT-OSS rises from 0.722 at zero shots to 0.738 at five; observed differences do not establish statistical superiority across repeated runs.Source: D. RQ4: Can LLMs assign instruction… · pp. 12–13
  • 0.818 F1, 86.81% subcategory agreement, and Cohen’s κ = 0.715 are achieved by one-shot Gemini against resolved human labels.Table IV summarizes classification performance; these are the highest observed F1 results.Source: D. RQ4: Can LLMs assign instruction… · p. 12, Table IV · p. 13
  • Gemini assigns Domain Context to 174 files versus 98 in the reference, with 83 false positives and seven false negatives.Domain Context is the largest disagreement; Rule Conformance differs on 79 files and Control Signal on 70.Source: D. RQ4: Can LLMs assign instruction… · p. 13
  • Gemini’s Domain Context errors are predominantly additional assignments, suggesting it may read background knowledge into an instruction.The example distinguishes text that specifies what to search for from text that supplies domain knowledge; it does not establish the model’s internal reasoning or show every additional label is unsupported.Source: D. RQ4: Can LLMs assign instruction… · p. 13

V. DISCUSSION AND IMPLICATIONS

The section introduces implications for developers and researchers and points to a repository workflow example.

V. DISCUSSION AND IMPLICATIONS

  • The discussion is framed around implications for developers and researchers.Source: §V · p. 13
  • A linked example is the check-emulator-updates workflow file.Source: §V · p. 13

A. For Developers

The discussion recommends treating copied or referenced workflow sources as evolving dependencies, considering operational guidance beyond task descriptions, and separating enforceable configuration limits from body instructions.

For Developers

  • 78.2% of files with at least 120 days of activity still receive updates in month 4, supporting treatment of copied or referenced workflow sources as evolving dependencies.Among 365 projects, 108 (29.6%) reference Markdown sources from another repository; size-normalized churn falls after month 1, and one-third of content-changing events leave line count unchanged.Source: A. For Developers · p. 13
  • Developers can consider resource budgets, prompt-injection defenses for workflows reading external content, and cross-run memory according to each workflow’s needs.The taxonomy documents these options but does not establish their effects on cost, security, or repeated work.Source: A. For Developers · p. 13
  • Supported configuration controls can enforce firm limits, while body instructions explain task-specific conditions and expectations.When a rule appears in both regions, the discussion recommends keeping them consistent as the workflow evolves.Source: A. For Developers · pp. 13–14

B. For Researchers

The discussion proposes controlled studies of pictographic symbols and instruction patterns, measuring agent performance and resource use rather than assuming Markdown guidance is uniformly useful.

For Researchers

  • 47.4% of files contain pictographic symbols, but their effects on task completion, token use, and interpretation errors remain an open research question.The discussion proposes controlled comparisons of icon-based and plain-text variants under matched task, model, and tokenizer settings.Source: B. For Researchers · p. 14
  • 89.9% of workflows provide resources, 68.1% examples, 63.9% communication guidance, and 19.4% a worked case, motivating controlled ablations of instruction patterns.Proposed outcomes include task completion, correctness, tool calls, runtime, and token usage; cited work reports additional context can increase exploration and inference cost by more than 20% with mixed task-success effects.Source: B. For Researchers · p. 14

VI. THREATS TO VALIDITY

The study addresses validity risks in coding, measurement, inference, and corpus coverage through calibration, explicit comparison choices, and bounded interpretation. Its findings are constrained by the selected files and projects, including workflows already observed to change over at least 120 days.

Internal Validity

  • Instruction labels involve coder interpretation; two authors calibrated on 30 files, used shared definitions, assessed agreement, and reconciled disagreements against Markdown bodies.For automatic labeling, target files and identical-body copies were excluded from examples, with prompts and examples held consistent across models.Source: §VI · p. 14

Construct Validity

  • Full-body FRE scores include code, Markdown syntax, and template expressions despite FRE being designed for prose, so they are descriptive outputs rather than validated comprehension measures.The study also required explicit textual support for instruction labels and left events with unreliable region comparisons unclassified.Source: §VI · p. 14

Conclusion Validity

  • Declining repository activity means fewer Markdown commits alone cannot establish workflow-specific slowdown; the study compared repository commits and used paired-project tests with Holm correction.Model rankings are reported as observed, not statistically superior, and GPT-OSS recovery requests changed generation settings, so those comparisons do not use identical reasoning budgets.Source: §VI · p. 14

External Validity

  • GitHub search limits may restrict corpus coverage; retrieval and sampling steps reduce missing results and file-selection bias but do not establish generalizability beyond the selection criteria.RQ2 is limited to 43 projects in the dataset that contain files meeting its observation-period requirement.Source: §VI · pp. 14–15
  • The RQ2 sample requires observed changes spanning at least 120 days, so its maintenance patterns do not establish how often all gh-aw workflows remain static after creation.This restriction selects already-maintained files.Source: §VI · p. 15

A. GitHub Actions Workflows

Prior work examines GitHub Actions maintenance, AI contributions to CI, and generated workflow configurations. This study instead examines Markdown sources specifying recurring tasks carried out by agents and separates instruction edits from execution-configuration changes.

A. GitHub Actions Workflows

  • Studies of GitHub Actions report continued workflow maintenance, often involving bug fixes, CI/CD improvements, small edits, and task specification or configuration,.Other work examines workflow reuse across repositories.Source: A. GitHub Actions Workflows · p. 15
  • Related AI-and-CI research examines review actions, agent-authored fixes, and generated YAML, including missing steps, misinterpreted requirements, and schema violations,,,, [58].Agent-authored fixes were faster in one study, while human developers remained involved in resolving failures.Source: A. GitHub Actions Workflows · p. 15
  • Unlike studies of agents creating or editing YAML pipelines, this work studies gh-aw Markdown sources and distinguishes execution-configuration changes from changes to agent instructions.These Markdown files specify recurring tasks performed by agents.Source: A. GitHub Actions Workflows · p. 15

B. Agent Instructions and Context Files

The conclusion reports that gh-aw workflows combine maintained instructions and configuration, while prior agent-context studies characterize guidance and test its effects. The paper positions its taxonomy and labeling evaluation as groundwork for later outcome-focused research.

B. Agent Instructions and Context Files

  • Prior studies characterize agent-facing Markdown and skill guidance [12], [13],,, while experiments report mixed effects of context files on runtime, token use, task success, and inference costs [52], [60].The studies caution against treating the presence of instructions as evidence of effectiveness.Source: B. Agent Instructions and Context Files · p. 15
  • GHAW-H links gh-aw Markdown histories to compiled YAML snapshots; this study complements it with a separately collected dataset, maintenance analysis, taxonomy, and automatic-labeling evaluation..Source: B. Agent Instructions and Context Files · p. 15
  • In gh-aw, instructions and execution configuration jointly specify workflows triggered by repository events or schedules,, distinguishing them from general context files or skills.The study evaluates whether LLMs identify instruction types, not whether instructions improve execution outcomes.Source: B. Agent Instructions and Context Files · p. 15

VIII. CONCLUSION

  • Among 348 files observed for at least 120 days, 78.2% received updates in month four; size-normalized churn fell after month one, while instruction bodies accounted for most changed lines.The share of repository commits devoted to workflow maintenance did not change significantly between adjacent months.Source: §VIII · p. 15
  • The taxonomy covers tasks, outputs, steps, constraints, context, and resources; safety instructions and resource budgets are less often explicit, without showing that runtime protections are absent.The authors conclude that maintained natural-language instructions warrant explicit attention during workflow review.Source: §VIII · p. 15
  • Future work should test how instruction patterns and revisions relate to task success, reliability, cost, and safety, and use those evaluations to inform validation and regression testing.The dataset and taxonomy characterize current specifications as a foundation for that work.Source: §VIII · p. 16
Loading 2609.27263v1…