PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Maintenance and Co-evolution of Agent Plugins: An Empirical Study of Claude Code Plugin Marketplaces

Ahmed Hereiz, Yingzhe Lyu, Hao Li, Bram Adams, Ahmed E. Hassan

arXiv:2608.28497v1cs.SEcs.AI

TL;DR

The study addresses limited empirical evidence about plugin marketplace structure, maintenance, and component co-evolution, including risks from failing to update co-evolved components together. It analyzes Claude Code plugin marketplaces and finds rapid expansion, Software Engineering dominance, and substantial multi-component plugin composition.

  • Problem

    Plugin marketplace structure, sustained maintenance, and co-evolution remain empirically unexplored, including whether multi-component plugins require coordinated updates to avoid inconsistencies and silent defects.

  • Method

    The paper presents a large-scale empirical study using three studies to characterize marketplace structure, development patterns, and component co-evolution.

  • Results

    84.5% of repositories were created in the six months after launch, plugin-touching commit activity grew 8.8×, Software Engineering plugins comprised 61.3%, and 34.4% combined multiple component types.

  • Takeaways & Limitations

    The marketplaces are young and accelerating, while multi-component plugins make coordinated component maintenance relevant to avoiding inconsistencies and silent defects.

  • Takeaways & Limitations

    The study treats ten stars as a noise-removal threshold, creating a tradeoff between marketplace coverage and noise, and calls for replication before generalizing to AI-native software broadly.

Abstract

from arXiv · show

AI coding agents, software tools that automate development tasks through reasoning and tool use, are increasingly extended through plugin marketplaces, yet the structure, maintenance, and co-evolution dynamics of these emerging repositories remain empirically unexplored. Unlike traditional software packages that deliver functionality through source code, agent plugins deliver functionality through a combination of natural-language instruction files, scripts, and configuration files, raising the question of whether these plugins are maintained artifacts that co-evolve across components, or one-off artifacts that developers write once and do not need to revisit. To study the maintenance and co-evolution of agent plugins, we conduct an empirical study of 1,926 repositories hosting Claude Code plugin marketplaces, analyzing 8,351 plugins and 77,773 commits across 2,018 marketplaces. We find that the marketplace is expanding rapidly, plugin-touching commit activity growing 8.8x over six months after the October 2025 launch, and plugins targeting Software Engineering tasks accounting for 61.3% of all plugins. Plugin development is predominantly feature-driven, with feature commits occurring at more than twice the rate of conventional open-source software (OSS) (39.6% vs. 17.2%). Claude co-authors 34.9% of all commits, and four commit types (docs, perf, style, and refactor) carry substantially different meanings in plugin repositories than in traditional software. Most component types evolve independently, but within skills directories, natural-language instruction files and implementation scripts co-evolve at above-chance rates, with 78% of co-changes being functionally coupled, representing a new class of maintenance dependency not observed in traditional software engineering.

1 Introduction

This study examines Claude Code agent plugins as a new artifact class combining natural-language instructions with scripts and configuration files. It establishes an empirical baseline for their marketplace structure, maintenance, and co-evolution, including whether traditional software-engineering methods transfer.

  • Agent plugins package reusable behavior for Claude Code through skills, commands, hooks, and MCP servers distributed in shared marketplaces.
  • Natural-language instruction files coexist with source-code artifacts as primary distributed components of agent plugins.
  • Updating co-evolved components separately can leave multi-component plugins inconsistent and introduce silent defects.
  • The paper asks whether source-code repository maintenance methods transfer to repositories where Claude reads Markdown instructions alongside scripts and configuration files.
  • The dataset covers 1,926 repositories, 8,351 plugins, and 77,773 commits to characterize plugin maintenance and component co-evolution.

RQ1: What are the structural and maintenance characteristics of the Claude Code plugin marketplaces?

The marketplace is expanding and actively maintained, with software-engineering plugins dominant and many plugins combining component types. Development practices differ from conventional OSS, while most components evolve independently except for functionally coupled Script–Markdown changes within skills.

  • 8.8× plugin-touching commit growth occurred within six months, while Software Engineering plugins accounted for 61.3% of plugins.
  • 34.4% of plugins combine multiple component types, making component co-evolution a relevant maintenance question.
  • Docs, performance, style, and refactor commits carry different meanings in agent plugin repositories than under conventional commit categories.
  • Most component types evolve independently, but 78% of Script–Markdown co-changes within skills are functionally coupled.
  • The study releases a dataset and establishes a first empirical baseline for this AI-native artifact class.

2 Background and Related Work

Claude Code plugins extend an agent through multiple component types that are loaded, invoked, or triggered during tasks. Prior work studied related natural-language artifacts and package ecosystems, but this study focuses on GitHub-hosted, multi-component agent plugins with commit histories.

  • Claude Code supports skills, agents, commands, hooks, MCP servers, LSP servers, and settings as distinct plugin components.
  • Skills and agents are model-invoked, whereas hooks, MCP servers, LSP servers, and settings operate through events, configuration, or external connections.
  • During a task, Claude can load skills, run hooks, delegate to isolated agents, and call MCP tools before reporting results.
  • Marketplaces catalog plugins through marketplace.json; users register a marketplace and install individual plugins for reuse across projects.
  • This study extends software-ecosystem research to repositories combining natural-language instructions, executable scripts, and configuration files.
  • Unlike AI assistant stores without versioned histories, Claude Code plugins enable commit-level analysis of development and component co-evolution.

3 Data Collection

The data pipeline discovers Claude Code marketplaces on GitHub, filters repositories and plugin declarations, then extracts plugin-touching commit histories. The resulting dataset contains 2,018 marketplaces, 8,351 valid plugins, and 77,773 plugin-touching commits across 1,926 repositories.

  • The pipeline performs repository discovery and filtering followed by marketplace parsing and data extraction.
  • GitHub Code Search identified repositories containing the required .claude-plugin/marketplace.json manifest, using divide-and-conquer to bypass the 1,000-result query limit.
  • 10,646 unique repositories remained after deduplicating 11,254 raw references, and a 10-star threshold retained community-adopted repositories.
  • 2,018 marketplaces yielded 9,565 raw plugins, which filtering reduced to 8,351 valid locally resolvable plugins.
  • The analysis extracts complete Git histories and records metadata and file changes for 77,773 plugin-touching commits across 1,926 repositories.
  • The median repository contains 13 plugin-touching commits, and only 6.0% contain a single commit, indicating ongoing development for most repositories.

4 RQ1: What Are the Structural and Maintenance Characteristics of the Claude Code Plugin Marketplaces?

Claude Code plugin marketplaces are rapidly expanding ecosystems built from diverse natural-language, executable, and configuration artifacts. Their structure is often simple, but component growth and commit activity indicate sustained, accelerating development.

  • Functionality: 61.3% of plugins target Software Engineering tasks, making it the dominant functionality category.
  • Growth: 84.5% of marketplace repositories were created during the six months after the October 2025 launch.Repository creation was 2.1× higher in Jan–Mar 2026 than in Oct–Dec 2025, with p < 0.001 and Cliff’s δ = 1.0.
  • Components: 22× growth brought skill instances from 1,776 at launch to 39,287 by March 2026, exceeding all other component types combined.Commands grew 3.7× and agents 3.9× over the same period; skills primarily contain natural-language instruction files.
  • Structure: 56.4% of plugins use one component type, while 34.4% combine two or more types and 49.8% contain multiple component files.Marketplace sizes and plugin structures are heavily right-skewed; three community aggregators account for 12.1% of all plugins.
  • File composition: Commands, agents, and skills are predominantly Markdown, whereas MCP, LSP, and settings components are exclusively JSON.Hooks are the most heterogeneous component type, combining shell, Python, JavaScript, and JSON files.
  • Maintenance activity: Plugin-touching commit activity grew 8.8× from 2,923 commits in October 2025 to 25,618 in March 2026, with no plateau.The upward trend was statistically significant (p = 0.003), with estimated growth of roughly 4,550 additional commits per month.

5 RQ2: How Are Plugins Developed and Maintained?

Agent plugin repositories are actively maintained, but conventional commit labels often describe natural-language behavior rather than source-code changes. Feature work dominates, while AI co-authorship is substantial and varies by commit type.

  • Commit semantics: 74% of docs commits modify instruction files Claude reads at inference time rather than human-readable documentation.After reclassification, only 1.7% of commits remain docs, while many become fix or feat by function.
  • Commit semantics: 49.4% of fix commits correct wrong behavioral text in instruction files rather than code bugs.Typical defects include incorrect tool references, workflow steps, and examples.
  • Development patterns: 39.6% of commits are feat, more than double the 17.2% share in conventional OSS.Feat, fix, and chore together account for 86.5% of classified commits.
  • Development patterns: 21.0% of commits are chore, showing developers return to maintain plugins after initial creation.Version increments and manifest updates comprise major chore activities.
  • Human-AI collaboration: 35.5% of plugin commits are co-authored by an AI coding agent, with Claude accounting for 34.9%.Co-authorship ranges from 40.1% for perf commits to 16.2% for revert commits.
  • Commit semantics: Docs, perf, style, and refactor acquire plugin-specific meanings because text edits can change runtime behavior, prompt efficiency, wording, or schemas.Perf and refactor retain their labels while targeting natural-language instructions; docs and style are often reclassified by developer intent.

6 RQ3: How Do Plugin Components Co-Evolve?

Plugin components usually co-change because skills are widespread, not because most component pairs exhibit specific coupling. Stronger dependencies appear within skills, where scripts and Markdown instructions frequently require synchronized updates.

  • Inter-component co-evolution: Agents–commands is the only component pair with above-chance co-change, reaching Lift 1.40 (p< 0.05).Agent definitions and commands reference one another, so renaming either requires corresponding updates.
  • Intra-component co-evolution: Scripts and Markdown files inside skills/ co-change above chance, with Lift values of 1.37–1.58.Confidence for script–Markdown pairs ranges from 55–64%, while hooks show no cross-file coupling.
  • Intra-component co-evolution: 78% of sampled script–Markdown co-changes are functionally coupled, driven mainly by interface and internal-logic changes.These changes require SKILL.md updates so agents can invoke scripts correctly.
  • Intra-component co-evolution: Variable and version synchronization can propagate one script change across many instruction files, including 24 SKILL.md files in one pull request.Renamed paths, filenames, and configuration formats similarly force matching instruction edits.
  • Inter-component co-evolution: Skills co-change with every other component type in at least 43.2% of pull requests, but most Lift values remain below 1.Skills’ prevalence explains much of this co-change frequency.

7 Implications

The findings imply that AI-facing Markdown requires software-maintenance analyses and consistency checks tailored to runtime behavior. The study also identifies scope boundaries around classifier transfer, lifecycle trends, and marketplace breadth.

  • Implications for researchers: AI-facing Markdown should be treated as a first-class software artifact because stale instructions can cause incorrect runtime invocations.Researchers are encouraged to develop semantic differencing, instruction linting, and consistency metrics.
  • Implications for researchers: Conventional OSS commit classifiers should not be applied directly to plugin repositories without semantic validation.Docs and perf labels frequently describe runtime AI behavior and prompt efficiency rather than conventional documentation or algorithmic speed.
  • Limitations and open questions: Whether these semantic shifts and structural patterns generalize beyond Claude Code remains untested.The paper calls for cross-platform comparisons and longitudinal tracking as other AI coding agent marketplaces mature.
  • Implications for researchers: The study’s human–AI collaboration evidence supports analyzing delegation and review dynamics, but lifecycle changes in co-authorship remain uncharacterized.Whether early high feat co-authorship shifts toward fix and perf co-authorship as plugins mature is unresolved.
  • Implications for plugin developers: Plugin developers should verify paired SKILL.md updates after script changes because 78% of script–Markdown co-changes are functionally coupled.Stale instructions may reference removed flags, renamed identifiers, or unsupported invocation patterns.
  • Marketplace scope: Raw plugin counts overstate supply breadth because 38.7% of normalized plugin names appear more than once, often through aggregator repositories.Software Engineering plugins currently represent 61.3% of all plugins, but this distribution may change as the marketplace grows.

8 Threats to Validity

The study’s validity is constrained by incomplete repository discovery, snapshot timing, platform-specific scope, and uncertainty in manual, LLM-based, and heuristic classifications. Sensitivity analyses support the robustness of qualitative component and commit-type orderings, but several findings remain bounded by these data and measurement choices.

  • Data coverage: Public-only GitHub indexing omits private or deleted repositories and cannot recover content that the API does not index.Recursive prefix partitioning mitigates the API cap but does not restore unindexed content.
  • Sensitivity analysis: Across 5-, 10-, and 25-star thresholds, Skills remains dominant, feat remains dominant, and component-type ordering stays consistent.This threshold analysis supports the robustness of the qualitative findings while retaining the coverage-versus-noise tradeoff of the ten-star choice.
  • Temporal scope: The April 2026 dataset is a snapshot of a rapidly evolving ecosystem, with a median repository age of only 80 days.Growth rates and adoption patterns may not generalize to later periods because many repositories lacked long commit histories.
  • Generalizability: Findings are specific to Claude Code marketplaces, and their generalization to Cursor, Copilot, Gemini, or broader AI-native ecosystems remains untested.The authors recommend replication before drawing conclusions about AI-native software in general.
  • Manual labeling: Manual coding involves subjective judgments about developer intent and boundaries between interface, logic, instruction-content, and behavioral changes.Inter-rater agreement was substantial, with κ=0.671 for 700 commits and κ=0.74 for 64 pull requests, but disagreements still required reconciliation.
  • LLM-based classification: LLM classifications may be sensitive to training data, prompts, model versions, and nondeterministic runs, despite validation against human labels.The reported classifications reflect a single run on the final model versions; validation achieved κ=0.707 on two independent 385-commit samples.
  • Agent detection: The 34.9% Claude attribution rate is a lower bound because developers may commit AI-generated content without adding detectable co-author signals.Detection relies on co-author trailers, email addresses, author names, and branch prefixes.

9 Conclusion

This study provides the first large-scale empirical characterization of Claude Code plugin marketplaces, showing rapid growth, feature-driven and AI-assisted development, and component-specific maintenance behavior. Most components evolve independently, while Script–Markdown co-evolution within skills forms a distinct maintenance dependency whose conventional software labels may mislead cross-ecosystem comparisons.

  • Study scope: 1,926 repositories, 8,351 plugins, and 77,773 commits underpin the first large-scale empirical study of an AI coding agent plugin marketplace.The study examines marketplace structure, development patterns, and component co-evolution.
  • Marketplace growth: 84.5% of repositories were created in the six months after launch, while plugin-touching commit activity grew 8.8× over the same period.These findings characterize the marketplaces as young and accelerating.
  • Marketplace composition: 61.3% of plugins target Software Engineering, and 34.4% combine multiple component types.Examples include skills paired with hooks or agents.
  • Development patterns: Feature development dominates plugin evolution, with feat commits at 39.6%, and Claude co-authors 34.9% of all commits.The results describe development as heavily AI-assisted.
  • Interpretation boundary: Cross-ecosystem comparisons using conventional commit labels risk incorrect conclusions because four CCS types carry different meanings in plugin repositories.The affected types are docs, perf, style, and refactor; natural-language instruction files are primary artifacts.
  • Component co-evolution: Most component pairs evolve independently, but Script–Markdown co-changes within skills are functionally coupled and represent a maintenance dependency without a traditional software analog.The Agents–Commands pair is the only other component pair reported to co-change above chance, with Lift = 1.40.

A Plugin Classification Prompt

The plugin-classification prompt directs a model to assign each plugin exactly one subcategory within six major categories using its name and description. It requires strict JSON output containing the selected category identifiers, names, and a brief English rationale.

  • Classification task: The prompt classifies each plugin into exactly one of six major categories and their corresponding subcategories.The definitions span software engineering, search, communication, data processing, content generation, and utilities.
  • Taxonomy: Software Engineering subcategories include code generation, debugging and analysis, version control, and infrastructure.Examples cover source-code generation, static analysis, Git workflows, cloud services, containers, and deployment.
  • Taxonomy: Other major categories cover web and academic search, live data streams, team communication, document systems, and task management.The prompt also includes data processing, mathematics, visualization, image or text generation, audio or video, and local utilities.
  • Classification task: The model must analyze the plugin name and description deeply before selecting the best-fitting subcategory.The input format supplies a plugin name and plugin description.
  • Output format: The required output is strict JSON with major and subcategory identifiers and names plus a brief English reasoning field.The prompt also explicitly prohibits conversational text.

B Conventional Commit Type Classification Prompt

The conventional-commit prompt classifies each plugin-repository commit into exactly one standardized type using the message as the primary signal and changed-file context for disambiguation. It returns a strict JSON result with the predicted type, rationale, and confidence.

  • Classification task: Each commit is assigned exactly one type from a 12-category taxonomy.The categories include feat, fix, docs, style, refactor, perf, test, build, ci, chore, revert, and other.
  • Decision procedure: Commit messages provide the primary classification signal, while changed files and metadata disambiguate documentation, testing, CI/build, refactoring, and chores.The prompt instructs the classifier not to invent missing details and to lower confidence when evidence is weak.
  • Label definitions: The prompt defines feat as a new feature, fix as a bug fix, docs as documentation-only changes, and style as changes that do not affect code meaning.It also specifies refactor, perf, test, build, ci, chore, revert, and other categories.
  • Output format: The output must be valid JSON containing an integer id, one output type, a brief reason, and a confidence score from 1 to 10.Per-commit context includes the message, components touched, and file counts.

C Commit Re-classification Prompt

The re-classification prompt revisits docs, ci, and style labels by reading diffs and judging whether changes alter agent behavior, pipeline automation, or only presentation. It returns a strict JSON record containing the corrected type, movement flag, rationale, and confidence.

  • General procedure: The rubric reclassifies commits originally labeled docs, ci, or style into their true Conventional Commit types using the diff.The prompt emphasizes reading the diff rather than relying on the commit message.
  • File-role distinction: AI-read files contain runtime instructions whose content changes agent behavior, whereas human-read files are ignored by the agent.AI-read examples include SKILL.md, agents/*.md, commands/*.md, skills/**, and references/*.md.
  • Reclassification rules: Changes adding or strengthening agent directives are feat, correcting existing directives is fix, and version bumps or metadata reorganization are chore.These rules apply to instructions, triggers, script-invocation procedures, and marketplace entries.
  • Docs rubric: A docs label remains docs for human-facing changes, inert metadata, or informational links, even when an AI-read file is touched.The rubric treats source-attribution and bare-reference link edits as informational documentation changes.
  • CI rubric: A ci label is retained only for actual workflow or CI-automation changes; CI-themed agent instructions are feat or fix instead.A workflow file changed remains ci even when plugin files are also modified.
  • Style rubric: Style denotes cosmetic formatting only, while wording changes that alter agent behavior are feat or fix.The decision rule asks whether the agent would behave differently after the change.
  • Output format: The output is strict JSON with true_type, moved, reason, and confidence fields, alongside the original label and file-role context.The prompt requires one of the allowed commit types and a short explanation.

D Script–Markdown Coupling Classification Prompt

The prompt classifies script–Markdown pull-request changes by first determining whether the edits are coupled, then assigning exactly one mechanism label when they are. It prioritizes observable diff relationships and applies explicit precedence rules for interface, value, repository, and internal-logic changes.

  • Classification procedure: The classifier reads the full script and Markdown diffs rather than relying on pull-request messages.Each pull request is presented to the model with its full script and Markdown diff.
  • Step 1: Determine coupling: Coupling requires the Markdown edit to closely reflect the specific script change; merely modifying both files is insufficient.Unrelated, cosmetic, pre-existing-behavior, or cross-subskill edits are classified as no_coupling.
  • Step 2: Select the mechanism: Coupled changes are split into surface changes, which preserve capabilities, and capability or behavior changes.Surface changes concern quoted values, files, paths, binaries, or configuration formats; behavior changes are resolved using Markdown signals.
  • Step 2: Select the mechanism: The four coupled labels are value_version_sync, repo_restructuring, interface_change, and internal_logic_change.Value synchronization covers input values such as versions, defaults, model identifiers, environment-variable names, and the skill's own name; repository restructuring covers files, paths, binaries, and formats.
  • Precedence rules: Input-surface changes take precedence: new or removed commands or flags outrank renamed input values, which outrank renamed files, paths, binaries, or formats.Only when none of these applies does the prompt assign internal_logic_change for output-contract changes, auto-detection, validation, or stub-to-real implementation changes.
  • Edge cases: Output-field renames, optional arguments created by auto-detection, and new output fields are classified as internal_logic_change rather than input-surface or value changes.The prompt distinguishes input values passed by the agent from output-contract changes documented in Markdown.
Loading 2608.28497v1…