MCRI: A Four-Dimensional Framework for Analyzing and Evaluating Agent Skills

1 Introduction

• Skills are becoming reusable building blocks in composite agent systems, but no structured perspective has been established for analyzing, organizing, and evaluating them.The introduction notes that LobeHub’s marketplace had collected more than 290,000 skills at the time of writing.Source: §1 · p. 1
• The paper asks how skills can be characterized, whether content-based scores capture ecosystem outcomes, and whether those scores can predict downstream performance and support selection.Source: §1 · pp. 1–2
• The paper proposes MCRI to characterize skills through information gain and behavioral constraint, and frames skills as intermediate between natural language and programming languages.Information gain adds external task context, knowledge, tools, examples, or resources; behavioral constraints reduce instability through procedures and boundaries.Source: §1 · p. 2
• 17.7, 22.8, and 19.6 percentile-point gains over the strongest baselines in top-1 downstream-performance rank occur on BigCodeBench, BFCL-Fundamental, and Mind2Web, respectively.The skill selected by MCRI-Eval ranks at the 78.0th percentile on BigCodeBench, 74.6 on BFCL-Fundamental, and 99.3 on Mind2Web.Source: §1 · p. 2
• MCRI-Eval scores are positively associated with ecosystem signals and achieve the highest downstream ranking agreement among the evaluated methods.The experiments compare associations with stars, downloads, and installs, then assess ranking agreement and top-k selection across downstream domains.Source: §1 · p. 2
• Full reproduction is estimated to require approximately 0.4 billion tokens, and the preprint source package omits datasets, code, logs, outputs, analyses, and test code.The paper says Appendix F describes the availability of these resources.Source: §1 · p. 3

2.1 Large Language Models and Agents Perspective

• LLM benchmarks assess model knowledge and reasoning, while agent evaluations focus on interactive behavior; neither directly evaluates skill content.Skills are reusable, pluggable artifacts, while agent outcomes also depend on host architecture, planning, and tool orchestration.Source: §2.1 · p. 3

2.2 Prompt Engineering Perspective

• Prompt-oriented evaluation methods do not transfer directly because skills package reusable capabilities with tools, resources, preconditions, and exception branches.Prompt evaluation generally assumes a flat input for a specific task instance.Source: §2.2 · p. 3

2.3 Software Engineering Perspective

• Software-quality models such as ISO/IEC 25010 offer useful context, but their evaluation objects and usage assumptions differ from natural-language capability modules embedded in agents.Source: §2.3 · p. 3

3.1 Where Does a Skill Sit Between Prompts and Code?

• Skills selectively constrain determinate behaviors while leaving other choices to model reasoning, occupying an intermediate position between natural language and programs.Natural-language requests leave procedures and outputs unspecified, whereas programs explicitly specify operations and branches.Source: §3.1 · pp. 3–4

3.2.1 Information Gain

• Information gain describes how a skill supplies task-relevant external information that expands the model’s effective knowledge and makes an appropriate behavioral path available.A PPTX-generation skill may provide python-pptx usage and template-manipulation code; model reasoning alone cannot recover information absent from its parameters.Source: §3.2.1 · p. 4

3.2.2 Behavioral Constraint

• Behavioral constraint describes how prescribed procedures compress permissible behavior and increase consistency in selecting a target path across repeated invocations.The example skill prescribes preprocessing, analysis, and a report format.Source: §3.2.2 · pp. 4–5

3.3 MCRI Framework: An Analytical Framework for Skills

• MCRI defines Metadata, Constraints, Resources, and Instructions as functional perspectives on invocation, permissible behavior, external information, and execution organization.The perspectives are not mutually exclusive: one artifact may act as a resource, instruction, and constraint for different functional reasons.Source: §3.3 · p. 5
• Removing Resources primarily weakens association with stars, while removing Instructions most strongly affects all-time installs, supporting functional differentiation without requiring orthogonal dimensions.Source: §3.3 · p. 5
• MCRI scores are proxies, not substitutes for downstream validation; they provide a structured pre-execution feature space to narrow candidates for execution-based evaluation.The framework focuses on identifiability, constraint sufficiency, resource relevance, and instruction executability before reasoning and tool-calling costs are incurred.Source: §3.3 · p. 5

4.1 From MCRI to MCRI-Eval

• MCRI-Eval converts the framework’s four layers into separate rubrics for task-description clarity, behavioral-constraint sufficiency, information gain, and instruction and interface completeness.The evaluation prompt supplies the framework definition and scoring rubric.Source: §4.1 · p. 6

4.2.1 Task Definition

• The community-signal analysis fits a linear model from four-dimensional skill scores to a target metric, then compares predicted and observed rankings; it also tests correlations and classification of signal extremes.The extreme groups are the top and bottom 15% of skills.Source: §4.2.1 · p. 6

4.2.2 Experimental Setup

• The community analysis uses 63,812 OpenClaw skills and four scoring models, with average daily downloads and selective filtering addressing exposure, availability, and long-tail concerns.Samples are randomly split into equally sized training and test sets, with the test set reserved for final evaluation.Source: §4.2.2 · p. 7

4.2.3 Results

• MCRI scores are positively associated with popularity across signals, models, and thresholds; correlations rise with stricter filtering, and all-time installs yield the most stable results.Removing Resources reduces the stars correlation from ρ = 0.13–0.21 to ρ = 0.04–0.06 for three models; removing Instructions most affects installs. Extreme-class performance follows the correlation-strength ordering.Source: §4.2.3 · pp. 7–8

4.3.1 Task Definition

• Downstream tests inject matched skills into benchmark executions; 83, 133, and 75 skills matched BigCodeBench, BFCL-Fundamental, and Mind2Web, producing 58,275 model calls.The comparison includes a name-and-summary Baseline, a Full-text Holistic Baseline, and MCRI-Eval; scoring and generation conditions are controlled as far as possible.Source: §4.3.1 · p. 9

4.3.2 Experimental Setup

• The downstream suites cover code generation, function calling, and multi-step web interaction; evaluation reports benchmark performance and each selected skill’s percentile within its candidate set.BigCodeBench uses test pass rate, BFCL-Fundamental function-calling success rate, and Mind2Web Step Success Rate.Source: §4.3.2 · p. 10

4.3.3 Experimental Results

• MCRI-Eval achieves the highest downstream ranking agreement across all three domains; top-1 selections improve candidate-pool percentile scores by 17.68 on BigCodeBench, 22.80 on BFCL-Fundamental, and 19.59 on Mind2Web.The Mind2Web top-1 selection has a Step Success Rate of 57.39%, versus 56.23% for the strongest baseline.Source: §4.3.3 · pp. 10–11, Table 6 · p. 11, Table 5 · p. 11

4.3.4 Stability Analysis

• MCRI-Eval has the lowest overall score standard deviation in all three domains, reducing it by 67.5% versus the Baseline and 74.3% versus the Full-text Holistic Baseline on average.Dimension-score standard deviations range from 0.103 to 0.163.Source: §4.3.4 · pp. 11–12, Table 9 · p. 12

5 Discussion and Conclusion

• No task-independent quality target exists because a skill’s value can vary across tasks, models, and user preferences.The paper instead examines whether MCRI scores exhibit expected associations with external signals.Source: §5 · pp. 12–13
• Community indicators measure popularity, while downstream performance is closer to task-specific utility and remains sensitive to task–skill alignment, sample size, and evaluation methodology.Source: §5 · p. 13
• MCRI provides a mechanism-driven, operationalizable, extensible lens for pre-execution skill selection, but does not fully define skill quality.The paper evaluates its utility through community indicators and downstream task performance.Source: §5 · pp. 12–13

A Additional Related Work

• Related work spans execution-based evaluation [9] [11] [18] [20], skill generation [15], surveys [4], and security auditing [7] [6]; the identified studies were available as arXiv preprints.Source: A Additional Related Work · p. 15
• These lines of work leave skill mechanisms, theoretically grounded quality criteria, and general pre-execution evaluation comparatively underexplored.Source: A Additional Related Work · p. 15

• The OpenClaw skill Hub API collection contained 63,820 records, with 63,812 retained after invalid or empty skills were removed.Records include skill metadata and observable signals such as stars, downloads, installs, comments, and versions.Source: B Dataset Construction and Filtering · p. 15
• Raw download and installation counts are affected by time since publication and platform exposure, so the analysis uses downloads per day.Source: B Dataset Construction and Filtering · p. 15
• The correlation analysis focuses on the top 3%, 1%, and 0.5% subsets, with N = 1,823, 606, and 305, respectively.The experimental resource package contains the filtered N = 1,823 subset and exact definitions for the N = 606 and N = 305 thresholds.Source: B Dataset Construction and Filtering · p. 15

• The supplementary evaluations reuse the BFCL-Fundamental, BigCodeBench, and Mind2Web settings from the main experiments.They introduce no new task set or evaluation metric.Source: C Supplementary Experiments · p. 15
• One supplementary evaluation compares MCRI-Eval with multi-stage heuristic scorers that have comparable or larger reasoning budgets.Source: C Supplementary Experiments · p. 15
• Other supplementary evaluations test downstream performance without skill injection and compare selectors based on community signals, text length, or file count.Source: C Supplementary Experiments · p. 15

• Each method assigns a selector score to every skill, and the outcome is downstream performance after that skill is injected.The outcomes include pass_rate for BFCL-Fundamental and BigCodeBench, and Step Success Rate for Mind2Web.Source: §C.1 · pp. 15–16
• MCRI-Eval attains the highest C-index and Spearman correlation on BFCL-Fundamental, BigCodeBench, and Mind2Web.Source: Table 10 · p. 16
• The protocol reports C-index, Spearman’s rank correlation and p-value, plus Top-k mean outcome and normalized rank score.The normalized rank runs from 0 for the worst position to 100 for the best; ties at the selector cutoff use expected Top-k outcome.Source: §C.1 · p. 16

C.2 Multi-Stage Heuristic Scorers

• The heuristic comparison uses three independent stage-one evaluations, then either summarizes them before stage two or pairs each evaluation with a stage-two run.Both variants compute selectors from three final scores and confidence products.Source: §C.2 · p. 16
• MCRI-Eval has the highest C-index and Spearman correlation in every benchmark and is the only method significant at p < 0.05 in all three; extra reasoning stages do not reproduce its ranking consistency.The heuristic variants do not consistently improve over FullTextBaseline, so the comparison does not support an explanation based solely on inference-step count.Source: §C.2 · pp. 16–17, Table 11 · p. 17

C.3 No-skill Reference

• The No-skill condition supplies only the task or function definition, with no skill text, and has no ranking-agreement statistics.Each benchmark’s fixed 50-task set is repeated three times.Source: §C.3 · p. 17
• On Mind2Web, the No-skill result is near the bottom of skill-injected outcomes, while MCRI-Eval Top-1 reaches a 99.32 rank score.On BigCodeBench, the No-skill mean is below MCRI-Eval Top-1; on BFCL-Fundamental, it is slightly above the average skill-injected level, but below MCRI-Eval Top-1.Source: §C.3 · p. 17, Table 12 · p. 17

C.4 Non-LLM Selectors

• Community selectors use stars, downloads, or all-time installs; text length counts injected skill-text characters, and file count counts packaged files.The comparison uses the same downstream outcomes and evaluation procedure as the LLM-based methods, and does not age-normalize community signals.Source: §C.4 · p. 18
• MCRI-Eval has the highest C-index and Spearman correlation among selectors on every benchmark and is the only method with a significant positive association in all three; popularity correlations range from −0.1557 to 0.0679.Popularity and task-specific post-injection performance are distinct criteria.Source: §C.4 · p. 18, Table 15 · p. 19
• On BigCodeBench, text length correlates with downstream performance at ρ = 0.2658, p = 0.0152, but its correlation is ρ = 0.0058 on BFCL-Fundamental and ρ = −0.0400 on Mind2Web.The passage offers a plausible interpretation involving reusable examples and library-specific information, but does not support a general rule that longer skills are better.Source: §C.4 · pp. 18–19
• File count is significantly negatively correlated with BFCL-Fundamental outcomes at ρ = −0.2237, p = 0.0096; its high Top-1 selection is an isolated outcome, not a reversal of the global ordering.Skills are injected as text without executing bundled scripts, which may leave behavior delegated to external files less directly usable in this setting.Source: §C.4 · p. 19

C.5 Summary of the Supplementary Evidence

• Across the comparisons, extra reasoning stages do not reproduce MCRI-Eval’s ranking consistency, arbitrary skill injection is not uniformly helpful, and surface signals are weak, task-specific, or unstable.MCRI-Eval is the only selector with a positive, statistically significant Spearman association in every benchmark setting.Source: §C.5 · p. 19

D Complete MCRI-Eval Definition and Scoring Rubric

• MCRI-Eval first scores Metadata, Constraints, Resources, and Instructions separately, then combines those scores with a benchmark description to produce a final quality score and confidence.The selector is the product of the mean final score and mean final confidence across repeated runs.Source: D Complete MCRI-Eval Definition and… · p. 19

D.1 Dimension Definitions

• Metadata concerns identifying when and why to invoke a skill, Constraints restrict permissible behavior, Resources supply external information, and Instructions organize execution and environment interaction.The dimensions are functional perspectives, not mutually exclusive components or statistically independent factors.Source: Table 16 · p. 20, §D.2 · p. 20

D.2 Stage A: Dimension Scoring

• Stage A counts only restrictive rules as Constraints, task-specific reusable information as Resources, and executable interfaces or orchestration as Instructions; it does not reward length or formatting.A tool mention without a call contract, parameters, ordering, or result handling does not warrant a high Instructions score.Source: §D.2 · p. 20
• Stage A evaluates the supplied skill name, summary, and full text, aggregates five sub-item scores per dimension, and maps them linearly to a 1–5 scale.This task-independent prompt excludes benchmark, popularity, downstream-performance, and external impressions from the evaluation.Source: §D.2 · pp. 20, 22
• Stage-A sub-item weights sum to one within each dimension.Source: Table 17 · p. 22

D.3 Task-Conditioned Stage-A Variant

• The downstream Stage-A variant evaluates each dimension’s usefulness for the current task rather than the skill’s task-independent general quality, while retaining the same construct definitions, sub-items, weights, and output schema.It appends a current-task description.Source: §D.3 · p. 22

D.4 Stage B: Overall Scoring

• Stage B outputs an overall quality score from 1 to 5 and a confidence discount factor applied to that score for the current task; confidence is neither skill quality nor task completion rate.The evaluation does not execute bundled scripts or resource files or call external APIs, browsers, network services, or other external tools declared by the skill.Source: §D.4 · p. 23

E.1 Concentration of Downstream Outcomes

• Pass rates span 73.33%–83.33% on BFCL-Fundamental and 47.33%–60.00% on BigCodeBench, while Mind2Web Step SR spans 50.14%–57.39%.These per-skill outcomes are concentrated in narrow intervals.Source: §E.1 · p. 23
• When many skills fall within a few percentage points, a 0.5-point difference can correspond to many ranking positions, so raw performance alone may understate selection differences.The paper therefore reports raw downstream performance alongside normalized rank score.Source: §E.1 · p. 24

E.2 Skill Text Length and File Count

• Median injected skill text is 15,402 characters on BFCL-Fundamental, 13,732 on BigCodeBench, and 14,028 on Mind2Web; file-count medians are 1, 3, and 3, respectively.Both text length and packaged file counts are right-skewed.Source: §E.2 · p. 24

F Resource Provenance

• The /Resources directory containing code, data, and intermediate outputs is absent from the preprint source package, and this version provides no public repository URL.Full computational reproduction additionally requires the separate experimental materials.Source: F Resource Provenance · p. 24