Source-linked AI summary
Towards Evaluation Engineering: An Empirical Study of ML Evaluation Harnesses in the Wild
Zhimin Zhao, Zehao Wang, Abdul Ali Bangash, Bram Adams, Ahmed E. Hassan
TL;DR
Operational challenges in ML evaluation harnesses have received limited software engineering attention. The paper empirically studies 57 harnesses and 16,560 classified issues to model workflows and root causes. It finds that challenges concentrate in specification and are dominated by capability, documentation, and validation gaps, establishing an empirical foundation for evaluation engineering as a distinct software engineering concern.
Problem
Software engineering research has not studied evaluation harnesses as software products, despite their role in operationalizing model evaluation.
Method
The study analyzes documentation, local execution, and GitHub issues from 57 harnesses to derive a workflow model and classify operational root causes.
Results
41.4% of issues concentrate in the Specification stage, while unimplemented features, documentation gaps, and missing input validation account for 61.7% of classified issues.
Takeaways & Limitations
The findings establish an empirical foundation for treating evaluation engineering as a distinct software engineering concern and evaluation reliability as a first-class concern.
Takeaways & Limitations
Issue counts characterize challenge frequency rather than severity, and single-label classification can underrepresent cascading or multi-cause challenges.
Abstract
from arXiv · showhide
Evaluation harnesses are software systems that orchestrate model evaluation by managing model invocation, data loading, metric computation, and result reporting. Despite their critical role in machine learning infrastructure, their operational challenges and engineering concerns have received limited attention so far. We present an empirical study of 57 evaluation harnesses, deriving a five-stage harness model and classifying 16,560 issues by workflow stage and root cause. Most harness operational challenges concentrate in the Specification stage (41.4% of issues), where harnesses integrate external models, datasets, and scoring judges. The three most frequent root causes of operational challenges are unimplemented features (24.3%), documentation gaps (20.3%), and missing input validation (17.2%), which together account for 61.7% of classified issues, spanning both defects in existing functionality and capability gaps that block intended workflows. Root causes also vary by workflow stage: environment incompatibility and external dependency breakage account for 36.2% of provisioning issues, whereas algorithmic error (25.9%) and validation gap (22.5%) dominate assessment issues. Together, these contributions establish an empirical foundation for treating evaluation engineering as a distinct software engineering concern.
1 Introduction
The paper frames evaluation harnesses as software products whose operational workflows and reliability concerns have been largely overlooked. It studies 57 harnesses and GitHub issues to model workflows, classify root causes, and identify where operational challenges concentrate.
- Evaluation harnesses orchestrate model invocation, data loading, metric computation, and result reporting across diverse evaluation scenarios.
- No prior software engineering work had examined evaluation harnesses as products, including their workflows, user challenges, or reliability-shaping engineering decisions.
- The study analyzes documentation, local execution, and GitHub issues from 57 harnesses to extract a unified workflow model and categorize operational challenges.
- The methodology combines open card sorting with large-scale GitHub issue mining across the collected evaluation harnesses.
- 41.4% of all issues occur in the Specification stage, while unimplemented features, documentation gaps, and missing input validation comprise 61.7% of classified issues.
- The study contributes a five-stage, nine-step, and 34-strategy workflow model, issue mapping across 57 harnesses, and a ten-category root-cause taxonomy.
2 Background and Related Work
Prior research emphasizes what ML evaluation measures, while software engineering concerns in the infrastructure that performs evaluation remain underexamined. The paper positions evaluation harnesses as distinct from benchmarks, MLOps checkpoints, and traditional testing infrastructure.
- Evaluation infrastructure enables reliable measurement of model progress, supporting advances on tasks with dependable verification mechanisms.
- Benchmark contamination, undocumented implementation details, incompatible frameworks, and annotation errors can undermine evaluation reliability and reproducibility.
- Benchmarks define evaluation tasks, datasets, references, and metrics, whereas harnesses operationalize measurement through invocation, resource management, error handling, aggregation, and reporting.
- Evaluation surveys catalog tasks, metrics, and methodological gaps but do not examine the software that executes evaluations.
- MLOps research treats evaluation as a pipeline checkpoint and offers limited guidance on dependency volatility, execution failures, and result integrity.
- Evaluation harnesses share operational concerns with testing infrastructure but manage heterogeneous models, datasets, and APIs while producing continuous aggregate metrics rather than binary outcomes.
- The paper addresses this operational gap through empirical analysis of 57 evaluation harnesses and 19,638 GitHub issues.
3 Methodology
The methodology combines systematic harness collection, inductive workflow extraction, and issue analysis. Researchers use documentation, source inspection, local execution, and iterative comparison to derive and validate a unified workflow model.
- An evaluation harness is defined as a software framework whose primary purpose is orchestrating ML model evaluation.
- The collection begins with 45 harnesses from a curated production ML list and adds 12 newly identified harnesses.
- Keyword-based GitHub searches expand coverage, while independent repository screening checks evaluation purpose, implementation, and quality criteria.
- Open card sorting of documentation is triangulated with source inspection and local execution, using constant comparison to consolidate a hierarchical workflow model.
- The workflow model distinguishes stages, steps, and strategies as progressively more detailed levels of evaluation operation.
- The final model contains 5 stages, 9 operational steps, and 34 implementation strategies, encoded in a 57 × 9 harness-step support matrix.
- Ward’s hierarchical clustering groups harnesses by strategy coverage, with k=6 selected because it produces the highest mean silhouette score among k=2–8.
3.3 GitHub Issues Collection
The study collects open and closed GitHub issues from the identified harness repositories to capture both unresolved problems and issues with documented resolutions. These reports support analysis of operational root causes and their workflow-stage distribution.
- 19,638 GitHub issues were retrieved from 59 repositories associated with 57 evaluation harnesses.Two harnesses maintain separate backend and frontend repositories.
3.4 GitHub Issues Analysis
The study uses a hybrid manual and LLM-based classification process to map workflow-relevant GitHub issues onto operational stages and root causes.
- Classification methodology: 377 randomly sampled issues were manually annotated to provide a statistically significant reference sample.The sample targets a 95% confidence level and 5% margin of error under maximum variability assumptions.
- Classification methodology: LLM classifications were calibrated against human consensus until substantial agreement exceeded κ> 0.8.The classifier used workflow definitions, annotation guidelines, and full issue context, truncated at 200K tokens when necessary.
- Workflow classification: 16,560 workflow-relevant issues were mapped to stages, steps, and implementation strategies using closed card sorting.Issues affecting multiple components received a primary label based on their most direct operational impact.
- Root-cause classification: Ten root-cause categories were developed through open card sorting of workflow-relevant sample issues and author negotiation.Root-cause annotation achieved κ=0.758 and 78% raw agreement, lower than workflow classification because attribution was more ambiguous.
- Illustrative classification: Issue #1407 illustrates the two-level scheme by linking a multi-GPU loading failure to both a workflow location and a missing harness capability.The issue was assigned to Specification, SUT preparation, and model-in-process execution before root-cause classification.
4 RQ1: Unified Workflow for Evaluation Harnesses
The unified workflow spans provisioning, specification, execution, assessment, and reporting, but harness support is strongest for offline batch evaluation and weakest for online monitoring and reporting.
- Workflow lifecycle: Five workflow stages progress from provisioning and specification through execution and assessment to reporting.The lifecycle moves from establishing the runtime environment and evaluation contract to producing stakeholder-facing insights.
- Provisioning: 100% of harnesses support git clone, 94.7% support Python packages, but only 21.1% support container images.Developers requiring reproducible isolated environments therefore need to maintain container configurations independently.
- Specification: 91.2% of harnesses use offline benchmark inputs and ground-truth references, while interactive agent evaluation reaches 28.1%.Production traffic sampling is supported by only 4 of 57 harnesses, or 7.0%.
- Execution and assessment: 94.7% of harnesses support batch inference, whereas assessment support is dominated by deterministic measurement at 89.5%.Support narrows for alternative execution and scoring strategies, including interactive, judge-based, embedding-based, and efficiency-based evaluation.
- Assessment: 96.5% support aggregate statistics, but only 22.8% support uncertainty quantification.Most harnesses therefore cannot indicate whether an observed score difference is meaningful or due to chance.
- Reporting: No reporting strategy exceeds 45.6% adoption, and regression alerting is used by only 8.8% of harnesses.Reporting is the least supported stage, with limited automated detection of performance degradation across runs.
- Harness archetypes: Standardized LLM Benchmark Suites and Narrow-Domain Metric Libraries together cover over 61% of harnesses while remaining restricted to static offline workflows.Full-Stack LLM Evaluation Platforms comprise 17.5% and are the only archetype spanning all five stages with every strategy present in at least one member.
5 RQ2: Root Causes of Operational Challenges
Operational challenges are dominated by capability, documentation, and validation gaps, while root-cause prevalence and secondary causes vary across harness archetypes.
- Root Cause Prevalence: 61.7% of issues come from unimplemented feature gap (24.3%), documentation deficiency (20.3%), and validation gap (17.2%).These categories outweigh interface contract mismatch, resource mishandling, and architectural constraint, which together account for 9.1%.
- Distribution Across Archetypes: Figure 5 compares root-cause distributions by archetype using normalized issue counts and within-archetype percentages.Rows represent archetypes, columns represent root causes, and normalized counts enable comparisons across archetypes of different sizes.
- Distribution Across Archetypes: Full-Stack LLM Evaluation Platforms reach 93.3 issues per harness for unimplemented feature gap, 90.6 for validation gap, and 89.9 for documentation deficiency.Standardized LLM Benchmark Suites show comparable normalized counts of 86.4, 57.0, and 70.3, respectively.
- Distribution Across Archetypes: Narrow-Domain Metric Libraries peak at 33.2 issues per harness for documentation deficiency, consistent with their single-metric scope limiting the components that can fail.This is substantially below the normalized issue volumes reported for Full-Stack platforms and Standardized LLM Benchmark Suites.
- Distribution Across Archetypes: Unimplemented feature gap leads in three archetypes at 27.8%, 42.9%, and 17.6% of within-archetype issues, while Narrow-Domain Metric Libraries are led by documentation deficiency at 31.8%.Secondary causes differ by archetype: external dependency breakage for Standardized LLM Benchmark Suites, algorithmic error for Task-Specific Capability Probes, and interface contract mismatch for Full-Stack platforms.
6 RQ3: Root Causes across Workflow Stages
Root causes localize differently across the evaluation workflow: early stages are more environment-related, while later stages concentrate scoring and validation concerns.
- Stage-Level Composition: 55.8% of unimplemented-feature-gap issues and 55.2% of documentation-deficiency issues occur in specification, compared with 5.1% and 4.9% in execution.Validation gap remains above 10% across all five stages and peaks at assessment at 22.7%.
- Stage-Level Composition: 54.3% of algorithmic-error issues occur in assessment, while 61.3% of resource-mishandling issues occur during execution.Resource issues arise when the system under test runs, whereas scoring issues surface when results are computed.
- Step-Level Localization: Resource mishandling peaks at SUT invocation (61.3%), environment incompatibility and external dependency breakage at harness installation (51.9% and 45.1%), and algorithmic error at individual scoring (43.3%).These concentrations provide distinct step-level targets for the corresponding root causes.
- Stage-Level Composition: Provisioning spreads issues across four root causes without one exceeding 21%, while specification and assessment concentrate nearly half or more of their issues in two causes.The study reports increasing root-cause concentration from provisioning toward specification and assessment.
- Stage-Level Composition: 36.2% of provisioning issues come from environment incompatibility (16.1%) and external dependency breakage (20.1%), whereas these causes each fall below 6% by assessment.Algorithmic error accounts for only 0.9% of provisioning issues but becomes prominent in later workflow stages.
- RQ3 Summary: Root-cause composition shifts from environment-related challenges in early stages to scoring-related challenges in later stages, although capability and documentation gaps persist across stages.Unimplemented feature gap and documentation deficiency together account for 59.9% of specification issues and 59.5% of reporting issues.
7 Implications
The implications emphasize that evaluation harness reliability requires engineering practices tailored to semantic compatibility, silent scoring failures, archetype-specific risks, and transparent capability reporting.
- Implications for Harness Developers: Validation gaps span all five stages and peak at assessment, where syntactically valid data can remain semantically incompatible with downstream components.The COMET22 example shows a valid configuration field silently defaulting to a different activation function.
- Implications for Harness Developers: Algorithmic errors concentrate in assessment and can produce plausible but incorrect scores without runtime exceptions.LM Eval reported ROUGE-L scores near 1.0 because of a metric computation bug discovered through comparison with published results.
- Implications for Harness Developers: Metamorphic and differential testing provide independent checks for metric implementations that ordinary unit tests may miss.Metamorphic tests encode input-output invariants, while differential testing compares independent implementations before release.
- Implications for Harness Developers: Maintenance priorities should follow archetype-specific failure modes rather than a uniform backlog policy.Recommended practices include dependency pinning for benchmark suites, metamorphic and differential testing for capability probes, and contract tests at platform service boundaries.
- Implications for Harness Developers: A machine-readable harness specification could expose workflow coverage, dependencies, and omitted capabilities before users select a harness.The paper reports that 77.2% of harnesses lack uncertainty quantification and 91.2% lack regression alerting.
- Implications for Researchers: Evaluation engineering remains an open software engineering problem because existing schema-level validation cannot enforce task-specific semantic contracts.The paper identifies semantic compatibility at stage boundaries as a research direction distinct from conventional deployment-focused MLOps concerns.
8 Threats to Validity
The study’s validity threats concern how issue counts, single-label classifications, repository selection, and public reporting shape the observed distributions.
- Conclusion Validity: Issue counts measure frequency rather than severity, so the distributions should not be read as a strict impact-based priority ordering.The analysis also assumes issue independence, although multiple issues may arise from one underlying infrastructure problem.
- Construct Validity: Assigning each issue to one stage and root cause simplifies aggregation but can underrepresent cascading and multi-cause challenges.In the sampled issues, 14.3% involved annotator disagreement on stage or root-cause assignment.
- External Validity: The GitHub sample favors open-source harnesses with at least 500 stars and recent maintenance, potentially excluding smaller industry-used systems.Popularity may also amplify the influence of high-traffic repositories on aggregate issue distributions.
- Internal Validity: Cross-stage issue volumes reflect different user populations because users blocked earlier may never reach later workflow stages.Within-stage root-cause compositions are less affected because they compare relative mixes among users who reach each stage.
- Internal Validity: Public GitHub issues omit privately resolved or abandoned problems, while issue-filing culture may vary across user communities.Temporal aggregation may also combine historical problems with the current state of evolving harnesses.
9 Conclusion
The conclusion establishes evaluation engineering as a software engineering concern through a workflow model, large-scale issue analysis, root-cause taxonomy, and identified capability gaps.
- Conclusion: The study models evaluation harnesses with five stages, nine steps, and 34 strategies, using 16,560 classified GitHub issues across 57 harnesses.It also derives ten challenge categories spanning software defects and capability gaps.
- Conclusion: Production-oriented capabilities such as uncertainty quantification and regression alerting remain adoption gaps across evaluation harnesses.The conclusion frames these gaps alongside the study’s empirical mapping of where operational challenges concentrate.
- Conclusion: The findings support treating evaluation reliability as a first-class concern of evaluation engineering.The paper states that improving reliability requires attention to workflow design across all stages, not only isolated metric or benchmark improvements.
- Conclusion: Two directions are proposed: structured transparency documents for cross-harness comparison and validity-oriented methods incorporating uncertainty, regression detection, and production-traffic assessment.These directions follow from the reported workflow coverage, dependencies, and capability gaps.
Continued on next page
The workflow continues by defining benchmark-input preparation, execution, assessment, and reporting activities, including test-data creation and result visualization.
- Workflow Model: Benchmark-input preparation acquires and configures test inputs used to evaluate the system under test.Inputs may come from pre-existing remote or local datasets or from manually specified custom test inputs, with optional preprocessing.
- Workflow Model: 40.4% of benchmark inputs are created on the fly through perturbation, augmentation, trajectory generation, and scenario synthesis.These generated inputs feed execution of the system under test.
- Workflow Model: Execution applies test inputs to the system under test to generate outputs or actions.The execution stage observes system behavior before assessment converts those observations into measurements.
- Workflow Model: Assessment computes individual-instance metrics and aggregates them into benchmark-level measures.Aggregate scoring is described as a fundamental operation supported by all evaluation harnesses.
- Workflow Model: Reporting makes results actionable through insight presentation, visualization, interactive interfaces, ranked tables, and leaderboard publication.Examples include radar charts, drift histograms, performance trends, filterable outcomes, and public or private leaderboards.