PaperlayerView exact revision on arXiv

Source-linked AI summary

A2Eval: Agentic and Automated Evaluation for Embodied Brain

Shuai Zhang, Jiayu Hu, Zijie Chen, Zeyuan Ding, Yi Zhang, Yingji Zhang, Ziyi Zhou, Junwei Liao, Shengjie Zhou, Yong Dai, Zhenzhong Lan, Xiaozhu Ju

arXiv:2602.01640v1cs.CL

TL;DR

Embodied VLM evaluation is limited by redundant, imbalanced benchmarks, distorted rankings, and high computational and annotation costs. A2Eval uses collaborative Data and Eval Agents to curate compact balanced suites and synthesize validated executable pipelines. Across 10 benchmarks and 13 models, it filters 85% redundant samples, reduces overall cost by 77%, accelerates evaluation 4.6×, and improves human preference alignment to 0.85 Spearman’s ρ.

  • Problem

    Existing embodied VLM evaluation relies on expert-defined, manually annotated benchmarks with severe redundancy and coverage imbalance, producing distorted rankings and high evaluation costs.

  • Method

    A2Eval uses a Data Agent to induce capability dimensions and construct balanced compact suites, while an Eval Agent synthesizes and validates executable evaluation pipelines.

  • Results

    85% redundant samples are filtered, overall cost decreases 77%, evaluation accelerates 4.6×, and human preference alignment reaches 0.85 Spearman’s ρ.

  • Takeaways & Limitations

    A2Eval provides autonomous benchmark curation and evaluation that mitigates coverage imbalance, ranking distortion, and computational cost.

Abstract

from arXiv · show

Current embodied VLM evaluation relies on static, expert-defined, manually annotated benchmarks that exhibit severe redundancy and coverage imbalance. This labor intensive paradigm drains computational and annotation resources, inflates costs, and distorts model rankings, ultimately stifling iterative development. To address this, we propose Agentic Automatic Evaluation (A2Eval), the first agentic framework that automates benchmark curation and evaluation through two collaborative agents. The Data Agent autonomously induces capability dimensions and assembles a balanced, compact evaluation suite, while the Eval Agent synthesizes and validates executable evaluation pipelines, enabling fully autonomous, high-fidelity assessment. Evaluated across 10 benchmarks and 13 models, A2Eval compresses evaluation suites by 85%, reduces overall computational costs by 77%, and delivers a 4.6x speedup while preserving evaluation quality. Crucially, A2Eval corrects systematic ranking biases, improves human alignment to Spearman's rho=0.85, and maintains high ranking fidelity (Kendall's tau=0.81), establishing a new standard for high-fidelity, low-cost embodied assessment. Our code and data will be public soon.

1. Introduction

Existing embodied VLM evaluation suffers from redundant, imbalanced benchmarks, distorted rankings, and prohibitive costs. A2Eval addresses these problems with autonomous benchmark construction and executable evaluation pipelines.

  • Evaluation Crisis: Over 3200 GPU hours are consumed when evaluating a single model across fragmented benchmarks.This computational burden makes evaluation cycles costly and constrains iterative research.
  • Evaluation Crisis: Up to 92% sample similarity and skewed task distributions create redundant coverage and underrepresent critical reasoning capabilities.Easy-to-annotate tasks dominate while capabilities such as PhysCausal receive fewer examples.
  • A2Eval Framework: A2Eval treats benchmark curation as an optimization problem that maximizes capability coverage and diversity while minimizing redundancy and human effort.Its process replaces the traditional expert-definition and manual-annotation paradigm with two collaborative agents.
  • A2Eval Framework: The Data Agent autonomously induces capability dimensions and uses diversity-aware sampling to build balanced, compact evaluation suites.This design targets coverage imbalance, redundancy, ranking distortion, and redundant computation while preserving comprehensive coverage.
  • A2Eval Framework: The Eval Agent synthesizes and validates executable inference and scoring pipelines through iterative sandbox refinement.The resulting process supports fully autonomous, high-fidelity evaluation without manual intervention.
  • Reported Contributions: 85% redundant samples are filtered, human preference alignment reaches 0.85 Spearman’s ρ, overall cost falls 77%, and throughput accelerates 4.6×.The reported experiments cover 10 benchmarks and 13 models.
  • Reported Contributions: 96.9% fidelity is achieved for autonomous executable evaluation without manual intervention.The Eval Agent automatically constructs validated inference and scoring logic.

2. Related Work

Embodied reasoning benchmarks cover diverse capabilities, while automated benchmark-construction methods have mainly focused on generating new domain-specific sets. A2Eval instead targets consolidation of existing heterogeneous benchmarks and their redundancy and cost problems.

  • Embodied VLM Benchmarks: Recent embodied benchmarks target placement, spatial understanding, grounding, manipulation, physical reasoning, and causal reasoning.Examples include Where2Place, VSI-Bench, RefSpatial, RoboSpatial, EmbSpatial, OmniSpatial, ERQA, and COSMOS.
  • Automated Benchmark Construction: Code2Bench and OKBench automate creation of new domain-specific benchmarks with contamination control and reproducibility protocols.Their focus is generating fresh evaluation sets rather than consolidating existing heterogeneous benchmarks.
  • A2Eval’s Distinction: A2Eval differs by combining automatic capability induction, balanced cross-domain sampling, and redundancy-aware benchmark consolidation.These distinctions position it around reducing redundancy and computational costs in existing evaluation ecosystems.

3. Method

A2Eval automates embodied VLM evaluation through a Data Agent that induces capability dimensions and constructs a balanced benchmark, and an Eval Agent that synthesizes validated executable pipelines. The resulting workflow covers benchmark construction and model assessment across capability dimensions.

  • Overview: A2Eval automates the evaluation lifecycle with a Data Agent for benchmark construction and an Eval Agent for executable model assessment.The Data Agent discovers unified dimensions and samples a compact benchmark, while the Eval Agent generates validated inference and scoring pipelines.
  • Data Agent: The Data Agent treats benchmark construction as a two-stage workflow: dimension induction followed by dimension assignment and diversity-aware sampling.Specialized agents collaboratively produce a capability taxonomy, after which examples are assigned to dimensions and sampled to reduce redundancy while preserving coverage.
  • Dimension Induction: The Proposer, Reviewer, and Assigner iteratively refine dimensions using critiques, historical memory, and empirical balance statistics.The Reviewer checks conceptual redundancy, coverage comprehensiveness, and dimension balance before the process stabilizes on a dimension set.
  • Benchmark Construction: Diversity-aware sampling embeds examples semantically and selects representatives near cluster centroids, retaining internal diversity while balancing dimensions.If a dimension contains fewer than K examples, all are retained; otherwise, centroid-based selection is used.
  • Eval Agent: The Eval Agent uses separate Evaluator and Scorer roles with sandbox execution to iteratively synthesize and validate inference and scoring logic.Inference code loads inputs, invokes the target model, and produces predictions; scoring code evaluates those predictions and is refined when execution fails.
  • Evaluation Execution: The validated inference and scoring artifacts are executed on the constructed benchmark to produce assessment results across all induced capability dimensions.The Evaluator generates predictions for the test cases, and the Scorer computes the final evaluation metrics.

4. Experiments

Across ten embodied vision–language benchmarks and diverse models, A2Eval constructs compact, balanced evaluations and preserves coverage, ranking alignment, efficiency, and pipeline fidelity.

  • The Data Agent induces eight capability dimensions spanning perception, spatial reasoning, physical understanding, and decision making.
  • 24,519 raw examples are reduced to 3,781 through dimension assignment and diversity-aware sampling.The sampled suite omits the negligible “Other” category.
  • Selected samples sparsely cover the semantic manifold while removing intra-dimension redundancy and preserving comprehensive capability coverage.This behavior is illustrated for the PercepObj and SpatGeo dimensions using UMAP projections.
  • Human validation yields average Cohen’s κ of 0.78 and inter-annotator agreement of 0.80 for automatic dimension assignments.
  • The rebalanced benchmark aligns with human rankings at ρ = 0.85 and τ = 0.72, improving over the original benchmark’s ρ = 0.83 and τ = 0.64.
  • A2Eval achieves 3.4× to 4.6× evaluation speedups, while the Eval Agent reaches 96.9% average fidelity against reference implementations.For Qwen3-VL-235B-A22B-Thinking, evaluation time falls from 412.9 to 89.4 hours, a 4.6× speedup.

5. Conclusion

A2Eval automates embodied benchmark curation and evaluation with collaborative Data and Eval Agents. Experiments show mitigation of coverage imbalance, ranking distortion, and computational cost.

  • A2Eval is an agentic framework in which the Data Agent induces capability dimensions and constructs balanced suites while the Eval Agent synthesizes executable pipelines.
  • Experiments show that A2Eval mitigates benchmark coverage imbalance, evaluation ranking distortion, and computational cost.

Impact Statement

A2Eval targets more efficient, high-fidelity, and fair embodied vision-language model evaluations by reducing computational costs and mitigating systematic ranking biases.

  • A2Eval enables more efficient, high-fidelity, and fair evaluations while reducing computational costs and mitigating systematic ranking biases.

A. Benchmark Description and Redundancy Analysis

The benchmark study documents substantial redundancy in existing evaluation resources and compares capability distributions before and after sampling.

  • The supplementary section describes source benchmarks and illustrates redundancy patterns in the evaluation landscape.
  • Table 12 summarizes the benchmarks used in the study, including their descriptions and references.
  • Pairwise semantic similarity reveals substantial overlap across benchmarks, with redundant examples documented across different sources.
  • Figures 5 and 6 show capability distributions before and after sampling and UMAP embeddings for each dimension.

B. Detailed Evaluation Setting

The evaluation uses family-specific decoding and image-processing settings, with results reported across capability dimensions and examples illustrating redundancy and rebalanced coverage.

  • Model settings: Qwen3-VL Instruct models use top-p = 0.8, top-k = 20, temperature = 0.7, and repetition penalty = 1.0.Thinking models use top-p = 0.95 and temperature = 0.6, with the same top-k and repetition penalty.
  • Reported evaluations: Table 7 reports model performance across all capability dimensions, with scores expressed as percentages and Avg denoting the mean across dimensions.
  • Reported evaluations: Table 8 presents four pairs of redundant samples across benchmarks.
  • Model settings: Qwen2.5-VL and InternVL-3.5 models use greedy decoding with temperature set to 0.01.InternVL-3.5-241B-A28B is deployed with the PyTorch backend.
  • Sampling analysis: The retained set exhibits substantially more balanced capability coverage than the highly skewed source distribution after diversity-aware sampling.

C. Implementation Details of the Evaluation Agents

The framework uses a Data Agent to induce and balance capability dimensions and an Eval Agent to generate executable inference and scoring pipelines through iterative feedback.

  • Data Agent: The Data Agent constructs capability dimensions and a structured benchmark through proposal–review–assignment iterations and diversity-aware sampling.Its procedure is specified in Algorithm 1.
  • Data Agent: The Proposer uses benchmark metadata, reviewer critiques, and historical memory to refine dimensions and avoid previously rejected configurations.
  • Data Agent: The Reviewer evaluates redundancy, coverage completeness, and balance, while the Assigner uses five voters, majority assignment, embedding-based clustering, and representative selection.Balance statistics identify under-represented dimensions for possible merging or refinement.
  • Data Agent: The dimension prompts require non-overlapping, mutually independent, concise, precise, and generalizable capability definitions.
  • Eval Agent: The Eval Agent synthesizes inference and scoring logic, then refines both through sandbox execution feedback until execution reliability and metric consistency are achieved.The Evaluator produces model predictions, while the Scorer computes metrics from test cases and predictions.

D.5. Results and Analysis

The evaluation uses real-world videos and instructions to assess embodied planning across multiple capability dimensions, with outputs constrained to visible evidence and physical feasibility.

  • Evaluation scenarios: Eight real-world scenarios cover Household, Grocery Store, School, Office, Warehouse, Hospital, Workshop, and Laboratory settings.Each video lasts approximately 15–30 seconds and contains 4–6 naturally instructed sub-tasks.
  • Evaluation scenarios: The household example combines shoe organization, table-trash disposal, and sofa-laundry loading into three sub-tasks.
  • Evaluation protocol: Models receive the same scenario video, system prompt, and task instruction, then produce explicit, physically feasible step-by-step plans.
  • Evaluation protocol: The required output addresses scene elements, goals, action order, spatial relations, object states, and relevant quantities.

E.6. Human Evaluation Procedure

Human evaluation constructs model rankings from multi-annotator scores across randomized real-world scenarios, then compares those rankings with benchmark-induced rankings.

  • Annotation procedure: Each model-scenario output is independently rated by at least 3–5 annotators, with scenario and model order randomized to mitigate ordering bias.
  • Ranking construction: Human-centric model scores average eight equally weighted dimension scores across annotators and scenarios to induce a human-perceived ranking.
  • Ranking comparison: Benchmark-induced and human-centric rankings are compared using Spearman’s ρ and Kendall’s τ to assess alignment with human judgments.

F. Detailed Fidelity Analysis

The analysis separately evaluates inference fidelity and scoring fidelity by comparing agent-generated components with reference implementations. Inference outputs achieve 93.6% average fidelity, while scoring fidelity exceeds 97% on average.

  • Fidelity evaluation: The evaluation separately analyzes inference fidelity and scoring fidelity for the Eval Agent.The complete pipeline includes inference logic F_e and scoring logic F_s, whose outputs are compared with reference implementations.
  • Inference fidelity: 93.6% average inference fidelity is achieved across all capability dimensions and model configurations.Inference fidelity uses normalized edit distance between agent-generated outputs and reference model results.
  • Scoring fidelity: Scoring fidelity exceeds 97% on average across all capability dimensions and model configurations.Scoring fidelity is measured using relative score difference between agent-generated and reference scores.
  • Pipeline specification: The evaluation pipeline takes benchmark information, test examples, a model, and capability dimensions as inputs, producing inference and scoring logic alongside predictions and scores.The pipeline assigns evaluator and scorer roles and executes the generated logic in a sandbox.
Loading 2602.01640v1…