PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models

Fan Zhang, Shulin Tian, Ziqi Huang, Yu Qiao, Ziwei Liu

arXiv:2412.09645v3cs.CVcs.AIcs.CL

TL;DR

Evaluating visual generative models can be costly, slow, rigid, and difficult to interpret. The Evaluation Agent uses human-like strategies to conduct dynamic, user-tailored evaluations with a small number of samples and detailed analyses. Experiments report over 90% lower evaluation time with accuracy comparable to standard benchmarks.

  • Problem

    Existing evaluation frameworks require many samples, incur high costs for slow-sampling diffusion models, constrain evaluation to predefined dimensions, and often provide only numerical scores.

    Source: §1 · pp. 1–2

  • Method

    The Evaluation Agent takes open-ended user input, uses appropriate tools, refines exploration based on intermediate results, and returns a detailed natural-language evaluation.

    Source: §1 · p. 2

  • Results

    Over 90% lower evaluation time and accuracy comparable to standard benchmarks are reported for the Evaluation Agent.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework is open-sourced, and its analyses are presented as cornerstones for future research on evaluating visual generative models.

    Source: §1 · p. 2

    Evaluation accuracy depends on toolkit reliability and LLM capability; tools may misalign with human perception and may not cover some user criteria or edge cases.

    Source: §7 · p. 9

Abstract

from arXiv · show

Recent advancements in visual generative models have enabled high-quality image and video generation, opening diverse applications. However, evaluating these models often demands sampling hundreds or thousands of images or videos, making the process computationally expensive, especially for diffusion-based models with inherently slow sampling. Moreover, existing evaluation methods rely on rigid pipelines that overlook specific user needs and provide numerical results without clear explanations. In contrast, humans can quickly form impressions of a model's capabilities by observing only a few samples. To mimic this, we propose the Evaluation Agent framework, which employs human-like strategies for efficient, dynamic, multi-round evaluations using only a few samples per round, while offering detailed, user-tailored analyses. It offers four key advantages: 1) efficiency, 2) promptable evaluation tailored to diverse user needs, 3) explainability beyond single numerical scores, and 4) scalability across various models and tools. Experiments show that Evaluation Agent reduces evaluation time to 10% of traditional methods while delivering comparable results. The Evaluation Agent framework is fully open-sourced to advance research in visual generative models and their efficient evaluation.

1 Introduction

Visual generative models now produce high-quality images and videos, but their evaluation can be slow, rigid, and difficult to interpret. The Evaluation Agent adopts human-like evaluation strategies to support efficient, user-tailored assessments with detailed explanations.

1 Introduction

  • Existing frameworks require many samples, incur high costs for slow-sampling diffusion models, constrain evaluation to predefined dimensions, and often return single scores.Users must invest additional effort to extract meaningful insights from those scores.Source: §1 · pp. 1–2
  • Human evaluators can form impressions from few samples and flexibly deepen evaluation through continued exploration of user-defined criteria.Examples include realism, creativity, and prompt adherence.Source: §1 · p. 2
  • The Evaluation Agent dynamically adjusts its evaluation pathway, accepts open-ended user input, provides detailed interpretable insights, and supports integration of new tools.It observes intermediate results to refine exploration and produces a natural-language summary of model capabilities; it can also compare or recommend models.Source: §1 · p. 2
  • Over 90% lower evaluation time and accuracy comparable to standard benchmarks are reported for the Evaluation Agent.Experiments also cover image and video generation and an open-ended user-query dataset.Source: §1 · p. 2

2 Related Work

Related work covers visual-generation evaluation, LLMs as evaluators, and agentic planning. The paper identifies scope and strategy gaps in these areas, including limited exploration of agents for evaluating visual generative models.

2.1 Visual Generation and Evaluation

  • Visual-generation evaluation lacks a definitive ground truth; metrics and multidimensional benchmarks are commonly used but can be time-intensive and limited in scope.The table characterizes the Evaluation Agent as supporting customized natural-language queries for text-to-image and text-to-video models, with dynamic multi-tool evaluation and textual analysis.Source: §2.1 · pp. 2–3, Table 1 · p. 3

2.2 LLM as a Judge

  • LLMs can reason about and explain evaluations, but their use for visual generative models remains largely unexplored and domain-specific evaluation strategies remain challenging.Prior work cited includes LLM evaluation for open-ended language tasks and downstream digital-agent tasks.Source: §2.2 · p. 3

2.3 Agent in Planning & Reasoning

  • Agentic systems customize task trajectories, yet automating visual-generative-model evaluation with agents remains largely unexplored.Prior approaches include Chain-of-Thought and Zero-shot-CoT prompting and ReAct’s interleaved thought-action-observation pattern.Source: §2.3 · p. 3

3 Methods

The Evaluation Agent uses LLM-based agents in iterative proposal and execution stages to assess visual generative models against user queries. Unlike fixed evaluation pipelines, it dynamically selects test cases and tools based on the evaluation process.

3.1 Preliminaries: Evaluation of Visual Generative Models

  • Existing evaluations use predefined test cases and dimensions, while the Evaluation Agent dynamically determines test cases and tools, usually ending with only a few cases.Existing benchmarks often contain hundreds or thousands of conditions, which require intensive sampling computation and time.Source: §3.1 · pp. 3–4

3.2 The Evaluation Agent Framework

  • LLM-based agents iteratively connect proposal and execution stages to evaluate models in response to user queries.The framework dynamically adapts evaluation based on intermediate observations.Source: §3.2 · p. 4, §3.2.3 · p. 5

3.2.1 Proposal Stage

  • The Plan Agent identifies and refines evaluation sub-aspects from intermediate feedback, while the PromptGen Agent creates tailored prompts for those aspects.PromptGen can also use established prompts from existing benchmarks.Source: §3.2.1 · p. 4

3.2.2 Execution Stage

  • Execution generates visuals from proposed prompts and evaluates them with an open, extensible toolkit that includes existing tools and VLM-based VQA assessments.Tables 2–3 report conclusion accuracy across 15 VBench and four T2I-CompBench dimensions, showing exact-range and one-range-error rates over ten trials.Source: §3.2.2 · pp. 4–5, Table 2 · p. 5, Table 3 · p. 5

3.2.3 Overall Pipeline

  • The multi-round pipeline adjusts exploration and prompt design based on intermediate observations, allowing evaluation to focus iteratively on model strengths and limitations.The process loops between proposal and execution stages.Source: §3.2.3 · p. 5

4 Experiments

The experiments assess Evaluation Agent on existing text-to-video and text-to-image benchmarks, then test its handling of open-ended user queries. Results report substantially lower evaluation costs alongside benchmark-consistent predictions and detailed user-focused analyses.

4.1.1 Experimental Setup

  • The benchmark setup uses four models per task, VBench for text-to-video and T2I-CompBench for text-to-image, restricting the agent to each benchmark’s tools and prompts.Source: §4.1.1 · pp. 5–6

4.1.2 Results Analysis

  • VBench evaluation time falls by over 10X; T2I-CompBench evaluations use about 26 samples and take five to eight minutes per dimension, with high accuracy within one range.Source: Table 4 · p. 6, §4.1.2 · p. 6, Table 5 · p. 6
  • Across VBench percentage dimensions, increasing samples from three to nine per round to 30 steadily improves both models’ results; Table 6 reports exact- and one-range predictions over ten trials.Source: Figure 3 · p. 6, §4.1.2 · p. 7, Table 6 · p. 7

4.2.1 Open-Ended User Query Dataset

  • The open-ended query dataset contains 100 manually labeled queries, tagged by ability, generality or specificity, and specific domain.Source: §4.2.1 · p. 7

4.2.2 Experimental Setup

  • For open-ended queries, a VLM answers questions generated alongside sub-aspect prompts, and its answers feed back to the Plan Agent for subsequent planning.Source: §4.2.2 · p. 7

4.2.3 Open-Ended User Query Evaluation

  • In the artwork example, the agent moves from assessing basic style replication to checking style preservation during detail changes, then synthesizes feedback for the user.Source: §4.2.3 · pp. 7–8

5 Further Discussions

The discussion contrasts dynamic, multi-step assessment with fixed-prompt benchmarks and considers VQA-based tools and broader applications. It emphasizes flexible, fine-grained evaluation while noting dependence on VLM capabilities.

  • Hierarchical evaluation progresses from simple to complex tasks and can reveal nuanced capability differences, while fixed prompts risk over-optimization and redundant testing.Source: §5 · p. 8
  • VQA-based VLM evaluation can use targeted questions to assess fine-grained details and produce textual outputs for LLM agents, but its effectiveness depends on VLM capabilities.Source: §5 · p. 8
  • The framework can compare models on specific dimensions, while accumulated capability results could support a recommendation system tailored to user needs.Source: §5 · p. 8

6 Conclusion

The conclusion presents Evaluation Agent as an efficient, promptable alternative to rigid evaluation pipelines. It mimics human evaluation strategies, adapts to user criteria, and is open-sourced to support further research.

  • Evaluation Agent is presented as an efficient, promptable approach that moves beyond rigid evaluation pipelines for visual generative models.Source: §6 · p. 8
  • The framework dynamically adjusts assessments to user-specified criteria, reducing evaluation time while using fewer samples and integrating across tools and models.Source: §6 · p. 8
  • The authors open-source the framework to inspire further research into flexible and efficient evaluation methods for visual generative models.Source: §6 · p. 8

7 Limitations

The framework’s performance depends on evaluation-tool reliability and LLM capability, with tool misalignment or incomplete coverage and LLM inconsistencies constraining evaluations. The authors expect stronger LLMs and more human-aligned toolkits to improve its utility.

7 Limitations

  • Toolkit reliability and LLM capability are the two factors identified as influencing Evaluation Agent performance.Toolkits may misalign with human perception or miss edge cases, while LLMs may produce inconsistent formats or struggle with numerical comparisons.Source: §7 · p. 9
  • Existing evaluation tools may not cover some edge cases or user-specific criteria, even though the framework accepts open-form queries.Tool misalignment with human perception can also negatively affect evaluation accuracy.Source: §7 · p. 9
  • The authors expect the framework’s utility to improve as stronger LLMs and more human-aligned evaluation toolkits are developed.Source: §7 · p. 9

8 Ethical Considerations

The authors warn that the Evaluation Agent could be misused to prompt visual generative models to create harmful content and advise caution with systems involving these models.

8 Ethical Considerations

  • The Evaluation Agent could potentially be used to prompt visual generative models to create unsafe or harmful content.Source: §8 · p. 9
  • Examples of potential misuse include deepfakes and offensive images or videos.Source: §8 · p. 9
  • The authors strongly advise caution when using any system involving visual generative models because improper use could spread harmful content.Source: §8 · p. 9

Supplementary

The supplementary material outlines the pipeline and supporting experiments, query dataset, related work, and additional results. The pipeline iterates between proposing evaluation directions and executing them, adapting exploration to user queries and intermediate observations.

Supplementary

  • The supplementary file maps its sections to pipeline details, experimental details, the open-ended query dataset, related work, and further results and analyses.Source: Supplementary · p. 13

A Detailed Explanation of Pipeline

A.1 Proposal Stage

  • The Plan Agent chooses exploration directions and analyzes results, while the PromptGen Agent designs prompts aligned with those directions.The Plan Agent uses the original query and intermediate observations to choose subsequent directions; both agents provide explanations for their decisions or prompt design.Source: §A.1 · p. 13

A.2 Execution Stage

  • The Execution Stage generates samples with the designed prompts, evaluates them using domain tools or VLM-based VQA, and returns results for Plan Agent analysis.Source: §A.2 · p. 13

A.3 Dynamic Looping.

  • Dynamic looping adjusts evaluation focus using intermediate results, potentially moving to more scenarios, complex situations, or intricate prompts until enough information is gathered.Source: §A.3 · pp. 13–14

B Experiment Implementation Details

The experiments specify an LLM configuration and procedures for T2V and T2I benchmark evaluations; a separate user study produced an open-ended evaluation-query dataset.

B Experiment Implementation Details

  • Main-text experiments used gpt-4o-2024-08-06 at temperature 0.7, with prompts inspired by CoT and ReAct for step-by-step reasoning and explanations.Source: B Experiment Implementation Details · p. 14
  • For T2V, the agent was compared with VBench across 15 dimensions and four open-source models, using five tiers to map metric scores into categorical assessments.The agent was restricted to VBench tools and prompts, and aggregated tiered results across rounds.Source: B Experiment Implementation Details · p. 14
  • For T2I-CompBench, experiments covered four dimensions and four models, excluding Spatial Relationships because of its statistical scoring, limited sample distribution, and lack of leaderboard-based tiering references.Source: B Experiment Implementation Details · p. 14

C.1 Building an Open-Ended User Query Dataset.

  • A user study and subsequent cleaning, filtering, and expansion produced a dataset of 100 open-ended queries about evaluating generative-model capabilities.Source: §C.1 · p. 14

C.2 Dataset Statistics.

The section describes the open-ended query dataset and reviews agent reasoning, tool-use, and recent LLM-based evaluation approaches. It also outlines Claude-backed comparisons with benchmark pipelines.

C.2 Dataset Statistics.

  • Queries were manually labeled by ability, General/Specific, and specific domain; Figure 5 reports a relatively balanced distribution across these dimensions.Source: §C.2 · pp. 14, 16, Figure 5 · p. 15
  • Claude-backed tables compare evaluation accuracy and time with VBench and T2I-CompBench pipelines, with the time tables reporting substantial reductions from the Evaluation Agent.Source: Table 9 · p. 15, Table 10 · p. 15, Table 7 · p. 15

D.1 Agents Planning & Reasoning Methods

  • ToT organizes reasoning as a tree searched breadth- or depth-first, while AoT, GoT, and DoT use algorithmic pathways, graph structures, and a directed acyclic graph, respectively.Source: §D.1 · p. 16
  • CoT methods reason step by step, while task decomposition and feedback-based approaches address single-path reasoning and intermediate errors; the paper identifies CoT as suitable for evaluation tasks.Source: §D.1 · p. 16

D.2 Agent in Action Modelling with Tool-Use

  • Tool use distinguishes agents through environmental perception and action, including selecting or generating API calls from natural-language inputs.Source: §D.2 · p. 16

D.3 Recent Works on LLM-based Evaluation for Visual Generation

  • Recent visual-generation evaluation work includes a scene-graph question-answering agent for text-to-image hallucinations and VideoGen-Eval for text-to-video quality.Source: §D.3 · p. 17

E.1 Experiments on Different Base Models

The experiments compare agent backbones and describe reliability safeguards and open-ended query cases. Claude-backed evaluations are reported as high quality with few samples and minutes, while Gemini showed recurring assessment failures.

E.1 Experiments on Different Base Models

  • Claude-3.5-Sonnet produced high-quality evaluations with few samples and a few minutes across VBench and T2I-CompBench dimensions.Source: §E.1 · p. 17
  • Gemini often chose the wrong evaluation tool, repeated sub-aspects without adapting to observations, and sometimes failed to produce a final response.Source: §E.1 · p. 17, Figure 7 · p. 17, Figure 8 · p. 18
  • The authors recommend GPT-4o for high-precision, aligned prompting, while Claude integration supports use across varied scenarios and system configurations.Source: §E.1 · p. 18

E.2 Agent Reliability and Error Handling

  • ReAct-style reasoning and required justifications are paired with a maximum-round constraint to halt loops and control evaluation cost.Source: §E.2 · p. 19

E.3 Evaluation Results of Open-Ended User Queries

Related papers

Loading 2412.09645v3…