PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounding Generative Verification in 3D Content

Shuanglin Li, Jiaying Lin, Shuquan Ye

arXiv:2610.22360v1cs.CV

TL;DR

3D assets can look convincing while failing geometric or functional requirements, making evidence-grounded verification important for 3D systems. The paper introduces a benchmark and verification-data pipeline, then examines verification for candidate selection and recursive refinement. Humans outperform the strongest tested model, and the downstream studies show headroom for verification-guided reflection.

  • Problem

    3D assets can appear convincing while failing requested geometric or functional requirements, and reliable verification across diverse 3D tasks remains underexplored.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces a ten-task benchmark evaluating decisions, explanations, and consistency, and develops verification data using multi-view fact extraction, controlled changes, and native annotations.

    Source: §1 · pp. 2–3, §3 · p. 3, §4 · p. 5

  • Results

    Humans achieve 87.8% task overall accuracy, exceeding GPT-6 Astra by 21.6 percentage points.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Oracle selection and human-guided recursive refinement show downstream headroom for using verification to support 3D generation.

    Source: §1 · p. 3

    The reported failure analysis presents representative cases in fine-grained geometry, spatiotemporal validity, and visual-evidence grounding, summarized as ten failure modes.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

Recent advances in multimodal large language models (MLLMs) and 3D generation have expanded 3D understanding and synthesis. However, beyond visual fidelity, task-specific correctness grounded in 3D evidence remains a major challenge. We argue that building reliable 3D systems requires not only stronger generation and reasoning, but also verification feedback that identifies unmet requirements and explains their violations, allowing systems to reflect on and refine their outputs. Such feedback must integrate multi-view observations and distinguish genuine inconsistencies from projection-dependent appearances. We make three contributions. First, we introduce 3DVerifierBench, a benchmark spanning 10 tasks that evaluates decisions, rejection explanations, and responses to controlled changes in specifications and candidates. A human study reveals a 21.6 percentage-point gap between humans and the strongest tested model, GPT-6 Astra, while explanation-aware analysis exposes weaknesses in rejection reliability. Second, we analyze verification across fine-grained factual categories, exposing errors in values and target binding. We develop a scalable pipeline combining multi-view fact extraction, controlled 3D editing, and native annotations to produce 14,126 training examples with contradiction explanations and linked counterfactuals. Third, we examine how verification could support 3D generation through candidate selection and recursive refinement. In human-labeled sixteen-candidate pools, oracle selection improves complete factual success from 26.7% to 90.0%. In a separate eight-input study, human-guided recursive refinement satisfies all seven requirements for every input, compared with three under parallel Oracle@8. Together, our benchmark, analysis, and data pipeline establish a foundation for models to reflect on 3D content through reliable verification feedback.

1 Introduction

3D content can look convincing while failing geometric or functional requirements, and reliable verification must explain contradictions across partial, projection-dependent views. 3DVerifierBench evaluates this capability against human judgments and tests whether feedback can support reflection.

1 Introduction

  • Multi-view verification must associate parts across views, resolve occlusion, distinguish geometry from viewpoint or appearance, and check task-specific constraints.The constraints include grasping, identity, and motion; views provide partial, projection-dependent observations rather than complete geometry.Source: §1 · p. 2
  • 3DVerifierBench provides 2,581 human-reviewed examples across ten tasks, with a common answer-and-explanation interface and linked counterfactuals.The benchmark covers object properties, parts, relations, affordance regions, grasps, identity, reference geometry, and motion; counterfactuals vary requirements or candidates.Source: §1 · p. 2
  • Humans achieve 87.8% task overall accuracy, exceeding GPT-6 Astra by 21.6 percentage points.The comparison is reported with Figure 1, bottom.Source: Figure 1 · p. 2, §1 · p. 3
  • In sixteen-candidate pools, oracle selection improves complete factual success by 63.3 percentage points; human-guided refinement satisfies all seven requirements for every input, versus three under parallel Oracle@8.The refinement result comes from a separate eight-input study.Source: §1 · p. 3

3 3DVerifierBench

  • The benchmark evaluates decisions, explanations, and consistency to assess whether verification feedback is correct, informative, and consistent.Balanced Accuracy is computed within each task, with equal task weights overall; explanation-aware scoring checks whether rejected negatives have supported explanations.Source: §3 · pp. 3–5

2 Related Work

  • Prior 3D work measures preference, hierarchical quality, visual verification, or asset consistency and artifact localization, whereas this benchmark targets verifier-side questions.It focuses on explicit requirements, controlled counterfactuals, heterogeneous 3D evidence, and verification-guided refinement.Source: §2 · p. 3, Figure 2 · p. 4

4 Dataset and Benchmark Construction

The construction pipeline combines existing 3D resources, controlled interventions, and native supervision to create task-specific verification data. Its counterfactual branches vary either the requirement or the candidate.

4 Dataset and Benchmark Construction

  • Three complementary strategies construct task-specific data by combining existing 3D resources, controlled interventions, and native supervision.The pipeline aims to identify which requirement evidence contradicts, not merely whether a candidate is wrong.Source: §4 · p. 5
  • Automated checks, model-assisted review, and human review contribute to producing the benchmark.The pipeline diagram presents these review stages after its dataset-construction stages.Source: §4 · p. 5
  • The pipeline prepares sources and multi-view rendered evidence, then constructs Spec-CF and Cand-CF examples.Spec-CF changes the claim with evidence fixed; Cand-CF changes the evidence with the claim fixed.Source: §4 · p. 5

B. Parametric Data Synthesis (Identity)

The data pipeline uses controlled counterfactuals, parametric identity pairs, expert-guided examples, and native task annotations. Human review and source-group exclusion support benchmark quality and training-data separation.

B. Parametric Data Synthesis (Identity)

  • Controlled counterfactuals start from reviewed positive anchors and create negatives by changing either a requirement or the candidate while holding the other fixed.Native annotations take precedence when available; candidate evaluation links are retained only when records share identical model-visible claims.Source: B. Parametric Data Synthesis (Identity) · p. 6
  • For Identity, FaceScape’s bilinear model generates 200 balanced face pairs, with expert-set thresholds validating perceptible identity differences.Expression and texture vary across four nuisance-factor conditions.Source: B. Parametric Data Synthesis (Identity) · p. 6
  • Experts guide FineGeo. and Motion data: they select mesh perturbations for photograph-CAD comparisons and retain motion samples only with unanimous classification.Motion labels distinguish Correct, Unnatural Motion, Incorrect Motion, and Deformation Artifacts.Source: B. Parametric Data Synthesis (Identity) · p. 6

4.1 Native Supervision and Quality Control

  • Six additional domain experts review annotation consistency, perturbation bounds, mesh validity, view coverage, rendering completeness, and temporal order where applicable.Benchmark examples also receive task-specific review for label correctness, evidence sufficiency, and explanation quality.Source: B. Parametric Data Synthesis (Identity) · p. 6, §4.1 · p. 6
  • When native labels define a distinction, the pipeline uses task-specific annotations and evidence rather than forcing every task through one editing backend.Examples include part and relation annotations, action semantics, grasp outcomes, identity parameters, reference photographs, and reviewed motion sequences.Source: §4.1 · p. 6
  • Excluding benchmark source groups and descendants from the construction pool yields 14,126 training examples across 4,011 source groups, with zero source-group overlap.The original construction pool contains 17,223 examples.Source: §4.1 · p. 6

5 Experiments

Experiments show that 3D verification remains difficult for current MLLMs, especially when explanations, fine-grained geometry, or temporal evidence are involved. The section also compares evidence representations and identifies recurring failure modes.

5.1 Setup

  • The experiments evaluate ten models on 2,581 examples, with human scores from 400 examples, 40 per task.Motion inputs use native MP4 when supported and a sixteen-frame sheet otherwise.Source: §5.1 · p. 7

5.2 How Good Are Current MLLMs in 3D Verification?

  • 87.8% human Overall score versus Astra’s 66.2% on a different test set; Table 1 reports Astra highest at 69.4% Overall, followed by Gemini-3.8-Flash at 68.4%.The human comparison uses different testing data from Table 1.Source: §5.2 · p. 7
  • 36.5% of 9,488 correct rejections failed the reference-semantic Judge, and Astra’s explanation-aware score was 65.8%.Across models, ExpOA fell 3.6 to 28.1 points below Overall score.Source: §5.2 · p. 7

5.3 Limitations of MLLMs in 3D Verification

  • Representative failures span fine-grained geometry, spatiotemporal validity, and grounding claims in visual evidence.Examples include accepting an incorrect cake-layer count and a stone-like appearance despite human assessments to the contrary.Source: §5.3 · p. 8, §5.4 · p. 9

5.4 How Does Evidence Representation Affect Verification?

  • Multi-view generally outperforms raw 3D input, while motion results vary by model and metric, with no representation consistently best.Native video reaches 59.7% in binary accuracy; sixteen frames reach 43.8% in 4-class accuracy.Source: §5.4 · p. 8

5 Part Relation

Examples expose errors across part relations, grasps, grounding, affordances, and identity. In generation studies, oracle selection and human-guided recursive refinement show headroom for satisfying explicit facts, though factual success does not ensure asset quality or source fidelity.

5 Part Relation

  • Astra accepted a direct-touch claim even though humans judged that the red and blue parts did not touch.Astra described the blue cushion as resting against the red backrest.Source: §5 · p. 9

7 Grasp Validity

  • Astra accepted a parallel-jaw grasp that humans judged would slip, and also accepted a handle claim when no handle was present.The jaws were on adjacent sides rather than providing opposing contacts.Source: §7 · p. 9

4 Part Grounding

  • Astra labeled the highlighted support box beneath a cushion as the seat cushion.This is a region-binding error: it assigns a plausible part description to the wrong region.Source: §4 · p. 9

6 Affordance Region

  • Astra accepted pressing as suitable for a highlighted camera dial, although human review found that the region did not support pressing.Source: §6 · p. 9

8 Identity Preservation

  • Astra judged that an edit preserved identity despite human-noted changes to the lower face and chin; motion verification uses native video or sixteen-frame sampling by default.Source: §8 · pp. 9–10

5.5 How Can Verification Feedback Help 3D Generation Models Reflect?

  • 8/8 human-guided recursive refinements met all seven facts, versus 3/8 for Parallel Oracle@8; human-oracle selection raised full factual success from 26.7% to 90.0%.Only 7/8 refined assets avoided severe defects, and 6/8 also preserved source fidelity.Source: §5.5 · pp. 10–11

6 Discussion and Conclusion

The paper frames reliable verification feedback as necessary for identifying unmet requirements and explaining violations in 3D outputs. It reports using generative AI tools in research and data construction, and specifies its facial-geometry source and exclusions.

6 Discussion and Conclusion

  • Reliable reflection on 3D outputs requires determining whether requirements are met and explaining violations; the paper connects verification feedback to selecting and revising assets.The authors note that satisfying explicit facts does not ensure overall quality or source fidelity.Source: §6 · p. 11

AI use statement

  • Generative AI tools assisted research ideation and execution, including parts of data construction, writing, and retrieval; the authors reviewed AI-assisted work and take responsibility for the final content.Source: AI use statement · p. 12

Ethics statement

  • Facial geometry uses a FaceScape bilinear statistical model compressing 847 subjects into a 50-dimensional PCA space, without using or redistributing source images, textures, meshes, demographic metadata, or identity-specific records.Source: Ethics statement · p. 12

A Tasks and Data Composition

The appendix defines the benchmark’s task types and inputs, describes its benchmark and training resources, and documents source grouping and preparation procedures.

A.1 Task Definitions

  • Tasks cover object, part, material, grounding, relations, affordances, grasps, identity, fine-grained geometry, and motion, with counterfactual pairs or triplets linking changed specifications or candidates.A counterexample counts as a measured counterfactual only when records, shared context, and sibling links are present.Source: §A.1 · p. 16

A.2 Model-Visible Inputs

  • Model inputs include frozen multi-view sheets, marked evidence, reference comparisons, or motion video and frames; reference answers and labels are excluded.Identity uses a 2-by-8 asset-pair sheet, while fine-grained geometry uses a 2-by-7 reference/render sheet.Source: §A.2 · p. 16

A.3 Benchmark and Training Resource

  • The benchmark contains 2,581 examples, while the source-disjoint Training Resource contains 14,126 examples after excluding benchmark source groups and descendants.The training resource has 5,660 True anchors and 8,466 False examples.Source: §A.3 · p. 16, Table A1 · p. 17

A.4 Source Grouping

  • Benchmark and training releases have zero source-group overlap, and source groups join assets with their counterfactual branches and edited descendants.The release contains 1,123 benchmark source groups and 4,011 training source groups.Source: §A.4 · pp. 16–17

B.1 Source Preparation

  • Preparation aligns task-specific views and inputs, including eight-camera face pairs, seven-direction geometry sheets, and chronological sixteen-frame motion sheets.The explanation Judge accepts paraphrases only when they identify the same violated fact and contradiction as the trusted reference.Source: §B.1 · pp. 17–18

B.2 Facts, Anchors, and Spec-CF

The pipeline creates specification- and candidate-side counterfactuals from reviewed 3D facts, then checks that changed requirements or assets produce valid, evidence-grounded contradictions. Expert review and linked supervision support data quality and evaluation of decisions, explanations, and consistency.

B.2 Facts, Anchors, and Spec-CF

  • Spec-CF changes a requirement while retaining the candidate, and its negative reference identifies the changed requirement and the candidate’s actual property.Source: §B.2 · pp. 18–19

B.3 Cand-CF: View Selection, Masks, and 3D Realization

  • Cand-CF selects a view exposing the intended edit, reviews masked or unmasked image edits, and verifies the realized change against the preserved anchor.The posterior change mask records the realized edit and is distinct from the conditioning mask for the intended edit.Source: §B.3 · p. 19

B.4 Human Review

  • Two domain experts spot-check 10% of outputs at construction checkpoints, and six independent experts verify benchmark candidates before retention.Review covers labels, evidence, perturbation validity, mesh and rendering quality, view coverage, and motion temporal consistency.Source: §B.4 · p. 19

B.5 Supervision and Expansion

  • The Training Resource retains task evidence, claims, labels, contradiction explanations, and source and counterfactual links across pairs and triplets.It contains 5,157 Spec-CF pairs, 3,309 Cand-CF pairs, and 2,807 complete triplets; these are distinct from benchmark consistency subsets.Source: §B.5 · pp. 19–20

C.1 Answer Scores and Aggregation

  • Balanced Accuracy averages True and False recall, and Overall Accuracy averages task-level Balanced Accuracy equally across ten tasks.Source: §C.1 · p. 20

C.2 Reference-Semantic Explanation Scores

  • Explanation-aware scores check whether negative explanations match the reference contradiction, with FalseExpAcc measuring acceptance among correct rejections.The Judge receives task, claim, model explanation, and reference explanation, but not visual evidence or model identity; True answers are not similarly checked.Source: §C.2 · p. 20

C.3 Counterfactual Pairs and Triplets

  • Consistency analysis pools 661 specification-side pairs and 727 fixed-claim candidate-side pairs, and evaluates triplet consistency on 615 complete triplets from six tasks, excluding Grasp Validity, Identity Preservation, and Motion Understanding.Fine-grained Geometry contains 30 reference-geometry pairs analyzed separately and not included in these consistency metrics.Source: §C.3 · pp. 20–21

C.4 Motion-Issue Accuracy

The evaluation combines task-level decision and explanation measures with clustered uncertainty estimates and a shared model-answer contract. Its results show that decision balance varies substantially across models.

C.4 Motion-Issue Accuracy

  • Motion-Issue Exact requires both a correct binary answer and the correct primary issue, while joint explanation recall also requires an accepted explanation.Source: §C.4 · p. 21

C.5 Uncertainty

  • Main benchmark intervals use 10,000 cluster-bootstrap replicates, while the human comparison uses 20,000 paired replicates on the same examples.Generation gain intervals resample input pools and retain all candidates and labels within each sampled input.Source: §C.5 · p. 21

D.1 Subject Models

  • Each model receives all 2,581 benchmark examples under a shared answer-and-explanation contract, while motion input format follows model capabilities.Some models receive native video and others ordered sixteen-frame sheets, which may omit continuous-motion and playback-speed cues.Source: §D.1 · pp. 21–22

D.2 Fact and Explanation Analysis

  • Fact-level analysis assigns each decision-relevant property to one of thirteen categories, describing observable decisions and rejection reasons rather than hidden reasoning.Source: §D.2 · p. 22

D.3 Role Separation

  • Subject models generate predictions, while humans separately provide baselines, targeted explanation review, generation labels, and recursive feedback.Source: §D.3 · p. 23

D.4 Explanation Judge Implementation

  • The frozen explanation Judge checks correctly rejected explanations against the reference using task and claim text, without images or subject-model identifiers.Source: §D.4 · p. 23

E.1 Overall Decision Behavior

  • Models differ substantially in decision balance: Gemini-3.8-Flash has True and False recall of 72.6% and 73.4%, while other models show strong directional biases.Manual inspection confirmed that the observed skew reflects model answers rather than errors in the judgment procedure.Source: §E.1 · p. 23

E.2 Explanation-Aware Verification

Explanation checking changes model comparisons and exposes failures in rejection explanations, while human comparisons reveal substantial remaining performance gaps. The paper also reports limits in evidence matching and task coverage.

E.2 Explanation-Aware Verification

  • The model ordering changes after explanation checking: GPT-5.6-Sol has lower OA than Gemini-3.8-Flash but higher ExpOA, reversing their relative standing under explanation-aware evaluation.Source: §E.2 · p. 23

E.3 Identity Preservation and Motion Understanding

  • Identity Balanced Accuracy ranges from 49.5% to 59.0%, and Motion Understanding has 134 accepted explanations among 428 correct rejections.Only 31.3% of correct motion rejections are accepted; deformation negatives account for 88 correct rejections and no accepted explanations.Source: §E.3 · p. 24, Table E4 · p. 25, Table E6 · p. 26

E.4 Native 3D Baselines

  • PointLLM-v1.2-7B reaches 50.0% Balanced Accuracy with zero False recall, while PointLLM-R-7B reaches 8.3% Balanced Accuracy.Neither native point-cloud model succeeds on an eligible counterfactual pair or triplet.Source: §E.4 · pp. 24–25, Table E7 · p. 26

E.5 Pair and Triplet Consistency Across Static Tasks

  • Gemini-3.8-Flash leads Triplet success at 49.6% (305/615), but Astra leads when both negative explanations must pass, at 40.5% (249/615).The explanation-aware requirement changes the ordering; Astra also leads both directional pair metrics.Source: §E.5 · pp. 26–27, Table E8 · p. 27

F Scope and Release

  • The benchmark covers task-specific evidence from finite views, marked regions, reference comparisons, and animations; the separate source-disjoint Training Resource contains 14,126 examples.The release has zero benchmark–training source-group overlap, and source assets and annotations remain subject to upstream licenses.Source: F Scope and Release · p. 27

G.1 Human Evaluation on the Same Task Subsets

  • On the same 400 examples, humans score 87.8% Overall Accuracy and GPT-6 Astra 66.2%, a 21.6-point difference.Motion is the only task with non-identical evidence formats, and the paired difference has a 95% interval of [16.8, 26.2] points.Source: §G.1 · pp. 28–29, Table G2 · p. 28

G.2 Public Image Conditions and Fixed Requirements

The generation study fixes image conditions and seven requirements before candidate review, then compares selection and human-guided refinement using factual success alongside asset quality and source fidelity.

G.2 Public Image Conditions and Fixed Requirements

  • For 40 screened source images, human annotators fixed seven claims—one identity and six local requirements—before generation and candidate review.The requirements cover counts, parts, geometry, relations, region binding, and surface appearance.Source: §G.2 · p. 29

G.3 Generation and Human Annotation

  • Two generators produced eight candidates per source, yielding 640 assets; multiple reviewers assessed facts, defects, and fidelity, with disagreements adjudicated.The final annotations include 4,480 fact judgments and 640 labels for each asset-level field.Source: §G.3 · pp. 29–30

G.4 Selection Metrics and Pooled Results

  • 36/40 inputs achieve full factual success under the human oracle, while 34/40 meet Strict Full, which also requires no severe defect and source fidelity.Oracle selection uses final human labels and is a post-hoc upper bound, not a deployed verifier.Source: §G.4 · p. 30, Table G4 · p. 30

G.5 Human-Guided Recursive Refinement

  • Across eight recursive cases, all eight final assets satisfy the seven facts, but only seven avoid severe defects and six also preserve source fidelity.Refinement can be non-monotonic, and the output-count-matched parallel prefixes do not match computational or human-effort budgets.Source: §G.5 · pp. 30–31

H Fine-Grained Facts and Explanation Reliability

The fine-fact analysis groups verification errors by the property being checked and examines both decision accuracy and whether rejection explanations identify the reference contradiction. It finds substantial weaknesses across categories and representative failures in binding, geometry, and motion judgments.

H.1 Analysis Procedure

  • The analysis contains 1,778 verification units, including 1,444 paired comparisons, 200 Identity comparisons, and 134 Motion Understanding sequences.The content-based analysis units differ from benchmark examples and eligible consistency comparisons.Source: §H.1 · p. 32

H.2 Thirteen Fine Fact Categories

  • Identity preservation concerns whether identity persists across editing, unlike identity/existence checks for category or presence in a single relation.Fine-grained Geometry and Motion Understanding contribute shape/geometry and motion-compliance categories, respectively.Source: §H.2 · p. 32, Table H1 · p. 32

H.3 Fact-Wise Verification Performance

  • Mean ExpBA is lowest for motion compliance at 31.6%, grasp validity at 35.7%, and identity preservation at 43.8%.Connectivity/contact reaches 67.8%; motion direction has only four pair units, so its rate is a small diagnostic group.Source: §H.3 · p. 32, Table H2 · p. 33, Table H3 · p. 33

H.4 Observable Errors and Fine Facts

  • Action compatibility, shape/geometry, and region binding contribute the largest shares of answer errors at 18.7%, 16.2%, and 11.4%, respectively.Incorrect values state the wrong property or state; target/binding errors associate a property with the wrong entity or region.Source: §H.4 · p. 34, Table H4 · p. 34

H.5 Correct Rejections with Reference-Mismatched Explanations

  • Across ten models, 36.5% of correct rejections fail the reference-semantic Judge; the rate is 12.9% for Astra and reaches 100.0% for Fine-grained Geometry.The pooled count is 3,462 of 9,488 correct rejections.Source: Table H5 · p. 34, §H.5 · p. 35, Table H6 · p. 35

H.6 Representative Failure Cases of GPT-6 Astra

  • Representative Astra failures include binding a plausible cushion description to the wrong furniture part and missing motion naturalness or deformation problems.Other examples show absent parts, invalid grasp contacts, and accepted reference-geometry mismatches.Source: §H.6 · p. 35

Related papers

Loading 2610.22360v1…