PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Visual Quality: Evaluating Physical Consistency under Ego-Motion with EgoGenEval

Yilin Long, Chenming Zhu, Zitang Gou, Jingli Lin, Tai Wang

arXiv:2609.11172v1cs.CVcs.AI

TL;DR

Visual generators can produce photorealistic images while failing to execute instructed ego-motion and preserve scene state, a gap existing benchmarks do not jointly measure. EgoGenEval provides a pose-free, geometry-grounded benchmark with separate CMG and SSP metrics, then uses EgoGen-Train to test supervised improvement. The results show a persistent motion–state gap: pairwise teacher-forced supervision improves camera-motion grounding more reliably than scene preservation, motivating trajectory-centric training.

  • Problem

    Existing benchmarks do not jointly evaluate pose-free camera-motion execution and target-view scene-state preservation during rollouts, despite the importance of both for spatial reasoning and visual simulation.

  • Method

    EgoGenEval uses 1,400 cases and 2,360 target views to score CMG and SSP separately across atomic, chained, and inverse-cycle ego-motion, then EgoGen-Train supports controlled SFT studies.

  • Results

    Across 16 pose-free systems, no evaluated system excels at both axes; pairwise SFT improves camera-motion grounding more than scene preservation and does not reliably improve both together.

  • Takeaways & Limitations

    Physical consistency requires coupling camera transformation with persistent cross-view scene state, motivating self-conditioned trajectory supervision with explicit pose and visibility supervision.

  • Takeaways & Limitations

    The benchmark covers static indoor scenes, four camera-action families, and rollouts of at most three steps, while learned evaluators may conflate evaluator limitations with generator failure.

Abstract

from arXiv · show

Recent visual generators produce high-fidelity images yet often violate physical consistency under ego-motion, limiting their use for spatial reasoning and embodied planning. Existing benchmarks largely focus on isolated images or single-step quality, leaving this challenge underexplored. We introduce EgoGenEval, a geometry-grounded, pose-free benchmark designed to evaluate the physical consistency of visual generators under ego-motion, and organize our study into two parts. (1) EgoGenEval contains 1,400 cases and 2,360 target views spanning single-step and multi-step ego-motion. It separately measures Camera Motion Grounding (CMG) and Scene State Preservation (SSP), with both metrics validated against blinded human judgments. Evaluating 16 pose-free generators together with two pose-conditioned references reveals that current models struggle to execute camera motion while maintaining scene state, and that no system performs well on both axes at once. (2) To examine whether benchmark-derived data can improve these capabilities, we build EgoGen-Train from the same geometry-grounded pipeline and run controlled SFT studies. These show that pairwise supervision does not reliably improve camera-motion grounding and scene-state preservation together: even at the full training pool and the longest budget, scene preservation gains a fraction of what camera motion does. This points to the pairwise teacher-forced objective itself as the binding constraint, motivating a trajectory-centric paradigm that couples self-conditioned rollouts with explicit pose and visibility supervision.

1 Introduction

EgoGenEval addresses the gap between visual quality and physical consistency by separately evaluating camera-motion execution and scene-state preservation in pose-free rollouts. Across benchmarking and controlled SFT studies, current systems show a persistent motion–state gap that pairwise supervision does not reliably resolve.

  • Benchmark findings: The benchmark contains 1,400 cases and 2,360 target views, measures CMG and SSP separately, and validates both scores against blinded human rankings.It spans atomic motions, chains, inverse cycles, and varying visual context.
  • Benchmark findings: EgoGenEval evaluates 16 pose-free generators and two pose-conditioned references, revealing that no system excels at both camera-motion grounding and scene-state preservation.Similar aggregate scores can conceal substantially different CMG–SSP profiles.
  • Benchmark findings: Current generators often execute the motion direction but miss its requested scale, while correctly reaching a viewpoint still does not ensure object, layout, or appearance preservation.Most direction-correct outputs miss the requested displacement by more than ±20%.
  • Benchmark findings: Rollouts amplify inconsistency: Chain cases reduce mean SSP by 0.164 and CMG by 0.059, while 66.3% of inverse-cycle return steps remain near-static.Increasing context from K=1 to K=4 raises mean CMG from 0.519 to 0.582 but lowers SSP from 0.539 to 0.504.
  • Training studies: EgoGen-Train studies show pairwise teacher-forced SFT improves camera-motion grounding far more than scene preservation and does not reliably improve both axes across backbones.On Qwen, gains are +0.303 in CMG versus +0.069 in SSP; the asymmetry persists with the full pool and longest budget.
  • Training studies: The results identify the pairwise teacher-forced objective as the binding constraint and motivate self-conditioned trajectory supervision with explicit pose and visibility supervision.This conclusion holds even on easiest single-step transitions, where error accumulation is precluded.

2 Related Work

Prior benchmarks cover image quality, geometry, camera control, or world-model behavior, but EgoGenEval combines pose-free language-conditioned camera motion with explicit cross-view scene-state scoring. Its scope comparison distinguishes support for context, camera control, multi-step evaluation, and state preservation.

  • Image-generation and geometric-consistency benchmarks: EgoGenEval complements image-generation and geometric-consistency benchmarks by jointly scoring language-instructed camera motion and target-view object state.Prior settings largely emphasize compositional prompts, single-output editing, or geometric residuals.
  • Benchmark design context: Figure 1 organizes EgoGenEval around atomic, chain, and inverse-cycle protocols, separate CMG and SSP axes, and leaderboard profiles that expose different performance patterns.Figure 2 further depicts geometry-grounded mining, language-conditioned protocols, filtering, and quota-balanced sampling.
  • Scope comparison: Table 1 compares six representative benchmarks across visual context, natural-language camera control, multi-step evaluation, and explicit cross-view scene-state scoring.The symbols ✓, ✗, and △ indicate full, absent, and partial coverage.
  • Novel-view synthesis and camera-controlled generation: Unlike novel-view synthesis and camera-controlled generation, EgoGenEval evaluates pose-free generators using only magnitude-specified natural-language camera instructions.Related methods typically assume calibrated views, known poses, or explicit camera trajectories.
  • World models and embodied evaluation: EgoGenEval differs from world-model and embodied benchmarks by evaluating pose-free, language-conditioned camera motion and target-view object-state preservation in static indoor scenes.Interaction dynamics and simulator task success remain outside its scope.

3 EGOGENEVAL: Benchmark Design and Construction

EgoGenEval evaluates magnitude-specified camera motions from current views, optional same-scene context, and controlled atomic or multi-step protocols. Its geometry-grounded construction mines valid transitions, forms rollout cases, and applies filtering before producing scored targets.

  • Task and context: The task maps a current view, a natural-language instruction specifying direction and approximate magnitude, and optional auxiliary views to a generated target view.The evaluated model receives images and language rather than camera poses.
  • Task and context: Visual context ranges from K=1 to K=4 images, with the current view first and additional same-scene views supplying broader evidence without matching an evaluated target.K=1 uses only the current view; larger K adds auxiliary views.
  • Motion taxonomy and protocols: The motion taxonomy retains dominant forward/backward translation, lateral translation, yaw, and pitch, while evaluating atomic transitions, three-step chains, and inverse cycles.Each chain transition and cycle step is scored separately.
  • Benchmark construction: Construction mines source–target pairs with controlled gaps, computes relative 6-DoF motion, retains transitions satisfying overlap and visibility constraints, and forms atomic, chain, and inverse-cycle cases.A frozen ranking procedure uses overlap, pose, and novelty criteria during case formation.
  • Benchmark composition: The final benchmark contains 1,400 cases from 771 scenes and 2,360 scored target views, with each unique ground-truth target annotated once and labels frozen for evaluation.The detection vocabulary combines generated annotations with a fixed 36-class indoor vocabulary.

4 Evaluation Setup and Metrics

EgoGenEval evaluates physical consistency under ego-motion by separating camera-motion realization from preservation of the same static environment. Its geometry-grounded metrics use pose-free image-and-language inputs, coverage-aware object comparisons, and human-alignment checks.

  • Evaluation Protocol: The benchmark evaluates 16 pose-free systems using images and language instructions, with two trajectory-conditioned world models as references.Each rollout step uses the anchor image initially and the preceding generated output thereafter; video models pass their final frame forward.
  • CMG: Is the Requested Motion Realized?: CMG measures whether generated transitions realize the requested ego-motion, using estimated relative motion for ground-truth and generated transitions.Translation is calibrated per target transition with a shared scale factor, while direction and rotation remain unchanged across systems.
  • CMG: Is the Requested Motion Realized?: CMG jointly scores directional agreement and magnitude accuracy, averaging step scores with nonzero translation and rotation instructions.The metric isolates the instructed dominant component; supplementary full-pose diagnostics capture unintended off-axis drift.
  • SSP: Is the Environment Preserved?: SSP measures environment consistency at the intended target view through object retention, pairwise spatial topology, depth ordering, and appearance integrity.Object matching uses fixed detection settings and one-to-one identity-aware matches; multi-step cases weight both average and worst-step performance.
  • Aggregation and Validation: Scores are normalized so higher is better, averaged within Atomic, Chain, and Cycle protocols, then macro-averaged; Overall weights CMG and SSP equally.A model-independent mask removes 44 unevaluable cases, retaining 1,356 cases and 2,265 steps for every system; RefSim and VisQual remain auxiliary diagnostics.
  • Aggregation and Validation: Both metrics align closely with blinded human rankings, while evaluator substitutions preserve system rankings across pose, depth, detection, vocabulary, and temporal components.Human alignment reaches Spearman ρ=0.943 and Kendall τb=0.867; robustness checks report ρ=0.943 for CMG and ρ=1.000 for SSP under key substitutions.

5 Experimental Results

EgoGenEval shows that motion execution and scene preservation are distinct, weakly coupled capabilities: systems may realize camera direction or viewpoint while distorting magnitude or losing scene state. Diagnostics further show that multi-step rollouts and extra context expose failures that conventional visual-quality scores obscure, while pairwise SFT improves motion much more reliably than preservation.

  • 5.1 Main Results: The best pose-free Overall reaches 0.662, yet no system is strong on both CMG and SSP, and performance declines from Atomic to Chain and Cycle.The leaderboard defines Overall as the unweighted mean of CMG and SSP protocol averages; the GT-target oracle reaches 0.940.
  • 5.1 Main Results: HY-WorldMirror achieves the highest CMG at 0.847 but only 0.481 SSP, whereas FLUX.2-dev reaches 0.554 SSP with CMG of 0.448.These profiles show why a single Overall score can conceal whether a system primarily lacks motion execution or scene preservation.
  • 5.1 Main Results: Among direction-correct outputs, 59.5%–70.7% miss the requested displacement by more than ±20%, with the error worsening from small to large motions.Direction accuracy is comparatively tractable, but magnitude remains a systematic weakness; HY-WorldMirror instead over-scales translations despite low angular error.
  • 5.1 Main Results: All 16 pose-free systems lose SSP from Atomic to Chain, with a mean drop of 0.164, while inverse cycles make 66.3% of return steps near-static versus 37.4% outbound.The return-step disparity isolates self-conditioning as a separate rollout challenge rather than a simple consequence of single-step quality.
  • 5.1 Main Results: Increasing context from K=1 to K=4 raises mean CMG from 0.519 to 0.582 but lowers SSP from 0.539 to 0.504, as models often imitate auxiliary views instead of fusing them.At K=4, 62.2% of outputs are CLIP-closest to an auxiliary view, and auxiliary-confused rows have lower SSP by 0.064.
  • 5.2 Supervised Fine-Tuning: Full SFT raises Qwen Overall from 0.495 to 0.681, but CMG gains +0.303 while SSP gains only +0.069, and matched-backbone controls find no scene-preservation gain.The asymmetry persists with longer training, cleaner data, and the full pool, supporting a limitation of pairwise teacher-forced supervision rather than dataset scale.

6 Discussion and Limitations

The discussion identifies persistent scene-state maintenance as a central bottleneck: models may realize camera direction without preserving metric motion, objects, layout, or appearance. The benchmark is limited to static indoor scenes and short rollouts, and its learned evaluators introduce an additional interpretive boundary.

  • Persistent scene state remains difficult to maintain even when models produce the appearance of a camera action.The observed failure involves coupling camera transformation with cross-view state, rather than establishing a specific internal representation.
  • Direction can be predicted without metric control, while auxiliary views may be imitated rather than integrated and self-generated views provide weak anchors for inverse actions.Small portable objects are also retained less often than large furniture.
  • EgoGenEval covers static indoor scenes, four camera-action families, and rollouts of at most three steps, leaving dynamic scenes, wider motions, and longer horizons open.
  • Because CMG and SSP rely on learned perception models, low scores can reflect evaluator limitations rather than true generator failures, despite tracking blinded human rankings.

7 Conclusion

The paper concludes that EgoGenEval provides an axis-specific, rollout-aware diagnostic of physical consistency, while EgoGen-Train studies show that pairwise teacher-forced supervision does not close the camera-motion and scene-preservation gap. These findings motivate self-conditioned trajectory training with explicit pose and visibility supervision.

  • EgoGenEval measures camera-motion and target-view environment fidelity in a 1,400-case benchmark with metrics validated against aggregate human system rankings.
  • Across 16 pose-free systems, similar Overall scores conceal distinct CMG–SSP profiles, scene state degrades in Chain rollouts, and extra views improve motion grounding without improving preservation.The benchmark therefore diagnoses physical consistency by axis and rollout protocol rather than by one aggregate score.
  • Pairwise teacher-forced supervision improves camera-motion grounding but does not reliably improve scene-state preservation across backbones, budgets, or even the easiest single-step cases.The asymmetry persists at the full training pool and longest budget.
  • EgoGenEval spans atomic, three-step chain, and inverse-cycle protocols with controlled visual context, while the primary evaluation comprises 16 pose-free generators and separately reported pose-conditioned references.
  • The benchmark uses posed indoor RGB-D observations from four datasets selected through feasibility-constrained balancing rather than population estimation.The final case counts are 1,003 ScanNet++, 210 ScanNet, 165 HyperSim, and 22 Matterport3D cases.

C.1 Evaluation scope

The evaluation runs fresh model calls in autoregressive short rollouts, using generated outputs as later current views and fixed natural-language motion instructions derived from ground-truth transitions. CMG and SSP then score motion realization and scene-state structure through separate aggregation and perception pipelines.

  • Each transition is a fresh model call, but later steps use the preceding generated output as the current view rather than a ground-truth intermediate.Cycles cannot close by supplying the answer view as an extra reference.
  • The benchmark evaluates forward/backward translation, lateral translation, yaw, and pitch across atomic transitions, three-step chains, and inverse cycles.Each chain contributes three scored targets and each cycle contributes two.
  • Instructions are deterministically generated from each ground-truth relative pose, specifying the dominant action’s direction and magnitude while withholding full pose, target images, depth, labels, and detector prompts.The pose-to-text fill rules hold for all 2,360 instructions with zero exceptions.
  • CMG scores the instructed component using direction and magnitude agreement after transition-specific translation calibration, while SSP evaluates detections, correspondences, planar relations, depth order, and object integrity.The two metrics use distinct evaluator pipelines and SSP is inspectable from detections through spatial and depth checks.
  • The benchmark’s CMG magnitude interpretation remains conditional on a learned pose backend and GT-assisted, transition-specific calibration shared across systems.
  • Case-level CMG averages step scores equally, so failures within multi-step cases can be diluted rather than isolated by a worst-step term.Atomic, chain, and cycle case means then receive equal protocol weight.

D.4 Full metric decomposition

The full decomposition shows that motion direction is easier than magnitude, while privileged trajectory control improves CMG without ensuring scene preservation. Human validation supports system-level use of the automatic metrics, but moderate item-level agreement limits fine-grained interpretation.

  • GPT-Image-2 reaches 0.887 Atomic direction accuracy but 0.582 gated magnitude, and every pose-free system shows the same direction-over-magnitude ordering.The gap between direction and gated columns measures credit attributable to sense alone.
  • Privileged 6-DoF control lifts the pose-conditioned references above pose-free systems on CMG, yet their SSP remains within the pose-free range.Trajectory control alone does not guarantee target-view content preservation.
  • Human validation ranks anonymized systems using motion direction, magnitude, target-view agreement, visible objects, organization, relative depth, and recognizable appearance rather than visual polish alone.
  • The reported 0.348 CMG mean correlation and moderate item-level values support case-, protocol-, and system-level comparison rather than exact per-example grading.

E.3 Comparison with conventional image metrics

Conventional image-quality metrics provide useful but limited alignment with human system rankings, while the composite CMG and SSP scores summarize aggregate capability rather than universally superior item-level metrics.

  • Comparison with conventional image metrics: Overall best matches the human system ordering, followed by CLIP-IQA, whereas NIQE and LPIPS provide weaker descriptive comparisons on the six-system panel.The comparison uses identical outputs and the same step-to-case-to-protocol aggregation; with six systems, the differences are not a powered significance claim.
  • Component-level comparison: CMG reaches mean ρ = 0.348 and SSP reaches 0.334 at item level, comparable to their strongest component or conventional metric alternatives.CMG is comparable to negative dominant-action absolute error, while SSP is comparable to Spatial, Integrity, and PSNR.
  • Interpretation: The composite scores are justified as pre-defined summaries for aggregate capability comparison, not as universally superior per-item metrics.Paired bootstrap differences for the compared item-level metrics include zero in several cases.
  • Uncertainty: The uncertainty analysis uses 2,000 parent-case bootstrap resamples, with a median Overall interval half-width of approximately 0.010.The intervals quantify benchmark case-sampling uncertainty only because each model-step has one frozen output and stochastic generation variance is excluded.

F Evaluator Robustness

Evaluator and benchmark-design perturbations largely preserve the main rankings, while temporal aggregation and matcher choices reveal narrower sensitivities that qualify interpretation of close comparisons.

  • Scope of robustness: The benchmark’s evaluator interventions establish rank robustness, not numerical equivalence or ground-truth accuracy of any single learned evaluator.Each intervention changes one component while holding outputs, ground-truth actions, and the remaining aggregation fixed.
  • Evaluator robustness: Replacing major CMG and SSP evaluator components preserves broad rankings, including CMG ρ = 0.943 with VGGT and SSP ρ = 1.000 across depth backends.Threshold, vocabulary, and temporal-aggregation substitutions also retain high correlations and often unchanged Top-1 or Top-3 results.
  • Temporal aggregation: Under matched temporal functionals, the robust finding is that Chain degrades both axes, while the relative size of CMG and SSP drops can change with aggregation.The official SSP-versus-CMG contrast is not aggregation-invariant, and a shared temporal-weight sweep changes its sign near a minimum-step weight of 0.26.
  • Sampling robustness: Scene-cluster resampling increases the median Overall interval half-width from 0.0100 to 0.0117, while the pose-free Top-1 comparison remains unambiguous.The primary case bootstrap stratifies by instruction subtype, whereas scene sampling preserves all cases from selected scenes.
  • Source sensitivity: Removing any source leaves rank correlation ρ = 0.968–1.000 with the full ranking, and GPT-Image-2 remains Top-1 even after removing ScanNet++.Source-specific common-protocol correlations remain ρ = 0.897–0.959, despite highly unequal source coverage.

G.4 Score-weight sensitivity and benchmark size

Benchmark conclusions are stable across broad score-weight ranges and modest subsampling, while added context improves motion grounding but slightly reduces scene preservation through imperfect view integration.

  • Score-weight sensitivity: GPT-Image-2 remains Top-1 for λ ∈ [0.25, 0.75] with minimum rank correlation ρ = 0.926, while extreme motion weighting changes the leader.Here Overallλ = λ CMG + (1 − λ) SSP; Kling becomes Top-1 at λ = 0.90.
  • Benchmark size: The 16-system ordering is already stable at 140 cases, with median ρ = 0.985 and Top-1 match 1.000; the full benchmark mainly reduces score error.MAE decreases from 0.0121 to 0.0023, but the check is not a future-system power analysis.
  • Context integration: At K = 4, 62.2% of outputs are feature-nearest to an auxiliary view but only 0.6% are near-pixel copies, indicating partial feature-space attraction rather than literal copying.Strict auxiliary-confused outputs score 0.064 lower SSP than other rows, while attraction and SSP changes correlate only weakly at ρ = −0.109.
  • Cycle behavior: Ten of 16 systems are near-static on more than half of their return steps, and 34.1% of Cycle cases move outbound but become near-static on return.The cycle diagnostic motivates free-running inverse-action supervision but does not identify whether suppression comes from the generator, visual drift, or pose evaluator.

H.6 Off-axis and full-pose error

Full-pose diagnostics show that dominant-axis action success can coexist with substantial residual or scale errors, while rollout galleries connect these errors to object loss and cycle-return failures.

  • Off-axis diagnostics: The residual translation diagnostic subtracts the complete ground-truth translation before measuring unintended motion, so legitimate off-axis motion is not penalized.For rotations, the SO(3) geodesic term captures roll error and axis coupling in addition to cross-axis error.
  • Atomic behavior: Atomic galleries show correct-direction outputs with large magnitude errors, while wrong-direction outputs and degraded renders can lose object matches despite plausible scenes.The gallery reports estimated action, direction agreement, and matched-over-evaluable ground-truth object counts across action families and systems.
  • Rollout behavior: Chain rollouts show matched-object counts falling toward zero by the final step, and cycles show correct outbound motion followed by wrong-direction or weak return.These examples make trajectory drift concrete even when individual steps are not grossly wrong.
  • Training-probe scope: The EgoGen-Train study uses 66,214 trajectories and 108,213 teacher-forced edit pairs, but its continuation changes the optimization budget and is not a pure data-scale comparison.The full-clean continuation adds 9,527 updates after the initial 4,000, totaling 13,527 updates.
  • Backbone-dependent SFT: Qwen gains in CMG but loses SSP F1 by −0.012, while OmniGen2 shows decreases of −0.037, −0.058, and −0.049 across SSP F1, spatial relations, and integrity.The second backbone therefore supports action–state divergence rather than a general claim that SFT improves CMG.

I.4 Budget, subgroup, and qualitative diagnostics

Controlled diagnostics show that pairwise SFT changes camera-motion grounding and scene-state preservation unevenly, with effects dependent on backbone, training budget, and subgroup. Qualitative audits and benchmark design details further delimit how these results should be interpreted.

  • Budget diagnostics: At longer budgets, the early Qwen clean-data advantage contracts, with Overall no longer separated from zero and SSP shifting from +0.003 to −0.009.The reported intervals are +0.024 Overall and +0.045 CMG at 4k updates, versus −0.009 SSP at 13.5k updates.
  • Subgroup diagnostics: Source and subtype analyses find heterogeneous effects, including Qwen SSP loss on ScanNet++ and OmniGen2 SSP declines on ScanNet and ScanNet++, while no headline effect is concentrated in a few scenes.The top 10% of scenes account for at most 43.9% of absolute headline contribution across four model–metric combinations.
  • Qualitative audit: Qualitative tradeoff cases show that positive Overall changes can hide stronger motion execution alongside fewer preserved target-view objects or relations.The top Qwen Cycle tradeoff is +0.502 CMG and −0.323 SSP; the OmniGen2 Atomic tradeoff is +0.956 CMG and −0.491 SSP.
  • Interpretation boundary: The interpretation boundary is that pairwise SFT changes the two axes at different rates, but the study does not establish universal cleaning gains, OmniGen2 CMG improvement, or seed-averaged causality.Teacher forcing conditions later steps on physical targets rather than model-generated histories, motivating tests with self-conditioned rollouts and explicit state-aware losses.
Loading 2609.11172v1…