PaperlayerView exact revision on arXiv

Source-linked AI summary

ShutterMuse: Capture-Time Photography Guidance with MLLMs

Jiayu Li, Yixiao Fang, Tianyu Hu, Wei Cheng, Ping Huang, Zheheng Fan, Gang Yu, Xingjun Ma

arXiv:2606.25763v1cs.CV

TL;DR

Capture-time photography guidance must address both framing decisions and subject pose, but existing benchmarks and models leave this capability incomplete. The paper introduces a benchmark, dataset, and unified MLLM, finding that ShutterMuse leads photographer-side performance while offering competitive pose recommendations at lower inference cost.

  • Problem

    Capture-time MLLM guidance for photographer framing and subject pose remains underexplored, while existing aesthetic-cropping benchmarks primarily assess post-hoc crop prediction.

  • Method

    The paper introduces CaptureGuide-Bench, constructs a 130K-sample annotated dataset, and develops ShutterMuse with supervised and reinforcement fine-tuning.

  • Results

    ShutterMuse achieves the best overall photographer-side performance among evaluated baselines and competitive subject-side pose recommendations with substantially lower inference cost.

  • Takeaways & Limitations

    The results suggest that MLLMs can serve as practical interactive assistants for photography during image capture.

Abstract

from arXiv · show

Real-world photography requires capture-time guidance for both camera framing and subject pose. Yet existing aesthetic cropping benchmarks mainly evaluate post-hoc crop prediction and overlook subject-side recommendations, leaving the capture-time guidance capabilities of multimodal large language models (MLLMs) underexplored. To address this gap, we introduce CaptureGuide-Bench, a benchmark with two complementary tasks: photographer-side composition decision and refinement, and subject-side scene-conditioned pose recommendation. Our evaluation reveals limitations: general-purpose MLLMs can make composition decisions but lack precise refinement localization, while specialized aesthetic cropping models localize crops effectively but are limited to refinement; neither provides actionable pose guidance. To support model development, we further construct CaptureGuide-Dataset, comprising 130K samples with textual rationales and structured visual annotations, and develop ShutterMuse, a unified MLLM trained with supervised and reinforcement fine-tuning. Experiments on CaptureGuide-Bench show that ShutterMuse achieves the best overall photographer-side performance among evaluated baselines and competitive subject-side pose recommendation with substantially lower inference cost, demonstrating the potential of MLLMs as interactive assistants for photography during image capture.

1 INTRODUCTION

ShutterMuse addresses the underexplored problem of capture-time photography guidance by introducing CaptureGuide-Bench, CaptureGuide-Dataset, and a unified MLLM for structured photographer- and subject-side recommendations. Experiments show strong photographer-side performance and efficient, competitive subject-side pose guidance.

  • Motivation: Although MLLMs have advanced visual understanding, aesthetic reasoning, and instruction following, their ability to guide photography during image capture remains underexplored.The work targets this gap with a benchmark focused specifically on capture-time assistance.
  • Benchmark: CaptureGuide-Bench evaluates capture-time guidance through photographer-side composition decisions and refinement plus subject-side pose recommendation.Photographer-side evaluation includes refine, keep, and reject decisions and valid framing boxes when refinement is needed.
  • Dataset: CaptureGuide-Dataset contains approximately 130K samples with textual rationales, structured composition boxes, pose keypoints, and visibility states.Its photographer-side subset spans five scenarios and six composition aspect ratios, while its subject-side subset covers five common human pose types.
  • Experimental results: ShutterMuse achieves the best overall photographer-side performance among evaluated baselines and competitive subject-side pose recommendations with substantially lower inference cost.These results support using MLLMs as interactive assistants during image capture.
  • Model: ShutterMuse is a unified MLLM trained with supervised fine-tuning and reinforcement fine-tuning to generate structured, interpretable capture-time guidance.The model is designed to provide both photographer-side and subject-side assistance.

2 RELATED WORK

Prior work frames aesthetic cropping primarily as post-hoc composition refinement, with proposal- and regression-based methods predicting improved crops. In parallel, text-conditioned motion-generation methods synthesize controllable body motions from language instructions.

  • Aesthetic image cropping is traditionally a post-hoc task that improves captured images by predicting better crops, as established by benchmarks including FCDB and FLMS (Chen et al., 2017).
  • Existing aesthetic cropping methods comprise proposal-based approaches that rank candidate crops using aesthetic cues and regression-based approaches (Su et al., 2024; Zeng et al., 2019; Zhang et al., 2026; Du et al., 2026).
  • Recent human motion-generation methods synthesize controllable body motions from language instructions, including discrete-token approaches such as T2M-GPT and MotionGPT and hierarchical tokenization in MoMask (Tevet et al., 2022; Chen et al., 2023; Zhang et al., 2023a; Jiang et al., 2023; Guo et al., 2024; Zhang et al., 2023b).

3 DATASET AND BENCHMARK

This section introduces CaptureGuide-Dataset, a roughly 130K-sample resource for capture-time photographer- and subject-side guidance, and CaptureGuide-Bench, a held-out benchmark spanning composition decisions and pose recommendations. It also defines the annotation pipelines and task-specific evaluation protocols for both guidance settings.

  • Dataset and benchmark: CaptureGuide-Dataset contains approximately 130K images, including 100K photographer-side guidance samples and 30K subject-side guidance samples, while CaptureGuide-Bench evaluates both tasks.The benchmark is built on top of the dataset for standardized evaluation and downstream method development.
  • Dataset construction: Photographer-side data combines three-way decisions—refine, reject, or keep—with refined composition boxes and comments explaining defects, strengths, and reframing strategies.A 12K-image expert seed set is labeled by 10 trained annotators with cross-review, then used in an expert-seeded, MLLM-verified self-distillation pipeline for scaling.
  • Dataset construction: Subject-side samples are triplets of person-free scene images, target human poses represented by keypoints, and textual rationales explaining scene-conditioned pose suitability.The generation pipeline removes people while preserving scene context, extracts COCO 17-keypoints, models visibility states, and applies filtering plus human correction.
  • Benchmark design: CaptureGuide-Bench holds out 421 photographer-side samples and 552 subject-side examples from training, with 3–5 ground-truth boxes per photographer-side refine image.Its two complementary subsets cover composition decision-making and subject-side pose guidance, and the samples are excluded from both SFT and RFT.
  • Evaluation metrics: Photographer-side evaluation measures decision accuracy alongside refinement localization using maximum IoU and minimum boundary displacement error, and supplements geometry with task-aware MLLM-Score.For non-reject predictions, scoring evaluates either the predicted refinement box or the full image for keep predictions against annotated compositional strengths and defects.
  • Evaluation metrics: Subject-side evaluation treats reference keypoints as plausible pose configurations rather than exact targets because multiple physically plausible, scene-aligned, and visually appealing poses may exist.This protocol reflects the practical goal of assessing pose plausibility, semantic alignment, and visual appeal instead of relying on a single geometric criterion.

4 SHUTTERMUSE

ShutterMuse is a unified Qwen3-VL-8B-based MLLM for capture-time photographer- and subject-side photography guidance, trained with supervised fine-tuning followed by reward-based reinforcement fine-tuning.

  • Framework: ShutterMuse unifies photographer-side and subject-side capture-time guidance in an MLLM built on Qwen3-VL-8B Bai et al. (2025).
  • Supervised Fine-Tuning: Supervised fine-tuning trains prompt-following generation of structured JSON responses from CaptureGuide-Dataset, with schemas dependent on guidance type.
  • Supervised Fine-Tuning: Subject-side outputs specify pose guidance with 17 normalized COCO keypoints and visibility labels distinguishing visible, occluded, and out-of-frame keypoints.
  • Reinforcement Fine-Tuning: Group Relative Policy Optimization further improves decision-making and output accuracy using 20,000 EMDP- and SGGP-based samples with task-specific rewards.
  • Reinforcement Fine-Tuning: Photographer-side rewards assess three-way decision correctness and, for refinements, salient-subject preservation inside the predicted composition box.
  • Reinforcement Fine-Tuning: Subject-side reinforcement uses visibility annotations to reward agreement between predicted and ground-truth 17-dimensional visibility vectors.

5 EXPERIMENTS

Experiments show that ShutterMuse provides the strongest overall photographer-side guidance, competitive subject-side pose recommendations, and benefits substantially from reinforcement fine-tuning and its full reward design. Qualitative and reliability analyses further examine its behavior and the EMDP data-generation pipeline.

  • Photographer-side guidance: ShutterMuse achieves the best overall photographer-side balance, with the highest IoU, lowest BDE, and best refinement success rate, while specialized and general-purpose baselines show complementary weaknesses.InstructCrop and Venus provide competitive crop quality but perform poorly on reject and keep decisions, whereas general-purpose MLLMs handle three-way decisions better but localize crops less accurately.
  • Subject-side guidance results: ShutterMuse attains a subject-side mean score of 0.34 across plausibility, interaction, and aesthetics, close to GPT-Image-2 at 0.35.Image-editing foundation models retain a slight advantage, attributed to larger capacity and broad pretraining that provide stronger priors for anatomy, feasibility, interactions, and aesthetics.
  • Qualitative comparisons: Qualitative comparisons show that ShutterMuse makes appropriate decisions, produces aesthetically refined results, and gives interpretable rationales across diverse capture-time scenarios, unlike Venus’s tendency to crop unnecessarily.The comparisons cover both photographer-side and subject-side guidance.
  • Effect of Training Strategy and Reward Design: GRPO raises IoU from 72.39% to 74.30%, RSR from 68.97% to 82.76%, KSR from 63.64% to 74.55%, and MLLM-Score from 0.56 to 0.64 over SFT-only training.It also improves subject-side plausibility and interaction while leaving aesthetics unchanged, indicating particular benefits for discrete capture-time decisions and composition-frame alignment.
  • Effect of Training Strategy and Reward Design: The full reward design achieves the strongest or tied-strongest performance across evaluated metrics, while removing the subject-side reward lowers plausibility.This indicates that Rsub improves consistency between pose recommendations and visible body-part constraints, and that each reward component contributes to final performance.
  • Reliability Analysis of EMDP: EMDP reliability is assessed over three rounds using 450 held-out expert-annotated samples excluded from seed training, expansion, SFT, RFT, verifier calibration, and hyperparameter selection.The analysis also evaluates verifier reliability using randomly sampled examples per round.

6 USER STUDY

A user study evaluates agreement between MLLM-based assessment and human preferences using blind rankings from six participants across sampled CaptureGuide-Bench examples. It compares ShutterMuse with distinct photographer-side and subject-side baselines.

  • Ranking agreement: Table 4 defines the reported comparison as ranking agreement between MLLM-Score and human preference, with lower ranks indicating better performance.
  • Study setup: The study randomly samples 100 test examples from each CaptureGuide-Bench subset and recruits six participants for blind evaluation.Rankings are aggregated across all examples and participants.
  • Photographer-side evaluation: The photographer-side comparison includes Gemini-3.0-Pro, GPT-5.5, InstructCrop, Venus, and ShutterMuse.These models are evaluated through aggregated human rankings.
  • Subject-side evaluation: The subject-side comparison includes Nano-Banana-Pro, GPT-Image-2, and ShutterMuse.The study aggregates rankings over all sampled examples and participants.

7 CONCLUSION

The paper introduces CaptureGuide-Bench and CaptureGuide-Dataset to support capture-time photography guidance, and proposes ShutterMuse, a unified MLLM trained with supervised and reinforcement fine-tuning. ShutterMuse achieves the best overall photographer-side performance among evaluated baselines, competitive subject-side pose recommendations, and substantially lower inference cost.

  • Benchmark: CaptureGuide-Bench evaluates MLLMs on both photographer-side composition refinement and subject-side pose recommendation during image capture.
  • Resources and model: The work constructs CaptureGuide-Dataset with structured annotations and textual rationales and proposes ShutterMuse as a unified MLLM trained with supervised and reinforcement fine-tuning.
  • Results: ShutterMuse achieves the best overall photographer-side performance among evaluated baselines while balancing refinement accuracy with keep/reject decisions, and provides competitive subject-side pose recommendations at substantially lower inference cost.These results are reported on CaptureGuide-Bench.

A ANNOTATION PROTOCOL · A.1 PHOTOGRAPHER-SIDE ANNOTATION GUIDELINES

The photographer-side protocol classifies images as keep, refine, or reject based on whether reframing can preserve or improve composition. Annotators provide compositional rationales for keep and refine cases and explain non-croppable defects for reject cases.

  • A.1 PHOTOGRAPHER-SIDE ANNOTATION GUIDELINES: Refine labels indicate usable subjects or content whose composition can improve through cropping or recomposition.Typical issues include weak subject prominence, shifted or unbalanced visual centers, undersized subjects, and distracting backgrounds.
  • A.1 PHOTOGRAPHER-SIDE ANNOTATION GUIDELINES: Keep labels indicate framing that is already compositionally appropriate, where further cropping would remove useful context or reduce image quality.Annotators describe strengths such as clear subject placement, balanced structure, suitable background context, and coherent scene layout.
  • A.1 PHOTOGRAPHER-SIDE ANNOTATION GUIDELINES: Reject labels exclude images with severe defects that cropping or recomposition cannot effectively correct.Examples include blur, occlusion, poor exposure, missing subjects, and tilted framing; annotators document these non-croppable issues.

A.2 SUBJECT-SIDE ANNOTATION GUIDELINES

Subject-side annotations follow the COCO-17 keypoint convention and combine structured pose metadata with natural-language rationales. Annotation quality is checked through pose visualization, visibility correction, and professional-photographer review.

  • A.2 SUBJECT-SIDE ANNOTATION GUIDELINES: The 17 keypoints follow COCO-17 order, with left and right defined from the subject’s anatomical perspective.
  • A.2 SUBJECT-SIDE ANNOTATION GUIDELINES: Each subject-side instance stores the task type, natural-language rationale, normalized keypoint coordinates, and visibility labels.
  • A.2 SUBJECT-SIDE ANNOTATION GUIDELINES: Visibility labels use 1 for visible in-image keypoints, 0 for occluded in-image keypoints, and -1 for keypoints outside the image boundary.
  • A.2 SUBJECT-SIDE ANNOTATION GUIDELINES: Annotators discard geometrically inconsistent poses, correct erroneous visibility flags, and revise or remove rationales that five professional photographers cannot use to reconstruct the intended pose.Guidance examples distinguish refine, reject, and keep cases according to whether recomposition is suitable, defects are non-croppable, or framing is already appropriate.

B MLLM-BASED EVALUATION PROMPTS · C BASELINE PROMPT TEMPLATES

The evaluation prompts standardize photographer-side composition assessment and subject-side pose assessment, while baseline templates enforce comparable cropping and pose-generation protocols. Subject-side outputs are converted to shared skeleton visualizations before MLLM evaluation.

  • B.1 PHOTOGRAPHER-SIDE MLLM-SCORE PROMPT: Photographer-side evaluation overlays each predicted composition frame as a red box and scores only its enclosed composition on a three-level scale: 0, 0.5, or 1.The prompt requires JSON output with a score, level, reason, whether original problems were fixed, and any composition technique used.
  • B.2 SUBJECT-SIDE MLLM-SCORE PROMPT: Subject-side evaluation independently measures physical plausibility, scene interaction, and pose aesthetics from a single skeleton visualization, normalizing scores to {0, 0.5, 1.0}.The visualization distinguishes full-body, three-quarter-body, and upper-body poses, with red body parts indicating predicted occlusions.
  • B.2 SUBJECT-SIDE MLLM-SCORE PROMPT: Physical plausibility rewards natural scale, credible scene contact, absence of floating or penetration, and poses that humans can reasonably imitate.The rubric tolerates visualization-specific head–torso separation, slight standing-pose floating, crossed legs, seated floating feet, and omitted lower-body keypoints when appropriate.
  • B.2 SUBJECT-SIDE MLLM-SCORE PROMPT: Scene interaction ranges from no meaningful interaction, through sitting or leaning, to distinctive object or scene interaction involving movement or gaze.Pose aesthetics similarly progress from ordinary static poses to detailed, dynamic, expressive actions adapted to the environment.
  • C.1 PHOTOGRAPHER-SIDE PROMPT: Specialized cropping baselines use official checkpoints, default settings, and original-paper prompts, whereas general MLLM baselines use a unified prompt for fair comparison.The unified photographer prompt classifies images as excellent, improvable, or unsalvageable, then requests a normalized target-ratio bounding box or <non>.
  • C.2 SUBJECT-SIDE PROMPT: Bounding boxes use normalized [x1,y1,x2,y2] coordinates and the specified aspect ratio; subject-side baselines instead request environment-compatible, aesthetic, physically plausible human meshes.Specialized cropping baselines are inapplicable to pose prediction, and generated meshes must preserve the scene while avoiding extra people, floating, penetration, stiff movement, and unsupported body–environment relationships.
  • C.2 SUBJECT-SIDE PROMPT: To avoid rendering, texture, lighting, and editing-artifact differences, baseline edits are converted into standardized skeleton visualizations before subject-side MLLM evaluation.This protocol extracts human keypoints from each edited image and re-visualizes them in the same skeleton format used by the proposed method.

D FAILURE CASE ANALYSIS FOR SUBJECT-SIDE GUIDANCE

Failure cases show that subject-side pose recommendations can appear to float near the feet because ankle-only keypoints poorly represent foot–ground contact. Despite this visualization artifact, users can still understand and imitate the intended poses, while denser or contact-aware keypoints could improve physical support modeling.

  • Failure Cases: Recommended poses may appear slightly floating near the feet in skeleton visualizations, revealing a limitation of the current subject-side guidance pipeline.The artifact is especially visible around the feet and creates a discrepancy between automatic evaluation and perceptual quality in some cases.
  • Failure Cases: The artifact largely arises because YOLO-style extraction uses 17 keypoints that localize ankles but not toes or the full foot-contact region.Consequently, the rendered skeleton may not accurately reflect the subject’s actual support on the ground.
  • Practical Impact: Despite the floating visualization, users can still understand the intended body configuration and imitate the recommended pose, so usability is not substantially affected.The limitation primarily concerns physical-contact representation rather than the intelligibility of the pose recommendation.
  • Future Work: Future improvements could use denser body keypoints, explicit foot keypoints, or contact-aware representations to model foot–ground support more accurately.These alternatives target the insufficiency of ankle-only keypoints for representing physical contact between the body and scene.
Loading 2606.25763v1…