Human-Centric Image Captioning with Subject-Centered Spatial Understanding

• H-SPACE achieves state-of-the-art performance on SPACE, with H-SPACE-GRPO leading representative Gemini, GPT, and Qwen baselines.The introduction attributes this result to a model trained with SFT and GRPO using a rubric-based spatial judge reward.Source: Figure 1 · p. 2, §1 · p. 3
• Existing human-centered evaluations do not explicitly test conversion from camera-centered observations to subject-centered descriptions, including left-right laterality.Reliable human-centric caption corpora also matter because their quality directly affects training downstream multimodal and generative models [12] [49].Source: §1 · p. 2
• SPACE evaluates human-centric captioning across 2,981 images and 19,892 keypoints, with Spatial & Orientation as its core category.Action & Pose and Appearance provide complementary coverage to distinguish coordinate-frame conversion failures from broader captioning weaknesses.Source: §1 · p. 2
• Mainstream MLLMs perform strongly on appearance and coarse actions but consistently struggle with directional attribution, especially left-right body-part distinctions.Source: §1 · p. 2
• The pipeline uses body-part hints to guide caption rewriting, expands the rewritten data with an open-source rewriter, then trains a hint-free model with SFT and GRPO.A rubric-based reward judge applies stronger penalties to structurally critical spatial errors during preference optimization.Source: §1 · p. 2, Figure 2 · p. 3

2.1 Human-Centric Benchmarking in MLLMs

• Existing human-centered benchmarks emphasize behavior and human-object relations or use QA, grounding, and other discriminative formats, leaving subject-centered spatial errors in generated captions unisolated.SPACE instead evaluates subject-centered spatial correctness in caption generation.Source: §2.1 · p. 3

2.2 Spatial Understanding and Frame of Reference in MLLMs

• Frame-of-reference studies find that MLLMs handle extrinsic frames better than intrinsic ones, while prior benchmarks mainly use VQA, classification, or embodied settings.This work studies coordinate-frame conversion in dense human-centric caption generation.Source: §2.2 · pp. 3–4

2.3 Data Construction and Preference Optimization

• Prior caption rewriting and person-centered synthesis improve descriptiveness, diversity, or retrieval-oriented discriminability, rather than correcting spatial errors with geometric hints.This work combines structured hints with a task-specific rubric reward for spatial omissions, factual mistakes, and critical orientation errors.Source: §2.3 · p. 4

3.1 Motivation and Overview

• SPACE targets structural errors in fine-grained human descriptions, especially subject-centered left/right and body-part/object bindings that generic descriptive accuracy can obscure.Source: §3.1 · p. 4

3.2 Benchmark Construction

• 2,981 image-caption pairs and 19,892 keypoint annotations result from filtering human images, manually rectifying captions to the subject’s perspective, and generating category-specific evaluation points.Images are sampled from CapRL-5M and LAION-400M; SPACE images are excluded from training-data generation.Source: §3.2 · pp. 4–5

3.3 Evaluation Protocol and Metrics

• A judge labels each predefined point correct, wrong, or not mentioned; Hit measures coverage-quality across all points, while Precision evaluates correctness conditional on mentioning a point.Source: §3.3 · p. 5

4.1 Structured Spatial Hint Generation

• YOLO keypoints anchor SAM3 part candidates, and side-aware distance checks retain supported assignments while leaving uncertain or crossed left-right parts unresolved.Verified boxes are rendered on the image and paired with a textual hint list for subsequent caption correction.Source: §4.1 · pp. 5–6

4.2 Two-Stage Caption Generation and Rewriting Pipeline

• Initial Qwen3-VL-235B captions are rewritten by Gemini-3-Pro using image and text hints, then distilled into an open model for bootstrapped corpus expansion.The hint-conditioned inputs guide rewriting; the resulting training pairs retain the raw-image input and rewritten caption.Source: §4.2 · pp. 6–7

4.3 Supervised Fine-Tuning and Preference Optimization

• The deployment model first learns rewritten captions with teacher-forced SFT, then GRPO uses judge-derived penalties that weight critical structural errors most heavily.The judge distinguishes missed facts, generic wrong facts, and critical errors such as left-right reversals and incorrect subject-object binding; inference uses raw images without training-time hints.Source: §4.3 · p. 7

5.1 Datasets and Implementation Details

• Evaluation uses Qwen3-VL-235B-A22B-Instruct as judge, while the deployment model is Qwen3-VL-8B-Instruct trained using rewritten human-centric captions and rubric-based GRPO.Data construction uses YOLO11x-Pose and SAM3; Gemini-3-Pro annotates approximately 60K samples, a Qwen3-VL-32B rewriter distills the rewriting behavior, and the rewritten corpus expands to approximately 237K samples. SFT uses 187K samples, while GRPO uses 50K samples and a Qwen3-VL-32B-Instruct reward judge.Source: §5.1 · p. 7

5.2 Main Results

• H-SPACE-GRPO reaches 83.6 overall Precision and 67.9 Hit, improves over Qwen3-VL-8B-Instruct by 13.0 and 11.2 points, and slightly exceeds Gemini-3.1-Pro at 83.6 vs. 82.5 Precision and 67.9 vs. 67.6 Hit.It also outperforms the larger Qwen3-VL-32B-Instruct and Qwen3-VL-235B-A22B-Instruct models, and achieves the highest Precision across the three top-level dimensions; Gemini-3.1-Pro retains higher Hit on Spatial & Orientation and Action & Pose.Source: §5.2 · p. 7

5.3 Fine-Grained Spatial Analysis

• Category 1.1 is especially difficult across model families, unlike the generally better-handled Category 1.5, because it requires subject-centered left-right discrimination of arms and hands.Several open-source models achieve only slightly above 50% Precision on Category 1.1; the pattern indicates concentrated difficulty in intrinsic-frame conversion and precise body-part grounding rather than uniformly weak human-centric captioning.Source: §5.3 · p. 8

5.4 Controlled Rewriting Analysis

• With source captions fixed, Category 1 Precision rises from 73.3 for Gemini-3.1-Pro without hints to 80.7 with hints, while the trained rewriter scores 80.0 with hints.The hint-conditioned trained rewriter essentially matches the proprietary rewriter at substantially lower cost.Source: §5.4 · p. 8

• SPACE evaluates whether models convert camera-centered observations into correct subject-centered descriptions, where current MLLMs still struggle despite strong broader visual-task performance.Source: §6 · p. 8
• The proposed scalable pipeline combines structured spatial hints, caption rewriting, and rubric-based preference optimization.Source: §6 · p. 8
• The resulting model substantially improves caption quality on SPACE, particularly in reducing subject-centered spatial errors, and remains competitive with strong closed-source systems.Source: §6 · p. 8