FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback

1 Introduction

• Existing figure-caption datasets often contain generic, uninformative captions, which can contribute to poor model generalization and weak alignment with reader preferences.Informative captions can represent and complement a figure while situating it in the article's context.Source: §1 · p. 1
• FigCaps-HF combines a benchmark and learning framework to align figure-caption generation with reader preferences and address feedback integration and scaling.The framework is designed to incorporate expert feedback without additional compute overhead and to scale feedback generation while minimizing annotation effort.Source: §1 · p. 1
• Offline Upside-Down Reinforcement Learning conditions caption-model fine-tuning on inferred feedback scores, after which the trained reward model is no longer needed during that training.The authors report that this reduces computational costs while maintaining performance.Source: §1 · p. 1, Figure 1 · p. 2
• A small human-rated set covering factors such as helpfulness, OCR content, and takeaway trains an auxiliary model to predict quality scores for a larger training set.The inferred scores improve scalability by reducing the need for human ratings across that larger set.Source: §1 · pp. 1–2
• 35.7% BLEU, 16.9% ROUGE-L, 9% METEOR, and 11.4% CIDEr gains over standard supervised fine-tuning were achieved by the best-performing model.Evaluations across multiple baseline models also found that the trained reward model generalized well to unseen samples.Source: §1 · p. 2
• The work contributes a method for predicting feedback scores from limited human ratings and releases a benchmark dataset for further RLHF research on figure-caption generation.The framework uses a small amount of human feedback to train an oracle model that supports large-scale score inference for unseen pairs.Source: §1 · p. 2

2 Background

• Earlier figure-captioning pipelines commonly paired CNN vision encoders with LSTM/RNN text decoders, while recent work uses pretrained transformer vision-language models for zero-shot inference or domain-specific fine-tuning.Examples of the earlier pipeline include Siegel et al. (2016), Qian et al. (2021, 2020), Chen et al. (2019, 2020a,b), and Hsu et al. (2021).Source: §2 · p. 2
• Prior work has used synthetic or document-extracted figure-caption pairs, while recent large multimodal model studies benchmark zero-shot and fine-tuned captioning settings (Roberts et al., 2024).FigCaps-HF instead focuses on reader-preference alignment and is described as model agnostic and applicable to any LMM.Source: §2 · p. 2
• Unlike prior on-policy RL approaches that maximize reward-weighted likelihood (Schulman et al., 2017), the framework uses offline reward-conditioned behavioral cloning (Emmons et al., 2021), an Upside-Down RL variant (Srivastava et al., 2019).The paper characterizes this approach as simpler and more controllable for human-preference alignment.Source: §2 · p. 3
• The feedback scheme can incorporate multiple feedback signals at different granularities as reward during model optimization.Figure question answering is a related task, but the proposed framework focuses on figure-caption generation.Source: §2 · pp. 2–3

3 Framework

• The framework presents a standard figure-captioning pipeline, a feedback-prediction model, and a feedback-aligned training strategy.These are introduced as the framework's three components.Source: §3 · p. 3

3.1 Preliminaries

• The captioning model encodes figures and generates text, trained with cross-entropy to predict each caption token from the figure and preceding tokens.The vision and text modules can use separate or jointly trained pretrained weights, or random initialization.Source: §3.1 · p. 3

3.2 Human Feedback Prediction Model

• A small expert-rated control set trains feedback-score regressors, which use fixed embeddings and mean-squared error to predict scores for the larger training dataset.The control set is assumed to be much smaller than the original dataset.Source: §3.2 · pp. 3–4

3.3 Reinforcement Learning with Human Feedback

• Offline UDRL treats figures as states, captions as actions, and predicted metric scores as rewards; reward-derived control tokens guide fine-tuning of the figure-to-caption policy.Scores are binarized as <|good|> or <|bad|> using a threshold.Source: §3.3 · p. 4

4 FigCaps-HF: Figure-Captioning with Human Feedback Benchmark

• 106,834 figure-caption pairs are scored on helpfulness, takeaway, visual-descriptiveness, and image-text (OCR) quality.The measures are intended to assess readers' ability to comprehend and draw inferences from the figure and caption.Source: §4 · p. 4
• 438 randomly sampled pairs receive expert ratings on four 5-point Likert measures, and a trained model predicts continuous scores for the remaining dataset.Users can apply their own thresholds to the continuous scores for fine-grained feedback.Source: §4 · p. 4
• Predicted rating medians align with actual feedback, and the model is reported to provide reliable scores even outside the typical rating range.The authors also describe the predicted ratings as diverse and not skewed toward a particular rating.Source: §4 · pp. 4–5

5.2 Qualitative Results

• High-scoring feedback predictions identify captions with specific takeaways and visual details, while low-scoring captions are vague and lack figure references.Source: §5.2 · p. 5, Figure 2 · p. 7

5.1 Results

• BLIP-RLHF improves over standard fine-tuning by 35.7% BLEU, 16.9% ROUGE-L, 9% METEOR, and 11.4% CIDEr; ViT+GPT2 gains 11.1% BLEU and 5.1% CIDEr.Source: §5.1 · p. 5, Table 2 · p. 6

5.2 Qualitative Results

• Compared with fine-tuned BLIP, BLIP-RLHF captions are more faithful to figures, capture semantic relations, and use visual attributes to explain them.Fine-tuned BLIP examples include incorrect, irrelevant, or uninformative captions.Source: §5.2 · p. 6, Figure 3 · p. 7

5.3 Ablation Study

• Five-fold validation on 438 annotated pairs yielded good validation results, while Helpfulness-prediction test performance saturated as training data increased and was good with half the original data.Source: Table 7 · p. 6, Table 8 · p. 7, §5.3 · p. 8
• Both five-level feedback and binary-then-multilevel feedback outperform simple binary feedback for BLIP-RLHF.The five levels are very bad, bad, neutral, good, and very good.Source: §5.3 · pp. 6–7, Table 3 · p. 6
• Training BLIP-RLHF with Takeaway, Visual, or OCR feedback outperforms training it with Helpfulness feedback.The authors characterize Helpfulness as subjective and Visual and Takeaway as objective measures.Source: Table 4 · p. 6, §5.3 · p. 7
• Alternative feedback-prediction representations outperform default MCSE, indicating that downstream captioning performance is sensitive to representation quality.The compared alternatives include BERT, SciBERT, and BLIP.Source: Table 5 · p. 6, §5.3 · pp. 7–8
• Prepending feedback labels conditions generation on them, and ViT+GPT2 and BLIP perform better with prepended than appended labels.Source: Table 6 · p. 6, §5.3 · p. 8

• The authors report that incorporating domain expert feedback improves figure-to-caption model performance and caption quality.Source: §6 · p. 8
• The framework is described as scalable through limited manual labeling and flexible in supporting multiple reward signals at different granularities.Source: §6 · p. 8
• The released benchmark is intended to support evaluation of human-feedback methods for figure-captioning and other image-to-text generation tasks.Source: §6 · p. 8

Limitations

• The UDRL-based caption model fine-tuning updates all parameters, which can use more compute than parameter-efficient methods such as Low-Rank Adaptation.The limitation concerns the caption-generation model fine-tuning scheme.Source: Limitations · p. 8
• The framework aligns feedback with general readers, while assistive tools for specific groups such as people with visual impairments require additional consideration in initial feedback annotations.Source: Limitations · p. 8

A Overview

• The overview describes quality metrics, experimental setup, baseline models, qualitative comparisons, dataset composition and collection, and supporting statements and plans.Source: A Overview · p. 11

A.1 Ethics Statement

• The ethics statement warns against presenting the research method as deployable and against generating human feedback without due consideration of human subjects; the dataset contains no personal information and is licensed for academic or non-commercial research.Source: §A.1 · p. 11

A.2 Description of metrics used for Feedback assessment

• Feedback assessment defines helpfulness by whether a caption informs readers about figure information and takeaway by whether it conveys a conclusion.Source: §A.2 · p. 11
• Visual-descriptiveness measures grounding in the figure, while image-text (OCR) assesses whether captions include textual elements such as titles, legends, and labels.Source: §A.2 · p. 12

A.3.1 Datasets

• All models use the same splits: 106,834 training pairs, 13,354 validation pairs, and 13,355 test pairs; baseline and RLHF training differ in caption feedback augmentation while images and test pairs stay the same.Source: §A.3.1 · p. 12
• Ten Ph.D. graduates and active graduate students with CV, NLP, or ML publication experience annotated 438 randomly selected pairs over two weeks.Source: §A.3.1 · p. 12
• Annotators rated each sample on a five-point Likert scale for OCR, visual-descriptiveness, takeaway, and helpfulness.Source: §A.3.1 · p. 12
• An auxiliary MLP regressor trained on MCSE embeddings of 400 annotated pairs predicts reader-preference scores for the larger set, whose captions are labeled good or bad around the median score.Source: §A.3.1 · p. 12

A.3.2 Evaluation Metrics

• ROUGE-L reports F1 using longest-common-subsequence recall, BLEU@4 uses n-gram precision with a brevity penalty, and METEOR combines generalized unigram precision and recall.Source: §A.3.2 · p. 12

A.3.3 Baselines

• Baselines are classified as figure-only, OCR-only, or figure-caption methods according to whether generation uses image, extracted text, or multimodal image-and-text inputs.Source: §A.3.3 · pp. 12–13
• The baseline set spans vision-encoder/text-decoder models, OCR and image-to-text models, multimodal models such as BLIP, and OCR summarization with Pegasus.Source: §A.3.3 · p. 13
• The fine-tuned baselines are ViT+RoBERTA, ViT+GPT2, BEiT+GPT2, GIT, BLIP, and CLIPCap, while Pegasus, TrOCR, PromptCap, and Flamingo-mini are evaluated zero-shot.Source: §A.3.3 · p. 13

Comparative analysis

• In the examples, BLIP-RLHF captures average epoch time rather than inferred edge counts, whereas BLIP hallucinates a caption for a phylogenetic tree and gives an unaligned caption for a chart.Source: §A.4 · p. 14
• The phylogenetic-tree example suggests generalization to chart types beyond line charts, the only chart type with actual human feedback in the study.Source: §A.4 · p. 14

Human-Evaluation of model generated captions

• 85% of the time, human evaluators selected BLIP-RLHF captions over BLIP captions in a 100-figure comparison rated by 10 subjects.Subjects considered helpfulness, visual-descriptiveness, OCR alignment, and takeaway.Source: §A.4 · p. 14

B.1 Motivation

• The dataset was created to help researchers develop and evaluate figure-to-caption pipelines for reader-preference-aligned caption generation.Source: §B.1 · p. 14
• Author and contact details are deferred until acceptance because of double-blind review, and the dataset received no funding.Source: §B.1 · p. 14

B.1.1 Author Statement

• The authors bear responsibility for distributing and maintaining the dataset.Source: §B.1.1 · p. 14

outside of the entity (e.g., company, institution,

• The dataset is public, and the benchmark and dataset-generation code are linked for access.Source: outside of the entity (e.g., company,… · p. 15

B.3 Maintenance

• The authors will host and maintain the dataset, announce updates, retain older versions, and allow extensions or contributions.Source: §B.3 · p. 15

B.4 Composition

• The dataset contains 06,834 training pairs, 13,354 validation pairs, and 13,355 test figure-caption pairs with feedback scores.Source: §B.4 · p. 15
• The dataset is self-contained, has three recommended splits, and provides each figure image with a caption and predicted feedback scores for helpfulness, OCR, visual, and takeaway metrics.Source: §B.4 · p. 15

B.5 Collection Process

• The authors were involved in curating data obtained from a publicly available source; the reported February 2023–May 2023 timeframe applies to data collection, not expressly to curation.Source: §B.5 · p. 15

B.6 Uses

• The dataset has been used for human-feedback-aligned figure captioning and may also support domain-specific evaluation of image-to-text models.Source: §B.6 · p. 15

B.7 Data Format

• Figure images are stored as PNGs, while captions and metadata are JSON; metadata includes figure identifiers, type, subfigure status, caption variants, and feedback fields.Feedback covers helpfulness, OCR, takeaway, and visual ratings; inferred feedback tokens are based on each metric's median score.Source: §B.7 · p. 15

B.7.1 Reading Data

• Image filenames link figures to their caption metadata across split folders, and a file-index JSON is an alternative way to enumerate images.The benchmark, augmented captions used during RLHF fine-tuning, and code for baseline training, feedback fine-tuning, and test evaluation are provided.Source: §B.7.1 · p. 16, §B.7.2 · p. 16