PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Take a Memorable Picture? Empowering Users with Actionable Feedback

Francesco Laiti, Davide Talon, Jacopo Staiano, Elisa Ricci

arXiv:2602.21877v2cs.CV

TL;DR

Prior memorability systems mainly predict scores or alter images, leaving users without actionable capture-time guidance. This paper introduces MemFeed, MemCoach, and MemBench, and reports consistent gains over zero-shot MLLMs across evaluation axes and backbones.

  • Problem

    Existing approaches predict memorability or edit images, but do not provide users actionable, interpretable guidance while taking photographs.

  • Method

    MemCoach is a training-free, model-agnostic teacher-student activation-steering method that distills memorability-improving feedback into MLLM behavior.

  • Results

    MemCoach consistently improves memorability-feedback performance over zero-shot strategies across multiple MLLMs and evaluation axes.

  • Takeaways & Limitations

    MemFeed and MemBench shift memorability research toward actionable guidance and provide a benchmark for evaluating feedback quality.

Abstract

from arXiv · show

Image memorability, i.e., how likely an image is to be remembered, has traditionally been studied in computer vision either as a passive prediction task, with models regressing a scalar score, or with generative methods altering the visual input to boost the image likelihood of being remembered. Yet, none of these paradigms supports users at capture time, when the crucial question is how to improve a photo memorability. We introduce the task of Memorability Feedback (MemFeed), where an automated model should provide actionable, human-interpretable guidance to users with the goal to enhance an image future recall. We also present MemCoach, the first approach designed to provide concrete suggestions in natural language for memorability improvement (e.g., "emphasize facial expression," "bring the subject forward"). Our method, based on Multimodal Large Language Models (MLLMs), is training-free and employs a teacher-student steering strategy, aligning the model internal activations toward more memorable patterns learned from a teacher model progressing along least-to-most memorable samples. To enable systematic evaluation on this novel task, we further introduce MemBench, a new benchmark featuring sequence-aligned photoshoots with annotated memorability scores. Our experiments, considering multiple MLLMs, demonstrate the effectiveness of MemCoach, showing consistently improved performance over several zero-shot models. The results indicate that memorability can not only be predicted but also taught and instructed, shifting the focus from mere prediction to actionable feedback for human creators.

1. Introduction

Memorability research has mainly predicted or altered images, leaving capture-time guidance unaddressed. The paper introduces MemFeed, MemCoach, and MemBench to provide and evaluate actionable memorability feedback.

  • Motivation: Prior work either predicts a scalar memorability score or automatically edits images, but neither paradigm gives users controllable capture-time guidance.Prediction models report memorability, while generative models directly change the image.
  • Memorability Feedback: MemFeed formalizes actionable, human-interpretable feedback that helps users improve an image’s memorability.The task uses natural-language suggestions describing concrete compositional or semantic changes.
  • MemCoach: MemCoach uses training-free teacher-student activation steering to redirect MLLM behavior toward memorability-aware suggestions.A teacher model distills transitions from less to more memorable images across views of the same scene.
  • MemBench: MemBench evaluates feedback using photoshoot-aligned images with memorability scores, editing-based improvement, and perplexity on effective feedback.The benchmark is built from multiple images from the same photoshoot and supports systematic evaluation of the novel task.
  • Results: Across four open-source MLLMs, MemCoach consistently improves performance over zero-shot models on memorability feedback.The reported evaluation combines editing metrics with perplexity.

2. Related Work

Related work established memorability as a measurable property of visual content and developed prediction, photographic critique, and MLLM steering methods. The paper distinguishes MemCoach by applying teacher-student steering to actionable memorability feedback.

  • Memorability: Memorability is a quantifiable visual property that can be stable across observers and is influenced by semantic and scene characteristics.Reported factors include faces, animals, indoor or less cluttered scenes, and negative emotions.
  • Photographic feedback: Existing photographic-feedback systems provide critique or rule-based guidance, but they do not consistently translate critique into executable instructions during shooting.Prior guidance has focused largely on aesthetic scoring and overlays.
  • MLLMs and steering: Activation steering modifies model behavior by shifting intermediate representations using contrasting samples and mean-difference vectors.These methods control target concepts at inference time under the linear activations hypothesis.
  • MLLMs and steering: MemCoach applies a teacher-student steering strategy to MLLMs for actionable memorability feedback, a perceptual-task use not previously studied in this line of work.The paper presents it as the first activation-steering strategy for MLLMs applied to perceptual tasks.
  • Benchmarking: MemBench evaluates feedback through editing-based memorability improvement and perplexity scoring in a paired photoshoot pipeline.Its construction groups same-scene images, ranks them by memorability, and generates actionable feedback.

3. Memorability Feedback

Memorability Feedback is framed as transforming a source image into a more memorable destination through actionable language. MemBench constructs same-scene ranked pairs and evaluates whether feedback produces memorability gains and resembles effective actions.

  • Task definition: Memorability Feedback seeks actionable language a that transforms source image xS with score mS into destination image xD with mD > mS.Scores are estimated by a memorability predictor M.
  • Benchmark construction: MemBench augments same-scene image pairs with semantic action descriptions explaining differences between lower- and higher-memorability photographs.It builds on the PPR10K portrait retouching dataset.
  • Benchmark construction: The data pipeline groups images by scene, predicts memorability, ranks photographs, and forms less-to-more memorable pairs.The predictor is a CLIP-feature regressor trained on public memorability datasets.
  • Feedback extraction: A captioning model generates semantic, on-the-fly actions such as facing each other rather than post-hoc corrections such as brightening an image.The feedback is elicited from source and destination image pairs.
  • Benchmark statistics: MemBench contains approximately 10K images across 1,570 scenes, averaging 6.5 images per scene.Source images have an average memorability score of 0.63.
  • Evaluation protocol: Evaluation covers editing effectiveness and the likelihood of producing memorable actions, using improvement ratio, relative memorability, and perplexity.The feedback model is evaluated on scenes held out by an 80-20 train/test split.

4. Method

MemCoach enables MLLMs to provide memorability-oriented feedback by steering a student model with activation patterns distilled from teacher-generated guidance. It builds contrasting feedback from same-scene images and applies the resulting steering vector at inference without additional training.

  • Task motivation: MemFeed targets actionable, human-interpretable suggestions because naively prompted MLLMs lack a concrete understanding of what makes images memorable.The intended output is natural-language guidance describing compositional or semantic changes to a user’s image.
  • Method overview: MemCoach is a training-free approach that uses knowledge-distillation activation steering to elicit memorability feedback from MLLMs.The method injects memorability information into a student model without retraining it.
  • Contrasting data generation: The method constructs paired samples by comparing teacher-generated memorability-aware actions with the student model’s neutral feedback for the same source image.Images from each scene are ranked by memorability, with the least memorable image serving as the source and a more memorable image defining the desired transformation.
  • Steering vector extraction: MemCoach extracts a steering vector from activation differences between memorability-aware and neutral feedback, representing the latent shift toward more effective suggestions.The vector is computed across student activations and acts as a distilled representation of the teacher’s memorability knowledge.
  • Inference with MLLM steering: At inference, the student’s intermediate activations are shifted by the extracted vector before subsequent layers generate memorability-oriented feedback.The steering strength is controlled by α, and the procedure operates at the activation level without relying on teacher information at inference.

5. Experiments

Experiments evaluate MemCoach across MLLM baselines, architectures, qualitative feedback patterns, and steering ablations. MemCoach improves memorability-oriented feedback, including quantitative gains, cross-backbone consistency, actionable suggestions, and strong low-data efficiency.

  • 5.1. Quantitative Results: MemCoach is compared with teacher-oracle, zero-shot, aesthetics-specialized, and editing-baseline MLLMs using editing and perplexity metrics.The implementation uses INTERNVL3.5 8B as teacher and student, with steering layer l = 12 and coefficient α = 55.
  • 5.1. Quantitative Results: +5% IR over GPT-5 MINI and +31.81% RM over base INTERNVL3.5 show MemCoach improves both the frequency and magnitude of memorability-increasing edits.The steered model also achieves lower perplexity on ground-truth feedback, indicating closer alignment with memorability-aware suggestions.
  • 5.1. Quantitative Results: MemCoach achieves the best results among the considered teacher-oracle, zero-shot, and aesthetics-specialized models.The comparison covers MemFeed performance across the reported evaluation metrics.
  • 5.1. Quantitative Results: Integrating MemCoach into different multimodal backbones consistently improves memorability-aware feedback, with the strongest IR gains for QWEN2.5VL and LLAVA-OV.The result is reported in the generalization analysis across architectures.
  • 5.2. Qualitative Evaluation: MemCoach produces concise, actionable suggestions spanning compositional adjustments and semantic interventions, while failure cases reveal normalization–distinctiveness trade-offs.Common instructions include altering gaze, pose, or hand position, and often use verbs such as “Bring”, “Stand”, and “Remove”.
  • 5.2. Qualitative Evaluation: Symmetry, balance, centered poses, and socially directed gaze recur among feedback patterns associated with improved memorability.Examples include holding with both hands, hands on the hips, looking at the camera, and looking at each other.
  • 5.3. Ablation Study: With only 1% of training data, MemCoach reaches performance on par with full-data fine-tuning and outperforms Low-Rank fine-tuning in the low-data regime.The DIFF(MEAN) ablation obtains 6.64 RM versus 7.21 RM for subtraction before averaging.

6. Conclusion

The conclusion frames Memorability Feedback as a shift from passive prediction to actionable guidance and presents MemBench and MemCoach for this setting. Experiments report more effective, human-aligned feedback than zero-shot strategies with minimal data.

  • 6. Conclusion: Memorability Feedback shifts memorability research from passive prediction toward actionable guidance for users.The paper formalizes this previously unstudied task and targets feedback at capture time.
  • 6. Conclusion: MemBench provides a benchmark and evaluation metrics for assessing the quality of memorability feedback.The benchmark accompanies the new MemFeed setting.
  • 6. Conclusion: MemCoach uses model-agnostic activation steering to distill memorability-improvement knowledge from an oracle teacher into a student MLLM.The framework is training-free and generates natural-language feedback.
  • 6. Conclusion: Experiments show that steering MLLMs toward memorability-aware activations yields more effective and human-aligned feedback than zero-shot strategies while requiring minimal data.The conclusion also proposes activation steering as a route toward perceptual skills and interactive visual guidance systems.

Supplementary Material

The supplementary material adds dataset examples, method details, user-study experiments, and cross-editing-model consistency analyses.

  • Supplementary Material: Section A presents qualitative MemBench examples and describes its construction pipeline.It provides additional context on the benchmark data.
  • Supplementary Material: Section B gives implementation details for MemCoach and discusses potential implications of the work.The section expands the description of the proposed method.
  • Supplementary Material: Section C reports preliminary user-study experiments.These studies supplement the main experimental evaluation.
  • Supplementary Material: Section D demonstrates framework consistency across different editing models.The supplementary analysis examines robustness to the editing component.

A.1. Data Examples

MemBench examples pair source and destination images with generated feedback and predictor scores. Scenes are ranked by memorability to create contrastive pairs, while evaluation uses unseen scenes starting from their least memorable images.

  • A.1. Data Examples: Each MemBench example shows a source image, destination image, generated feedback, and memorability scores assigned by predictor M.The source and destination are visually distinguished with red and blue frames.
  • A.1. Data Examples: Images within each scene are ranked by predictor M, and least–most memorable pairs form the contrastive training data.The pair construction uses images from the same scene.
  • A.1. Data Examples: Evaluation uses a random held-out set of unseen scenes, generating feedback from each scene’s least memorable image.This separates evaluation scenes from the contrastive training construction.
  • A.1. Data Examples: PPR10K RAW files are converted to JPEG while preserving aspect ratio to reduce storage and support MLLM vision processors.Conversions use rawpy and PIL with default parameters.

A.4. Memorability Predictor

The memorability predictor M regresses continuous image memorability scores from visual features and is trained across three datasets. It achieves the highest Spearman rank correlation among the evaluated memorability predictors.

  • Predictor architecture: M uses a frozen OpenCLIP visual feature extractor followed by an MLP regression head to output a memorability score in [0, 1].The predictor follows the approach of a frozen visual encoder with a trained regression head.
  • Training data: The regressor is trained on LaMem, MemCat, and SUN, using ground-truth memorability scores for each image.OpenCLIP ViT-SO400M-14-SigLIP-384 embeddings provide the regression input.
  • Validation: M achieves the highest Spearman’s rank correlation among the evaluated memorability models on the LaMem test set.Unlike most prior approaches, M leverages LaMem, MemCat, and SUN.
  • Prompt design: Early prompt refinement forbids image-referential phrasing and requires each action description to be self-contained.This change addresses feedback that described matching a target image instead of specifying the transformation.
  • Structured feedback: The feedback-generation pipeline uses constrained decoding to enforce a predefined JSON schema for structured action outputs.The schema supports reliable parsing and systematic storage, while allowing output fields to be modified when extending the format.
  • Feedback analysis: GPT-5 MINI categorizes atomic feedback sub-actions into Framing, Lighting, Posing, Semantics, Intent, and Aesthetics.The taxonomy enables downstream analysis of category frequencies and co-occurrence patterns.

B. MemCoach Additional Details

MemCoach steers a multimodal model’s internal activations toward memorability-aware behavior without training the model. Preliminary experiments report improved human memorability and evaluate the method across model configurations.

  • Core method: MemCoach is training-free and uses activation steering to modulate multimodal model representations.Experiments use PyTorch on a single NVIDIA A100 GPU.
  • Steering-vector extraction: The extraction stage captures residual-module activations from a selected language Transformer layer and averages them over the input sequence.Each input sequence is tokenized before activation capture and aggregation.
  • Inference: During inference, the steering vector is scaled by α and injected uniformly across the sequence at a selected layer l.The altered activation patterns steer the model toward memorability-aware behavior.
  • Model configurations: The optimal layer index l and steering coefficient α are tuned separately for each evaluated open-source multimodal model.Hyperparameters are optimized on a held-out training split.
  • Implications: Memorability optimization raises risks of manipulation, undue influence, and homogenized visual expression, while assistive contexts may support communication and learning recall.The paper emphasizes transparency, user agency, and context-aware application.
  • Preliminary experiments: In a preliminary memorability experiment, Figure 9 reports that both INTERNVL3.5 and MemCoach increase average memorability relative to source images, with MemCoach showing a larger improvement.The study used 47 valid users and averaged 15.6 annotations per image.

C.2. Human-in-the-loop Evaluation

The human-in-the-loop evaluation tests whether users can apply MemCoach’s feedback in real-world scenes. Across 27 scenes, the feedback consistently improves predicted memorability despite domain shift.

  • Results: Figure 9 compares human memory performance across three different image settings.The figure is used to assess memorability across settings in the human evaluation.
  • Study setup: A mobile app lets users request either a memorability score or a score accompanied by MemCoach’s actionable textual feedback.Users point the phone at a scene, capture a frame, and receive feedback through the live-camera workflow.
  • User interaction: Users can implement the displayed suggestion by adjusting framing or the scene and recapturing the image while observing score changes.The phone is kept as stable as possible when feedback concerns scene objects or subjects.
  • Results: 0.52 IR and +4.9% RM were achieved across 27 scenes, with MemCoach consistently improving memorability despite domain shift.The improvements reflect both the frequency and magnitude of successful enhancements.

C.3. Feedback Quality Evaluation

The feedback-quality evaluation measures clarity, scene relevance, and real-world feasibility, then analyzes factors associated with improved or worsened memorability. MemCoach provides interpretable and implementable guidance, but greater generality can reduce scene-specificity.

  • Evaluation design: Twenty-eight participants provided 381 annotations rating feedback on Clearness, Relevance, and Feasibility using 1–5 Likert scales.The ratings assess clarity of steps, scene-specificity, and realism of applying the advice.
  • Feedback quality: MemCoach maintains strong clearness and improves feasibility, while showing a slight reduction in scene-specificity.The results indicate a trade-off between precision and practical applicability.
  • Factors behind outcomes: 74.9% of positive effects are attributed to posing, followed by emotional saliency at 23.11% and framing at 1.59%.Failures are predominantly associated with over-normalization at 76.19%.
  • Evaluation proxy: Ground-truth and edited-image evaluations achieve 0.52 IR, +4.9% RM and 0.55 IR, +2.19% RM, respectively.The moderate correlation between destination-image memorabilities is ρ = 0.51, supporting the editor as an evaluation proxy.
  • Robustness: MemCoach remains effective across different memorability predictors and editing backbones, including Qwen-Image Edit1 and FLUX.2klein2.The cross-predictor experiment is intended to test whether improvements are artifacts of a single predictor.
  • Content scope: Preliminary non-human experiments on objects and landmarks report MemCoach performing on par with QWEN2.5VL at 0.79 IR.These experiments use the Yo’LLaVA dataset and the editing-proxy pipeline.
  • Qualitative feedback: The qualitative examples illustrate generated action sequences involving framing, posing, semantics, and lighting changes.Examples include repositioning subjects, changing viewpoints, modifying backgrounds, and adjusting lighting.
Loading 2602.21877v2…