PaperlayerView exact revision on arXiv

Source-linked AI summary

Focus Where It Counts: A Salience-Driven Vision-Language Model for Low Vision Assistance

Jiazhao Liang, Hao Huang, Shuaihang Yuan, Congcong Wen, Geeta Chandra Raju Bethala, Giles Hamilton-Fletcher, Yu Hao, John-Ross Rizzo, Mengyu Wang, Anthony Tzes, Yi Fang

arXiv:2608.28218v1cs.CV

TL;DR

Existing VLM captioning does not explicitly model the perceptual priorities that matter to low vision users, especially when safety-relevant information must be conveyed early. The paper builds salience-aware datasets and Salience-LLaVA, then evaluates salience ordering and deploys the system on assistive glasses, reporting consistent improvements across three benchmarks. Its main practical constraint is that inference remains computationally heavy and server-dependent.

  • Problem

    Existing VLMs and captioning datasets do not explicitly represent human perceptual priorities or sufficiently realistic low vision contexts, limiting emphasis on relevant scene information.

  • Method

    The paper curates Salience COCO, Salience Flickr, and Salience VizWiz, integrates saliency rankings into Salience-LLaVA, and evaluates ordering with SCMI.

  • Results

    Salience-LLaVA shows consistent improvements over existing methods across standard captioning metrics on three benchmark datasets and improves salience alignment measured by SCMI.

  • Takeaways & Limitations

    The framework produces descriptions that better align with human attention and intent and demonstrates real-time assistive applicability through deployment on a customized wearable device.

  • Takeaways & Limitations

    The model remains computationally heavy, requiring server-based inference and limiting fully mobile use when network access is unreliable or unavailable.

Abstract

from arXiv · show

Vision-language models (VLMs) are rapidly progressing and offer promising capabilities for assistive technologies supporting persons with blindness or low vision. However, existing VLMs are primarily designed for general-purpose captioning and do not explicitly model human perceptual priorities, thereby limiting their ability to emphasize the most relevant information in a scene. To address this gap, we propose a salience-driven captioning framework that prioritizes scene elements according to their importance for human-centered assistance. We curate three salience-aware datasets, namely, Salience COCO, Salience Flickr, and Salience VizWiz, with object-level salience annotations designed to reflect the visual information most relevant to low vision users across different environments. Building on these datasets, we introduce Salience-LLaVA, a salience-aware VLM that incorporates salience cues to generate captions in which important elements are mentioned in the order of importance. Our work makes four main contributions. We build salience-aware datasets verified by low vision participants, propose Salience-LLaVA to describe objects in the order of importance, introduce SCMI to evaluate ordering accuracy, and deploy the system on assistive glasses to demonstrate real-world practicality. Code and datasets are available at: https://github.com/topo-focus/Topofocus

I. INTRODUCTION

Existing VLMs can describe scenes but do not reliably prioritize information according to low vision users’ perceptual and safety needs. Salience-LLaVA addresses this gap by integrating human-centered salience rankings so important elements appear earlier in captions.

  • General-purpose VLMs may miss subtle but safety-critical cues such as uneven curbs, oncoming vehicles, and misaligned tactile paths.The paper frames this limitation as especially important for independent mobility.
  • Human annotation ranks Crosswalk, Traffic Light, People, and Bicycle by importance, whereas original LLaVA begins with People and misorders Vehicles and Bicycle.This example illustrates the mismatch between generic captioning order and human-centered relevance.
  • Low visual acuity, limited field of view, and night blindness can make small but critical objects easy to miss.Delayed or buried descriptions may reduce the practical value of otherwise complete captions for safety and navigation.
  • The proposed framework curates salience-annotated datasets and integrates saliency rankings from a dedicated module so relevant elements are described first.The approach targets both dataset curation and model architecture.

1) Datasets With Salience-Aware Captions Tailored to Low Vision Users:

The paper introduces three datasets designed to capture human-centered salient objects and integrates salience features into a VLM. The resulting captions prioritize visually crucial details that generic VLMs may overlook, with audio feedback delivered through glasses hardware.

  • Salience COCO, Salience Flickr, and Salience VizWiz capture complex, real-world salient objects missing from existing captioning benchmarks.
  • The salience-aware VLM integrates image salience features to generate descriptions that prioritize visually crucial details often overlooked by generic VLMs.
  • The model is integrated with glasses that capture visual input and deliver audio feedback for assistive use.The implementation is intended to provide an accessible and intuitive experience for visually impaired users.

A. VLM for Image Captioning

The paper positions salience-aware captioning as a response to datasets and VLM pipelines that do not represent low vision users’ perceptual priorities or complex everyday scenes. Its figures and discussion connect salience maps, object ranking, and dataset statistics to human-centered caption generation.

  • Salience maps highlight influential image regions or features in ways intended to align model focus with human visual priorities.
  • Figure 2 depicts processing an image into a salience map, detecting and reordering salient objects, and generating a salience-aware description.
  • Existing captioning datasets often lack salience information, sufficient scene complexity, and contexts representative of low vision users’ daily activities.
  • The dataset curation rationale emphasizes multi-object, semantically rich everyday environments over less complex scenes.Examples include kitchens, living rooms, and urban streets; outdoor sports are excluded because they differ functionally from cluttered everyday settings.
  • Figures 3 and 4 present examples of images, salience masks, captions, word counts, and sentence-length statistics for the three datasets.

1) Salience COCO:

Salience COCO repurposes COCO by filtering for complex, low-vision-relevant scenes and generating captions from ranked object salience. The same configuration supports consistent salience-aware annotation across the related datasets.

  • Salience COCO filters COCO images to retain at least two supercategories and scenes relevant to everyday low vision needs.The selection focuses on categories such as indoor objects and crossroads.
  • A pre-trained QAGNet with a 0.7 confidence threshold supplies saliency scores that rank retained objects by mean saliency intensity.
  • Janus Pro 7b receives the ranked object list and original image to generate ground-truth captions expressing relative object importance.
  • Salience COCO contains 6,636 training images and 1,659 test images.
  • The confidence threshold, saliency ranking procedure, and caption template remain fixed when generating Salience Flickr and Salience VizWiz.

3) Salience Vizwiz:

The repurposed VizWiz data pipeline adds object-level annotations and filters images for contextual richness, producing datasets intended to reflect low vision users’ everyday visual challenges. Human verification found good agreement between model-generated salience rankings and low vision participants’ judgments.

  • Dataset construction: VizWiz images were annotated with COCO-style bounding boxes, labels, supercategories, and item categories using YOLO-World.Images were then filtered to retain at least two supercategories for contextual richness.
  • Dataset construction: The repurposed datasets address missing salience information, insufficient scene complexity, and nonrepresentative contexts in existing captioning data.The selection process targeted everyday visual challenges encountered by low vision users.
  • Human verification: Human verification assessed whether annotations ranked objects according to their perceived importance for low vision users, rather than only checking object presence.The evaluation followed established agreement protocols.
  • Human verification: 120 images were evaluated by four low vision participants after Institutional Review Board approval.The sample included 40 images from each of the three salience datasets.
  • Human verification: Cohen’s κ was 0.70 across verification images, indicating good agreement between model-generated salience annotations and low vision participants’ annotations.The reported agreement supports the alignment and reliability of the salience rankings.
  • Model overview: Salience-LLaVA combines a CLIP-ViT vision branch, a QAGNet salience branch, and Vicuna for instruction-conditioned caption generation.The architecture is designed to generate salience-aware image captions.

A. Vision Backbone Branch

The vision backbone extracts patch-level visual representations with CLIP-ViT, while the saliency branch uses multiscale ResNet-50 features, learnable instance queries, and QAGNet to rank salient objects contextually.

  • Vision Backbone Branch: CLIP-ViT resizes and tokenizes an image into N patches, producing patch features with dimension d = 768.An MLP maps these features to Vicuna’s input dimension D = 4096.
  • Multiscale Saliency Branch: The saliency branch processes ResNet-50 multiscale feature maps with a Mask2Former-inspired pixel decoder to preserve semantic and spatial information.The decoder fuses successive convolutional-layer features into a coherent representation.
  • Multiscale Saliency Branch: N learnable instance queries propose potential salient instances before entering QAGNet for saliency instance ranking.QAGNet uses human fixation data to align ranking signals with natural visual perception.
  • Multiscale Saliency Branch: QAGNet’s hidden layers construct single-scale, multiscale, and global relationship graphs to model instance-level cues and relationships.These graph stages combine information across scales and instances.
  • Multiscale Saliency Branch: The graph module outputs a saliency feature z ∈ R^N×D, which a linear rank head converts into relative saliency scores for detected instances.The representation integrates intrainstance multiscale details and interinstance relational information.

C. Saliency Map Construction and Fusion

Salience-LLaVA projects and fuses saliency features with original visual features, combines them with tokenized instructions, and feeds the unified representation to Vicuna for caption generation.

  • Saliency Map Construction and Fusion: A two-layer MLP projects the saliency feature z so salient cues can be integrated with the original visual representation.The projection uses two linear layers with a ReLU activation.
  • Saliency Map Construction and Fusion: The projection parameters are represented by weight matrices W1 and W2 and bias vectors b1 and b2.The cited description identifies these as the parameters of the two linear layers.
  • Saliency Map Construction and Fusion: Projected saliency features are concatenated with original visual features to form fused features for Vicuna caption generation.The resulting fused representation has dimension R^N×(D+D_s).
  • Caption Generation: Tokenized instructions and projected fused visual features are concatenated into c ∈ R^(L+N)×D_e before entering Vicuna.This unified representation enables salience-aware caption generation.
  • Loss Function: Caption tokens are trained with cross-entropy loss from the unified image-instruction representation.The target caption contains T tokens generated sequentially from c.
  • Loss Function: LoRA updates attention weights with low-rank matrices while only the new saliency projection layer is trained and other model parameters remain frozen.This preserves pretrained knowledge and reduces the risk of overfitting.

V. EXPERIMENT

The evaluation combines conventional captioning metrics with SCMI, which measures whether generated captions preserve human-centered salience ordering. Results are compared with several recent caption-generation and salience-oriented baselines.

  • Evaluation metrics: Conventional metrics include BLEU, CIDEr, ROUGE, and METEOR, providing complementary measures of caption linguistic quality.
  • Evaluation metrics: SCMI evaluates whether captions identify important object instances in a salience-consistent order.It is based on the longest common subsequence between ground-truth and predicted sequences.
  • Evaluation metrics: Ground-truth sequences rank object instances by salience, while predictions provide the word sequence evaluated for order consistency.
  • Baselines: The evaluation compares Salience-LLaVA with DeCap, GRIT, Tag2text, LLaVA variants, SCOPE, and WalkVLM.The compared methods include zero-shot, fine-tuned, salience-driven, and low-vision-oriented approaches.

B. Results

Across Salience COCO, Flickr, and VizWiz, salience-aware modeling improves caption quality and alignment with human salience ordering. The strongest gains are visible in SCMI and Wu-Palmer SCMI, including under challenging VizWiz conditions.

  • Overall results: 0.60 SCMI and 0.63 Wu-Palmer SCMI indicate stronger salience-driven object ordering than n-gram metrics alone capture.The authors report that salience cues strengthen linguistic accuracy and perceptual alignment.
  • Salience Flickr: On Salience Flickr, BLEU-1 rises from 0.44 to 0.50 and CIDEr from 0.42 to 0.45 over LLaVA-FT.METEOR increases from 0.37 to 0.42, ROUGE-1 from 0.51 to 0.57, and SCMI from 0.73 to 0.79.
  • Salience VizWiz: On Salience VizWiz, BLEU-4 improves from 0.13 to 0.17, CIDEr from 0.31 to 0.36, and SCMI from 0.60 to 0.71 over LLaVA-FT.WalkVLM matches the method’s Wu-Palmer SCMI at 0.79 on this challenging dataset.

C. Ablation Study

Ablations show that the learned MLP projection module is important for integrating salience and vision features. The experiments also include real-world, low-light, and alternative-salience-network evaluations, alongside smart-glasses deployment diagrams.

  • Additional evaluations: The ablation section reports additional evaluations using real-world data, low-light conditions, and alternative salience networks.
  • Deployment: The real-world experiment uses smart glasses equipped with smart sensors, an earphone, and a camera.
  • Deployment: A sequence diagram describes interaction with the backend captioning model through an ESP32 S3.
  • MLP ablations: Replacing the MLP with a CNN reduces BLEU scores by up to 0.15, while CIDEr and SCMI fall by 0.28 and 0.15.The authors associate these reductions with the MLP’s role in aligning salience and vision features.

D. Robustness Under Low-Light Conditions

Low-light conditions degrade salience ranking more than surface-level caption quality. The largest reported semantic-ordering drop occurs on the already noisy Salience VizWiz images.

  • Evaluation setup: Low-light testing uses 50 manually selected images from each dataset and compares them with the full test sets.The selected images represent reduced brightness and weaker contrast relevant to low-vision assistance.
  • Results: On Salience COCO, BLEU-1 remains 0.52 while SCMI drops from 0.60 to 0.53 and Wu-Palmer SCMI from 0.63 to 0.53.
  • Results: On Salience Flickr, BLEU-1 stays at 0.50 while Wu-Palmer SCMI decreases from 0.83 to 0.78.
  • Results: On Salience VizWiz, Wu-Palmer SCMI falls from 0.79 to 0.63 under low light.The authors attribute the larger degradation to the combination of poor lighting and already challenging image quality.
  • Interpretation: Overall, low light affects salience estimation more than language generation.The authors suggest improving visual perception before ranking through low-light enhancement or illumination-aware saliency estimation.

F. Real-World Experiment

The real-world experiment deploys Salience-LLaVA on a wearable device for near-real-time auditory scene analysis. The system outperforms original LLaVA on standard captioning metrics, but low-resolution edge-device images reduce SCMI through less precise object labels.

  • System deployment: The wearable device captures images and provides almost real-time scene analysis through auditory feedback.Its interaction flow connects the user, wearable hardware, and backend captioning model.
  • Evaluation: In the real-world experiment, Salience-LLaVA outperforms original LLaVA on BLEU, CIDEr, METEOR, and ROUGE, while SCMI remains relatively low.The experiment uses YOLO-World for COCO-style annotations and Janus Pro-7b for ground-truth captions.
  • Evaluation: Low-resolution edge-device images reduce visual detail, causing less specific detections and lower SCMI than on higher-resolution benchmark datasets.Visually similar categories may collapse into broader labels, limiting exact object classification.
  • Practical validation: The framework is presented as practically effective through deployment on customized wearable glasses for real-time assistive scenarios.The deployment demonstrates applicability in everyday settings, within the system’s hardware and inference constraints.
Loading 2608.28218v1…