PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Large Language Models for Generalized and Specialized Applications

Yifan Li, Zhixin Lai, Wentao Bao, Zhen Tan, Anh Dao, Kewei Sui, Jiayi Shen, Dong Liu, Huan Liu, Yu Kong

arXiv:2501.02765v1cs.CVcs.AI

TL;DR

Prior surveys provide limited coverage of VLLM applications, limitations, and future directions. This survey organizes applications into vision-to-text, vision-to-action, and text-to-vision and examines scenarios, social impacts, challenges, and ethical concerns. It concludes that addressing scalability, interpretability, and cross-modal understanding remains important to VLLM development.

  • Problem

    Previous surveys have not provided a comprehensive overview of VLLM applications, limitations, and future directions across vision modalities.

    Source: §I · p. 2

  • Method

    The survey categorizes VLLM applications as vision-to-text, vision-to-action, or text-to-vision and analyzes scenarios, social impact, challenges, and ethical concerns.

    Source: §I · p. 2

  • Results

    The survey highlights application scenarios, remaining challenges, and potential directions for future VLLM research.

    Source: §IV · p. 18

  • Takeaways & Limitations

    Addressing scalability, interpretability, and cross-modal understanding is described as crucial to realizing VLLMs’ potential.

    Source: §IV · p. 18

Abstract

from arXiv · show

Visual-language models (VLM) have emerged as a powerful tool for learning a unified embedding space for vision and language. Inspired by large language models, which have demonstrated strong reasoning and multi-task capabilities, visual large language models (VLLMs) are gaining increasing attention for building general-purpose VLMs. Despite the significant progress made in VLLMs, the related literature remains limited, particularly from a comprehensive application perspective, encompassing generalized and specialized applications across vision (image, video, depth), action, and language modalities. In this survey, we focus on the diverse applications of VLLMs, examining their using scenarios, identifying ethics consideration and challenges, and discussing future directions for their development. By synthesizing these contents, we aim to provide a comprehensive guide that will pave the way for future innovations and broader applications of VLLMs. The paper list repository is available: https://github.com/JackYFL/awesome-VLLMs.

I. INTRODUCTION

The introduction traces the shift from task-specific VLMs to instruction-following VLLMs and positions this survey as a broad account of their applications, challenges, and future directions. It then reviews image-to-text applications across general tasks, specialized grounding, document understanding, and retrieval.

I. INTRODUCTION

  • VLLMs use a vision encoder, connector, and LLM decoder to process visual inputs and generate instruction-following responses, extending VLM use to generalized and specialized applications.The survey describes this development after conventional task-specific VLMs and pretrained foundation-model approaches.Source: §I · pp. 1–2
  • The survey addresses earlier surveys’ limited coverage of VLLM applications, limitations, and future directions with a three-part taxonomy and analysis of social impact, challenges, and ethical concerns.Its categories are vision-to-text, vision-to-action, and text-to-vision.Source: §I · p. 2

A. Image-to-text

  • Image-to-text applications span captioning, VQA, dialogue, referring-expression tasks, OCR, and specialized-domain uses, with pretrained vision encoders improving contextual visual interpretation.Instruction-tuned models are described as handling multiple general-domain tasks rather than one discrete task.Source: A. Image-to-text · p. 2, 1) General domain application · p. 2

1) General domain application:

  • For segmentation, LISA decodes a <SEG> embedding into a mask, while other approaches use multiple target and rejection tokens or directly generate segmentation coordinates.The survey also describes mask or region extractors for fine-grained details and specialized backbones for segmentation.Source: 1) General domain application · p. 4
  • REC systems encode locations as text tokens or use region-level data and feature modules, while external localization pipelines provide another route to grounding.The survey describes Shikra [86] and Kosmos-2 [85] as using instruction-tuning datasets, and Ferret and GPT4RoI [88] as using region-focused modules.Source: 1) General domain application · p. 4
  • OMG-LLaVA [80] combines a universal perception module with an LLM for image-, object-, and pixel-level understanding, reporting state-of-the-art performance on refCOCO, refCOCO+, refCOCOg, and GranD.The passage names those datasets but provides no metric values.Source: 1) General domain application · p. 5
  • Document models address high-resolution OCR through adaptive cropping, frequency-domain inputs, token filtering, layout-aware tuning, and dynamic resolution up to 4K HD.InternLM-XComposer2-4KHD is reported to achieve state-of-the-art performance on DocVQA, ChartQA, InfoVQA, and TextVQA.Source: 1) General domain application · p. 5
  • Image-to-text retrieval includes single-stage shared-embedding selection and hierarchical retrieval that first finds documents, then extracts relevant passages for response generation.Wiki-LLaVA is cited as an example of hierarchical retrieval for tasks requiring external knowledge.Source: 1) General domain application · p. 5

2) Specific Domain Application:

The survey describes specialized image-to-text applications as requiring domain-focused datasets or structural changes to meet distinct task demands. It reviews applications in remote sensing, medicine, science and mathematics, graphic design and UI understanding, and financial analysis.

2) Specific Domain Application:

  • Remote-sensing VLLMs address differences in image color, resolution, context, and object scale through task-specific datasets, including datasets spanning multiple image types and tasks.EarthGPT incorporates SAR and infrared images, while H2RSVLM [277] includes unanswerable questions in its RS-VQA data.Source: 2) Specific Domain Application · p. 6
  • Medical VLLMs support image-question answering and report generation, with methods using multimodal inputs, medical instruction tuning, few-shot learning, and retrieval-augmented generation.RaDialog [128] is described as enabling interactive report modification and expert collaboration.Source: 2) Specific Domain Application · pp. 6–7
  • A two-stage strategy combining instruction tuning and reinforcement learning focuses on abnormal regions in medical images and outperforms other open-source models on abnormality recognition.The passage does not provide metric values for the comparison.Source: 2) Specific Domain Application · p. 7
  • Science and mathematics applications include science VQA and figure captioning, alongside geometry and mathematical reasoning models trained on specialized datasets and evaluated with dedicated benchmarks.Geo170K contains 170,000 geometry image-caption and question-answer pairs; MathVista consolidates over 6,000 examples from 31 datasets.Source: 2) Specific Domain Application · p. 7
  • Graphic-design and UI systems generate layouts or adapt visual encoders to recognize small interface elements, while financial models use chart datasets for analysis and stock-movement prediction.FinTral is reported to achieve state-of-the-art performance in Stock Movement Prediction, outperforming GPT-4V [255], Gemini-Pro [300], and LLaVA-NEXT.Source: 2) Specific Domain Application · p. 7

B. Video-to-text

Video-to-text applications cover diverse video understanding tasks, but translating sparse video content into dense language representations remains an open challenge.

B. Video-to-text

  • Video-to-text tasks include video captioning, video question answering, and video conversation.Inputs may be seconds-long or span a couple of hours, and may come from cached databases or real-time streams.Source: B. Video-to-text · p. 7
  • Figure 6 illustrates video-to-text applications.Source: Figure 6 · p. 8
  • The sparse content of videos and abstract nature of language create a semantic gap that makes video-to-text tasks challenging.Embedding long videos’ sparse causal information, including temporal activity evolution and physical motion dynamics, into a dense, tokenizable space remains open-ended.Source: B. Video-to-text · p. 8

1) General-purpose Video Understanding:

General-purpose video understanding models use large-scale pretraining and language generation to serve multiple tasks. Long-form video encoding remains a challenge, with methods exploring token selection, memory, and duration-aware strategies.

1) General-purpose Video Understanding:

  • General-purpose video LLMs use large-scale visual-text pretraining to develop knowledge for downstream video understanding tasks.Autoregressive modeling is reported to have better capability than masked language modeling, which is described as critical to video generation and classification.Source: 1) General-purpose Video Understanding · p. 8
  • LaViLa, Vid2Seq, and VideoLLaMA apply language-modeling approaches to activity recognition, captioning benchmarks, and multimodal temporal understanding, respectively.Other models following similar architectures include LLaMA-VID, MiniGPT4-Video, ST-LLM, LLaVA-Next-Video, LLaVA-OV, and Aria.Source: 1) General-purpose Video Understanding · p. 8
  • Long-form video models must encode video into visual tokens and project them into text-token space for language-model decoding.LLaMA-VID uses context and content tokens per frame; LVChat adjusts token count by duration; MA-LMM stores historical features in a memory bank; LongVU selects temporal frames and spatial tokens to reduce redundancy.Source: 1) General-purpose Video Understanding · p. 8
  • Efficiently encoding long video-text context into a unified token space for video conversation remains an active challenge.Source: 1) General-purpose Video Understanding · p. 8

2) Video Conversation:

Video conversation treats the language model as a human-facing conversational agent, while egocentric applications address long-term activity understanding and action programming. The survey describes systems spanning video chat, action anticipation, and executable programs.

2) Video Conversation:

  • Video conversation has attracted interest as a setting where the language model serves as a conversational agent for people.It entails multi-round reasoning over complex video content and ambiguous intent in long dialogue contexts; many video assistants exist, though most general-purpose video LLMs are not engineered specifically for conversation experience.Source: 2) Video Conversation · p. 8

3) Egocentric Understanding:

  • Egocentric video understanding supports applications such as helping workers complete complex mechanical repairs through visual understanding and plain-language interaction.Prior CLIP-based approaches are described as having fewer than one billion parameters and being too small for complicated video tasks.Source: 3) Egocentric Understanding · p. 8
  • Figure 7 illustrates vision-to-action applications.Source: Figure 7 · p. 9
  • Egocentric systems address action recognition, long-term anticipation, embodied tasks, natural-language queries, and executable action programs.Examples include GPT4Ego’s zero-shot action recognition, PALM and AntGPT for long-term anticipation, and LEAP for generating programs from egocentric videos.Source: 3) Egocentric Understanding · p. 9

C. Vision-to-action

Vision-to-action VLLMs take visual inputs conditioned on language instructions and generate actions for vehicles, robots, or software. The survey organizes these applications around autonomous driving and embodied AI tasks.

C. Vision-to-action

  • Vision-to-action VLLMs condition on language instructions and visual inputs—including images, videos, depth, or 3D data—to generate actions for vehicles, robots, or software.The survey groups applications into autonomous driving, embodied AI, and tool management.Source: C. Vision-to-action · p. 9

1) Autonomous Driving:

  • Autonomous-driving applications span perception, planning, and prediction using inputs such as multi-view images, video, BEV maps, and LiDAR.The survey notes that recent driving systems have shifted from rule-based to data-driven approaches and that many use LLMs for perception, planning, and prediction.Source: 1) Autonomous Driving · p. 9
  • Driving perception work includes VQA benchmarks, graph-structured reasoning, BEV scene understanding, risk-object localization, video-and-control understanding, and LiDAR scene description.HiLM-D uses separate branches for high-resolution perception and low-resolution reasoning; Dolphins introduces Grounded Chain-of-Thought with object spatial information.Source: 1) Autonomous Driving · p. 9
  • Driving planners generate maneuvers, control signals, or structured plans, using approaches such as interleaved vision-action prediction, hierarchical planning, memory, and end-to-end multimodal control.The described systems include ADriver-I, DriveVLM, DiLu, DriveGPT4, SurrealDriver, and LMDrive.Source: 1) Autonomous Driving · pp. 9–10
  • Driving prediction work forecasts object trajectories, 3D bounding boxes, and surrounding agents’ actions or potential dangers.NuPrompt prompts trajectory prediction, while BEV-InMLLM combines BEV features and multi-view video for perception, prediction, and planning.Source: 1) Autonomous Driving · p. 10

2) Embodied AI:

  • Embodied AI controls physical embodiments interacting with environments and is organized into perception, manipulation, planning, and navigation streams.The survey notes that conventional robot agents often focus on constrained tasks and lack common-sense knowledge.Source: 2) Embodied AI · p. 10
  • Embodied perception combines visual, depth, or 3D information for question answering, affordance grounding, object detection, scene understanding, and action grounding.Examples include OpenEQA’s episodic-memory and active tasks, 3DVG’s fusion of point-cloud geometry and image details, and 3D-LLM’s 3D question answering, captioning, and navigation.Source: 2) Embodied AI · p. 10
  • Embodied manipulation methods range from continuous-input planning and large robot datasets to API-based, imitation-learned, and instruction-tuned approaches.Open X-Embodiment includes 527 skills and more than 1M real-robot trajectories; LLARVA uses structured prompts and reports benefits from intermediate visual traces for aligning vision and action spaces.Source: 2) Embodied AI · p. 10

3) Automated Tool Management:

The survey extends tool management beyond text-only tasks to visual signals and physical-environment interactions, and organizes VLLM tool systems by architecture, integration mode, and application area.

3) Automated Tool Management:

  • The scope extends beyond simple logical tasks to include visual signals and interactions with physical environments as task complexity increases.The survey points readers to [334] for a review of pure text-based tool management.Source: 3) Automated Tool Management · p. 11
  • The taxonomy organizes VLLM tool management by system architecture, mode of integration, and application areas.It is intended to organize existing methodologies and provide insight into their functionalities and potential applications.Source: 3) Automated Tool Management · p. 11
  • Tool-managed systems can incorporate vision models, visual-language models, web-search APIs, and software libraries as callable tools.These tools support visual processing, responses to visual queries, information retrieval, code generation, software testing, and design tasks.Source: 3) Automated Tool Management · p. 11
  • System architectures range from tightly integrated and modular components to self-instructing systems and trajectory synthesis with self-exploration.Synthesized trajectories may serve as retrieval-based in-context examples or be distilled into smaller VLLMs.Source: 3) Automated Tool Management · pp. 11–12
  • Tool integration ranges from direct invocation in model outputs to indirect use through intermediary steps or processing layers.CLOVA is described as iteratively updating tool use through reflective learning.Source: 3) Automated Tool Management · p. 12

D. Text-to-vision

Text-to-vision applications turn prompts into images, 3D content, and videos. Within text-to-image, the survey describes approaches for generation, editing, synthesis, and visual-story creation.

D. Text-to-vision

  • Text-to-vision applications include generating images, 3D models, and videos from text, with text-to-image covering generation, editing, synthesis, and visual stories.The section describes these as generative applications of VLLMs.Source: D. Text-to-vision · p. 12, 1) Text-to-image · p. 12

1) Text-to-image:

  • Image-generation methods refine prompts, use visual tokenizers, or fuse modalities to connect language models with image-generation systems.GILL predicts fixed-size visual embeddings aligned with CLIP space; SEED and LaVIT use discrete visual tokens, while MiniGPT-5 [226] uses generative vokens.Source: 1) Text-to-image · p. 12
  • DiffusionGPT [231] uses domain-specific trees to guide selection of an image-generation model rather than translating prompts only into embeddings.Source: 1) Text-to-image · p. 12
  • Image-generation models can adapt to editing when given image inputs, with InstructPix2Pix data supporting more precise, contextually accurate modifications.Kosmos-G [227] and CoDi-2 [228] leverage data constructed by InstructPix2Pix.Source: 1) Text-to-image · p. 12
  • LaVIT supports image synthesis by generating visual tokens autoregressively, while Kosmos-G [227] supports zero-shot subject-driven generation from interleaved multi-image and text inputs.Source: 1) Text-to-image · p. 13
  • Visual-story generation creates image sequences that follow a narrative; [232] introduces a visual SEED tokenizer with discrete visual tokens carrying high-level semantics and aligned with text.The passage identifies coherent, contextually relevant narrative progression as the task's main challenge.Source: 1) Text-to-image · p. 13

2) Text-to-3D:

Text-to-3D work covers scene generation and editing as well as mesh generation, while text-to-video work combines language-model planning with video-generation methods and training strategies.

2) Text-to-3D:

  • Text-to-3D applications include scene generation, editing, and mesh generation, with spatial coherence and realistic interactions among complex elements identified as challenges.Source: 2) Text-to-3D · p. 13
  • For 3D scenes, LI3D [233] uses LLMs to interpret layouts and LLaVA for visual feedback, while 3D-GPT [234] decomposes modeling into procedural tasks.ShapeGPT [235] discretizes shapes into shape words, sentences, and multimodal paragraphs; GPT4Point aligns instructions with point-cloud data.Source: 2) Text-to-3D · p. 13
  • Additional assets and tools can support 3D scene understanding, planning, self-debugging, and memory; GPT4Point introduces Pyramid-XL to annotate 3D object-text pairs.Source: 2) Text-to-3D · p. 13
  • MeshGPT [238] uses a decoder-only transformer and latent quantized-embedding vocabulary to generate compact, detailed triangle meshes autoregressively.This approach bypasses traditional dense methods.Source: 2) Text-to-3D · p. 13

3) Text-to-video:

  • Text-to-video approaches use LLMs to produce scene layouts, multi-scene scripts, or video plans that guide generation and support consistency across scenes.LVD uses layouts to inform diffusion-based video creation; VideoDrafter [240] uses shared entities and reference images to maintain visual consistency.Source: 3) Text-to-video · pp. 13–14
  • Video-generation work also aligns visual tokenization with LLM learning; MAGVITv2 reports LLMs outperforming diffusion models on standard video-generation benchmarks.SVD [244] identifies text-to-image pretraining, video pretraining, and high-quality video fine-tuning as three training stages; VideoPoet [246] uses a decoder-only autoregressive transformer for multimodal inputs.Source: 3) Text-to-video · p. 14

III. ETHICS CONSIDERATION, CHALLENGES, AND FUTURE WORK

The survey discusses potential labor displacement, data-linked biases, and training and inference efficiency as ethical concerns and deployment challenges for VLLMs.

1) Impact on Labor Markets:

  • VLLM automation could displace work in graphic design, video editing, and medical imaging, while oversight and AI-human collaboration may create new roles.The survey identifies reskilling as one response to displacement risks.Source: 1) Impact on Labor Markets · p. 14

2) Bias:

  • Nonrepresentative training data can produce sampling bias, while gender, racial, and language biases may yield unequal outputs across groups and languages.The survey notes concerns for high-stakes uses such as hiring, law enforcement, and healthcare, and for multilingual visual analysis.Source: 2) Bias · p. 14

1) Efficiency:

  • VLLM efficiency challenges span both training and inference, with inference efficiency especially relevant to real-time systems and edge-device deployment.Source: 1) Efficiency · p. 14
  • Training DeepSeek-V3 [62] from scratch is reported to cost approximately $5.6M for 180,000 H800 GPU hours, consuming around 54,000 kWh and producing 25.65 metric tons of CO2.Source: 1) Efficiency · p. 14

2) Interpretability and Explainability:

The survey discusses interpretability and explainability challenges, hallucination risks and mitigation approaches, spatial-understanding gaps, and security and privacy priorities for VLLMs.

2) Interpretability and Explainability:

  • Interpretability concerns internal model properties, while explainability uses external post-hoc accounts; VLLMs' scale and complexity make both difficult.The survey links these difficulties to intricate decisions, large parameter counts, and extensive training data.Source: 2) Interpretability and Explainability · p. 14

3) Hallucination:

  • Hallucinations can stem from encoder misreadings, ambiguous inputs, biased or insufficient training data, and adversarial inputs.The survey also identifies spatial-relation and subtle-visual-cue errors as possible sources.Source: 3) Hallucination · p. 15
  • In sensitive domains, misinformation risks poor decisions, safety hazards, and lost trust; proposed mitigations include better data, cross-modal attention, constrained decoding, and factual checks.The cited risks include medical diagnosis, autonomous driving, and security systems.Source: 3) Hallucination · pp. 15–16

4) Spatial Understanding:

  • Current VLLMs face spatial-understanding challenges with egocentric visual data, including relational reasoning and egocentric-to-allocentric view transformations.The survey says progress in this area would benefit embodied agents and autonomous driving systems.Source: 4) Spatial Understanding · p. 16

1) Security & Pravacy:

  • Security priorities span privacy-preserving data handling, adversarial robustness, regulatory compliance, and secure post-deployment operation.Suggested directions include image anonymization, encrypted processing, adversarial-input detection, privacy impact assessments, and trusted execution environments.Source: 1) Security & Pravacy · p. 16

2) Efficiency:

The survey reviews efficiency directions for VLLMs across token use, training data, fine-tuning, and inference, then organizes interpretability and explainability work around data, model, and training and inference.

2) Efficiency:

  • Token-reduction methods prune or merge visual tokens, but automatic tuning across tasks without extra hyperparameters or performance loss remains challenging.Approaches include merging neighboring tokens, cross-modal attention guidance, and layer-wise pooling or pruning during training.Source: 2) Efficiency · p. 16
  • VLLM dataset distillation remains limited, despite the potential to improve training efficiency by reducing the data required for training.Prior data-distillation research has addressed CNNs, ViTs, diffusion models, and text-only LLMs.Source: 2) Efficiency · pp. 16–17
  • PEFT adapts pretrained models without tuning all parameters, but VLLM-specific methods remain a research direction.LoRA reduces trainable parameters and memory use; ReLoRA periodically updates pretrained weights with low-rank adaptors.Source: 2) Efficiency · p. 17
  • Inference-time token reduction without tuning remains difficult to achieve without significant performance loss, and current merging lacks instruction-specific adjustment.LLaVA-PruMerge merges similar visual tokens using CLS-token scores before the LLM decoder.Source: 2) Efficiency · p. 17
  • FastV prunes unnecessary visual tokens after selected layers, but retaining the initial predictions' KV cache creates memory overhead.The survey identifies KV-cache management for visual tokens as a research opportunity, especially for long visual contexts.Source: 2) Efficiency · p. 17

3) Interpretability & Explainability:

  • Data-focused interpretability studies connect visual inputs to decisions through regions of interest, attention maps, causal inference, attribution maps, and information-theoretic metrics.The survey describes these methods as examining input attributes and model outputs.Source: 3) Interpretability & Explainability · p. 17
  • Model-focused analyses examine visual and visual-textual tokens and embeddings, neurons, and layers to study processing, concepts, tasks, and model workflow.The survey notes that token-based methods can also address visual redundancy and hallucination issues.Source: 3) Interpretability & Explainability · p. 17
  • Training and inference analyses cover pretraining, alignment, gradients, regularization, Chain-of-Thought, and In-Context Learning.The survey describes gradient analysis and regularization as promising for reducing hallucination, and CoT and ICL as ways to examine reasoning and decision-making.Source: 3) Interpretability & Explainability · p. 17

4) Complex Reasoning:

The survey reviews methods for complex reasoning and examples of VLLMs in specialized applications, then concludes that scalability, interpretability, and cross-modal understanding remain limitations to address.

4) Complex Reasoning:

  • For complex reasoning, the survey discusses CoT, ToT, visual prompting, and tree search, alongside LLaVA-o1's staged reasoning and stage-level beam search.It also identifies reward models for planning and multi-VLLM judgment as promising directions.Source: 4) Complex Reasoning · p. 17

5) Other Applications:

  • Across specialized domains, nuanced patterns and domain-specific data or prior knowledge matter; face work addresses instruction-data scarcity and privacy with tailored datasets and models.Examples include EmoLA with FABA-Instruct, facial-prior-informed EMO-LLaMA, and CoT prompts for facial expression recognition.Source: 5) Other Applications · pp. 17–18
  • Anomaly-detection work includes an interactive video-anomaly interpretation tool, a training-free VAD method using captioning and prompting, and a benchmark for video-anomaly causation.The benchmark also introduces an evaluation metric designed to better align with human preferences.Source: 5) Other Applications · p. 18
  • Gaming applications range from Atari policies and basic-to-medium ARPG combat to instruction-following and long-horizon Minecraft tasks, as well as 40-minute RDR2 missions.CRADLE plans from screenshots and translates them into keyboard and mouse controls.Source: 5) Other Applications · p. 18

IV. CONCLUSION

  • The survey concludes that addressing scalability, interpretability, and cross-modal understanding limitations is crucial to realizing VLLMs' potential across applications.It presents the survey as a resource on application scenarios, challenges, and future research directions.Source: §IV · p. 18

Related papers

Loading 2501.02765v1…