A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges

1. Introduction

• Text-only models face limited high-quality text supplies and limitations on tasks requiring joint visual, spatial, temporal, and linguistic reasoning.VLMs combine visual and text inputs, but introduce visual hallucination, brittle grounding, and shallow multimodal reasoning.Source: §1 · p. 1
• VLMs already support visual question answering, web and GUI agents, document understanding, robotics, and autonomous driving.The survey also notes failure modes including visual hallucination, brittle grounding, and shallow multimodal reasoning.Source: §1 · p. 1
• The survey focuses on representative frontier families and benchmark directions that clarify how the field is changing, rather than cataloguing every release.Its three emphases are architectural evolution, expanded evaluation, and post-training and reinforcement learning in multimodal reasoning.Source: §1 · pp. 1–2

2. State-of-the-Art VLMs

• Progress is now organized around families with multiple reasoning modes, context tiers, and deployment targets, rather than isolated model releases.Figure 1 tracks public model scale from sub-billion contrastive systems to frontier multimodal families; Table 1 summarizes representative releases.Source: §2 · p. 2
• Architecture has moved from contrastive or bridged two-tower systems through LLM-backbone adapters to native multimodal pretraining with image, video, audio, and text sharing a trunk.The simple account of adding a vision encoder to an LLM no longer describes this progression.Source: §2 · p. 2
• The survey uses this architectural view to organize evaluation, applications, and discussion of post-training, alignment, safety, and data-quality bottlenecks.Evaluation covers spatial, temporal, embodied, and calibration-aware assessment; later sections connect capabilities to agents, document intelligence, media generation, and embodied systems.Source: §2 · p. 2

3. Building Blocks and Training Methods

• Modern VLMs span contrastive or bridged two-tower, projector-based LLM-backbone, native multimodal-input, unified input/output, and emerging world-action architectures.The key differences are where modality fusion occurs, whether outputs remain text-only, and whether models are expected to generate and act as well as describe.Source: §3 · p. 2, Figure 2 · p. 4

3.1. Common Architecture Components

• Vision encoders extract image or video features for integration with language representations, while some models use dedicated text encoders and others rely on an LLM.CLIP aligns image and text embeddings contrastively; newer LLM-backbone models can integrate vision through projection or cross-attention mechanisms.Source: §3.1 · p. 2

3.2. Building Blocks of Training From Scratch

• From-scratch contrastive training maps visual and textual inputs into a shared embedding space, bringing related image-text pairs closer and separating unrelated pairs.CLIP, BLIP, and ALIGN use this approach on large-scale image-text datasets.Source: §3.2 · p. 5

3.3. Building Blocks of Using LLMs as Backbone

• Projectors map visual features into an LLM-compatible embedding space; backbone systems may jointly train components or selectively freeze them to preserve pretrained knowledge.The explicit projector may shrink, become distributed, or disappear in newer native multimodal families.Source: §3.3 · p. 5

3.4. Newer Architectures

• Adapter-era models inject visual features through a projector, resampler, or cross-attention bridge, while native multimodal trunks fuse modalities earlier and may generate non-text outputs.Unified models use decoder heads for image, video, or audio generation; world-action systems extend the trunk toward predicting observations and selecting actions.Source: §3.4 · pp. 5–6

3.5. VLM Alignments

• Alignment combines supervised fine-tuning for formats and tool use, preference optimization for response quality and safety, and reinforcement learning for hard multimodal reasoning.GRPO-style rule-based reinforcement learning is a prominent 2025–2026 trend for visual reasoning, spatial reasoning, and multimodal generation.Source: §3.5 · p. 6
• Multimodal alignment must assess partially linguistic evidence and distinguish visual grounding from plausible but unsupported answers.Approaches include human preference pairs, rule-based correctness rewards, spatial-faithfulness rewards, and teacher or verifier guidance; reported rule-based methods achieve stable accuracy and response-length improvements with sparse rewards and minimal filtering.Source: §3.5 · p. 6
• Reinforcement Learning with Verifiable Rewards uses a direct verification function instead of a trained reward model to evaluate correctness.The passage describes this as streamlining the reward process while maintaining alignment with task correctness criteria.Source: §3.5 · p. 6

4. Benchmarks and Evaluation

• Recent VLM benchmarks span temporal reasoning, embodied interaction, GUI/computer use, and evaluation reliability in realistic settings.The survey identifies faithful measurement of grounded multimodal competence as the hardest open evaluation problem.Source: §4 · p. 6, Table 3 · p. 8, Table 4 · p. 9

4.1. How Are Benchmark Data Collected

• Benchmark datasets use fully human annotation, synthetic-data scaling with partial human validation, or synthetic-data scaling with full human validation.Fully human-annotated examples include MMMU questions collected by 50 college students and VCR questions written by Mechanical Turks.Source: §4.1 · p. 6

4.2. Evaluation Metrics

• Multiple-choice and yes/no formats are easy to score, but models may guess above chance and popular evaluations remain far from broader general-intelligence assessment.MMLU covers 57 distinct tasks; emerging benchmarks also target judge reliability and calibration.Source: Figure 3 · p. 7, §4.2 · p. 10

4.1. How Are Benchmark Data Collected

• Synthetic question generation scales datasets with less human effort, but caption- and text-based generation can yield questions answerable without substantial visual reasoning.Pipelines use automatic filters and crowdworker validation to address inaccurate or unfaithful generated content.Source: §4.1 · pp. 7–8
• Simulator interaction traces support agent training and evaluation, while transferring simulator-trained agents to real applications may expose a sim2real gap.Rule-based acquisition is used where human-labeled or synthetic datasets are hard and expensive to obtain.Source: §4.1 · pp. 8–9

4.2. Evaluation Metrics

• Answer matching suits short-form answers, but exact-match and token-overlap metrics can misjudge semantically equivalent or complex generative responses.LLM judges can assess long-form answers, yet their outputs may vary with model updates or prompt changes.Source: §4.2 · p. 9

5.1. Computer-Use, GUI, and Web Agents

• GUI and web agents combine fine-grained OCR, visual-element grounding, long-horizon planning, tool use, and error recovery to act on screens and websites.Benchmarks increasingly assess computer use and reward-model quality, not only answer accuracy.Source: §5.1 · pp. 10–11

5.2. Document Intelligence and Accessibility

• Document-centered applications combine OCR, layout understanding, numerical extraction, and grounded language, while accessibility systems turn documents into diagrams or visual descriptions.ChartLLaMA uses specialized post-training to improve chart interpretation where generic models may hallucinate values or axes.Source: §5.2 · p. 11

5.3. Generative Media and Design

• VLMs serve both lightweight social-content creation, such as memes, and richer media generation, including video systems that point toward storyboarding, editing, and synthesis.Shared multimodal trunks that understand and generate visual content narrow the boundary between analysis, editing, and creation.Source: §5.3 · p. 11

5.4. Robotics and Autonomous Systems

• Robotics, navigation, and driving use VLMs for manipulation, exploration, object-goal search, scene understanding, and planning, but expose calibration, latency, sim-to-real, and action-reliability weaknesses.These settings can convert model outputs into physical or safety-critical behavior.Source: §5.4 · p. 11

5.5. High-Stakes Domain Assistants

• In healthcare, education, agriculture, science, and climate, VLMs are explored as expert copilots where humans remain in the loop.Across domains, copilots must ground perception, reason over domain context, and communicate uncertainty appropriately.Source: §5.5 · p. 11

6. Challenges of VLMS

• VLM challenges include hallucination, safety, fairness, alignment, training efficiency, and data scarcity, all relevant to reliable and ethical deployment.The survey emphasizes particular concern for marginalized users.Source: §6 · p. 11

6.1. Hallucination

• Hallucination benchmarks assess nonexistent-object claims through caption metrics, yes-no object-existence questions, fine-grained VQA samples, and tests of image changes or dependent, unanswerable questions.CHAIR, POPE, M-HalDetect, BEAF, and HallusionBench represent these approaches.Source: §6.1 · pp. 11–12
• Hallucination datasets also include GPT-4-generated GAIVE with 400K samples and Hal-Eval with 2M image-caption pairs, alongside LLM-free AMBER.Halle-Switch studies data amount, quality, and granularity; Hallu-Pi provides 1,260 annotated images, while AutoHallusion automates benchmark generation.Source: §6.1 · pp. 11–12

6.2. Safety

• Safety evaluations span harmful-query scenarios, image-text attacks, malicious image queries, face-spoofing detection, and methods that bypass or break multimodal refusal alignment.Examples include SafeBench, MM-SafetyBench, JailbreakV, SHIELD, HADES, and imgJP.Source: §6.2 · p. 12

6.3. Fairness

• VLM bias benchmarks examine religion, nationality, disability, sexual orientation, and medical-image disparities; reported findings include group preferences and cultural-performance differences.CulturalVQA reports better performance for North American cultures and worse performance for African and Islamic ones.Source: §6.3 · p. 12

6.4. Alignment

• Multimodal alignment work includes self-critique, projectors, paired-data-free semantic alignment, unified encoders, and benchmarks for text-image misalignment.The section also describes physics and video-generation evaluation, while noting challenges in advanced video evaluation and real-world fidelity.Source: §6.4 · p. 12
• Efficient VLM adaptation includes LoRA and feedback-based alignment, but rule-based reinforcement learning raises computation costs, while PPO and DPO require extensive human annotation.The cited trade-offs are between computation and annotation resources.Source: §6.4 · p. 12

6.5. Data Scarcity

• VLM reliability depends on training-data availability and diversity, while model scale and scarce high-quality data make continued improvement difficult.Proposed directions include self-supervised learning, rule- or tool-generated synthetic data, simulator interactions, and human-demonstration videos; further insight into mass-scale training and alternatives to internet-scale data remains needed.Source: §6.5 · p. 13

• The survey reviews notable VLM architectures, trends, applications, and challenges as a roadmap for practitioners.Source: §7 · p. 13

• The survey covers representative families, benchmarks, and post-training methods through 2026, but new releases can outpace the manuscript; its repository and website offer nearer-term updates.Source: §8 · p. 13