PaperlayerView exact revision on arXiv

Source-linked AI summary

From Training-Free to Adaptive: Empirical Insights into MLLMs' Understanding of Detection Information

Qirui Jiao, Daoyuan Chen, Yilun Huang, Yaliang Li, Ying Shen

arXiv:2401.17981v3cs.CVcs.AI

TL;DR

MLLMs struggle to interpret fine visual details, and most text-based detection-infusion methods have been used without training. This paper compares training-free integration, retraining, and fine-tuning, finding that fine-tuning performs best and retains benefits after detector replacement.

  • Problem

    Most studies infuse textual detection information without training, leaving whether adaptive training improves MLLMs’ comprehension and performance insufficiently explored.

    Source: §1 · p. 2

  • Method

    The study compares training-free infusion, retraining from scratch, and fine-tuning a pretrained MLLM with textual detection information, while examining original abilities and detector interchangeability.

    Source: §1 · p. 2

  • Results

    Fine-tuning outperforms training-free infusion and retraining, improving performance by up to 6.71% over training-free integration across ten benchmarks, and retains benefits after detector replacement.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Fine-tuning a pretrained MLLM with textual detection information can yield better performance than leaving it untrained and support retaining gains after detector replacement.

    Source: §6 · pp. 10–11

    The DINO detector used in prior experiments is closed-set and can detect only 80 common object categories, limiting coverage of uncommon objects and specific entities.

    Source: §4.5 · p. 9

Abstract

from arXiv · show

Despite the impressive capabilities of Multimodal Large Language Models (MLLMs) in integrating text and image modalities, challenges remain in accurately interpreting detailed visual elements. Vision detection models excel at recognizing fine-grained image details, prompting researchers to use them to enhance MLLMs. One effective strategy is to infuse detection information in text format, which has proven simple and effective. However, most studies utilize this method without training, leaving the potential of adaptive training largely unexplored. Adaptive training could significantly enhance MLLMs' comprehension of unique inputs while filtering out irrelevant information. This paper addresses the crucial question: How does training impact MLLMs' understanding of infused textual detection information? We systematically experiment with various representative models to evaluate the effects of training-free, retraining, and fine-tuning strategies. We also examine the influence of training on MLLMs' original abilities and the interchangeability of detection models. Our findings indicate that fine-tuning a pre-trained MLLM to incorporate textual detection information delivers superior results compared to training-free and retraining methods, improving performance by 6.71% across 10 widely recognized benchmarks. Furthermore, fine-tuning enables MLLMs to retain performance enhancements even when detection models are swapped, indicating improved understanding of formatted textual data. We release our codes to support further exploration of fusion strategies for vision detection models and the enhancement of MLLMs' fine-grained multimodal capabilities.

1 INTRODUCTION

MLLMs can produce coherent but image-misaligned responses and struggle with fine visual details. This work compares training strategies for adding textual detection information and finds that fine-tuning outperforms training-free integration and retraining.

1 INTRODUCTION

  • MLLMs can miss fine image details, contributing to coherent responses misaligned with image content.The paper describes this as hallucination.Source: §1 · p. 1
  • Most prior work applies textual detection information without training, leaving adaptive training’s effect on MLLMs’ comprehension insufficiently explored.The paper asks whether adaptive training can improve performance beyond training-free integration.Source: §1 · p. 2
  • The study compares training-free, retraining, and fine-tuning strategies, and examines effects on original image understanding and detection-model interchangeability.Experiments cover representative models including LLaVA-1.5, DINO, PaddleOCRv2, and Grounding DINO.Source: §1 · p. 2
  • Fine-tuning improves LLaVA-1.5 by up to 6.71% over training-free integration across ten benchmarks, outperforming both training-free and retraining strategies.Detection-model outputs are converted to text and supplied to the LLM.Source: §1 · p. 2
  • Retraining with textual detection information harms MLLMs’ interpretation of image-encoder features, whereas fine-tuning avoids this problem.The paper links retraining to reduced performance on tasks unrelated to detection.Source: §1 · p. 3
  • Fine-tuned MLLMs retain performance improvements when the deployed detection model is replaced, and can discard irrelevant information after replacement.The reported replacement experiment also finds further enhancement with an open-set detector and supports dynamic detection guided by user queries.Source: §1 · p. 3
  • The work provides empirical evidence on training strategies for textual detection infusion and makes its code publicly available for further research.The authors frame the study as addressing a gap in evidence on adaptive versus training-free methods.Source: §1 · p. 3

2 BACKGROUND AND MOTIVATION

MLLMs struggle with fine-grained visual targets, while detection models can provide complementary information. This section reviews approaches to improving detection capabilities and motivates studying adaptive training for text-form detection outputs.

2.1 MULTIMODAL LARGE LANGUAGE MODELS (MLLMS)

  • LLMs are primarily designed for text-based tasks, so modality-bridging modules reconcile the representational differences between text and images.This background frames the MLLM architectures discussed in the section.Source: §2.1 · p. 3
  • MLLMs bridge image and text representations using learnable queries, projection-based interfaces, or parameter-efficient tuning modules.Examples include Flamingo and BLIP-2, LLaVA, and LLaMA-Adapter, respectively.Source: §2.1 · p. 3
  • MLLMs can miscount sheep, miss a partially obscured pedestrian, and misread a license plate, illustrating object-counting, localization, and text-recognition weaknesses.The paper notes that detection and OCR models perform better on detection and recognition tasks.Source: §2.1 · p. 3

2.2 ENHANCING DETECTION CAPABILITIES FOR MLLMS

  • Detection-capability methods include expanding datasets, restructuring image encoders, integrating detectors into model outputs, and supplying detector outputs as text.The text-based category includes methods that provide textual grounding clues, classes, or locations to LLMs.Source: §2.2 · pp. 3–4
  • Text-based approaches require less extensive training data for new detection modules than encoder-restructuring or feature-alignment methods, while still delivering commendable results.The paper therefore identifies text-based methods as likely to be used more often in practical applications.Source: §2.2 · p. 4
  • Most text-based detection-infusion research is training-free, while prior LLM work suggests that fine-tuning can improve comprehension of specially formatted text and help discard redundant information.The paper proposes investigating adaptive training using the studied MLLMs’ limited but high-quality original training data.Source: §2.2 · p. 4

3 INVESTIGATION METHODOLOGY FOR THE INFUSION OF TEXTUAL DETECTION INFORMATION

The study converts object-detection and OCR outputs into text, combines those features with image features, and compares three infusion strategies across ten benchmarks.

3.1 TEXT-BASED DETECTION INFORMATION CONSTRUCTION

  • Object-detection text encodes object classes and center coordinates, grouping objects by category; OCR text combines recognized content with its center coordinates.Each format is preceded by an instruction sentence, and the authors report that the construction method compresses the text while expressing useful information.Source: §3.1 · p. 5

3.2 STUDIED MODEL ARCHITECTURE

  • The architecture concatenates image-level features from CLIP-ViT and detection features from DINO and PaddleOCRv2 before inputting them to the backbone LLM.The detection outputs are converted to sentences and embedded with the backbone LLM’s embedding layers.Source: §3.2 · p. 5

3.3 STUDIED INFUSION STRATEGIES

  • LLaVA-1.5 is the main-page representative because its architecture aligns with many leading MLLMs; appendix experiments on Qwen-VL show similar trends.The study also considers other models in the appendix.Source: §3.3 · p. 5
  • Training-free infusion adds detection text without training; retraining starts from scratch, while fine-tuning trains LoRA modules for one epoch on a pretrained MLLM.The retraining pipeline also trains the MLP module; the fine-tuning strategy freezes the pretrained MLP and LLM weights.Source: §3.3 · p. 6

3.4 QUANTITATIVE EVALUATION SETTINGS

  • The evaluation uses ten benchmarks spanning VQA, perception, reasoning, text recognition, complex fine-grained tasks, and object localization.GQA is evaluated on an unambiguous-question subset, GQA*; the authors also compute average percentage improvement over the original models across benchmarks.Source: §3.4 · p. 6

4 MAIN RESULTS AND ANALYSIS

Across the evaluated strategies, fine-tuning with textual detection information performs best while preserving the original model’s image-feature understanding. The experiments also find that fine-tuning retains benefits after replacing the detector, though the open-set detector introduces noise.

4.1 OVERVIEW AND ORGANIZATION

  • 3.99% and 3.30% are the respective overall improvements of FTBI-7B and FTBI-13B over LLaVA-1.5; FTBI-13B exceeds TFI-13B by 6.71%.Table 1 compares the infusion strategies and original models on 10 benchmarks.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 LESSON 1: THE ORIGINAL MLLM STRUGGLE WITH COMPREHENDING TEXTUAL DETECTION INFORMATION

  • Training-free infusion improves some object-detection and OCR benchmarks but produces fluctuations elsewhere, consistent with detection text sometimes acting as noise.TFI-7B improves on some benchmarks, while TFI-13B declines; both improve on POPE, and both perform robustly on MME-Cognition.Source: §4.2 · p. 7

4.3 LESSON 2: RETRAINING HAS ADVERSE EFFECTS ON COMPREHENDING VIT FEATURES

  • Retraining improves several benchmarks, but without detection information its models lag LLaVA-1.5, indicating degraded use of image-encoder features.RBI models gain on benchmarks including POPE, MMBench, and Seed-Bench; the authors attribute the feature-comprehension issue as possibly due to redundant detection information.Source: §4.3 · pp. 7–8

4.4 LESSON 3: SUITABLE FINE-TUNING ACHIEVES GOOD TRADE-OFFS BETWEEN VIT FEATURES AND TEXTUAL DETECTION INFORMATION

  • Both FTBI-7B and FTBI-13B outperform the original models by up to 3.99% and training-free models by up to 6.71%; without detection information at inference, their performance is comparable to LLaVA-1.5.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 LESSON 4: SUITABLE FINE-TUNING HELPS MLLMS BETTER UNDERSTAND SPECIALLY FORMATTED DETECTION INFORMATION

  • With Grounding DINO replacing closed-set DINO, FTBI-7B still outperforms TFI-7B, although low detection accuracy introduces noise and lowers both models’ scores versus LLaVA-1.5-7B.Grounding DINO can detect targets specified by user-input, unlike DINO’s limitation to 80 common object categories.Source: §4.5 · pp. 9–10

5 OVERVIEW OF MORE EXPERIMENTS

The appendix tests whether FTBI’s gains come from detection infusion and examines model, detector, and benchmark generality. It also reports additional evaluations of the models.

APPENDIX

  • The appendix provides experimental details and parameter settings, and reports further tests intended to validate the universality of the results.Its overview also lists detector-accuracy, data-removal, visual-encoder, and broader-detection-scope experiments.Source: §5 · p. 10

Appendix C, Further Experiments and Analysis on the FTBI Model:

  • FTBI’s exceptional performance is primarily attributed to infused detection information rather than additional fine-tuning alone.The appendix also evaluates FTBI models using only OCR information or only object-detection information, finding that each can improve relevant tasks.Source: §5 · p. 10

6 CONCLUSION

The conclusion identifies fine-tuning with infused textual detection information as the most effective tested strategy and reports retained gains after detector replacement. Appendix experiments probe examples, implementation, model generality, and additional benchmarks.

6 CONCLUSION

  • Fine-tuning the original MLLM for an additional epoch while infusing textual detection information outperforms training-free and retraining strategies.After replacing a closed-set detector with an open-set detector, the fine-tuned model retains the training effect and performs better than the updated training-free model.Source: §6 · p. 10
  • The conclusion recommends using a small amount of general VQA data, potentially the MLLM’s own instruction-tuning data, for additional fine-tuning when infusing detection outputs as text.The authors state that this can yield better performance than models not subjected to training and support more comprehensive use of detection information.Source: §6 · pp. 10–11

Appendix A, More Demonstrative Examples:

  • The appendix includes examples where FTBI-13B answers correctly when LLaVA-1.5 does not, and examples illustrating construction of textual detection information.The examples cover object counting, detection, and text recognition.Source: Appendix A · p. 17

Appendix B, Implementation Details:

  • The appendix analyzes textual detection-information length to demonstrate compression efficacy, introduces the LLaVA-1.5 instruction-following dataset, and reports branch input resolutions, detection-output filtering thresholds, training hyperparameters, and model-training time consumption.Source: Appendix B · p. 17

Appendix C, Further Experiments and Analysis on the FTBI Model:

  • Fine-tuning without detection information indicates that FTBI’s exceptional performance is primarily due to the infused information, while OCR-only or object-detection-only infusion improves relevant tasks.The appendix also assesses inference efficiency.Source: Appendix C · p. 17

Appendix D, Model Architecture Rationale:

  • Experiments on Qwen-VL and LLaVA-NeXT support the versatility of findings from LLaVA-1.5, while tests also examine detector choice, detector accuracy, and broader detection scope.Experiments with LoRA modules in the visual encoder yield results consistent with the main conclusions.Source: Appendix D · p. 17

Appendix E, Model Performance and Additional Evaluation Benchmarks:

  • Additional evaluations cover MME-Perception, MME-Cognition, DocVQA, InfographicVQA, and VALSE; VALSE further confirms fine-tuning’s advantage over training-free infusion.The appendix also explains the motivations and modifications behind GQA∗.Source: Appendix E · p. 18

A.1 EXAMPLES ON WHICH LLAVA-1.5-13B FAILS WHILE THE MODEL INFUSED WITH TEXTUAL DETECTION INFORMATION SUCCEEDS.

The appendix presents examples of FTBI-13B successes, demonstrates the format of textual detection information, and documents its compression and implementation. It also describes the dataset and processing constraints used in the experiments.

A.1 EXAMPLES ON WHICH LLAVA-1.5-13B FAILS WHILE THE MODEL INFUSED WITH TEXTUAL DETECTION INFORMATION SUCCEEDS.

  • FTBI-13B gives accurate answers on showcased cases where LLaVA-1.5 answers incorrectly, spanning counting, object detection, and text recognition.Figure 5 illustrates these contrasting examples.Source: §A.1 · p. 19, Figure 5 · p. 19

A.2 EXAMPLES OF TEXTUAL DETECTION INFORMATION

  • Textual detection information combines instruction, detected object or class names, and position text, with examples generated using DINO and PaddleOCRv2.The examples include object counts and central coordinates, as well as OCR-recognized text and coordinates.Source: Figure 6 · p. 20, Figure 7 · p. 20, §A.2 · p. 20

B.1 LENGTH OF TEXTUAL DETECTION INFORMATION

  • 118.5 and 29.4 are the average object-detection and OCR information lengths, respectively, before excluding empty sequences; compression leaves only about 0.6% and 0.67% above 512.After excluding empty sequences, the reported averages are 125.1 for object detection and 97.5 for OCR; information longer than 1,024 is excluded.Source: §B.1 · p. 21, Table 5 · p. 21

B.2 LLAVA-1.5’S INSTRUCTION-FOLLOWING DATASET

  • The LLaVA-1.5 instruction-following dataset combines curated multitask data, but has about 467K image samples, including 116K for object detection and about 80K for text recognition.Its pretraining dataset is not used for FTBI because it predominantly emphasizes image-level captioning, whereas detection information focuses on fine-grained details.Source: §B.2 · p. 21

B.3 IMAGE RESOLUTION

  • CLIP-ViT preprocessing resizes and pads images to 336x336, while DINO and Grounding DINO accept arbitrary aspect ratios subject to shortest- and longest-side limits.PaddleOCRv2 processes images of any resolution autonomously.Source: §B.3 · p. 22

B.4 THRESHOLD SETTING FOR DETECTION MODELS

  • Valid detections require confidence above 0.3 for DINO, bounding-box confidence above 0.6 for PaddleOCR, and both thresholds above 0.35 and 0.25 for Grounding DINO.Source: §B.4 · p. 22

B.5 TRAINING HYPERPARAMETERS

The appendix specifies hyperparameter naming for RBI and FTBI and reports training-time and memory costs when detection models are deployed.

B.5 TRAINING HYPERPARAMETERS

  • Table 6 reports training hyperparameters derived from Vicuna and LLaVA-1.5; Pretrain-RBI denotes RBI vision-language alignment pretraining, Finetune-RBI its visual-instruction-tuning fine-tuning, and Finetune-FTBI FTBI fine-tuning.Source: §B.5 · p. 22, Table 6 · p. 22

B.6 TIME CONSUMPTION AND MEMORY REQUIREMENTS

  • On four NVIDIA A100 GPUs, fine-tuning LLaVA-1.5-7B increases from 16 to 22 hours and LLaVA-1.5-13B from 26 to 33 hours; detection deployment adds up to 4GB GPU memory per GPU.Pretraining time also rises from 6 to 11 hours for 7B and from 11 to 17 hours for 13B.Source: §B.6 · pp. 22–23

C.1 FINE-TUNING ON LLAVA-1.5 WITHOUT DETECTION INFORMATION

The appendix tests whether detection-informed fine-tuning gains depend on the detection input, the information type, inference cost, or model and detector choice. Across the reported comparisons, fine-tuning benefits persist across settings and models.

C.1 FINE-TUNING ON LLAVA-1.5 WITHOUT DETECTION INFORMATION

  • Fine-tuning without detection information performs on par with LLaVA-1.5 but falls below FTBI on almost all benchmarks.The authors attribute FTBI’s stronger performance more to supplemented textual detection information than to an extra fine-tuning epoch.Source: §C.1 · p. 23, Table 7 · p. 23

C.2 PERFORMANCE OF FTBI MODELS EXCLUSIVELY WITH OCR OR OBJECT DETECTION INFORMATION

  • Object-detection information boosts object-localization and object-hallucination benchmarks, while OCR information improves text-recognition benchmarks.The comparisons assess FTBI models infused exclusively with OCR or object-detection information.Source: §C.2 · p. 23, Table 8 · p. 23, Table 9 · p. 23

C.3 INFERENCE EFFICIENCY

  • DINO accounts for 38% of inference time with object detection alone, and PaddleOCRv2 accounts for 25% with OCR alone; text compression reduces inference time by 9% and 58%, respectively.The compression reductions are reported for object-detection and OCR information, respectively.Source: §C.3 · p. 24

D.1 HOW LLAVA-1.5 REPRESENTS OTHER MLLMS?

  • Qwen-VL-IT-FTBI improves across all 10 benchmarks over Qwen-VL-IT and Qwen-VL-IT-TFI, and remains comparable to Qwen-VL-IT without detection input at inference.The reported pattern mirrors the main-page FTBI findings.Source: §D.1 · p. 25, Table 10 · p. 25, Table 11 · p. 25
  • LLaVA-NeXT-8B-FTBI outperforms both the original LLaVA-NeXT-8B and its TFI version overall, with gains on object-detection and text-recognition benchmarks.Detection information improves LLaVA-NeXT results on those benchmark types.Source: §D.1 · p. 25, Table 12 · p. 25

D.2 HOW DINO AND PADDLEOCRV2 REPRESENT OTHER DETECION MODELS?

  • Replacing DINO with YOLOv8 yields similar FTBI performance improvements when the detectors have similar functionality and performance.The comparison covers FTBI-7B-DINO and FTBI-7B-YOLOv8.Source: §D.2 · p. 26, Table 13 · p. 26

D.3 EXPERIMENTS ON DETECTORS WITH VARYING PERFORMANCE

These experiments vary detector quality, fine-tuning data, and whether the visual encoder is unfrozen. Fine-tuning remains more robust to detector noise and retains its comparative advantage in the reported settings.

D.3 EXPERIMENTS ON DETECTORS WITH VARYING PERFORMANCE

  • Under training-free use, YOLOv5N degrades performance, while its gains on object-detection tasks are smaller than YOLOv11L’s.The authors describe this as limited adaptability to low-performance detectors.Source: §D.3 · p. 26, Table 14 · p. 26
  • Fine-tuning with either YOLOv5N or YOLOv11L yields comprehensive improvements over the original LLaVA-1.5-7B, indicating greater robustness to erroneous detector information than training-free use.The authors report that fine-tuning helps the MLLM distinguish noise from useful detection information.Source: Table 14 · p. 26, §D.3 · p. 27

D.4 MODEL FINE-TUNING WITHOUT THE USE OF DETECTION DATA

  • Without detection-related fine-tuning data, FTBI still significantly surpasses the original model and training-free model, and falls only slightly below FTBI trained with detection data.The dataset excludes coordinate-question and text-recognition samples, reducing its size from 665K to 450K.Source: §D.4 · p. 27, Table 15 · p. 27

D.5 MODEL FINE-TUNING WITH AN UNFROZEN VISUAL ENCODER

  • With the visual encoder unfrozen, the training-free, retraining, and fine-tuning strategies retain their reported ordering, and FTBI achieves the best result on 9 of 10 benchmarks.RBI outperforms training-free, while FTBI further surpasses RBI.Source: §D.5 · pp. 27–28, Table 16 · p. 27
  • Without detection input at inference, the unfrozen-encoder FTBI model remains comparable to LLaVA-1.5-7B, whereas RBI performs worse than the original model.The authors interpret this as FTBI better balancing visual-encoder outputs and detection information.Source: Table 17 · p. 27, §D.5 · p. 28

D.6 EXPERIMENTS ON A DETECTOR WITH BROADER DETECTION RANGES

The broader-detector experiments show that category coverage alone does not guarantee stronger FTBI performance, while the discussion and benchmark analyses describe the scope of the training comparison and its evaluation.

D.6 EXPERIMENTS ON A DETECTOR WITH BROADER DETECTION RANGES

  • With Co-DETR-LVIS, training-free performance is below the DINO-based TFI model on almost all benchmarks; fine-tuning improves over its training-free counterpart by mitigating detector noise.The authors attribute the training-free decline to redundant detected objects.Source: §D.6 · p. 28, Table 18 · p. 28
  • The Co-DETR-LVIS-based FTBI model performs worse than the DINO-based FTBI model across all 10 benchmarks, despite Co-DETR-LVIS covering more object categories.The detector can introduce redundant-object noise, which the authors suggest may lower scores.Source: §D.6 · p. 28, Table 18 · p. 28

D.7 FURTHER DISCUSSION ON RELATED WORKS

  • The discussion identifies a gap in comprehensive comparisons of training-free and adaptive training for deployed detection models, distinguishing this setup from special-token approaches.The paper describes deployed detectors as providing real-time detection details, whereas special-token approaches use manually supplied detection information as task guidance.Source: §D.7 · p. 29
  • The authors report that adaptive training outperforms training-free use, that fine-tuning with a small amount of high-quality VQA data improves performance, and that gains persist after detector replacement.They describe the study as a systematic investigation of adaptive training for detection-model assistance.Source: §D.7 · p. 29

E.1 MODIFICATION ON THE GQA BENCHMARK

  • GQA∗ filters evaluation to yes-or-no and choice questions, leaving 5,677 samples, approximately half the original evaluation set.The selection is intended to avoid correct answers being omitted because they do not exactly match reference wording.Source: §E.1 · pp. 29–30

E.2 MME BENCHMARK IN TABLE 1

  • Supplementary OCR information is associated with higher MME-Cognition scores, while FTBI scores on MME-Perception exceed those of TFI and RBI despite fluctuations.The authors link the cognition result to questions about text embedded in images.Source: §E.2 · p. 30, Table 19 · p. 30

E.3 PERFORMANCE ON DOCUMENTVQA BENCHMARKS

On DocumentVQA benchmarks, the reported comparisons favor adaptive training and show that fine-tuned models retain performance when detection information is absent at inference.

E.3 PERFORMANCE ON DOCUMENTVQA BENCHMARKS

  • Detection models, particularly OCR, produce a significant score improvement on DocVQA.These benchmarks evaluate visual question answering using text within document images.Source: §E.3 · p. 30, Table 20 · p. 31
  • On DocVQA and InfographicVQA, FTBI outperforms RBI, which in turn outperforms TFI.The authors report that adaptive training enables better use of input detection information.Source: §E.3 · p. 30, Table 20 · p. 31
  • Without detection information at inference, FTBI performs similarly to the original LLaVA-1.5, while RBI performs significantly worse than FTBI.The authors state that fine-tuning preserves comprehensive VQA capabilities by balancing image-encoder output and textual detection information.Source: §E.3 · p. 30, Table 21 · p. 31

E.4 PERFORMANCE ON THE VALSE BENCHMARK

The VALSE evaluation compares training-free and fine-tuned models on visual grounding across six linguistic phenomena. FTBI-7B outperforms LLaVA-1.5-7B in all reported aspects and TFI-7B on all but Existence.

E.4 PERFORMANCE ON THE VALSE BENCHMARK

  • VALSE tests visual-linguistic grounding across existence, plurality, counting, spatial relations, actions, and entity co-reference.Each instance pairs an image and caption with a foil, and models must distinguish correct captions from foils.Source: §E.4 · p. 31
  • The evaluation adapts VALSE scoring to yes-or-no judgments of whether the image matches a caption and its foil.It reports accuracy, precision, foil precision, and pairwise ranking accuracy.Source: §E.4 · pp. 31–32
  • FTBI-7B outperforms LLaVA-1.5-7B in all VALSE aspects and TFI-7B on every metric except Existence.The authors interpret the results as evidence that fine-tuning is more effective than training-free approaches for understanding and using textual detection information.Source: Table 22 · p. 32, §E.4 · p. 32

Related papers

Loading 2401.17981v3…