PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model

Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, Kavya Srinet, Babak Damavandi, Anuj Kumar

arXiv:2309.16058v1cs.LGcs.CLcs.CV

TL;DR

AnyMAL addresses multimodal LLMs limited to text and one other modality by aligning diverse inputs to an LLM’s text space and tuning it with multimodal instructions. It reports new state-of-the-art performance across diverse tasks, while noting that image grounding remains imperfect.

  • Problem

    Prior multimodal LLM research focused on text paired with one other modality or proprietary language models, while AnyMAL targets diverse modality inputs.

    Source: §1 · p. 1

  • Method

    AnyMAL aligns modality-specific encoders to an LLM’s text embedding space and fine-tunes with manually collected and synthetic multimodal instruction data.

    Source: §3.1 · p. 3, §3.2 · p. 5

  • Results

    AnyMAL reports new state-of-the-art performance across diverse tasks and modalities, including +14.5% CIDEr on AudioCaps versus models available in the literature.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The approach uses scalable training to bring LLaMA-2 language reasoning into multimodal settings and supports interleaved multimodal prompting.

    Source: §1 · p. 1, §6 · p. 14

    Image grounding can be weak during generation, allowing language-model biases to produce responses inaccurate for the image; addressing this may require architectural changes or unfreezing LLM parameters at higher computational cost.

    Source: §7 · p. 14

Abstract

from arXiv · show

We present Any-Modality Augmented Language Model (AnyMAL), a unified model that reasons over diverse input modality signals (i.e. text, image, video, audio, IMU motion sensor), and generates textual responses. AnyMAL inherits the powerful text-based reasoning abilities of the state-of-the-art LLMs including LLaMA-2 (70B), and converts modality-specific signals to the joint textual space through a pre-trained aligner module. To further strengthen the multimodal LLM's capabilities, we fine-tune the model with a multimodal instruction set manually collected to cover diverse topics and tasks beyond simple QAs. We conduct comprehensive empirical analysis comprising both human and automatic evaluations, and demonstrate state-of-the-art performance on various multimodal tasks.

1 Introduction

AnyMAL extends LLM-based multimodal work beyond text-plus-one-modality systems and proprietary models by aligning diverse modalities to an LLM. It combines large-scale modality alignment with multimodal instruction tuning and reports strong performance across tasks.

  • Prior multimodal LLM work focused on text plus one other modality [3] or proprietary language models; AnyMAL aligns image, video, audio, and IMU data to an LLM text space.Source: §1 · p. 1
  • AnyMAL aligns diverse modality datasets to LLaMA-2-70B-chat, supporting interleaved multimodal in-context prompting across 200M images, 2.2M audio samples, 500K IMU time-series, and 28M videos.Source: §1 · p. 1
  • Its manually collected multimodal instruction set covers image, video, and audio tasks beyond simple QA and also serves as a benchmark for complex reasoning.Source: §1 · p. 1
  • AnyMAL reports new SOTA with +7.0% relative accuracy, +8.4% CIDEr on zero-shot COCO image captioning, and +14.5% CIDEr on AudioCaps versus literature models.Source: §1 · pp. 1–2
  • Figure 1 illustrates free-form responses to vision, audio, and motion inputs, including joint reasoning over interleaved image and IMU signals.Source: Figure 1 · p. 2

2 Related Work

Prior multimodal LLM work includes vision-language systems that combine visual and linguistic inputs, often using pretrained LLMs and task-specific fine-tuning. AnyMAL extends this line toward diverse modalities, manually collected instruction tuning, and 70B-scale pretraining.

  • Vision-language models have been applied to image captioning and visual question answering, with recent approaches harnessing pretrained LLMs and task-specific fine-tuning.Source: §2 · p. 2
  • Prior systems typically integrate vision and language, whereas AnyMAL extends multimodal inputs beyond vision and introduces manually collected multimodal instruction-tuning data.Source: §2 · p. 2
  • Figure 2 depicts modality-specific encoders projecting into the LLM embedding space, followed by instruction tuning that links queries and system instructions to multimodal contexts.Source: Figure 2 · p. 3

3 Methods

AnyMAL first aligns modality encoders with an LLM’s text embedding space, then fine-tunes with multimodal instructions. The methods pair frozen-LLM alignment and quantized training with manually collected and synthetic instruction data.

3.1 Pre-training

  • Each modality adapter projects input signals into the LLM text-token embedding space, using 64–256 tokens per input while the underlying LLM remains frozen.Text-aligned encoders and modality-specific projection modules support the alignment objective.Source: §3.1 · p. 3
  • Four- and eight-bit quantization lets the frozen 70B LLM train with modality tokenizers on one 80GB GPU at batch size 4, using half the GPU resources of FSDP at the same throughput.Training and validation loss were higher than with FSDP, but the authors report no impact on inference generation quality.Source: §3.1 · p. 3

3.2 Fine-tuning with Multimodal Instruction Datasets

  • MM-IT contains manually collected image, video, and audio instructions across diverse domains, grounded so queries require the accompanying modality context; it serves as both training data and an evaluation benchmark.The table presents examples spanning creative writing and open-ended reasoning.Source: Table 1 · p. 4
  • Fine-tuning concatenates instructions with modality tokens and uses 60K manually annotated examples plus 150K synthetic image instruction-response pairs generated with LLaMA-2-70B.The authors report that results improve with their fewer, balanced, higher-quality manual examples.Source: §3.2 · p. 5

4 Experiments

Experiments assess zero-shot captioning and multimodal reasoning and instruction-following, with separate reporting for pretrained and instruction-tuned models because MM-IT includes images from public benchmarks. Results span image, audio, video, and IMU tasks, alongside human evaluations and interleaved-modality examples.

4.2 Quantitative Analysis

  • AnyMAL variants significantly outperform baselines on COCO and MM-IT-Cap image captioning, with no significant performance gap between the 13B and 70B variants.The authors attribute captioning more to data scale and alignment methods than to underlying LLM capability.Source: §4.2 · p. 5, Table 2 · p. 6
  • 41.1% win against human-annotated responses, versus 34.4% for LLaVA and 27.0% for MiniGPT4; full-set instruction tuning had the highest preferential-win rate.The pairwise comparisons use 1K MM-IT test samples.Source: Figure 3 · p. 6, §4.2 · p. 7
  • AnyMAL shows the strongest performance across multiple zero-shot image QA tasks, while video QA results are competitive with baseline vision-language models.ViT-G scores higher on most vision datasets, but ViT-L’s 336x336 resolution outperforms ViT-G’s 224x224 on TextVQA; DinoV2 performs worse than text-guided vision encoders, and the 70B base model is strongest among base LLMs.Source: Table 4 · p. 7, Table 6 · p. 8, §4.2 · pp. 8–9
  • 12.2% relative improvement over LLaVA in response accuracy and relevance came from the model tuned with both manually collected and synthetic instruction data.Instruction-tuned models also showed a decline in detailed object recognition and descriptions, which the authors attribute to promoting concise responses.Source: Table 3 · p. 7, §4.2 · p. 8
  • Increasing resampler depth from 2 to 6 layers significantly reduces training loss, whereas increasing batch size or visual tokens yields minimal improvement.The 70B variants also show lower training loss overall than the 13B variants.Source: Figure 4 · p. 9, §4.2 · p. 10
  • AudioCaps: AnyMAL exceeds prior models by +10.9pp CIDEr and +5.8pp SPICE, with the 70B model notably stronger than the 7B and 13B variants.The authors interpret the larger model’s performance as showing the importance of the reasoning module for audio captioning.Source: Table 5 · p. 8, §4.2 · p. 10
  • 52.5 CIDEr and 23.2 ROUGE-L on held-out Ego4D show the feasibility of generating text descriptions from IMU motion signals, a task not previously reported.Source: §4.2 · p. 10

4.3 Qualitative Analysis

  • Qualitative examples show visual recognition and secondary reasoning, and demonstrate combining image and IMU inputs to use motion context in a response.The authors note that AnyMAL’s outputs are not consistently more accurate than baseline responses; Table 11 presents interleaved-modality examples.Source: §4.3 · pp. 10, 13, §4.2 · p. 10, Table 11 · p. 13

5 Safety

AnyMAL uses layered safeguards across inputs, generated text, and image-text associations. Training-data filtering and inherited LLaMA-2 safeguards supplement inference-time checks.

Inference-time checks

  • Input images and text prompts flagged for integrity violations are rejected using RegNetY-based image and RoBERTa-based text classifiers.Image checks cover graphic material, violence, hate symbols, bullying, and harassment; text checks cover violence, harassment, and hate speech.Source: §5 · p. 13
  • Generated text is checked with the same text classifier, sentence by sentence for streaming use cases.The classifier detects integrity problems in model outputs.Source: §5 · p. 13
  • A multimodal classifier checks image-text associations that may be problematic even when the image and text are individually harmless.This check addresses the uncommon case where the combination creates an issue.Source: §5 · p. 13

Training-time safeguards

  • Pretraining datasets are filtered for harmful language or images, while AnyMAL carries over the base LLaMA-2 language-generation safeguards.The cited LLaMA-2 safeguards include negative-example fine-tuning and reinforcement learning with human feedback.Source: §5 · p. 13

6 Conclusions

AnyMAL combines multimodal perception with LLaMA-2 reasoning, and the paper introduces MM-IT and reports empirical analysis of multimodal reasoning-model design choices.

6 Conclusions

  • AnyMAL is presented as a large-scale multimodal LLM built with open-sourced resources and scalable solutions for multiple modalities.The paper describes interaction through combined visual, auditory, and motion cues, with questions that presume shared understanding between user and agent.Source: §6 · p. 14
  • MM-IT is introduced as a collection of high-quality manual annotations for multimodal instruction data.Source: §6 · p. 14
  • The paper's empirical analysis reports insights into efficient and scalable multimodal reasoning-model construction across LLMs and modeling choices.Source: §6 · p. 14

7 Limitations

The paper identifies limitations in image grounding, visual-concept coverage, and the scope of modalities evaluated.

7 Limitations

  • During generation, AnyMAL can prioritize generated text over the input image, producing LLM-biased output that may be inaccurate against the image.The authors say addressing this may require architectural adjustments or unfreezing LLM parameters, at higher computational cost.Source: §7 · p. 14
  • Visual concept and entity understanding remains constrained by the quantity of paired image-text data used in training.The authors mention external knowledge retrieval as a potential way to alleviate this limitation.Source: §7 · p. 14
  • The work adapts LLMs for image, video, audio, and IMU signals; effectiveness on other modalities remains unsubstantiated.The authors say broader application could be possible where paired datasets exist.Source: §7 · p. 14

A Human Evaluation

Human feedback is collected using rubrics for image recognition, factual response accuracy, integrity, and pairwise comparison of answers.

Human feedback rubrics

  • Human ratings assess image recognition, factual response accuracy, and integrity, with rubric descriptions specifying what scores indicate.Recognition scores distinguish complete misinterpretation (0), partial recognition (1), and correct interpretation (2); the other listed rubrics describe scores 0 and 2.Source: A Human Evaluation · p. 19

Pairwise comparison

  • Pairwise comparison asks evaluators to choose the response that better answers the question and is more contextually appropriate and accurate.Source: A Human Evaluation · p. 19

Future use of feedback

B Additional Notes on Experiments

Appendix B describes task-specific prompt formats and shows examples of modality-interleaved inference. Prompts adapt to the task and modality, while Figure 5 illustrates interleaved inputs.

B Additional Notes on Experiments

B.1 Multimodal Prompts

  • Image prompts tailor outputs by task: one-word VQA, answer-or-unanswerable VizWiz, yes/no meme classification, generic JSON captions, and single-letter ScienceQA.Source: §B.1 · p. 19
  • Video prompts request direct answers with one selected option, IMU prompts request concise motion descriptions, and the audio prompt asks for a sound description.Source: §B.1 · p. 20

B.2 Multimodal Inputs

B.3 Hyperparameters

Appendix B reports pretraining hyperparameters, describes the MM-IT finetuning setup, and lists implementation and hardware details.

B.3 Hyperparameters

  • Table 12 reports the hyperparameters used for AnyMAL pretraining.Source: §B.3 · p. 20, Table 12 · p. 20
  • Finetuning updates the resampler and LoRA adapters on MM-IT for 3k steps, using rank 64, α = 16, batch size 128, and learning rate 1 × 10−5.Source: §B.3 · p. 20

B.4 Code Base & Hardware

  • The implementation uses PyTorch and extended transformer-model code, with training on varying numbers of Nvidia A100 3 GPUs.Source: §B.4 · p. 21

B.5 More Qualitative Examples

Tables 13 and 14 provide qualitative comparisons between AnyMAL and several baselines. The examples include fluent baseline responses, but their accuracy is not consistent across visual understanding and secondary reasoning.

B.5 More Qualitative Examples

  • Some baseline responses are reasonable and fluent, but accuracy is not consistently ensured.Source: §B.5 · p. 21
  • The reported accuracy inconsistency concerns both visual understanding, such as identifying image objects, and secondary reasoning, such as checking image integrity-policy violations.Source: §B.5 · p. 21
  • Tables 13 and 14 present additional qualitative output comparisons between AnyMAL and BLIP-2, InstructBLIP, OpenFlamingo, Mini-GPT4, and LLaVA.Source: Table 13 · p. 22, Table 14 · p. 23

Related papers

Loading 2309.16058v1…