PaperlayerView exact revision on arXiv

Source-linked AI summary

Hijacking Context in Large Multi-modal Models

Joonhyun Jeong

arXiv:2312.07553v2cs.AIcs.CL

TL;DR

Off-the-shelf LMMs can let a small fraction of incoherent image-text context bias responses away from the intended context. The paper proposes preemptively removing hijacked contexts and investigates reforming them by having GPT-4 replace irrelevant sentences and DALLE-3 generate corresponding images. Reforming hijacked contexts still does not ensure coherent LMM responses, with generated text or images conjectured to retain or combine original and hijacked content.

  • Problem

    Off-the-shelf LMMs can let a small fraction of incoherent image-text context bias responses away from the intended context.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    The paper proposes preemptively removing hijacked contexts and investigates reforming them by having GPT-4 replace irrelevant sentences and DALLE-3 generate corresponding images.

    Source: §4 · p. 4, §6 · p. 5

  • Results

    Reforming hijacked contexts still does not ensure coherent LMM responses, with generated text or images conjectured to retain or combine original and hijacked content.

    Source: §1 · p. 2, §4 · p. 5

  • Takeaways & Limitations

    The paper identifies context hijacking as an LMM reliability problem and presents filtering or replacing irrelevant context as directions for addressing it.

    Source: §1 · p. 2, §6 · p. 5

    Results are qualitative examples on VIST, and the proposed scheme is fragile when irrelevant contexts are the majority.

    Source: §5 · p. 5

Abstract

from arXiv · show

Recently, Large Multi-modal Models (LMMs) have demonstrated their ability to understand the visual contents of images given the instructions regarding the images. Built upon the Large Language Models (LLMs), LMMs also inherit their abilities and characteristics such as in-context learning where a coherent sequence of images and texts are given as the input prompt. However, we identify a new limitation of off-the-shelf LMMs where a small fraction of incoherent images or text descriptions mislead LMMs to only generate biased output about the hijacked context, not the originally intended context. To address this, we propose a pre-filtering method that removes irrelevant contexts via GPT-4V, based on its robustness towards distribution shift within the contexts. We further investigate whether replacing the hijacked visual and textual contexts with the correlated ones via GPT-4V and text-to-image models can help yield coherent responses.

1 INTRODUCTION

The paper identifies context hijacking: a small fraction of incoherent image-text context can bias an off-the-shelf LMM away from the intended context. It proposes filtering or reforming irrelevant context with foundation models, but reforming does not ensure coherent responses.

1 INTRODUCTION

  • A single irrelevant image-text pair can make an LMM disregard the preceding visual story and respond only about the hijacked context.The paper presents this as a reliability concern because real-world contexts are not guaranteed to be free of noise or irrelevant information.Source: §1 · p. 1, Figure 1 · p. 2
  • Reforming the hijacked text and image still does not ensure coherent responses, with mixed original and hijacked content and image-style differences offered as possible explanations.The authors conjecture that generated captions and images may retain both contexts, while generated-image texture or style may differ from the original sequence.Source: §1 · p. 2
  • The paper identifies context hijacking as a new off-the-shelf LMM limitation and frames replacing hijacked context as an open research question.Its qualitative illustration contrasts an emergency story with a hijacking football-game context.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper proposes using foundation models to suppress hijacked contexts and investigates whether replacing them with correlated image-text content improves response coherence.GPT-4 identifies irrelevant sentences and supplies alternatives; a text-to-image model then generates a corresponding image.Source: §1 · pp. 1–2

2 RELATED WORKS

Related work covers language-model in-context learning, vision-enabled language models, and prior hijacking or adversarial-robustness studies. The paper distinguishes its focus on irrelevant image-text context in multimodal in-context learning.

Large language models

  • LLM-related work describes in-context learning alongside broad language-task versatility, while this paper focuses on LMM vulnerability to incoherent or irrelevant context.The cited examples include few-shot learning and long-term dialogues.Source: §2 · pp. 2–3

Large multi-modal models

  • Vision-enabled LLMs use pretrained visual encoders and linear layers to map visual tokens into a form the frozen language model can use.FROMAGe demonstrated answering a query image given consecutive image-text context pairs.Source: §2 · pp. 2–3

Hijacking context in language and vision tasks

  • Prior work reports that LLMs can be hijacked by irrelevant context, character perturbations, or adversarial suffixes, while vision studies examine adversarial pixel changes to LMM inputs.The paper describes its work as the first investigation of hijacking by irrelevant image-text sequences in an in-context learning scenario.Source: §2 · p. 3

3 CONTEXT-HIJACKING IN LARGE MULTI-MODAL MODELS

The section outlines how LLMs predict text and how LMMs incorporate image features, then describes FROMAGe's use of image-text context pairs. In this setting, irrelevant context can bias the response away from the intended visual story.

Preliminary: Large Language Models

  • An autoregressive LLM is trained to predict the next text token from the preceding token sequence.The section expresses this training objective as maximizing next-token log-likelihood.Source: §3 · p. 3

Preliminary: Large Multi-modal Models

  • An LMM maps pretrained visual-encoder features into text-embedding space and prepends them to text tokens so a frozen LLM can interpret images and answer instructions.The section describes the image-captioning objective as conditioning on an image and generating its caption.Source: §3 · p. 3

3 CONTEXT-HIJACKING IN LARGE MULTI-MODAL MODELS

  • FROMAGe uses concatenated image-text pairs as context for answering a query image, but irrelevant pairs can bias responses toward the irrelevant context.The section describes this as a loss of coherence with the preceding visual story.Source: §3 · p. 3

4 DISCUSSIONS

Hijacking context closer to the query image makes LMM responses more likely to favor the irrelevant context, while GPT-4V remained aligned with the original context in the tested example. Replacing hijacked contexts with GPT-4 and DALLE-3 did not ensure coherent responses, with mixed content and image-style differences offered as possible explanations.

Location of hijacked context

  • As hijacking context moves closer to the query image, LMM responses tend to become incoherent with earlier context and biased toward the hijacking context.In the example, the earlier context is “family” and the hijacking context is “football.”Source: §4 · p. 3

Robustness of GPT-4V

  • GPT-4V’s response to a hijacking context inserted between sequences 4 and 5 stayed with the original context rather than the hijacking context.The response described Auntie Marie supervising children on an indoor swing set.Source: §4 · p. 4
  • The example indicates GPT-4V is robust to distribution shift within the contexts and suggests distilling its robust knowledge as a future direction.Source: §4 · p. 4

Reforming contexts via large foundation models

  • GPT-4 reformulates irrelevant text and DALLE-3 generates a corresponding synthetic image using the underlying visual and textual context.Source: §4 · p. 4

Limited impact of reforming hijacked contexts

  • Reforming hijacked contexts still did not ensure coherent LMM responses; the authors conjecture that mixed family/football content and differences in image style or texture may confuse the model.Source: §4 · p. 5

5 LIMITATIONS

The evaluation is limited to qualitative examples on VIST, and the proposed scheme is fragile when irrelevant contexts make up most of the input.

5 LIMITATIONS

  • Results are qualitative examples on VIST because quantitative evaluation is labor-demanding, and generated text requires human checks for image alignment and story coherence.Source: §5 · p. 5
  • The straightforward scheme is fragile when irrelevant contexts become the majority, a setting in which even GPT-4 may be confused.Source: §5 · p. 5
  • Rigorous quantitative evaluation across various LMMs and more secure filtering or replacement of numerous hijacked contexts are left for future work.Source: §5 · p. 5

6 CONCLUSION

The paper identifies context hijacking in existing LMMs and introduces preemptive removal and reforming of hijacked contexts as mitigation approaches.

6 CONCLUSION

  • Irrelevant visual and textual contexts can lead existing LMMs to produce biased, incoherent responses that disregard the original context.Source: §6 · p. 5
  • The paper introduces a simple method to preemptively remove hijacked contexts and further reform them into coherent ones.Source: §6 · p. 5
  • The work promotes future research on generating more coherent visual and textual information.Source: §6 · p. 5

A APPENDIX

Figure 4 presents a process for reforming visual-story image-text pairs: GPT-4 replaces irrelevant content, then DALLE-3 generates an image for the reformed text in context.

A APPENDIX

  • For VIST visual-story pairs (Ferraro et al., 2016), GPT-4 replaces irrelevant images or descriptions with coherent alternatives, then DALLE-3 generates an image for the revised text under the underlying context.Source: Figure 4 · p. 8

Related papers

Loading 2312.07553v2…