PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training

Michal Chudoba, Sergey Alyaev, Petra Galuscakova, Tomasz Wiktorski

arXiv:2606.11854v1cs.LGcs.AIcs.CL

TL;DR

Existing PEFT methods can conflict with the rigid computational graphs of high-throughput LLM serving. ART instead adapts a frozen multimodal model by optimizing an input image, matching or beating LoRA on mathematics and tool-use benchmarks.

  • Problem

    Existing PEFT methods create engineering friction in production serving because high-throughput engines rely on optimized kernels and rigid CUDA graphs.

  • Method

    ART freezes the multimodal model and optimizes a single task-specific input image routed through its standard visual pathway.

  • Results

    ART matches or beats weight-space LoRA on standard mathematics and tool-use benchmarks.

  • Takeaways & Limitations

    ART is particularly attractive for locally served small multimodal language models because it is more efficient than LoRA during training and inference.

  • Takeaways & Limitations

    Experiments use only the Qwen3.5 0.8B and 2B models, so generalization to other vision-language architectures remains future work.

Abstract

from arXiv · show

There are two main Parameter-Efficient Fine-Tuning (PEFT) techniques for Large Language Models (LLMs). While Low-Rank Adaptation (LoRA) introduces additional weights between the LLM layers, Soft Prompting introduces additional fine-tuning-specific raw tokens to an LLM input. However, both require modification to the computational graphs of precompiled, preoptimized LLMs. As a result, neither is fully supported in high-throughput engines like vLLM. We propose fine-tuning with ART (Art-based Reinforcement Training). The method injects information into a frozen Multimodal Large Language Model (MLLM) by optimizing only its raw visual input, thus enabling the soft-token approach on pre-compiled computational graphs. It relies on backpropagation of gradients back into a plain pixel array and thus supports any fine-tuning objective. Moreover, the optimized visual input can be stylized as task-relevant computational artworks. The approach's effectiveness is confirmed for different sizes of a popular open Qwen architecture and for several textual benchmarks. Specifically, ART reaches accuracy competitive with LoRA across mathematics and structured-tool-use benchmarks.

h ART-BASED REINFORCEMENT TRAINING

The section identifies ART as concerning multimodal language models, visual prompting, parameter-efficient fine-tuning, and reinforcement learning. It also situates the method within high-throughput serving, generative and computational art, and steganography.

  • ART is associated with multimodal large language models, visual prompting, and parameter-efficient fine-tuning.
  • The method involves reinforcement learning, including Group Relative Policy Optimization.
  • The section connects ART to high-throughput serving, generative art, computational art, and image steganography for AI.

1 Introduction

ART adapts frozen multimodal LLMs by optimizing a single task-specific visual input, avoiding architectural changes required by LoRA and soft prompting. Across mathematics and structured tool-use benchmarks, ART matches or beats LoRA while producing computational artworks that encode fine-tuning information.

  • Introduction: ART supports end-task reward optimization for text-based tasks including mathematics, scientific question answering, code execution, and structured tool use.Experiments compare ART with text baselines, random-image controls, and LoRA weight tuning across established benchmarks.
  • Introduction: LoRA and soft prompting require modifications to model weights or token inputs that create engineering friction in optimized high-throughput serving engines.Soft prompting also under-performs LoRA on many downstream tasks while requiring continuous-embedding injection.
  • Introduction: ART adapts a frozen multimodal model by optimizing only a single task-specific input image through the standard vision pathway.Fine-tuned prompts remain plain multimodal requests for serving infrastructure.
  • Introduction: ART matches or beats weight-space LoRA on standard mathematics and tool-use benchmarks, while also identifying tasks where it falls behind.The reported benchmark examples include GSM8K, GPQA, and ToolMind.
  • Introduction: ART images simultaneously encode model fine-tuning, with lossless PNG file-size growth used as a proxy for stored information.The optimized artifacts are presented as task-relevant computational artworks and a form of AI steganography.

2 Related Work

ART builds on visual prompt engineering, adversarial reprogramming, parameter-efficient fine-tuning, and reinforcement learning. Its related work motivates pixel-space adaptation, standard-input compatibility, and group-relative policy optimization.

  • Visual Prompt Tuning, Reprogramming, and Adversarial Steering: Adversarial Reprogramming repurposed ImageNet classifiers for out-of-domain tasks by adding a single optimized pixel perturbation.Exploring Visual Prompts extended pixel-space learning to frozen CLIP vision encoders, while Visual Prompt Tuning introduced learnable intermediate-layer tokens in Vision Transformers.
  • Visual Prompt Tuning, Reprogramming, and Adversarial Steering: Visual-channel studies found that visual inputs exert an outsized influence on multimodal language-model text generation.
  • Continuous Soft Prompting: Soft Prompting and Prefix-Tuning optimize virtual embeddings prepended to frozen LLM inputs but require custom token-pipeline engineering that breaks high-performance engines such as vLLM.ART instead uses standard multimodal inputs to circumvent these inefficiencies.
  • Group Relative Policy Optimization: GRPO removes PPO’s critic model by computing advantages relative to groups of sample rollouts, and ART combines this optimization with DAPO.GRPO was introduced by DeepSeekMath and popularized by DeepSeek-R1.

3 Art-based Reinforcement Training (ART)

ART fine-tunes a frozen multimodal LLM by optimizing a differentiable image in pixel space rather than model weights. Its two-pass rollout and backward procedure produces a portable PNG artifact that standard multimodal serving infrastructure can process without model or graph modifications.

  • Core formulation: ART freezes the multimodal LLM and optimizes the external input image X_pixel as trainable parameters.The method updates pixel-space inputs instead of the model weights θ.
  • Core formulation: The ViT and cross-modal projection form a frozen, differentiable mapping from raw pixels to the model’s embedding space, enabling output steering through gradient descent.The image-processing pipeline remains continuous and differentiable with respect to input pixels.
  • Optimization procedure: The input tensor supports any differentiable training objective, including supervised fine-tuning, RLHF, policy gradients, SFT, DPO, and GRPO/DAPO.The objective is agnostic to the training algorithm because gradients are propagated to the plain input tensor.
  • Optimization procedure: ART uses two passes: quantized-image rollouts generate and score multiple completions, then a frozen-model backward pass updates the raw pixel tensor.The rollout uses vLLM, while the backward pass substitutes the continuous normalized tensor into a frozen model and applies a clipped objective.
  • Practical properties: The learned artifact is a native pixel-space input, serialized as a standard 8-bit RGB PNG and served through the unmodified multimodal pipeline.This avoids custom weight managers, specialized kernels, adapter-loading overhead, and serving-time CUDA-graph rebuilds.

4 Benchmark Datasets

ART is evaluated on three domains—mathematics, grade-level reasoning, and structured API tool calling—using benchmark-specific scoring procedures. The benchmarks assess numeric accuracy, expert science multiple-choice reasoning, and structured XML function-call generation.

  • Evaluation domains: ART is evaluated across mathematics, grade-level reasoning, and structured API tool calling.These domains are intended to test different cognitive capabilities.
  • Mathematics: GSM8K measures exact-match accuracy on multi-step grade-school math word problems with a binary reward R ∈{0, 1}.The scorer extracts the final numeric value after the #### delimiter.
  • Grade-level reasoning: GPQA evaluates exceptionally difficult expert-authored science questions as multiple-choice reasoning with binary scoring.The standard gpqa_extended split is divided evenly between image optimization training and held-out evaluation.
  • Structured tool use: ToolMind evaluates structured XML API calls by strictly matching the generated function name and required arguments against ground truth.Correct calls receive 1, while incorrect calls receive 0; training additionally uses a lightly shaped reward.

5 Experimental model setups

The experiments isolate ART’s visual-prompting behavior with a dense baseline grid on Qwen3.5-0.8B and Qwen3.5-2B using an NVIDIA A100. Setups compare text-only, random visual and textual prefixes, fixed semantic images, LoRA, and optimized-image ART under controlled training conditions.

  • Experimental scope: The study evaluates Qwen3.5-0.8B and Qwen3.5-2B with a dense grid of baselines on an NVIDIA A100 GPU.The grid is designed to isolate ART’s unique visual-prompting behavior.
  • Baselines: Baseline uses standard text-only generation without an image prefix.
  • Baselines: Random Image prepends a fresh unique random 256 × 256 RGB image to each query, controlling for continuous visual tokens.
  • Baselines: Random String prepends 64 random whitespace-separated text tokens, providing a text-space token-overhead baseline for random images.
  • Baselines: Fixed Initial Image uses static recognizable seed imagery: math.png for GSM8K, brain.png for GPQA, and tool.png for ToolMind.
  • Trainable setups: LoRA updates decoder projections with rank r = 16 and α = 32, while ART optimizes raw pixels initialized from the corresponding seed image; both use DAPO loss and run 100 steps with group size 8 and effective batch size 32.LoRA uses AdamW with LR = 1 × 10−5, whereas ART uses AdamW with LR = 0.1, a constant-with-warmup schedule, and 5 warmup steps; the vision encoder remains frozen for LoRA.

6 Results and Discussion

ART improves performance on procedural benchmarks by optimizing raw visual inputs, outperforming LoRA in the smallest model while avoiding computational-graph modifications. Its artifacts remain portable 8-bit images, though benefits vary by task and ART increases training efficiency.

  • Baseline visual-input effects: Random 256 × 256 images raise 0.8B accuracy from 39.65% to 54.59% on GSM8K and from 36.65% to 63.10% on ToolMind.The corresponding absolute gains are +14.94% and +26.45%.
  • Baseline visual-input effects: 64-token random text prefixes instead reduce accuracy from 39.65% to 25.25% on GSM8K and from 36.65% to 24.45% on ToolMind.This contrasts visual inputs with text prefixes of the same post-spatial-merge length.
  • Baseline visual-input effects: The results attribute visual-input benefits to ViT decoder activation and improved attention routing rather than generic continuous prefix padding.The random-image benefit contracts to +3.71% on GSM8K and +0.60% on ToolMind for the larger decoder.
  • ART and PEFT comparison: ART reaches 58.53% on 0.8B GSM8K and 73.80% on 0.8B ToolMind, improving over baseline by +18.88% and +37.15%, respectively.These gains are also +3.94% and +10.70% over the random-image condition and fall outside its 95% confidence intervals.
  • ART and PEFT comparison: LoRA reaches only 49.51% on 0.8B GSM8K under identical reward and rollout conditions, below both ART and the 54.59% random-image control.ART provides adaptation without low-level architecture hooks.
  • Limitations: ART degrades GPQA performance across both models, including a 0.8B decrease from 23.44% to 20.15%, although the difference is not statistically decisive.The held-out set contains n=273 examples and has wide confidence intervals.
  • Artifacts and efficiency: Optimized artifacts develop visible high-frequency structure and can grow from 8.5 KB to 98.0 KB, a +1047% increase, while deployment retains only 8-bit PNG values.Accuracy is evaluated after reloading the quantized PNG through the unmodified vision pathway.
  • Artifacts and efficiency: ART trains roughly twice as fast as LoRA on GSM8K and more than three times as fast on ToolMind on a single NVIDIA A100.Inference-time comparisons are less conclusive because reasoning effort and output length may vary.

7 Conclusions

ART adapts frozen multimodal LLMs for text tasks by optimizing runtime-prepended input images without modifying model weights or runtime pipelines. It matches or exceeds LoRA on mathematics and tool-use benchmarks while producing computational-art artifacts that store task information in images.

  • Conclusions: ART adapts frozen multimodal LLMs by optimizing input images prepended to the visual input at runtime.This non-intrusive approach improves task-specific performance without changing the model itself.
  • Conclusions: ART matches or beats LoRA on mathematics and tool-use benchmarks while avoiding model-weight and runtime-pipeline modifications.The passages identify this efficiency as especially attractive for locally served small MLLMs.
  • Conclusions: ART stores fine-tuning information as high-frequency image structures while preserving overall visual structure, creating computational-art artifacts and AI steganography.Growth in PNG file size indicates that task information is stored in the optimized images.

Limitations and Future Work

The study is limited to the Qwen3.5 0.8B and 2B families, leaving ART’s behavior across other vision-language backbones unresolved. Future work will compare ART with continuous Soft Prompting, test cross-model visual transferability, and explore visual feature fusion as an alternative to LoRA weight merging.

  • Limitations: Experiments cover only the Qwen3.5 architecture family, specifically the 0.8B and 2B models.ART’s architecture-agnostic mechanism may nevertheless show different random-image boosts, pixel-space optimization effectiveness, and reasoning-task boundaries on LLaVA, InternVL, or proprietary models.
  • Future Work: Future work will benchmark ART directly against continuous Soft Prompting under matched capacity.This comparison is intended to contrast visual and embedding-space prefixes.
  • Future Work: Future work will test cross-model visual transferability by applying images optimized on Qwen-0.8B directly to Qwen-2B.The goal is to test projection invariance and whether the artifact functions as a portable parameter tensor.
  • Future Work: Future work will explore visual feature fusion as an alternative to LoRAs weight merging.This direction examines another way to use the optimized visual artifact.

8 Appendix

The appendix establishes the visual-prefix token budget for a 256 × 256 optimized image and matches it with the random-string control. After patchification and 2 × 2 spatial merging, the image yields exactly 64 continuous visual prefix tokens.

  • Visual-prefix capacity: A 256 × 256 image is patchified into a 16 × 16 = 256 grid using 16px patches, then processed through ViT and a 2 × 2 spatial merge layer.These operations determine the visual prefix capacity used for token-budget matching.
  • Budget matching: Exactly 64 continuous visual prefix tokens result, matching the random-string control’s 64 prepended text tokens.A 64-embedding soft-prompting baseline is left for future work.
Loading 2606.11854v1…