PaperlayerView exact revision on arXiv

Source-linked AI summary

Medical Image Alignment Assessment as a Test of Generalist Visual Reasoning in Frontier Multimodal Models

Ross Callaghan, Niannu Gao, Hojjat Azadbakht, Hui Zhang

arXiv:2610.06896v1cs.CV

TL;DR

The paper asks whether frontier MLLMs can make visual medical image-alignment judgments without task-specific optimisation, and evaluates them alongside local task-specific models. Recent frontier models perform strongly across most tested settings, while local models show a trade-off between tuned-task performance and transfer.

  • Problem

    Medical image alignment is important to downstream analysis, but manual visual assessment requires trained operators and is impractical to scale.

    Source: §1 · p. 2

  • Method

    The study evaluates frontier MLLMs on MRI-PET and T1w-FLAIR alignment tasks, varying prompting and presentation, and compares them with a locally fine-tuned MLLM and task-specific CNN.

    Source: §1 · p. 3, §2.1 · p. 3, §2.3 · p. 6

  • Results

    Recent frontier models performed strongly in almost all settings, with GPT-6 Astra best in 7 of 8; local models can match or exceed frontier models on their tuned task but transfer less effectively.

    Source: §4 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    Medical image alignment is a useful test bed for generalist visual reasoning, and the results support MLLMs as potential visual assessors in medical-imaging pipelines.

    Source: §5 · p. 9

    The study covers only two exemplar alignment tasks, and its datasets were kept reasonably small for practicality and cost reasons.

    Source: §4 · p. 9

Abstract

from arXiv · show

Frontier multimodal large language models (MLLMs) are increasingly positioned as general purpose visual reasoners as part of the quest for artificial general intelligence. A key test of this generality is whether they can perform novel visual judgments that humans can make reliably from visual evidence and task instructions, without task-specific parameter optimisation. We investigate this question through the task of medical image alignment assessment, where the goal is to establish whether there is anatomical correspondence between two images. Human visual assessment of image alignment is still the gold standard and most common approach; however, it requires trained operators and is impractical to scale for large datasets. We evaluate recent generations of MLLMs on two exemplar medical image alignment tasks, varying both prompting strategies and image-presentation methods. We compare against a locally fine-tuned MLLM and a task-specific CNN to examine the trade-off between frontier general purpose models and smaller models that require specific task optimisation but can be used locally. We show that are reaching an inflection point, where frontier MLLMs can now perform effective visual assessment of medical image alignment. Models released only a few months ago generalise poorly and, in some settings, perform barely above chance, whereas GPT-6 achieves over 85% across almost all scenarios tested. Fine-tuned local models can match or exceed frontier-model performance on the tasks on which they are trained, but transfer substantially less effectively to unseen settings. These findings identify medical image alignment as a useful test bed for generalist visual reasoning and suggest that frontier multimodal models are beginning to acquire capabilities that could support a common quality-control mechanism across heterogeneous medical-imaging pipelines.

1 INTRODUCTION

The paper asks whether frontier MLLMs can perform novel visual judgments from evidence and instructions without task-specific optimisation. It studies medical image alignment assessment, a task important to image-analysis pipelines but difficult and time-consuming to assess manually.

1 INTRODUCTION

  • Alignment assessment is critical because errors can propagate to downstream image analysis, including multimodal fusion and longitudinal analysis.Source: §1 · p. 2
  • Manual visual inspection is the gold standard, but it requires trained assessors and is impractical for large datasets.Source: §1 · p. 2
  • Existing automated approaches are often tailored to particular modalities, anatomy, or registration tasks and do not readily generalise to new settings.Source: §1 · p. 2
  • MRI-PET alignment is challenging for automated metrics because corresponding voxel intensities lack a simple relationship, yet human readers can use anatomical cues to detect misalignment.Source: §1 · p. 2
  • The study tests frontier MLLMs on two exemplar medical-imaging settings and compares them with a locally fine-tuned MLLM and task-specific CNN.Source: §1 · p. 3

2 EXPERIMENTAL DESIGN

The experimental design tests frontier MLLMs on two medical image alignment settings, then compares them with locally trained task-specific models. The study varies model, prompting, and image-presentation choices.

2 EXPERIMENTAL DESIGN

  • Figure 2 illustrates the two modality pairs and how image-presentation techniques affect the visibility of different misalignments.Source: Figure 2 · p. 3
  • Experiment 1 tests MLLMs as medical image alignment assessors, and Experiment 2 compares frontier models with locally trained task-specific models.Source: §2 · p. 3
  • The first setting uses challenging MRI-PET alignment, where the images come from the same subject but convey different information.Source: §2 · p. 3
  • The second setting tests transfer to T1w-FLAIR alignment, described as theoretically simpler because both images show aspects of brain structure.Source: §2 · p. 3

2.1 EXPERIMENT 1A: MRI-PET ALIGNMENT

Experiment 1a benchmarks MLLMs on MRI-PET misalignment detection, varying model choice, prompting, and image presentation. It uses manually verified aligned pairs, introduces synthetic rigid misalignments, and evaluates prediction accuracy.

2.1 EXPERIMENT 1A: MRI-PET ALIGNMENT

  • The experiment benchmarks MLLMs on MRI-PET misalignment detection, a task described as easy for human assessors but difficult to implement robustly with traditional metrics.Source: §2.1 · p. 3
  • The study varies model choice, zero-shot versus few-shot prompting, and alpha-blend versus contour image presentation.Source: §2.1 · p. 4

2.1.1 DATA

  • The main test set contains 52 same-session MRI-PET scans from 29 ADNI subjects using 18F-AV-45 PET and T1w MRI.Source: §2.1.1 · p. 4

2.1.2 GROUND TRUTH ALIGNMENT

  • The study selects an alignment pipeline from three rigid-registration algorithms using mutual information and brain-mask Dice, then manually inspects each output.Source: §2.1.2 · pp. 4–5

2.1.3 INTRODUCED MISALIGNMENT

  • Synthetic misalignments are random translations of 2.5–10 mm and rotations of 3.5–10 degrees, with a classification threshold of 2.5 mm and 3.5 degrees.Source: §2.1.3 · p. 5

2.1.4 PERFORMANCE EVALUATION

  • Accuracy is the primary metric, evaluated on 104 cases comprising one aligned and one randomly misaligned case for each MRI-PET pair.Source: §2.1.4 · p. 5

2.2 EXPERIMENT 1B: T1W-FLAIR MRI ALIGNMENT

Experiment 1b tests transfer to T1w-FLAIR alignment using selected models and the best-performing MRI-PET presentation and prompting settings. Its data span three studies, and alignment uses a single ANTs pipeline.

2.2 EXPERIMENT 1B: T1W-FLAIR MRI ALIGNMENT

  • The T1w-FLAIR evaluation uses selected models and limits testing to alpha-blend presentation with zero-shot or few-shot prompting.Source: §2.2 · p. 5

2.2.1 DATA

  • The test data include 11 ADNI sessions, 13 A4 sessions, and 16 sessions from the MICCAI 2016 multiple sclerosis lesion segmentation challenge.Source: §2.2.1 · pp. 5–6

2.2.2 GROUND TRUTH ALIGNMENT

  • Unlike the MRI-PET setup, T1w-FLAIR alignment uses a single ANTs pipeline on skull-stripped images prepared with SynthStrip.Source: §2.2.2 · p. 6

2.3 EXPERIMENT 2: MLLM COMPARISON AGAINST LOCAL MODELS

The experiment compares locally fine-tuned models with frontier MLLMs on MRI-PET misalignment detection, then tests transfer across modality and image-presentation settings. The local models use distinct training and validation subjects, separate from the test subjects.

2.3 EXPERIMENT 2: MLLM COMPARISON AGAINST LOCAL MODELS

  • MedGemma and ResNet-18 are trained on MRI-PET misalignment detection with contour-based presentation, with training-set size varied before testing transfer to T1w-FLAIR contours and MRI-PET alpha-blend.The two transfer tests probe generalisation across modalities and image-presentation methods, respectively.Source: §2.3 · p. 6

2.3.1 DATA

  • The training set contains 2,143 sessions from 1,006 subjects and validation uses 518 sessions from 125 subjects, with all subjects distinct from the 29 test subjects.Source: §2.3.1 · p. 6

3 RESULTS

Local models can match or exceed frontier MLLMs on MRI-PET when tuned for that task, but generalisation to new settings is more limited. Frontier-model performance varies across models, prompting, presentation methods, and tasks, while improving broadly over time.

3.1 EXPERIMENT 1A: MRI-PET ALIGNMENT

  • Across MRI-PET scenarios, GPT-6 Astra performs strongly and Claude Fable 5.1 ranks first or second in four of six; performance has broadly improved over time, with Gemini 3 Flash strong zero-shot on alpha-blend.The Figure 4 caption identifies its scope as misalignment detection performance over time across all six scenarios tested.Source: §3.1 · p. 6, Figure 4 · p. 7

3.2 EXPERIMENT 1B: T1W-FLAIR MRI

  • Performance is similar between T1w-FLAIR and MRI-PET experiments; Gemini performs comparatively strongly on T1w-FLAIR, and models that perform well generally do so across data subsets.Source: §3.2 · p. 6

3.3 EXPERIMENT 2: COMPARISON TO LOCAL MODELS

  • Both MedGemma and ResNet-18 match or exceed frontier MLLMs on tuned MRI-PET, show reasonable T1w-FLAIR generalisability that falls as training data increases, and generalise poorly to MRI-PET alpha-blend.Source: §3.3 · p. 6

4 DISCUSSION

The discussion finds that frontier performance has improved over time, while results vary with task, image presentation, and prompting. Local models can excel on trained settings, but transfer and study scope constrain broader conclusions.

Model performance over time

  • GPT-6 Astra performs best in 7 of 8 settings and second best in the remaining one; frontier-model performance has generally improved over time.The discussion also identifies Claude Fable 5.1 as a strong performer in almost all settings.Source: §4 · p. 7

Comparison to local models

  • On MRI-PET contours, both local models eventually exceed 98% accuracy and match or beat frontier models at almost all training sizes, while T1w-FLAIR performance falls as training grows.Exceptions occur at 1% training data for ResNet-18 and 2% for MedGemma; the figure also compares local performance with the best frontier MLLM on MRI-PET contours.Source: Figure 5 · p. 8, §4 · p. 9

Zero-shot vs. few-shot

  • Few-shot examples generally improve frontier MLLM performance, especially with alpha-blend presentation, but additional misaligned examples bring only marginal gains.Gemini 3 Flash is an exception: its strong zero-shot alpha-blend performance degrades with examples.Source: §4 · p. 8

Image presentation

  • Alpha-blend presentation enables more accurate misalignment detection than contours for almost all models; only the newest frontier models perform strongly on contours.Alpha-blend may expose edge misalignment, whereas contours may help reveal subtle internal brain misalignment.Source: §4 · p. 8

T1w-FLAIR assessment

  • MLLMs generally perform slightly better on T1w-FLAIR than MRI-PET, with exceptions among few-shot GPT-6 Astra and Claude Fable 5.1 results.The discussion attributes the expected difficulty difference to PET measuring physiological and molecular information rather than MRI anatomy.Source: §4 · p. 8

Data subsets

  • Few-shot results vary across A4, ADNI, and MICCAI; MICCAI is usually easiest, while examples can reduce Gemini performance on other subsets.The few-shot example comes from MICCAI, and the discussion says example selection should be considered carefully.Source: §4 · p. 8

Comparison to local models

  • Neither local model generalises well to MRI-PET alpha-blend; MedGemma predicts aligned throughout, while ResNet-18 performance changes in opposite directions across presentation settings as training data grows.MRI-PET alpha-blend performance rises with dataset size while T1w-FLAIR contours performance falls for ResNet-18.Source: §4 · p. 9

Limitations and practical considerations

  • Practical choice involves API privacy and cost trade-offs, while evidence covers two tasks, reasonably small datasets, and synthetically introduced rigid misalignments.The authors consider the synthetic approach reasonable for rigid-transform alignment, but say other tasks such as nonlinear deformation require a more careful approach.Source: §4 · p. 9

5 CONCLUSION

The authors present medical image alignment assessment as a useful test bed for generalist visual reasoning, with the most recent frontier models performing strongly across most scenarios. Contour-based presentation remains challenging, while locally fine-tuned models trade broader transfer for task-specific performance.

5 CONCLUSION

  • Recent frontier models perform strongly across most scenarios, but contour-based presentation reaches at most 75% accuracy in the zero-shot setting.The results support evaluating medical image alignment as a test bed for generalist visual reasoning.Source: §5 · p. 9
  • Locally fine-tuned models can match or outperform frontier MLLMs on their training tasks, but generalize less effectively to different tasks.They keep data local but require careful tuning and validation for each use case.Source: §5 · p. 9
  • The results support using MLLMs for visual alignment assessment in medical imaging pipelines, while evaluation across more modalities and tasks remains a next step.The authors describe strong frontier-model performance on these exemplar tasks as evidence of promise for visual medical image assessment.Source: §5 · p. 9

T. Rohlfing. Image Similarity and Tissue Overlaps as Surrogates for Image Registration Accuracy:

This reference-list block includes work on image similarity and registration accuracy, MRI-ultrasound error mapping, multimodal benchmarks, and medical-imaging studies.

T. Rohlfing. Image Similarity and Tissue Overlaps as Surrogates for Image Registration Accuracy:

EXAMPLE PROMPTS

The example prompts ask models to classify medical-image alignment from brain views, using either alpha-blended images or anatomical contours. Some prompts provide labeled examples as guidance.

EXAMPLE PROMPTS

  • The prompts identify three views of the same brain and ask whether the PET image and T1-weighted MRI are spatially aligned.Source: EXAMPLE PROMPTS · p. 18
  • The contours example describes colored anatomical contours from a T1-weighted MRI segmentation over a grayscale PET image.Source: EXAMPLE PROMPTS · p. 18
  • The alpha-blend example uses PET as the base image and overlays colored T1-weighted MRI at 0.5 opacity.Source: EXAMPLE PROMPTS · p. 19
  • One prompt supplies known classifications as guidance and asks for a label for the test case.Source: EXAMPLE PROMPTS · p. 19
Loading 2610.06896v1…