Source-linked AI summary
Benchmarking Vision-Language Models on Synapse Detection and Proofreading in Connectomics
Yicong Li, Junjie Wang, Leander Lauenburg, Ella Hugie, Alexandra Irger, Wanhua Li, Donglai Wei, Hanspeter Pfister
TL;DR
Annotators make many small visual decisions when detecting synapses and proofreading EM images, but whether VLMs can make those decisions and how to adapt them is largely unknown. We benchmarked VLMs across these tasks and adaptation settings. Most models were at or near chance zero-shot, while LoRA adaptation and cross-species transfer produced the strongest results.
Problem
Whether VLMs can make annotators’ decisions from raw EM, and what adaptation they require, is largely unknown despite their use of examples or short finetuning runs.
Source: §1 · p. 1Method
We benchmarked 19 open and two closed VLMs on public fly and mouse synapse tasks, and three open and two closed models on ConnectomeBench2 proofreading, including transfer to human and zebrafish.
Source: §1 · p. 1, §2.1 · p. 1, §2.2 · p. 3Results
Most models were at or near chance zero-shot; LoRA brought open models level with specialists, and the best adapted VLMs beat specialists on unseen-species merge errors.
Source: §1 · p. 1, §4 · p. 4, §3.1 · p. 4Takeaways & Limitations
VLMs’ clearest advantage is away from the trained setting: their adapters transferred to unseen species, where the best beat specialists on merge errors.
Source: §4 · p. 4
Abstract
from arXiv · showhide
We benchmarked vision-language models (VLMs) on the decisions annotators take when inspecting electron microscopy images in connectomics: synapse detection (presence and polarity) and proofreading (split errors and merge errors). For synapse detection, we evaluated 19 open and 2 closed models across various architectures and sizes under zero-shot, four-shot in-context learning and LoRA settings, against specialist models, on datasets constructed by us using public resources. For proofreading, we evaluated 3 open and 2 closed models on the ConnectomeBench2 dataset, with cross-species transfer from fly and mouse to human and zebrafish. Most models were at chance zero-shot; a few examples helped mainly the closed and largest open ones. LoRA on a few thousand labels brought open models level with specialist models. When evaluated on unseen species, the best adapted VLMs outperformed specialist models trained on the same data in identifying merge errors. The project will be publicly available upon acceptance.
1. INTRODUCTION
Connectomics depends on annotators making small visual decisions on EM images, but whether VLMs can make those decisions—and how to adapt them—is largely unknown. This study benchmarks VLMs on synapse detection and proofreading against specialist models.
1. INTRODUCTION
- Connectomics requires annotators to check synapse presence and polarity and proofread segmentations, where split or merge errors propagate through a neuron’s circuits.FlyWire required about three million proofreading edits, and MICrONS proofreading continued years after imaging.Source: §1 · p. 1
- Whether VLMs can make annotators’ decisions from raw EM, and what adaptation they require, remains largely unknown.VLMs can be steered with examples or short finetuning runs, but they struggle on general microscopy benchmarks, even with modality identification.Source: §1 · p. 1
- Prior work addresses synapse detection and proofreading with task-specific supervised models, while earlier proofreading language-model evaluations used mesh renderings rather than EM images.ConnectomeBench2 released expert-labelled EM sites and published supervised ViT results, but had not evaluated language models.Source: §1 · p. 1
- Most open models were at chance zero-shot, while LoRA brought them level with specialists; adapted VLMs also appeared more resilient to cross-species transfer.The benchmark covered synapse detection and proofreading, including transfer to unseen species.Source: §1 · p. 1
- The study benchmarks 19 open and two closed VLMs on synapse detection, and evaluates VLMs on ConnectomeBench2 proofreading against specialist models.Synapse detection uses zero-shot, four-example in-context learning, and LoRA settings; proofreading includes comparisons with closed VLMs and specialists on mouse and fly subsets.Source: §1 · p. 1
2. BENCHMARK
The benchmark uses public fly and mouse connectomes for synapse detection and ConnectomeBench2 for proofreading, evaluating VLMs under zero-shot, few-shot, and LoRA settings against specialist references. It tests both within-species performance and proofreading transfer to unseen species.
2.1. Task 1: Synapse detection
- Synapse detection uses single-section EM crops from public fly and mouse connectomes, with query-point presence and partner-polarity subtasks and separate training, validation, and test tissue.Fly data use region-separated CREMI volumes; mouse data use spatially separated blocks from MICrONS.Source: §2.1 · pp. 1–2
2.2. Task 2: Proofreading
- Proofreading reuses ConnectomeBench2 sites and matched controls across four species, with tinted segment masks defining split-error and merge-error decisions.The oblique view is omitted for merge-error evaluation because its orientation could leak the answer.Source: §2.2 · p. 2
2.3. Evaluation protocol
- Proofreading is assessed on the full test split, balanced mouse-and-fly subsets, and transfer to human and zebrafish, using balanced accuracy and shared images and prompts.Four-shot examples come from the same species and subtask; specialist references use the adapters’ image and label pools.Source: §2.2 · p. 3, §2.3 · p. 3
2.4. Models
- The synapse benchmark spans 19 open models across five model families and two closed models, while proofreading uses three mid-sized open models and two closed references.Open VLMs are compared with specialist ResNet-50, ConvNeXt-Tiny, and ViT-S/14 networks trained on corresponding adapter data.Source: §2.4 · p. 3
3. RESULTS
LoRA adaptation brought VLMs close to specialist performance on synapse detection and proofreading, and adapted VLMs outperformed specialists on unseen-species merge errors. Zero-shot performance was generally weak, while in-context examples helped on several tasks.
3.1. Synapse detection
- 50.0 was the median zero-shot score across 76 open-model cells, with 62 below 55; no open model passed 67 on mouse.Gemma 4 31B reached 73.6 on fly presence and Gemma 4 12B reached 75.0 on fly polarity; GPT-5.6 Sol reached 77.2 on fly polarity but 59.7 on fly presence.Source: §3.1 · p. 4
- 83.2 on fly polarity: Gemma 4 12B exceeded the best specialist’s 82.6 with four examples; Gemma 4 31B scored 81.8 on fly presence.The best mouse cells stayed at 64.4 and 66.9; GPT-5.6 Sol scored 79.4 and 83.1 on the two fly subtasks, while Claude Opus 5 scored 68.2 and 65.7.Source: §3.1 · p. 4
- 85.5 on fly presence, 91.1 on fly polarity, 93.5 on mouse presence, and 94.5 on mouse polarity were the best-adapter-per-cell scores.Overall, 63 of 76 LoRA cells reached 70; adapters were ahead of or level with specialists on fly and within three points on mouse.Source: §3.1 · p. 4
3.2. Proofreading
- 77.3 and 82.7 versus specialist scores of 66.9 and 74.7: Gemma 4 12B led on human and zebrafish merge errors.On transferred split errors, Qwen3.5-9B scored 83.1 and 88.0, against ConvNeXt-Tiny’s 85.4 and 89.3.Source: §3.2 · p. 4
- 95.0 and 90.9: Gemma 4 12B’s full-ConnectomeBench2 LoRA scores were within about one point of the dataset’s ViT-B scores, 94.2 and 91.1.The other two adapted open models were also within about one point of the ViT-B on both tasks.Source: §3.2 · p. 4
- On balanced mouse and fly proofreading subsets, LoRA models scored 83.1–93.3 on split errors and 74.5–84.6 on merge errors.They were within 1.6 points of references on mouse and 3.5 points behind on fly split errors, but led on fly merge errors, 80.1 against 79.7.Source: §3.2 · p. 4
4. DISCUSSION AND CONCLUSION
Without adaptation, most open VLMs were at or near chance and unsuitable for annotation pipelines, while closed models did better zero-shot but remained below adapted open models. A few thousand labels with LoRA brought open models level with purpose-trained networks, and adapters transferred to unseen species, outperforming specialists on merge errors.
4. DISCUSSION AND CONCLUSION
- A few thousand labels with LoRA brought open VLMs level with purpose-trained networks on synapse detection.On proofreading, adapted VLMs also reached the dataset’s ViT-B level on the full ConnectomeBench2 evaluation.Source: §4 · p. 4
- On unseen species, the best adapted VLMs outperformed specialist networks on merge errors.The discussion reports that adapters generally lost less accuracy than specialists when transferred to unseen species.Source: §4 · p. 4
- A VLM’s advantage is strongest when many decisions or species without training data are involved; a small purpose-trained network is cheaper for one fixed, well-labelled decision.One adapter per task answered both questions in both species, while specialist models needed one network per question.Source: §4 · p. 4
6. COMPLIANCE WITH ETHICAL STANDARDS
The study used publicly released, de-identified datasets under open licenses and collected no new human or animal data. Ethical approval was therefore not required.
6. COMPLIANCE WITH ETHICAL STANDARDS
- The study used publicly released, de-identified CREMI, MICrONS, and ConnectomeBench2 datasets under their open licenses.This included the human tissue sites in ConnectomeBench2’s public release.Source: §6 · p. 5
- No new human or animal data was collected.Source: §6 · p. 5
- Ethical approval was not required for the study.Source: §6 · p. 5