Source-linked AI summary
Taking Shortcuts for Categorical VQA Using Super Neurons
Pierre Musacchio, Jaeyi Jeong, Dahun Kim, Jaesik Park
TL;DR
Categorical VQA classifiers usually rely on the full VLM, but the paper investigates whether individual scalar activations contain sufficient answer information. It probes and thresholds raw activations to identify Super Neurons, which improve classification and enable extreme early exiting, including a reported 5.10× speedup. The authors limit their conclusions to categorical VQA and leave open-ended prompts and reasoning for future study.
Problem
The paper asks whether individual scalar activations in over-parameterized VLMs can provide satisfactory categorical VQA answers without using the full model.
Method
The method probes VLM activations, thresholds individual neurons into binary predictions, selects high-scoring neurons, and aggregates their outputs.
Results
SNs outperform base models across diverse categorical VQA benchmarks and enable first-layer, first-token early exiting with up to 5.10× speedup.
Takeaways & Limitations
Scalar activations can function as strong categorical classifiers while substantially reducing VLM inference time through extreme early exit.
Takeaways & Limitations
It remains unknown whether SNs provide useful signals for complex open-ended prompts and reasoning, or retain robustness and early exiting there.
Abstract
from arXiv · showhide
Sparse Attention Vectors (SAVs) have emerged as an excellent training-free alternative to supervised finetuning or low-rank adaptation to improve the performance of Vision Language Models (VLMs). At their heart, SAVs select a few accurate attention heads for a task of interest and use them as classifiers, rather than relying on the model's prediction. In a similar spirit, we find that directly probing the raw activations of the VLM, in the form of scalar values, is sufficient to yield accurate classifiers on diverse visually grounded downstream tasks. Shifting focus from attention vectors to scalar activations dramatically increases the search space for accurate parameters, allowing us to find more discriminative neurons immediately from the first generated token. We call such activations Super Neurons (SNs). In this probing setting, we discover that enough SNs appear in the shallower layers of the large language model to allow for extreme early exiting from the first layer of the model at the first generated token. Compared to the original network, SNs robustly improve the classification performance while achieving a speedup of up to 5.10x.
1 Introduction
The paper proposes Super Neurons (SNs), scalar activations that can classify categorical VQA answers without training. SNs outperform base models across diverse benchmarks and support first-layer, first-token early exiting.
- Motivation: VLMs contain micro-level scalar activations that may encode task-specific answer information despite their macro-level architecture.The paper contrasts scalar activations with representations formed by aggregating token interactions.
- Approach: SNs are discovered by probing VLM activations and thresholding individual neurons into classification predictions.The strategy is training-free and converts raw activations into categorical answers.
- Findings: SNs serve as strong categorical classifiers and outperform the base models on diverse VQA benchmarks.The authors benchmark probed neurons across a broad suite of categorical VQA tasks.
- Efficiency: 5.10× speedup is achieved through extreme early exit while maintaining model-level performance.SNs can operate from the first layer during generation of the first token.
2 Related work
Related work spans efficient VLMs, explainability, and sparse attention methods. The paper instead studies raw neuron activations as training-free categorical classifiers and uses them for early exit.
- Efficient VLMs: Efficient VLM research includes parameter pruning, token reduction, visual-contribution estimation, scale-down methods, and supervised early exiting.These approaches modify parameters, tokens, encoders, or inference depth to improve efficiency.
- Positioning: The paper shifts from clustered attention heads and macro-level representations to individual neural activations at the micro level.This preserves the training-free spirit of SAVs while changing the representation being probed.
- Search space: For LLaVA-v1.5-7b, the activation-scalar search space contains 131,072 candidates versus 1,024 attention-head candidates.The comparison is expressed as layers multiplied by dimensionality for SNs versus layers multiplied by heads for SAVs.
- Paper approach: The method studies individual neurons without adding a learning component or altering model weights.Raw activations are repurposed as categorical predictions, and their locations are analyzed for early answering.
3 Method
The method discovers Super Neurons by thresholding scalar LLM activations into categorical predictions, selecting high-scoring neurons, and aggregating their validation predictions. It also measures agreement with the VLM and examines where accurate neurons emerge across layers.
- A VLM forward pass produces layer-wise LLM activations from encoded image and text inputs.
- Probing set: The probing procedure gathers a dataset, runs end-to-end inference, stores activations, and evaluates neuron-level statistics against ground-truth labels.The probing set is typically built from training data used to optimize the task model.
- Discovering Super Neurons: Super Neurons are discovered by converting raw activations into binary predictions with an activation threshold α.The threshold is instantiated empirically, and each neuron is evaluated on the probing set using a metric μ.
- Discovering Super Neurons: Neurons whose probing-set metric exceeds the predetermined SN threshold are selected and stored in an index map Σ.The index function returns tensor values meeting the thresholding requirement.
- Evaluating Super Neurons on validation data: On validation data, selected SN activations are thresholded into predictions and aggregated by averaging or majority voting.The aggregation combines the predictions of K selected Super Neurons into one final prediction.
- Agreement rate: Agreement rate measures how frequently SN predictions match the VLM prediction across samples and selected neurons.The metric can be reported for different SN thresholds, such as AR@0.8.
4 Experiments
The experiments evaluate Super Neurons across diverse categorical VQA datasets, models, baselines, and ablations. SNs generally outperform or match base models, emerge in shallow layers, and enable substantial early-exit speedups.
- Datasets: The evaluation covers seven categorical VQA datasets spanning hallucination, occlusion, depth, visual understanding, geometry, knowledge, and scientific reasoning.The datasets include Pope, InstaOrder (Occ.), InstaOrder (Depth), VizWiz, Clevr, A-OKVQA, and ScienceQa.
- Datasets: SNs outperform the base models in all Table 1 cases using the single best probed neuron, with accuracy and F1 optimized separately.The probing set contains 3,000 samples except VizWiz, which contains 942 binary questions.
- Neuron analysis: Accurate SNs appear in shallow LLM layers, indicating that individual neurons can participate in decisions earlier than expected.The scalar-level search reveals many accurate neurons, including in the shallowest layers.
- Main evaluation: SNs largely outperform or match their source models across validation datasets and remain strong on challenging multiple-choice benchmarks.This pattern holds for both LLaVA-v1.5-7b and Qwen3-VL-4b-Instruct, including improved performance on occlusion understanding and A-OKVQA and ScienceQa.
- Comparison with baselines: SNs compare positively with n-shot prompting and SAVs, while balanced VizWiz evaluation reveals lower accuracy for SAVs despite high recall.The comparison with SAVs uses the same number of probing samples and a balanced yes-no VizWiz validation set.
- Ablation studies: 5.10× faster inference is achieved by stopping after the first layer and first generated token while maintaining model-level performance.SNs perform better on the first token, allowing the autoregressive process to be skipped; performance generally improves with more probing data and exceeds the base model above 100 samples.
5 Conclusion
Super Neurons are presented as strong, robust categorical classifiers that improve VLM runtime efficiency, while their usefulness for open-ended prompts and reasoning remains unresolved.
- Super Neurons serve as strong and robust categorical classifiers while improving VLM runtime efficiency.They compare favorably with base models and sparse attention vectors.
- SNs enable extreme early exiting during first-token generation from the first LLM layer, substantially reducing inference time.
- Future research must test whether SNs provide useful signals for complex open-ended prompts and reasoning.Robustness and early exiting also require evaluation in those contexts.
Supplementary Materials
The supplementary materials identify the paper’s authors and their institutional affiliations.
- Pierre Musacchio, Jaeyi Jeong, Dahun Kim, and Jaesik Park are listed as the paper’s authors.
- The authors are affiliated with Seoul National University, EPFL, and Google DeepMind.
- The listed contact addresses use Seoul National University and Jaesik Park’s email domain.
A Datasets and prompts
The supplementary materials describe the VQA datasets, prompt handling, model configurations, and comparison setup used in the experiments.
- Datasets: Pope contains 2.9K object-presence VQA questions built on COCO to evaluate VLM hallucinations.
- Datasets: InstaOrder evaluates instance-wise occlusion and depth-order understanding through two dataset splits.The occlusion split tests geometry-related prompts, while the depth split tests instance-wise depth order.
- Datasets: VizWiz contains visually constrained questions, and the experiments retain only yes-no answers.
- Datasets: Clevr tests spatial understanding across counting and positional questions using classification examples.
- Datasets: A-OKVQA is a four-choice multiple-choice dataset requiring commonsense reasoning grounded in visual scenes.
B Baselines
The baseline section fixes the pretrained configurations and compares Super Neurons with finetuned alternatives across annotation budgets.
- Baseline configurations: The experiments retain the reported LLaVA-v1.5-7b and Qwen3-VL-4b-Instruct configurations unless otherwise specified.
- Finetuned baseline comparison: With a small annotated finetuning budget, Super Neurons obtain better results than finetuned alternatives while remaining training-free.
- Finetuned baseline comparison: As the annotation budget grows, fully finetuning the model becomes a better alternative than Super Neurons.
C Profiling
Profiling on an NVIDIA A100 shows that Super Neurons substantially reduce answer-generation wall time by skipping autoregressive computation and Hugging Face post-processing.
- SNs dramatically decrease the wall time required to obtain an answer.The benchmark uses finer-grained measurements on an NVIDIA A100 while varying the model’s maximum generated token count.
- The speedup primarily comes from bypassing the transformer’s autoregressive process.
- Avoiding Hugging Face’s post-processing routine contributes substantially to the wall-time reduction.
D Robustness
SNs remain robust to prompt wording but depend on meaningful image–prompt grounding. Adversarial shuffling causes their F1 score to collapse, while random comparative strings still outperform the vanilla model.
- Prompt sensitivity: SNs maintain similar F1 scores across alternative relational words in InstaOrder (Occ.).This indicates robustness to the choice of initial prompt template.
- Prompt sensitivity: SNs outperform the vanilla model on random comparative strings formed from 3–10 sampled characters.The result suggests that SNs encode general in-domain knowledge for the task.
- Adversarial prompting: F1 score collapses when validation images or prompts are shuffled so that the prompt is not grounded in the image.The adversarial evaluation uses Pope-Im and Pope-Txt datasets containing shuffled VQA pairs.
E Location of SNs
SNs are more numerous on datasets where LLaVA-v1.5-7b struggles, and many occur in shallow layers. Some neurons overlap across tasks, especially between InstaOrder (Occ.) and Clevr.
- Dataset-wise: Harder datasets produce many more SNs than datasets the model already answers well.The visualization reports only SNs that outperform the network itself.
- Dataset-wise: Many SNs on difficult datasets appear in the shallower layers of the model.
- Cross-dataset: Some neurons capture useful information across multiple datasets, indicating overlap in task expertise.The overlap analysis considers neurons that exceed the model’s performance on every indicated dataset.
- Cross-dataset: InstaOrder (Occ.) and Clevr share the largest number of SNs, consistent with their shared focus on object-wise geometric understanding.The passage also reports neurons exceeding the model’s performance on three datasets and a single neuron doing so across multiple datasets.