Source-linked AI summary
Rethinking Projector Training in Multimodal LLMs
Fabian Gröger, Naël Ouerghemi, Shuo Wen, Maria Brbić
TL;DR
Projector alignment is widely used in multimodal LLM training, but its role remains poorly understood. The paper proposes PORTAL, which constructs a projector from per-modality statistics without paired data or gradient optimization. Across evaluated model pairs, PORTAL matches the standard pipeline and exceeds it in three of seven settings.
Problem
The role of the widely used projector-alignment stage in multimodal LLM training remains poorly understood.
Source: §1 · p. 1Method
PORTAL constructs a projector using optimal transport between Gaussian approximations of modality feature distributions, relying on per-modality statistics rather than paired captions or gradient optimization.
Source: §1 · p. 2Results
Across seven vision-encoder/LLM pairs, PORTAL matches the 558k-image-caption-pair pipeline and exceeds it in three settings, peaking at 106% of its trained-baseline score.
Source: §1 · p. 2Takeaways & Limitations
The results suggest projector alignment can often be replaced by or reduced to lightweight feature-distribution matching, lowering paired-data requirements and alignment cost.
Source: §6 · p. 10The study focuses on frozen vision and language backbones during alignment, single-image inputs, and simple projector architectures.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Multimodal LLMs are typically trained by first aligning a vision encoder with a language model via projector training on large image-caption datasets, followed by joint training of the full multimodal model. Despite its widespread use, the role of the projector training stage remains poorly understood. We find that (i) using only $10$ to $20\%$ of the data for training the projector already recovers most downstream performance gains, and (ii) projectors trained on different datasets can be linearly interpolated while largely preserving performance. These results suggest that projector training mainly performs a coarse alignment, which many different projectors achieve equally well. Motivated by these findings, we propose PORTAL, a training-free projector initialization that requires neither paired image-caption data nor gradient-based optimization, but only per-modality summary statistics. PORTAL computes an optimal-transport-based map between Gaussian approximations of the vision and language feature distributions on a shared principal subspace, so that the projector training stage can be skipped entirely. Without any paired data, PORTAL matches the standard pipeline trained on 558k image-caption pairs across six LLM backbones, two vision encoders, and 16 benchmarks. It outperforms that pipeline in $3$ of $7$ (vision encoder, LLM) settings and reaches an average $98\%$ of its score across the other four.
1 Introduction
Projector alignment is widely used, but its role remains poorly understood amid conflicting findings about whether it can be skipped or benefits from more data. The paper argues that it performs coarse alignment and introduces PORTAL, a training-free alternative that matches the standard pipeline across evaluated model pairs.
1 Introduction
- Projector alignment is standard, yet its role is unclear amid conflicting reports that skipping it can improve performance, scaling its data can help, or resolution and data mixing matter more.The standard pipeline freezes the vision encoder and language model while training the projector on image-caption pairs.Source: §1 · p. 1, Figure 1 · p. 2
- Using only 10–20% of the standard alignment corpus achieves similar performance to full-data training, while projectors trained on different datasets remain linearly interpolable.These findings support the view that alignment performs coarse mapping into a language-model-compatible space rather than learning a unique solution.Source: §1 · p. 1
- PORTAL constructs a projector from per-modality statistics using Bures-Wasserstein transport between Gaussian approximations on a shared top-k principal subspace.It requires neither paired captions nor gradient-based optimization and removes the projector-alignment training stage.Source: §1 · pp. 1–2
- Across seven vision-encoder/LLM pairs, PORTAL matches the 558k-pair pipeline and exceeds it in three settings, reaching 106% of its trained-baseline score.When used as an initialization, PORTAL reaches the baseline with 135× fewer alignment FLOPs; performance gaps concentrate on OCR-heavy benchmarks.Source: §1 · p. 2
2 Related Work
Prior work leaves the role and design of multimodal connectors unsettled, while related approaches vary in their use of learned components, paired data, or alignment objectives. PORTAL is positioned as a closed-form alternative based on marginal feature statistics.
Projector alignment in multimodal LLMs
- Prior studies disagree about Stage-I alignment: Prismatic reports it can be skipped, Cambrian-1 finds scaling to 2.5M pairs helps, and MM1 emphasizes resolution and data mixing.The paper frames geometric conditioning, rather than fine-grained semantic learning, as a way to reconcile these findings.Source: §2 · p. 2
Connector designs and projector geometry
- Connector designs range from LLaVA’s MLP and Q-Former to Perceiver Resampler, locality-enhanced abstractors, and token-reduction schemes [16] [5].Related analyses investigate information degradation, projector information loss, norm discrepancies [19], and where cross-modal alignment emerges.Source: §2 · pp. 2–3
Optimal transport for feature alignment
- PORTAL applies Bures-Wasserstein optimal transport between Gaussian approximations on a shared top-k principal subspace, extending second-order-statistic alignment to multimodal projector construction.Unlike CORAL, PORTAL uses the transport map as a closed-form projector rather than a regularizer.Source: §2 · p. 3
Training-free and efficient connector construction
- PORTAL requires neither paired data, gradient steps, learned caches, nor a vocabulary bottleneck, distinguishing it from other training-free or efficient connector approaches.It constructs the connector solely from per-modality summary statistics; Gupta et al. instead share parameters across unpaired modalities to improve a unimodal representation.Source: §2 · p. 3
3 What Does the Projector Alignment Actually Do?
The experiments suggest that projector alignment saturates with a small data fraction and admits a broad solution space across projectors trained on different data. Projectors from different random initializations, however, do not show the same interpolation behavior.
3.1 Projector Alignment Saturates with a Small Number of Samples
- 10–20% of LCS-558k captures most downstream benefit, and all categories plateau by 30–40%; larger ShareGPT-1.2M does not do better, peaking at average 38.1 with 40% of its data.The protocol uses a LLaVA 1.5 setup and evaluates 16 benchmarks grouped into Knowledge, General, OCR, and Vision-Centric categories.Source: §3.1 · p. 3, Figure 2 · p. 4
3.2 Projector Alignment Admits a Broad Solution Space
- Projectors trained on different datasets largely preserve downstream performance along the full linear interpolation path, supporting a broad solution space.The test interpolates projectors trained on LCS-558k and ShareGPT-1.2M at α values from 0 to 1.Source: §3.2 · p. 4
- Every interior cross-data blend improves on both trained endpoints in the aggregate, with α=0.5 optimal; category optima differ, with OCR closer to ShareGPT and Knowledge closer to LCS.The interpolation path combines projectors trained on LCS-558k and ShareGPT-1.2M without additional Stage-I optimization.Source: Figure 3 · p. 4, §3.2 · p. 4
- Cross-seed interpolation collapses captioning CIDEr from about 80 at either endpoint to 0.7 at α=0.5, unlike the cross-data interpolation path.The two cross-seed projectors share training data but start from different random initializations.Source: §3.2 · p. 4
- Limited-data saturation and cross-data interpolation suggest that alignment places visual features in a broadly compatible LLM input region rather than requiring one precise projector solution.The section tests data scaling and interpolation as evidence bearing on whether projector solutions are narrow or broad.Source: §3.1 · p. 3, §3.2 · p. 4
4 PORTAL: Training-Free Projector via Optimal Transport
The section motivates a training-free alternative because projector alignment appears to condition visual features into a broadly compatible LLM input region. PORTAL uses marginal statistics to construct a closed-form transport map, with moment matching offering a rationale for its effectiveness beyond Gaussian data.
4 PORTAL: Training-Free Projector via Optimal Transport
- PORTAL estimates each modality’s marginal feature statistics and uses a closed-form optimal-transport map to align visual features with a compatible language-feature distribution.The method is motivated by projector alignment as geometric conditioning that admits many possible solutions.Source: §4 · p. 5
4.1 Closed-Form Projector Initialization
- The projector maps source features to target embeddings; PORTAL fixes the second MLP layer to identity and its bias to zero, constructing the first layer and bias in closed form.The construction uses Gaussian approximations and a transport map on the top-k PCA subspaces, with principal coordinates matched by eigenvalue order.Source: §4.1 · p. 5
- The affine map whitens source features, expresses them in the target’s principal axes, matches target spread, and recenters the result.The statistics are computed separately for each modality; the method requires no Stage-I paired captions or optimization, and uses k=512 by default.Source: §4.1 · p. 5
- The Gaussian assumption does not match the full distributions because real source and target features deviate from Gaussianity in tail weight and marginal shape.The paper argues that the construction remains effective despite those deviations.Source: §4.1 · p. 5
4.2 Why Moment Matching Survives Non-Gaussianity
- Because the evaluated LLMs normalize tokens before attention, attention logits are unchanged by token rescaling and depend on token directions; PORTAL therefore targets their spread.The paper uses cosine dispersion ω to measure directional spread, and notes that covariance determines it when token norms are equal.Source: §4.2 · p. 6
- When token norms are equal, ω² equals 1/deff(Σ), so covariance alone determines directional spread; PORTAL’s matching of target mean and covariance then inherits the target’s ω in this limit.The result does not assume Gaussianity, but real token norms vary, so the bound does not precisely predict ω for the studied features.Source: §4.2 · p. 6
- PORTAL reduces cosine dispersion ω from 0.23 at the source to 0.09–0.10 on three CLIP–LLM pairs, toward the target’s 0.05–0.06; a random projector leaves it at the source value.The paper cautions that attention weights also depend on learned query and key projections, which ω does not describe.Source: §4.2 · p. 6
5 Experiments
The experiments compare PORTAL with a 558k-caption-trained projector under identical Stage-II tuning, then test its alignment budget, projector architecture, image pool, rank, and target layer. PORTAL matches or exceeds the trained baseline in several settings, while the ablations examine how robust its performance is to these choices.
5 Experiments
- The evaluation inserts PORTAL into the LLaVA pipeline and compares it with trained and random projectors across seven vision-encoder/LLM pairs and 16 benchmarks.All configurations use the same Stage-II instruction tuning; the trained baseline uses 558k paired captions for Stage I.Source: §5 · pp. 6–7
5.1 PORTAL Matches Trained Projector Alignment Without Paired Data
- 106.2% of the trained baseline on LLaMA 3.2 1B/CLIP, 106.1% on Qwen 2.5 0.5B/CLIP, and 102.7% on LLaMA 3.1 8B/CLIP; the four other settings reach 96.3% to 99.3%.PORTAL uses no paired captions or gradient updates; all configurations receive identical LLaVA-665k instruction tuning.Source: §5.1 · p. 7, Table 1 · p. 7
- On SigLIP, PORTAL reaches 99.3% of the trained projector versus Random’s 82.5%; two settings fall below the trained baseline’s bootstrap CI, with gaps concentrated on OCR and Chart benchmarks.Across categories, PORTAL retains 100.7%, 102.0%, 102.4%, and 100.4% of baseline performance on Knowledge, General, OCR, and Vision-Centric tasks, respectively.Source: §5.1 · pp. 7–8
5.2 PORTAL Reduces the Projector Alignment Budget
- Around 4k paired captions let PORTAL match the 558k-trained baseline at roughly 135× fewer alignment FLOPs; PORTAL stays above Random at every budget.With as few as 10k captions, PORTAL exceeds the full-data baseline and continues widening the gap up to 100k.Source: Figure 4 · p. 8, §5.2 · p. 8
5.3 Ablations: Projector Architecture, Image Pool, and Rank
- With a linear projector, PORTAL reaches 111.8% of the trained baseline versus Random’s 94.2%, widening their gap to 17.6 percentage points; with the standard MLP, it reaches 106.2% versus 99.9%.With the approximately 19× larger C-Abstractor, PORTAL remains essentially indistinguishable from conventional Stage-I training (42.93 vs. 42.98 average score).Source: §5.3 · p. 9, Table 2 · p. 9
- COCO val2017’s 1,024-image pool matches or exceeds the 4,096-image Cambrian pool on 12 of 15 benchmarks and has a higher average score (44.0 vs. 43.4); every tested rank beats the trained baseline by at least ∼4%.Rank performance is flat at ∼43.3 average from k=64 to 256, peaks at 44.2 for k=512, and decreases marginally to 44.0 at k=1024.Source: §5.3 · p. 9
5.4 Target Layer Choice
- Targeting the embedding table reaches 102.3% of the trained baseline on LLaMA 3.1 8B/CLIP, versus 101.9% at layer 4 and 101.4% at layer 8.The embedding table is the default target; PORTAL-ctx extends the construction to hidden states after an LLM block.Source: §5.4 · p. 9, Table 3 · p. 9
6 Conclusion
The findings support viewing projector alignment as coarse rather than fine-grained correspondence learning. PORTAL offers a training-free alternative, while the study’s conclusions remain bounded by its evaluated scope.
6 Conclusion
- 10 to 20% of LCS-558k recovers most downstream gains, and projectors trained on different datasets remain linearly interpolable, supporting a coarse-alignment view.The authors describe the mapping as visual features entering a language-model-compatible space, rather than fine-grained visual-language correspondences.Source: §6 · p. 10
- PORTAL uses per-modality summary statistics without paired captions or gradient optimization, remains close to the 558k-trained baseline, and reaches that baseline with roughly 135× lower alignment compute when used as initialization.The conclusion suggests alignment can often be replaced or reduced to lightweight feature-distribution matching.Source: §6 · p. 10
- The study examines projector alignment with frozen vision and language backbones, single-image inputs, and simple projector architectures; broader settings remain future work.The authors identify richer connectors, long-context grounding, multi-image reasoning, and audio, video, and 3D modalities as extensions to investigate.Source: §6 · p. 10
A Limitations
The analysis and method address Stage-I projector alignment with frozen vision and language backbones. The authors frame the results as an initial existence proof, not a general claim across multimodal-LLM regimes.
A Limitations
- The empirical analysis uses LLaVA 1.5 with a 2-layer MLP; richer connectors are untested and may not show the same linear interpolability.The authors also do not study end-to-end fine-tuning that updates the vision encoder.Source: A Limitations · p. 14
- PORTAL’s Gaussian approximation captures means, principal directions, and per-axis spread but omits higher-order structure, which the authors hypothesize may contribute to OCR and Chart gaps.The proposed missing structure includes character-level spatial layout.Source: A Limitations · p. 14
- The 16-benchmark suite does not stress long-context grounding or multi-image reasoning, and PORTAL has only been instantiated for vision-to-language alignment.Audio and video encoders are untested.Source: A Limitations · p. 14
- The authors treat geometric conditioning as the simplest hypothesis consistent with the data, not the unique mechanism behind Stage-I alignment.Other interpretations remain possible.Source: A Limitations · p. 14
- The closed-form construction exceeds trained Stage-I or stays within 1.6 average points across tested configurations; the authors make no claim that this extends to all regimes.Richer benchmarks or backbones may reveal settings where trained Stage-I outperforms PORTAL.Source: A Limitations · p. 14
B Training Details and Compute Resources
The appendices document the training resources and protocol, evaluation conventions, and the setup used for data-fraction and interpolation experiments.
B Training Details and Compute Resources
- Stage-I and Stage-II runs use the standard LLaVA-1.5 recipe, AdamW, and 8×A100 GPUs; PORTAL construction takes under 30 seconds on one RTX 3090.Stage-I takes under one hour for MobileLLaMA 1.4B and under four hours for Vicuna 7B; Stage-II takes 2–10 hours depending on scale. Runs use β1=0.9, β2=0.999, and no gradient clipping; Table 4 is identified as identical to LLaVA-1.5.Source: B Training Details and Compute Resources · p. 14, Table 4 · p. 14
C Evaluation Setup: Benchmarks and Protocol
- Evaluation covers 16 benchmarks in four Cambrian-1-adapted categories, with Lite SeedBench, normalized scores, arithmetic-mean averages, and paired bootstrap confidence intervals.The suite replaces OCRBench with OCRBench-v2 and MMVP with BLINK; SeedBench uses its 500-sample Lite split, and averages cover all 16 benchmarks.Source: C Evaluation Setup: Benchmarks and… · pp. 14, 16, Table 6 · p. 15
C.1 Setup for the Data-Fraction and Interpolation Experiments
- The data-fraction and interpolation experiments use LLaVA 1.5, with LCS-558k Stage-I projector training by default and identical LLaVA-665k instruction tuning across configurations.Section 3 uses CLIP ViT-L/14 and Vicuna 1.5 7B; the data-fraction experiment uses MobileLLaMA 1.4B, and the study also tests LCS-558k and ShareGPT-1.2M fractions.Source: §C.1 · p. 16
C.2 Cross-Seed vs. Cross-Data Interpolation
Cross-data projectors trained from a shared initialization interpolate smoothly, unlike the tested cross-seed pair. Additional experiments compare unpaired alignment methods, with PORTAL achieving the strongest reported recovery among those considered.
C.2 Cross-Seed vs. Cross-Data Interpolation
- Cross-data projectors interpolate smoothly and peak in aggregate score at α = 0.5, while the cross-seed pair’s captioning CIDEr falls from approximately 80 at either endpoint to 0.7 at α = 0.5.The cross-seed pair is farther apart in weight space (92.4 versus 63.7 Frobenius distance), yet only the cross-data pair stays within a low-loss interpolation region.Source: §C.2 · p. 16
- The authors interpret initialization as selecting a loss basin and training data as selecting a solution within that basin; PORTAL is intended to land in the compatible region without paired data.The comparison is between cross-seed projectors trained on the same corpus and cross-data projectors trained from the same initialization.Source: §C.2 · p. 16
D.1 LCS-558k Pretraining
- Table 7 reports per-benchmark results across LCS-558k data fractions, including four category groupings, an overall average, and its ratio to the 100% reference.The table uses CLIP ViT-L/14 and MobileLLaMA 1.4B with a two-layer MLP projector.Source: Table 7 · p. 17
D.2 ShareGPT-1.2M Pretraining
- Table 8 reports per-benchmark results across ShareGPT-1.2M data fractions using the same layout as Table 7.The corpus used for Stage-I pretraining is ShareGPT-1.2M.Source: Table 8 · p. 17
E Additional Experiments
- The additional experiments probe PORTAL with higher-capacity connectors, larger-scale instruction tuning, task-category variation, alternative unpaired methods, and object hallucination.The alternative-method comparison includes CORAL and Gromov-Wasserstein alignment.Source: E Additional Experiments · p. 18, §E.1 · p. 18
E.1 Comparison with Alternative Unpaired Alignment Methods
- PORTAL recovers 98.8% of the trained baseline, ahead of CORAL at 96.5% and GW at 98.0%, while requiring neither paired samples nor projector optimization.The table caption describes the compared alignment methods and notes that all except the conventional Stage-I baseline use no paired image-caption samples.Source: §E.1 · p. 18, Table 9 · p. 18
E.2 Performance Retention Across Task Categories
Across task categories, heavier Stage-II training, object hallucination, and a higher-capacity connector, PORTAL remains comparable to or better than conventional Stage-I alignment. The appendix also specifies its training-free projector construction from source and target feature statistics.
E.2 Performance Retention Across Task Categories
- PORTAL retains 100.7%, 102.0%, 102.4%, and 100.4% of trained-baseline performance on Knowledge, General, OCR, and Vision-Centric tasks, respectively.The aggregate is not driven by one category, though deficits vary by backbone and benchmark.Source: Table 11 · p. 18, §E.2 · p. 19
E.3 Large-Scale Multimodal Instruction Tuning
- With 3.2M Stage-II samples, PORTAL scores 47.32 versus 46.49 for conventional Stage-I initialization, a significant +0.83 points (95% CI [+0.36, +1.35]).This supports effectiveness beyond the standard LLaVA-665k Stage-II regime.Source: §E.3 · p. 19, Table 12 · p. 19
E.4 Object Hallucination
- Across all seven configurations, PORTAL matches or beats the trained baseline on POPE, with gains from +0.2 to +2.2 F1 points.In these experiments, replacing paired Stage-I projector training does not increase object hallucination.Source: §E.4 · p. 19, Table 13 · p. 19
E.5 Higher-Capacity Connector
- With C-Abstractor, PORTAL scores 42.93 versus 42.98 for Stage-I training, while random initialization scores 41.46.The connector has approximately 19× more parameters than the two-layer MLP; the authors caution that this does not establish identical behavior for every architecture.Source: §E.5 · p. 20, Table 14 · p. 20
F PORTAL Construction: Pseudocode
- PORTAL computes source and target feature statistics separately, then returns a two-layer LLaVA projector whose first layer is a Gaussian optimal-transport map and whose second layer is identity.The construction uses separate PCA operations and requires neither paired captions nor Stage-I optimization.Source: F PORTAL Construction: Pseudocode · p. 21
G Contextualized Targets: Methodology Details
The appendix describes contextualized target features for PORTAL and examines the Gaussian approximation underlying its construction. It audits modality distributions and relates cosine variability to effective dimension when token lengths vary little.
G Contextualized Targets: Methodology Details
- PORTAL-ctxℓ uses post-block-ℓ hidden states as the target distribution; ℓ=0 recovers standard PORTAL targeting the input embedding table.For supported models, hidden states come from a cache; otherwise the concepts.txt corpus is run through the model.Source: G Contextualized Targets: Methodology… · p. 22
H Cosine Dispersion: Theory and Empirical Checks
- PORTAL uses Gaussian approximations for a closed-form map even though measured source and target distributions are not exactly Gaussian.The audit samples 5,000 tokens, projects onto 128 principal components, and measures skewness, kurtosis, Gaussianity rejection, and effective dimension.Source: §H.1 · p. 22, H Cosine Dispersion: Theory and… · p. 22, Table 15 · p. 22
H.1 Distributional Audit of Source and Target Modalities
- Skewness is at most 0.003; vision-feature excess kurtosis is 0.28 to 0.49, versus 0.07 to 0.13 for LLM input embeddings.Anderson-Darling rejects Gaussianity on almost all top marginals; embedding-table effective dimension is 830 to 1,713, versus 4.5 for CLIP patches.Source: §H.1 · pp. 22–23
H.2 Proof of Proposition 1
- When token lengths vary little, Proposition 1 places cosine variance near 1/deff(Σ); the empirical ω statistic converges to Var(c) as sample size grows.The proof starts from the dot-product second moment, which depends only on Σ, then bounds the effect of normalizing by token lengths.Source: §H.2 · p. 23
H.3 Cosine Dispersion of PORTAL Projector Outputs
PORTAL moves population cosine dispersion from the source toward the target, while the appendix compares preactivation and GELU outputs with trained projectors. The reported geometric association is not claimed to explain downstream performance by itself.
H.3 Cosine Dispersion of PORTAL Projector Outputs
- PORTAL reduces population ω from the source value of 0.23 to 0.09–0.10, toward the target value of 0.05–0.06; a random projector roughly preserves source ω.Intra-image ω does not separate the projectors because patches in one image share a common offset; the authors say the measurement is correlative and ω is not the sole geometric explanation of downstream performance.Source: §H.3 · p. 24, Table 16 · p. 24
H.4 Post-Projection Statistics: Linear vs. MLP-GELU Output
- GELU roughly preserves PORTAL preactivation effective dimension and ω despite an order-of-magnitude increase in spectrum error; trained Stage-I projectors remain distributionally distinct despite comparable downstream averages.PORTAL preactivation matches target mean and top-128 spectrum, while the trained Stage-I projector deviates substantially from the target on reported statistics.Source: §H.4 · p. 24
I Image Pool for PORTAL Construction
In the LLaMA 3.2 1B/CLIP ablation, a smaller single-domain COCO image pool compares favorably with a larger, more diverse Cambrian pool. The results support the sufficiency of generic natural-image second-order statistics in this configuration.
I Image Pool for PORTAL Construction
- The 1024-image COCO pool matches or exceeds the 4096-image Cambrian pool on 12 of 15 benchmarks, with 2 ties and 3 Cambrian wins; its aggregate gains are +0.6 Avg and +1.3 %BL.The authors infer that increasing source diversity provides no additional benefit once the statistics have converged, for this configuration.Source: I Image Pool for PORTAL Construction · p. 25, Table 18 · p. 25
J Projector Architecture: Linear vs. MLP
This section examines PORTAL with linear and MLP projectors and across PCA ranks, and reports disclosures on crowdsourcing, human subjects, and LLM use.
J Projector Architecture: Linear vs. MLP
- PORTAL exceeds Random by 7.1 points under the linear projector and 2.7 points under the MLP, supporting transfer across projector architectures.Under the MLP, Random init scores 41.5 Avg versus 41.6 for the trained baseline; PORTAL scores 44.2 Avg.Source: J Projector Architecture: Linear vs. MLP · pp. 25–26
- 44.7 Avg (111.8% of the linear baseline) is the highest macro average among six configurations; PORTAL beats Random on 16/16 linear and 15/16 MLP benchmarks.The comparison uses LLaMA 3.2 1B/CLIP with Stage-II tuning on LLaVA-665k.Source: J Projector Architecture: Linear vs. MLP · pp. 25–26, Table 19 · p. 26
K PCA Rank Sensitivity
- 44.2 Avg at k=512 is the peak in the rank sweep; scores are approximately 43.3 Avg for k=64–256 and 44.0 at k=1024.The authors attribute the low-rank plateau to dominant PCA directions capturing most cross-modal structure, and the k=1024 decrease to noisy eigenvectors diluting the transport map.Source: Figure 5 · p. 26, K PCA Rank Sensitivity · p. 26
2. Limitations
- The checklist states that limitations are discussed in the Conclusion and Appendix A.The supplied passage does not specify what those limitations are.Source: §2 · p. 27
NeurIPS Paper Checklist
- The checklist reports 95% confidence intervals from 1,000 paired bootstrap resamples and says experimental details and hyperparameters are provided in the setup and appendices.It also states that code is available and that negative societal impacts are not expected because the method enables projector alignment without paired data while being computationally efficient.Source: §2 · pp. 28–30
2. Limitations
- The research involved no crowdsourcing or human subjects.The paper marks human-subject research as not applicable.Source: §2 · p. 32
- The research did not develop LLMs or use them beyond minor editing.The paper says LLMs were not an important, original, or non-standard component of its methods.Source: §2 · p. 32
Related papers
This paper cites
- Learning Transferable Visual Models From Natural Language Supervision2021
- Microsoft COCO: Common Objects in Context2014
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models2023
- Sigmoid Loss for Language Image Pre-Training2023
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks2023
- Towards VQA Models That Can Read2019