Vector-ICL: In-context Learning with Continuous Vector Representations

1 INTRODUCTION

• LLM ICL is generally learned and used through natural language because LLMs are trained on discrete language tokens, limiting use with non-textual data.Many modalities, including sensor readings, financial time series, and scientific measurements, lack a natural text representation.Source: §1 · p. 1
• Projector pretraining with a next-token objective enables Vector-ICL, while downstream fine-tuning further improves its effectiveness.The approach aligns continuous inputs with the LLM’s embedding space through embedding projection.Source: §1 · p. 1
• Figure 1 depicts Vector-ICL demonstrations as projected box tokens representing text, numerical data, brain fMRI, time series, or graphs.Inputs are encoded and transformed through embedding projection before being supplied in prompts during inference.Source: Figure 1 · p. 2
• Vector-ICL offers a more effective approach to function approximation, particularly for large numbers spanning multiple text tokens.The paper identifies tokenization schemes as a limitation for precise number processing.Source: §1 · p. 2
• Vector-ICL often surpasses few-shot ICL and task-specific models or tuning across diverse tasks and modalities.The study includes text reconstruction, numerical function regression, text classification, summarization, molecule captioning, time-series and graph classification, and fMRI decoding.Source: §1 · pp. 1–2

Empirical results of in-context learning

• Empirical studies examine ICL performance, demonstration properties, instruction tuning, model size, and methods such as explanations or chained calls.Work also reports ICL in multimodal models and tabular data.Source: §2 · p. 2

Learning to learn in-context

• Figure 2 depicts projector pretraining and task-specific fine-tuning as stages for enabling Vector-ICL and improving its performance against established baselines.The figure contrasts general-language-modeling or modality-to-text pretraining with downstream task-specific fine-tuning.Source: Figure 2 · p. 3

Understanding ICL

• Analyses connect ICL in specific settings to linear models, discrete functions, algorithms, and optimization-like processes, while other work studies generalization bounds.A separate proposal suggests ICL may arise from parallel structures in pretraining data.Source: §2 · p. 3

Learning to learn in-context

• MetaICL meta-trains pretrained LLMs across diverse tasks, while TNP and PFNs train transformers for in-context prediction over function families.Other work meta-learns transformers to model group preferences in context.Source: §2 · p. 3

3.1 EMBEDDING PROJECTION

• An encoder maps each input to an abstract representation, which a projector transforms into a box token for use by the LLM.Inputs may come from any modality, provided an encoder maps each data point to a vector space; raw inputs may already be continuous vectors.Source: §3.1 · p. 3
• Linear projection uses a matrix to produce projected embeddings, while a two-layer MLP is available when greater expressive power is needed.The MLP includes an input projection that maps the encoder dimension to the LLM embedding dimension.Source: §3.1 · p. 4

3.2 PROJECTED EMBEDDINGS AS CONTEXT

• Projected embeddings replace original inputs in prompts, serving as demonstrations or the query input for tasks such as sentiment classification and summarization.The prompt can pair projected examples with their labels and ask for the corresponding output for a projected query.Source: §3.2 · p. 5

3.3 TRAINING THE EMBEDDING PROJECTORS

• General language-modeling pretraining enables vector-context ICL, and task-specific fine-tuning further improves performance.For text, the projected first half of a snippet conditions next-token prediction on its intact second half, with the encoder and LLM frozen and the projector updated. For non-text data, general objectives can include number reconstruction, basic algebra, or next-token prediction from brain fMRI embeddings; fine-tuning uses structured prompts and conditional generation loss.Source: §3.3 · p. 5

Baselines

• Text tasks compare Vector-ICL with few-shot ICL and soft prompt tuning, while non-text tasks use cross-modal few-shot ICL or tuned encoders.Soft prompts serve as a lightweight comparison with the base LLM frozen; non-text inputs use numeric sequences or textual graph descriptions.Source: §4 · p. 5

Text Pretraining

• Text projectors are pretrained on WikiText-103 by projecting embedded text and applying next-token prediction loss to the continuation.The corpus contains over 100 million tokens, and snippets are split at random sentence boundaries.Source: §4 · p. 5

Text and Molecule Tasks

• The evaluation includes text reconstruction, classification, summarization, and molecule captioning, with BLEU, accuracy, and RougeL used for the respective tasks.Molecule captioning uses molecule sequence-caption pairs from LPM24 and evaluates performance by BLEU.Source: §4 · pp. 6–7

Arithmetic and Function Regression

• Numerical experiments use 10-digit numbers for reconstruction and addition, then evaluate function regression by predicting the integer part of f(x, y) = √x√y using mean relative error.The digits are represented as concatenated one-hot vectors; the regression error is the normalized ℓ1 difference from the ground truth.Source: §4 · p. 6

Brain fMRI Decoding and Classification

• fMRI experiments use 200-dimensional PCA responses and next-token pretraining; time-series and graph tasks use Chronos-base and Graphormer embeddings, respectively.fMRI decoding uses randomly sampled training context; the graph dataset is imbalanced, and its baseline encoder is finetuned on the target data.Source: §4 · p. 7

Text Reconstruction and Function Regression

• Projected text embeddings support reconstruction, with performance improving as the number of examples increases, while Vector-ICL consistently outperforms raw-number few-shot ICL on function regression.The number task uses 10-digit inputs that span multiple text tokens.Source: §5 · p. 8

Text Classification and Summarization

• Vector-ICL outperforms few-shot ICL and soft prompt tuning on text classification and gives more effective continuous context than text input and soft prompts for summarization.The reported classification comparison aggregates mean accuracy across five datasets, four encoders, and three LLMs; summarization aggregates RougeL across two datasets and three encoders.Source: §5 · p. 8

Cross-Modal Results

• Across the evaluated tasks, Vector-ICL often improves on task-specific or few-shot baselines, including text, molecules, time series, and graphs.The results cover molecule captioning, time-series classification, and graph classification in addition to text tasks.Source: §5 · pp. 8–9

Brain fMRI Decoding and Classification

• On fMRI decoding, LLMs surpass random baselines using projected 200-dimensional PCA factors, and performance improves with more context.The projectors are pretrained with next-token generation loss before decoding text from test-set fMRI signals.Source: §5 · p. 9

6.1 THE IMPACT OF ENCODER QUALITY ON VECTOR-ICL PERFORMANCE

• Across 15 configurations, reconstruction and classification rankings correlate positively; 4 correlations approach 1, supporting reconstruction as a predictor for encoder selection.The analysis spans 5 datasets and 3 LLMs.Source: §6.1 · p. 9, Figure 4 · p. 10

6.2 CASE STUDY: WHAT HAS BEEN LEARNED IN THE PROJECTIONS?

• Projected number embeddings preserve local similarity and separate numerically distant values, while block patterns reflect shared digit patterns across decimal places.The authors suggest this block structure may help LLMs process numerical relationships by preserving place-value hierarchy.Source: §6.2 · p. 9, Figure 4 · p. 10

6.3 CASE STUDY: CAN SYNTHETIC DATASET BE USEFUL IN CROSS-MODAL PRETRAINING?

• Synthetic time-series pretraining generalizes to new tasks; its QA corpus describes trends, anomalies, and stability using statistical analysis.The pipeline uses linear regression, z-scores, and variability thresholds, generating descriptive statements and binary questions for identified properties.Source: §6.3 · pp. 9–10, Figure 4 · p. 10

7 DISCUSSION

• The experiments do not cover every combination of variables and use only single-token encoders; variable-sized embeddings and instruction-tuning effects remain open questions.The authors leave broader exploration of modalities, tasks, encoder-decoder configurations, and approach limitations for future research.Source: §7 · p. 10
• Across the tested tasks and modalities, Vector-ICL often surpasses few-shot ICL and task-specific models or tuning.The experiments include text reconstruction, numerical function regression, classification, summarization, molecule captioning, time-series and graph classification, and fMRI decoding.Source: §7 · p. 10
• Linear projections are often sufficient, but cross-modal tasks such as time-series or graph inputs may require nonlinear transformations.Source: §7 · p. 10

A.1 DETAILED EXPERIMENT SETUP

• Text reconstruction uses Parallel Sentence Talks’ English subset and Quora to test recovery of messages from projected embeddings.Source: §A.1 · p. 16
• Numerical projectors are pretrained on 3- or 10-digit numbers encoded as flattened one-hot vectors; function regression tests f(x, y) = √x · √y using in-context examples.The 10-digit setup is challenging because such numbers span multiple tokens across the tested models.Source: §A.1 · p. 16
• Text classification covers five sentiment or emotion datasets and compares few-shot ICL and soft prompt tuning, using test accuracy.Soft prompt tuning uses 20 virtual tokens and one training epoch.Source: §A.1 · pp. 16–17
• Summarization uses XSum (Narayan et al., 2018) and XLSum’s English subset, evaluated with RougeL against reference summaries.Source: §A.1 · p. 17
• Molecule captioning uses the 161K-pair LPM24 dataset and evaluates generated captions against references with BLEU.The dataset contains molecule strings paired with captions.Source: §A.1 · p. 17
• Time-series classification uses Chronos-base representations on FordA and FordB, while synthetic QA pretraining describes trends, anomalies, and temporal stability.The synthetic corpus uses linear regression, z-score analysis, and variability thresholds; the baseline is the base encoder with a trained classification head.Source: §A.1 · p. 18
• Graph classification uses Graphormer embeddings on ogbg-molhiv and weighted sampling for its roughly 3% positive class; performance is measured by F1.The finetuned Graphormer serves as the baseline.Source: §A.1 · p. 18

A.2 CASE STUDY: WHAT INFORMATION WAS PERCEIVED FROM PROJECTED BRAIN FMRI?

• Physical Actions and Movements and Cognitive and Reflective Aspects have higher mean accuracy after next-token-only unsupervised pretraining on noisy fMRI inputs.The figure groups questions about the underlying text by category and reports mean accuracy for each.Source: §A.2 · p. 18, Figure 5 · p. 19

• Llama3.1-8B is the common LLM, and NV-Embed-v1 is the text encoder for Vector-ICL.Source: §A.3 · p. 19
• ICL and soft-prompt methods receive up to 50 shots as context.Source: §A.3 · p. 19
• Soft-prompt tokens are trained over the entire training set.Source: §A.3 · p. 19
• Table 3 lists hyperparameters for V-ICL training.Source: Table 3 · p. 20

A.4 ADDITIONAL EXPERIMENTAL RESULTS

• Figures 6 and 7 cover text reconstruction and function regression with 10-digit and 3-digit settings.Source: Figure 6 · p. 20, Figure 7 · p. 20
• Figures 8–10 cover text classification with pretrained projectors, finetuned projectors, and few-shot ICL.Source: Figure 8 · p. 21, Figure 9 · p. 21, Figure 10 · p. 22
• Figures 11 and 12 cover text summarization with pretrained and finetuned projectors.Source: Figure 11 · p. 22, Figure 12 · p. 22

A.5 BRAIN FMRI QUESTION CATEGORIES

• Table 4 presents brain fMRI question categories and their associated questions.Source: Table 4 · p. 23