PaperlayerView exact revision on arXiv

Source-linked AI summary

NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference

Aurélien Lac, Tony Wu

arXiv:2609.01657v1cs.IRcs.AIcs.CV

TL;DR

Multimodal document retrievers often repurpose generative or separate-tower architectures, incurring unnecessary complexity for encoding. NeoMME uses a single long-context bidirectional encoder for multilingual text and image patches, and its retriever outperforms evaluated models below 800M parameters while improving throughput and compressing embeddings.

  • Problem

    Multimodal encoders often repurpose separate or generative architectures, creating a need for shared, efficient encoders for non-generative retrieval.

    Source: §1 · p. 2, §8 · p. 25

  • Method

    NeoMME maps multilingual text and raw image patches into one long-context bidirectional Transformer, pretrained with masked discrete-diffusion text denoising conditioned on visible patches.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    0.523 and 0.556 nDCG@10 are achieved by the 260M and 800M retrievers, while the 260M model delivers about 2× throughput and 255× embedding compression with over 95% retrieval quality retained.

    Source: §1 · p. 2, §8 · p. 25

  • Takeaways & Limitations

    NeoMME provides a compact shared backbone for visual-document and text retrieval, supporting flexible representations and efficient large-corpus deployment.

    Source: §8 · p. 25

    NeoMME lacks an objective that directly predicts or reconstructs image content, leaving the effect of its visual training objective unresolved.

    Source: §7 · p. 24

Abstract

from arXiv · show

Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal language models. Visual document retrievers such as ColPali repurpose these models as encoders, carrying over the parameter and compute overhead of a VLM for a non-generative task. We introduce NeoMME, a family of 260M and 800M-parameter Multimodal and Multilingual bidirectional Encoders that process multilingual text and raw image patches in a single bidirectional Transformer encoder. Both models are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both support a 16,384-token context, enough to encode up to two standard 4K UHD images. To demonstrate its downstream capabilities, we fine-tune NeoMME with jointly trained dense and late-interaction heads. On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress late-interaction multimodal document embeddings by 255x while preserving over 95% of baseline nDCG@10. We contribute NeoMME to Hugging Face Transformers and release the pretrained backbone and retrieval-compatible checkpoints under Apache 2.0 at https://hf.co/collections/Hcompany/neomme.

1. Introduction

NeoMME is a family of multilingual, multimodal-native bidirectional encoders that share one Transformer across text and raw image patches, trained from scratch with masked discrete-diffusion objectives. Its retrieval variant supports dense and late-interaction representations, achieves strong ViDoRe v3 results, improves encoding throughput, and enables substantial embedding compression.

Architecture

  • NeoMME uses a single shared bidirectional Transformer in which modality-specific projections feed text embeddings and raw 32 × 32 pixel patches into every layer.Dynamic-resolution images preserve aspect ratio, while long context and 32-pixel patches support tractable high-resolution inputs.Source: §1 · p. 2

Downstream results

  • 0.556 and 0.523 nDCG@10 are achieved by NeoMME-Retriever 800M and 260M, respectively, on ViDoRe v3.The models are fine-tuned for visual document retrieval with both dense and late-interaction heads.Source: §1 · p. 2

Compression

  • 255× compression reduces each NeoMME-260M ViDoRe v3 document embedding from ∼1.5 MB to 6 kB while retaining more than 95% of original retrieval quality.Hierarchical token pooling at factor 8 is combined with asymmetric int8-query and binary-document quantization.Source: §1 · p. 2

Efficiency

  • 1.97× throughput lets NeoMME-260M encode 2048 × 2048 pages at 51.3 pages per second on an NVIDIA L40S versus ColModernVBERT.The comparison uses the same input size for both models.Source: §1 · p. 2

Pretraining

  • 260M and 800M NeoMME backbones are trained from scratch on multilingual text, code, mathematics, natural images, and document images with long-context dynamic-resolution encoding.Text-only pretraining uses masked discrete diffusion, while image–text pretraining conditions the same text-denoising objective on visible patches.Source: §1 · pp. 2–3

Retrieval system

  • NeoMME-Retriever outputs both dense and late-interaction representations from one backbone forward pass and is compatible with Sentence Transformers for deployment flexibility.The authors also release fused Late-Interaction Kernels that reduce peak VRAM and runtime for late-interaction training and inference.Source: §1 · pp. 2–3

2. Related work

Prior work spans bidirectional and converted encoder representations, masked diffusion, shared multimodal architectures, visual document retrieval, and late-interaction efficiency. NeoMME combines these directions through raw-patch, vision-tower-free bidirectional processing and dense plus late-interaction retrieval.

2.1. Text representation models

  • Encoder research progressed from masked bidirectional pretraining and dense or late-interaction retrieval to longer-context, multilingual models, while decoder-to-encoder conversion enables bidirectional representations.Ettin finds task-specific differences: tested encoders are stronger on classification and retrieval, whereas decoders are stronger on generation.Source: §2.1 · p. 3

2.2. Masked diffusion for representations

  • Masked diffusion extends denoising across sampled noise trajectories, and diffusion-pretrained states now support text embeddings, multilingual encoders, and multiple retrieval representations.Related multimodal systems either condition on separate image encoders or use externally tokenized images, primarily for generation; NeoMME instead combines raw patches, a shared bidirectional Transformer trained from scratch, image-conditioned diffusion, and visual-document retrieval.Source: §2.2 · pp. 3–4

2.3. Vision–language architectures

  • Vision-language architectures range from separate vision and language towers to shared backbones, while recent tower-free models feed patches into decoder-oriented systems and establish components of shared multimodal processing.Existing shared-backbone approaches include pretrained or modality-specific components, discrete image codes, and from-scratch raw-patch decoders.Source: §2.3 · p. 4

2.4. Visual document representation and retrieval

  • Visual document encoders increasingly embed pages directly as images, avoiding OCR while preserving layout, tables, figures, and other evidence for visual retrieval-augmented generation.Recent systems improve this recipe with compact encoders or larger embedding spaces but retain inherited visual components such as pretrained vision towers.Source: §2.4 · p. 4

2.5. Late-interaction capacity and efficiency

  • Late interaction offers greater expressive capacity than dense retrieval but increases storage and scoring costs, motivating pruning, pooling, representation transformation, and fused MaxSim kernels.The cited theoretical and synthetic results do not establish universal late-interaction superiority or explain visual-document retrieval in every task.Source: §2.5 · p. 5

3. Architecture

NeoMME is a single bidirectional Transformer encoder that jointly processes multilingual text and RGB image patches in a shared hidden space. Its architecture combines efficient tokenization, dynamic-resolution vision inputs, multimodal positional encoding, and long-context sparse attention in 260M- and 800M-parameter models.

3. Architecture

  • NeoMME jointly processes text tokens and RGB image patches through modality-specific input layers mapped into one shared hidden space.Both modalities are handled by a single bidirectional Transformer rather than separate towers or a decoder-based visual-language architecture.Source: §3 · p. 5

3.1. Tokenizer

  • NeoMME emits 44.4% fewer tokens than ModernBERT, 39.4% fewer than LFM2.5-Encoder-230M, 6.3% fewer than mmBERT-base, and 16.9% fewer than EuroBERT-210m.The tokenizer uses a 131,072-entry byte-level BPE vocabulary trained on multilingual text, code, mathematics, and machine-produced image transcripts, but coverage is weaker outside the 14 target languages.Source: §3.1 · p. 5

3.2.1. Text tokens

  • Text tokens use a 256-dimensional factorized embedding projected to model width, and the tied masked-token decoder reuses both factors without extra output parameters.This factorization reduces model size while preserving a shared input-output parameterization.Source: §3.2.1 · p. 6

3.2.2. Image patches

  • Images are converted to RGB, split into non-overlapping 32 × 32 patches, and projected with a layer-normalized two-layer MLP trained from scratch.The pipeline randomly samples a longest-side cap between 1,024 and 2,048 pixels while preserving aspect ratio; 32-pixel patches produce approximately one quarter as many tokens as 16-pixel patches.Source: §3.2.2 · p. 6

3.2.3. 2D rotary position embeddings

  • NeoMME extends RoPE to two coordinate axes, using diagonal coordinates for text and row-column coordinates for image patches.Global layers use partial RoPE, while sliding-window layers use full RoPE with distinct bases for the two attention patterns.Source: §3.2.3 · p. 7

3.3. Modern bidirectional backbone

  • NeoMME uses 256- and 1,024-token half-windows as design choices; Figure 3 shows the layer sequence for both model sizes.Source: §3.3 · p. 7

3.4. Weight initialization

  • NeoMME initializes attention-output and MLP down-projections to zero, making each residual branch an exact no-op and each block an identity at initialization.The factorized word table uses a dimension-dependent scale, while token-indexed value embeddings and exclusive-self-attention coefficients start at zero.Source: §3.4 · p. 9

4. Pretraining

NeoMME pretrains a shared backbone from scratch on text-only and multimodal data using a masked discrete-diffusion objective conditioned on visible image patches. The training design mixes diverse sources, optimizes long packed sequences efficiently, and demonstrates that both model sizes use image evidence during masked-text recovery and generation.

4.1. Dataset mixture

  • The planned pretraining mixture contains about 524B packed tokens, with 55% from text-only data and 45% from multimodal data spanning documents, OCR, natural images, and varied text sources.The text stream includes 14 datasets across 58 configurations, while source sampling follows the weights defined in Table 2.Source: §4.1 · p. 9

4.2. Data pipeline and systems optimization

  • Documents are packed into 16,384-position streams with isolated attention boundaries, background image patchification, prefetched batches, and compute-aware rank balancing to keep GPU utilization high.FlashAttention-3, Liger Kernel’s fused linear cross-entropy, and torch.compile support the optimized training pipeline.Source: §4.2 · p. 9

4.3. Pretraining objective

  • NeoMME uses discrete masked-diffusion denoising over text, with visible image patches conditioning masked-token prediction and no image-reconstruction loss.At inference, the same denoiser can iteratively reveal masked tokens, optionally conditioned on visible image patches.Source: §4.3 · pp. 9–10
  • Multimodal segments use corruption rates U(0.30, 1), while text-only segments use U(0, 1), reducing textual shortcuts and encouraging image-based predictions.Selected eligible text positions are always replaced by mask tokens, and loss is computed only at those positions.Source: §4.3 · p. 10, §4.5 · p. 12

4.4. Training settings

  • Both 260M and 800M models train for 500,000 steps at 1,048,576 packed input tokens per global step, totaling 524.288 billion planned tokens.Runs use AWS p5.48xlarge5 instances with EFA networking and a warmup–stable–decay schedule whose final decay reaches 1% of peak learning rates.Source: §4.4 · pp. 11–12

4.5. Text masking and image sensitivity

  • 38.4 and 40.5 percentage points of image gain are reached at 90% masking by NeoMME-260M and NeoMME-800M, respectively, with positive gains at every tested rate.Figure 6 tracks pretraining loss and 90% image gain; image gain rises from near zero during early training, while the final loss decline coincides with WSD annealing.Source: §4.5 · pp. 12–13, Figure 6 · p. 13

4.6. Image-conditioned generation

  • Unprompted image-conditioned decoding iteratively generates captions or document text from visible image patches and a fully masked canvas, but the examples do not measure average generation quality.Experiments compare uniform and blockwise LLaDA-style decoding with cosine mask schedules, greedy decoding, or nucleus sampling.Source: §4.6 · p. 13

5. Retrieval

This section presents NeoMME-Retriever’s joint dense and late-interaction approach for retrieving visual documents and text, along with evaluations of benchmark performance, resolution trade-offs, compression, and indexing efficiency. It also examines qualitative similarity-map localization around relevant chart text.

5.2. Retrieval architecture

  • A single backbone pass produces dense and late-interaction representations, enabling direct late-interaction retrieval or dense ANN retrieval followed by late-interaction reranking.The late-interaction head projects each token or image patch to 128 dimensions, while the dense head mean-pools and L2-normalizes backbone states.Source: §5.2 · p. 15

5.3. Dataset mixture

  • Retrieval training combines text-only and multimodal query-document streams, with decontamination against all evaluated benchmarks and self-mined visual hard negatives.Generated page-image queries increased the multimodal stream from 760,826 to 952,741 queries without adding pages.Source: §5.3 · p. 16

5.4. Contrastive objective

  • The contrastive objective jointly weights dense and late-interaction losses, uses full in-batch negatives via GradCache, and omits teacher logits and knowledge distillation.Late-Interaction Kernels reduce MaxSim memory at 4,096 document tokens from 672 MB to 193 MB and backward time from 1.82 ms to 0.54 ms.Source: §5.4 · p. 17, §5.5 · p. 17

5.7. Main results

  • Late-interaction outperforms dense retrieval on BEIR-15 by 18.3 points for NeoMME-260M and 14.4 points for NeoMME-800M, whose scores are 0.513 and 0.369.The shared encoder therefore supports both page-image and text retrieval, with late-interaction the stronger representation.Source: Table 6 · p. 19, §5.7 · p. 19
  • 0.523 nDCG@10 makes NeoMME-260M the strongest model below 300M parameters and strictly below 800M across ViDoRe benchmarks, improving over the best sub-300M model by 26.1, 11.5, and 5.4 points on v3, v2, and v1.Its ViDoRe v3 score is within 0.2 points of 3.75B-parameter ColQwen2.5 while using 14.4× fewer parameters.Source: §5.7 · pp. 18–19, Table 5 · p. 19
  • 0.556 nDCG@10 is reached by NeoMME-800M on ViDoRe v3, within 0.9 points of ∼850M Vultron Flash and 3.2 points above 3.75B ColQwen2.5.Scaling from 260M to 800M improves ViDoRe v3, v2, and v1 by 3.3, 3.7, and 1.5 points, respectively.Source: Table 5 · p. 19, §5.7 · p. 19

5.8. Resolution impact

  • Reducing the image cap from 2,048 to 1,536 pixels lowers ViDoRe nDCG@10 by at most 1.8% while using about 57% as many vectors and float32 bytes.At 1,024 and 768 pixels, relative quality losses range from 3.0–12.7% and 11.7–36.1%, respectively.Source: §5.8 · pp. 19–20

5.9. Compression and storage

  • 255.5× compression retains 95.19% of baseline nDCG@10 using pool factor 8 with int8 queries and binary documents; int8 and binary indexes alone are 3.9× and 32× smaller.Pooling factor 7 retains more than 99% of ViDoRe v3 nDCG@10 while reducing document-vector storage by about 7×.Source: §5.9 · pp. 20–21

5.11. Interpretability

  • Similarity maps show strong responses around chart titles, axis labels, and numbered hours, suggesting emergent OCR-like behavior without a separate OCR pipeline.The visualization uses per-token cosine similarities and MeanMaxSim to identify relevant document positions.Source: Figure 14 · p. 23, §5.11 · p. 23
  • The maps are qualitative diagnostics because MeanMaxSim also uses nonspatial structural positions and background responses may reflect high-norm register-like tokens rather than local visual evidence.They therefore do not fully explain the retrieval score.Source: §5.11 · p. 23

6. Other downstream tasks

Beyond retrieval, NeoMME-260M is evaluated as a shared initialization for language, document, and natural-image tasks, emphasizing adaptability rather than modality-specialized superiority. It transfers strongly to language and document images after fine-tuning, while frozen natural-image representations remain weak.

6. Other downstream tasks

  • The downstream experiments test whether one compact backbone supports task-specific text, document-image, and multimodal fine-tuning, measuring adaptability rather than superiority over specialized models.Retrieval remains NeoMME’s principal downstream application.Source: §6 · p. 23

6.1. Language transfer

  • NeoMME-260M achieves a 75.3 mean across 17 language tasks, within 4 points of LFM2.5-Encoder-230M’s 79.3, while exceeding it on five tasks but reaching only 46.4 Matthews correlation on CoLA.The suite contains eight GLUE, four SuperGLUE, and five multilingual tasks; the comparison follows aligned task definitions and seeds but not an identical training harness.Source: §6.1 · p. 23
  • 85.5±0.2 accuracy on LEDGAR and 89.3 entity F1 on CoNLL-2003 demonstrate transfer to long-document classification and token labeling.LEDGAR uses an 8K context, while the CoNLL-2003 result comes from a one-seed experiment.Source: §6.1 · p. 23

6.2. Visual transfer

  • Fine-tuning on only 6,000 RVL-CDIP examples raises accuracy from 51.6±0.3 with a frozen first-token probe to 81.5±0.6, indicating rapid document adaptation.The result should not be read as clean out-of-distribution generalization because RVL-CDIP contains annotation noise and train–test overlap.Source: §6.2 · p. 24
  • Frozen 16-shot natural-image probes average 13.2 accuracy, whereas task-specific fine-tuning reaches 77.1±0.3 on Food101, 63.9±0.2 on Oxford Pets, and 46.8±6.3 on Stanford Cars.These exploratory diagnostics reflect that pretraining uses visible patches for masked transcript recovery without pixel reconstruction, image classification, or image-level contrastive learning.Source: §6.2 · p. 24

7. Limitations

The study is limited by constrained compute and data, missing objective and supervision comparisons, incomplete mixed-modality evaluation, and gaps in multilingual visual-document coverage. Additional limitations concern unisolated design choices, absent distillation, data bias, and safety risks.

Pretraining scale

  • NeoMME processes about 524B packed tokens, including 290B text-only, versus ModernBERT’s roughly 2T text tokens, limiting conclusions about pretraining scale.ModernBERT sees roughly seven times more text-only data, so longer runs with more text, images, and training steps remain untested.Source: §7 · p. 24

Visual objective

  • The masked-text objective provides no direct image-content target, potentially explaining weak frozen natural-image results and motivating matched visual-objective studies.Image-token prediction, pixel reconstruction, and image-level contrastive objectives could separate objective effects from data mixture and architecture.Source: §7 · p. 24

Retrieval supervision

  • NeoMME uses far less retrieval supervision than mLateOn, with about 430K text and 850K image-query examples versus 660M contrastive and 16M hard-negative examples.Matched larger training sets are needed to distinguish data-scale effects from architecture, objectives, and evaluation code in BEIR differences.Source: §7 · p. 24

Mixed-modality corpora

  • NeoMME-Retriever was not evaluated on mixed-modality benchmarks, despite supporting one candidate pool for text chunks and page images.UniDoc-Bench covers 70,000 PDF pages, while MixBench covers images, texts, and multimodal documents; multilingual extensions also remain future work.Source: §7 · p. 24

Visual document data

  • Visual retrieval lacks broad multilingual document data, especially for low-resource languages, because translating real PDFs while preserving layout and visual content is difficult.The visual stream covers English, French, German, Italian, and Spanish, with Portuguese generated queries, while synthetic generation could expand scripts and regions.Source: §7 · pp. 24–25

Additional limitations

  • Major architecture, scaling, data, and training effects remain unisolated, while distillation, bias, and safety evaluations are also absent.Future work includes matched ablations, distilling 800M into 260M, measuring web and synthetic-data biases (Singh et al., 2025), and testing harmful or private text across inputs and decoding methods.Source: §7 · p. 25

8. Conclusion … A.3. Processor

NeoMME combines a shared multilingual, multimodal bidirectional encoder with flexible retrieval representations, achieving strong throughput and highly compressed document embeddings. The paper also specifies its training mixtures, retrieval setup, processor format, and released implementation.

8. Conclusion

  • A single bidirectional Transformer maps multilingual text tokens and raw 32 × 32 image patches, using dynamic resolution and one masked-diffusion objective across text-only and image-conditioned examples.The 260M and 800M models share this architecture and train on multilingual text, code, mathematics, natural images, and document images.Source: §8 · p. 25
  • 255× compression reduces late-interaction multimodal embeddings from roughly 1.5 MB to 6 kB per page while retaining more than 95% of original retrieval quality.NeoMME-Retriever 260M also outperforms all evaluated models strictly below 800M parameters and encodes matched 2048×2048 pages at about 2× ColModernVBERT’s throughput on an NVIDIA L40S.Source: §8 · p. 25
  • The release provides pretrained NeoMME checkpoints and a day-zero Hugging Face Transformers implementation for efficient multimodal and multilingual representation-model research.The checkpoints and implementation are intended as a foundation for practitioners building representation models.Source: §8 · p. 25

A.1. Pretraining data mixture

  • Pretraining uses separate text-only and visual–text streams whose configured source-sampling weights define within-stream distributions rather than observed shares of tokens, documents, images, or pixels.The text mixture covers 58 datasets, while the visual–text stream covers natural images, rendered documents, PDF pages, charts, diagrams, and synthetic image-caption sources.Source: §A.1.1 · p. 37, Table 8 · p. 37, Table 9 · p. 38

A.2. Retrieval training settings

  • Retrieval fine-tuning uses one p5.48xlarge5 node with eight H100 accelerators, with settings summarized separately for both model sizes.The detailed pretraining settings appear in the main pretraining section, while retrieval settings are reported in Table 12.Source: §A.2 · p. 39, Table 12 · p. 39

A.3. Processor

  • The processor represents queries and documents with explicit token and patch-grid notation, including query and document lengths and patch coordinates over an H-row by W-column grid.These conventions define the processor’s structural format rather than additional model inputs.Source: §A.3 · p. 40, Table 13 · p. 40
  • The processor inserts [QUERY] or [DOC], adds [IMG] and row-delimiting [ROW] tokens for image documents, and appends 10 learned [MASK] tokens to queries.Image patches are arranged in row-major order, and the query augmentation follows ColBERT’s design; its effect is not isolated in an ablation.Source: §A.3 · p. 40, Table 13 · p. 40

A.4. Training mixture details · A.5. Contrastive training details · B. Supporting pretraining results

NeoMME training combines large text-only and multimodal retrieval streams with multilingual augmentation, staged visual hard-negative mining, and memory-efficient contrastive computation. The supplied passages specify dataset composition, query generation, candidate filtering, and implementation optimizations, but provide no supporting-pretraining results.

A.4. Training mixture details

  • The text-only stream contains 1,581,039 queries from five source pools spanning English, multilingual, code, organic, and translated mLateOn data.Sampling weights within the text stream are defined in Table 14.Source: §A.4 · p. 40, Table 14 · p. 40

B. Supporting pretraining results

  • The supplied passages contain no reported supporting-pretraining results for the merged B. Supporting pretraining results section.Accordingly, no benchmark values or comparisons are attributed to that section.Source: §A.4 · p. 40

A.4. Training mixture details

  • The multimodal stream combines ColPali, multilingual document-image, and VisRAG datasets, with equal source weights and original coverage in five languages.Query augmentation adds Portuguese, expanding coverage to six languages.Source: §A.4 · pp. 40–41, Table 15 · p. 41
  • Query augmentation generated 191,915 filtered queries from 100,000 sampled pages, raising the multimodal query count from 760,826 to 952,741 without adding pages.Queries varied across six task types, three formats, and six languages before Qwen3-VL-Reranker-8B filtering.Source: §A.4 · p. 41

A.5. Contrastive training details

  • Visual hard-negative mining first trains with in-batch negatives, then mines 32-candidate windows and retrains with one positive plus seven filtered negatives.Cross-rank gathering expands candidates, while negatives scoring at least 0.98× the positive are removed using stored mining scores.Source: §A.5 · p. 41
  • GradCache enables contrastive training over gathered candidates in smaller activation chunks, while LIK avoids materializing the full token-similarity tensor.LIK averaged 1.6% higher NeoMME-260M throughput in the control, although run-to-run variation overlapped.Source: §A.5 · p. 41

B.1. Tokenizer evaluation … C.1. Evaluation protocol details

The merged sections specify tokenizer comparisons, qualitative image-conditioned generation examples, and the retrieval evaluation protocol for ViDoRe and BEIR-15. They report that NeoMME’s tokenizer is less compressed than mmBERT and EuroBERT across the evaluated multilingual language-script pairs.

B.1. Tokenizer evaluation

  • The tokenizer evaluation compares NeoMME with ModernBERT, LFM2.5-Encoder-230M, mmBERT-base, and EuroBERT-210m on fixed multilingual and domain text inputs.Language evaluation uses 1,012 aligned FLORES-200 devtest sentences per language; domain evaluation uses 2,000 documents per source capped at 4,000 characters, with lower tokens per UTF-8 byte preferred.Source: §B.1 · pp. 41–42, Table 16 · p. 42, Table 17 · p. 42
  • NeoMME emits 65.29% more tokens than mmBERT and 15.28% more than EuroBERT across all 204 FLORES-200 language-script pairs, with largest deficits in Tibetan, South Asian, and Southeast Asian scripts.The comparison uses identical texts without added special tokens, and the aggregate reduction ratios sum token counts over 14 languages.Source: §B.1 · pp. 41–42

B.2. Image-conditioned generation examples

  • NeoMME-260M3 generation examples are qualitative demonstrations rather than task-level captioning or optical character recognition measurements.The examples are presented as selected image inputs and generated text, with truncation marked by [...].Source: §B.2 · p. 42, Table 18 · p. 43

C. Retrieval evaluation and complete results

  • The retrieval study reports complete metric and domain breakdowns, trained Matryoshka widths, and supporting ablations.The section frames these analyses as part of the retrieval evaluation rather than as separate task definitions.Source: C. Retrieval evaluation and complete… · p. 44

C.1. Evaluation protocol details

  • The retrieval evaluation covers ViDoRe v1, v2, and v3 document retrieval benchmarks plus BEIR-15 under the MTEB 2.18.7 definition and mLateOn task list.BEIR-15 contains 14 original zero-shot BEIR datasets plus MS MARCO development, omitting four datasets requiring corpus reconstruction and counting CQADupstack’s 12 child corpora once through its official aggregate.Source: §C.1 · p. 44
  • Visual retrieval uses downscale-only resizing to a maximum 2,048-pixel image side with bfloat16 execution and exact all-pairs MeanMaxSim scoring for ViDoRe.BEIR-15 uses the official MTEB scorer over full corpora with 8,192-token query and document limits, title followed by body, compressed multi-vector indexing, and exact top-candidate rescoring.Source: §C.1 · p. 44
  • Reported retrieval results use 260M and 800M models at step 20,000, averaging metrics over judged queries and weighting every task equally.ViDoRe v3 reports eight tasks, while BEIR-15 reports 15 tasks.Source: §C.1 · p. 44

C.2. Complete retrieval results … D.1. Efficiency measurement details

The complete retrieval evaluation reports NeoMME-800M’s ViDoRe v3 mean, examines representation-width and dual-head effects, and details a controlled cross-device efficiency protocol. The analyses show measurable late-interaction gains from joint training while defining important benchmarking conditions and limitations.

C.2. Complete retrieval results

  • The retrieval aggregates use 2,048-pixel maximum image sides, full 128-dimensional late-interaction vectors, and full-width dense representations.These settings define the aggregate visual-retrieval measurements reported at training step 20,000.Source: §C.2 · p. 44, Table 20 · p. 46
  • 0.5560 mean nDCG@10 is reported for NeoMME-800M on ViDoRe v3 across domains and tasks.Table 22 reports task-level ViDoRe v3 results, while the benchmark comparison covers all included models and retrieval heads.Source: §C.2 · pp. 44, 46, Table 22 · p. 47

C.3. Matryoshka results

  • Matryoshka training evaluates NeoMME-260M and NeoMME-800M at multiple representation widths on ViDoRe v1, v2, and v3 using task-macro nDCG@10.The results report retrieval quality at every trained width for both model sizes.Source: Table 24 · p. 48

C.4. Dual-head training ablation

  • Joint training increases late-interaction nDCG@10 by 1.38 points on ViDoRe v3 but changes dense retrieval by only 0.01 point.Across 14,514 judged queries, the paired late-interaction gain is 1.39 points with a 95% confidence interval of 1.11–1.67 points and p=10^-4, whereas the dense change is not significant.Source: §C.4 · p. 48, Table 25 · p. 48
  • On BEIR-15, joint training raises late-interaction nDCG@10 by 1.07 points but reduces dense retrieval by 1.85 points, indicating one-directional rather than general transfer.Each ablation run uses one training seed, so the paired intervals do not measure run-to-run variation.Source: §C.4 · pp. 48–49

D.1. Efficiency measurement details

  • Efficiency measurements time transfer, encoding, unpadding, float32 conversion, and serialization while excluding image processing, decoding, storage I/O, compression, and index construction.Benchmarks use deterministic RGB squares, calibrated batch sizes, and square inputs of 768, 1,024, 1,536, and 2,048 pixels; H100 and L40S use 128 images, while M5 Pro uses 16.Source: §D.1 · p. 49, Table 26 · p. 50, Table 27 · p. 50
  • Throughput values are median pages per second, and comparisons are restricted to models measured on the same device because execution environments differ.CUDA uses bfloat16 and torch.compile; M5 Pro includes eager-execution cases where compilation could not generate a model graph, and unsupported hardware targets are marked with dashes.Source: §D.1 · pp. 49–50

D.2. Query-encoding latency · D.3. Late-Interaction Kernels (LIK)

NeoMME delivers low batch-one query-encoding latency across CPU, CUDA, and MPS devices, while its late-interaction kernels compute exact MaxSim scores without materializing the full similarity tensor. The LIK evaluation focuses on memory and short-run step time under specified hardware and training configurations, not convergence or retrieval quality.

D.2. Query-encoding latency

  • Batch-one query latency uses 649 NanoBEIR queries, one warmup pass, three measured passes, and timings for tokenization, forward computation, and host-memory transfer.CPU tests use 128 cores, 1 TB RAM, OMP_NUM_THREADS=128, and float32; L40S and M5 Pro tests use bfloat16.Source: §D.2 · p. 50
  • Query encoding excludes candidate generation, approximate search, MeanMaxSim scoring, and end-to-end request latency.Thus, the reported measurements isolate query encoding rather than complete retrieval serving.Source: §D.2 · p. 50
  • 21.0 ms on L40S, 78.3 ms on CPU, and 15.9 ms on M5 Pro make NeoMME-260M second only to ColModernVBERT on every device.Other measured visual-language-model retrievers range from 30.6–52.5 ms, 113.8–899.1 ms, and 20.1–129.1 ms, respectively.Source: §D.2 · p. 50, Figure 17 · p. 51
  • NeoMME-800M is faster than the larger visual-language-model retrievers on every device, yielding low query-encoding latency across CUDA, MPS, and CPU paths.The benchmark reports device-specific results because the execution environments differ.Source: §D.2 · pp. 50–51

D.3. Late-Interaction Kernels (LIK)

  • LIK computes exact MaxSim scores in tiles while retaining running token maxima, avoiding writes of the full query-token-by-document-token similarity tensor to high-bandwidth memory.During training, it stores winning document-token indices for gradient routing.Source: §D.3 · p. 51
  • LIK measurements use one H100 80 GB SXM with float32 accumulation and bfloat16 training inputs, covering memory and short-run step time rather than convergence or downstream retrieval quality.Long-document tests use float16 inputs with 16 queries, 32 documents, and 32 query tokens; training comparisons use specified ColQwen2 and PyLate configurations.Source: §D.3 · p. 51

E. Retrieval case demonstrations

Three demonstrations show NeoMME-Retriever 260M ranking pages for multilingual queries across three ViDoRe v3 task corpora. Each case reports MeanMaxSim scores and page-level relevance judgments.

Case overview

  • The demonstrations span HR, industrial, and pharmaceutical ViDoRe v3 tasks, using NeoMME-Retriever 260M late-interaction rankings with MeanMaxSim and relevance labels.Pages are labeled fully relevant, critically relevant, or unjudged; green and red frames distinguish relevant and unjudged pages.Source: E. Retrieval case demonstrations · p. 52

ViDoRe v3 retrieval case 1

  • For a French PES-network anniversary query, the HR corpus results have MeanMaxSim scores 0.7068, 0.7205, and 0.6989, all critically relevant.The corpus contains 1,110 pages.Source: E. Retrieval case demonstrations · p. 53

ViDoRe v3 retrieval case 2

  • For a plastic-panel paint-restoration query, the industrial corpus results score 0.7275, 0.8502, and 0.7146, with one fully relevant and two unjudged pages.The corpus contains 5,244 pages.Source: E. Retrieval case demonstrations · p. 53

ViDoRe v3 retrieval case 3

  • For an ISMP-versus-FDA error-reporting query, the pharmaceutical corpus results score 0.7804, 0.7850, and 0.7572, with one critically relevant and two unjudged pages.The corpus contains 2,313 pages.Source: E. Retrieval case demonstrations · p. 54

Related papers

Loading 2609.01657v1…