PaperlayerView exact revision on arXiv

Source-linked AI summary

NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference

Aurélien Lac, Tony Wu

arXiv:2609.01657v1cs.IRcs.AIcs.CV

TL;DR

NeoMME addresses limitations of multimodal encoders built by repurposing separate image and text encoders. It uses a single long-context bidirectional Transformer backbone for multilingual text and raw image patches, achieving strong visual-document retrieval results with faster encoding and highly compressed embeddings.

  • Problem

    NeoMME addresses common limitations of multimodal encoders built by repurposing separate image and text encoders.

  • Method

    NeoMME maps multilingual text tokens and raw image patches into a shared hidden space for joint processing in one long-context bidirectional Transformer, with a shared masked-diffusion denoising objective.

  • Results

    NeoMME-Retriever is Pareto-optimal on ViDoRe v1, v2, and v3, while NeoMME-260M achieves about 2× ColModernVBERT’s throughput and 255× embedding compression with over 95% retrieval quality retained.

  • Takeaways & Limitations

    The released checkpoints and Hugging Face Transformers implementation support building efficient multimodal and multilingual representation models.

  • Takeaways & Limitations

    NeoMME has no objective that predicts or reconstructs image content, and matched image-objective studies are needed to separate objective effects from data mixture and architecture.

Abstract

from arXiv · show

Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal language models. Visual document retrievers such as ColPali repurpose these models as encoders, carrying over the parameter and compute overhead of a VLM for a non-generative task. We introduce NeoMME, a family of 260M and 800M-parameter Multimodal and Multilingual bidirectional Encoders that process multilingual text and raw image patches in a single bidirectional Transformer encoder. Both models are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both support a 16,384-token context, enough to encode up to two standard 4K UHD images. To demonstrate its downstream capabilities, we fine-tune NeoMME with jointly trained dense and late-interaction heads. On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress late-interaction multimodal document embeddings by 255x while preserving over 95% of baseline nDCG@10. We contribute NeoMME to Hugging Face Transformers and release the pretrained backbone and retrieval-compatible checkpoints under Apache 2.0 at https://hf.co/collections/Hcompany/neomme.

1. Introduction

NeoMME introduces multimodal, multilingual bidirectional encoders that share one Transformer across text and raw image patches, avoiding separate vision towers and causal decoders. Fine-tuned NeoMME-Retriever models combine dense and late-interaction retrieval with strong benchmark, throughput, and compression results.

  • Motivation: A shared Transformer gives text and image tokens the same computational path and unifies pretraining, fine-tuning, parallelization, and serving.This contrasts with architectures using separate modality towers or a visual encoder followed by a causal language model.
  • Introduction: NeoMME trains 260M and 800M bidirectional single-tower encoders from scratch on multilingual text and raw image patches.The models use modality-specific input projections before sharing every Transformer layer.
  • Pretraining: NeoMME uses masked discrete-diffusion text denoising for text-only examples and image-conditioned denoising for multimodal examples.The objective retains visible image patches while predicting corrupted text.
  • Downstream retrieval: NeoMME-Retriever produces both dense and late-interaction representations from one backbone forward pass for deployment flexibility.The retrieval heads are jointly trained and support dense and multi-vector retrieval through Sentence Transformers.
  • Results: 0.523 nDCG@10 for NeoMME-Retriever 260M and 0.556 for 800M are reported on ViDoRe v3.The 260M model outperforms all evaluated models strictly below 800M parameters.
  • Efficiency: 255× compression reduces NeoMME-260M embeddings from approximately 1.5 MB to 6 kB while retaining more than 95% of original retrieval quality.The result combines hierarchical token pooling at factor 8 with asymmetric int8-query and binary-document quantization.

2. Related work

Related work spans bidirectional encoders, diffusion objectives, shared multimodal Transformers, visual document retrieval, and dense versus late-interaction representations. NeoMME combines these strands by pairing raw patches and shared bidirectional processing with masked-text diffusion and retrieval heads.

  • Encoders and adaptations: Bidirectional encoders support transferable text representations, while decoder models can be adapted into encoders through bidirectional and masked or contrastive training.The related work contrasts native masked encoders with causal-decoder adaptations.
  • Masked diffusion: Masked diffusion extends masked-language modeling by sampling corruption across a noise trajectory rather than using one fixed corruption rate.Prior work applies discrete diffusion to BERT-style denoising, generation, and retrieval embeddings.
  • Shared multimodal architectures: Shared multimodal Transformers process image and text through common layers, but prior systems differ in initialization, image representation, modality sharing, and generation versus encoding goals.Examples include ViLT, OneR, M3AE, Uni-Perceiver, and tower-free vision-language models.
  • Positioning: NeoMME combines continuous raw patches, a vision-tower-free shared bidirectional Transformer, image-conditioned masked-text diffusion, and dense plus late-interaction retrieval.The cited related-work review identifies this combination as absent among the systems it surveys.
  • Visual document retrieval: Visual document retrieval can preserve layout, tables, figures, and other evidence lost by text extraction by embedding document pages directly as images.DSE uses dense page embeddings, while ColPali retains image-patch granularity with late interaction.
  • Retrieval representations: Late interaction offers greater expressive capacity than dense retrieval but increases storage and scoring costs because representations retain token-level vectors.The literature addresses these costs with pruning, pooling, representation transforms, and fused MaxSim kernels.

3. Architecture

NeoMME’s architecture maps multilingual text and RGB image patches into one long-context bidirectional Transformer with dynamic-resolution processing and efficient attention. Its tokenizer and backbone design target compact multilingual multimodal representations, while the reported tokenization analysis exposes weaker coverage outside the original target languages.

  • Architecture: NeoMME uses modality-specific input layers to map text tokens and RGB image patches into a shared hidden space processed jointly by one long-context bidirectional Transformer.The architecture is optimized for multimodal inputs without a separate vision tower.
  • Tokenizer: 44.4% fewer tokens than ModernBERT are emitted over 14 FLORES-200 target languages, but coverage is weaker across the full 204-language set.The comparisons also report 39.4% fewer than LFM2.5-Encoder-230M, 6.3% fewer than mmBERT-base, and 16.9% fewer than EuroBERT-210m.
  • Image patches: 32 × 32 RGB patches contain 3,072 values and are projected through layer normalization and a 2-layer MLP trained from scratch.No patch-merging module or pretrained vision encoder is used.
  • Image patches: Dynamic resolution samples a longest-side cap between 1,024 and 2,048 pixels, preserves aspect ratio, and varies patch count with image dimensions.Images are downsampled only when they exceed the sampled cap.
  • Long-context attention: 16,384-token context supports up to two standard 3,840 × 2,160 4K UHD images using mostly symmetric sliding-window attention with periodic global layers.The sliding-window half-size alternates between 256 and 1,024 tokens, while every sixth and final layer use global attention.
  • Modern bidirectional backbone: The backbone uses grouped-query attention, independently RMS-normalized queries and keys, squared-ReLU MLPs, sigmoid attention gates, and residual branch scaling.Each residual branch is multiplied by (2L)^-1/2, while NeoMME does not claim the complete Depth-μP parameterization.

4. Pretraining

NeoMME pretraining combines text-only and multimodal data in long packed sequences, using masked discrete diffusion with visible image patches to encourage image-grounded text recovery. The training pipeline and sensitivity probes show that image use increases as textual context is removed.

  • Pretraining setup: NeoMME is pretrained from random initialization on text-only and multimodal examples, including document and natural images paired with text.The shared backbone is trained with a masked discrete-diffusion denoising objective, optionally conditioned on visible image patches.
  • Pretraining setup: 55% of packed input tokens come from the text-only stream and 45% from the multimodal stream, with a planned total of about 524B packed input tokens.The streams cover web and PDF text, synthetic educational material, encyclopedic text, mathematics, question answering, code, OCR, documents, and natural images.
  • Training system: Packed 16,384-position streams, background image patchification, prefetched batches, and fused training kernels keep GPU utilization high during steady-state training.The runs process 1,048,576 packed input tokens per global step for 500,000 steps.
  • Text masking and image sensitivity: Multimodal segments sample corruption rates from U(0.30, 1), while text-only segments use U(0, 1), reducing reliance on visible text during image-conditioned prediction.Eligible text positions are independently replaced by mask tokens; image patches and structural markers are excluded.
  • Text masking and image sensitivity: Both models show positive mean image gain at every tested corruption rate, reaching 38.4 percentage points for NeoMME-260M and 40.5 percentage points for NeoMME-800M at 90% masking.Image gain measures the improvement in masked-token recovery when visible page patches are supplied instead of zero tensors.
  • Training dynamics: The NeoMME-260M loss enters a lower regime around step 36,000, while NeoMME-800M shows a visible change around step 25,000; image gain rises from near zero early in training.The final loss decline coincides with scheduled WSD decay to 1% of the peak learning rate.

5. Retrieval

NeoMME-Retriever uses one backbone to produce dense and late-interaction representations for visual-document and text retrieval. Its models offer strong retrieval quality across scales, faster page encoding, and substantial storage reductions through pooling and quantization, with interpretability visualizations requiring caution.

  • Retrieval architecture: A single backbone pass produces both dense and late-interaction representations, enabling direct late-interaction retrieval or dense ANN retrieval followed by late-interaction reranking.The dense head adds only mean pooling and normalization beyond the late-interaction computation.
  • Visual-document retrieval: NeoMME-260M is the strongest model below 300M parameters and improves nDCG by 26.1 points on ViDoRe v3 versus the best other model below 300M.Its ViDoRe v3 score is within 0.2 points of 3.75B-parameter ColQwen2.5 while using 14.4× fewer parameters.
  • Visual-document retrieval: 0.556 nDCG@10 is achieved by NeoMME-800M on ViDoRe v3, 3.2 points above 3.75B-parameter ColQwen2.5 and within 0.9 point of Vultron Flash.Scaling from 260M to 800M adds 3.3 points on ViDoRe v3, 3.7 points on v2, and 1.5 points on v1.
  • Text retrieval: Late-interaction exceeds dense retrieval by 18.3 points for NeoMME-260M and 14.4 points for NeoMME-800M on text retrieval.NeoMME-800M reaches 0.513 with late-interaction and 0.369 with dense retrieval on BEIR-15.
  • Resolution trade-offs: Higher image-resolution caps improve retrieval quality but increase encoder FLOPs, late-interaction storage, and indexing cost.Reducing the cap from 2048 to 1536 pixels lowers MeanMaxSim nDCG@10 by at most 1.8% while a square page uses about 57% as many vectors and raw float32 bytes.
  • Compression and storage: 255.5× storage compression retains 95.19% of baseline nDCG@10 with pool factor 8, int8 queries, and binary documents.Pool factor 10 with int8 queries and documents achieves 39.4× compression while retaining 99.16% of baseline nDCG@10.
  • Efficiency: 1.97× ColModernVBERT throughput is reached by NeoMME-260M at 2048×2048, encoding 51.3 pages per second on an NVIDIA L40S.NeoMME-800M reaches 21.2 pages per second at 2048×2048 for retrieval-quality results.
  • Interpretability: Similarity maps provide only qualitative diagnostics because MeanMaxSim also uses nonspatial structural positions and background responses may reflect register-like tokens.The visualization therefore does not fully explain the retrieval score.

6. Other downstream tasks

NeoMME-260M is evaluated as a shared-backbone initialization across language, document, and natural-image tasks, measuring adaptability rather than specialization. It performs near a dedicated text encoder on the 17-task suite while showing both strengths and weaknesses across individual tasks.

  • The evaluation tests whether one compact shared backbone supports task-specific text, document-image, and multimodal fine-tuning rather than outperforming modality-specialized models.
  • The 17-task suite comprises eight GLUE tasks, four SuperGLUE tasks, and five multilingual tasks.
  • NeoMME-260M reaches a 17-task mean of 75.3, compared with 79.3 reported for the dedicated LFM2.5-Encoder-230M text model.The comparison follows closely aligned task definitions and reporting seeds, but is not a same-harness reproduction.
  • NeoMME exceeds the peer on PAWS-X, MASSIVE Intent, SeaHorse, MRPC, and WSC, but reaches only 46.4 Matthews correlation on CoLA.
  • The same backbone reaches 85.5±0.2 accuracy on LEDGAR with 8K context and 89.3 entity F1 on a downstream task.

7. Limitations

The authors identify limitations involving pretraining scale and objectives, retrieval supervision, evaluation coverage, data availability and language coverage, bias and safety, and missing ablations. These boundaries leave several factors for future matched studies to isolate.

  • NeoMME processes about 524B packed input tokens, including about 290B text-only tokens, while ModernBERT uses about 2T text tokens.The paper notes that accounting methods differ and estimates ModernBERT sees roughly seven times more text-only data.
  • NeoMME has no direct image-content objective, predicting masked text while image patches remain visible; matched image-objective studies are proposed.The authors connect this choice to weak frozen natural-image results, while noting that objective, data mixture, and architecture could interact.
  • Retrieval training uses about 430K pure-text and 850K image-query examples, versus mLateOn’s roughly 660M contrastive and 16M hard-negative examples.Larger matched training sets could test how much performance differences reflect supervision scale rather than architecture, objectives, or evaluation code.
  • NeoMME-Retriever was not evaluated on mixed-modality corpora despite training text chunks and page images into one embedding space.The authors propose evaluation on UniDoc-Bench and MixBench and extension to multilingual page-image corpora.
  • Visual retrieval lacks a large multilingual dataset covering broad document types and layouts, and its data has narrower language coverage than text retrieval.The visual stream covers English, French, German, Italian, and Spanish, with generated queries adding Portuguese; synthetic generation is proposed for broader coverage.
  • The study does not isolate major architecture, scaling, data, and training choices through matched ablations.The authors also note possible social, geographic, topical, and language biases in web and machine-generated data, plus potential harmful, private, or biased sampled text.

8. Conclusion

NeoMME is a shared, long-context bidirectional encoder for multilingual text and raw image patches, trained from scratch and adapted for retrieval. Its retriever combines competitive retrieval quality, higher throughput, and substantial embedding compression, while released checkpoints and tooling support further use.

  • NeoMME maps multilingual text tokens and raw 32 × 32 image patches into one long-context bidirectional Transformer encoder trained from scratch.Dynamic resolution preserves image aspect ratios, and the shared masked-diffusion objective supports text-only and image-conditioned denoising.
  • NeoMME-Retriever produces both dense and late-interaction representations in one forward pass for visual-document and text retrieval.
  • The 260M model outperforms all evaluated models below 800M parameters and encodes matched 2048×2048 pages at about twice ColModernVBERT’s throughput.
  • The authors release NeoMME checkpoints and a day-zero Hugging Face Transformers implementation for building efficient multimodal and multilingual representation models.

A.4. Training mixture details

Retrieval training uses separate text-only and multimodal streams, combining multilingual and code-search sources with page-image datasets. Query augmentation, hard-negative mining, and memory-efficient kernels support the training pipeline.

  • Text-only stream: The text-only retrieval stream contains 1,581,039 queries across English, multilingual, code, organic, and translated mLateOn pools.
  • Multimodal stream: The multimodal stream combines ColPali, multilingual document-image, and VisRAG datasets, with each source assigned a configured weight of 0.25.The original pairs cover English, French, German, Italian, and Spanish.
  • Multimodal stream: Query augmentation adds Portuguese, bringing the retrieval data to six languages.
  • Query augmentation: The pipeline generates two queries for each of 100,000 sampled pages, using six query types and question, instruction, or keyword formats.
  • Hard-negative sampling: Visual hard-negative mining first trains with positive and in-batch negatives, then mines a fixed 32-candidate window before a second fine-tuning stage.
  • Memory-efficient training: GradCache processes gathered candidates in smaller activation chunks, while LIK fuses dot products with online maximum reduction to reduce MaxSim memory use.

B.1. Tokenizer evaluation

NeoMME’s tokenizer is evaluated for multilingual compression across FLORES-200 languages and document, math, and code sources. The evaluation also compares tokenization behavior with other encoders under matched-text protocols.

  • Evaluation protocol: All tokenizers encoded identical texts without added special tokens, making token-count comparisons consistent across languages.The language evaluation used all 1,012 aligned sentences per language in FLORES-200 devtest.
  • Evaluation protocol: The document evaluation used 2,000 documents per source, capped at 4,000 characters, while the tokenizer covered document, math, and code sources.The table reports tokens per UTF-8 byte, where lower values are better.
  • Multilingual compression: 65.29% more tokens than mmBERT and 15.28% more than EuroBERT were emitted across all 204 FLORES-200 language-script pairs.The largest deficits occurred in Tibetan, South Asian, and Southeast Asian scripts.
  • Qualitative examples: NeoMME’s selected image-conditioned generations are qualitative examples rather than task-level captioning or optical character recognition measurements.The examples use NeoMME-260M3 pretraining generations.

C.2. Complete retrieval results

NeoMME-Retriever is evaluated across visual-document retrieval, representation widths, throughput, and joint retrieval-head objectives. The results report strong ViDoRe v3 performance alongside efficiency measurements and an ablation showing asymmetric effects from joint training.

  • Complete retrieval results: 0.5560 nDCG@10 was achieved by NeoMME-800M on ViDoRe v3.The benchmark reports task-level nDCG@10 results by domain.
  • Retrieval heads: NeoMME-Retriever jointly trains dense and late-interaction heads on one shared backbone for deployment flexibility.The same backbone forward pass produces both a pooled dense representation and a late-interaction multi-vector representation.
  • Retrieval-head ablation: 1.38 points of ViDoRe v3 late-interaction task-macro nDCG@10 improvement came from joint training, while dense retrieval changed by 0.01 point.The paired analysis estimated a 1.39-point late-interaction gain with a 95% confidence interval of 1.11 to 1.67 points.
  • Retrieval-head ablation: 1.07 points of late-interaction improvement and 1.85 points of dense-retrieval reduction occurred on BEIR-15 under joint training.The results are consistent with one-directional transfer on ViDoRe v3 rather than general improvement to both heads.
  • Evaluation caveat: Each dual-head ablation run used one training seed, so its paired intervals do not measure run-to-run variation.The runs otherwise shared the same data seed, query order, negative draws, data mixture, schedule, and evaluation protocols.

D.2. Query-encoding latency

The query-latency evaluation measures batch-one encoding across CPU, CUDA, and MPS hardware while excluding downstream retrieval operations. NeoMME-260M is consistently near the fastest evaluated visual retriever, and NeoMME-800M is faster than larger visual-language retrievers.

  • Measurement protocol: The benchmark measured tokenization or query processing, model execution, and output transfer for 649 fixed NanoBEIR queries.Each row used one warmup pass followed by three measured passes.
  • Measurement scope: Candidate generation, approximate search, MeanMaxSim scoring, and end-to-end request latency were excluded from the query-encoding measurements.The reported values therefore characterize query encoding only.
  • Measured latency: 21.0 ms, 78.3 ms, and 15.9 ms were NeoMME-260M’s mean query-encoding latencies on the L40S, CPU host, and M5 Pro, respectively.NeoMME-260M ranked second only to ColModernVBERT on every device.
  • Cross-device comparison: NeoMME-800M was faster than the larger visual language model retrievers on every tested device.The comparison covered CUDA, MPS, and CPU execution paths.

E. Retrieval case demonstrations

The retrieval demonstrations show NeoMME-Retriever 260M ranking pages for multilingual and domain-specific ViDoRe v3 queries. The examples pair each retrieved page’s MeanMaxSim score with its graded relevance status.

  • Demonstration setup: Three ViDoRe v3 cases show NeoMME-Retriever 260M late-interaction rankings from the full task corpora.The cases cover industrial, pharmaceutical, and HR-related document collections.
  • Relevance labels: ViDoRe v3 labels pages as fully relevant, critically relevant, or unjudged based on whether they answer the query independently or provide needed facts.Green frames indicate relevant pages and red frames indicate unjudged pages.
  • HR example: 0.7205 was the highest displayed score in the HR example, where the retrieved pages were critically relevant.The query asks when the PES network’s activities marked its tenth anniversary and through which segment it was broadcast.
  • Industrial example: 0.8502 was the highest displayed MeanMaxSim score in the industrial restoration example, whose page was fully relevant.The other displayed industrial pages were unjudged, with scores of 0.7275 and 0.7146.
  • Pharmaceutical example: 0.7572 was assigned to the critically relevant page in the pharmaceutical example, alongside unjudged pages scoring 0.7804 and 0.7850.The example concerns differences between ISMP and FDA error reporting.
Loading 2609.01657v1…