PaperlayerView exact revision on arXiv

Source-linked AI summary

MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image

Alan Arazi, Eilam Shapira, Shoham Grunblat, Mor Ventura, Elad Hoffer, Gioia Blayer, David Holzmüller, Lennart Purucker, Gaël Varoquaux, Frank Hutter, Roi Reichart

arXiv:2605.10616v1cs.LGcs.CLcs.CV

TL;DR

Multimodal tabular models often rely on frozen embeddings that may discard task-specific information from text and images. MulTaBench introduces 40 curated image-tabular and text-tabular datasets to evaluate target-aware representation tuning, which outperforms frozen embeddings across diverse learners, encoder scales, and embedding dimensions.

  • Problem

    Existing multimodal tabular models lack joint target-aware tuning for unstructured representations, while generic embeddings may discard fine-grained signals needed for prediction.

  • Method

    MulTaBench curates 40 balanced image-tabular and text-tabular datasets using criteria for complementary modality signals and task-aware representations, evaluated across tabular learners.

  • Results

    Target-aware representations outperform frozen embeddings across established benchmarks, with gains generalizing across modalities, tabular learners, encoder scales, and embedding dimensions.

  • Takeaways & Limitations

    MulTaBench provides a benchmark for evaluating multimodal tabular architectures that target-aware representations for unstructured modalities.

  • Takeaways & Limitations

    The curation pipeline entangles dataset selection with the algorithmic solution, making eligibility difficult to predict and preventing fair evaluation of the curation models because of selection bias.

Abstract

from arXiv · show

Tabular Foundation Models have recently established the state of the art in supervised tabular learning, by leveraging pretraining to learn generalizable representations of numerical and categorical structured data. However, they lack native support for unstructured modalities such as text and image, and rely on frozen, pretrained embeddings to process them. On established Multimodal Tabular Learning benchmarks, we show that tuning the embeddings to the task improves performance. Existing benchmarks, however, often focus on the mere co-occurrence of modalities; this leads to high variance across datasets and masks the benefits of task-specific tuning. To address this gap, we introduce MulTaBench, a benchmark of 40 datasets, split equally between image-tabular and text-tabular tasks. We focus on predictive tasks where the modalities provide complementary predictive signal, and where generic embeddings lose critical information, necessitating Target-Aware Representations that are aligned with the task. Our experimental results demonstrate that the gains from target-aware representation tuning generalize across both text and image modalities, several tabular learners, encoder scales, and embedding dimensions. MulTaBench constitutes the largest image-tabular benchmarking effort to date, spanning high-impact domains such as healthcare and e-commerce. It is designed to enable the research of novel architectures which incorporate joint modeling and target-aware representations, paving the way for the development of novel Multimodal Tabular Foundation Models.

1 Introduction

The introduction identifies a gap in multimodal tabular learning: tabular foundation models lack native support for text and images, while generic embeddings may discard task-critical information. It presents MulTaBench, a 40-dataset benchmark curated to test complementary multimodal signal and the value of target-aware representations.

  • Motivation: Tabular Foundation Models have become state-of-the-art for supervised tabular learning but are trained exclusively on structured data.They have surpassed historically leading gradient-boosted decision trees.
  • Motivation: Existing tabular models lack inherent visual support, while multimodal extensions can compromise tabular performance and LLMs/VLMs do not fit tabular relational structure.These limitations motivate methods designed specifically for multimodal tabular learning.
  • Task-aware representations: Generic embeddings are lossy summaries optimized for broad semantic content, so they may omit details needed for task-specific prediction.The introduction argues that representations should be aligned with the prediction objective rather than treated as universally sufficient.
  • Findings: Target-aware representations outperform frozen embeddings across established multimodal tabular benchmarks, with gains varying substantially by dataset.The curation pipeline requires joint prediction to outperform unimodal baselines and target-aware representations to improve on frozen embeddings.
  • MulTaBench: MulTaBench includes 40 datasets balanced between image-tabular and text-tabular tasks, explicitly prioritizing datasets requiring task-aware representations.The benchmark also spans classification and regression objectives, varied sample sizes and feature counts, and diverse domains.
  • Findings: Target-aware tuning generalizes across independent tabular learners, encoder scales, and embedding dimensions.The authors position MulTaBench as a resource for developing architectures that contextualize unstructured-modality representations.

2 Related Work · 3 Benchmarking Multimodal Tabular Learning

The benchmark defines multimodal tabular learning as prediction from structured data paired with text or images, then curates tasks requiring both joint multimodal signal and task-aware representations. Its protocol evaluates four feature-and-representation conditions and filters datasets using learner-based performance criteria.

  • 3 Benchmarking Multimodal Tabular Learning: MMTL combines numerical and categorical columns with an unstructured modality such as text or image for prediction.The analysis assumes one unstructured modality paired with tabular data, while noting that the logic extends to trimodal datasets.
  • 3.1 Desiderata for Multimodal Tabular Learning datasets: Challenging MMTL datasets must exhibit joint signal and task-awareness.These properties determine whether a seemingly natural dataset, such as pneumonia diagnosis from clinical metadata and notes or X-rays, is genuinely challenging.
  • 3.1 Desiderata for Multimodal Tabular Learning datasets: Joint signal requires each modality to provide independent target information, so joint performance exceeds both unimodal alternatives.The criterion may also capture cross-modal interactions in which one modality becomes discriminative only once combined with another.
  • 3.1 Desiderata for Multimodal Tabular Learning datasets: Task-awareness means the optimal unstructured-modality representation depends on task context rather than only global semantics.Generic encoders may discard exact wording or fine-grained spatial textures, motivating target-aware representations that preserve target-relevant details.
  • 3.1 Desiderata for Multimodal Tabular Learning datasets: Task-aware representations are unnecessary when the predictive signal is coarse enough for task-agnostic embeddings to capture.For example, categorizing scan type may not benefit from preserving pathology-specific visual patterns.
  • 3.2 The Curation Pipeline: 4 experimental conditions vary the included features and unstructured-modality representation to connect theoretical desiderata with empirical curation.The conditions are summarized in Table 1 and Figure 1.
  • 3.2 The Curation Pipeline: Embeddings use e5-v2-small for text and DINO-v3-small for images, while TAR finetunes the last 3 layers on the prediction target using LoRA.The encoders were selected for high performance-to-parameter efficiency.
  • 3.2 The Curation Pipeline: A dataset passes the Joint Signal filter when, across at least 3 out of 5 learners, Joint Frozen outperforms both unimodal structured and unimodal unstructured variants.This ensures the unstructured modality is relevant without reducing the task to a purely language- or vision-based problem.

4 MulTaBench

MulTaBench curates text-tabular and image-tabular datasets using joint-signal and task-awareness criteria, producing balanced benchmark subsets. The curation retains 20 datasets per modality and shows that target-aware representations consistently outperform frozen embeddings on text-tabular tasks.

  • Benchmark Composition: Datasets range from 400 to 114,000 rows and from 1 to 245 structured features across domains including medical and e-commerce.The benchmark is uploaded to Kaggle with a unified API linking tables and images.
  • Text-Tabular Curation: 56 unique text-tabular datasets are aggregated, normalized across classification and regression, and evaluated with 95% confidence intervals.AUC and R2 scores are min-max scaled to [0, 1] and averaged across datasets.
  • Text-Tabular Curation: Target-Aware Representations consistently outperform frozen representations across the text-tabular benchmark datasets.Figure 3 compares normalized scores for Joint TAR and Joint Frozen across all datasets and the MulTaBench subset.
  • Curation Criteria: Approximately 23% of datasets fail the Joint Signal criterion, 36% of the remainder fail Task-awareness, and 41% pass both.The passing datasets are subsampled to 20 to match the size of the image-tabular subset.
  • Image-Tabular Curation: Only 5 of 16 valid image-tabular datasets from existing literature meet the criteria, a 31% acceptance rate.Additional qualifying datasets are manually curated from Kaggle, producing 20 image-tabular datasets.

5 Robustness Analysis

The robustness analysis shows that Target-Aware Representations (TAR) retain their benefits across tabular learners, embedding scales, and PCA dimensions. Qualitative attention analysis further examines how TAR changes image-encoder focus across MulTaBench datasets.

  • New Tabular Learners: Target-aware embeddings consistently outperform frozen embeddings across new tabular learners and both text-tabular and image-tabular modalities.The analysis notes that GBDTs exhibit the most substantial gains, while extending improvements beyond the curation models supports the robustness of the benchmark criteria.
  • Embedding Model Scale: TAR Small can outperform Frozen Large, showing that greater representational capacity does not guarantee retention of target-relevant signals.The result indicates that representation tuning remains necessary even when the embedding model is larger.
  • Embedding Model Scale: TAR variants outperform frozen variants for both small and large embedding models.The large variants use approximately 10 times more parameters and a final dimension of 1024, compared with 384-dimensional small embeddings.
  • Embedding Dimension: TAR variants remain stronger than Frozen variants with 15, 30, and 60 PCA components.The analysis tests whether TAR gains are artifacts of the standard 30-component compression and finds tuning effective across embedding dimensions.
  • Qualitative Analysis: Attention-map analysis examines how target-aware adaptation reshapes DINO-v3’s focus across four image datasets.For CheXpert and Glaucoma, the reported attention shifts from arbitrary anatomical borders toward the right lower lung.

6 Towards Multimodal Tabular Foundation Models

MulTaBench exposes a gap between current tabular learners and multimodal tabular learning demands because existing architectures cannot jointly tune unstructured representations for target labels. The paper frames future multimodal tabular foundation models around combining target-aware contextualization with the strengths of existing paradigms.

  • Current architectures cannot jointly tune unstructured representations for target labels, revealing a significant gap between tabular learners and multimodal tabular learning demands.
  • The proposed foundation-model vision retains four desiderata: mixed-type columns, cross-dataset modeling, textual context and metadata, and equivariance to column order.The four desiderata are labeled D1–D4 in the paper’s framework.
  • PFNs support structured learning but struggle to unlock target-aware representations with multimodal encoders while preserving in-context learning without parameter updates.
  • Joint modeling approaches such as AutoGluon-Multimodal and TabSTAR achieve target-awareness through finetuning, contrasting with PFN approaches that avoid parameter updates.
  • MulTaBench isolates datasets demanding task-specific representations and supports developing architectures that combine target-aware contextualization with the complementary strengths of current paradigms.

7 Discussion and Conclusion · A Curation Pipeline · A.1 Target-Aware Representations

MulTaBench introduces a 40-dataset benchmark for challenging image-tabular and text-tabular learning, showing the importance of task-aligned representations while identifying curation limitations and future directions. Its Target-Aware Representations use task-specific encoder adaptation with fixed training procedures for multimodal tabular learning.

  • 7 Discussion and Conclusion: MulTaBench contains 40 image-tabular and text-tabular datasets targeting challenging tasks that benefit from Joint Modeling and TAR.The authors describe it as the largest image-tabular benchmark to date.
  • 7 Discussion and Conclusion: The curation pipeline entangles the computational problem with the algorithmic solution, creating advance-prediction difficulty and selection bias.Consequently, curation models cannot be fairly evaluated, although the authors view the benchmark as an important step.
  • 7 Discussion and Conclusion: Future work includes text-image-tabular benchmarks, additional modalities such as audio and video, and prompting strategies that steer embeddings toward targets.MulTaBench is intended to support Multimodal Tabular Foundation Models, with architecture and training data identified as major challenges.
  • A.1 Target-Aware Representations: Target-Aware Representations finetune the encoder’s top 3 transformer layers with LoRA and map its 384-dim output to output classes.Finetuning is preprocessing performed independently of structured features and the downstream tabular learner, using only the training split.
  • A.1 Target-Aware Representations: Both encoders use LoRA with r = 16, α = 32, dropout 0.1, fixed across datasets, while learning rates are 10^-4 for e5 and 0.001 for DINO.Training uses AdamW, batch size 256, weight decay 0.01, early stopping after 3 unimproved validation-loss epochs, and up to 100 DINO or 50 e5 epochs.
  • A.1 Target-Aware Representations: For regression, continuous targets are discretized into 20 equal-frequency bins and optimized with cross-entropy, which is more stable than direct regression finetuning.The authors attribute the stability to reduced sensitivity to outliers, while noting the choice could be optimized further.
  • A.1 Target-Aware Representations: For text datasets, one e5 model is jointly finetuned across multiple string columns by treating each row-column pair as a labeled training example.Each example uses the format “col_name : col_val” paired with the row’s target, enabling shared representation learning across text features.

A.2 Curation Experimental Setup · A.3 Formal Acceptance Criteria · B MulTaBench Datasets

MulTaBench curates datasets using five tabular learners across multimodal conditions, then accepts candidates only when joint and task-awareness gains exceed a threshold for a learner consensus. The resulting benchmark reports dataset statistics and high-level descriptions, with exact preprocessing available in released code.

  • A.2 Curation Experimental Setup: Five tabular learners—LightGBM, CatBoost, TabM, TabPFNv2, and TabPFN-2.5—are evaluated over five random seeds and four conditions.Training is capped at 10,000 examples per fold; classification uses AUC and regression uses R2.
  • A.2 Curation Experimental Setup: Models use default configurations, except CatBoost, which uses early_stopping_rounds = 50, od_pval = 0.001, and iterations = 2000.The CatBoost settings follow previous work, while the other learners use their default implementations or parameters.
  • A.3 Formal Acceptance Criteria: For each learner, performance is represented as average AUC or R2 under each evaluated condition.D denotes a candidate dataset, M the pool of five curation learners, and S_m(Condition) the learner’s average predictive performance.
  • A.3 Formal Acceptance Criteria: The Joint gain measures improvement of the joint model over the strongest unimodal baseline, while the Awareness gain measures Joint TAR over Joint Frozen.The unimodal baselines are UnimodalStructured and UnimodalUnstructured.
  • A.3 Formal Acceptance Criteria: A dataset is accepted only when both gains exceed δ for at least a fraction ρ of learners, evaluated jointly per learner.The selection uses δ = 0.001 and ρ = 3/5, so a learner counts only if both gains exceed the threshold.
  • B MulTaBench Datasets: MulTaBench presents dataset statistics, including row counts and feature-type breakdowns, alongside concise high-level descriptions of each dataset.Exact preprocessing logic is provided in the released code.

B.1 Image-Tabular Dataset Descriptions … C.2 Empirical Results for Curation Conditions

MulTaBench spans diverse image-tabular and text-tabular prediction tasks, pairing structured features with complementary visual or textual information across healthcare, commerce, social media, and other domains. Its text-tabular curation deduplicates four benchmark sources into 56 candidates, while empirical results show more consistent condition ordering for the selected subset.

  • B.1 Image-Tabular Dataset Descriptions: Image-tabular datasets cover healthcare, e-commerce, social media, biology, and other domains, with targets spanning classification, diagnosis, ratings, prices, weights, and continuous scores.Examples include mammography, chest X-ray, glaucoma, pet adoption, fashion, movie, artwork, and plankton datasets.
  • B.1 Image-Tabular Dataset Descriptions: The image-tabular collection includes datasets from 400 to 104,072 examples, with structured metadata ranging from lesion morphology and clinical records to product, post, and specimen descriptors.Several medical datasets contain substantial missingness, including over 85% missing pathology labels in CheXpert and over 99% missing fields in Glaucoma SMDG.
  • B.2 Text-Tabular Dataset Descriptions: Text-tabular datasets span jobs, toxicity, crowdfunding, restaurants, products, books, salaries, media, and scientific publishing, using text fields alongside metadata for classification or regression.Targets include fraud, toxicity, funding success, sentiment, genre, readability, prices, salaries, ratings, sales, and impact measures.
  • C Text-Tabular Curation: The text-tabular curation draws candidates from 4 benchmark sources and produces 56 unique datasets after deduplication and exclusion of unavailable datasets.The sources are the Multimodal AutoML Benchmark, Grinsztajn et al., CARTE, and TextTabBench.
  • C Text-Tabular Curation: Each curated candidate is evaluated with 5 tabular learners over 5 folds under 4 experimental conditions, with Structured and Unstructured serving as unimodal baselines.The evaluated conditions include Structured, Unstructured, Joint Frozen, and Joint TAR.
  • C.1 Existing Benchmarks: Deduplication extends Arazi et al.’s procedure to TextTabBench, addressing substantial overlap because the four source benchmarks reuse identical or similar-enough datasets.Table 4 documents datasets shared across more than one existing text-tabular benchmark.
  • C.2 Empirical Results for Curation Conditions: The MulTaBench subset exhibits a consistent ordering across all 4 conditions, more pronounced than in the full 56-candidate corpus.Figure 8 reports normalized scores for Structured, Unstructured, Joint Frozen, and Joint TAR.

C.3 Benchmark Acceptance Breakdown · C.4 Per-Dataset Curation Results · D Image-Tabular Curation

Text-tabular curation retains 20 of 56 candidates after applying joint predictive criteria, with acceptance varying by source benchmark. Per-dataset results show heterogeneous model agreement, including datasets approved by two models and others by only one.

  • C.3 Benchmark Acceptance Breakdown: 23 of 56 candidates pass all criteria, and 20 datasets are retained for MulTaBench.Acceptance rates are highest for Grinsztajn et al. and the AutoML Multimodal Benchmark sources.
  • C.3 Benchmark Acceptance Breakdown: CARTE has the lowest acceptance rate at 33%, reflecting its focus on short knowledge-graph strings and high-cardinality categorical columns.
  • C.4 Per-Dataset Curation Results: Each candidate is evaluated by five curation models against both criteria jointly, with pass, fail, or unavailable outcomes.Unavailable outcomes occur for highly multiclass problems that TabPFN variants cannot run on.
  • C.4 Per-Dataset Curation Results: The curation grid includes LightGBM, CatBoost, TabM, PFNv2, and PFN-2.5, and reports each dataset’s total number of passing models.
  • C.4 Per-Dataset Curation Results: 2 models approve Google Q&A Type, American Eagle Prices, JC Penney Products, Wikiliq Alcohol, Chocolate Bar Ratings, Wine Vivino Spain, California House Prices, SF Permit Applications, and FIFA22 Wages.
  • C.4 Per-Dataset Curation Results: 1 model approves IMDB Genre, Melbourne Airbnb, Bike Price Bikewale, Car Price Cardekho, Polish Wine Prices, ML/DS Job Salaries, and Books Goodreads.The listed per-dataset results illustrate limited agreement across the five curation models.

D.1 Existing Benchmarks · D.2 Curation Logic

Existing image-tabular benchmarks are limited and difficult to reproduce, motivating a curation process that yields 5 qualifying datasets from 16 unique candidates. MulTaBench enforces consistent image handling, target engineering, and reproducible dataset packaging.

  • D.1 Existing Benchmarks: Existing image-tabular benchmarks are substantially more limited than text-tabular benchmarks.MuG reports 8 text-image-tabular datasets corresponding to only 4 underlying datasets, while Tang et al. curate 22 datasets across modality combinations.
  • D.1 Existing Benchmarks: External image links, deleted source datasets, and undocumented preprocessing create serious reproducibility problems.Seattle image URLs are unreachable, KARD references a deleted Kaggle dataset, and remaining candidates are difficult to replicate faithfully.
  • D.1 Existing Benchmarks: 16 unique datasets remained after deduplication and availability filtering, but only 5 passed the curation filter.The authors avoid reporting curation statistics at the same detail level as for text-tabular datasets because some rejected candidates were difficult to assess.
  • D.2 Curation Logic: Curation decisions aim to make the image feature sufficiently important and interesting for a relevant true image-tabular task.The datasets were found in the wild and then evaluated against this curation goal.
  • D.2 Curation Logic: Each dataset contains exactly one image column, and rows with absent or corrupt images are dropped without imputation.Multiple image fields are reduced to one, while placeholder images are excluded because they would inject noise into encoding.
  • D.2 Curation Logic: Target engineering includes log transformation and quantile binning to satisfy curation criteria.Amazon Bestseller retail price uses log(1 + price), while CS:GO Skin Price, PetFinder listed age, and HubMAP HPA donor age use 10, 8, and 10 equal-frequency bins, respectively.
  • D.2 Curation Logic: All 20 image-tabular datasets are preprocessed and uploaded to Kaggle under the MulTaBench organization for reproducible, uniform access.Each upload includes a flat images/ directory, data.csv, relative image paths, and a unified loading API.

E Text-Image-Tabular Datasets · F Extended Results

The paper identifies PetFinder and Amazon Packages as the only text-image-tabular candidates satisfying all criteria for at least three learners. Amazon Packages further shows that target-aware representation tuning over both modalities dominates across models.

  • E Text-Image-Tabular Datasets: Eight of 20 image-tabular datasets also contain text columns, motivating evaluation as potential text-image-tabular datasets.The full text curation pipeline and independent selection test are applied to both image and text modalities.
  • E Text-Image-Tabular Datasets: Only PetFinder and Amazon Packages satisfy all criteria for at least three learners among eight candidates.The other six candidates primarily fail because text TAR does not improve over the frozen joint baseline.
  • E Text-Image-Tabular Datasets: The criterion requiring both modalities’ TAR to improve over the frozen joint baseline may be overly strict.Future benchmarks could require gains from representation tuning in at least one modality instead.
  • E Text-Image-Tabular Datasets: Amazon Packages predicts the total weight of an Amazon bin using a warehouse photograph, product description, and expected item quantity.It is a regression task, with results reported in Table 7.
  • F Extended Results: Table 7 reports mean R2 (%) for structured, image, and text conditions across models.The table denotes the modalities as S=Structured, I=Image, and T=Text.
  • F Extended Results: TAR over both modalities dominates for all models on Amazon Packages.This result supports joint target-aware tuning of the image and text representations in the dataset’s multimodal regression setting.

F.1 Main Results Breakdown … 6. Experimental setting/details

Across the appendix analyses and checklist responses, TAR consistently outperforms Frozen across task types, modalities, learners, encoder sizes, and embedding dimensions, while the paper documents computational costs, limitations, reproducibility details, and asset handling.

  • F.1 Main Results Breakdown: TAR consistently outperforms Frozen for both classification and regression tasks across image and text modalities.The result indicates that target-aware representations are not specific to one task type or modality.
  • F.1 Main Results Breakdown: TAR beats Frozen in the large majority of dataset-fold runs across all evaluated models and both modalities.Win rates are computed over (dataset, fold) pairs with 95% confidence intervals; models without separate TAR conditions are excluded from corresponding columns.
  • F.2 Missing Baselines: The benchmark excludes autoregressive LLMs and VLMs because of prohibitive inference costs and memorization risk, while noting unavailable or inflexible relevant baselines.TIME had not released code, and MultimodalTabPFN was described as insufficiently flexible for serving.
  • F.3 Computation Costs: Text TAR is substantially more expensive: e5-small TAR takes roughly ten times longer than frozen, while e5-large approaches three hours per run.The gap partly reflects text-tabular datasets containing more than one text column, increasing their effective dataset size.
  • F.3 Computation Costs: Experiments use a single fixed LoRA configuration without per-dataset encoder or learner tuning, so reported gains are conservative lower bounds.Hyperparameter optimization would require fold-specific encoder fine-tuning for every trial to prevent data leakage.
  • F.4 Encoder Scale by Task Type / F.5 No PCA Variant: TAR’s advantage generalizes across small and large encoders and persists with raw 384-dimensional embeddings without PCA.Larger encoders move from roughly 30M to 300M parameters; the no-PCA analysis covers 33 datasets with CatBoost and LightGBM.
  • G Additional Attention Maps: Target-Aware attention focuses on semantically meaningful, prediction-relevant regions, whereas Frozen attention remains scattered across task-irrelevant areas.This pattern appears across additional test examples from four datasets, including lungs, animal facial features, the optic disc and nerve-fiber region, and human facial features.
Loading 2605.10616v1…