Source-linked AI summary
Large Language Models for Automated Cross-Domain Machine Learning Task Type Identification: A Benchmark Dataset and Evaluation
Petros Tsialis, Steffen Limmer, Tobias Rodemann, Martin Heckmann
TL;DR
The paper asks whether an LLM can infer a dataset’s domain and prediction task from dataset-level information when the target feature is specified. It introduces a benchmark and system for this task, outperforming AutoGluon on tabular data while cross-domain identification is harder.
Problem
Task identification is often performed manually, although preprocessing, model selection, and evaluation depend on the task formulation.
Source: §1 · p. 1Method
Results
Qwen3-14B with DD+ST reaches 0.98 F1 macro versus AutoGluon’s 0.93 on tabular data, while performance decreases when time series datasets add domain identification.
Source: §5 · p. 7, §6 · p. 10Takeaways & Limitations
The findings support treating task identification as an explicit AutoML component, while smaller local models involve an accuracy–deployability trade-off.
Source: §7 · p. 10Incorrect predictions may propagate to later pipeline stages, so the system should be used as transparent, inspectable decision support under expert oversight.
Source: §7 · p. 11
Abstract
from arXiv · showhide
Machine learning task type identification is essential for constructing valid ML pipelines, yet in practice it is typically specified manually. We investigate whether large language models (LLMs) can infer both the data domain and the downstream prediction task directly from dataset-level information when only the target feature is provided by the user. Together with our LLM-based system we also release an annotated benchmark comprising 625 public tabular and time series datasets. We evaluate the proposed approach in three settings: (i) tabular datasets in comparison with established AutoML heuristics, (ii) cross-domain evaluation across tabular and time series datasets, and (iii) a practical deployment scenario using smaller local models. The results show consistent advantages for LLM-based task type identification, with increasing difficulty in heterogeneous and resource-constrained settings. LLM-based approaches outperform AutoGluon in the tabular setting, reaching 0.98 F1 macro compared to 0.93. In the cross-domain setting, the best model achieves 0.90 F1 macro, while smaller locally deployable models reach 0.75, indicating a trade-off between deployment feasibility and accuracy.
1 Introduction
Identifying the downstream ML task is necessary for valid pipelines, but it is often manual and existing heuristics mainly address tabular tasks rather than domain identification across modalities. The paper proposes an LLM-based alternative and frames evaluation around tabular comparisons, cross-domain capability, and practical deployment.
1 Introduction
- Task formulation matters because preprocessing, model selection, and evaluation depend on it, yet it is often handled manually in heterogeneous, poorly documented settings.Existing heuristic methods mainly target tabular task identification and generally do not identify domains across heterogeneous modalities.Source: §1 · p. 1
- The paper introduces an openly available benchmark and an LLM-based system to infer data domain and downstream task from dataset-level information.The system uses structured dataset representations, target-specific statistics, and optional textual descriptions.Source: §1 · pp. 1–2
- The evaluation asks how LLMs compare with AutoML systems on tabular datasets, perform across tabular and time series domains, and fare in local deployment.These are the paper’s three research questions: baseline comparison, cross-domain capability, and practical deployment.Source: §1 · p. 2
2 Related Work
Prior LLM research establishes capabilities relevant to structured inputs, classification, and prompt design, while LLM-assisted AutoML typically starts from a task specified by the user. This paper instead treats task identification as a standalone inference problem.
2 Related Work
- Prior work reports LLM capabilities in processing structured data and classification, with prompting strategies that include examples, explicit reasoning, and dataset meta-information.These capabilities motivate exploring LLMs for task inference, but do not themselves address the paper’s standalone identification setting.Source: §2 · p. 2
- Existing LLM-assisted AutoML systems generally receive an explicit task specification or descriptions that already state the prediction problem.Examples include systems that coordinate workflows, decompose subtasks, evaluate agents, generate semantic features, or recommend solutions.Source: §2 · p. 2
- This paper assumes only the target feature is known and infers domain and task from dataset information, with experiments both with and without dataset descriptions.The input includes feature names, a serialized dataset excerpt, and target-specific statistics; descriptions generally characterize datasets rather than define the task.Source: §2 · p. 2
3 Datasets
The paper introduces an openly available benchmark for identifying ML task types, built from public datasets with heterogeneous sources and formats. Its two-level taxonomy supports evaluation of both domain and downstream-task identification.
3 Datasets
- The collection is presented as the first openly available benchmark specifically designed for machine learning task type identification.It is intended to evaluate whether models infer both domain and downstream task from dataset content and meta-information.Source: §3 · pp. 2–3
- The benchmark contains 625 datasets: 299 Time_Series and 326 Tabular, each categorized by binary classification, multiclass classification, or regression/forecasting.The time-series counts are 101 regression, 61 binary, and 137 multiclass; tabular counts are 140 regression, 86 binary, and 100 multiclass.Source: §3 · p. 3, Figure 1 · p. 3
- Datasets come from OpenML, UCI, Kaggle, and the Time Series Classification Archive, and the loading pipeline preserves original structure where possible instead of imposing extensive uniform preprocessing.The collection includes original datasets and tabular metadata such as domain, task, target variable, and dataset description.Source: §3 · p. 3
- The taxonomy distinguishes discrete-target classification from continuous-target regression, with time-series tasks defined by ordered observations and continuous forecasting grouped with regression.Time-series classification targets categorical labels, while forecasting targets typically represent future values at a defined horizon.Source: §3 · pp. 3–4
4 Methodology
The system combines a dataset and specified target feature with target statistics, a structured dataset representation, and optional description to prompt an LLM for domain and task predictions. Its experiments compare model and prompting configurations under a fixed train-validation-test protocol.
4 Methodology
- The workflow takes a dataset and explicitly specified target feature, optionally adds a textual description, and prompts an LLM to predict domain and downstream task.The target feature defines the prediction objective and is treated as a mandatory input.Source: §4 · p. 4, Figure 2 · p. 4
- Target-specific statistics include value counts, unique values, mean, standard deviation, minimum, and maximum, with additional text-count features for text-like targets.These statistics are intended to provide compact information about the prediction objective.Source: §4 · p. 4
- The serialized input preserves row-column structure, original feature names, and an explicitly marked target feature; group-wise sampling keeps selected contiguous samples in original order.Sampling reduces computational cost and context-window pressure while retaining short-range patterns and the target feature.Source: §4 · pp. 4–5
- The prompt combines system instructions with the serialized dataset, target information, statistics, and optional description; few-shot prompts also include labeled examples.The study evaluates both zero-shot and few-shot prompting.Source: §4 · p. 5
- Configurations are selected using stratified 20% training, 20% validation, and 60% test partitions, with final configurations evaluated once on the held-out test split.The paper uses 1,000 bootstrap resamples of validation and test splits to estimate means and standard deviations rather than cross-validation.Source: §4 · p. 5
- The evaluated configuration space varies model, reasoning mode, prompting strategy, and dataset information, with GPT-5.3 and Qwen models spanning cloud and local deployment settings.Qwen3-14B targets high-memory GPUs, while compact 4B FP8 variants target consumer-grade hardware.Source: §4 · p. 5, Table 1 · p. 6
5 Experimental Results
Across three experiments, LLM-based task identification outperformed AutoGluon on tabular data and generalized across tabular and time-series datasets, while smaller local models achieved lower performance.
Experiment 1
- 0.98 F1 macro (Qwen3-14B, DD+ST) exceeds AutoGluon’s 0.93 F1 macro for tabular task identification.ST-only LLM configurations also outperformed AutoGluon, reaching 0.95–0.97 F1 macro.Source: Table 2 · p. 6, §5 · p. 7
Validation and configuration selection
- Validation favored DD+ST in tabular and local-model settings, while semantic dataset information was most helpful across domains.For small local models, few-shot prompting performed slightly better than zero-shot prompting; both were retained for test evaluation.Source: §5 · pp. 6, 8, Figure 3 · p. 7
Error patterns
- AutoGluon tended to confuse regression with multiclass classification, while the selected LLM configuration produced only rare misclassifications.In the cross-domain setting, frequent errors involved the same downstream task across tabular and time-series domains.Source: §5 · pp. 7–9, Figure 4 · p. 7, Figure 5 · p. 9
Experiment 2
- 0.90 F1 macro was achieved by GPT-5.3 with reasoning in the few-shot DD+ST cross-domain setting.Cross-domain identification was harder than tabular identification; DD+ST remained the most robust configuration, and reasoning benefited GPT-5.3.Source: Table 3 · p. 8, §5 · p. 8
Experiment 3
- 0.75 F1 macro and 0.74 balanced accuracy were achieved by Qwen3-4B-Instruct with DD+ST in the few-shot local-model setting.With ST-only, its few-shot F1 macro fell from 0.75 to 0.50; the authors conclude small local models require semantic context to resolve domain ambiguities.Source: Table 4 · p. 9, §5 · p. 9
6 Discussion
LLM task identification performs better than heuristic AutoML on tabular data, but cross-domain performance falls when time-series data require domain recognition. Input representation affects stability, and evaluation does not yet measure downstream pipeline effects.
6 Discussion
- LLM approaches outperform the heuristic AutoML baseline on tabular datasets, while cross-domain performance decreases mainly because models must also recognize the data domain.Domain identification is therefore an important component for future AutoML systems.Source: §6 · p. 10
- The most stable results use row- and column-structured data with feature names and a marked target, enriched by descriptions and target-specific statistics.Target-specific statistics alone remain competitive for stronger models; descriptions improve cross-domain performance and reduce variance, while smaller local models are less robust without semantic context.Source: §6 · p. 10
- The evaluation does not measure how task-type errors affect downstream preprocessing, model selection, training, evaluation, or overall pipeline performance.The benchmark is mainly curated public datasets, and contamination through dataset content or metadata remains possible despite mitigation steps.Source: §6 · p. 10
7 Conclusion
The paper introduces an LLM-based task-identification system and benchmark, finding advantages over heuristic AutoML in tabular settings and generalization across tabular and time-series data. It also emphasizes expert oversight because incorrect predictions may affect later pipeline stages.
7 Conclusion
- LLMs outperform heuristic AutoML task inference on tabular data and generalize across tabular and time-series datasets, while smaller local models trade accuracy for deployability.The conclusion supports treating task identification as an explicit AutoML component rather than a fixed user-provided input.Source: §7 · p. 10
- Incorrect predictions may propagate into unsuitable preprocessing, model choices, or evaluation metrics, so the system should remain transparent, inspectable decision support under expert oversight.The paper notes that such errors may ultimately contribute to biased or unreliable outcomes.Source: §7 · p. 11
A Additional Dataset Details
Table 5 lists repository links for the benchmark dataset, implementation, and LM Contamination Index.
A Additional Dataset Details
- Table 5 lists repository links for the benchmark dataset, implementation, and LM Contamination Index.The caption identifies the three linked resources but does not provide their URLs.Source: Table 5 · p. 20
A.2 Dataset Metadata
The benchmark records dataset-level metadata and distinguishes tabular prediction tasks from time-series tasks by data structure and target type.
A.2 Dataset Metadata
- The metadata table records dataset name, domain, downstream task, target, description, multi-target status, target type, dataset properties, storage path, and download link.For time series it also records whether the data are univariate or multivariate and contain one or multiple series.Source: §A.2 · p. 20
A.3 Dataset Types
- Tabular datasets are independent samples with feature columns and a specified target, categorized as binary classification, multiclass classification, or regression.Source: §A.3 · p. 20
- Time-series datasets have sequential or temporal structure; forecasting is treated as continuous-value regression, while classification uses categorical targets.The benchmark includes univariate and multivariate series, as well as single-series and multiple-series datasets.Source: §A.3 · p. 20
A.4 Runtime and Cost Considerations
The runtime and token statistics compare the practical costs of AutoGluon and LLM approaches. The loading pipeline preserves heterogeneous dataset structure, while the prompt template asks models to identify both data domain and prediction task.
A.4 Runtime and Cost Considerations
- 0.0004s average inference time for AutoGluon is several orders faster than LLM-based approaches, while some non-thinking Qwen models run in a few seconds per prompt.The comparison covers latency and cost by model, baseline, and in-context learning strategy.Source: §A.4 · p. 21, Table 6 · p. 21
A.5 Token Statistics
- 2,502-token average zero-shot prompts (537–15,585) rise to 7,400 with few-shot examples; user prompts remain unchanged at 2,252 tokens on average.Source: §A.5 · p. 21, Table 7 · p. 22
A.6 Loading Pipeline
- The loader handles heterogeneous file formats and time-series layouts with format-specific parsers, preserving original structure rather than forcing a common schema; target metadata is retained.Source: §A.6 · pp. 21–22
B Prompt Templates
- The prompt requests a Tabular or Time_Series domain and binary, multiclass, or regression task, using dataset structure and semantic context rather than datatype alone.Source: B Prompt Templates · pp. 22–23
C Scaling Behavior
Model performance scales positively with parameter count. Larger models perform better, while smaller models show a sharper decline on complex cross-domain tasks.
C Scaling Behavior
- Figure 7 links larger parameter counts with higher F1 macro: GPT-4.1 and GPT-5.3 outperform smaller models, while 4B variants decline notably on complex cross-domain tasks.The 14B Qwen variants achieve intermediate performance; the passage attributes the broader trend to larger models' stronger handling of semantic and structural differences.Source: Figure 7 · p. 25, C Scaling Behavior · p. 25