PaperlayerView exact revision on arXiv

Source-linked AI summary

Solving In-Table Prediction Problems by Deep Neural Networks with Performance Evaluation Using Synthetic Data

Xiao Zhao, Daniela Oelke

arXiv:2609.01262v1cs.LG

TL;DR

The paper asks whether neural networks can predict arbitrarily masked columns in continuous-feature tables, a task distinct from standard supervised learning and missing-value imputation. It uses self-supervised masking, trainable embeddings, and synthetic relationships to compare three architectures. Attention-based models perform best with larger training sets, but the findings are limited to controlled synthetic settings.

  • Problem

    The paper addresses whether neural networks can predict arbitrarily masked table columns from the remaining known columns, rather than only a predefined supervised target or missing entries.

  • Method

    The study randomly masks columns for self-supervised prediction, embeds numeric and missing continuous values with a neural layer, and evaluates MLP, Resnet, and attention-based architectures on synthetic data.

  • Results

    Attention-based architectures achieve the best performance overall, particularly with larger training sets; increasing embedding dimension consistently benefits attention-based models but has limited influence on MLP and ResNet.

  • Takeaways & Limitations

    Under the tested synthetic conditions, attention-based structures are preferable when sufficient training data and larger embedding dimensions are available.

  • Takeaways & Limitations

    The conclusions come from two synthetic three-column relationships with predefined dependencies and may not transfer directly to larger or real-world tables.

Abstract

from arXiv · show

Tabular deep learning (TDL) leverages neural networks (NN) to extract patterns from tabular data. Traditional TDL methods follow a supervised learning paradigm, where a target feature is explicitly given. In this work, however, we explore a different approach by employing deep NNs to learn relationships among individual columns within a given table. We investigate whether NNs can predict the values of arbitrarily selected columns in a given table based on the remaining known columns. We call this problem In-Table Prediction (ITB), which is slightly different from table imputation methods and the pretraining task of TDL. Three potential usage scenarios are identified, which, to our best knowledge, have not been extensively studied in the literature. A self-supervised learning approach is applied to address this problem by randomly selecting columns to be masked out and used as learning targets. This work focuses on tabular datasets containing only continuous features. To handle missing values in continuous features, a novel neural layer is proposed to embed both numerical and empty values. Synthetic data is generated based on predefined column relationships, with empty values inserted using two distinct mechanisms. Additionally, an adapted masking strategy is employed to create test data. Performances of three NN architectures, namely MLP, Resnet and Transformer, are evaluated using the generated synthetic data. We conclude that, the attention-based structure outperforms the other two networks, when a sufficiently large number of training examples is available and a relatively large embedding length is chosen. We stress that these findings are obtained under controlled, synthetic conditions with a small number of columns and it should therefore be regarded as an initial, narrowly-scoped investigation rather than a general characterization of ITP on real-world tabular data.

1 Motivation and contribution

This work introduces In-Table Prediction (ITP), a tabular-data task that predicts arbitrarily masked values rather than performing conventional supervised prediction or missing-value imputation. It proposes neural approaches and evaluates them on synthetic continuous-feature data across usage scenarios and experimental conditions.

  • ITP predicts missing values at user-defined table positions from the remaining values in an incomplete row.Unlike MVI, the task targets arbitrarily masked positions rather than all existing empty values.
  • ITP supports completing newly created rows, recovering lost columns, and checking whether masked positions in an existing table are correct.
  • The study proposes a neural layer that embeds numeric and empty continuous-feature values as trainable representations for downstream DNN layers.Categorical features are excluded from the study.
  • Synthetic data with predefined column relationships is used to evaluate whether MLP, Resnet, and Transformer-based structures can solve ITP.The controlled relationships make performance results more directly interpretable than evaluation on real data.
  • The study examines performance across dataset sizes, missing-value insertion mechanisms, embedding lengths, and masked-value positions.

2 Problem Analysis

The paper formulates a continuous-feature table with numerical, empty, and masked states for each cell. ITP trains on unmasked rows to reconstruct masked positions while preserving empty values as potentially meaningful states.

  • A table T contains N rows and K continuous-feature columns, with rows treated as independent and identically distributed examples.
  • Each cell may contain a numerical value, an empty value called NaN, or a masked placeholder ζ that the model must predict.The masked target may ultimately be numerical or empty.
  • Rows are partitioned into Tfull, containing no masked values, and Tmasked, containing at least one masked value.Both partitions may contain empty values.
  • The model learns hidden relationships among columns from Tfull and uses them to predict masked positions in Tmasked.
  • Unlike MVI, ITP treats empty values as special states that may themselves need to be recovered rather than replaced with numeric imputations.

3 Related Work

Related tabular deep learning work has largely focused on supervised targets, transfer learning, or masked pretraining, while missing-value imputation seeks complete datasets. This paper distinguishes ITP by using arbitrary targets and recovering original masked content, including empty values.

  • Most tabular deep learning methods predict a predefined target feature in supervised learning tasks.
  • Self-supervised pretraining masks input columns and reconstructs them to learn representations for downstream tasks.
  • Missing-value imputation replaces missing entries with substituted values to produce a complete dataset for analysis.
  • Compared with Gorishniy et al., this work adds a neural layer for embedding missing values and masks in continuous variables and uses MLM-based pretraining.
  • ITP uses arbitrary column subsets as targets and seeks to recover masked content that may be numeric, categorical, or empty, unlike standard MVI for continuous features.

4 Our approach

The approach masks selected table entries, embeds numerical values, empty values, and masks, then uses MLP, Resnet, or Transformer blocks to reconstruct masked positions. Training optimizes losses only at masked positions, with outputs representing empty-value confidence and numerical predictions.

  • Model structure: The method applies a masking layer before embedding each numerical value, empty value, or mask into a trainable D-dimensional vector.The resulting embeddings are processed by one of three neural architectures.
  • Model structure: MLP and Resnet flatten all column embeddings, whereas the Transformer processes the K embeddings as a sequence without positional encoding.The MLP and Resnet route uses a standard block after flattening; the Transformer route uses an encoder followed by per-column linear layers.
  • Model structure: Each column output contains two empty-value confidence scores and one regressed value, with the regression valid only when the column is predicted as numerical.The output vector for each column has dimension three.
  • Training strategy: The loss is computed only for masked positions, and its regression term is activated only when a masked ground-truth value is numerical.The model predicts both whether a position is empty and, when applicable, its numerical value.
  • Training strategy: Training permits masking up to K − 1 columns in a row, unlike approaches using a fixed masking portion.This setting supports predicting the remaining values from a single known input column during prediction.
  • Training strategy: Network depth and width are fixed because of limited computational resources, while early stopping selects the best validation-loss model.The study does not tune the number of layers or neurons.

5 Design of the experiment

The experiment uses synthetic continuous tables with predefined relationships and inserted missing values, then evaluates masked-value prediction under a deterministic test construction. Performance is assessed with NRMSE for numerical predictions and accuracy for empty-value classification.

  • Synthetic data generation: Synthetic tables are generated by first creating complete continuous-feature tables and then inserting empty values.The predefined relationships make the underlying column dependencies known for performance interpretation.
  • Synthetic data generation: The complete synthetic datasets use a linear function and a multivariable exponential function, with Dataset B containing three continuous columns.The linear dataset includes noise controlled by τ, while the exponential dataset uses x0, x1, and x2.
  • Missing-value mechanisms: Empty values are inserted using MAR and MNAR mechanisms; MCAR is excluded because it provides no hidden relationship for the neural network to learn.Under MAR, missingness depends on the other features; under MNAR, it depends on the feature’s own value.
  • Masking strategy: Training and validation rows receive a random number of masks from 0 through nmax using masking by maximal number nmax.The validation data supports early stopping, while the training data updates network parameters.
  • Masking strategy: Test data uses masking for deterministic relationship, retaining rows where masked values can be completely derived from unmasked positions.Rows with incompatible empty-value patterns are removed, and masking follows the invertibility of each predefined relationship.
  • Evaluation metrics: NRMSE evaluates numerical predictions at masked non-empty positions, while accuracy evaluates whether masked positions are correctly classified as empty or numerical.The index set Φ/Θ excludes masked positions whose ground truth is empty from the NRMSE calculation.

6 Results of the experiment

The experiments use a grid search over synthetic datasets, masking strategies, architectures, training sizes, and embedding lengths. Across most tested settings, attention-based networks perform best with larger datasets and sufficiently large embeddings, while masking and missing-value mechanisms affect performance.

  • Experimental setup: The grid search varies datasets, insertion strategies, model types, inserted masks, noise level, training size, and embedding length.Each training uses random initialization, early stopping, and reports NRMSE and empty-value-position accuracy on test data.
  • Model structure: For training sizes N ≥10000, the attention-based structure performs better than MLP and Resnet in most cases.For the exponential dataset, NRMSE advantages generally appear above 10,000 samples, with the MAR case reported at N = 50000.
  • Training size: Performance increases with training size, reaching saturation around N = 10000 for the linear dataset but not clearly for the exponential dataset.The authors do not rule out further improvement from additional training examples for the exponential dataset.
  • Masking strategy: For larger datasets, mask ≤1 outperforms mask ≤2 in most cases, consistently for the linear dataset and nearly always for the exponential dataset.The exponential dataset has one reported exception: MAR with N = 10000.
  • Missing-value mechanism: MNAR accuracy is significantly lower than MAR under equal training sizes and mask ≤1, indicating a more challenging learning scenario.The paper attributes this difficulty to inferring the unknown value before determining whether the observed value should be missing.
  • Embedding length: Increasing embedding length improves the attention-based network, with saturation near D = 128 for linear data and D ≥256 for several exponential-data cases.For exponential data under MNAR, saturation is not clearly observed for D ≤512; MLP and Resnet are largely unaffected by D.

7 Conclusion

The study evaluates neural approaches for In-Table Prediction using synthetic continuous-feature data with predefined relationships and missing-value mechanisms. Attention-based networks generally perform best, especially with larger datasets and embeddings, but the conclusions remain limited to controlled synthetic settings.

  • The study evaluates MLP, Resnet, and attention-based architectures on synthetic datasets generated from predefined inter-column relationships.Missing values are introduced through two mechanisms, and the evaluation compares model behavior under controlled conditions.
  • The attention-based architecture achieves the best overall performance, particularly when larger training sets are available.Performance generally improves as the amount of training data increases.
  • Single-mask training is preferable in most tested cases, while larger embedding dimensions consistently improve the attention-based model.Embedding size has limited influence on MLP and Resnet, but is especially beneficial for the exponential dataset.
  • The conclusions are limited to two synthetic three-column relationships with predefined dependencies and may not transfer directly to larger or real-world tables.The authors characterize the study as an initial investigation under controlled synthetic conditions rather than a general claim about ITP.

A Applied network structure and training parameters

The appendix documents fixed network structures and training parameters for the MLP, Resnet, and attention-based architectures. The models use established building blocks, with task-specific processing added around them.

  • Applied network structure: The appendix documents the applied network structures for MLP, Resnet, and attention-based networks.The network structures are fixed as defined in the corresponding tables.
  • Applied network structure: The MLP structure is specified in Table 3, while the Resnet and attention-based structures are specified in Tables 4 and 5.These tables provide the fixed architectures used in the experiments.
  • Training parameters: Training parameters are fixed for the experiments.The appendix introduces the fixed training-parameter configuration before presenting the varying evaluation parameters.

B Selected parameters for performance evaluation and comparison

The performance evaluation varies datasets, missing-value insertion strategies, model types, masking amounts, training sizes, and embedding lengths. A grid search tests every possible combination of the selected parameter values.

  • The evaluation varies datasets, insertion strategies, model types, maximum inserted masks, training sizes, and embedding lengths.These parameters define the experimental comparison space.
  • Table 6 lists the parameters and their possible values used for performance evaluation and comparison.The table serves as the specification of the candidate experimental settings.
Loading 2609.01262v1…