PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Correctness: Evaluating Semantic Knowledge in Cross-Table Transfer

Seokyong Sheem, Hochang Lee, Suyeong Lee, Daekyum Kim

arXiv:2609.34098v1cs.LG

TL;DR

Heterogeneous schemas complicate cross-table transfer, and semantic ablations can confound sensitivity to altered content with the predictive benefit of intended knowledge. The paper distinguishes these quantities and evaluates them with claim-matched controls; experiments show that large altered-condition gaps can coexist with little predictive utility over a suitable reference. It concludes that predictive claims require suitable references and that measured utility depends on what the reference can recover.

  • Problem

    As semantic knowledge is increasingly used to bridge heterogeneous schemas, how much it improves prediction remains an open question, and common ablations do not necessarily isolate that benefit.

    Source: §1 · pp. 1–2

  • Method

    The paper distinguishes content sensitivity from predictive utility and uses claim-matched evaluation, requiring references to remove the tested component while preserving other inputs and the modeling setup.

    Source: §1 · p. 2

  • Results

    Across an existing semantic tabular model, real transfer tasks, and controlled experiments, altered semantics sometimes produce large performance gaps despite little predictive utility, which varies with what the reference can recover.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Altered controls assess sensitivity to semantic content, while claims of predictive benefit require a suitable reference and should account for reference sensitivity.

    Source: §5 · pp. 9–10

    The bounded audit evaluates what reported controls identify for stated claims, rather than the overall correctness of the audited studies or methods.

    Source: §4.5 · p. 9

Abstract

from arXiv · show

Semantic knowledge is increasingly used to bridge heterogeneous schemas in tabular learning, but how much does that knowledge actually improve prediction? Studies in tabular learning commonly answer this question through semantic ablations that modify or suppress the supplied semantic knowledge. We show that these ablations can lead to misleading conclusions about predictive benefit: poor performance under altered semantics may be taken as evidence that the intended knowledge is beneficial. Across real and controlled experiments, altering semantic content can produce large performance differences even when the model gains little predictive benefit from having that semantic knowledge in the first place. To separate these effects, we distinguish two quantities: content sensitivity and predictive utility. Content sensitivity measures the change in performance when semantic content is altered, whereas predictive utility measures the benefit of the intended semantic knowledge relative to a suitable reference without that knowledge. This distinction motivates an evaluation framework in which the control is chosen according to the question being asked: altered controls assess sensitivity to semantic content, whereas claims that semantic knowledge improves prediction require a suitable reference. Even then, predictive utility is not fixed; it varies across suitable references and decreases when the reference can more easily recover the tested knowledge from other inputs or labeled examples. In a bounded audit of 25 semantic-ablation comparisons across nine studies, only one of 18 explicit predictive-utility claims is paired with a control that clearly isolates the tested semantic contribution. Together, these findings motivate a simple evaluation principle: semantic-ablation controls should be chosen and interpreted according to the question they are intended to answer.

1 INTRODUCTION

Heterogeneous schemas make cross-table transfer difficult, and semantic knowledge is increasingly used to bridge them. The introduction distinguishes sensitivity to altered semantic content from predictive utility, arguing that claims of predictive benefit require a suitable reference.

1 INTRODUCTION

  • Semantic knowledge can help interpret values, identify corresponding variables, and introduce predictive structure, but its contribution to performance remains an open question.It may come from feature names, documentation, codebooks, knowledge graphs, or domain rules.Source: §1 · p. 1
  • Intended–altered performance gaps can reflect degradation under altered semantics, rather than benefit from the intended knowledge over a condition without it.Thus, semantic ablations that modify or suppress inputs can be misread as evidence of predictive benefit.Source: §1 · p. 2
  • Content sensitivity measures performance change under altered semantic content, whereas predictive utility measures benefit over a suitable reference without the tested knowledge.The reference must remove the tested component while preserving other inputs and their presentation, and keep the model and learning procedure unchanged.Source: §1 · p. 2
  • Across an existing semantic tabular model, real transfer tasks, and controlled experiments, altered semantics sometimes produce large gaps despite little predictive utility over a suitable reference.In several settings, degradation under the altered condition explains much of the intended–altered gap; measured utility also varies with what the reference can recover from other inputs or labeled target examples.Source: §1 · pp. 2–3
  • In a bounded audit of 25 comparisons across nine studies, only one of 18 explicit predictive-utility claims used a control clearly satisfying all three suitability requirements.Eleven controls failed at least one requirement and six were indeterminate; this does not show that the corresponding knowledge is unhelpful, only that the comparison does not isolate its utility.Source: §1 · p. 2
  • The paper proposes matching semantic-ablation controls to the question being asked, distinguishing sensitivity assessments from claims about predictive benefit.Its conceptual, methodological, and empirical contributions include criteria for suitable references and analysis of how utility varies with reference information.Source: §1 · p. 3

2 RELATED WORK

Related work uses semantic information to support transfer across heterogeneous schemas, while leaving open whether particular uses improve prediction. The paper connects this gap to evaluation work showing that conclusions depend on what controls change, preserve, and use as a reference.

2.1 SEMANTIC KNOWLEDGE IN CROSS-TABLE TRANSFER

  • Cross-table methods use feature descriptions or open-vocabulary representations, while other semantic tabular methods serialize feature names and values; whether these uses improve prediction remains open.Semantic information can also come from documentation, schema matching, knowledge graphs, generated features and rules, or task-derived constraints.Source: §2.1 · p. 3

2.2 EVALUATING THE CONTRIBUTION OF SEMANTIC KNOWLEDGE

  • Prior work on ablations, probing, removal, and attribution emphasizes that conclusions depend on the control, preserved information, and reference; transfer studies also find value can diminish with more labeled target support.The paper responds with claim-matched controls that distinguish content sensitivity from predictive utility and examine variation across suitable references.Source: §2.2 · p. 3

3 EVALUATING THE CONTRIBUTION OF SEMANTIC KNOWLEDGE

The evaluation holds the task and modeling setup fixed while comparing intended semantic knowledge with altered or removed conditions. It defines criteria for suitable references and constructs measurement, correspondence, and relation knowledge from documentation and other specified sources.

3.1 EVALUATION DESIGN

  • Experiments compare semantic conditions on the same evaluation data and hold the model, training procedure, and other components fixed; target labels support adaptation in transfer tasks.In the TabLLM reanalysis, semantic knowledge varies within the model input.Source: §3.1 · p. 3
  • The design tests measurement, correspondence, and relation knowledge: interpreting coding or scales, matching compatible variables, and specifying structure among variables.These roles organize test cases and are not presented as a full taxonomy.Source: §3.1 · p. 3
  • For real-data experiments, intended knowledge comes from documentation or domain sources and is not assumed ground-truth correct; controlled transformations and outcome design specify intended knowledge in the benchmark.For correspondence, the reference separates source and target values into distinct columns rather than a shared matched column.Source: §3.1 · p. 4

3.2 CONSTRUCTING A REFERENCE

  • A suitable reference removes the tested component while preserving other information and its presentation—including feature identity, input width, and format—and leaves the model and learning procedure unchanged.Changing feature names can otherwise change a key–value input into a values-only input.Source: §3.2 · p. 4
  • Different suitable references may yield different predictive-utility values; the paper reports reference sensitivity when multiple references qualify and treats utility as not separately evaluable when none qualifies.An effect is unresolved when a suitable reference exists but its 95% confidence interval does not lie entirely above or below zero.Source: §3.2 · p. 4

3.3 CONSTRUCTING SEMANTIC KNOWLEDGE

  • Measurement knowledge comes from documentation; correspondence proposals use documented variable information and validation checks, while relation candidates are fixed before evaluation and reviewed for support afterward.Target labels and predictive performance do not create, select, or revise the semantic specifications.Source: §3.3 · p. 5

4 EXPERIMENTS AND RESULTS

Across TabLLM, real cross-table transfer, and controlled experiments, semantic-ablation gaps can reflect degradation under altered controls rather than benefit from intended knowledge. Predictive utility varies across suitable references and falls when the reference can recover more of the tested information; a bounded audit finds that most explicit utility claims are not isolated by their controls.

4.1 EXPERIMENTAL SETTINGS

  • The evaluation spans a TabLLM reanalysis, real cross-table transfer, and controlled experiments, with paired conditions sharing data splits and samples.The settings include clinical and hydrological tasks; the controlled benchmark varies schema mismatch and prediction-task structure independently.Source: §4.1 · pp. 5–6

4.2 CONTENT SENSITIVITY DOES NOT IDENTIFY PREDICTIVE UTILITY

  • TabLLM shows content sensitivity can overstate or understate predictive utility, while real-data relation experiments show altered controls can change sensitivity without changing utility.For Credit-g, sensitivity is +.066 versus utility −.003; for Bank, sensitivity is −.032 versus utility +.089. Random rather than reversed directions sharply reduce sensitivity in all three real-data relation settings.Source: §4.2 · p. 6

4.3 PREDICTIVE UTILITY CAN VARY ACROSS REFERENCES

  • Utility varies across suitable references: the nine-dataset mean remains positive under all 24 TabLLM identifier assignments, but dataset-level ranges cross zero for four datasets.Only Jungle has simultaneous 95% confidence bands confirming a sign change; the other three crossings are unresolved point-estimate results.Source: §4.3 · pp. 7–8, Figure 3 · p. 7

4.4 PREDICTIVE UTILITY DEPENDS ON WHAT THE REFERENCE CAN RECOVER

  • Utility rises when tested information is harder for the reference to recover and falls as labeled target support increases, with most variation coming from reference performance.At the largest measurement mismatch and ntarget = 32, XGB utility reaches +.259 to +.260; across the three prespecified target structures it falls from about +.26 to +.03 as target support increases. For TabLLM it falls from +.088 to +.007, a decrease of .081 (95% CI [+.025, +.148]).Source: §4.4 · pp. 8–9, Figure 4 · p. 8

4.5 IMPLICATIONS FOR PUBLISHED SEMANTIC ABLATIONS

  • Only one of 18 explicit predictive-utility claims in the bounded audit uses a control meeting all three suitability requirements; 11 fail at least one and six are indeterminate.The audit assesses whether each reported control isolates the stated claim, not the overall correctness of the study or method.Source: §4.5 · p. 9

5 CONCLUSION

Altered semantic controls measure sensitivity to content, not by themselves the predictive contribution of semantic knowledge. Predictive-utility claims require a suitable reference, and utility can vary with the reference and information available without the tested knowledge.

5 CONCLUSION

  • Altered controls can reveal dependence on supplied semantic content, but do not directly estimate predictive utility.Predictive utility depends on the reference, available information, labeled target support, and model training setup.Source: §5 · p. 9
  • Predictive-utility claims require a suitable reference that removes the tested knowledge while preserving other predictive information.When several suitable references are available, report sensitivity to the reference choice.Source: §5 · pp. 9–10
  • Reported controls often fail to isolate the claimed predictive utility, motivating controls chosen and interpreted for the question they address.The audit identifies this issue in published practice.Source: §5 · p. 10

DATA USE AND AVAILABILITY

The study uses secondary data from named clinical, survey, hydrological, and benchmark sources. Some prepared-input provenance could not be fully reconstructed, and source data are not redistributed with the repository.

DATA USE AND AVAILABILITY

  • Secondary data came from NHANES, KNHANES, HRS, CAMELS-US, CAMELS-GB v2, and nine public TabLLM benchmark datasets.The listed datasets form the study’s data sources.Source: DATA USE AND AVAILABILITY · p. 10
  • The controlled NHANES benchmark uses source cycles from 2001–2002 through 2013–2014 and target data from later cycles through August 2023.The final target release is labeled 2023 in local preprocessing; these cycle ranges do not describe the separate real clinical-transfer inputs.Source: DATA USE AND AVAILABILITY · p. 10
  • The exact raw KNHANES release and HRS byte-level lineage to frozen analysis snapshots could not be fully reconstructed.For HRS, the preparation chain identifies the RAND HRS Longitudinal File 2022 and 2016 biomarker and venous blood products.Source: DATA USE AND AVAILABILITY · p. 10
  • The repository does not redistribute participant-level clinical data, model weights, or row-level predictions; restricted data must be obtained from providers under their terms.Some data require registration or supplemental authorization.Source: DATA USE AND AVAILABILITY · p. 10

REPRODUCIBILITY STATEMENT

The supplement and repository document experimental settings, analysis provenance, and reproducible outputs. They also provide code and frozen records, with mappings linking reported visuals to the scripts and outputs that generated them.

REPRODUCIBILITY STATEMENT

  • The supplement provides experimental settings, inference procedures, analysis provenance, reference constructions, and robustness analyses.These materials support reproduction of the reported comparisons.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The repository includes code, frozen semantic records and reference libraries, manifests, and environment information.It also maps reported tables and figures to their generating scripts and outputs.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Data-access boundaries, product identifiers, provenance limitations, and reproducible derived outputs are documented in the supplement and repository.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The supplementary material describes the authors’ use of generative AI and documents experimental construction, inference, provenance, evaluated panels, and reference criteria. Its scope includes specified datasets and settings, with some source-data lineage limitations.

AI USE STATEMENT

  • Language models proposed semantic candidates from schema metadata and documentation; authors fixed specifications before evaluation and verified scientific decisions and analyses.Generative AI also provided research-development feedback and helped refine manuscript phrasing and structure.Source: AI USE STATEMENT · p. 10

A SUPPLEMENTARY EXPERIMENTAL AND CONSTRUCTION DETAILS

  • The supplement documents settings, inference, provenance, reference constructions, robustness analyses, and repository materials for reproducing comparisons.Reported results are specific to the evaluated datasets, learners, interventions, and references.Source: A SUPPLEMENTARY EXPERIMENTAL AND… · p. 14

A.1 COMMON EXPERIMENTAL AND INFERENCE SETTINGS

  • A.1 distinguishes intended–altered content sensitivity from intended–reference utility, pairs conditions on split and seed, and uses 10,000-resample bootstrap intervals unless stated otherwise.Target-only models are reported separately and are not references for individual semantic components.Source: §A.1 · pp. 15–16

A.2 ANALYSIS PROVENANCE

  • A.2 classifies analyses by timing; exact KNHANES release composition and HRS lineage to the frozen snapshot could not be fully reconstructed.HRS features and endpoints should not be interpreted as measurements from a single visit.Source: §A.2 · pp. 16–17

A.3 CONTROLLED BENCHMARK CONSTRUCTION

  • A.3 uses an eight-feature NHANES panel and 20 realizations per outcome family, varying synthetic mechanisms, schema mismatches, and altered measurement, correspondence, or relation knowledge.Source: §A.3 · pp. 17–18

A.4 EVALUATED FEATURE AND ENDPOINT PANELS

  • A.4 evaluates binary survey-history endpoints in focused and extended panels, alongside 13 eligible examination endpoints.The focused survey panel has three endpoints; the extended panel has ten.Source: §A.4 · p. 18, Table A3 · p. 18

A.5 ADDITIONAL DETAILS ON SUITABLE REFERENCES

  • A.5 defines suitable references by removal, preservation, and setup; survey omission fails preservation, while permuted correspondence fails removal.All comparisons listed in Table A4 satisfy the setup criterion.Source: §A.5 · p. 19, Table A4 · p. 19

A.6 SEMANTIC-KNOWLEDGE CONSTRUCTION AND AUDITS

The audit reports how documentation-based semantic candidates are generated, checked, and evaluated, while documenting limits in both model audits and the evidence supporting selected relations.

A.6 SEMANTIC-KNOWLEDGE CONSTRUCTION AND AUDITS

  • Candidate generation receives schema metadata and selected documentation, while fixed mechanical checks validate schemas, evidence, leakage risks, operations, units, duplicates, and candidate limits.LLM confidence, predictive performance, and human semantic judgments are not used to accept candidates.Source: §A.6 · p. 19
  • The experiments use the stored candidate set from one recorded LLM run; deterministic regeneration from a new call is not claimed.The stored prompt, schemas, validation rules, response, manifest, and hashes preserve the executed run.Source: §A.6 · p. 19
  • Five of seven evaluated models produce complete measurement-code records, and four also produce valid-code sets covering the target codes needed for evaluation.This audit tests extraction consistency from documentation, not predictive utility.Source: §A.6 · pp. 19–20
  • Names, units, and codebook descriptions recover all 9 known correspondences in the 1,315-variable candidate set; a single threshold retains 8 true matches and yields no false matches in nine no-match cases.Names alone recover 2 of 9 matches, and adding units recovers 4 of 9.Source: §A.6 · p. 20
  • The closed 9-by-9 matching task recovers all 9 matches, indicating an easier correspondence setting than matching against the full target schema.The relation set was fixed before predictive evaluation, and subsequent literature review documented its evidence without changing the set.Source: §A.6 · p. 20
  • The external relation-evidence table documents support of varying strength and does not establish that every relation is a universal constraint.The long-term-to-daily scale mismatch further limits direct transfer of some evidence.Source: Table A5 · p. 20, §A.6 · p. 20
  • Relation support has boundaries: some evidence concerns BMI rather than weight, or population associations that vary by conditioning set and are not directly established by joint use in LDL estimation.Hydrology evidence often concerns annual or long-term behavior, while prediction uses daily data; the long-term relation is weak and not uniformly negative.Source: §A.6 · p. 20

B.1 TABLLM FEATURE-NAME ABLATION

Across these analyses, intended–altered performance differences do not by themselves establish predictive benefit: results vary by semantic component, altered control, and reference construction.

B.1 TABLLM FEATURE-NAME ABLATION

  • For TabLLM, the macro intended–altered difference is +.107 AUROC, but the values-only comparator also removes template structure and is not a suitable reference for feature-name utility.Credit-g has +.066 content sensitivity but −.003 predictive utility, while Bank has −.032 content sensitivity but +.089 utility.Source: §B.1 · pp. 21–22, Table A6 · p. 22

B.2 MEASUREMENT KNOWLEDGE

  • With documented measurement information under coding-polarity mismatch, content sensitivity and predictive utility are positive; in the ten-endpoint extension, utility remains positive but the sensitivity result fails its coverage requirement.Cancer history in KNHANES-to-NHANES transfer has only 4 of 10 finite seed pairs, so that broader sensitivity result is descriptive.Source: §B.2 · pp. 22–23, Table A7 · p. 22

B.3 CORRESPONDENCE KNOWLEDGE

  • In the six-endpoint examination panel, both correspondence sensitivity and utility are positive, but sensitivity is larger; across 13 endpoints, utility is smaller and unresolved.The width-matched reference preserves other predictive information, and the altered condition's lower performance accounts for part of the larger intended–altered gap.Source: §B.3 · p. 23
  • Across 13 examination endpoints, correspondence content sensitivity is +.057 (95% CI [+.025, +.093]), while predictive utility is unresolved at +.014 (95% CI [−.013, +.042]).The width-matched reference preserves input width and underlying values while removing the source–target correspondence.Source: §B.3 · p. 23, Table A8 · p. 23

B.4 RELATION KNOWLEDGE

  • For real-data relation transfer, intended relations outperform reversed relations substantially while utility over the unconstrained reference stays near zero; random relations sharply reduce the sensitivity gap in two settings.In the controlled additive setting with ntarget = 32, sensitivity is +.304 against reversed relations and +.126 against random relations, while utility is +.024.Source: §B.4 · p. 24, Table A9 · p. 24

C.1 EVALUATED TABLLM REFERENCE CONSTRUCTIONS

  • Across 24 evaluated anonymous-name references, mean TabLLM utility ranges from +.088 to +.124, while dataset-level point-estimate ranges cross zero for four of nine datasets.The evaluated library varies only the mapping of anonymous identifiers to fixed field lines.Source: §C.1 · p. 25, Table A10 · p. 25

C.2 REFERENCE VARIATION AND EVALUATION UNCERTAINTY

Reference choice, recoverability, and labeled target support all affect measured predictive utility. Across the evaluated settings, utility tends to be greater when supplied semantic information is harder for the reference to recover.

C.2 REFERENCE VARIATION AND EVALUATION UNCERTAINTY

  • After accounting for evaluation uncertainty, 79.2–99.7% of between-reference variance remains; only Jungle has a resolved sign change, supported for 1 of 9 datasets.Jungle's lowest utility is −.0196 (95% CI [−.0311, −.0081]) and its highest is +.0828 (95% CI [+.0694, +.0962]).Source: §C.2 · p. 26, Figure A2 · p. 26

C.3 ADDITIONAL REFERENCE CHECKS

  • Among four datasets with closer published-result reproduction, mean utility across references ranges from +.120 to +.175, and point-estimate ranges cross zero for one dataset.Mean utility under the primary anonymous reference is +.141.Source: §C.3 · p. 26

D.1 MEASUREMENT MISMATCH

  • Across all three outcome families and evaluated models, predictive utility rises with greater measurement mismatch; at the largest mismatch it is approximately +.26 for XGB and HistGB with ntarget = 32, versus +.02 to +.04 with ntarget = 512.The decrease from ntarget = 32 to 512 has a 95% paired interval below zero for every model and outcome family.Source: §D.1 · p. 27, Figure A3 · p. 27

D.2 RECOVERABILITY OF RELATION VALUES

  • When relation constituents are removed, pairwise XGB utility with ntarget = 32 rises from +.002 to +.242, while intended-condition performance changes comparatively little and reference performance deteriorates.The same qualitative pattern appears across the sparse family, both tree models, and both target sample sizes.Source: §D.2 · p. 28, Figure A4 · p. 28
  • With fixed input width, pairwise XGB reconstruction R2 falls from .987 to .032 as utility rises from +.002 to +.266; for the sparse family, R2 falls from .988 to .146 as utility reaches +.231.This was a post-result controlled diagnostic; available real-data cases did not cleanly isolate relation-value recoverability.Source: §D.2 · p. 29

D.3 LABELED TARGET SUPPORT

  • TabLLM feature-name utility with the primary anonymous reference falls from +.088 at zero-shot to +.007 at nadapt = 512, although only two of nine dataset trajectories decrease monotonically.Seven of nine datasets have lower utility at nadapt = 512 than at zero-shot.Source: §D.3 · pp. 29–30, Table A12 · p. 29, Figure A5 · p. 30
  • At the largest measurement mismatch, XGB utility falls from +.259–+.260 with ntarget = 32 to +.028–+.037 with ntarget = 512; examination correspondence utility falls from +.069 at ntarget = 64 to +.018 at ntarget = 1024.In the examination panel, absolute nRMSE improves in both conditions, with a larger improvement for the reference.Source: §D.3 · pp. 30–31, Table A13 · p. 30
  • Across TabLLM, controlled measurement, and examination correspondence settings, additional labeled target information reduces semantic-knowledge utility as the reference improves more than the intended condition.The support manipulations are not quantitatively equivalent across examination and controlled settings.Source: §D.3 · pp. 29, 31

E.1 MEASUREMENT ROBUSTNESS

Robustness and extension analyses show that measured semantic utility varies with representation, learner, labeled-target support, and how semantic knowledge is incorporated. The audit analyses also report coding agreement and the criteria used to evaluate published controls.

E.1 MEASUREMENT ROBUSTNESS

  • After rank alignment, utility averages +.228 at ntarget = 32 versus +.260 under the original representation, and falls to +.008 to +.019 for XGB outcome families at ntarget = 512.The polarity-invariant check also reports +.276 in the three-endpoint panel and +.156 in the ten-endpoint extension; the authors conclude the positive utility is not explained solely by AUROC polarity.Source: Table A14 · p. 31, §E.1 · p. 31

E.2.1 TREE CAPACITY AND MLP

  • For tree models, adding intended directional constraints yields +.009 to +.018 utility, but reducing XGBoost depth from 6 to 2 cuts utility roughly tenfold for additive and pairwise families.The sparse-family mean becomes slightly negative; the MLP has +.006 to +.013 utility for pairwise and sparse families, while additive utility remains unresolved.Source: Table A15 · p. 32, §E.2.1 · p. 32

E.2.2 TRANSTAB AND CARTE

  • TransTab favors shared over domain-specific anonymous identifiers, while CARTE shows positive measurement and correspondence utility in some ntarget = 512 families and unresolved relation utility.CARTE supplies relation knowledge through derived value-plus-text columns, not the tree learners’ directional constraints; results therefore are not a direct replication.Source: Table A16 · p. 33, §E.2.2 · p. 33

E.3 FINE-GRAINED AND UNIT-LEVEL ANALYSES

  • Correspondence sensitivity varies by representation: the intended–altered difference is +.065 with explicit source–target pairing and approximately zero with aggregated columns, while survey comparisons are positive for both XGB (+.045) and HistGB (+.051).For CAMELS, flipping precipitation increases log-RMSE by +.191, whereas flipping PET has approximately no effect (−.001); aggregate results therefore do not show equal contributions from both relations. Endpoint and basin results also vary, with relation sensitivity positive across evaluated HRS endpoints and original CAMELS basins while utility varies around zero.Source: §E.3 · pp. 33–34

E.4 EXPLORATORY CROSS-COMPONENT INTERACTION

  • The controlled benchmark’s measurement–correspondence interaction decreases as more source labels are changed, but initial real-data probes do not show the same decrease.A later check finds a decrease under target-label corruption, but not source rendering or source-column permutation; the authors treat the analysis as exploratory, not evidence for a general mechanism.Source: Figure A6 · p. 34, §E.4 · pp. 34–35

F.1 SELECTION AND CODING PROCEDURE

  • The retrospective audit retains qualifying semantic ablations, codes the claims made and what controls support, and assesses predictive-utility controls using Removal, Preservation, and Setup criteria.Preservation requires P1–P4 checks to pass, and Setup requires P5–P6; a failed required check makes a criterion unsupported, while unresolved required checks can make it indeterminate.Source: §F.1 · pp. 35–36

F.2 CODING AGREEMENT

  • The two coders agreed on 15 of 25 Preservation decisions (κ = .265), 105 of 150 fixed checks (70.0%), and 15 of 18 predictive-utility support labels (83.3%, κ = .693).Disagreements were reviewed against source passages and resolved by consensus; the final consensus codes are not treated as an inter-rater agreement measure.Source: §F.2 · p. 36

F.3 FULL AUDIT RESULTS

The audit distinguishes controls that support explicit claims from those that fail the criteria or remain indeterminate. Its detailed decisions turn on whether controls preserve other predictive information, input structure, and procedure.

F.3 FULL AUDIT RESULTS

  • Of 18 explicit predictive-utility claims, 1 control satisfies all three requirements, 11 fail at least one, and 6 are indeterminate; three of four content-sensitivity claims have supporting controls.These counts describe the audited comparisons, not the wider literature, and unsupported controls do not establish that the underlying semantic knowledge is unhelpful.Source: §F.3 · pp. 36–37, Table A20 · p. 37

F.4 DETAILED CODING DECISIONS

  • Several controls fail Preservation because they alter more than the tested semantics, including removing task descriptions and categorical-value verbalizations, changing template structure, or changing feature representations.For ConTextTab C08-E and C08-F, the published description does not establish whether the same learning procedure was used, leaving Setup indeterminate.Source: §F.4 · pp. 37–38
  • Other coding decisions identify altered numerical information as a confound, including discretizing continuous values before permutation; missing procedural details also leave some controls indeterminate.For FeatLLM C06-A, whether descriptions were also removed from the rule-parsing prompt is unreported; for TabSTAR C09-B, the effect on separate numerical input is unstated.Source: §F.4 · p. 38

F.5 SOURCE EVIDENCE FOR CODED CLAIMS

  • For every comparison coded as making an explicit or unclear predictive-utility claim, Table A21 records the interpretation used and its location in the original paper.Comparisons with no identified predictive-utility claim remain coded as having no such claim.Source: §F.5 · p. 38

G LIMITATIONS AND FUTURE WORK

The authors bound their conclusions by the settings and models examined, the availability of suitable references, and the scope of their retrospective audit. Supplementary materials also document a replication and available artifacts.

G LIMITATIONS AND FUTURE WORK

  • The experiments focus on semantic components identifiable from feature names, documentation, codebooks, known schema structure, or explicit relations; isolating components in opaque or high-dimensional representations is less straightforward.Predictive utility can vary with the chosen reference and with how readily the learner recovers removed information from other inputs or labeled target examples; not all models or representations are covered.Source: G LIMITATIONS AND FUTURE WORK · p. 38
  • The published-ablation analysis is a bounded audit, not a systematic review, and its counts are not intended to estimate how common particular evaluation practices are across the literature.The authors suggest a broader systematic review could examine these patterns across more methods and domains.Source: G LIMITATIONS AND FUTURE WORK · p. 38

H.1 SOFTWARE ENVIRONMENT AND DATA-ACCESS BOUNDARIES

  • The supplied one-thread replication reproduces all 400 comparable stored HistGradientBoosting nMSE values exactly, but does not claim cross-thread bitwise identity.A planned second HRS measurement–correspondence interaction experiment was not run because new authorization for licensed HRS microdata was required.Source: §H.1 · p. 39, Table A22 · p. 40

H.2 SUPPLEMENTARY ARTIFACT MAP

  • Table A23 maps the reproducibility artifacts available in the public repository.Source: Table A23 · p. 40

Related papers

Loading 2609.34098v1…