Source-linked AI summary
Adversarial Attacks on Tables with Entity Swap
Aneta Koleva, Martin Ringsquandl, Volker Tresp
TL;DR
CTA evaluation datasets contain entity leakage, while perturbation vulnerability in tabular language models has not been investigated. The paper proposes a black-box, similarity-based entity-swap attack and reports up to a 70% performance drop when all entities in a column are replaced.
Problem
Entity leakage in CTA evaluation datasets complicates assessment on unseen entities, and tabular language models' sensitivity to perturbations has not been investigated.
Source: §1 · p. 1Method
The paper proposes a black-box CTA attack that swaps column entities for similar-sampled, unseen entities of the same semantic type.
Source: §1 · p. 1, §3.1 · p. 2, §3 · p. 2Results
70% drop in performance is reported when replacing all entities in a targeted column with the most dissimilar same-type entities.
Source: §1 · p. 1Takeaways & Limitations
The evaluation finds that similarity-guided entity modifications can substantially change TaLM predictions and affect F1 score.
Source: §5 · p. 4The attack's imperceptibility condition requires swapped entities to retain the column's most specific class.
Source: §3 · p. 2
Abstract
from arXiv · showhide
The capabilities of large language models (LLMs) have been successfully applied in the context of table representation learning. The recently proposed tabular language models have reported state-of-the-art results across various tasks for table interpretation. However, a closer look into the datasets commonly used for evaluation reveals an entity leakage from the train set into the test set. Motivated by this observation, we explore adversarial attacks that represent a more realistic inference setup. Adversarial attacks on text have been shown to greatly affect the performance of LLMs, but currently, there are no attacks targeting tabular language models. In this paper, we propose an evasive entity-swap attack for the column type annotation (CTA) task. Our CTA attack is the first black-box attack on tables, where we employ a similarity-based sampling strategy to generate adversarial examples. The experimental results show that the proposed attack generates up to a 70% drop in performance.
1. Introduction
Tabular language models perform strongly on table interpretation, but their robustness to adversarial perturbations and their generalization to unseen entities remain unclear. The introduction motivates an entity-swap attack with entity leakage in CTA evaluation datasets and reports substantial performance drops under swapping.
1. Introduction
- TaLMs are state-of-the-art for table interpretation, but their sensitivity to input perturbations and the perturbations that remain imperceptible are unstudied.The introduction identifies table-to-class annotation, entity linking, and CTA as example tasks.Source: §1 · p. 1
- WikiTables and VizNet CTA evaluations have train-to-test entity leakage, making generalization to unseen entities difficult to judge from existing high F1 scores.The paper frames entity swapping as an adversarial setup involving new, unseen entities of the same semantic type.Source: §1 · p. 1
- The last 15 WikiTables types have 100% entity overlap between training and test sets.Table 1 reports entity overlap per type; the supplied text gives no overlap values for the top five classes.Source: §1 · p. 1, Table 1 · p. 1
- 6% to 70% drop in performance is reported when swapping 20% to all entities in targeted columns using the most dissimilar same-type entities.The evaluation varies the swapped share from 20% to 100%.Source: §1 · p. 1
2. Related Work
Prior work has examined adversarial attacks on language models, while vulnerabilities in tabular data and tabular language models remain largely unexplored. The paper situates its work against text attacks and attacks on traditional tabular models.
2. Related Work
- Text attacks have been studied, but tabular data remained unexplored for adversarial vulnerability research.The related work notes challenges in attacking discrete text and describes attacks such as BERT-Attack and a gradient-based method.Source: §2 · p. 2
- Prior attacks on tabular data target traditional machine-learning models and commonly evaluate numerical datasets such as financial or healthcare data [11].Those works aim to preserve feature-distribution consistency when generating adversarial examples.Source: §2 · p. 2
- This paper targets tabular language models used for table interpretation, which the authors identify as a first for adversarial attacks on these models.The paper distinguishes its work by model, adversarial-sample generation technique, and evaluation task.Source: §2 · p. 2
3. CTA Adversarial Attack
The section formalizes CTA as multilabel semantic-type assignment to a table column and defines an attack that changes predicted labels while preserving entity types. Its entity-swap procedure selects important column entries and replaces them with same-class, dissimilar candidates using prediction scores.
3. CTA Adversarial Attack
- A CTA attack transforms a correctly classified table-column input so its predicted labels no longer overlap, while swapped entities retain the column’s semantic class.The attack is black-box: it uses classifier prediction scores.Source: §3 · p. 2, §3.1 · p. 2
3.1. Entity Swap Attack
- Entity-swap selects a controllable percentage of key entities in the target column, then replaces each with an adversarial entity likely to change the prediction.The attack accesses classifier prediction scores rather than model internals.Source: §3.1 · p. 2
3.2. Key Entities
- Each entity’s importance is the ground-truth-class logit difference between the original column and a version with that entity masked; the highest-scoring percentage is swapped.For the multilabel task, the importance score takes the maximum across ground-truth classes.Source: §3.2 · pp. 2–3, Figure 2 · p. 3
3.3. Adversarial Entities
- Adversarial value candidates share the attacked column’s most specific class, and an embedding model selects the candidate least similar to each key entity.Candidate pools are WikiTables test entities or a filtered set excluding entities that appear in training.Source: §3.3 · p. 3
4. Evaluation
The evaluation tests entity-swap attacks on TURL for CTA, comparing importance-based entity selection and similarity-based adversarial sampling with random alternatives. Increasing the share of swapped entities reduces performance, and perturbing column names also degrades the model’s metrics.
4. Evaluation
- The evaluation fine-tunes TURL for CTA on WikiTables and reports F1 score, precision, and recall while comparing proposed sampling and selection strategies with random alternatives.The experiments assess the influence of adversarial-entity sampling and the choice of entities to swap.Source: §4 · p. 3
4.1. Results
- As the share of swapped entities increases, TURL’s performance drops even when replacements retain the original entities’ semantic type.The reported F1 decline is attributed to a sharp recall decline.Source: §4.1 · pp. 3–4, Table 2 · p. 4
- Importance-based entity selection produces an F1-score drop around 3% higher than random selection, consistently at 20% and 80% swaps.The authors interpret this consistency as evidence that the scores identify entities with greater influence on model performance.Source: Figure 3 · p. 4, §4.1 · p. 4
- Similarity-based sampling produces a sharper F1-score drop than random sampling for adversarial entities drawn from either the test set or the filtered set.Figure 4 compares both sampling strategies across those two candidate sources.Source: Figure 4 · p. 4, §4.1 · p. 4
- Perturbing column names with synonyms causes all evaluation metrics to decline as the percentage of perturbed names increases.This attack targets TURL’s use of table metadata.Source: Table 3 · p. 4, §4.1 · p. 4
5. Conclusion
The paper formalizes adversarial attacks on tabular language models and identifies an issue in CTA evaluation. Its evaluation finds susceptibility to similarity-guided entity changes, and future work will extend the attacks to other table-interpretation models.
5. Conclusion
- The paper formalizes an adversarial attack targeting tabular language models and identifies an issue in CTA evaluation.The conclusion does not specify the evaluation issue in this passage.Source: §5 · p. 4
- The evaluation finds that tabular language models are susceptible to adversarial attacks, with similarity-guided entity modifications changing predictions and affecting F1 score.Source: §5 · p. 4
- Future work will extend evaluation to more sophisticated attacks and other models used for table interpretation tasks.Source: §5 · p. 4