PaperlayerView exact revision on arXiv

Source-linked AI summary

N\"urnberg NLP at ChildSafeAds 2026: Structurally Dissimilar Voter Ensembles under Four Levels of Data Access

Philipp Steigerwald, Eric Rudolph, Jens Albrecht

arXiv:2609.34986v1cs.CLcs.LG

TL;DR

Monitoring sponsored content in child-facing YouTube videos is framed as a measurement problem across levels of data access. The paper uses per-subtask ensembles of nine structurally dissimilar voters, selected with channel-disjoint cross-validation. The system wins two subtasks, with the best ST2 and ST3 scores among 22 entries, and places third on the official mean.

  • Problem

    The task asks what a monitoring system can achieve when assessing commercial content in child-facing YouTube videos with different levels of data access, a setting where sponsored segments are barely monitored.

    Source: §1 · p. 1

  • Method

    The system combines nine structurally dissimilar voters per subtask, using cross-validation fold-models that vary by backbone, adaptation method, and class scope.

    Source: §1 · p. 1

  • Results

    The system wins two of three subtasks, with the best ST2 (0.8243) and ST3 (0.6530) scores among 22 entries, and places third on the official mean (0.7079).

    Source: §1 · p. 1

  • Takeaways & Limitations

    Across evaluated configurations, L2 raised pooled voter scores on every subtask; no L3 gain was measured, while L4 gains were concentrated in ST1 and ST2, with little added ST3 gain beyond L2.

    Source: §8 · p. 7

    The scores measure agreement with one operationalisation of the law, not legal compliance.

    Source: Limitations · p. 7

Abstract

from arXiv · show

We describe the N\"urnberg NLP system for ChildSafeAds 2026. The shared task asks what a monitoring system for commercial content in child-facing YouTube videos can achieve at a given level of data access. We answer with per-subtask ensembles of nine voters, organised into three branches that differ in backbone, adaptation method and class scope. Selection rests on channel-disjoint cross-validation, with the development set as a transfer check. The system wins two of the three subtasks. Its product-category score (ST2, 0.8243) and its compliance-flag score (ST3, 0.6530) are the best of the 22 final entries, and it places third on the task mean (0.7079). We further compare four access levels and report the cost at test-set scale.

1 Introduction

The paper frames monitoring commercial content in child-facing YouTube videos as a measurement problem across four levels of data access. It presents per-subtask ensembles of structurally dissimilar voters and reports the best ST2 and ST3 scores among 22 entries, with third place on the official mean.

1 Introduction

  • The task measures commercial type (ST1), product category (ST2), and EU consumer-law compliance risk flags (ST3) at four data-access levels.Inputs range from transcript alone (L1) through video context (L2) and channel name (L3) to the promoted product’s web page (L4); hidden test channels are disjoint from training channels.Source: §1 · p. 1
  • Each subtask uses an ensemble of nine voters, combining structurally dissimilar branches that differ in backbone, adaptation method, and class scope.Each fold-model casts one vote, and the branches may be generalists, subtask specialists, or minority-class specialists.Source: §1 · p. 1, Figure 1 · p. 1
  • ST2 (0.8243) and ST3 (0.6530) were the best scores among 22 entries, while the system placed third on the official mean (0.7079).It won two of the three subtasks; the gap to the winning entry lay entirely in ST1.Source: §1 · p. 1
  • The paper contributes an aggregated voter-level comparison across access levels, crosschecked against deployed ensembles, and a training-and-inference cost account for the default nine-voter architecture.It also contributes the leaderboard-best ST2 and ST3 systems.Source: §1 · pp. 1–2

2 Related Work

Prior computational work on sponsored content has largely focused on disclosure, with newer work addressing compliance and legal-label classification. The paper situates ChildSafeAds among research on advertising, minors, and legal NLP while distinguishing label classification from detecting a legal wrong.

2 Related Work

  • Computational sponsored-content research has mostly studied disclosure, while more recent work shifts toward compliance assessment.Gui et al. (2025a) report a marked performance drop on ambiguous advertising posts and identify cues often misidentified as hidden advertising.Source: §2 · p. 2
  • ChildSafeAds differs from post-based advertising studies by using crowd-sourced sponsored-segment identification and spoken transcripts with metadata rather than captions.Research on minors has focused more on harmful than commercial content, while children’s advertising exposure on video platforms is mainly documented through measurement studies.Source: §2 · p. 2
  • Legal-NLP precedents classify expert-designed legal labels, but accurate label classification does not itself establish detection of a legal wrong.The passage cites terms-of-service, cookie-banner, and dark-pattern work as related examples, and notes ethical limits in legal NLP.Source: §2 · p. 2

3 Task and Data

The task uses channel-disjoint training, development, and test splits of sponsored segments from child-facing YouTube channels. It labels each segment on three axes and provides four cumulative levels of input access.

3 Task and Data

  • The corpus contains 3,360 sponsored segments, with one segment per video, split by channel into 2,353 training, 504 development, and 503 test datapoints.The splits contain 632, 154, and 153 channels respectively, and test labels are withheld.Source: §3 · p. 2

3.1 Subtasks and Labels

  • ST1 assigns one of five commercial types, ST2 assigns 12 product categories, and ST3 assigns eight compliance-risk flags to the same segment.Label sets are steeply skewed, and ST3 includes standalone and mutually exclusive labels that systems must respect.Source: §3.1 · p. 2

3.2 Access Levels

  • The four cumulative access levels add fields in sequence: transcript (L1), video context (L2), channel name (L3), and promoted-product web-page text (L4).At full access every datapoint fits within 8,192 tokens without truncation; the maximum is 8,039 tokens.Source: §3.2 · pp. 2–3

4 System

The system builds voters across backbone, adaptation-method, class-scope, and access-level choices, then deploys selected fold-models in three branches. Its design combines channel-disjoint cross-validation with majority voting and task-specific taxonomy rules.

4.1 Backbones and Adaptation Methods

  • Untrained-backbone methods trailed the best trained method by 0.15, 0.07, and 0.15 F1 top3 cv on ST1, ST2, and ST3, respectively.Only voters whose backbone had seen the task reached a deployed ensemble.Source: §4.1 · p. 3

4.2 Class Scopes

  • G trains jointly on all three subtasks, S on one, and MCS on one subtask after removing its most frequent labels.For ST1, MCS removes the two dominant classes; for ST2 and ST3 it removes label columns while retaining majority-only datapoints as all-negative training signal.Source: §4.2 · p. 3

4.3 Cross-Validation and Branches

  • Channel-disjoint CV5 scores each configuration on held-out folds; its three best folds form a branch, with branches collectively covering all five folds.Three fold-models are deployed per branch, mainly to reduce the cost from fifteen voters to nine.Source: §4.3 · pp. 3–4

4.4 Vote Aggregation

  • Each of three branches contributes three votes, and labels reaching five of nine votes enter the raw prediction set.On ST1, classes compete for votes; on ST2 and ST3, each label has an independent five-vote threshold.Source: §4.4 · p. 4
  • The ensemble must span at least two backbones and both generative and discriminative paradigms; later submissions favor full-label G and S branches over MCS.An MCS cannot name majority labels, so at the five-vote threshold it can arbitrate only where the full-label branches disagree.Source: §4.4 · p. 4

4.5 Taxonomy Rules

  • ST1 falls back to physical_goods when no class reaches five votes; ST2 leaves the vote set unchanged, while ST3 repairs taxonomy conflicts.For ST3, no_flag and insufficient_context stand alone, and the two disclosure flags exclude each other.Source: §4.5 · p. 4

5 Submissions and Results

The system’s final entry placed third on the official mean and led the field on ST2 and ST3, while falling short on ST1. The authors selected submissions using cross-validation and treated the development set as a transfer check.

5.1 Submission Strategy

  • The final upload combined the best already-scored subtask fields: ST1 and ST3 from Submission 4 and ST2 from Submission 3.All five uploads were used, and ensembles were hand-picked from branches with the highest F1 top3 cv, with one development-set exception.Source: §5 · p. 4, §5.1 · p. 5

5.2 Leaderboard and the ST1 Gap

  • The entry had the best ST2, ST3, and ST3-family scores among 22 final entries and placed third on the official mean at 0.7079.It led the best other entry by +0.021 on ST2 and +0.038 on ST3, but fell 0.203 short of the best ST1 entry; the authors have not established why.Source: §5.2 · p. 5

5.3 Validation on the Hidden Test

  • Development-set rankings correlated only r=0.06 on ST1, 0.03 on ST2, and 0.37 on ST3 across 300 candidates, so the authors treated development as a transfer check.Submission 3’s development-picked ST1 ensemble scored best on development and worst on test at 0.5339.Source: §5.3 · pp. 5–6
  • The four-vote ST3 threshold that improved development performance scored 0.6483 on test, below Submission 2’s 0.6512.Refitting over training and development helped ST1 by 0.026 and ST3 by 0.002, but cost 0.049 on ST2.Source: §5.3 · p. 6

6 What Each Access Level Contributes

Across pooled voter-level comparisons, added access does not produce a monotonic score increase: L2 and L4 improve the mean, while L3 is near zero. The deployed ensemble shows a different ordering of the L2 and L4 gains.

6 What Each Access Level Contributes

  • The pooled mean changes by +0.033 at L2, −0.006 at L3, and +0.072 at L4.Table 3 pools complete G and full-label S configurations by access level and reports 95% t-confidence intervals.Source: Table 3 · p. 6, §6 · p. 6
  • L4 gains are concentrated in ST1 (+0.080) and ST2 (+0.116), while ST3 rises by only 0.013.Successive L1–L3 intervals overlap, but the L123 and L1234 intervals do not; a matched comparison reproduces the ordering (+0.026, −0.010, +0.073). The authors treat these heterogeneous-configuration results as descriptive, not causal.Source: §6 · p. 6
  • On the deployed ensemble, L2 gives +0.074 [+0.024, +0.106], L3 +0.005 [−0.011, +0.019], and L4 +0.031 [+0.013, +0.050].These development-set comparisons use fifty best ensembles per level and a paired bootstrap over 154 channels with 2,000 draws; L2 and L4 change places because nine L12 votes already recover much of what one L1234 voter holds alone.Source: §6 · p. 6

7 Cost at Test-Set Scale

The cost accounting covers the selected nine-voter Submission 2 system, distinguishing training across fold-models from test-scale inference on deployed folds.

7 Cost at Test-Set Scale

  • Submission 2 trains the fold-models for three retained configurations per subtask across all five CV folds, including reused backbone training but excluding CPU-only head fits.The reported training column sums logged one-GPU wall time; runs used mixed A100, H200, and L40S accelerators, so totals are raw GPU-hours.Source: §7 · p. 6
  • Inference deploys the selected three folds of each configuration, for nine voters per subtask, with costs reconstructed on one A100-80GB.Measured single-pass times are approximately 2.02 s per segment for frozen Phi-4, 0.25 s for frozen ettin-1b, and 0.52 s for a Ministral-8B pass; no step uses a paid API.Source: §7 · pp. 6–7
  • Table 4 reports training and test-scale inference costs for Submission 2, excluding the twelve-voter ST2 extension.That extension’s hidden-test gain was only 0.0039.Source: §7 · p. 6, Table 4 · p. 6

8 Conclusion

The system’s nine structurally dissimilar voters win two subtasks and place third on the mean, while ST1 remains weak. The access-level comparisons are descriptive, and full-access inference labels the test set in under three GPU-hours on one A100.

8 Conclusion

  • The system wins two of three subtasks with field-best ST2 and ST3 scores, but places third on the mean because ST1 is weak.The conclusion says the gap to the winning entry lies entirely in ST1 and leaves explaining that weakness for future work.Source: §8 · p. 7
  • Video context (L2) raised pooled voter scores on every subtask, while the system measured no gain from channel name (L3).Product-page text (L4) gave voters the largest gain, concentrated in ST1 and ST2; ST3 gained little beyond L2. These findings are descriptive, not causal estimates.Source: §8 · p. 7
  • At full access, the ensemble labels the 503-datapoint test set in under three GPU-hours on one A100.Source: §8 · p. 7
  • Cross-validation was a pragmatic selector because it scored candidates on 2,353 training datapoints through held-out folds, versus 504 in development.The authors do not claim it is demonstrably better than development-set selection; the development set could not separate the top candidates.Source: §8 · p. 7
  • Strong ST2 and ST3 selections did not compensate for a weak ST1 candidate pool.The conclusion frames selection quality as insufficient to make up for weak ST1 candidates.Source: §8 · p. 7

Limitations

The access-level comparison is descriptive rather than a controlled ablation, and its confidence intervals and internal CV scores have important limits. The paper also leaves ensemble gains over individual voters unmeasured and bounds its claims to a particular language, platform, and flag taxonomy.

Limitations

  • The access-level comparison aggregates heterogeneous configurations, so its results are descriptive rather than a controlled ablation.Its t-confidence intervals assume independent observations, but shared folds and related configurations break that assumption; the matched comparison is a robustness check, not an independent test set.Source: Limitations · p. 7
  • Internal F1 top3 cv values are optimistic because thresholds are tuned on the folds that score them.They serve as branch-selection signals, not estimates of hidden-test performance.Source: Limitations · p. 7
  • No submission went to a single-voter baseline, leaving the ensemble’s test-set gain over its best member unmeasured.The paper also did not measure this gain on cross-validation or test whether plurality voting improves ST1 in validation.Source: Limitations · p. 7
  • The text pipeline cannot observe visual presentation and timing, which affect whether a child recognizes a disclosure as advertising.The scores measure agreement with one operationalisation of the law, not compliance itself.Source: Limitations · p. 7
  • Results are limited to English-language YouTube content and the paper’s flag taxonomy.The development set’s 504 datapoints also bound what its low split-half reliabilities can certify.Source: Limitations · p. 7

Ethics Statement

The authors bound how the benchmark’s compliance labels should be interpreted and depict how access-limited inputs feed the deployed ensemble. Figure 5 supplies the task-specific values behind Table 3.

Ethics Statement

  • ST1 scores can lose a fifth of their value to one rare class, so the authors say results should inform human review, not replace it.The compliance labels are a research benchmark, not legal advice or a finding of unlawfulness; reported figures are aggregate and no test datapoint was manually labelled.Source: Ethics Statement · p. 7

A The System End to End

  • Figure 4 traces a sponsored segment and its access level through the voter pool and cross-validation to a deployed ensemble of three branches of three fold-models.The branches’ votes produce one commercial type, product categories, and compliance flags.Source: Figure 4 · p. 10, A The System End to End · p. 10

B Complete Access-Level Results

C Trained and Untrained Methods

The section compares trained and untrained methods, describes the task-specific prompt rules, and specifies training and threshold choices. At full access, every trained method beats every untrained one on every subtask.

D Prompts

  • Task prompts add subtask-specific decision rules and verbatim official taxonomy definitions; OPRO searches system messages by subtask, class scope, and access level.Its search is scored on a class-balanced training sample, not the development set or scored fold.Source: D Prompts · p. 10
  • Figure 5 reports voter-level F1 top3 cv by access level, per subtask and pooled, with whiskers showing 95% t-CIs.Source: Figure 5 · p. 11

C Trained and Untrained Methods

  • At full access, every trained method beats every untrained one on every subtask; OPRO’s best ST3 across all levels is 0.373 (L123).Figure 6 compares the best fully finished five-fold configuration of each training method at full access.Source: C Trained and Untrained Methods · p. 10, Figure 6 · p. 11

E Training and Thresholds

  • Decoder voters use 4-bit NF4 QLoRA and encoders train all weights; thresholds are tuned per label on each fold-model’s held-out fold, with no firing for labels lacking validation positives.ST1 fold-models use argmax.Source: E Training and Thresholds · p. 11
Loading 2609.34986v1…