Source-linked AI summary
Discrete Diffusion Language Models Are Training-Free Multi-Label Classifiers
Pawan Kumar
TL;DR
Multi-label text classification with masked-diffusion language models faces slot-position artefacts and scaling limits for large label spaces. dLLM-SetScore addresses these issues with per-label entailment scoring and shortlist-based evaluation, while formalizing permutation invariance and decision ceilings. Instruct checkpoints outperform Base checkpoints across most shared evaluation cells, although the source of this improvement remains unexplained.
Problem
Masked-diffusion language models had not been extended straightforwardly to multi-label text classification, where long all-masked answer suffixes can create training-distribution artefacts and label subsets grow exponentially.
Method
dLLM-SetScore asks a yes/no question for each candidate label, compares the two answer-token probabilities at one masked position, and uses per-label scoring with a shared shortlist for large label spaces.
Results
Instruct checkpoints outperform corresponding Base checkpoints on 9 of 10 shared macro-F1 cells and 8 of 10 shared micro-F1 cells across LLaDA and Dream evaluations.
Takeaways & Limitations
Per-label entailment scoring removes slot-position asymmetry through permutation invariance, while shortlist retrieval imposes recall and F1 ceilings for prompt-based methods.
Takeaways & Limitations
The study documents Base-to-Instruct differences but does not provide a mechanistic explanation for why instruction tuning improves discriminative quality.
Abstract
from arXiv · showhide
We present dLLM-SetScore, a training-free method that uses discrete masked-diffusion language models for multi-label text classification. For each candidate label, it asks a short yes/no question and compares the probabilities of the two answer tokens at one masked position. The method uses no task-specific fine-tuning or training on textual-entailment datasets; a 200-example labelled validation slice selects thresholds, temperature, and prompt wording. We first show that placing all labels in one prompt creates a strong slot-position asymmetry: the first answer slot is predicted positive on $99.4\%$ of GoEmotions examples and $100\%$ of Reuters examples. Per-label scoring places every label in the same syntactic position, making predictions invariant to label order and avoiding this artifact. We evaluate LLaDA-8B and Dream-7B on six datasets against NLI models, an autoregressive LLM, SetFit, and supervised classifiers. On the five datasets shared by both diffusion families, Instruct checkpoints improve macro-F1 in 9 of 10 comparisons and micro-F1 in 8 of 10, although these comparisons do not identify the cause. Within our protocol, LLaDA-Instruct records the highest training-free values for both Reuters and ECtHR metrics. We prove permutation invariance, characterize thresholded decisions under weighted Hamming loss, and derive shortlist ceilings for recall and F1. An exploratory local Joint Set Refinement step lowers F1 from biased and unbiased initializations and is retained as a negative result.
1 Introduction.
dLLM-SetScore applies discrete masked-diffusion language models to multi-label text classification without task-specific fine-tuning, using short per-label yes/no prompts. Per-label scoring removes the slot-position asymmetry of all-masked multi-slot scoring, while the paper also studies shortlist constraints and Joint Set Refinement.
- Motivation: Multi-label text classification requires selecting among 2^m label subsets, while masked-diffusion language models can suffer artefacts from long all-masked answer suffixes.The paper extends diffusion-based classification from prior single-label image settings to multi-label text.
- Method: dLLM-SetScore uses no task-specific fine-tuning or textual-entailment training; a 200-example validation slice selects thresholds, temperature, and prompt wording.Each candidate label receives a short yes/no query scored at one masked answer position.
- Failure mode: 99.4% of GoEmotions examples and 100% of Reuters examples receive positive predictions in the alphabetically-first slot under all-masked multi-slot scoring.The paper attributes this collapse to a mismatch between masked-diffusion training and long all-mask suffixes.
- Method: Per-label entailment scoring queries every label in the same syntactic position, making predictions permutation-invariant to label ordering and removing slot-position asymmetry.Prompt-template, lexical, and label-description biases remain possible.
- Refinement: Local Joint Set Refinement is not guaranteed to improve the pseudo-likelihood surrogate, and the full-gain variant was not run in the main experiments because of extra query costs.The passages characterize JSR design and computational limitations rather than presenting it as an improvement.
- Shortlisting: Shortlisting scales scoring linearly in shortlist size, but EURLEX57K retains 31.2% of gold labels with k=32, capping prompt-based micro-F1 at approximately 47%.This retriever ceiling affects prompt-based methods that score only the shortlist.
4 Theoretical Properties of Per-Label Scoring.
The theoretical analysis establishes permutation invariance for per-label scoring, decision properties under threshold-matched weighted Hamming loss, and upper bounds imposed by label shortlists. It complements rather than replaces the empirical evaluation, and does not establish optimality for micro- or macro-F1.
- Permutation invariance: Per-label scores remain unchanged under label permutations when each label retains its threshold, ruling out dependence on global answer-slot index.The result assumes a prompt constructor and deterministic score extractor applied independently to document-label pairs.
- Permutation invariance: Permutation invariance excludes slot-position asymmetry but not lexical biases from label text or biases introduced by the prompt template.These remaining biases are studied empirically elsewhere in the paper.
- Weighted Hamming loss: Threshold-matched weighted Hamming loss assigns label-specific costs to false positives and false negatives, with ordinary Hamming risk recovered when all thresholds equal one half.The loss is defined coordinatewise across labels.
- Bayes optimality: The Bayes-optimal coordinate decision predicts a label according to whether its conditional positive probability exceeds its threshold.The theorem also provides pointwise stability and an excess-risk bound controlled by average probability-estimation error.
- Shortlist ceilings: Shortlists impose upper bounds on recall, micro-F1, and macro-F1 whenever labels outside the shortlist are forced negative.The bounds depend on positive-label retention, including per-label retention rates.
- Shortlist ceilings: Shortlisting contributes a retriever-only Hamming-risk term for positives excluded from the shortlist, separating retrieval error from scorer-controlled error.This formalizes an irreducible error component determined by retrieval.
- Scope: The theory does not prove optimality for micro- or macro-F1 and does not analyze all-masked scoring or local JSR; empirical results remain primary.The theoretical scope is the per-label scorer and its structural properties.
5 Experiments.
Experiments compare training-free diffusion classifiers and baselines across six datasets, while diagnosing positional asymmetry, checkpoint effects, prompt tuning, shortlist limits, and refinement failures.
- Setup: The evaluation covers six datasets, two diffusion backbone families, NLI and autoregressive baselines, SetFit, and supervised reference systems.EURLEX57K uses a 32-label SBERT shortlist, whereas AAPD evaluates the full 54-label inventory.
- Checkpoint comparison: Instruct checkpoints improve macro-F1 in 9 of 10 shared dataset-family cells and micro-F1 in 8 of 10.The regressions occur on Dream Reuters micro-F1 and EURLEX micro/macro-F1; AAPD deltas are treated as within noise.
- Positional asymmetry: Slot 1 is predicted positive for 99.4% of GoEmotions inputs and 100% of Reuters inputs under all-masked scoring.On Reuters, this collapse yields 41.6 micro-F1 / 10.9 macro-F1; per-label scoring recovers macro-F1 to 38.2 with essentially unchanged micro-F1.
- Limitations and ablations: EURLEX shortlist recall is 31.2%, structurally capping prompt-based micro-F1 at 47, while local JSR degrades test F1 from biased and unbiased seeds.The paper therefore recommends per-label scoring alone and treats JSR as a negative result.
- Positional asymmetry: Per-label scoring is permutation-invariant, whereas reordering labels moves the all-masked collapse to the new first slot.Every label is queried in the same syntactic position, removing this specific positional artefact.
6 Discussion and Conclusion.
The paper identifies per-label entailment scoring and Instruct checkpoints as central practical choices, while formalizing key properties and documenting important limitations.
- Theorems formalize permutation invariance, Bayes-optimal decisions under weighted Hamming loss, and retrieval-imposed ceilings without replacing empirical claims.
- Use per-label entailment scoring as the default training-free classifier and prefer Instruct over Base.
- Shortlisting can cap recall; EURLEX57K at k=32 has 31.2% recall.
- Local-JSR is retained as a negative result because the evaluated refinement is harmful.
1 Introduction.
The method uses per-label masked-diffusion yes/no scoring with validation-only calibration, while addressing positional asymmetry and implementation constraints.
- dLLM-SetScore queries each candidate label in a short yes/no prompt and converts masked-answer log-odds into calibrated binary predictions.
- Per-label construction avoids the slot-position asymmetry observed with long all-masked answer suffixes.
- The all-masked unary stage uses one document prompt with one masked answer slot per label, whereas per-label scoring uses one document-label prompt.
- Documents are truncated to 600 tokens, preserving over 95% of examples except in EURLEX57K and ECtHR Task A.
- A 200-example validation slice selects threshold strategy, temperature, and prompt template, while diffusion forward passes remain deterministic under fixed weights and masking.
3 Detailed dataset descriptions.
The study spans six multi-label benchmarks across domains, label cardinalities, and document lengths, with evaluation choices that affect interpretation of results.
- EURLEX57K uses an SBERT k=32 shortlist, whereas AAPD uses its full 54-label inventory.
- The six benchmarks cover emotion, news, EU legal, human-rights legal, toxicity, and academic domains with 6 to 100 labels.
- Micro-F1 weights every document-label decision equally and is dominated by frequent labels, while macro-F1 weights labels equally.
- Reuters unary scoring obtains micro 41.6 and macro 10.9, while per-label entailment raises macro-F1 to 38.2 with essentially unchanged micro-F1.
- LLaDA-Instruct differs from Base through additional supervised instruction-following fine-tuning, while Dream is adapted from Qwen2.5-7B with masked-diffusion fine-tuning.
6 Some Remarks.
The remarks clarify how the method compares with other classifiers, where it fails or depends on design choices, and which limitations remain open.
- LLaDA-8B-Instruct has higher macro-F1 than Qwen2.5-7B-Instruct on Reuters, ECtHR, and Jigsaw, while Qwen scores higher on GoEmotions and Reuters micro-F1.
- Per-label scoring uses masked denoising and instruction-following supervision rather than task-specific NLI training.
- Long all-mask suffixes create positional bias because they differ from the model’s richer-context training distribution; per-label prompts place labels identically.
- Two JSR sweeps reduce GoEmotions micro-F1 from 26.79 to 19.78 and Reuters micro-F1 from 60.70 to 47.52 from an unbiased per-label seed.
- On Dream-7B, Instruct improves macro-F1 over Base on 4 of 5 datasets, but two checkpoint families cannot establish a general property.
- EURLEX retrieval recall is 31.2%, creating a reported micro-F1 ceiling of about 47% for shortlist-restricted methods.
- Reuters template tuning boosts micro-F1 from 60.6 to 80.5 using a 200-example validation slice.
- The method costs about 117 ms per example and 15.7 GB of GPU memory, compared with BART-MNLI’s 101 ms and 2.6 GB.
7 Negative results and what we think went wrong.
The paper records several unsuccessful alternatives and attributes them to prompt bias, out-of-distribution joint contexts, decoding limits, backbone incompatibilities, and shortlist constraints. It retains local-JSR only as a negative result while identifying retriever quality, calibration, and controlled checkpoint studies as future directions.
- Joint Set Refinement: JSR reduced GoEmotions micro-F1 from 17.16 at S=0 to 8.89 after three sweeps from an all-masked unary seed.With per-label entailment initialization, GoEmotions micro-F1 fell from 26.79 to 19.78 and Reuters from 60.70 to 47.52 after two sweeps.
- Masking variants: Permutation averaging did not reliably remove slot bias: at P=2, GoEmotions fell to 7.6 / 6.7 and Reuters to 19.1 / 19.3.The paper attributes this to too few random permutations to cancel the biased first slot.
- Direct set decoding: Greedy direct label-set decoding underperformed per-label entailment on GoEmotions, Reuters, and Jigsaw because of token repetition and label-name fragmentation.The attempted decoder used 64 masked tokens after a prompt listing candidate labels.
- Text-to-set baselines: T5 text-to-set decoding failed with OverflowError on EURLEX, ECtHR, Jigsaw, and AAPD, where long or rare-label outputs exceeded generation limits.Decoding completed only on GoEmotions and Reuters; chunked decoding or label-id prediction would be needed.
- Backbone compatibility: BD3-LMs could load but required clean x_0 inputs unavailable to the adapter, showing that masked-diffusion checkpoints are not interchangeable.MDLM-OWT also encountered an unconditional flash_attn failure on sm_120 silicon.
- Future directions: EURLEX shortlist recall was 31.2% at k=32, creating a ceiling that stronger label encoders or joint retriever–scorer training might raise.AAPD uses the full label inventory and therefore has no retrieval ceiling.
10 Comparison of masking variants.
The comparison distinguishes all-masked, permutation-averaged, and per-label scoring, with the recommended per-label layout removing dependence on label order. The theory covers permutation invariance, thresholded decisions, weighted Hamming risk, and shortlist ceilings, while local-JSR remains exploratory.
- Scoring variants: Per-label entailment queries each label in the same syntactic position, making predictions invariant to label ordering and free of slot-position asymmetry.The paper contrasts this with all-masked unary and permutation-averaged unary scoring.
- All-masked unary: All-masked unary builds an ordered prompt, samples mask sets over answer slots, and aggregates verbalizer evidence into real-valued log-odds.Thresholding is a separate calibration step.
- Per-label entailment: Per-label entailment computes one yes-versus-no log-odds score for each document–label prompt before calibration.The recommended algorithm returns a score vector, with temperature and thresholds tuned on the validation slice.
- Risk analysis: Under threshold-matched weighted Hamming loss, the thresholded decision is analyzed label-wise and its conditional excess risk is bounded by probability-estimation error.The ordinary Hamming-risk result is expressed through an expected absolute estimation-error bound.
- Shortlist ceilings: Shortlisting imposes recall and F1 ceilings because labels excluded from Λ(x) are always predicted negative.The macro-F1 ceiling is obtained label-wise and then averaged over positive labels.
- Shortlist ceilings: The Hamming-risk decomposition isolates errors caused by the retriever when a positive label is excluded from the shortlist.This separates retrieval limitations from downstream scoring errors.
13 Extended results matrix.
The extended matrix shows that no training-free method dominates all six datasets, while supervised rows remain higher on most datasets. Results also depend on prompt choice, checkpoint choice, and whether evaluation uses deployable or oracle settings.
- Overall comparison: Table 9 reports micro-F1 / macro-F1 percentages for all method variants, with bold entries marking the best training-free method per column.The convex-blend block is an oracle evaluation-slice analysis rather than a deployable setting.
- Prompt selection: On Jigsaw, the LLaDA-I per-label best-prompt operating point is 45.1/28.5 for the micro-best template, whereas the macro-best template gives 43.5/30.2.The split bests are not one operating point.
- Overall comparison: No single training-free method achieves the highest score on all six datasets.BART-MNLI leads Reuters before prompt-tuned diffusion scoring, while SetFit leads EURLEX57K and AAPD.
- Checkpoint and prompt effects: The largest LLaDA Base-to-Instruct differences occur on Reuters, ECtHR, and Jigsaw, but the paper does not establish their mechanism.Prompt wording can also encode dataset-specific relations, as illustrated by the Reuters “main topic” row.
14 Supervised baselines.
Supervised encoders and text-to-set models provide reference points for the remaining gap to task-specific training. Their strengths vary by dataset, and T5 decoding fails on four datasets rather than yielding zero scores.
- Supervised reference points: Table 10 compares BERT-base, RoBERTa-base, and T5 text-to-set as supervised upper bounds trained with task labels.BERT and RoBERTa use independent sigmoid heads, while T5 generates label sets as text.
- Encoder baselines: RoBERTa is strongest on Reuters and ECtHR, whereas BERT is slightly better on Jigsaw and AAPD micro-F1.The supervised models still show low macro-F1 on GoEmotions and EURLEX, indicating rare-label thresholding remains difficult.
- T5 text-to-set: T5 is strongest on the two datasets where decoding completed, but its missing entries reflect OverflowError failures on EURLEX, ECtHR, Jigsaw, and AAPD.The failures occur during label-set generation rather than representing zero scores.
- Calibration: Oracle label-wise calibration is not test performance; its gap from honest global-threshold calibration measures ranking signal lost during binary decisions.The largest single headroom occurs for LLaDA per-label entailment.
- Calibration: +3 to +11 micro-F1 of representative calibration headroom suggests that global thresholds tuned on 200 validation examples leave useful score signal unused.These values compare honest validation-tuned global thresholds with oracle test-tuned label-wise thresholds.
- Per-label effects: GoEmotions gains are largest for lexically cued labels, including 62 F1 points for gratitude and 45 for love, but are not uniform across emotions.Regressions occur among nearby positive-affect categories.
- Per-label effects: Reuters macro improvement comes mainly from better coverage of rare commodity and macroeconomic topics, while the frequent earn category remains at 84 F1.The tail labels include soybean, vegetable oil, livestock, sugar, and GNP.
17 Prompt sweep (full).
Prompt wording materially changes per-label classification, while calibration and architecture comparisons reveal dataset-dependent behavior. The analysis also shows that shortlist recall and inference cost constrain prompt-based methods, and that per-label rates diagnose bias without measuring example-level correctness.
- 17 Prompt sweep (full).: Each prompt-sweep row is a separate paired measurement, so Jigsaw’s best micro- and macro-F1 values must not be combined.The selected template should be chosen using validation performance because the sweep varies only the question relation.
- 18 Multi-seed reproducibility.: GoEmotions and ECtHR Instruct results are seed-stable, while Reuters varies by approximately 1.9 micro-F1 points on the default template and 2.7 on the tuned variant.The zero standard deviation for GoEmotions and ECtHR means all three calibrated runs produced the same final score.
- 19 Dream-7B replication.: Dream-Instruct improves both metrics on GoEmotions, ECtHR, and Jigsaw, but declines on EURLEX and loses AAPD micro-F1; this does not establish a universal causal instruction-tuning effect.On Reuters it trades 1.2 micro-F1 points for a 6.7-point macro gain.
- 20 EURLEX shortlist recall analysis.: EURLEX’s SBERT k=32 shortlist retains 31.2% of gold labels per document and imposes an approximately 47% micro-F1 ceiling on shortlist-restricted methods.The ceiling applies before language-model scoring or thresholding; SetFit avoids it by using the full 100-label inventory.
- 21 Reuters per-label positive-rate plot.: Instruct reduces Reuters mean absolute per-label rate error from 10.45% to 5.54%, moving predictions closer to gold across common and rare labels.These marginal rates explain macro-F1 differences but do not measure example-level correctness; the largest remaining mismatch is trade.
- 22 Accuracy–latency Pareto front.: Supervised encoders and T5 occupy the low-latency, high-micro-F1 edge, whereas JSR costs roughly 1.3–2.7 seconds per example without enough accuracy recovery to justify that cost.Per-label LLaDA and permutation-averaged unary move farther right because they require many masked-token queries.
23 Verbatim prompt templates.
The appendix specifies the exact prompt forms, verbalizers, evaluation assets, and experimental caveats. It documents negative JSR results, preliminary scaling evidence, and reproducibility details rather than presenting a new main result.
- 23 Verbatim prompt templates.: The all-masked template asks for one token per label in order, with labels listed separately and answer slots written as repeated [MASK] positions.The per-label template instead asks, “Does this document express <label>? Answer: [MASK]”.
- 23 Verbatim prompt templates.: LLaDA and Dream use “yes” and “no” as single-token verbalizers, including a leading space for LLaDA.Dream uses the Qwen tokenizer, but the same verbalizers remain single tokens.
- 24 MDLM and BD3-LMs compatibility note.: MDLM and BD3-LMs are left unevaluated because the available interfaces are incompatible with the RTX 5090 setup and zero-shot adapter.Dream-7B is used as the second-backbone replication instead.
- 25 Ethical considerations.: Prompt wording materially influences predictions, while pretrained diffusion biases, transparency risks, and sensitive legal or emotion text create deployment concerns.The paper frames training-free convenience as insufficient to guarantee harmlessness in legal triage, moderation, safety monitoring, or affective computing.
- 26 JSR sweep tables (negative results).: Local-JSR lowers both metrics from the all-masked seed; by S=3, GoEmotions loses 8.43 micro-F1 and 4.19 macro-F1 points, while Reuters loses 9.07 and 2.16 points.The empirical decline is distinct from the analytical counterexample, but both concern local updates that need not improve set-level prediction.
- 26 JSR sweep tables (negative results).: Local-JSR also degrades the unbiased per-label seed: two sweeps reduce GoEmotions by 7.01 micro-F1 and 5.30 macro-F1, while Reuters loses 13.18 micro-F1 and 2.96 macro-F1 points.The result places the problem in the local refinement objective rather than only in positional bias.
- 27 Preliminary results with LLaDA2.0-mini (16B MoE, CPU).: LLaDA2.0-mini shows a preliminary Reuters prompt gain from 56.8 to 67.1 micro-F1, but its CPU int4 setup takes approximately 67 seconds per example and is not a controlled scaling study.Checkpoint training, quantisation, slice size, and architecture differ from the LLaDA-8B reference.
- 28 Reproducibility checklist.: The release includes code, configurations, saved score arrays, public datasets, supervised checkpoints, hardware details, three-seed headline replication, and CLI commands for main-table cells.These materials support reproduction of the reported experiments under the listed environment.
29 Full-gain JSR is monotone; tie-stable updates terminate.
Full-gain JSR updates are monotone for the pseudo-likelihood surrogate and terminate under tie-stable updates, unlike the local update used in the negative empirical results. A two-label counterexample demonstrates why local-JSR can decrease the surrogate.
- 29 Full-gain JSR is monotone; tie-stable updates terminate.: Full-gain JSR selects the coordinate value that maximizes the entire pseudo-likelihood surrogate with all other coordinates fixed.This differs from local-JSR, which maximizes only the coordinate’s local term.
- 29 Full-gain JSR is monotone; tie-stable updates terminate.: Each full-gain coordinate update and every complete sweep are monotone non-decreasing in the pseudo-likelihood surrogate.The proof follows because each update chooses the best available coordinate value while holding the other coordinates fixed.
- 29 Full-gain JSR is monotone; tie-stable updates terminate.: Tie-stable full-gain sweeps terminate after finitely many coordinate changes at a coordinate-wise local optimum.The finite binary state space and strict surrogate increase on every actual state change establish termination.
- 29.1 Counter-example: local-JSR can decrease the surrogate.: A two-label example shows local-JSR changing y_1 from 0 to 1 can reduce the surrogate from -1.1 to -10.1.The local term improves from -1.0 to -0.1, but the other term worsens by enough to decrease the total.
- 29.1 Counter-example: local-JSR can decrease the surrogate.: The counterexample explains why the local-JSR curves in the empirical sweeps move in the wrong direction: local improvement need not increase the set-level surrogate.The full-gain update would keep y_1 unchanged and preserve the higher surrogate value.