Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation

1 Introduction

• ViT pixel-level active learning remains challenging because heavy learned decoders such as DPT are difficult to train from extremely sparse supervision.Prior pixel-level methods use CNN backbones whose lightweight per-pixel heads train reliably under sparse supervision.Source: §1 · p. 2
• Upsampled token features can make boundary pixels appear diverse or uncertain because neighboring descriptors are mixed, rather than because of genuine semantic ambiguity.Figure 1 contrasts pixel-space selections near grid-aligned boundaries with TEAL’s more representative interior selections and cleaner segmentation maps under the same sparse label budget.Source: §1 · p. 2, Figure 1 · p. 2
• TEAL first uses MaxHerding on ViT tokens for diverse candidate coverage, then refines candidates with uncertainty on a finer decoder grid to query individual pixels.This token-to-pixel design preserves the semantic geometry of ViT features during acquisition.Source: §1 · p. 2
• With DINOv3, TEAL outperforms eDALD with DDPM features by up to 24.36 mIoU across four datasets and gains 0.50–2.47 mIoU over matched-backbone baselines at one pixel per image.At 512 × 512 resolution, the matched-backbone gains widen up to 4 mIoU.Source: §1 · p. 2

2.1 Semantic Segmentation

• Self-supervised ViT backbones, including DINOv3, provide spatial and semantic structure useful for dense visual tasks under limited supervision.Diffusion features have also been used for label-efficient segmentation, but their extraction is computationally expensive and acquisition is usually dense pixel-space.Source: §2.1 · p. 3

2.2 Active Learning

• Pixel-level active learning reduces annotation cost beyond image- and region-level selection, with methods including uncertainty-based PixelPick and Bayesian or balanced uncertainty approaches.These acquisition functions operate directly on dense pixel grids.Source: §2.2 · p. 3
• Coverage-based methods represent unlabeled-pool structure, and diffusion-driven two-stage active learning combines MaxHerding candidate selection with uncertainty refinement.For ViT backbones, dense pixel-space selection can misalign with patch tokens because interpolation mixes descriptors and may create artificial boundary diversity.Source: §2.2 · p. 3

3 Method

• TEAL separates semantic coverage in token space from spatial refinement on a finer decoder grid to select final pixel queries.This aligns acquisition with the ViT’s native representation rather than conducting both stages in dense pixel space.Source: §3 · p. 4

3.1 Preliminaries

• The setup uses a frozen DINOv3 backbone and a Mini-DPT head trained on accumulated queried pixels, targeting budgets of 1 and 10 pixels per image on average.The head uses four transformer layers and predicts on a 64 × 64 grid.Source: §3.1 · pp. 4–5

3.2 Coverage (Stage 1—Diversity Selection)

• Stage 1 applies two-level MaxHerding to native token descriptors, selecting representative tokens per image before global selection forms the candidate set.Herding descriptors use normalized features from layers 8–11, chosen for their more semantically mature distances.Source: §3.2 · pp. 5–6

3.3 Stage 2: Uncertainty Refinement

• Stage 2 ranks valid decoder cells by margin uncertainty and queries the center pixels of the most uncertain cells.Cells must overlap a candidate token and have an unlabeled center pixel; the decoder grid is 64 × 64.Source: §3.3 · p. 6

4.1 Datasets

• The evaluation covers CamVid, Cityscapes, ADE20K, and Pascal Context, including urban, indoor, outdoor, and scene-parsing settings.Pascal Context uses the 33-class setting; inputs are resized and center-cropped to 256 × 256, with a specified aspect-ratio-preserving resize for ADE20K.Source: §4.1 · p. 6

4.2 Implementation details

• Experiments use 1- and 10-pixel-per-image budgets, two-stage MaxHerding candidate selection, and a mini-DPT head trained with Adam at 1 × 10^-4.Training runs for 10 epochs with early stopping at 95% training accuracy; the validation set supplies the fully supervised reference.Source: §4.2 · p. 7

4.3 Main Results

• +3.24 to +24.36 mIoU over the strongest baseline: TEAL beats PixelPick and 2-Stage eDALD across all four datasets and both annotation budgets.At 10 pixels per image, TEAL reaches 90.2% of the dense-label ceiling on CamVid, 87.7% on Cityscapes, 88.6% on ADE20K, and 97.1% on Pascal Context.Source: §4.3 · p. 7, Table 1 · p. 7
• With DINOv3 shared across methods, TEAL remains best on every dataset and budget, exceeding the pixel-space baseline by up to 3.2 ceiling points on CamVid at 10 pixels per image.The matched-backbone comparison holds the representation fixed; the table’s final column reports the fraction of dense-label performance recovered at the 10-pixel budget.Source: §4.3 · pp. 7–8, Table 2 · p. 8

Acquisition-stage analysis

• At 1 pixel per image, token-to-cell refinement recovers 1.12 mIoU and token-space selection another 1.08; at 10 pixels, they add 0.98 and 1.57, respectively.The decomposition compares pixel herding with TEAL on CamVid; both stages contribute at both budgets.Source: §5 · p. 8, Figure 3 · p. 9

Resolution scaling

• At 512 × 512 on Cityscapes, TEAL exceeds eDALD DINO by +1.32 mIoU at 1 pixel per image and +2.32 mIoU at 10 pixels per image.On CamVid, the gains over eDALD DINO are +5.63 and +3.56 mIoU, respectively; on Pascal Context they are +0.81 and +0.57.Source: Figure 4 · p. 9, §5 · p. 9

Budget specialization

• At 1 pixel per image, only 2% of TEAL’s labeled tokens receive multiple labels; at 10 pixels, that share is 29.8%, versus 1.8% for eDALD.TEAL’s rare-class shares are 14.5% versus 10.1% for eDALD at one pixel, and 12.7% versus 7.5% at ten pixels.Source: §5 · p. 9

Sensitivity and design choices

• Across local-budget and RBF-bandwidth sweeps, TEAL varies by under 1.2 mIoU; final DINOv3 layers [8, 9, 10, 11] outperform shallow and balanced layer selections.Shallow layers score 30.30 ±3.85 and balanced layers 48.80 ±0.58; the deep configuration scores 49.54 ±0.37, while late layers score 50.21 ±1.29.Source: §5 · pp. 9–10

Normalization

• Fixed normalization scores 49.54 ± 0.37 mIoU, compared with 47.90 ± 0.36 without it and 48.20 ± 1.21 with learned normalization.The authors hypothesize that learned normalization is difficult to estimate reliably with few labels and may destabilize the feature space.Source: §5 · p. 10

Acquisition function

• At one pixel per image, replacing margin sampling with BALD or eBALD lowers TEAL’s CamVid score from 49.54 ±0.37 to 44.94 ±0.93 or 45.02 ±1.79 mIoU.The gap narrows at ten pixels per image as more labels stabilize the head.Source: §5 · p. 10

Backbone ablation

• On CamVid at one pixel per image, TEAL scores rise from 43.78 ± 0.56 with GoogleViT to 47.58 ± 0.87 with DINOv2 and 49.54 ± 0.37 with DINOv3.The paper relates stronger herding results to more consistent semantic neighborhoods and clearer object boundaries in the feature space.Source: §5 · p. 10

Limitations

• TEAL becomes more expensive as image resolution increases, while its lightweight Mini-DPT decoder can struggle with complex scenes and precise segmentation boundaries.The paper identifies the DINO forward pass as the main scaling bottleneck and notes that higher-resolution inputs increase token counts and self-attention cost.Source: §5 · p. 10

6 Conclusion

• TEAL combines native-token-space diversity selection with cell-level uncertainty refinement, and improves over strong low-budget baselines across CamVid, Cityscapes, ADE20K, and Pascal Context.The conclusion says matched-backbone comparisons indicate that both token-space acquisition and cell-level refinement contribute to the gains.Source: §6 · p. 10

A.1 Hyperparameter sensitivity

• The hyperparameter sweeps vary one parameter at a time on CamVid at 1 pixel per image, with other settings held at their defaults and results averaged over three seeds.Source: §A.1 · p. 14

A.1.1 Local budget k

• Performance peaks at k = 20 (49.54 mIoU); k = 5 falls to 46.15, while k = 80 reaches 48.40.Too few tokens cannot cover an image’s variation; larger pools degrade more gently, likely because they add redundant tokens.Source: §A.1.1 · p. 14

A.1.2 Global retention ratio

• Retention ratios from 0.25 to 0.75 yield 49.29 to 50.43 mIoU; keeping all candidates (1.00) drops performance to 48.80 ±1.21.The chosen 0.50 ratio halves the candidate set and stays within 0.9 mIoU of the best setting.Source: §A.1.2 · p. 14

A.1.3 RBF bandwidth scale factor s

• Scaling the RBF bandwidth by the median pairwise token distance keeps the kernel meaningful across feature magnitudes; performance is generally robust except at s = 1.0.Absolute bandwidths such as [0.5, 3.0] are far below typical feature distances and drive the kernel to zero.Source: §A.1.3 · p. 14

A.1.4 Herding layers

• Layer choice has the largest hyperparameter effect: shallow layers score 30.30 mIoU, versus 48.80 ±0.58 for Mini-DPT default, 50.21 ±1.29 for late layers, and 49.54 ±0.37 for four consecutive layers.The deep configuration is used for its low variance; shallow features mainly encode low-level appearance.Source: §A.1.4 · p. 15

• TEAL beats pixel herding on every dataset and budget, by 1.19–2.55 mIoU at one pixel per image and 1.31–2.55 at ten.Table 7 reports mIoU at both budgets under matched backbones, with means and standard deviations over three seeds.Source: §B.1 · p. 15, Table 7 · p. 15

B.2 Results at higher resolution

• At 512×512, TEAL beats both baselines across datasets and budgets, with eDALD DINO gains at 1/10 px of +5.63/+3.56 mIoU on CamVid and +0.81/+0.57 on Pascal Context.On Cityscapes, the gain over eDALD DINO grows from +1.32 to +2.32 mIoU as the budget rises from one to ten pixels per image.Source: §B.2 · p. 16, Table 8 · p. 16

B.3 Query allocation

• At ten pixels per image on CamVid, token herding improves IoU for most classes, especially localized categories; pixel herding is slightly better on road and pavement.The query-allocation analyses also track selected-pixel boundary proximity, rare-class share, and the fraction of labeled tokens receiving multiple labels.Source: Table 10 · p. 16, Table 9 · p. 16, Figure 5 · p. 17

B.4 Compute Cost

• A profiled CamVid round fits below 1.2 GB of VRAM; global cross-image MaxHerding is the acquisition memory bottleneck at 1.14 GB.Per-image herding peaks at 0.36 GB, and fully supervised decoder training peaks at 0.47 GB.Source: §B.4 · p. 17, Table 11 · p. 17

C.1 Mini-DPT head: changes from the Original Mini-DPT

• The Mini-DPT head replaces bilinear resizing plus a 3 × 3 convolution with DPT reassembly and uses a 64 × 64 grid instead of 128 × 128.The grid defines Stage 2 cells; the modified head reaches 62.1 mIoU on CamVid and 43.0 on ADE20K, versus 61.7 and 42.6 for the original Mini-DPT.Source: §C.1 · pp. 17–18, Table 12 · p. 18

C.2 Algorithm

• Stage 1 selects k representative tokens per image, then globally retains half the candidates; Stage 2 ranks valid decoder cells by margin uncertainty and labels their center pixels.The algorithm starts from frozen DINOv3 token features and a Mini-DPT head trained on the existing labeled pixels.Source: §C.2 · p. 18

Dataset Input GT eDALD (DDPM) eDALD (DINOv3) TEAL (Ours)

• At the final active-learning round, TEAL produces cleaner object regions and fewer boundary artifacts than eDALD with DDPM or matched-backbone DINOv3 features.The figure caption says these visual patterns are consistent with quantitative gains in the low-budget setting.Source: Figure 6 · p. 19

E Broader Impact

• Reducing annotation costs could make dense prediction models more accessible to teams with limited labeling resources.The passage names disaster response, rare disease diagnosis, and manufacturing inspection as domains where reliable annotations can be difficult to obtain.Source: E Broader Impact · p. 19
• For high-stakes or sensitive applications, the authors urge careful testing, fairness checks across groups, and human involvement because sparse supervision may reproduce training-data biases or vulnerabilities.Examples include law enforcement, medical decision-making, and applications involving private information.Source: E Broader Impact · p. 19