PaperlayerView exact revision on arXiv

Source-linked AI summary

AHMAD: Adaptive Hybrid Multi-task Vision Learning with Assisted Distillation for Keypoint Detection

Mohammad Mahdi, Nedyalko Prisadnikov, Yuqian Fu, Carmelo Scribano, Danda Pani Paudel, Luc Van Gool

arXiv:2609.35490v2cs.CV

TL;DR

Vision multitask learning remains difficult because different tasks require different output structures. AHMAD unifies five vision tasks through a shared encoder-decoder and lightweight task-specific projectors, reporting PQ 53.1 and mIoU 66.5 for COCO panoptic and semantic segmentation.

  • Problem

    Vision multitask learning remains challenging because dense and sparse tasks require different output structures, making a unified framework difficult to design.

    Source: §1 · p. 1

  • Method

    AHMAD uses a shared encoder-decoder with lightweight task-specific projectors for five vision tasks, and distills keypoint features to enable single-pass full-image detection.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 6

  • Results

    AHMAD reports strong performance across vision tasks, including PQ 53.1 and mIoU 66.5 for COCO panoptic and semantic segmentation.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    KD* reduces keypoint detection forward passes by a factor of 2.7x, supporting unified processing of the same image across tasks.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Generalist multitasking vision models aim to unify multiple vision tasks within a single framework, enabling more efficient and versatile learning. However, handling diverse vision tasks -- spanning dense and sparse predictions -- remains challenging due to their inherently varying output structures. In this paper, we propose AHMAD, a simple yet effective framework for generalist multitask learning that integrates different key vision tasks: semantic segmentation, instance segmentation, depth estimation, keypoint detection, and object detection. Our approach incorporates these five tasks into a unified structure: a shared encoder-decoder with several lightweight task-specific projectors. Under the multitask learning paradigm, we observed a complementary performance gain, achieving a state-of-the-art PQ of 53.1 and an mIoU of 66.5 for COCO-val panoptic and semantic segmentation, respectively. Additionally, for top-down keypoint detection, which typically incurs high computational overhead due to multiple forward passes, we introduce a knowledge distillation-based method that enables a single forward pass over the entire image, greatly improving efficiency. Ultimately, our model delivers a lightweight yet effective generalist multitask learning framework, demonstrating strong performance across five vision tasks.

1. Introduction

AHMAD unifies five vision tasks in a shared multitask framework while addressing the repeated image passes required by top-down keypoint detection. The paper reports strong COCO segmentation results and competitive keypoint and object detection results.

1. Introduction

  • Vision multitask learning is challenging because dense and sparse tasks require different output structures and optimization objectives.Examples include pixel-level segmentation, continuous depth regression, and structured bounding-box prediction.Source: §1 · p. 1
  • Prior output-homogenization and serialization approaches introduce complex pre- and post-processing and sophisticated training requirements.These trade-offs motivate a framework that accommodates varied task outputs without extra components.Source: §1 · pp. 1–2
  • AHMAD unifies panoptic segmentation, depth estimation, keypoint detection, and object detection without restricting task output shapes or requiring extra components.Its shared multitask structure allows each task’s ideal single-task approach.Source: §1 · p. 2
  • Knowledge distillation enables top-down keypoint detection in a single forward pass over the entire image instead of separate runs for each person.Repeated runs are especially expensive with large, computationally intensive encoder backbones.Source: §1 · p. 2
  • The paper reports strong performance among generalist models, with PQ 53.1 on COCO panoptic segmentation and mIoU 66.5 on semantic segmentation, plus competitive keypoint-detection and object-detection results.Source: §1 · p. 2

2. Related Work

The related work surveys Vision Transformers, transformer-based segmentation and detection, and generalist models that unify diverse vision tasks.

Vision Transformers

  • ViTs [21] [37] serve as foundations for multitask and multimodal vision, while Swin Transformer [28] uses shifting windows for efficient long-range modeling.Self-supervised methods including DINO [3] and DINOv2 [34] use distillation on unlabeled data to learn transferable representations.Source: §2 · p. 2

Segmentation and Object Detection

  • Transformer models support dense segmentation and open-vocabulary detection, including MaskFormer [7], Mask2Former [8], OneFormer [19], SAM, and Grounding DINO [27].SAM2 [38] adds a memory bank and attention; Grounded SAM [42] combines Grounding DINO and SAM for segmentation and detection.Source: §2 · p. 2

Multitask Generalist Models

  • Painter [51] frames dense vision tasks as image inpainting, with outputs in the same RGB space as input images; while Unified-IO [29] [30] introduced an encoder-decoder architecture using stacked transformer layers.At inference, Painter adapts tasks through random, searched, or learned prompts.Source: §2 · p. 2
  • Pix2Seq [4], Pix2Seq-D [6], GiT [48], and DINO-X [40] extend prompt- or region-based approaches to diverse vision outputs and open-world understanding.Pix2Seq-D handles semantic and instance segmentation; GiT merges sparse responses from image subregions, and DINO-X supports text, visual, and customized prompts.Source: §2 · p. 3

3. Method

AHMAD trains multiple vision tasks through a shared encoder-decoder and lightweight task-specific projectors, and uses knowledge distillation to enable single-pass, whole-image keypoint inference.

3. Method

  • A shared encoder-decoder learns a robust feature map, while shallow task-specific projectors adapt its output shape to each task; training samples a random task and minimizes its loss.The backbone is ViT-L initialized with self-supervised DINOv2 weights, and the decoder has four transposed convolutional layers.Source: §3 · p. 3

3.1. Task-specific Projectors

  • Four intermediate backbone layers are concatenated into a latent representation that the decoder maps to a dense per-pixel feature map with 256 channels.Task-specific projectors use this shared map to optimize their task losses.Source: §3.1 · p. 3, Figure 2 · p. 4
  • Edge Distance Sampling weights pixels near boundaries for semantic, instance, and depth tasks, excluding unlabeled regions and mitigating loss imbalance between small and large objects.The method sets the minimum weight for pixels far from boundaries to zero and uses a distance parameter D = 20.Source: §3.1 · p. 3
  • Semantic segmentation uses per-pixel weighted cross-entropy, while instance segmentation regresses embedded centroid coordinates and assigns unlabeled pixels a void encoding.Panoptic segmentation is decomposed into semantic and class-agnostic instance segmentation.Source: §3.1 · pp. 3–5
  • Depth estimation predicts normalized inverse depth through a sigmoid convolution, while object detection uses a YOLO-style [39] grid head to predict boxes and class vectors.Each detection grid cell predicts multiple bounding boxes and a category vector.Source: §3.1 · p. 5
  • The keypoint projector predicts one unified heatmap alongside 17 joint-classification channels, trained with heatmap regression and classification losses at ground-truth locations.At inference, a class-aware argmax selects the highest-scoring location for each joint and filters low-confidence detections.Source: §3.1 · p. 5

3.2. KD*:Single-pass Top-down Keypoint Detection

  • KD* replaces repeated per-person network runs with one full-image forward pass by distilling cropped-person features from the full-image feature map.The teacher processes cropped RGB inputs, while the student crops corresponding regions from the full-image features.Source: §3.2 · pp. 5–6
  • A training-free upscaler rearranges channels into a 4×4 spatial grid, increasing feature resolution fourfold while reducing channels by 16× to preserve information for small crops.This addresses spatial-information loss when cropping small person boxes from the full-image feature map.Source: §3.2 · p. 6

4. Experiments

Experiments evaluate AHMAD’s five-task framework on COCO-val, reporting strong segmentation results and efficient keypoint inference through knowledge distillation.

4.1. Setup and Implementation

  • Training uses three stages: IS-only at 280×280, all tasks at the same resolution, then fine-tuning at 616×616 with reduced augmentation and fewer epochs.The stages run for 280, 150, and 60 epochs, totaling 5 days; most training uses low-resolution data to reduce computational cost.Source: §4.1 · p. 6

4.2. Main Results on COCO

  • PQ 53.1 and mIoU 66.5 surpass other models; panoptic performance gains are 22.3% over Painter and 5.5% over Pix2Seq-D.The paper reports these results for panoptic and semantic segmentation, respectively.Source: §4.2 · p. 7
  • Semantic segmentation exceeds LDM by ∆mIoU 6.4 and GiT by ∆mIoU 14.1; keypoint OKS-mAP is 68.12 and detection mAP is 44.6.Depth estimation is compared only between AHMADsingle and AHMADmulti because other models use different ground-truth labels. Figure 4 shows predictions across the five tasks.Source: §4.2 · p. 7, Figure 4 · p. 7
  • KD* distills information from the teacher feature map to the student feature map, reducing forward passes by 2.7x and enabling unified multi-task processing of the same image for different tasks in one pass.Source: §4.2 · p. 7
  • The bypassed path alone achieves strong metric scores, supporting DINO-V2’s suitability; top-down keypoint crops use 20% noise on ground-truth boxes to approximate real-world inference.Source: §4.2 · p. 7

5. Ablation Study

The ablations examine EDS and positional embeddings for segmentation, post-processing choices for detection, and loss functions and training strategies for knowledge distillation. Reported findings favor joint-aware argmax and dIoU-based NMS, while the KD ablation identifies a performance benefit from TFU.

5. Ablation Study

  • The semantic segmentation ablation investigates EDS, while the instance segmentation ablation examines positional embeddings across different L values.Table 3 describes a panoptic segmentation ablation of EDS and positional embedding on ViT-Base at 420 × 420 resolution, including direct (u, v) regression.Source: §5 · p. 7, Table 3 · p. 8
  • Joint-aware argmax over heatmap values outperforms clustering for keypoint detection, and dIoU-based NMS performs better than IoU-based NMS for object detection.The keypoint comparisons include DBSCAN and KMeans for obtaining final joints from candidates.Source: §5 · p. 7
  • The KD ablation compares architectures and loss functions on OKS-mAP, and reports that TFU enhances performance.CID-1 and CID-2 use no distillation; CID-2 concatenates the cropped image with the input feature map, CID-3 shares a projector between student and teacher, and CID-5 uses different but architecturally identical projectors.Source: §5 · p. 7, Table 4 · p. 8

6. Conclusion

AHMAD unifies five vision tasks in one framework and uses distillation to enable single-pass keypoint detection. The paper reports strong performance across tasks, including state-of-the-art panoptic and semantic segmentation, and identifies several directions for future work.

6. Conclusion

  • AHMAD integrates semantic segmentation, instance segmentation, depth estimation, keypoint detection, and object detection in one model without complex serialization or pre/post-processing.The framework is designed to handle heterogeneous task outputs efficiently.Source: §6 · p. 8
  • Distillation enables keypoint detection in a single forward pass instead of the multiple passes required by traditional top-down methods; AHMAD reports state-of-the-art panoptic and semantic segmentation.The paper also reports strong performance across all five tasks.Source: §6 · p. 9
  • Future work will refine distillation, incorporate higher-resolution training, and expand AHMAD to additional vision tasks.Source: §6 · p. 9

Related papers

Loading 2609.35490v2…