PaperlayerView exact revision on arXiv

Source-linked AI summary

MoTE: Mixture of Task Experts for Multi-Task Video Understanding

Muhammad Asad Ali, Umar Khan, Nadia Robertini, Didier Stricker

arXiv:2608.24763v1cs.CVcs.LG

TL;DR

Procedural video-language models must handle heterogeneous tasks over shared visual evidence, while dense decoders and token-level routing do not provide stable task-specific computation. MoTE converts decoder FFNs into task experts with shared multimodal components and sample-level routing; across five COIN tasks, it outperforms dense, learned sparse, and larger VideoLLM baselines with fewer active parameters.

  • Problem

    Procedural video understanding combines heterogeneous tasks with shared visual evidence, while dense decoders can entangle task behavior and token-level routing is not naturally aligned with task objectives.

  • Method

    MoTE converts language-model FFNs into task-specific experts while retaining a shared backbone, routing each video–prompt sample through one task route.

  • Results

    Across five COIN video-understanding tasks, VideoLLM-MoTE outperforms dense all-expert activation, learned sparse-routing variants, and reported larger VideoLLM baselines despite using fewer active parameters.

  • Takeaways & Limitations

    Task-structured routing provides an interpretable, compute-efficient decoder alternative and supports modular capability expansion while retaining existing routes in the tested settings.

  • Takeaways & Limitations

    The evaluation does not cover ambiguous, overlapping, or compositional requests, and freezing old routes during expert addition reduces plasticity for new tasks.

Abstract

from arXiv · show

Procedural video-language models must solve heterogeneous tasks from the same visual evidence, including action recognition, forecasting, and procedure prediction. Dense transformer decoders share the same feed-forward networks across tasks, which can entangle task behavior and make controlled capability expansion difficult. Sparse Mixture-of-Experts (MoE) decoders provide conditional computation, but token-level learned routing is not naturally aligned with task-level procedural objectives. We propose MoTE (Mixture of Task Experts), a decoder architecture that converts large language model feed-forward networks into task-specific experts while keeping the multimodal backbone shared. Each example follows one sample-level task route, so active task-expert computation remains independent of the number of stored task experts. We instantiate this design as VideoLLM-MoTE and evaluate it on five COIN benchmarks using explicit task routes. The five-expert model activates ~2B LLM parameters per sample and achieves higher average top-1 accuracy than recent VideoLLM baselines. Under the same expert topology, it improves over dense all-expert activation and learned sparse-routing controls. These results show that task-structured routing provides an interpretable and compute-efficient decoder alternative for multi-task video-language learning.

1 Introduction

Procedural video-language models must support diverse tasks over shared visual evidence, but dense decoders can entangle task-specific transformations. MoTE makes task identity an explicit routing variable while preserving shared components and enabling modular task expansion.

  • Motivation: Procedural videos require recognizing current actions, forecasting future steps, identifying goals, and producing procedure-level responses from shared visual evidence.
  • Motivation: Dense multi-task decoders can create negative transfer because dissimilar tasks update shared feed-forward parameters despite differing temporal horizons and output structures.
  • MoTE: MoTE replaces shared decoder FFNs with task-specific experts and an always-active shared expert while retaining the shared video-language backbone and attention path.
  • MoTE: Each video–prompt example follows one sample-level task route, keeping active task computation independent of the number of stored task experts.
  • Modularity: Task experts can be added while retaining previously learned knowledge in the shared backbone and existing experts, with routes supervised by task labels or selected from expert descriptions.
  • Evaluation: VideoLLM-MoTE improves over recent VideoLLM baselines and learned sparse-routing controls, while related experiments extend the expertization principle to OCR and receipt KIE.

2 Related Work

Prior MoE systems primarily route tokens or patches with learned gates, while task-aware methods move routing toward task semantics. MoTE instead uses sample-level task routing over decoder FFN experts for modular video-language decoding.

  • Sparse MoE and Task-Level Routing: Prior sparse transformers primarily route individual tokens or image patches with learned gates to provide conditional capacity.
  • Sparse MoE and Task-Level Routing: Task-level MoE methods align routing more closely with task semantics through task-conditioned gating features, but are not designed for modular video-language decoders.
  • Sparse MoE and Task-Level Routing: MoTE routes complete samples through task-specific decoder FFN experts rather than routing individual tokens, with explicit task routes used in the reported experiments.
  • Instruction Tuning and Task Conflict: Instruction-tuned vision-language models connect pretrained visual encoders to language models through lightweight projectors, enabling diverse task templates and improved held-out task generalization.

Method

MoTE replaces decoder FFNs with shared and task-specific experts while preserving shared multimodal processing. It routes each video–prompt sample through one task expert, supports prompt-based selection and modular expansion, and reports higher COIN average accuracy than baselines.

  • Architecture: MoTE retains the shared vision-language backbone and attention path while converting decoder FFNs into a shared expert plus task-specific experts.The shared expert remains active for every task, while the selected expert specializes to task supervision and output formats.
  • Architecture: Task experts are initialized from pretrained dense FFNs, allowing shared and task-specific paths to diverge during training.This dense-to-expert initialization avoids starting task experts from random transformations.
  • Task-route selection: Each example follows one sample-level task route, sending all visual and textual tokens through the same task expert at every decoder layer.Known task identities supervise routing during training, and only the selected task expert plus the shared expert receive gradients.
  • Task-route selection: Prompt-conditioned inference selects the top-1 expert by matching the prompt with natural-language expert descriptions.The selector uses normalized text embeddings and nearest-neighbor matching, while the description registry can add or remove routes without changing decoder parameters.
  • Modular Task Expert Adaptation: MoTE adds or removes task capabilities by attaching or dropping task experts without rewriting the shared backbone or remaining experts.For a new task, existing modules are frozen and only the new experts are updated, isolating new-task learning from prior task modules.

4 Experiments

Experiments evaluate MoTE on five COIN procedural-video tasks, cross-dataset video extension, and document understanding. Task-routed experts improve accuracy and support compute-efficient, modular capability expansion, with scope limited by the tested settings and prompt-selection assumptions.

  • Experimental Scope: The evaluation uses a 1B decoder as the default backbone because of limited computational resources, and ambiguous or compositional prompts remain outside evaluation.These conditions constrain how broadly the routing and prompt-conditioned selection results should be interpreted.
  • Comparison with Prior COIN Baselines: 62.9% five-task average accuracy is achieved by VideoLLM-MoTE-1B+5E, compared with 61.8% for the strongest 8B VideoLLM baseline.The model achieves the highest accuracy on all five COIN tasks under the same five-task protocol.
  • Efficiency: 2B LLM parameters are active per sample, reducing cost and latency versus VideoLLM-online-8B-v1+ while increasing decoding throughput.Relative to 98.7 TFLOPs, 2.8 seconds, and 27.8 tokens/s, MoTE uses 28.9 TFLOPs, 1.8 seconds, and 37.7 tokens/s.
  • Modular Task Adaptation via Expert Addition and Removal: Sequential expert addition preserves earlier task accuracies, but reaches 61.0% final average accuracy versus 62.9% with joint training.The comparison exposes a stability–plasticity trade-off: frozen additions preserve existing behavior, whereas joint training yields higher peak accuracy.
  • Cross-Dataset Task Extension: A single expanded checkpoint retains online narration while adding five COIN routes, although standalone Ego4D baselines achieve lower PPL and TimeDiff.This is evidence for backward-compatible expansion across related video-language datasets, not unrestricted transfer or superior Ego4D-only narration.
  • Auxiliary Cross-Domain Evaluation: 87.90% and 95.79% Micro-F1 are achieved on SROIE and CORD KIE, while the retained OCR route preserves OmniDocBench performance after conversion.The auxiliary GLM-OCR-MoTE evaluation demonstrates applicability beyond video backbones without catastrophic forgetting in the tested setting.
  • Effect of Routing Granularity: 62.9% average accuracy exceeds 61.5% with dense all-expert activation and 61.4% with token routing under the same converted decoder capacity.The results support sample-level task routing as better aligned with specialized procedural objectives than dense fusion or token-level routing.

5 Conclusion

MoTE transforms decoder FFNs into task-specific experts while preserving a shared backbone, isolating specialized knowledge and activating only the expert required for each task. Across five COIN tasks and an auxiliary document-understanding experiment, it outperforms evaluated alternatives while using fewer active parameters.

  • MoTE transforms LLM decoder FFNs into task-specific experts while preserving a shared backbone.
  • Across five COIN video-understanding tasks, VideoLLM-MoTE outperforms dense all-expert activation, learned sparse-routing variants, and larger VideoLLM baselines.The model uses fewer active parameters than these evaluated alternatives.
  • An auxiliary document-understanding experiment shows that MoTE adds structured-output capabilities while retaining foundational OCR performance.
  • The results support explicit task specialization and route preservation as a parameter-efficient alternative to monolithic model expansion in the evaluated settings.

A Supplementary Material

The supplementary material documents the experimental workflow, including accounting, training, evaluation, routing, diagnostics, and an auxiliary document-understanding study.

  • Sections A.1–A.3 document model accounting, training configuration, and data sampling.
  • Section A.4 defines the evaluation protocols, while Section A.5 presents rendered model inputs.
  • Section A.6 covers the expert-description registry and prompt-conditioned routing evaluation, followed by the expert-swap diagnostic in Section A.7.
  • Section A.8 presents the auxiliary document architecture, protocol, and full benchmark results.

A.1 MoTE LLM Parameter Accounting

MoTE increases stored parameters by replicating FFNs across routed and shared experts, while top-1 sample-level routing keeps active computation fixed at the shared path plus one routed path.

  • Table 3 reports LLM-only parameter counts, excluding the shared visual encoder and projector.
  • Replacing one dense FFN path with T routed experts and S shared experts determines the stored-parameter expansion.Here, T denotes stored routed experts and S denotes shared experts.
  • For VideoLLM-MoTE-1B+5E, T = 5, S = 1, and top-1 routing activates k = 1 routed expert per sample.
  • The resulting configuration rounds to 5.3B total parameters and 2.0B active parameters in Table 3.
  • Adding task routes increases total parameters linearly by PFFN, while active parameters remain fixed because each sample executes one shared and one routed FFN path.

A.2 Training Configuration

VideoLLM-MoTE trains COIN tasks as offline answer-generation problems, while Ego4D narration retains streaming silence supervision to control when narration is produced.

  • COIN tasks are trained as offline answer-generation problems with visual-frame positions masked from the streaming silence loss.Supervision covers the assistant answer plus EOS.
  • Ego4D narration retains the streaming silence term so the narration route produces short narrations at annotated response points and remains silent elsewhere.
  • In the main COIN results, task labels provide the routing signal.

A.3 Data Sampling Strategy

The sampler organizes examples by routed expert and selects buckets in proportion to their remaining data, while rank-aligned groups ensure every distributed rank processes the same expert route at each step.

  • A.3 Data Sampling Strategy: Buckets group training examples by their attached expert id, whose unequal sizes reflect different numbers of task examples.A uniform-over-experts schedule could overtrain smaller buckets and undersample larger ones.
  • A.3 Data Sampling Strategy: At each local step, the sampler selects an expert bucket with probability proportional to its remaining fresh samples.This preserves the training set’s task-size bias while limiting overexposure of small buckets.
  • A.3 Data Sampling Strategy: The sampler draws G = bR indices from the selected bucket, then distributed sharding gives every rank a microbatch for the same routed expert.Here b is the per-rank local batch size and R is the number of data-parallel ranks.
  • A.3 Data Sampling Strategy: Exhausted buckets are reshuffled and reused only as needed, reducing both small-bucket overexposure and large-bucket underexposure.This also avoids different ranks receiving different routed experts at the same local step.

A.4 Evaluation Protocols

The evaluation uses task-specific decoding and scoring protocols across COIN, Ego4D narration, and auxiliary receipt-document benchmarks, with metrics matched to each task format.

  • A.4 Evaluation Protocols: COIN outputs are mapped from free-form text to valid task labels before computing top-1 accuracy.Step, Next, and Task use exact matching followed by Levenshtein-distance assignment when needed.
  • A.4 Evaluation Protocols: Ego4D narration is evaluated online using PPL, TimeDiff, and Fluency diagnostics.Lower PPL and TimeDiff are better, whereas higher Fluency is better.
  • A.4 Evaluation Protocols: Figure 3’s sampler forms global groups of G = bR indices before rank-local sharding, aligning distributed evaluation-related data handling with routed experts.The figure specifies bucket selection, global grouping, and rank-local microbatches.
  • A.4 Evaluation Protocols: Receipt KIE reports strict or relaxed field-level F1 and micro-F1, while OCR preservation uses the original OmniDocBench metrics.SROIE and CORD use separate KIE reporting conventions.

A.5 Prompt Templates

The prompt templates define route-specific user requests and response formats for COIN recognition and forecasting tasks, plus online Ego4D narration instructions interleaved with video frames.

  • A.5 Prompt Templates: The shared system prompt frames the interaction as an assistant receiving a conversation interleaved with video frames.The same system prompt applies across routes.
  • A.5 Prompt Templates: Step recognition asks for the action in the video and returns a concise label such as “Cut potato into strips.”The prompt forbids extra text output.
  • A.5 Prompt Templates: Next-step forecasting asks for the next action, while procedure forecasting requests the next three actions as a numbered list.The procedure example predicts “Soak them in water,” “Dry strips,” and “Put in the oil to fry.”
  • A.5 Prompt Templates: Task recognition asks for the overall activity and returns a concise response such as “Make french fries.”The template specifies no extra text output.
  • A.5 Prompt Templates: The Proc.+ template adds the task goal, asking what the next three actions are for making french fries.Its response format remains a numbered list with one action per line.
  • A.5 Prompt Templates: Ego4D narration requests concise real-time narration, uses “C” for the camera wearer, and interleaves additional frame chunks with timestamped assistant narrations.The stream begins with an initial frame chunk followed by the narration instruction.

A.6 Prompt-conditioned Routing Evaluation

Prompt-conditioned routing selects among registered task experts, while expert-swap and document-domain experiments examine specialization and architectural transfer; the evaluation remains limited to stable predefined task boundaries.

  • A.6 Prompt-conditioned Routing Evaluation: The prompt-conditioned selector uses natural-language expert descriptions rather than decoder prompts to define expert-selection routes.Table 11 records the task-to-description registry used by the selector.
  • A.6 Prompt-conditioned Routing Evaluation: A MiniLM-L6-v2 selector reaches 100% test F1 for each of five predefined COIN intents, making prompt-selected scores equal to explicit-route scores.The evaluation uses 250 generated prompts split 70/10/20 into training, validation, and held-out test sets.
  • A.6 Prompt-conditioned Routing Evaluation: The held-out confusion matrix uses rows for ground-truth intents and columns for selected expert routes.This layout makes route-selection errors comparable across the five intents.
  • A.6 Prompt-conditioned Routing Evaluation: The prompt-routing result is a controlled selector sanity check because generated variants preserve clear task intent and do not test ambiguous, overlapping, or compositional requests.Table 12 shows three of 50 generated variants per task.
  • A.7 Offline Expert-Swap Routing: Expert swapping causes accuracy collapses primarily on the swapped task pair while untouched routes remain near base values, supporting task-localized FFN transformations.The diagnostic is a specialization stress test, not an expert-removal or unlearning evaluation.
  • A.8.1 Architecture: In the document architecture, PP-DocLayoutV3 crops regions, CogVIT and the GLM-OCR attention path remain shared, and the switch selects one task expert alongside the shared expert.Solid lines denote active routes and dotted lines inactive routes in the illustrated example.
  • A.8.2 Detailed Evaluation: GLM-OCR-MoTE uses separate OCR and receipt-KIE routes, with KIE predicting structured fields directly from receipt images and OCR evaluated independently.The comparison groups document baselines by input modality, and Tables 14 and 15 provide detailed comparisons.
  • A.9 Limitations: MoTE is most effective when task boundaries are meaningful and stable; ambiguous, overlapping, or compositional requests may require softer, hierarchical, or compositional routing.Freezing old modules during expert addition protects existing routes but can reduce plasticity for new tasks.
Loading 2608.24763v1…