PaperlayerView exact revision on arXiv

Source-linked AI summary

Breaking Babel: A Self-Evolving Multi-Agent System for Long-Form Subtitle Translation

Haibo Jin, Xinjie Li, Najmeh Sadoughi, Yang Liu, Yibo Wang, Zhu Liu, Yuzong Liu

arXiv:2609.38660v1cs.CLcs.AIcs.CVcs.MA

TL;DR

Long-form subtitle translation must preserve discourse, cultural context, terminology, and style across episodes, but existing methods have limited long-range context and adaptive coordination. SMART combines persistent series memory, dynamic routing, tools, and judge-guided test-time updates; it achieves the best Overall MQM score across all 15 Subtitle Arena directions, with a 6.9% lower average penalty than the strongest agent baseline.

  • Problem

    Long-form subtitle translation depends on discourse and cultural context across episodes while maintaining terminology and style, but existing methods have limited series-level context, adaptive coordination, and contextual retrieval.

    Source: §1 · pp. 1–2

  • Method

    SMART uses persistent series memory, dynamic routing among translators and tools, and judge-refiner feedback to update prompts and routing during test-time training without changing LLM parameters.

    Source: §1 · p. 2, §3.1 · p. 3, §3.4 · p. 5

  • Results

    SMART achieves the best Overall MQM score in all 15 Subtitle Arena directions, with a 6.9% lower average penalty than the strongest agent baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments show consistent improvement over strong baselines, supporting treatment of long-form translation as a stateful and adaptive process rather than independent sentence-level decisions.

    Source: §6 · p. 10

    SMART uses substantially more test-time computation than a single-call translator, limiting use in latency-bound or high-volume settings.

    Source: I LIMITATIONS AND FUTURE WORK · p. 46

Abstract

from arXiv · show

Long-form subtitle translation requires reasoning over discourse and cultural context spanning episodes or entire series, while maintaining consistent terminology and style. Existing single-LLM methods are largely sentence-level, and multi-agent systems often use static workflows that do not adapt to scene complexity or production context. We propose SMART, a Self-evolving Multi-Agent system for long-foRm subtitle Translation. During test-time training, SMART builds persistent series-level memory and translates a subset of sentences through a dynamic router and Mixture-of-Agents layer with tools for terminology verification, subtitle constraint validation, and contextual retrieval. A judge-refiner loop scores candidates and uses textual critiques to update agent prompts and routing policies without retraining the underlying LLMs. During test-time inference, the evolved configuration translates the remaining series. We also introduce Subtitle Arena, covering 14 genres, 2--198 episodes per series, production years 1959--2023, and 15 target locales, together with SubMQM, a subtitle-adapted MQM framework with seven dimensions and 19 error categories. SMART achieves the best overall MQM score in all 15 Subtitle Arena directions, reducing average penalty by 6.9% over the strongest competing agent system. On the public MuSC benchmark, SMART obtains the best model result across all four language pairs and also achieves the best human-evaluation result, with an overall score of 4.50/5.

1 INTRODUCTION

Long-form subtitle translation must preserve meaning and consistency across extended narrative context while meeting display constraints, yet existing approaches have limited long-range memory, adaptive coordination, and contextual adaptation. SMART addresses these challenges with test-time evolution, and the paper introduces Subtitle Arena and SubMQM to evaluate long-form translation.

1 INTRODUCTION

  • Long-form subtitles require discourse and cultural context across speakers, scenes, and episodes while preserving terminology and narrative coherence under line-length and reading-speed constraints.The paper identifies limited series-level context, static agent workflows, and limited dynamic use of production context as key challenges.Source: §1 · pp. 1–2
  • SMART evolves agent prompts and routing during test-time training, combines dynamic specialist selection with persistent series memory and tools, then applies the evolved configuration to remaining content.Its judge–refiner loop converts translation errors into textual feedback without updating underlying LLM parameters.Source: §1 · p. 2
  • Subtitle Arena covers series across 14 genres, production years 1959–2023, 2–198 episodes per series, and 15 target locales; SubMQM covers seven dimensions and 19 error categories.The benchmark targets discourse-aware translation, consistency across extended narratives, and subtitle display constraints.Source: §1 · p. 2
  • SMART achieves the best Overall MQM score in all 15 Subtitle Arena directions, with a 6.9% lower average penalty than the strongest agent baseline.It also achieves the best model result across all four MuSC language pairs and the best human-evaluation result, with an overall score of 4.50/5.Source: §1 · p. 2

2 RELATED WORK

Prior work includes single-LLM adaptation, role-based translation agents, self-evolving agents, and LLM-based judges. SMART distinguishes itself by evolving translation instructions, routing, and contextual knowledge during a long-form task.

Multi-Agent Systems and Translation Agents

  • Translation agents commonly use role-based generation, evaluation, and refinement or persistent context, while LLM judges typically score, rank, or locally refine outputs.Single-LLM approaches also use subtitle-specific fine-tuning or context-aware prompting.Source: §2 · pp. 2–3

Key Differences

  • Existing translation-agent systems mainly use predefined workflows or fixed agent behaviors, whereas SMART evolves instructions, routing, and contextual knowledge at test time.SMART targets long-form translation and treats prompts and agent behavior as persistent system state updated through translation-specific feedback.Source: §2 · p. 3

Self-Evolving Agents and Prompt Optimization

  • Related self-evolving agents improve workflows, reusable skills, or memory, while prompt-optimization methods typically optimize future tasks or fixed objectives.SMART instead adapts during a single long-form task while preserving previously acquired behavior.Source: §2 · p. 3

3 METHODOLOGY

SMART represents long-form translation as a stateful process with persistent series memory, adaptive agent and tool selection, and test-time updates to prompts and routing. Its inference configuration is frozen after training, while memory continues to accumulate.

3.1 PROBLEM FORMULATION

  • SMART’s system state comprises a fixed pool of translator roles, their prompts, a routing policy, and persistent series-level memory.It adapts prompts and routing from translation feedback without updating the underlying LLM parameters.Source: §3.1 · p. 3

3.2 SYSTEM OVERVIEW

  • A persistent memory pool, adaptive translation graph, and judge–refiner loop jointly retain series information, route sentences, and select and refine candidates.Episodes are split 3:7 between test-time training and held-out inference; memory continues to grow during inference.Source: §3.2 · p. 3, Figure 1 · p. 4

3.3 PERSISTENT SERIES MEMORY

  • Series memory stores terminology, character profiles, domain and scene knowledge, and idioms, carrying confirmed information across episodes for retrieval.SMART also builds a content profile and may retrieve evidence for ambiguous jargon, cultural references, and slang, interpreting it in the current scene before storage.Source: §3.3 · p. 4

3.4 ADAPTIVE MULTI-AGENT TRANSLATION

  • A dynamic router selects translators according to sentence properties, and active translators can use tools for retrieval or verification before a judge scores, selects, and refines candidates.Tools include context, terminology, subtitle constraints, domain knowledge, idiom lookup, and target-language fluency checks; an episode-level pass repairs residual consistency errors.Source: §3.4 · pp. 4–5

3.5 SELF-EVOLUTION AT TEST-TIME TRAINING

  • Batch-aggregated scores and critiques update prompts and routing in natural language during test-time training, after which the configuration is frozen for inference.The underlying LLM parameters remain fixed, while memory continues updating during inference to preserve consistency.Source: §3.5 · p. 5

4 SUBTITLE ARENA: BENCHMARK AND EVALUATION

Subtitle Arena is organized around television series and is designed to test long-range discourse, consistency, and subtitle display requirements. SubMQM supplies a fine-grained subtitle-adapted evaluation rubric.

4.1 BENCHMARK CONSTRUCTION

  • Subtitle Arena contains 192 television series, 6,267 English source episodes, and 70,664 aligned bilingual episode pairs across 15 target locales.It spans 14 genres, with six locales exceeding 5,000 aligned episodes and every locale retaining more than 3,200 episode pairs.Source: §4.1 · p. 5
  • The series-level benchmark tests discourse-aware translation, consistency of terminology, character references and style, and subtitle display requirements across genres and locales.Its table reports aligned bilingual episode coverage by target locale against 6,267 English source episodes and series distribution across 14 genres.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 LLM-AS-A-JUDGE EVALUATION WITH SUBMQM

  • SubMQM assigns each error type a penalty of 0, 5, or 10, with lower penalties indicating better results; its rubric covers 19 error types.The complete rubric is provided in Appendix E.Source: Table 2 · p. 6

5 EXPERIMENTS

Experiments compare SMART with subtitle, single-call, and agentic baselines on Subtitle Arena, then assess robustness, component effects, human ratings, and contextual examples.

5.1 EXPERIMENTAL SETUP

  • The comparison includes community subtitles, single-call LLMs, and reimplemented agentic baselines; per-series training uses the first 30%, with the remaining 70% held out and updates frozen.Persistent series memory continues to accumulate during held-out inference.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS

  • Across 15 English→locale directions, SMART lowers mean Overall SubMQM penalty from 1.20 (TransAgent) to 1.11, a 6.9% reduction; locale→English, it lowers the mean from 0.41 to 0.37, a 10.0% reduction, achieving the lowest Overall score in all 15 directions.Source: §5.2 · pp. 7–8

5.3 BACKBONE AND EVALUATOR ROBUSTNESS

  • SMART reduces Overall penalty on en→zh from 1.46 to 0.67 with Gemma 3 4B and from 0.93 to 0.53 with DeepSeek-V3.2.Replacing the judge changes scores only marginally, including 0.48 versus 0.47 on en→zh and 0.87 versus 0.87 on en→de.Source: Table 5 · p. 8, §5.3 · p. 9

5.2 MAIN RESULTS

  • 0.63 to 0.57: SMART + MM lowers average SubMQM penalty across six directions and outperforms ViDove and Hermes.The multimodal extension uses video and audio tools powered by Qwen3-Omni.Source: §5.2 · p. 8
  • SMART outperforms TransAgent in all six evaluated directions on 200 TV series released in 2025–2026, including titles beyond Claude Sonnet 4.6’s reported knowledge cutoff.The paper says this suggests gains extend to temporally shifted content beyond Subtitle Arena.Source: §5.2 · pp. 8–9

5.4 ABLATION STUDY

  • Removing dynamic routing and self-evolution raises penalties by 0.27 and 0.36; replacing MoA with one Combined Prompt raises them by 0.41.The authors attribute MoA’s benefit to independently generated specialist hypotheses rather than combining instructions in one prompt.Source: Table 6 · p. 9, §5.4 · p. 9

5.5 HUMAN EVALUATION

  • 4.50: SMART ranks first on all four human-evaluation criteria, with its best average rank on Consistency at 4.55.The blind study used 20 continuous passages, 267 aligned lines, and 20 annotators; the authors retain SubMQM for fine-grained error analysis given the study’s limited scale and coarse criteria.Source: Table 7 · p. 9, §5.5 · p. 9

5.6 QUALITATIVE VISUALIZATION

  • SMART localizes the humor/humerus pun into Chinese and uses web retrieval to resolve “Bullets” as Lt. Grace Billets rather than ammunition.The examples illustrate non-literal wordplay adaptation and entity disambiguation using external knowledge.Source: Figure 2 · p. 10, §5.6 · p. 10

6 CONCLUSION

SMART treats long-form translation as a stateful, adaptive process, combining persistent series memory and test-time evolution without changing model parameters. Its related-work distinctions center on adapting agent behavior during translation rather than relying on fixed workflows or objectives.

6 CONCLUSION

  • SMART combines persistent series-level memory, adaptive routing, tool-augmented translation, and test-time evolution without updating model parameters.The conclusion frames long-form translation as a stateful and adaptive process rather than a sequence of independent sentence-level decisions.Source: §6 · p. 10

A EXTENDED RELATED WORK

  • Unlike translation agents with fixed workflows, SMART evolves instructions, routing, and contextual knowledge at test time for long-form translation.It treats prompts and agent behaviors as persistent state updated through translation-specific feedback.Source: A EXTENDED RELATED WORK · p. 15
  • Prior self-evolving-agent work mainly evaluates gains on subsequent tasks, with less attention to whether new behaviors interfere with previously learned ones.The distinction matters for long-form translation, where later improvements should not destabilize earlier terminology, register, or conventions.Source: A EXTENDED RELATED WORK · p. 15
  • Prompt-optimization methods typically refine prompts for a fixed task or objective, whereas SMART evolves agent instructions throughout an ongoing translation process.SMART maintains those instructions as persistent system state.Source: A EXTENDED RELATED WORK · p. 15
  • Existing LLM-as-a-judge work uses judgments for evaluation or local refinement, while SMART turns translation-specific judgments into persistent system updates.Structured MQM feedback can provide more diagnostic information than scalar quality scores.Source: A EXTENDED RELATED WORK · p. 15

B SMART IMPLEMENTATION DETAILS

This section describes SMART’s implementation, combining context-aware subtitle processing, persistent series-level memory, specialized translators, and verification tools with judge-feedback updates to prompts, routing, and tool assignments. It also shows how terminology memory, tool-guided reasoning traces, and natural-language prompt updates support consistency and pragmatic translation choices and address recurring agent failures, without changing LLM weights.

B.1 SUBTITLE PARSING AND CONTEXT CONSTRUCTION

  • SRT blocks are augmented with neighboring lines, scene metadata, and series context; interrupted sentences are grouped for translation, then restored to their original timestamp boundaries.This lets the system translate sentence-level meaning rather than treating every subtitle block independently.Source: §B.1 · p. 16

B.2 MEMORY CONSTRUCTION AND CONTENT PROFILING

  • Series memory stores confirmed terminology, character information, domain facts, and idioms; initial profiling establishes series context, while uncertain expressions can trigger contextually interpreted web retrieval.Existing persistent memory is reused, and the worked example shows an early-episode name rendering retrieved in a later episode.Source: §B.2 · p. 16, §C.3 · p. 21

B.3 ROUTING POLICY AND TRANSLATOR ROLES

  • A feature-based router selects translator subsets using length, duration, tone, emotion, slang, and display-budget pressure, with roles specialized for fidelity, fluency, expression, length, or colloquial language.Training logs routing outcomes and judge scores to inform policy updates.Source: §B.3 · p. 16

B.4 TOOL INTERFACES

  • Eight callable tools support terminology lookup and registration, memory and context retrieval, subtitle-constraint checks, web search, idiom lookup, and fluency checks.Tool access is restricted by translator assignment; web results pass through contextual interpretation before use.Source: §B.4 · pp. 16–17

B.5 JUDGE–REFINER AND RESCUE STEP

  • The judge scores candidates for accuracy, fluency, register, relevant expressiveness, terminology consistency, and subtitle constraints; scores below 8.0 trigger a critique-guided rescue pass.The accepted translation updates local history and, when appropriate, persistent series memory.Source: §B.5 · p. 17

B.6 EPISODE-LEVEL CONSISTENCY PASS

  • An episode-level editor checks overlapping windows for terminology, coherence, references, and translationese; examples show selective routing, contextual retrieval, and memory reused across episodes.Consistency rounds are skipped when an earlier round makes no changes; contextual retrieval separates literal meaning from scene-appropriate register.Source: §B.6 · p. 17, §C.1 · p. 20, §C.2 · p. 20

B.7 TEST-TIME TRAINING LOOP

  • SMART trains on each series’ first 30%, freezes prompts and routing for the remaining 70%, and evolves text-based configuration over fixed epochs without changing LLM weights.Translation and series memory continue updating during inference; the final epoch’s configuration is used unchanged.Source: §B.7 · pp. 17–18

B.8 INSTANTIATION OF F Π AND F ρ

  • Prompt rewrites target flagged agents while preserving their roles and workflow; routing rewrites keep categories fixed and retain at least two agents per category.SMART runs three epochs, then freezes the resulting prompts, routing policy, and tool assignments for inference.Source: §B.8 · p. 18

C.3 SERIES MEMORY EXAMPLE

  • Series memory lets later episodes reuse established terminology, while accumulated occurrence counts give the judge a consistency reference.For example, the mapping Harry Bosch →哈里·博斯 is loaded from memory in episode five, and the existing term count grows from 14 to 27.Source: §C.3 · p. 21

C.4 AGENT REASONING TRACE

  • In the reasoning trace, the natural translator resolves sarcasm by retrieving scene context, selecting an ironic Chinese phrasing, and checking its register with tools.The interleaved reason-act record lets the judge and self-evolution stage attribute a low score to a specific decision rather than to the sentence as a whole.Source: §C.4 · p. 22

C.5 PROMPT UPDATE EXAMPLE

  • During test-time training, the evaluator uses recurring agent deficiencies to rewrite prompts in natural language, without changing LLM weights.One batch flagged literal interjections, missing context for short ambiguous lines, and flattened emotional register; the example’s mean score was 4.2. After an update, a previously unseen “Oh, boy. Here we go again” example scored 8.6 versus 4.1 before, changing from 哦，孩子。我们又来了。 to 唉，又来了。Source: §C.5 · pp. 22–23

C.6 ROUTING POLICY UPDATE EXAMPLE

The routing example shows how judge scores inform category membership and tool assignments. The benchmark reconstruction organizes aligned subtitles into episode streams, retaining episode boundaries and presentation timing while filtering unusable alignments.

C.6 ROUTING POLICY UPDATE EXAMPLE

  • The routing policy and tool assignments are updated from per-category judge statistics, with memory_search and web_search added for the colloquial agent.The revised routing table reassigns agents within existing categories, and the evolved configuration is retained for inference.Source: §C.6 · pp. 23–24
  • After routing changes, the colloquial winner’s judge score is 8.7, compared with 6.2 for the earlier faithful candidate.The update removes the underperforming faithful agent from that category; no agent is retrained.Source: §C.6 · p. 24

D.1 SOURCE CORPUS AND EPISODE RECONSTRUCTION

  • Subtitle Arena groups aligned English and target-language documents by the canonical (IMDb, season, episode) key, preserving episode boundaries.This converts sentence-level parallel data into episode-level subtitle streams.Source: §D.1 · p. 24

D.2 SRT RECONSTRUCTION AND FILTERING

  • SRT reconstruction restores timestamped subtitle blocks in presentation order, merges continuation blocks, and discards empty, one-sided, or unparsable episode streams.Timestamps are normalized to hh:mm:ss,ms.Source: §D.2 · p. 24

D.3 LOCALE NORMALIZATION

  • The benchmark reports 15 target locales, but OpenSubtitles does not provide a distinct subtitle stream for every regional variety evaluated.Source: §D.3 · p. 25

D.4 BENCHMARK STATISTICS AND TRANSLATION DIRECTIONS

Subtitle Arena covers television series across many episodes, genres, and production eras, with evaluation in both directions for 15 target locales. Unlike sentence- or segment-level resources, it organizes subtitles around complete episodes and series to evaluate long-range context and consistency.

D.4 BENCHMARK STATISTICS AND TRANSLATION DIRECTIONS

  • 192 series, 6,267 English source episodes, and 70,664 aligned bilingual episode pairs span 14 genres and production years 1959–2023.Series contain 2–198 episodes, averaging 32.6 episodes per series.Source: §D.4 · p. 25
  • The 15 target locales yield 30 evaluation directions: 15 English→locale and 15 locale→English.Each episode pair aligns English with one target locale.Source: §D.4 · p. 25

D.5 RELATION TO EXISTING SUBTITLE BENCHMARKS

  • Unlike prior resources organized around sentences or multi-line utterances, Subtitle Arena organizes bilingual data around complete episodes and series.This makes persistent discourse context and long-horizon consistency evaluation targets, paired with an error-typed subtitle evaluation protocol.Source: §D.5 · p. 25

D.6 SUBTITLE ARENA VISUALIZATION

  • Figure 3’s left panel shows aligned bilingual episode coverage by target locale against 6,267 English episodes; its right panel shows 192 series across 14 genres.The figure visualizes the language coverage and genre composition summarized in Table 1.Source: Figure 3 · p. 26, §D.6 · p. 26

E MQM EVALUATION PROTOCOL

SubMQM scores subtitle errors across seven weighted dimensions, and SMART has lower Overall penalties than TransAgent in both translation directions. The protocol aligns references by timestamps to avoid penalizing benign re-segmentation.

E MQM EVALUATION PROTOCOL

  • SubMQM assigns each of 19 error subcategories a 0, 5, or 10 penalty within a sliding window; lower scores indicate fewer and less severe errors.Non-applicable subcategories default to zero.Source: E MQM EVALUATION PROTOCOL · p. 26, Table 10 · p. 27
  • Episode-level dimension penalties average subcategory penalties across windows, and Overall is a weighted average of seven dimensions emphasizing Accuracy, Terminology, and Fluency.Their weights are 0.3, 0.2, and 0.2, respectively.Source: E MQM EVALUATION PROTOCOL · p. 26
  • Model outputs retain source block segmentation, while human references are aligned by timestamp overlap and scored by passage to avoid penalizing benign re-segmentation.The evaluation also reports reductions in recurring errors including mistranslation, terminology inconsistency, and line-formatting violations.Source: E MQM EVALUATION PROTOCOL · p. 26, §F.1 · p. 27

F.1 ENGLISH-TO-LOCALE RESULTS

  • SMART achieves the lowest Overall SubMQM penalty in all 15 English→locale directions, averaging 1.11 versus TransAgent’s 1.20, a 6.9% relative reduction.The fine-grained results report lower mean Accuracy, Fluency, and Technical penalties as well.Source: §F.1 · pp. 26, 27, 29, 31, Table 11 · p. 27, Table 12 · p. 29

F.2 LOCALE-TO-ENGLISH RESULTS

  • In locale→English, SMART has the lowest Overall penalty in all 15 directions, averaging 0.37 versus TransAgent’s 0.41, a 10.0% relative reduction.Mean Accuracy falls from 0.75 to 0.67 and mean Technical from 0.13 to 0.07; other dimensions also improve on average.Source: §F.2 · pp. 31, 33, 35, Table 13 · p. 32, Table 14 · p. 34
  • In locale→English, error categories are not uniformly better; SMART’s overall gains instead accompany reductions in higher-weight semantic, consistency, and subtitle-formatting errors.Vividness and Profanity show small mixed changes in some directions.Source: §F.2 · p. 31

F.3 BACKBONE AND JUDGE ANALYSIS

The backbone and judge analysis finds that SMART’s scaffold improves results with smaller backbones, while changing the evaluator yields smaller differences. Additional analyses report transfer to newly released series, computational cost, and the scope of the human evaluation.

F.3 BACKBONE AND JUDGE ANALYSIS

  • Applying SMART to Gemma 3 4B or DeepSeek-V3.2 reduces Overall penalty versus direct translation with the same model; changing the judge produces smaller differences.For en→zh, the judge change shifts Overall from 0.48 to 0.47, while direct versus SMART-wrapped Gemma 3 4B scores 1.46 versus 0.67.Source: §F.3 · pp. 35–36, Table 15 · p. 36, Table 17 · p. 38

F.4 TEMPORAL GENERALIZATION TO NEWLY RELEASED TV SERIES

  • On 200 newly released, title-disjoint TV series, SMART beats TransAgent in all six directions, averaging 0.73 versus 0.77 Overall penalty.The set targets content released beyond Claude Sonnet 4.6’s reported May 2025 knowledge cutoff.Source: §F.4 · p. 41, Table 19 · p. 42

F.5 ABLATION SETTINGS

  • The ablation settings separately remove dynamic routing, test-time evolution, persistent memory, retrieval and idiom resources, sliding-window consistency, or Mixture-of-Agents candidate generation.A combined-prompt variant retains specialist guidance but removes independent candidate generation and selection.Source: §F.5 · p. 42

F.6 COMPUTATIONAL COST AND LATENCY

  • An average episode of 611.7 subtitle sentences requires about 4,970 model calls, 14.03M input tokens, 487.7K output tokens, and 3,501 tool calls; serial runtime is 25.94 minutes.The paper frames this as stronger translation quality at substantially greater test-time computation, with runtime of approximately 22–27 minutes across six languages.Source: §F.6 · p. 43, Table 20 · p. 43

F.7 HUMAN EVALUATION DETAILS

  • The human study ranks five anonymized systems on four broad criteria and Overall Preference across 20 continuous passages from 13 series, using 20 annotators.The criteria are Fidelity, Consistency, Language Quality, and Subtitle Quality.Source: §F.7 · p. 43
  • Human evaluation complements rather than replaces SubMQM because several localized or sparse fine-grained errors are difficult to estimate reliably at this study scale.SubMQM remains the tool for scalable, fine-grained diagnostic analysis.Source: §F.7 · p. 43

G MULTIMODAL EXPANSION

SMART’s multimodal extension exposes video and audio evidence as selectively invoked tools for subtitle ambiguities. Across six directions, the extension lowers Overall SubMQM penalties relative to text-only SMART.

G MULTIMODAL EXPANSION

  • SMART primarily processes text-based SRT subtitles but can incorporate video and audio evidence without changing its underlying translation workflow.Multimodal reasoning is invoked selectively rather than applied to every sentence.Source: G MULTIMODAL EXPANSION · pp. 44–45
  • Using Qwen3-Omni, tools retrieve timestamp-matched video clips or audio spans and return textual descriptions that translators can query when subtitle text leaves ambiguity unresolved.Example uses include identifying speakers, resolving visually grounded references, and interpreting emotional delivery or sarcasm.Source: G MULTIMODAL EXPANSION · p. 44
  • Across six directions, SMART + MM achieves the lowest Overall SubMQM penalty, reducing the average from 0.63 for text-only SMART to 0.57, about 9.5%.The largest improvements tend to occur in context-dependent dimensions, while surface-form dimensions change little.Source: G MULTIMODAL EXPANSION · p. 44
  • In the “See the striation?” example, audiovisual evidence identifies a single mark on a bone, enabling the translation to name it rather than use a generic plural.The passage connects this accuracy error to the dimension with the largest average gain.Source: G MULTIMODAL EXPANSION · pp. 44–45

H QUALITATIVE ANALYSIS ON RENDERED FRAMES

The rendered-frame analysis composites system subtitles onto identical video frames to make viewer-facing differences visible. Its six examples illustrate semantic contrasts but are not evidence of error frequency.

H QUALITATIVE ANALYSIS ON RENDERED FRAMES

  • Six English→Chinese lines from Nikita S01E01 are rendered for five systems, with each row using the identical frame so the target subtitle is the only variable.Claude Sonnet 4.6 is SMART’s backbone, while TransAgent provides a fixed-agent comparison.Source: H QUALITATIVE ANALYSIS ON RENDERED… · p. 45, Table 22 · p. 46
  • The six lines cover one direction, genre, and episode and illustrate error types rather than how often they occur; they also do not assess sparse formatting and convention errors.The passage identifies SubMQM, rather than visual inspection, as the appropriate instrument for those error types.Source: H QUALITATIVE ANALYSIS ON RENDERED… · p. 45
  • Four panels turn on correct source interpretation, including two involving tradecraft knowledge, while two show cases all systems understand where SMART writes more subtitle-like Chinese.The examples locate SMART’s margin in semantic dimensions, consistent with the table-level findings described in the passage.Source: H QUALITATIVE ANALYSIS ON RENDERED… · p. 45

I LIMITATIONS AND FUTURE WORK

The authors qualify SMART’s evaluation and applicability through limits in validation, computational cost, benchmark coverage, and persistent multimodal context.

Evaluation

  • Human-study and judge-swap checks cover only a subset of directions, and community Online subtitles are not a controlled human upper bound.These checks therefore do not establish judge robustness across every evaluated direction or provide a controlled human ceiling.Source: I LIMITATIONS AND FUTURE WORK · p. 46

Cost

  • SMART uses substantially more test-time computation than a single-call translator, limiting use in latency-bound or high-volume settings.Reducing model calls without losing consistency gains is identified as the most direct extension.Source: I LIMITATIONS AND FUTURE WORK · p. 46

Coverage

  • Subtitle Arena draws on one pre-2024, English-pivoted corpus, leaving non-English pairs and locales absent from it untested.The temporal split probes only the corpus's pre-2024 boundary; qualitative analyses cover one direction and few episodes.Source: I LIMITATIONS AND FUTURE WORK · p. 46

Modality

  • Multimodal evidence is optional and invoked per sentence; persistent series memory does not yet include speaker diarization, longer-horizon visual context, or audio prosody.Source: I LIMITATIONS AND FUTURE WORK · p. 46

Related papers

Loading 2609.38660v1…