BOOM: Beyond Only One Modality KIT's Multimodal Multilingual Lecture Companion

1 Introduction

• Providing educational content in learners’ native languages is presented as a way to improve accessibility and engagement with complex concepts.The paper identifies this need in the context of increasingly global higher education.Source: §1 · p. 1
• Lecture materials combine audio, text, and visual content, so multilingual localization must account for more than spoken language.Slides can contain definitions, formulas, diagrams, and specialized terminology that provide context for translation and downstream tasks.Source: §1 · p. 1
• BOOM extends the Lecture Translator with OmniFusion for slide-aware speech translation and adds an open-source pipeline that translates text embedded in slides while preserving layout.Together, these components form a multimodal lecture-localization pipeline.Source: §1 · p. 2
• Including images in speech translation improves downstream NLP-task performance across different LLMs and language pairs.The paper also evaluates OCR models and unimodal versus multimodal NMT for image translation.Source: §1 · p. 2

2 System Description

• The system performs multimodal speech translation, uses the resulting transcripts for downstream LLM tasks, and translates slide text while preserving layout and visual coherence.The downstream tasks include summarization and question answering.Source: §2 · p. 2
• A PDF viewer synchronizes captions with the presenter’s selected slide and identifies the slide associated with each audio segment to supply context for multimodal speech translation.The viewer displays slides with overlaid captions.Source: §2 · p. 2
• The image-translator overview indicates that its steps are model-based except for drawing, which uses heuristic rules.The figure’s arrows indicate inputs to each step.Source: Figure 2 · p. 3

2.1 Multimodal Speech Translation

• For live lectures, the system adopts multilingual OmniFusion, fine-tunes it on noisy data, and follows the Lecture Translator policy combining voice-activity detection with Local-Agreement.The paper identifies existing systems as either unimodal or multimodal without multilingual support, while many are trained offline with fixed segmentation.Source: §2.1 · p. 3
• The PDF viewer tracks the slide shown during each audio segment, enabling a mid-segment screenshot to provide visual cues that improve translation, especially for technical content.The same interface lets participants follow translations in real time.Source: §2.1 · p. 3

2.1.1 Summarization & Question-Answering

• For summarization, translated lectures are divided into chapters to limit context-window overflow and reduce topic drift; chapter-level compressed transcripts and length-controlled summaries are generated first in English, then translated.Transcript compressions use ratios such as 50, 70, and 90 percent.Source: §2.1.1 · p. 3
• For question answering, the chapter-organized English transcript is queried with an LLM using Retrieval-Augmented Generation, and the resulting answers are translated into target languages.Source: §2.1.1 · p. 3

2.2 Slide Translation

• Editable slide text is extracted with a PowerPoint parser and translated with unimodal machine translation, while embedded image text requires a pipeline that detects, recognizes, translates, and re-renders it.The image pipeline aims to preserve layout and visual meaning.Source: §2.2 · p. 4

2.2.1 Optical Character Recognition

• PaddleOCR v5 extracts recognized text and bounding boxes from slide images, but its detections need layout analysis to form coherent translation units.Source: §2.2.1 · p. 4

2.2.2 Layout Analysis

• Hi-SAM groups OCR boxes into block- and line-level segments while preserving structural cues such as grouping, font size, and color for re-rendering.Bullet items and diagram labels are examples of text grouped to retain consistent formatting.Source: §2.2.2 · p. 4

2.2.3 Multimodal Translation

• Text blocks are translated with OmniFusion, adapted from Qwen Omni 2.5 7B and SeedX PPO 7B, using the slide image as context for short, ambiguous, or visually grounded text.Source: §2.2.3 · p. 4

2.2.4 Inpainting

• Simple-LaMa removes the original text regions by inpainting the background before translated text is inserted.The lightweight model is described as reconstructing the background with minimal artifacts.Source: §2.2.4 · p. 4

2.2.5 Drawing

• A heuristic drawing module estimates the original text styling and positions translated text in the same layout and line structure because repeated diffusion-based edits degraded clarity.The approach is intended to preserve alignment, spatial organization, and visual coherence.Source: §2.2.5 · p. 4

3.1 Evaluation Data & Metrics

• Evaluation uses VISTRA for image translation and MCIF ACL talks for downstream tasks, with CER, TER, latency, translation metrics, and normalized BERTScore.VISTRA provides ground-truth OCR and translations; MCIF supports summarization and question-answering evaluation.Source: §3.1 · pp. 4–5

3.2 Image Translation

• PaddleOCR v5 offers the selected trade-off among accuracy, latency, and language coverage, while block-level segmentation improves translation over line-level segmentation.Layout analysis and translation are the slowest components; OCR and image rendering add relatively minor overhead. Figure 3 illustrates context-sensitive translation of identical text.Source: §3.2 · pp. 5–6, Figure 3 · p. 5
• OmniFusion outperforms unimodal translation in most languages, but unimodal translation performs better in Russian; ground-truth OCR and segmentation yield the best performance.Block-level segmentation improves translation over line-level segmentation.Source: §3.2 · p. 5, Table 2 · p. 5

3.3 Downstream Tasks

• Audio+image transcripts improve summaries across most language–model settings, with slight Chinese degradation, while QA gains are smaller and regress in 4 of 12 combinations.The downstream evaluation reports BERTScore on MCIF.Source: §3.3 · p. 6, Table 3 · p. 6

4 Related Work

• Earlier lecture-translation systems primarily use audio, whereas this work incorporates slide cues and produces translated audio and slides in multiple languages.The related-work discussion describes image-to-image translation as under-explored, with much prior work focused on text translation inside images rather than aligned re-rendering.Source: §4 · p. 6
• The image-translation pipeline is a modular foundation for integrating OCR, translation, and rendering models without implementing additional components.The cited discussion identifies the gap as aligned re-rendering of visual content.Source: §4 · pp. 6–7

5 Conclusion

• The system translates lectures across modalities and languages, using multiple input modalities to produce multiple output modalities.Source: §5 · p. 7
• Future work includes human evaluation of translated slides and audio to guide targeted system improvements.Source: §5 · p. 7

Limitations

• VISTRA is a proxy that does not fully reflect lecture-domain translation quality.Source: Limitations · p. 7
• Rendered-slide quality, including layout preservation and visual coherence, still requires human evaluation.Source: Limitations · p. 7
• Summarization and question answering are evaluated after the whole talk, not in a live-lecture setting; timeline-aligned questions would better match the use case.Source: Limitations · p. 7

• Diasumm is listed as flexible summarization of spontaneous dialogues in unrestricted domains.Source: §A.2 · p. 10
• Patimt-bench is described as a multiscenario benchmark for position-aware text-image machine translation in large vision-language models.Source: §A.2 · p. 10
• InImageTrans is listed as a multimodal LLM-based text-image machine translation work by Min Zhang (2025).Source: §A.2 · p. 10

• Table 4 reports inference time for each pipeline step when translating the image shown in Figure 2.Source: Table 4 · p. 10

• TTS output can play simultaneously during a talk or consecutively, pausing when the system recognizes speaker input.VITS/VITS2 and Kokoro-82M generate audio, with a rule-based streaming algorithm segmenting the input text.Source: §A.1 · p. 10
• Figure 4 shows English and German versions of the Feynman video.Source: Figure 4 · p. 11
• The summarization and question-answering interface displays summaries for each chapter in all languages.Source: Figure 5 · p. 12
• The slide viewer supports multilingual navigation and independent slide browsing, with a sync toggle to realign to the live presentation.Source: Figure 6 · p. 12
• The interface displays translations in multiple languages alongside the current slide.Source: Figure 7 · p. 12
• The full-screen participant view shows slides with German caption overlays.Source: Figure 8 · p. 13