Source-linked AI summary
TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming
Yibo Hu, Yu Qian, Mao Gu, Yingfan Tao, Yuhao Chen, Yongdong Luo, Zhuoqun Liu, Meiguang Jin, Junfeng Ma
TL;DR
E-commerce live streams distribute product evidence across noisy, temporally extended modalities, creating a need for faithful omni-modal understanding. TLive-Omni combines unified multimodal representations, Per-vGrid temporal organization, staged supervision, and Faithful-RFT, achieving strong live-commerce performance and broad generalization.
Problem
E-commerce live streaming requires faithful integration of product evidence distributed across speech, video, images, text, and user queries over time.
Method
TLive-Omni combines a unified multimodal model, Per-vGrid audio–video organization, three-stage supervision, Faithful-RFT, and capability-oriented data construction.
Results
TLive-Omni shows strong performance across live-commerce tasks and leading results on several general-purpose multimodal benchmarks while remaining competitive on others.
Takeaways & Limitations
The model provides broad live-commerce understanding across speech, audio, product, text, temporal, video, and omni-modal QA capabilities.
Takeaways & Limitations
TLive-Omni focuses on understanding rather than generation or full-duplex real-time interaction, with future work targeting broader, noisier, and more ambiguous streams.
Abstract
from arXiv · showhide
E-commerce live streaming requires omni-modal understanding of noisy, temporally extended streams, where product facts are distributed across speech, video frames, product images, overlaid text, and user queries. We present TLive-Omni, an omni-modal understanding model tailored to live-commerce scenarios. It maps image, video, audio, and text inputs into a unified representation space. For long-form live streaming analysis, we introduce Per-vGrid, a timestamped token organization that groups each video grid with its temporally corresponding audio within explicit boundary tokens to facilitate temporal alignment. We design a three-stage supervised training recipe that progressively develops live-commerce understanding, from omni-modal perception to instruction-following responses. We then propose Faithful-RFT, a reinforcement fine-tuning stage that further improves answer faithfulness and expression quality while meeting real-time demands, scoring final responses directly with task-verifiable feedback rather than optimizing for reasoning-style exploration during rollout. Moreover, TLive-Omni is supported by a scenario-oriented atomic capability taxonomy and a compact data production engine that converts live-commerce audio, image, and video streams into training signals for speech recognition, speaker analysis, product visual grounding, text recognition, temporal grounding, video dense caption, and omni-modal QA, etc. For scalable training, a synchronized length-grouped sampler reduces padding while preserving comparable workloads across workers, while a lightweight dynamic sampling strategy regenerates rollout groups with near-zero reward variance to maintain meaningful relative advantages for GRPO. Experiments on e-commerce live streaming benchmarks demonstrate strong performance across live-commerce domain tasks, together with excellent generalization on general benchmarks.
1 Introduction
E-commerce live streams require models to integrate heterogeneous, temporally distributed evidence across speech, video, images, text, and queries. TLive-Omni addresses this setting with unified multimodal modeling, capability-oriented supervision, and strong benchmark performance.
- Live-commerce product facts are distributed across speech, video frames, product images, overlaid text, and user queries over different moments.The setting requires joint interpretation, temporal alignment, and faithful answers for tasks including recognition, grounding, and question answering.
- TLive-Omni integrates heterogeneous modalities through a unified interface and supports up to 256K tokens for extended live-streaming segments.Its Per-vGrid organization places temporally matched video grids and audio together with explicit boundaries and timestamps.
- The model uses a scenario-oriented taxonomy and data engine to convert live-commerce streams into capability-specific supervision for staged training.Covered capabilities include speech recognition, speaker analysis, visual grounding, text recognition, temporal grounding, dense captioning, and omni-modal QA.
- TLive-Omni achieves strong performance on business-oriented live-commerce tasks and leading results on several general-purpose benchmarks while remaining competitive on the rest.
2 Architecture
TLive-Omni combines a Qwen3.5 vision-language backbone with an aligned AuT audio encoder. Per-vGrid preserves timestamped audio–video correspondence by organizing matched evidence into explicit local spans.
- TLive-Omni uses Qwen3.5 as its vision-language backbone and maps merged visual features into the backbone embedding dimension.Images and sampled videos are represented as spatially merged visual tokens before alignment.
- External ASR-only processing would discard speech–video timing and paralinguistic cues, so TLive-Omni retains audio as a first-class input modality.The design targets product facts carried by host speech that may not be visible in video frames.
- The AuT audio encoder consumes 128-dimensional mel features at 16 kHz and compresses speech to approximately 13 tokens per second.A two-layer aligner projects audio features into the backbone embedding space.
- Per-vGrid places each temporal video grid and its corresponding audio segment in one contiguous span with explicit boundaries and textual timestamps.Timestamps and audio spans follow the realized sampled frame indices rather than only the requested sampling rate.
3 Supervised Fine-Tuning: Data and Recipe
TLive-Omni constructs modality-specific supervision and trains it through a staged fine-tuning recipe. Synchronized length grouping addresses heterogeneous batching costs by aligning workloads without merging source samples.
- Data Construction: Audio, image, and video pathways use separate filtering and quality-control processes to transform noisy sources into task-grounded supervision.Figure 2 summarizes source, construction, filtering, and output stages for the three-stage recipe.
- Data Construction: The audio pathway addresses fast, domain-specific speech and overlapping speakers through preprocessing and cross-model agreement voting.These conditions include brand names, materials, colors, model numbers, and short interjections.
- Data Construction: The image pathway uses a VLM Detector–Judger loop and source-consistency checks to construct grounding, caption, selling-point, and Image QA supervision.
- Data Construction: The video pathway combines shot splitting, ASR, visual description, and language-model aggregation for dense captioning, Video QA, and temporal grounding.
- Supervised Fine-Tuning Recipe: The three-stage SFT recipe progresses from audio–language alignment to audio understanding and then joint multimodal adaptation.The progression separates modality alignment, capability learning, and full multimodal adaptation.
- Synchronized Length-Grouped Sampling: Synchronized length-grouped sampling sorts samples by modality and token length into fixed-size global batches, reducing padding and balancing worker workloads.Workers share epoch-dependent randomness and receive disjoint local batches from the same global batch.
4 Faithful-RFT
Faithful-RFT extends the three-stage supervised model with task-verifiable reinforcement fine-tuning that directly scores final responses, targeting faithful, timely live-commerce answers. It organizes multimodal tasks and rewards conditionally, while dynamic rollout resampling preserves informative GRPO preference signals.
- Framework: Faithful-RFT applies GRPO after Stage 3, using task-verifiable rewards that directly score final responses instead of rewarding reasoning length or visible reasoning traces.This is intended to improve answer faithfulness and expression quality while avoiding unnecessary generation overhead for real-time live streaming.
- Data organization: Four mixed data streams preserve modality-specific task schemas: image, video with audio, video without audio, and audio.Examples include image QA, visual grounding, OCR, captioning, Omni QA, temporal grounding, ASR, and audio QA.
- Rollout strategy: Dynamic resampling retains groups with nonzero reward variance and regenerates near-identical groups, keeping GRPO updates informative.Identical rewards across a response group provide no relative preference signal for the group-relative advantage.
- Reward design: Task-conditioned reward routing assigns each example only the reward functions applicable to its task, accommodating different notions of correctness within multimodal batches.Each reward function declares an applicable task set, while inapplicable outputs are treated as invalid.
- Reward design: Rule-based rewards evaluate deterministic or machine-checkable tasks such as multiple-choice questions, visual grounding, and OCR, while some LLM-based extraction remains deterministically scored.The reward pool is organized by evaluation mechanism and applied conditionally across the four data streams.
- Model variants: The supervised recipe and Faithful-RFT produce TLive-Omni-4B and TLive-Omni-9B variants.The variants result from combining the three-stage supervised recipe with Faithful-RFT.
5 Evaluation
TLive-Omni is evaluated on live-commerce capabilities and broad multimodal benchmarks. It performs strongly across audio, image, video, and omni-modal tasks, with broad generalization beyond the target domain.
- Evaluation scope: The evaluation covers live-commerce tasks spanning speech, speaker-attributed ASR, audio understanding, product grounding, text understanding, temporal grounding, video captioning, video QA, and shot understanding.General benchmarks additionally test image reasoning, hallucination, OCR, grounding, spatial reasoning, long-context video understanding, and omni-modal perception.
- Live-Commerce Evaluation: TLive-Omni-9B achieves the lowest CER on live-commerce ASR, while both variants show strong speaker-attributed ASR performance by cpWER.CER measures character errors; cpWER minimizes total word error over speaker assignments.
- Live-Commerce Evaluation: The two variants achieve the highest Prod AP, text localization, and classification scores, plus the lowest recognition edit distances among evaluated models.They also remain competitive with Gemini 3.5 Flash on Live AP.
- Live-Commerce Evaluation: TLive-Omni-9B leads live-commerce temporal grounding, Video QA Accuracy, and Dense Caption Accuracy while achieving the lowest Dense Caption Hallucination Rate.TLive-Omni-4B records the second-best open-source results on these four metrics, while the variants also rank highly on shot understanding.
- General Benchmark Evaluation: On general benchmarks, TLive-Omni maintains strong multimodal generalization and improves over Qwen3.5 4B and 9B backbones on most reported benchmarks.The 9B model leads open-source models on several video benchmarks, while the 4B model achieves the highest open-source temporal-grounding mIoU on three TimeLens-Bench datasets.
- Overall findings: Across both evaluation axes, the 9B variant achieves the best open-source results on many metrics, while the 4B variant also reaches best or second-best open-source results across multiple benchmarks.The combined pattern spans audio, image, video, and omni-modal perception and reasoning rather than only live-commerce tasks.
6 Conclusion and Limitations
TLive-Omni combines scenario-oriented design, data construction, training, and evaluation for e-commerce live-stream understanding. It performs strongly across target and general benchmarks, while remaining scoped to understanding rather than generation or full-duplex interaction.
- TLive-Omni unifies image, video, audio, and text understanding with Per-vGrid, staged supervision, Faithful-RFT, and an atomic capability taxonomy.The taxonomy supports capabilities spanning speech, visual grounding, text recognition, temporal grounding, dense captioning, and omni-modal QA.
- The model achieves strong performance across live-commerce tasks and competitive results across general image, video, and omni-modal benchmarks.The reported live-commerce coverage includes transcription, speaker-attributed ASR, grounding, text understanding, temporal grounding, captioning, video QA, and shot understanding.
- TLive-Omni remains focused on understanding rather than generation or full-duplex real-time interaction.Future work targets broader public-benchmark coverage, robustness to longer and noisier streams, and calibration under incomplete or ambiguous multimodal inputs.
B Evaluation Metrics
The evaluation defines task-specific metrics for speech, OCR, grounding, temporal understanding, captioning, and structured shot understanding. These metrics combine standard error, overlap, classification, and evaluator-based measures.
- Speech metrics: CER measures character substitutions, deletions, and insertions relative to the ground-truth transcript after transcript normalization.CER is computed as (S + D + I)/N, where N is the number of ground-truth characters.
- Speech metrics: cpWER concatenates speaker-specific transcripts and minimizes total word error over predicted-to-ground-truth speaker assignments.Empty streams are added when the two sides contain different numbers of speakers.
- Grounding and OCR: Product grounding uses AP at IoU 0.5, while OCR localization uses matched-box F1 and recognition uses normalized edit distance on IoU-matched text boxes.Recognition distance is lower-is-better and normalizes edit distance by the maximum predicted or ground-truth text length.
- Temporal and video metrics: Temporal grounding averages interval IoU, while audio description and dense video captioning are evaluated through annotation-grounded multiple-choice questions.The questions cover entities, attributes, actions, scenes, events, and temporal relations.
- Temporal and video metrics: Generated descriptions are scored by an evaluator LLM without original-input access, using answer accuracy and hallucination rate.Shot understanding separately evaluates layout, shot size, camera angle, and content category tags, with partial credit for multi-choice subsets.
C Additional In-Context ASR Results
In-Context ASR tests whether domain-specific keyword prompts improve recognition of product and commerce terminology. Results are reported across keyword-list sizes using recall and CER.
- In-Context ASR supplies a candidate keyword list before transcription to test recognition of product names, brands, and other domain terms.Keyword-list sizes range from 0 to 1000, with zero keywords serving as the no-context baseline.
D Training and Faithful-RFT Details
The training details specify shared optimization settings for the three supervised stages and Faithful-RFT hyperparameters. The supplied passages also identify the benchmark-prompt appendix and an In-Context ASR results table.
- Supervised training: All three SFT stages use AdamW, cosine learning-rate scheduling, weight decay, gradient clipping, ZeRO-3 sharding, and gradient checkpointing.Each stage is trained for one epoch over its stage-specific data mixture.
- Additional In-Context ASR: Table 9 reports In-Context ASR recall and character error rate for different keyword-list sizes.Best values are bolded and second-best values are underlined.
- Faithful-RFT: Faithful-RFT uses groups of G = 8 candidate responses, reward smoothing ϵs = 10^-4, policy-ratio bounds ϵl = 0.2 and ϵh = 0.28, and KL coefficient β = 0.1.
- Evaluation prompts: The general-benchmark appendix groups evaluation prompts by input modality.
E.1 Image Benchmarks
The image benchmarks use task-specific answer instructions spanning multiple-choice selection, structured JSON responses, yes/no answers, OCR transcription, and visual grounding coordinates.
- MathVista and similar tasks require selecting the correct answer from listed options.
- CharXiv and SimpleVQA require step-by-step reasoning followed by a final answer in a specified format.
- DynaMath provides a question-only prompt without an additional answer-format instruction.
- Some benchmarks require JSON containing a detailed solution and a concise short answer, with the answer represented by its choice letter.
- HallusionBench, OCRBench, CC-OCR, and RefCOCO respectively request yes/no answers, image text, text-only transcription, and bounding-box coordinates.
E.2 Video Benchmarks
The video benchmarks specify distinct response formats for multiple-choice, open-ended, and temporal-localization tasks, including final-answer markers and timestamp ranges.
- VideoMMMU presents options and instructs the model to end with an answer letter, while also labeling the task open-ended.
- Charades-TL, ActivityNet-TL, and QVHighlights-TL request reasoning followed by a final answer appended to the response.
- Temporal-localization queries require returning exactly one timestamp range in mm:ss-mm:ss format.