SimulEval: An Evaluation Toolkit for Simultaneous Translation

1 Introduction

• Inconsistent latency evaluations use imprecise text and speech segmentation and often lack released decoding and measurement implementations, hindering comparison and progress.Source: §1 · p. 1
• SIMULEVAL is an open-source toolkit that simulates real time and evaluates quality and latency for simultaneous text and speech translation.Source: §1 · p. 1
• Its server supplies text or audio on request while a client agent runs the policy and state tracks execution information and generates translations.Source: §1 · p. 2
• The toolkit supports built-in quality and latency metrics, customized evaluation functions, and discussion of adapting text latency metrics to speech.Built-in latency metrics include AP, AL, and DAL.Source: §1 · p. 2
• An interactive interface visualizes the agent’s online decoding, and the toolkit is suitable for shared tasks such as IWSLT 2020.Source: §1 · p. 2

2 Task Formalization

• The evaluation corpus pairs a source sequence X with a reference Y*, while the system takes X as input and generates hypothesis Y.Source: §2 · p. 2
• For text, each source segment is a word; for speech, each segment is a raw audio chunk with duration Tj.Source: §2 · p. 2
• A simultaneous system begins generating from partial input and alternates between reading a source segment and writing a target segment.Source: §2 · p. 2
• Translation quality compares the completed hypothesis with the reference, while latency metrics are calculated from the sequence of output delays.Source: §2 · p. 2

3.1 Existing Text Latency Metrics

• AP is not length-invariant or evenly distributed on [0, 1], so equal metric changes can represent different levels of difficulty.For a wait-3 model, AP is 0.72 when |X| = |Y| = 10 but 0.52 when |X| = |Y| = 100.Source: §3.1 · p. 2
• AL compares a system with an ideal wait-0 policy matched to its predictions, and its value directly describes lag behind that policy.The section characterizes AL as length-invariant and intuitive.Source: §3.1 · pp. 2–3
• DAL imposes a minimum delay of 1/γ after each operation and includes tokens generated after the source is fully read.The minimum delay prevents DAL from recovering from lag once incurred.Source: §3.1 · p. 3

3.2 Adapting Metrics to the Speech Task

• For speech, AP is adapted straightforwardly to use the speech-task input.The section introduces an adaptation of the three text latency metrics for simultaneous speech translation.Source: §3.2 · p. 3
• The straightforward speech AL adaptation can misrepresent models that stop early, since they cease reading input and can appear to lag less than the ideal policy.Early end-of-sentence generation can occur during a long pause, even when the source has not been consumed; the section instead defines the ideal policy using the reference rather than the system hypothesis.Source: §3.2 · p. 3, Figure 1 · p. 3
• Speech DAL retains its text-task equations but uses a newly defined γ.The section states that speech DAL still uses Eqs. (4) and (5).Source: §3.2 · p. 3

4 Architecture

• The server and client can run separately or jointly and communicate through RESTful APIs.Figure 2 presents the client as executing the policy and the server as operating the evaluation.Source: §4 · p. 4, Figure 2 · p. 4

4.1 Server

• On a READ action, the server sends a source segment; on a WRITE action, it receives a predicted segment and records its delay before evaluating the instance.The server collects hypotheses, delays, and speech durations for corpus-level metrics after all hypotheses are complete.Source: §4.1 · p. 4

4.2 Client

• The client separates a user-defined agent, which executes the policy and generates hypotheses, from a state component that handles processing and context.The state provides preprocessing, postprocessing, and context memorization functions.Source: §4.2 · p. 4

5.1 User-Defined Agent

• A user-defined agent implements a policy that chooses read or write actions and a predict function that produces target output after a write decision.Agents may also preprocess inputs and postprocess outputs, such as extracting speech features or merging subwords.Source: §5.1 · pp. 4–5

5.2 User-Defined Client

• Users can replace the default client with a custom implementation and connect it to the server through RESTful APIs.The APIs can support clients written in another programming language or implementations designed for greater efficiency.Source: §5.2 · p. 5

5.3 Evaluation

• SIMULEVAL can start evaluation from a defined agent, either in one command or through separate client and server commands.It saves intermediate results and corpus-level metrics and can resume an interrupted evaluation.Source: §5.3 · p. 5

5.4 Visualization

• A web interface visualizes online decoding and lets users inspect the translation corresponding to a selected point in the process.The paper illustrates the interface with simultaneous speech translation.Source: §5.4 · p. 5

5.5 Case Study: IWSLT 2020

• The authors encourage researchers to use SIMULEVAL when reporting latency to reduce inconsistencies in metric computation and support fair comparisons.Source: §5.5 · p. 5
• For the IWSLT simultaneous speech translation task, participants submitted Dockerized systems with their own agent files, allowing organizers to score submissions consistently without releasing the test source.The shared evaluation also covered systems implemented in different frameworks.Source: §5.5 · p. 6
• The IWSLT evaluation used a client-server design so organizers could run submissions under the same conditions while keeping the test set, including its source side, unreleased.Source: §5.5 · p. 6

• SIMULEVAL evaluates simultaneous text and speech translation in a simulated real-time scenario using a server-client scheme and user-defined agents.It also provides a visualization interface for tracking online decoding.Source: §6 · p. 6
• The toolkit's general design allows evaluation across different frameworks.Source: §6 · p. 6
• The authors encourage future simultaneous speech and text translation research to use SIMULEVAL for accurate, standardized latency comparisons between systems.Source: §6 · p. 6