Source-linked AI summary
Pearmut: Human Evaluation of Translation Made Trivial
Vilém Zouhar, Tom Kocmi
TL;DR
Human evaluation is often skipped because existing multilingual evaluation tools impose substantial setup and operational burdens. Pearmut addresses this gap with a lightweight platform supporting standardized translation-evaluation workflows, and case studies provide evidence of ease of use while noting remaining engineering requirements.
Problem
Human evaluation is frequently omitted because tool usability, setup complexity, annotator access, and operational overhead discourage multilingual researchers from using it.
Method
Pearmut provides a lightweight, pip-installable platform for multilingual and machine-translation evaluation with standardized protocols, document-level interfaces, and configurable assignment strategies.
Results
Pearmut’s case studies show ease of use and a strong directional signal for translation-evaluation suitability, while 55 of 82 ACL 2025 translation-model papers reported no human evaluation.
Takeaways & Limitations
Pearmut is intended to make standardized human evaluation a practical component of multilingual model development, diagnosis, and benchmarking.
Takeaways & Limitations
Pearmut still requires engineering skills to prepare campaign data and set up its running instance.
Abstract
from arXiv · showhide
Human evaluation is the gold standard for multilingual NLP, but is often skipped in practice and substituted with automatic metrics because it is notoriously complex and slow to set up with existing tools with substantial engineering and operational overhead. We introduce Pearmut, a lightweight yet feature-rich platform that makes end-to-end human evaluation as easy to run as automatic evaluation. Pearmut removes common entry barriers and provides support for evaluating multilingual tasks, with a particular focus on machine translation. The platform implements standard evaluation protocols, including DA, ESA, and MQM, and is extensible to support new protocols. It features document-level context, absolute and contrastive evaluation, attention checks, ESAAI pre-annotations and both static and dynamic assignment strategies. Pearmut enables reliable human evaluation to become a practical, routine component of model development and diagnosis rather than an occasional effort.
1 Introduction
Human evaluation is often omitted because existing tools make multilingual evaluation costly, difficult to configure, and hard to reproduce. Pearmut addresses these barriers with a platform supporting standardized multilingual evaluation workflows.
- Existing usability limitations, alongside annotator cost and access, discourage researchers from running human evaluations.
- 55 of 82 ACL 2025 translation-model papers (68%) reported no human evaluation.
- Ad-hoc, poorly documented protocols can make human evaluations irreproducible, unauditable, and noisier.
- Pearmut supports multilingual evaluation with standardized protocols, span-level error marking, pre-annotations, tutorials, attention checks, monitoring, and statistical analysis.
- The platform targets rapid prototyping, larger-scale evaluations, academic papers, and production-system benchmarking.
2 Related Work
Related tools cover different parts of human evaluation but leave a gap between deployability, domain specialization, and translation-specific functionality. Pearmut is positioned as a lightweight, translation-focused alternative with built-in statistical analysis.
- Human judgments can be solicited through scales, rankings, error marking, or combinations of these annotation protocols.
- Prior work improves annotation quality and efficiency through AI-prefilled error spans and selective allocation of human-annotated items.
- Appraise supports shared-task evaluation but is difficult to deploy or modify, while ChatbotArena targets crowdsourced free-form chat comparisons.
- General-purpose tools provide strong classification or span-tagging interfaces but lack translation-specific logic, state management, or annotation infrastructure.
- Pearmut combines lightweight deployment with multilingual specialization and includes statistical analysis using a two-sided t-test with α=0.05.
3 Pearmut
Pearmut is a minimal, configurable platform for machine-translation evaluation that combines standard protocols, document-level and contrastive interfaces, assignment strategies, and campaign monitoring. Its design prioritizes practical setup and efficient annotation under varied evaluation conditions.
- Pearmut is a pip-installable package with reasonable machine-translation defaults, requiring three commands to run a campaign.
- Annotation protocols: The platform distinguishes interfaces from protocols and supports DA, MQM, and ESA with different trade-offs in speed, depth, and annotation quality.
- Annotation protocols: All supported protocols can show multiple source documents and parallel outputs, providing document-level contrastive evaluation.
- Assignment: Pearmut offers task-based, single-stream, and optional dynamic assignment strategies for controlling item allocation and annotation efficiency.
- Dashboard: The dashboard monitors progress, annotation time, attention-check success, existing annotations, task redistribution, and model rankings with statistical analysis.
- Additional features: Additional features include non-destructive re-annotation, tutorials, attention checks, multimodal inputs, custom sliders, and approximate alignment.
4 Case Study with Researchers
A small practitioner study found a strong directional signal that Pearmut is suitable for translation evaluation and supports rapid setup. The platform trades some generic flexibility for a more specialized multilingual workflow.
- Five NLP practitioners compared Pearmut with four existing tools on setup time, annotation, ease of use, customizability, translation fitness, and future willingness to use.
- The study showed a strong directional signal for Pearmut’s translation-evaluation suitability, although its sample size prevents statistical generalization.
- Pearmut sacrifices generic flexibility through lower customizability to provide a more specialized translation and multilingual experience.
- An LLM-coding agent successfully wrote a setup script for Pearmut and Label Studio but not for the other evaluated tools.
5 Case Study with Annotators
Pearmut supported annotators performing translation error annotation comparably to Appraise while receiving higher satisfaction and supporting substantially greater server capacity. The tool has also been used for live multilingual evaluation and large-scale shared-task evaluation.
- Study design: Ten bilingual speakers annotated 540 English-to-multilingual translation outputs in Pearmut and Appraise using ESA.Interfaces were switched after every three documents, with randomized starting interfaces and different documents across tools to avoid priming.
- Results: Annotators performed ESA similarly fast and in-depth in Pearmut and Appraise, based on annotated errors, model separation, and inter-annotator agreement.Table 2 reports annotation times, model results, and 11-point-scale answers averaged across ten annotators and language pairs.
- Results: Annotators attributed Pearmut’s much higher satisfaction to responsiveness, parallel-reading layout, annotation-level switching, and optional character alignment.Some annotators found approximate parallel character alignment distracting when enabled.
- Adoption: Pearmut has been used with live annotators for speech-model diagnosis and for large-scale WMT 2026 evaluation across many language pairs.The cited deployments extend beyond the controlled annotator comparison.
- Scalability: Pearmut responds at least 3× faster for key requests and supports an expected maximum concurrent-user count 12× higher than the 158 users in large-scale WMT 2025 evaluation.The response-speed measurements covered next-item display, dashboard display, and result downloads.
6 Conclusion
Pearmut is an evaluation platform for practitioners conducting human evaluation of multilingual tasks and translation. Its purpose is to remove entry barriers, support good evaluation practices, and provide an easy-to-use workflow demonstrated through two case studies.
- Conclusion: Pearmut is an evaluation platform designed for practitioners human-evaluating multilingual tasks and translation.The conclusion frames the platform’s scope around these evaluation needs.
- Conclusion: Pearmut aims to make human evaluation as easy to use as automatic metrics while enabling unified evaluation and statistical testing.The platform is intended to support these practices without requiring practitioners to introduce new annotation protocols.
- Conclusion: Two case studies involving practitioners and annotators provide evidence of Pearmut’s ease of use.The conclusion summarizes the paper’s empirical evaluation of the platform.
Limitations
The evaluation studies used modest participant numbers, and Pearmut still requires engineering skills for campaign preparation and instance setup. These constraints may exclude some groups.
- Study size: The two user studies included 5 and 10 participants, respectively, although these samples exceed those reported for several other frameworks.The paper compares its participant counts with prior framework studies reporting 0+6, 0+1, and 0+0 participants.
- Accessibility: Pearmut still requires engineering skills to prepare campaign JSON data and set up the running instance.Setup may require basic command-line operations or port forwarding, potentially excluding some groups.
Ethics Statement
The user studies used informed consent, disclosed their experimental setup, and relied on benign data from an existing human-evaluated dataset. Collected data excluded annotator-identifiable information, while experiment managers remained responsible for obtaining consent.
- Ethics Statement: Expert participants gave informed consent, and the full experiment setup was disclosed without deception.The study texts came from WMT25, an existing human-evaluated dataset, and were deemed benign and exempt from ethics board approval.
- Data handling: Collected data contained no annotator-identifiable information, and user IDs were randomly generated strings.The data was stored on the server and accessible only through a privileged link.
- Ethics Statement: Experiment managers were responsible for ensuring that all annotators gave proper consent.This responsibility is stated alongside the description of data storage and access.
A.1 Architecture
Pearmut combines a lightweight client-server architecture with configurable evaluation protocols, assignment strategies, quality controls, and statistical reporting. Its design emphasizes responsive operation, reliable annotation storage, and efficient comparison of translation systems.
- Architecture: Pearmut comprises a server for static files and API requests, frontend annotation templates, and a monitoring dashboard.
- Architecture: Static client-side pages query the server directly, improving responsiveness and flexibility for implementing new protocols.
- Architecture: Runtime data is held in memory and synchronously flushed to an append-only log, avoiding heavy database overhead while preserving durability.
- Evaluation design: Dynamic assignment uses an initial cold-start estimate and ranking-based sampling to identify the best model with fewer costly evaluations.
- Evaluation design: Pearmut supports contrastive evaluation that compares models close in performance, concentrating annotations where they provide more comparative signal.
- Dashboard and quality: The dashboard hides intermediate model results until explicitly requested to reduce accidental decisions that could bias an ongoing campaign.
- Dashboard and quality: Pearmut provides predefined statistical comparisons and tutorial or attention checks to support consistent reporting and annotation quality.
- Evaluation setup: The researcher case study measured installation, campaign setup, annotation completion, result retrieval, usability, customizability, and translation fit.
C Evaluation of Agentic Workflow Compatibility
The agentic-workflow evaluation compared Pearmut with other annotation tools by asking researchers and an AI agent to configure translation campaigns and interpret results. The reported findings attribute Pearmut’s successful integration to its standard package installation and JSON configuration.
- Agentic workflow: The evaluation instructed an AI agent to prepare an English-to-German translation campaign using tool documentation and example translation inputs.
- Results: Pearmut’s reported success was linked to a pip-installable CLI and JSON configuration that fit standard package-management practices and AI-assisted research pipelines.
- Human evaluation: Researchers evaluated annotation interfaces by marking error spans across alternating tools and rating speed, clarity, and interaction effort.
- Experimental design: The comparison used translations with intentionally varied quality so participants could assess whether each interface supported the same functional task.
D.3 Results
Pearmut's evaluation study found comparable annotation quality signals to Appraise, while its logging, scalability model, and simple setup support practical deployment. The platform also provides configurable assignment, validation, and tutorial features for multilingual annotation campaigns.
- Annotation results: Pearmut and Appraise produced comparable inter-annotator correlation signals in the modest English→Czech study.Pearson correlation was used for global and group-by-model analysis, while Kendall’s τ_b was used for group-by-item ranking.
- Annotation results: Detailed action logging records when annotators create or delete spans and assign scores across document segments.The time diagram exposes annotation behavior that similar tools often cannot reveal because they lack detailed logging.
- Scalability: 50ms per request implies a theoretical maximum throughput of 2600 requests per second under perfect timing.This calculation uses an average annotation time of 130 seconds per item and assumes single-thread execution with worst-case cold reload.
- Scalability: Pearmut’s 99%-within-1000ms service target yields an estimated capacity of 15.39 requests per second and about 2000 parallel users.The model uses a single-thread M/M/1 queue with a 20 requests-per-second service rate.
- Campaign configuration: Pearmut campaigns can be launched with three commands and support shared-pool or task-based assignment strategies.Task-based assignment can enable contrastive evaluation, shuffle model order to reduce positional bias, and attach tutorials or attention checks through validation rules.
G Usage of Human Evaluation in Papers
The paper’s review found that human evaluation remains uncommon in recent translation research, with many papers reporting no human evaluation. The authors characterize this as an observation of missing evaluation and relate it partly to setup barriers and reliance on automated metrics.
- Review results: Among 108 NAACL, ACL, and EMNLP 2025 translation-related papers, 55 papers, or 51%, lacked human evaluation.The review classified 26 papers as containing human evaluation and 27 as not requiring it.
- Review results: The reviewed papers used diverse human-evaluation protocols, including direct assessment, ranking, error categorization, pairwise judgments, and qualitative analysis.Examples included DA, MQM, terminology verification, adequacy, fluency, comprehensibility, and would-share decisions.
- Interpretation: The authors stress that their categorization observes missing human evaluation rather than criticizing the classified papers.They also note that the analysis used a smaller scale, conservative criteria, and excluded papers that might not need human evaluation.