Source-linked AI summary
Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction
Tencent WorkBuddy Bench Team, Siqi Cai, Shaopeng Chen, Xiang Fei, Yong Mao, Zihan Xu, Zhiheng Lyu, Zhijian Shao, Yuchen Shi, Shuwen Zhang, Chaofan Qiu, Linjie Che, Xiaoxi Zhao, Feng Wu, Kai Zhang, Chaofan Zhu, Yubin Qi, Xiaoyun Liang, Peijie Dong, Yunhao Zhang, Yuanjie Zhu, Ling Jiang, Xianjun Zhang, Zhehang Chu, Anyuan Sang, Zhen Feng, Sen Nie, Shi Wu, Yuanzhen Xu, Xin Li, Ning Yang, Zhiqiang Dong, Hande Dong, Qiang Lin, Yi Liu, Yunsheng Wu, Ke Li, Xing Sun
TL;DR
Existing coding-agent benchmarks are either public and vulnerable to web-search contamination or closed and difficult to audit across real organizational work. Tencent WorkBuddy Bench addresses this gap with an openly released, multi-domain benchmark built from distribution-informed, role-played tasks and reports a split leaderboard: Claude Opus 4.8 leads five boards, GLM-5.2 leads two, and GPT-5.5 leads one.
Problem
Existing benchmarks offer limited evidence for contamination-resistant, openly auditable evaluation across the diverse work domains and real-work distributions faced by coding agents.
Method
Tencent WorkBuddy Bench constructs and openly releases role-played tasks across Code, Web, Office, and Security under a shared reproducible protocol with domain-specific scoring.
Results
Claude Opus 4.8 leads five boards, GLM-5.2 leads two Security boards, and GPT-5.5 leads one Office board, showing board-dependent leadership.
Takeaways & Limitations
The benchmark provides an openly auditable comparison of coding agents across complementary work domains, while preserving separate scores because the subsets use different scoring instruments.
Takeaways & Limitations
Because task directories, tests, and reference solutions are public, future model training may scrape released content and erode contamination resistance over time.
Abstract
from arXiv · showhide
We introduce Tencent WorkBuddy Bench, a multi-domain evaluation suite for coding agents; this report documents its construction methodology, scoring protocol, and a cross-model leaderboard. At its core is a unified evaluation framework for constructing and running distribution-informed coding-agent tasks across four work domains - Code, Web, Office, and Security. Rather than adapting public issue text, every task is reverse-engineered from a real commit, pull request, or business scenario and rewritten as a short, colloquial, role-played request, so that a task's prompt is not recoverable by web-searching the underlying issue, pull request, or commit thread. Because the dataset is released openly - task directories, environment images, evaluation harness, tests, and reference solutions - contamination resistance rests on this construction together with dataset versioning rather than on secrecy. The four subsets - repository-level engineering, front-end development, office and business workflows, and red-/blue-team security - probe complementary facets of real work, each with its own verification style. All are packaged in a uniform task-directory format and run, under a uniform and reproducible protocol, on two agent harnesses (CodeBuddy Code and Claude Code); the full open release makes the benchmark reproducible end to end and directly auditable, since any third party can re-run each task and inspect its content. Because each subset uses a different scoring instrument, scores are not comparable across subsets and the suite reports no suite-wide average. We report a cross-model leaderboard across several model families.
1 Introduction
Tencent WorkBuddy Bench is an open, reproducible, four-domain benchmark for coding agents designed to reflect real organizational work while resisting contamination from web-searchable prompts. It combines shared task infrastructure with subset-specific scoring and reports cross-model results without a suite-wide average.
- Motivation: The benchmark addresses public suites’ memorization risk and closed production benchmarks’ lack of inspectability, distribution transparency, and generalizability.Public suites expose problem statements and often solutions, while closed benchmarks prevent external inspection of task distributions and selection bias.
- Benchmark scope: Tencent WorkBuddy Bench evaluates repository engineering, front-end development, office workflows, and security workflows under one reproducible harness.The four parallel subsets share a common task-directory format, admission protocol, and execution infrastructure, and run on two agent harnesses.
- Evaluation protocol: Scoring is domain-specific: Code uses hidden tests, Web combines rule checks and multiple judges, and Office combines deterministic checks with an LLM Judge.The tests are hidden from agents during solving but included in the open release; scores therefore are not directly comparable across subsets.
- Contamination resistance: Each task is reverse-engineered from a real commit, pull request, or business scenario and rewritten as a colloquial, role-played request that withholds solution-revealing context.The construction specifically targets web-searchable prompt contamination by omitting the root cause, reference diff, and framing that would reveal the solution.
- Distribution-informed construction: Task distributions match internal usage taxonomies rather than reusing raw user requests, enabling full public release without exposing prompts, sessions, or user data.The methodology matches category, task-mode, and difficulty distributions while keeping the underlying real-usage data private.
2 Task Construction
Tencent WorkBuddy Bench constructs tasks from real artifacts or business scenarios, then rewrites them as realistic, deliberately underspecified requests in isolated, self-contained workspaces. Evaluation assets remain hidden during solving but are publicly released afterward, making the benchmark reproducible and auditable while resisting searchable-prompt contamination.
- Task sources: Tasks originate from real commits, pull requests, historical CVEs, or concrete business scenarios, with category and difficulty mixes matched to aggregate real-request distributions.No raw user prompt, session, or other user data enters construction.
- Rewriting protocol: Requests are short, colloquial, role-played, and deliberately underspecified rather than copied from public issue text, requiring agents to recover missing context from the workspace.They may omit target files, schemas or interfaces, edge-case handling, and the precise change boundary.
- Post-episode evaluation isolation: Evaluation assets are withheld during each episode and introduced only after the agent finishes, while remaining public in the released dataset.This isolation applies across subset-specific checks, rubrics, and evaluation procedures.
- Contamination-resistant task construction: Searchable-prompt contamination is mitigated by reverse-engineering source artifacts into role-played requests, so the instruction cannot be recovered by searching the underlying issue, pull request, or commit thread.The benchmark’s resistance is established at task-writing time rather than through post-release secrecy.
- Open release: Everything needed to run, grade, and audit tasks—including directories, environments, harness code, tests, and reference solutions—is released publicly for third-party reruns and inspection.The open release makes the benchmark fully reproducible and openly auditable.
3 The Benchmark
Tencent WorkBuddy Bench is a four-subset benchmark—Code, Web, Office, and Security—with a shared task format but domain-specific realistic agentic work and verification. This section introduces the subsets’ scopes and the Code subset’s repository-scale construction, validation, composition, and scoring.
- Benchmark scope: The benchmark spans Code, Web, Office, and Security, using a common task format and scoring philosophy while targeting distinct realistic agentic tasks.The Code subset is introduced here, with Web, Office, and Security covered in subsequent sections.
- Code subset: Code contains 80 repository-level tasks requiring agents to interpret role-played engineering requests, explore full repositories, modify code, and keep hidden tests green.Tasks begin from baseline commits and are not pre-diagnosed bug reports or single-file exercises.
- Code subset: Each Code task passes oracle-gated admission requiring baseline reward ≤0.3 and oracle reward = 1.0 after applying its diagnostic gold patch.This two-run validation filters out tasks whose baseline already satisfies too much of the contract or whose gold patch fails to solve it.
- Other subsets: Web requires a runnable front-end artifact at a declared output path, Office requires consistent deliverables across mixed-format workspace files, and Security covers discovery, exploitation, malware analysis, operations, and agent-security assessment.Web tasks span generation, modification, analysis, and quality assurance; Office tasks also require state updates and reviewable evidence; Security tasks use sandboxed environments where appropriate.
2 SECURITY TASK TYPES EVALUATION
This section presents reproducible security tasks constructed from real vulnerabilities and authored scenarios, covering red-team discovery, safe exploitation, and agent assessment.
- Task Construction: Security tasks are constructed from real vulnerabilities.
- Red Team: Red-team tasks cover discovery, safe exploitation, and agent assessment.
- Reproducible Security Tasks: The security tasks are designed for reproducible evaluation.
- Concrete Origins: Task origins include historical CVEs and authored scenarios.
BLUE TEAM
The Security subset spans diverse red- and blue-team tasks, including blackbox binary exploitation, web exploitation, agent security, and security operations. Tasks use deterministic per-task verification in isolated containers, with anti-cheat safeguards and results averaged across three runs under two agent harnesses.
- Blue-team workflows: Blue-team tasks emphasize detection and defense hardening through IOC, YARA, and SOC-report findings.The listed workflows include SOC triage and detection rules, alongside reversing, IOC, and YARA outputs.
- Security task coverage: Security covers six red- and blue-team task types, totaling 38 red-team and 22 blue-team tasks.The types are whitebox source audit, blackbox binary exploitation, web exploitation, agent security, malware analysis, and security operations.
- Verification and scoring: Security tasks are scored by deterministic programs in isolated Docker containers using task-specific evidence and correctness checks.The scoring protocol includes PoC verification and matching-report scoring, with averages over three independent runs.
- Anti-cheat infrastructure: A five-layer anti-cheat system blocks hardcoded answers, filename dependence, tampering, plaintext-only detection, and blind enumeration.Its checks include banned-literal scanning, renamed-input tests, overlay/tamper tests, encoding-dependence tests, and low-weight decoy fields.
- Evaluation protocol: Security is evaluated under both the CodeBuddy Code and Claude Code harnesses in think mode, with scores averaged over three runs.The results are reported in Section 5.
4 Evaluation Harness and Scoring
Tencent WorkBuddy Bench uses a uniform, sandboxed task-directory harness with fully public evaluation artifacts, while reporting results under both CodeBuddy Code and Claude Code. Scoring is track-specific rather than suite-wide: Code, Web, Office, and Security use different verification instruments, with model-judged components explicitly treated as a risk.
- Evaluation infrastructure: Every task runs in an isolated container through one shared harness, with grading assets withheld until agent execution ends and all evaluation artifacts publicly released.The open release includes task directories, environment images, evaluation code, grading tests, and reference or gold solutions.
- Evaluation infrastructure: All four tracks are reported under CodeBuddy Code and Claude Code, because model rankings can shift between harnesses.Claude Code is supported where a model’s protocol permits that route.
- Track-specific scoring: Track rewards use distinct verification methods: hidden-test pass rates for Code, rubric penalties and fatal-zero rules for Web, Rule–Judge blends for Office, and deterministic three-term scoring for Security.Security scores are averaged over three runs; Web uses scored rubric items, while Office combines deterministic rule checks with semantic LLM-judge checks.
- Track-specific scoring: Security uses hidden-test verification without an LLM judge, covering exploitation, malware analysis, YARA rules, and SOC reports through programmatic task scorers.Each Security task ships a scoring.py that writes a numeric reward inside an isolated container, and each score averages three independent runs.
- Scoring limitations: Model-judged components appear in Web and Office, while Code’s LLM-judge score is reference-only; the stated risk is judge bias toward particular styles or model families.Web’s judged items include LLM/VLM and agent-judge rubrics, and Office uses an LLM Judge for semantic-quality checks.
5 Results
Results show that performance varies substantially by model, harness, domain, and task type: no model leads every board, while Code, Web, Office, and Security expose distinct strengths and failure modes. Efficiency and reliability also diverge, with output budgets, refusals, and harness integration affecting observed results.
- Leaderboard: Claude Opus 4.8 leads five of eight boards, while GLM-5.2 leads Security under both harnesses.Claude Opus 4.8 leads Code under both harnesses, Web under both harnesses, and Office under cbc; GLM-5.2 leads Security under both harnesses.
- Harness sensitivity: Code rankings shift across harnesses, while Web, Office, and Security show different magnitudes and directions of model-score changes.Web’s signed mean shift is −1.14, Office’s median absolute shift is 0.86, and Security shows the largest reordering.
- Code: 0.47 is the mean reward for both bug_fix and api_contract, versus 0.94 for feature_pipeline and 0.88 for testing.Bug-fix tasks require diagnosing context-dependent upstream regressions, while api_contract tasks often fail when required fields, names, parameters, or output formats do not match the contract.
- Web: Web results are strongest for visual design and analytical reporting, but weakest for interaction, statefulness, and data-visualization semantics.Models often render plausible pages while failing to maintain consistency among state source, display, persistence, final payload, or source-to-output evidence.
- Office: 84.6/80.3/73.1 under cbc and 83.9/78.9/72.0 under cc are the cross-model Office means for easy, medium, and hard tasks.Office submissions commonly contain inconsistent deliverables, unclear source or state links, unparsable structured files, or artifacts submitted without validation.
- Efficiency and reliability: 6.9k tokens per run makes GPT-5.5 the lowest-output-budget cbc model while it still posts top-tier scores; Security is the heaviest track at 30–89 turns per run.Under cc, DeepSeek-V4-Flash emits 28.6k tokens versus GPT-5.5’s 8.7k while scoring 61.89 versus 76.63; Claude Opus 4.8 recorded 13 Security refusals under cc.
6 Related Work
Tencent WorkBuddy Bench is positioned against existing benchmarks across code, web, office, and security, with comparisons focused on task design rather than measured agent performance. Its differentiation is breadth and framing: colloquial, role-diverse, contamination-resistant tasks and unified coverage of complementary work settings.
- Comparative framing: The related-work comparison is qualitative and design-time, not a head-to-head measured evaluation of agents across benchmark suites.The benchmark positions itself against existing agent benchmarks while reserving detailed methodology, subset design, and harness descriptions for later sections.
- Code: Code tasks use short, colloquial teammate-style requests rather than detailed GitHub issues or test-driven specifications, while spanning multiple requester roles.The Code subset is compared with SWE-bench, SWE-bench Verified, and Commit0 on instruction style and role diversity.
- Web: Web benchmarks differ in emphasis, covering static reproduction, interactive generation, browser-agent operation, or runnable front-end artifacts.Design2Code and Interaction2Code use reference designs; FrontendBench broadens front-end generation; WebArena and VisualWebArena evaluate agents operating existing browser environments.
- Office: Office benchmarks target heterogeneous files, multi-application interaction, long-horizon workflows, or complex spreadsheets, whereas WorkBuddyBench-Office evaluates complete local-workspace handoffs.Its agents must preserve source information, file and state consistency, and execution constraints under CodeBuddy Code or Claude Code.
- Security and contribution: Security benchmarks span CTF solving, interactive execution, real-world CVE exploitation, and model cybersecurity risks, while the suite emphasizes breadth and framing rather than a new task type.The suite’s Code contribution includes repository-scale tasks, five requester roles, and an 18-category taxonomy using colloquial asks designed to resist searchable-prompt and leaked-answer contamination by construction.
7 Limitations and Conclusion
Tencent WorkBuddy Bench has material limitations involving comparability, language coverage, contamination exposure, judging, serving conditions, and Office scope. It is nevertheless fully open for audit, with near-term work on calibration and broader public leaderboard coverage.
- Limitations: Claude Opus 4.8’s Code score under Claude Code uses an added do-not-ask, complete-in-one-pass instruction, so it is reported separately from the harness-shift aggregate.All seven models are scored under both harnesses across all four tracks, but this configuration differs from the other runs.
- Limitations: The Code subset is dominated by Python, so its findings should not be assumed to generalize across programming languages or ecosystems.Cross-language coverage is limited to a small number of tasks porting behavior from JavaScript, TypeScript, or Rust projects into Python.
- Limitations: Because task directories, images, evaluation code, tests, and reference solutions are public, post-release contamination remains possible despite dataset versioning.Versioning can retire or replace tasks showing contamination symptoms, mitigating but not eliminating the exposure.
- Limitations: Judge-based Web, Office, and diagnostic Code components may introduce model-judge bias, while scores also depend on serving endpoints and harness builds.Office preserves rule-check outcomes independently, but metrics may shift with endpoint configurations, request handling, or evolving harness versions.
- Limitations: Office evaluates text-first local, mixed-format workflows rather than OCR, visual perception, pixel-level layout judgment, or native desktop-GUI interaction.Its results therefore apply to file handling, state updates, and evidence-based workflow completion.
- Conclusion: The suite is fully open for offline third-party testing and audit, with near-term plans for rubric calibration, standardizing the modified run, and expanding the public leaderboard.The release uses one task-directory format, one admission protocol, and one execution harness across four subsets.