PaperlayerView exact revision on arXiv

Source-linked AI summary

onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction

Lei Yang, Mengyin Liu, Jia Wang, Hangyu Guo, Liang Zhao, Zheng Ge, Kang An, Binxing Jiao, Qi Han, Daxin Jiang, Siqi Shen, Xiangyu Zhang

arXiv:2609.24983v1cs.CLcs.HCcs.LG

TL;DR

Alignment annotation must balance cost, on-policy fidelity, and supervision granularity, and interactive correction of agent reasoning and tool calls across environments remains lacking. onPanda lets annotators correct the first inappropriate token with a model candidate or free-form text, then continues generation from the corrected prefix. onPanda’s median annotation time was 330 s per prompt, 51.5% less than POTATO’s 681 s and on par with Argilla’s 336 s.

  • Problem

    Alignment annotation must balance cost, on-policy fidelity, and supervision granularity, and interactive correction of agent reasoning and tool calls across environments remains lacking.

    Source: §1 · pp. 1–2

  • Method

    onPanda lets annotators correct the first inappropriate token with a model candidate or free-form text, then continues generation from the corrected prefix.

    Source: §1 · p. 2

  • Results

    onPanda’s median annotation time was 330 s per prompt, 51.5% less than POTATO’s 681 s and on par with Argilla’s 336 s.

    Source: §3.1 · p. 5

  • Takeaways & Limitations

    Token-level corrections retain position-specific supervision, and the authors release Panda-CVL with a benchmark for token-level correction research.

    Source: §5 · p. 7

    The downstream training benefit of token-level correction signals remains unverified by training experiments.

    Source: Limitations · p. 10

Abstract

from arXiv · show

We present onPanda, an interactive tool for efficiently annotating LLM alignment data and agent trajectories. onPanda adopts token-level correction as its core interaction: while reading a model response, the annotator locates the first inappropriate token and either picks a substitute from the model's candidate tokens or types the correct text via free-form editing. The system then truncates everything after that position and continues generation from the corrected prefix, repeating this locate-correct-continue loop until a satisfactory response is obtained. This mechanism lets annotators precisely steer model outputs at low cost: a small controlled study suggests that onPanda reduces median annotation time by 52% over manual post-editing. Since the vast majority of tokens in the final response are generated by the model itself, the resulting data largely preserves the model's sampling distribution and is well suited for constructing on-policy SFT and preference data. Furthermore, the token-level corrections recorded during annotation provide fine-grained supervision with precise positions and naturally paired positive--negative samples. onPanda also connects to external tools and harnesses, enabling interactive trajectory annotation in realistic environments. In addition, we release Panda-CVL, a dataset annotated with onPanda, together with a benchmark for token-level correction.

1 Introduction

onPanda addresses the cost, on-policy fidelity, and supervision-granularity trade-offs in alignment annotation with token-level correction. It supports LLM responses and agent trajectories, and the paper reports experimental validation, a Panda-CVL release, and an accompanying benchmark.

1 Introduction

  • Existing annotation pipelines trade off cost, on-policy fidelity, and supervision granularity, while interactive correction of reasoning and tool calls across diverse environments remains lacking.Manual writing and post-editing are costly and yield off-policy data; preference data provides coarse response-level supervision and limited correction when good responses are outside the model’s candidates.Source: §1 · pp. 1–2
  • onPanda uses a locate–correct–continue loop: annotators replace the first inappropriate token with a model candidate or free-form text, then generation resumes from the corrected prefix.The system truncates subsequent content after each correction, so annotators need not reread the entire response after a fix.Source: §1 · p. 2
  • Most response tokens remain rollout-model generated, while logged correction positions and positive–negative samples provide fine-grained supervision for preference or process-reward training.Free-form edits can inject content beyond the model’s candidates, trading a limited distributional deviation at those positions for correctness.Source: §1 · p. 2
  • For agent settings, onPanda connects through MCP to tools and harnesses including Claude Code, Codex, and OpenClaw for interactive trajectory annotation in realistic environments.The paper also describes support for multimodal and agent-trajectory annotation.Source: §1 · p. 2
  • The authors release Panda-CVL, a Chinese vision–language dataset annotated with onPanda, alongside a benchmark for token-level correction.The release is intended to facilitate community research on this data type.Source: §1 · p. 2

2 Design and Implementation

onPanda is a componentized front-end library whose token-level correction workflow preserves session history and supports structured agent and multimodal annotation.

2.1 System Overview

  • The componentized front end works as a standalone web app or can be embedded in a data platform; it requires an inference API that continues assistant prefixes and returns top-k token candidates with probabilities.Session messages, annotations, operation logs, and probability caches are serialized in a single .panda.json file.Source: §2.1 · p. 3

2.2 Token-Level Correction Engine

  • Corrections operate on readable grapheme-aligned chunks while preserving token-precise records; each fix truncates later text and continues generation from the corrected prefix.Free-form edits can trigger a probability refresh for the response, and pasting text or switching models can reveal the current model’s token probabilities.Source: §2.2 · p. 3

2.3 Annotation Tree and Data Protocol

  • Each correction forks a dialog node in an annotation tree, retaining intermediate versions and operation logs for traceable provenance.The companion parser exports is_good=Y nodes as SFT samples and can pair positive and negative nodes into preference data or correction-position token triples.Source: §2.3 · p. 3

2.4 Agentic and Multimodal Annotation

  • Response templates render structured reasoning, content, and tool calls as correctable native token streams, then parse the stream back into messages for storage and execution.MCP-connected tools can await approval; corrected calls may execute, rejected trajectories can be kept as negatives, and returned tool results re-enter context.Source: §2.4 · pp. 3–4

3 Evaluation

The evaluation compares complete annotation workflows, reports time, quality, fidelity, workload, and production-use measures, and describes onPanda’s model-inspection capabilities.

3.1 Annotation Efficiency and Data Quality

  • onPanda’s median annotation time was 330 s per prompt, 51.5% less than POTATO’s 681 s and on par with Argilla’s 336 s, while its pairwise win rate was 66.7%.Human evaluators preferred onPanda over POTATO in 54.8% of pairs. Because each platform differs in both interface and annotation paradigm, it is difficult to determine whether the gains arise from the interface, the paradigm, or their combination.Source: §3.1 · pp. 4–5
  • onPanda reached PPL 1.181 (∆PPL +0.86%) against a baseline of 1.171, while POTATO reached 1.596 (+36.31%); onPanda and Argilla were within 1% of baseline.In the user study, onPanda’s workload score was 3.1 versus Argilla’s 5.4 and POTATO’s 6.8; deployment data included about 388K token-level corrections, with 97.0% of qualified-response tokens model-generated.Source: §3.1 · p. 5, §3.2 · p. 5

3.3 Model Inspection

  • Token-probability visualization and entropy statistics help users inspect candidate distributions and diagnose whether faulty rollouts originate from the model or the pipeline.Users can combine this inspection with token-level correction to steer generation along alternative paths and audit data quality.Source: §3.3 · p. 5

4 Dataset and Benchmark

Panda-CVL is a predominantly Chinese vision-language dataset of annotation sessions, filtered to general-purpose tasks. Its benchmark evaluates whether models can judge response quality, locate the first error, and correct it.

4 Dataset and Benchmark

  • Panda-CVL contains 7,491 sessions (6,839 training and 652 test), focused on general-purpose visual tasks and images created in-house or openly licensed.The tasks exclude internal business knowledge and cover visual perception, image description, and visual reasoning.Source: §4 · p. 5
  • The benchmark asks models to accept a satisfactory response or locate and correct its first inappropriate token, using human correction positions and replacements as references.The task is decomposed into response-quality judgment, first-error localization, and correction by candidate selection or manual editing.Source: §4 · p. 6
  • The 652 test conversations yield 2,126 evaluation instances: 652 acceptable final responses and 1,474 intermediate responses labeled not good.Correction triples are compared using the Qwen3.6 tokenizer to enable tokenizer-agnostic evaluation across models.Source: §4 · p. 6
  • F1 is the harmonic mean of GoodAcc and Corr.-NG, combining correct acceptance of good responses with end-to-end correction performance.The benchmark assigns zero F1 to always predicting is_good and always attempting a correction because each strategy scores zero on one component.Source: §4 · p. 6
  • Multi-reference analysis finds human location agreement far above chance, with measurable variation in correction positions and replacement tokens.Source: §4 · p. 7

5 Conclusion

onPanda uses a locate–correct–continue loop to steer model generation with a few clicks and edits. Experiments and a user study report reduced annotation time and workload while preserving data quality.

5 Conclusion

  • onPanda centers annotation on token-level correction, using a locate–correct–continue loop to steer model generation with a few clicks and edits.Source: §5 · p. 7
  • The tool produces alignment data and agent trajectories close to the rollout model’s distribution while retaining each correction as fine-grained supervision.Source: §5 · p. 7
  • The authors report reduced annotation time and workload while preserving data quality, and release onPanda, Panda-CVL, and its benchmark.Source: §5 · p. 7

Ethics Statement

The study reports voluntary, informed participation by salaried annotators, aggregate-only survey reporting, and safeguards for released dataset images.

Ethics Statement

  • Efficiency-study and user-study participants were full-time institutional data annotators; participation was voluntary and informed, and surveys are reported only in aggregate.Source: Ethics Statement · p. 7
  • Annotation tasks used image-description content without offensive or high-risk material.Source: Ethics Statement · p. 7
  • Released images were created in-house or openly licensed, manually reviewed to exclude personal information and inappropriate content, and intended for research use only.The authors note that data properties depend on users’ annotation guidelines and encourage responsible curation.Source: Ethics Statement · p. 7

A Additional Results and Statistics

On Panda-CVL, token-level correction remains challenging across evaluated models: the best overall F1 is 17.09%, while different models lead on individual metrics.

A Additional Results and Statistics

B Annotation Agreement

On Panda-MultiRef-21, four independent annotations per image-description prompt show substantial but incomplete agreement about correction locations and replacements.

B Annotation Agreement

  • 30.95% exact-match location agreement exceeded the 0.20% uniform random-location baseline.The authors interpret the gap as evidence of shared localization judgments supporting reference reliability.Source: B Annotation Agreement · p. 9
  • 44.44% location agreement within four tokens compared with approximately 1.84% for the uniform random-location baseline.Source: B Annotation Agreement · p. 9
  • The study covered 21 image-description prompts and 84 annotations, with replacement-token agreement of 69.44% conditional on identical correction positions.Agreement remained incomplete, reflecting diversity in the annotation distribution.Source: B Annotation Agreement · p. 9

C Related Work

Related work spans configurable annotation platforms, interactive agent editing, fine-grained supervision, and earlier correct-and-continue methods; onPanda combines these directions for broader annotation settings.

C Related Work

  • POTATO and Argilla (Argilla) support configurable post-editing or preference annotation, while POTATO 2.0 also supports agent-trajectory annotation.Source: C Related Work · p. 9
  • Reptile edits terminal-based SWE outputs for execution, whereas onPanda selects probability-scored tokens and supports structured reasoning and tool calls.The authors compare workflows rather than benchmark the systems head-to-head because their tasks and action representations differ.Source: C Related Work · pp. 9–10
  • Unlike PRM800K (Lightman et al., 2024) step labels and TLDR (Fu et al., 2025) synthetic token labels, onPanda records position-precise, naturally paired corrections during response construction.Source: C Related Work · p. 10
  • Correct-prefix-and-regenerate-suffix interaction predates LLMs in Predictive Translation Memory and INMT; onPanda extends it to multimodal and structured-agent annotation.Annotation-tree branches export as SFT, preference, and token-level correction data.Source: C Related Work · p. 10

Limitations

onPanda’s operation depends on model-interface capabilities and sparse correction needs, while the paper’s empirical validation has limited scope and does not verify downstream training benefits.

Limitations

  • On-policy annotation requires prefix continuation and top-k candidate logprobs; some APIs lacking both capabilities cannot support it.Reasoning and tool-call continuation for a new model also requires a one-time response-template adaptation.Source: Limitations · p. 10
  • Token-level correction’s efficiency and on-policy advantages depend on locally sound responses needing only sparse edits, and on using data with the rollout model current at annotation time.Dense corrections diminish both advantages; changing the trained model’s weights also erodes the data’s on-policy benefit.Source: Limitations · p. 10
  • Evidence is bounded by a modest, single-rollout-model user study with in-house participants, no controlled agent study, and no training experiments verifying downstream benefits.Quality evaluation relies primarily on LLM-as-a-judge, which may retain judge-specific preferences despite position-swapped comparisons.Source: Limitations · p. 10

D Future Work

Future work develops training methods around token-level corrections and explores annotation-training feedback loops and retrospective supervision from later textual feedback.

D Future Work

  • The authors are developing a token-level correcting model and token-level correction optimization, alongside post-training methods using correction data.The released benchmark evaluates correction capability and provides a testbed for the correcting model.Source: D Future Work · pp. 10–11
  • A closed annotation-training loop could retrain and update the served rollout model after each batch, making later corrections sparser while keeping data on-policy with current weights.A correcting model could also pre-locate suspicious tokens and propose fixes for annotator review.Source: D Future Work · p. 11
  • Annotating earlier turns using later tool errors or user complaints could convert retrospective feedback into token-level corrections without scalar rewards.The paper frames this as a prospective data source for online learning from textual feedback.Source: D Future Work · p. 11

Related papers

Loading 2609.24983v1…