PaperlayerView exact revision on arXiv

Source-linked AI summary

HaPRL: Human-Anchored Process Reinforcement Learning for Visual Search Agent

Zhangquan Chen, Yaoxin Niu, Xiang An, Mingze Sun, Zhumei Wang, Chih-Ting Liao, Hongkun Cao, Ruqi Huang

arXiv:2609.37190v1cs.CV

TL;DR

Outcome-only reinforcement learning leaves visual-search processes unsupervised, so correct answers can receive full credit despite faulty searches. HaPRL trains against human search traces with answer-gated, task-adaptive process rewards; it outperforms outcome-based RL and supports stronger subsequent outcome-based scaling.

  • Problem

    Outcome-based rewards score only final answers, so correct results reached through searches that miss target evidence can receive full credit.

    Source: §1 · p. 2

  • Method

    HaPRL collects human search traces and uses a frozen judge to score trajectories with task-adaptive weights, gating process scores by answer correctness.

    Source: §1 · pp. 2–3

  • Results

    HaPRL consistently outperforms outcome-based RL across four backbones and six benchmarks; early process supervision yields 6.7× (6.13 vs. 0.91) more subsequent outcome-based improvement.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results highlight the importance of aligning agent behavior with human process annotation signals and offer practical insights for work beyond visual search.

    Source: §1 · p. 3

    Extending human-process alignment beyond visual search, including to 3D rigging and GUI agents, is framed as future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Multi-turn visual search agents answer questions about high-resolution images by iteratively deciding where to look. Reinforcement learning for these agents rewards only the final answer, leaving the search process unsupervised. Consequently, faulty routes in which the reasoning process is erroneous yet the final result is correct arise frequently, which in turn leads to ineffective training, i.e., scaling along the wrong paths. In this paper, we introduce HaPRL, the first framework to reinforce the search process with human search behavior. We first build an annotation platform and collect 1K+ human-annotated data with fine-grained behavioral signals. During training, a carefully designed judge scores each rollout with task-adaptive weights, anchored on the distilled trace of how a human annotator actually searched the same image. Extensive experiments show that HaPRL consistently outperforms outcome-based RL, and early-stage process supervision yields 6.7x more improvement in subsequent outcome-based scaling. Our results also demonstrate the importance of aligning model behavior with human process annotation signals, which offer new insight into the training of foundation models.

1 INTRODUCTION

Outcome-only rewards can credit correct answers reached through faulty searches, leaving visual-search processes unsupervised. HaPRL aligns trajectories with human search traces, and experiments report stronger performance than outcome-based RL.

1 INTRODUCTION

  • Outcome-only rewards give full credit to correct answers even when agents miss target evidence, reinforcing faulty search routes.On multiple-choice and short-answer tasks, an agent can inspect irrelevant regions and guess correctly.Source: §1 · p. 2
  • Visual search needs trajectory-level evaluation because the usefulness of each crop depends on the question, prior views, and progress toward the target.Human behavior records regions attended, view refinement, and where evidence was found.Source: §1 · p. 2
  • The authors collected 1,104 human-annotated examples and introduced HaPRL, which uses answer-gated, task-adaptive rewards anchored on human traces.The traces summarize attention and evidence acquisition round by round.Source: §1 · pp. 2–3
  • Across four backbones and six benchmarks, HaPRL consistently outperforms outcome-based RL; early process supervision yields 6.7× (6.13 vs. 0.91) more improvement in subsequent outcome-based scaling.The comparison is with the same subsequent outcome-based stage without early process supervision.Source: §1 · p. 3

2 RELATED WORK

Prior work trains visual-search interactions, provides process supervision for reasoning, and uses rubric-based reinforcement learning. HaPRL evaluates complete search trajectories against matched human traces.

Visual Search Agents

  • Visual-search methods use varied tools and training recipes, but primarily rely on terminal correctness or hand-designed auxiliary signals.HaPRL instead uses human search traces as instance-specific references for evaluating evidence-acquisition trajectories.Source: §2 · p. 3

Process Supervision for Reasoning

  • Process-supervision methods assess individually evaluable reasoning steps, whereas visual search requires evaluating the complete trajectory because each action changes later evidence.HaPRL uses this trajectory-level signal for reinforcement learning.Source: §2 · p. 3

Rubric-Based Reinforcement Learning

  • Rubric-based reinforcement learning commonly evaluates final responses; HaPRL applies task-adaptive criteria to search, anchors them on a matched human trace, and gates process scores by answer correctness.Source: §2 · p. 3

3 METHOD

HaPRL treats human search traces as reward anchors rather than demonstrations, using them to assess trajectories and train visual-search policies. Its pipeline collects and distills human interaction data, scores search processes, and applies answer-gated group-relative optimization.

3 METHOD

  • HaPRL uses matched human traces to score sampled trajectories, while answer correctness gates process scores and traces remain outside the policy context.The resulting reward provides graded preferences among correct trajectories.Source: §3 · p. 4, Figure 2 · p. 4

3.1 HUMAN PROCESS DATA COLLECTION

  • The platform logs timestamped human pointer movements, hovers, and zooms, then distills them into round-by-round traces of inspected regions and discovered evidence.Traces preserve meaningful dwell, redirection, and coarse-to-fine refinement, while suppressing redundant pointer samples.Source: §3.1 · pp. 4–5

3.2 HUMAN-ANCHORED PROCESS JUDGE

  • A frozen vision-language judge scores trajectories on five dimensions, including target semantics, evidence acquisition, search progress, tool discipline, and communication discipline.Task-adaptive profiles weight criteria differently for OCR/text, count/relation, attribute, and general-search tasks.Source: §3.2 · p. 5

3.3 ANSWER-GATED PROCESS OPTIMIZATION

  • Answer correctness gates process rewards: incorrect trajectories receive zero, while correct ones are ranked by evidence-acquisition quality.Group-relative optimization preserves search-quality differences among all-correct samples; inference uses neither judges nor human traces.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments compare HaPRL with Outcome-RL across four backbones and six benchmarks, assessing answer accuracy and search quality separately. HaPRL improves benchmark accuracy and search quality, and scaling and ablation experiments examine process-data volume, early process supervision, and the human trace.

4.1 BENCHMARKING OUTCOME-RL-BASED VLMS

  • HaPRL wins all 24 backbone–benchmark cells over Outcome-RL and base models, including +30.3% on Qwen3-VL-4B (44.26 vs. 33.98) and +15.3% on LLaVA-OneVision-1.5-8B (47.80 vs. 41.46).The reported accuracy is the unweighted mean across six benchmarks.Source: §4.1 · p. 6, Table 1 · p. 7
  • Cold-start SFT costs 3.00 to 8.71 points, while HaPRL clears each base model by +4.19 to +7.40 points; Outcome-RL fails to repay the SFT loss on half the backbones.On Qwen3-VL-4B, HaPRL reaches 56.50 on HR-Bench 4K after the cold start gives up 6.62 points and Outcome-RL loses a further 1.75.Source: §4.1 · p. 6
  • HaPRL adds +71.8% on VisualProbe Hard on average; its correctness-gated rubric ranks correct rollouts where binary outcome reward supplies no gradient.The reported hard-search gains peak at +144.4% on Qwen3-VL-4B and +127.7% on LLaVA-OneVision-1.5-8B.Source: §4.1 · p. 6

4.2 SEARCH QUALITY UNDER DIFFERENT SETTINGS

  • On answer-correct rollouts, HaPRL adds +2.78 (84.77) and improves all six benchmarks; Outcome-RL adds +0.08 (82.08 vs. 81.99) and falls on two.On VisualProbe Hard, Outcome-RL raises accuracy by +28.6% (8.49 vs. 6.60) while its process score drops 8.4% (65.56 vs. 71.54).Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 SCALING BEHAVIOR

  • Accuracy rises from 36.62 at 200 annotated prompts to 44.26 at 1,104 (+20.9%); 18% of the prompts already beats Outcome-RL trained on all prompts (36.62 vs. 33.98).The reported accuracy rises monotonically at every annotation budget and does not flatten at the largest collected budget.Source: §4.3 · p. 7, Table 3 · p. 8
  • A 600-prompt outcome-only continuation adds +16.7% after HaPRL (42.76 vs. 36.63), versus +2.8% after Outcome-RL (33.54 vs. 32.63).The stage-2 training is held the same across the two arms; stage-1 reward choice differs.Source: §4.3 · p. 8

4.4 ABLATION STUDY

  • Mis-pairing human traces costs 35.3% (28.63 vs. 44.26), while removing the trace from the judge leaves only +2.43 over Outcome-RL.The rubric supplies judgment axes; the trace supplies where good search sits for the particular image.Source: §4.4 · pp. 8–9
  • Perturbing 10% of dwell and hover events costs 5.7% (41.74 vs. 44.26) but retains +7.76 over Outcome-RL; the trace’s useful region ordering survives event jitter.The paper characterizes the process reward as a channel for human search behavior whose value tracks how faithfully the trace is paired with its trajectory.Source: §4.4 · p. 9

5 CONCLUSION AND LIMITATION

HaPRL uses human search traces to supervise visual-search processes and outperforms matched outcome-based reinforcement learning across benchmarks. The authors identify extension to other domains as future work.

5 CONCLUSION AND LIMITATION

  • HaPRL distills 1,104 human search traces into round-by-round evidence descriptions, then uses a frozen judge with task-adaptive, human-trace-anchored weights and answer-correctness gating to score rollouts.Across all benchmarks, it consistently outperforms matched outcome-based RL in both accuracy and search quality.Source: §5 · p. 9
  • The authors identify extending human-process alignment to more domains, especially those where human annotation can play a central role, as future work.Examples include 3D rigging and GUI agents.Source: §5 · p. 9

AI USE STATEMENT

The authors report using generative AI to polish author-written text and assist with some code, with all AI-assisted work manually verified. They also state that the study follows the ICLR Code of Ethics.

AI USE STATEMENT

  • Generative AI tools were used to polish manuscript wording after an author-written draft and assist with some training and inference code.The authors state that they take responsibility for the final content.Source: AI USE STATEMENT · p. 10
  • The authors manually verified all AI-assisted text and code.Source: AI USE STATEMENT · p. 10
  • The authors state that the study follows the ICLR Code of Ethics and uses publicly available, fully anonymized datasets.They state that the datasets do not involve human subjects, privacy infringement, or harmful discrimination concerns.Source: ETHICS STATEMENT · p. 10

A ADDITIONAL METHOD DETAILS

The appendix details how human interactions become compact search traces and how HaPRL uses a structured, task-adaptive process score gated by answer correctness. The resulting rewards are standardized within groups for GRPO updates.

A.1 ALGORITHMS

  • Algorithm 1 logs annotator interactions, distills them into round-by-round traces, and suppresses pointer samples that do not change the inspected region.It retains dwell, redirection, and coarse-to-fine refinement; the policy observes only the image and question.Source: §A.1 · p. 13
  • Algorithm 2 samples trajectories, scores answer correctness, and evaluates process quality only for answer-correct rollouts before group-relative GRPO updates.The process reward multiplies the answer reward; updates use clipped GRPO and KL regularization toward a reference policy.Source: §A.1 · p. 13

A.2 HUMAN-TRACE REPRESENTATION

  • Human interaction streams are distilled offline into ordered textual traces that record inspected regions and revealed evidence while omitting motion that does not change the view.The matched trace anchors reward computation without requiring exact action or token alignment.Source: §A.2 · p. 13

A.3 STRUCTURED PROCESS RUBRIC

  • The process rubric scores five dimensions, and task-profile weights are normalized before the weighted total is recomputed outside the judge response.The dimensions are target semantics, evidence acquisition, search progress, tool discipline, and communication discipline.Source: §A.3 · p. 14, Table 6 · p. 14

A.4 REWARD COMPUTATION

  • HaPRL sets process reward to zero for an incorrect answer; otherwise, it evaluates the rollout against its matched trace and multiplies process and answer rewards.Correct rollouts therefore retain a continuous preference signal that outcome-only supervision discards.Source: §A.4 · p. 14

B.1 BENCHMARKS

The appendix specifies the benchmarks, visual-token budget, cold-start filtering and selection, reinforcement-learning comparison, and judge setup. The two RL conditions differ only in the optimization total.

B.1 BENCHMARKS

  • Evaluation covers VisualProbe Easy, Medium, and Hard, V∗Bench QA, and HR-Bench 4K and 8K, with one greedy rollout capped at six tool rounds.All models share the visual-token budget; V∗Bench QA uses open-ended grading, and search quality is measured only on answer-correct rollouts.Source: §B.1 · pp. 14–15

B.2 COLD START

  • Cold-start filtering reduces the 7,267-trajectory Mini-o3 corpus to 6,057, and initialization is selected by rollout quality on 20 held-out prompts rather than cold-start loss.The filters cap assistant turns at six and remove trajectories dominated by recovery phrases or repeated long sentences.Source: §B.2 · p. 15

B.3 REINFORCEMENT LEARNING

  • The two reinforcement-learning conditions differ only in the optimization total, while both call the answer and process judges and log all five rubric dimensions.Process quality is observable in the outcome-only condition but does not contribute to its gradient.Source: §B.3 · p. 15, Table 7 · p. 15

B.4 JUDGE

  • Both judges use a frozen Qwen3-VL-30B-A3B-Instruct model; the answer judge returns a binary semantic-correctness score from a text-only prompt.The process judge receives the image, full rollout, and matched human trace; judge fallback rates are monitored because degraded judging can flatten process rewards.Source: §B.4 · p. 15

B.5 PROCESS ANNOTATION

The annotation platform records human searches and distills them into reusable, question-specific reference traces for judging rollouts. Collecting the traces takes about 27.6 person-hours, and the study compares the human trace with alternative reward references.

B.5 PROCESS ANNOTATION

  • The platform logs timestamped interactions as annotators search full-resolution images, and annotators mark the evidence regions they settle on.The event stream includes canvas entry, hover movement, zoom start, and zoom motion, along with cursor position and the inspected region.Source: §B.5 · pp. 15–16
  • Each event stream is distilled into a short, round-by-round trace of inspected regions and findings for the judge to use as its human reference.The judge does not receive raw streams; traces are collected once per question and reused across runs without entering the policy context.Source: §B.5 · p. 16
  • One annotator produced the collection in about 27.6 person-hours, with the annotation cost paid once across runs and backbones.Each question takes about 1.5 minutes on average.Source: §B.5 · p. 16
  • The ablations shuffle question-to-trace mappings, perturb 10% of dwell and hover events, or remove the reference trace while retaining the rubric and task-profile weights.The shuffled-label variant preserves the label distribution but destroys question-to-trace correspondence.Source: §B.5 · p. 16

C ADDITIONAL EXPERIMENTS

Additional experiments compare human traces with cheaper reward references, test whether gains depend on the cold start, and repeat two-stage training across seeds. They find a 6.6× larger subsequent-stage gain for process-initialized training, alongside distinct contributions from initialization and process supervision.

C ADDITIONAL EXPERIMENTS

  • The experiments separate human-trace contributions from generic evidence specifications, a better cold start, and seed variance.The cheaper-reference comparison finds that every substitute improves on outcome-only reward, while their gains differ with how much of the search they describe.Source: C ADDITIONAL EXPERIMENTS · p. 16, §C.1 · p. 16

C.1 CHEAPER REWARD REFERENCES

  • The human trace adds 4.41 points over the best substitute, while the teacher path reaches 39.85 without human labor.With training held identical, the tool-use bonus recovers 1.74 points and the final evidence box 4.63 points over outcome-only reward.Source: §C.1 · p. 16, Table 8 · p. 17

C.2 COLD-START DEPENDENCE

  • Recipe B preserves the unfiltered 7,267-trajectory corpus, masks loss beyond the six-round cap, and stops after two epochs; its Qwen3-VL-4B cold start scores 39.84, within 0.23 of the 40.07 backbone score.This provides a second cold start that does not degrade the backbone, allowing process-supervision gains to be separated from recovery of cold-start losses.Source: §C.2 · pp. 16–17, Table 9 · p. 17
  • Recipe B with HaPRL reaches 49.02, 4.76 points above the best Recipe A result; the HaPRL advantage over outcome-only reward is 5.87 points from Recipe B and 10.28 from Recipe A.The authors attribute 4.41 points of the main-table advantage to recovery from weak initialization and the remaining 5.87 to process supervision.Source: §C.2 · p. 17, Table 9 · p. 17

C.3 TWO-STAGE SEED VARIANCE

  • The three-seed schedule comparison repeats each two-stage training path, extending the single-run comparison reported in Table 4.Table 4 reported stage-2 gains of 6.13 points after process-trained stage 1 and 0.91 after outcome-only stage 1.Source: §C.3 · p. 17
  • Across three seeds, the process-initialized path gains 5.83 ± 0.26 points in stage 2 versus 0.88 ± 0.06 for outcome-only training, a 6.6× ratio.Both schedules use the same 600 prompts and reward in stage 2; their 3.94-point gap grows to 8.89 points, with seed intervals reflecting optimization and rollout stochasticity rather than data partitioning.Source: §C.3 · p. 17

D QUALITATIVE CASES

Six qualitative cases contrast cold start, Outcome-RL, and HaPRL rollouts on the same questions. HaPRL answers all six correctly with fewer rounds and crops on average, while the examples illustrate rather than measure the frequency of these behaviors.

D QUALITATIVE CASES

  • The figures render each condition’s reasoning, crop, and resulting observation round by round, making the route to each answer visible.Each question is held fixed across the cold start, Outcome-RL, and HaPRL conditions from the same backbone.Source: D QUALITATIVE CASES · p. 17
  • The six cases are hand-picked to span task profiles and evaluation sources, so they illustrate measured mechanisms but do not establish their frequency.The examples cover text recognition, spatial relations, and attribute recognition.Source: D QUALITATIVE CASES · p. 17
  • Outcome-RL is correct twice but uses extra rounds on redundant or misplaced crops; its four failures include repeated crops, a truncated date, and reporting the wrong flag’s colors.The examples include overshooting a text target before reading it, repeating a crop after reading the answer, and exhausting the round budget without a valid response.Source: D QUALITATIVE CASES · p. 18
  • HaPRL answers all six cases correctly using 2.7 rounds and 1.7 crops on average, fewer than either control, and reaches each answer with the shortest evidence path.The cases span text recognition, spatial relation, and attribute recognition across all three evaluation sources.Source: D QUALITATIVE CASES · p. 18, Table 11 · p. 18
  • HaPRL’s examples show direct evidence targeting: one crop reads the displayed date, a joint sign crop reveals a relation, and a direct flag crop avoids the salient distractor.The shelf example first locates the queried word and then tightens the crop while keeping it visible.Source: D QUALITATIVE CASES · p. 18

Related papers

Loading 2609.37190v1…