PaperlayerView exact revision on arXiv

Source-linked AI summary

SSL: Sweet Spot Learning for Differentiated Guidance in Agentic Optimization

Jinyang Wu, Changpeng Yang, Yuhao Shen, Fangzhi Xu, Bolin Ni, Chonghua Liao, Yuchen Liu, Hongzhen Wang, Shuai Nie, Shuai Zhang, Haoran Luo, Jiaming Xu

arXiv:2601.22491v1cs.CL

TL;DR

Binary rewards obscure quality differences among successful trajectories and provide limited directional guidance for agent optimization. SSL introduces progressively amplified, tiered rewards across diverse tasks, preserving solution ordering and improving optimization; experiments show consistent gains and up to 2.5× sample-efficiency improvement.

  • Problem

    Binary rewards group successful and failed trajectories together within each outcome class, obscuring quality differences and creating optimization ambiguity, learning inefficiency, and policy fragility.

  • Method

    SSL assigns progressively amplified, tiered rewards based on trajectory proximity or incremental progress, using distance-tiered modeling for perception and progress-tiered rewards for reasoning.

  • Results

    SSL consistently improves performance across GUI perception, short- and long-term planning, and complex reasoning, matching or surpassing GRPO with 40% of its samples and achieving up to 2.5× data efficiency.

  • Takeaways & Limitations

    SSL provides a unified reward principle for differentiated agent optimization, with theoretical guarantees on solution ordering and gradient optimization and empirical transferability across diverse tasks.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards has emerged as a powerful paradigm for training intelligent agents. However, existing methods typically employ binary rewards that fail to capture quality differences among trajectories achieving identical outcomes, thereby overlooking potential diversity within the solution space. Inspired by the ``sweet spot'' concept in tennis-the racket's core region that produces optimal hitting effects, we introduce \textbf{S}weet \textbf{S}pot \textbf{L}earning (\textbf{SSL}), a novel framework that provides differentiated guidance for agent optimization. SSL follows a simple yet effective principle: progressively amplified, tiered rewards guide policies toward the sweet-spot region of the solution space. This principle naturally adapts across diverse tasks: visual perception tasks leverage distance-tiered modeling to reward proximity, while complex reasoning tasks reward incremental progress toward promising solutions. We theoretically demonstrate that SSL preserves optimal solution ordering and enhances the gradient signal-to-noise ratio, thereby fostering more directed optimization. Extensive experiments across GUI perception, short/long-term planning, and complex reasoning tasks show consistent improvements over strong baselines on 12 benchmarks, achieving up to 2.5X sample efficiency gains and effective cross-task transferability. Our work establishes SSL as a general principle for training capable and robust agents.

1. Introduction

RLVR commonly uses binary rewards that treat successful trajectories equally despite quality differences. SSL addresses this gap with tiered, proximity-aligned rewards and reports consistent gains across diverse agent tasks.

  • Motivation: Binary rewards obscure quality differences among successful trajectories, creating optimization ambiguity, learning inefficiency, and policy fragility.The paper contrasts efficient three-action GUI solutions with eight-action detours that receive identical rewards.
  • Approach: SSL guides policies toward high-quality solutions using progressively amplified, tiered rewards based on trajectory proximity to task completion.Distance-tiered rewards model proximity in GUI perception and navigation, while progress-tiered rewards capture incremental reasoning progress.
  • Theory: SSL preserves relative solution-quality ordering and improves gradient signal-to-noise ratios by providing more informative policy-update feedback.The paper links this granular feedback to accelerated convergence and improved sample efficiency.
  • Results: Up to 2.5× data efficiency is reported, with SSL matching or surpassing GRPO using only 40% of its samples.The result is presented alongside consistent performance improvements and transferability across diverse tasks.
  • Results: Experiments span 12 benchmarks covering GUI perception, short- and long-term planning, and complex reasoning, showing consistent gains over strong baselines.The benchmarks are used to evaluate efficiency, performance, and transferability.

2. Related Work

Prior RLVR methods commonly rely on binary terminal rewards, while reward-shaping approaches provide more detailed feedback but often require task-specific constructions. SSL instead organizes solution quality into ordered zones for generalizable learning.

  • RLVR for Agent Optimization: RLVR commonly applies binary terminal rewards, treating successful trajectories equally despite differences in solution quality.The paper gives three-step and eight-step GUI paths as an example of this limitation.
  • Reward Shaping in RLVR: Existing reward-shaping methods include handcrafted potentials, navigation distance rewards, reasoning milestones, and hindsight relabeling for sparse rewards.These approaches augment sparse terminal signals with more detailed feedback.
  • Sweet Spot Learning: SSL assigns tiered rewards through quality-ordered zones based on proximity to optimality while preserving optimal solution ordering.This formulation is presented as enabling generalizable and efficient learning.

3. Methodology

SSL converts fine-grained step evaluations into trajectory-level proximity scores, discretizes them into ordered sweet-spot zones, and uses the resulting tiers as structured rewards. Task-specific proximity designs support GUI grounding and other agent settings while integrating with standard RLVR optimization.

  • Core Principle: SSL assigns progressively higher rewards to trajectories that approach task-optimal regions by partitioning solution space into hierarchical proximity zones.The framework is designed to provide directional guidance toward high-quality solutions.
  • Reward Construction: SSL defines a discretized sweet-spot value measuring trajectory closeness to high-quality completion and combines it with binary verification using guidance strength α.The verification term encodes correctness, while the discretized value guides refinement toward superior solution regions.
  • Zone Design: Sweet-spot zones are ordered intervals defined by boundaries 0 = b_K < ··· < b_1 < b_0 = 1, with each zone assigned a value s_k.The intervals specify proximity ranges and their corresponding tier values.
  • Proximity Scoring: SSL computes step proximity h(s_t, a_t) to assess how each action aligns with desirable task behavior.These local evaluations are aggregated into a unified trajectory-level proximity measure.
  • Trajectory Scoring: Trajectory-level aggregation captures global progress, and subsequent discretization suppresses small noisy fluctuations while producing more discriminative reward tiers.The resulting feedback is intended to provide clearer learning signals for policy optimization.
  • GUI Instantiation: In GUI grounding, SSL uses a Gaussian spatial field and normalized Mahalanobis distance to assign graded proximity to predicted points inside target bounding boxes.Points outside the target box receive zero proximity and are treated as incorrect.
  • GUI Planning: Refined grounding supervision transfers to GUI planning, where spatial components receive graded feedback while high-level action types retain binary correctness evaluation.The planner conditions its next operation on interaction history.

2 Complex Reasoning Agents: Blockwise Sweet-Spot

SSL extends sweet-spot rewards to complex reasoning by scoring localized partial correctness, while its theory preserves quality ordering and improves gradient information.

  • Construction: Complex reasoning tasks share a blockwise structure that enables localized partial-correctness evaluation across maze navigation, Sudoku, and ARC-AGI.The construction applies across pathfinding, constraint satisfaction, and symbolic induction despite their different semantics.
  • Construction: Maze solutions are partitioned into 3 × 3 blocks, whose matched-cell counts receive high, medium, or low sweet-spot values.The values are 1 for 7–9 matches, 2/3 for 4–6, and 1/3 for 0–3 matches.
  • Construction: The same blockwise scoring rewards local agreement in Sudoku and ARC-AGI before global constraints or transformation rules are fully satisfied.Partial digit agreement and local symbolic structure contribute to trajectory-level feedback.
  • Theoretical Analysis: For policies with equal success rates, SSL ranks them by expected sweet-spot score, recovering distinctions that binary rewards cannot make.The formal condition is JSSL(π1) > JSSL(π2) iff µS(π1) > µS(π2).
  • Theoretical Analysis: SSL increases gradient signal-to-noise when higher sweet-spot scores align with performance-improving gradient directions.Its centered advantage amplifies meaningful components while suppressing noise.
  • Theoretical Analysis: Unlike dense rewards that uniformly minimize distance, SSL concentrates learning signals within an informative sweet-spot region.The weighting parameter α controls the relative emphasis on task success and solution-space proximity.

4. Experiments

Experiments evaluate SSL across GUI perception, planning, and complex reasoning using twelve benchmarks and task-specific metrics. SSL consistently improves performance, spatial grounding, sample efficiency, and transferability across settings.

  • Experimental Setup: Experiments cover GUI perception, short- and long-term planning, and complex reasoning across twelve benchmarks.GUI planning uses Type, GR, and SR; other tasks primarily use accuracy.
  • Performance Enhancement: SSL-3B reaches 82.41% average accuracy in short-term planning and 57.11% in long-term planning, improving over RL-Binary-3B by 9.0% and 14.6%.The reported gains span both planning horizons.
  • Performance Enhancement: Spatial grounding improves by +11.0% GR/SR on OmniAct-Desktop and +23.9% SR on AndroidControl-Low for the 3B model.Because SR requires correct actions and precise parameters, these results are linked to reduced grounding bottlenecks.
  • Performance Enhancement: SSL-7B achieves 85.31% short-term and 56.92% long-term average accuracy, exceeding RL-Binary-7B by +4.1% and +3.6%.Benefits hold across model sizes, platforms, and planning horizons.
  • Performance Enhancement: On ScreenSpot-Pro, SSL-3B gains +5.7% on Office interfaces, with similar improvements for the 7B model.The results are reported as evidence of improved precise spatial alignment and multimodal representations.
  • Performance Enhancement: SSL improves complex reasoning benchmarks including Sudoku, Maze Navigation, and ARC-AGI, with a reported +100.0% gain on Sudoku for the 3B model.The comparison includes binary and cell-level continuous reward baselines.
  • Sample Efficiency Analysis: Using 40% of training data, SSL-3B matches or exceeds full-data RL-Binary-3B, achieving up to 2.5× data efficiency.SSL maintains its advantage from 20% through 100% of training data.
  • Transferability: Cross-task results and transfer experiments support SSL’s generalization from perception to planning and across domains.Maze and ARC results further indicate that the tiered reward principle generalizes beyond GUI tasks.

5. Conclusion

The paper presents Sweet Spot Learning as a unified framework for differentiated guidance through tiered, proximity-aligned rewards. Experiments report consistent gains across diverse tasks, supporting SSL as a general principle for agentic optimization.

  • Conclusion: SSL addresses optimization ambiguity, learning inefficiency, and policy fragility by discretizing solution spaces into hierarchical zones.The framework provides differentiated guidance through tiered, proximity-aligned rewards.

Impact Statement

SSL is framed as operating within RLVR using verifiable rewards, without novel data collection or architectural requirements beyond standard agent post-training. It may improve accessibility and reduce computational barriers through more sample-efficient training, while more capable autonomous agents may introduce misuse concerns.

  • Ethical Considerations: SSL uses verifiable reward signals within the existing RLVR paradigm and introduces no novel data collection or architectural requirements.The paper states that experiments follow responsible AI practices and that SSL raises no ethical concerns in its motivation, design, or implementation.
  • Societal Implications: More capable autonomous agents may raise misuse concerns despite SSL having no inherent malicious intent.The paper presents this as a potential concern associated with improved agent capabilities, not as an inherent property of SSL.
  • Societal Implications: Differentiated rewards may accelerate assistive, accessibility, and productivity applications by enabling more sample-efficient agent training.The stated potential applications include assistive technologies, accessibility tools, and productivity systems.
  • Societal Implications: Improved sample efficiency may reduce computational costs and lower barriers for resource-constrained researchers and practitioners.

A.2. Proof of Proposition 2.1: Quality Ordering

The theoretical analysis shows that SSL preserves quality ordering among equally successful policies and can improve gradient quality under an alignment condition. The section also describes task-specific proximity modeling, practical hyperparameters, and plug-and-play integration with GRPO and related policy-gradient methods.

  • Quality Ordering: SSL strictly preserves sweet-spot quality ordering between policies with identical success rates.For equal success rates, one policy has a higher SSL objective exactly when it has a higher mean sweet-spot score.
  • Projected SNR Improvement: When sweet-spot scores align with gradient directions, SSL improves gradient quality and supports faster convergence and better sample efficiency.The claim is conditioned on alignment between sweet-spot scores and gradient directions.
  • Practical Design: Experiments report α ∈[0.1, 0.5] as effective across diverse tasks, while K = 3 to K = 5 zones balance noise suppression and quality differentiation.α balances binary correctness against sweet-spot guidance, and zone discretization suppresses noise in continuous proximity scores.
  • Task-Specific Sweet-Spot Modeling: GUI grounding uses Gaussian-field proximity to the target box, while maze, Sudoku, and ARC-AGI use block-based matching schemes.Maze and Sudoku use 3×3 structures with thresholded match scores; ARC-AGI additionally normalizes grids and evaluates pixel equality and spatial alignment.
  • Integration with RLVR Algorithms: SSL integrates with GRPO by replacing binary rewards while leaving the remaining GRPO components unchanged.The same replacement strategy is stated to apply to PPO, REINFORCE, and other policy-gradient algorithms.

C. Training Data and Evaluation Details

The evaluation spans GUI perception, short- and long-term planning, spatial and combinatorial reasoning, and abstract reasoning benchmarks. Dataset descriptions cover interfaces, mazes, Sudoku, and ARC-AGI tasks with varied scales and difficulty levels.

  • Evaluation Protocol: Evaluation metrics for most datasets follow their official standards, including accuracy and success rate.
  • GUI Perception and Planning: The benchmark suite evaluates GUI action prediction and grounding across web, desktop, and mobile interfaces.It includes GUI-Act, OmniAct, AndroidControl, GUI-Odyssey, ScreenSpot, and ScreenSpot-Pro.
  • GUI Perception and Planning: AndroidControl separates short-horizon tasks requiring 1-3 actions from long-horizon tasks requiring 5-15 sequential actions.Both splits contain 7,708 samples.
  • Spatial and Combinatorial Reasoning: Maze Navigation contains 1,000 training and 1,000 test mazes with varying sizes, obstacle densities, and path tortuosity.Each instance requires a valid path from a designated start to a goal while avoiding walls.
  • Spatial and Combinatorial Reasoning: The Sudoku benchmark contains 1,000 training puzzles and 100 test puzzles across easy, medium, hard, and extreme difficulty levels.Solutions must satisfy row, column, and 3×3 subgrid constraints.
  • Complex Reasoning Benchmarks: ARC-AGI evaluation uses ARC-AGI-1 with 400 samples and ARC-AGI-2 with 120 samples to assess abstract reasoning and few-shot generalization.Tasks require inducing transformation rules from 2-4 input-output demonstrations and applying them to test inputs.

D. Statistical Significance Analysis

The statistical analysis compares SSL with the RL-Continuous baseline using paired t-tests across three independent runs for each benchmark. Reported improvements are statistically significant across all task categories, with particularly strong gains in complex reasoning.

  • Statistical Significance: p<0.05 across all task categories indicates statistically significant SSL improvements over the RL-Continuous baseline.The tests use paired t-tests across 3 independent runs for each benchmark.
  • Statistical Significance: The analysis reports particularly strong gains in complex reasoning tasks, where differentiated rewards provide clearer learning signals.
  • Statistical Significance: Table 8 reports average success rate (%) across benchmarks within each task category for 3B models.
  • Detailed Benchmark Results: Tables 9 and 10 provide complete results for long-term planning and GUI grounding tasks, respectively.Their captions define relative improvements over the RL-Binary baseline using SSL RL-Binary −1.

E.3. Why Tiered Rewards Over Continuous Rewards?

The paper argues that tiered rewards can outperform continuous shaping because discretization reduces gradient noise under finite-sample, stochastic optimization. SSL combines task-aware zones with theoretical and empirical evidence for more efficient, transferable training.

  • Gradient noise: Small continuous reward differences can produce opposing update directions for trajectories with negligible quality differences.For click offsets of 45px and 50px, distance ratios differ by only 0.02-0.05, yet batch baseline subtraction can amplify the difference.
  • Gradient noise: 2.1×-3.5× higher gradient variance is observed with continuous rewards than tiered rewards across tasks.Continuous scores scatter samples across [0,1], whereas tiered scores cluster trajectories into discrete groups with more stable advantage estimates.
  • Granularity trade-off: Finite batches make very fine reward partitions statistically unreliable because each zone receives too few samples.With N≈128 and K=10+ zones, each zone contains only 10-15 samples on average.
  • Granularity trade-off: K≥8 zones risk over-fitting spurious correlations, such as screen-region effects replacing semantic target learning in GUI tasks.The paper frames finer differentiation as a sample-efficiency–discrimination trade-off rather than an unconditional benefit.
  • Design rationale: SSL’s zone selection is task-aware and systematic, using Gaussian σ-levels for spatial tasks and quartiles for matching.The authors report that these designs require minimal manual tuning and are less arbitrary than continuous shaping.
  • Design rationale: SSL preserves solution ordering and improves gradient signal-to-noise ratios by aggregating similar-quality trajectories and filtering noisy alignment.This supports more directed optimization while retaining useful distinctions among informative trajectories.
  • Scope conditions: Continuous rewards may be preferable with unlimited data, noise-free environments, or well-calibrated learned reward models.The paper’s setting instead uses finite data, stochastic policy sampling, and verifier-based rewards.

H.1. Discussion on Reward Hacking and Misalignment

The discussion identifies reward hacking as a limitation of block-wise sweet-spot rewards under global constraints, especially in Sudoku. It describes correctness gating and discretization as mitigations, while distinguishing SSL from learned reward models in data, flexibility, and interpretability.

  • Reward hacking: Block-wise rewards can misalign local optimization with global constraint satisfaction, enabling reward hacking.An agent may maximize block matches while violating row and column constraints.
  • Reward hacking: Approximately 8% of Sudoku training trajectories achieve S(τ) > 0.7 while failing global verification C(τ) = 0.These failures include repeated digits across row or column boundaries and locally correct blocks that create global contradictions.
  • Mitigation: Binary correctness gating makes sweet-spot rewards amplify correct trajectories while assigning failed trajectories a weaker signal.The proposed reward is RSSL = C(τ) + α · ˆS(τ), with failed trajectories receiving RSSL = α · ˆS(τ) < 1.
  • Mitigation: Zone discretization reduces sensitivity to spurious local optima that continuous rewards might overfit.The paper also recommends moderate α, cross-block consistency checks, and monitoring high-score failures.
  • Empirical balance: SSL’s imperfect sweet-spot signals still correspond to a +100% improvement for the 3B Sudoku model over sparse binary rewards.The authors present this as evidence that differentiated signals can accelerate learning despite the observed failure mode.
  • Learned reward models: Direct PRM/ORM comparison is infeasible for GUI and spatial reasoning because suitable annotations, domain transfer, and open-source baselines are lacking.Math-oriented PRMs operate on text chain-of-thought, whereas GUI tasks require multimodal visual-spatial scoring.
  • Learned reward models: SSL derives quality from geometric or structural proximity rather than learned models, making it more sample-efficient and interpretable but potentially less flexible.Zone thresholds have clear semantic meaning, while proximity can be difficult to define for some tasks.
Loading 2601.22491v1…