PaperlayerView exact revision on arXiv

Source-linked AI summary

Steering LLMs via Scalable Interactive Oversight

Enyu Zhou, Zhiheng Xi, Long Ma, Zhihao Zhang, Shihan Dou, Zhikai Lei, Guoteng Wang, Rui Zheng, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang

arXiv:2602.04210v2cs.AIcs.LG

TL;DR

Complex, long-horizon AI tasks create a supervision gap because non-experts may be unable to fully specify intent or verify outputs. The paper proposes Scalable Interactive Oversight, which gathers structured feedback through a recursive decision tree and achieves up to 54% higher alignment in website-requirement evaluation, while online user feedback can train the interaction agent.

  • Problem

    Non-experts may lack the expertise, cognitive bandwidth, and ability to verify complex outputs needed to align strong models with their latent intent.

  • Method

    Scalable Interactive Oversight decomposes complex intent into a recursive tree and elicits low-burden user feedback that is accumulated into guidance for model execution.

  • Results

    54% improvement in alignment was achieved in website-development evaluation, and online supervision signals from non-expert humans were sufficient to improve system alignment through reinforcement learning.

  • Takeaways & Limitations

    Non-expert humans can effectively steer LLMs toward expert-level, aligned outputs, while expert evaluator signals can further enhance performance.

  • Takeaways & Limitations

    Accumulated preferences may amplify early misunderstandings or biases, and the framework is not intended for safety-critical domains requiring stronger guarantees and expert oversight.

Abstract

from arXiv · show

As Large Language Models increasingly automate complex, long-horizon tasks such as \emph{vibe coding}, a supervision gap has emerged. While models excel at execution, users often struggle to guide them effectively due to insufficient domain expertise, the difficulty of articulating precise intent, and the inability to reliably validate complex outputs. It presents a critical challenge in scalable oversight: enabling humans to responsibly steer AI systems on tasks that surpass their own ability to specify or verify. To tackle this, we propose Scalable Interactive Oversight, a framework that decomposes complex intent into a recursive tree of manageable decisions to amplify human supervision. Rather than relying on open-ended prompting, our system elicits low-burden feedback at each node and recursively aggregates these signals into precise global guidance. Validated in web development task, our framework enables non-experts to produce expert-level Product Requirement Documents, achieving a 54\% improvement in alignment. Crucially, we demonstrate that this framework can be optimized via Reinforcement Learning using only online user feedback, offering a practical pathway for maintaining human control as AI scales.

1. Introduction

As LLMs take on complex, long-horizon tasks, users become weak supervisors because they struggle to specify intent and verify outputs. Scalable Interactive Oversight addresses this by eliciting low-burden feedback through a recursive interaction tree before execution.

  • Vibe coding lets non-experts describe high-level software requirements while AI handles implementation, lowering the barrier to sophisticated software development.
  • Users face specification and verification gaps because precise intent is difficult to articulate and complex model outputs are difficult to validate.
  • Existing critique and debate approaches mainly evaluate outputs after generation, leaving ambiguity unresolved before long-horizon execution.
  • Scalable Interactive Oversight decomposes complex intent into a recursive tree of manageable decisions and guides users through structured elicitation rather than open-ended prompting.Users provide low-burden feedback at each node, allowing the agent to translate vague intent into precise specifications before execution.
  • 54% alignment improvement was achieved in website-development evaluation against standard vibe-coding and vanilla-interaction baselines.The evaluation used non-expert users producing professional-grade Product Requirement Documents under the sandwich protocol.
  • The framework is presented as an agentic approach whose interaction signals can support reinforcement-learning optimization from online user feedback.

2. Preliminary & Problem Setup

The paper formalizes scalable oversight as aligning a strong model with a non-expert’s latent intent when the user cannot fully specify or verify the desired outcome. It evaluates this problem by comparing weak-supervision alignment with expert-evaluated target intent.

  • Sandwiching experiments place a non-expert supervisor, a capable but potentially misaligned model, and an expert evaluator in one oversight setting.
  • The expert provides a good-faith evaluation signal without performing the task, serving as an approximate upper bound for alignment under correct supervision.
  • The setup models a non-expert user with latent intent who cannot fully specify requirements or reliably verify execution outcomes.
  • The model generates a Product Requirement Document from the user’s high-level query, while its output may not fully match the user’s underlying intent.
  • The research objective is to make alignment under weak human supervision approach alignment achievable with expert-level supervision.Alignment is measured by how closely the generated document approaches the target intent according to expert evaluation.

3. Method: Scalable Interactive Oversight

The method iteratively decomposes a long-horizon task into a tree, gathers low-burden feedback at leaf nodes, and propagates accumulated preferences to adapt future interactions. Once all nodes are resolved, the resulting global preference state guides downstream generation.

  • The interaction agent runs a decomposition-interaction loop that provides scalable oversight throughout the interaction process.
  • Given a natural-language query, the agent creates a tree-structured interaction plan whose leaves represent concrete development modules.In web-development requirements, examples include UI design as a parent and display rule as a child.
  • Users supervise each leaf through low-burden, mainly selection- or ranking-based queries tailored to the current subtask.Users can also respond with DontCare or DontKnow, allowing the interaction to adapt.
  • After each leaf interaction, the agent summarizes feedback into a node preference and updates the interaction plan using the cumulative preference state.
  • The loop continues until every node is visited, producing a global preference state for downstream generation.
  • The framework simplifies supervision by replacing full natural-language requirement specification with comparative judgments and recursively amplifies signals across the interaction tree.

4. Empirical Validation of Scalable Interactive Oversight Framework at Test Time

The test-time evaluation measures alignment between generated PRDs or websites and target user intent across simulated and human interactions. Scalable Interactive Oversight outperforms comparison methods, with gains increasing as preferences accumulate and interaction components are added.

  • Evaluation setup: 37 test cases evaluate PRDs built from real-world websites, using rubric-based alignment against decomposed target requirements.The evaluation covers modules including authentication methods and responsive design, with an LLM judge used at scale.
  • Baselines: The comparison includes direct PRD generation with vibe-coding frameworks and vanilla multi-turn free-form interaction.GPT-5, Claude Sonnet 4.5, and Gemini 2.5 Pro serve as document generators under the sandwich setting.
  • PRD alignment: +54% relative gain on Gemini-2.5-pro, increasing average alignment from 0.359 with vanilla interaction to 0.554.On GPT-5, the method improves over vanilla interaction and codex by 33% and 39%, respectively; Claude Sonnet 4.5 reaches 0.618 on average.
  • Module-level results: Module 2 shows the largest and most consistent gains, including +0.24 on GPT-5 versus codex and +0.15 on Gemini-2.5-pro over vanilla interaction.The method also improves later-stage performance, reaching up to +0.21 on Module 5.
  • Interaction scaling: Alignment increases as interaction nodes and accumulated preferences grow, in both simulated-user and human-user evaluations.Intermediate PRDs are evaluated every five interaction nodes for simulated users and every three nodes for the human-user experiment, which covers 10 cases.
  • Final implementations: The method consistently improves alignment in final full-stack websites under both LLM-based and human evaluation.Websites are generated from PRDs produced by the no-interaction baseline and the proposed method, then assessed with the same rubric.
  • Case study: Execution traces show that preferences reconfigure the tree, clarification cycles address ambiguity, and proposed options preserve advancement requirements.The case study connects interactive preference elicitation with dynamic tree updates and ambiguity control.
  • Ablation study: Ablations show 4.7% improvement from easier closed-format feedback and +11.7% on Module 2 from tree-based preference propagation.The lightweight o4-mini tree-update variant retains a substantial gain, indicating benefits beyond controller model scaling.

5. Training Interactive Agents with RL from Online Human Feedback

The paper optimizes the interactive oversight agent with reinforcement learning from online user feedback, optionally combined with expert outcome rewards. Training improves alignment, generalizes to untrained modules and settings, and reduces interaction turns.

  • Reward design: Online user feedback can serve as a reward for optimizing the interaction agent, avoiding reliance on users comparing or rating completed outputs.The user reward penalizes DontCare responses and is purely online.
  • Training rollout: The rollout repeatedly updates the requirement tree from simulated-user preferences until all nodes are resolved into a PRD.Each query can produce multiple training sequences because interaction traces are multi-turn and do not share prefixes.
  • Training results: User-only reinforcement learning increases alignment as training reward rises and DontCare responses decrease, demonstrating optimization from weak supervision alone.The reported relationship is based on training with online feedback from non-expert users.
  • Reward design: The combined user and expert reward achieves higher alignment, while the online reward produces more stable training dynamics.The expert component includes outcome-based evaluation of the generated PRD, alongside progressive node-level reward.
  • Generalization: 0.518 is the average M3–M5 score under User+Expert Reward versus 0.500 under the same-as-training setting, despite those modules being excluded from RL training.Under the GPT-5 setting, the untrained-module average increases from 0.606 to 0.640.
  • Interaction efficiency: RL training reduces both total interaction rounds and rounds per node, indicating more efficient information acquisition during interaction.The downward trends are reported across both efficiency measures.

6. Related Work

The related work situates the paper within scalable oversight and the shift toward agentic software development through vibe coding. It frames sandwiching experiments as an empirical test of whether weak supervision can align more capable models.

  • Scalable oversight: Scalable oversight reframes supervision as an easier task so humans can oversee models beyond their direct capabilities.The paper places its framework among approaches such as AI critique.
  • Vibe coding: Vibe coding describes an agentic software-development approach that emphasizes behavioral validation of AI-produced systems over direct source-code examination.The related-work discussion connects this approach to the transformation of AI from code-writing aid to collaborator.

7. Conclusion & Future Work

The paper frames Scalable Interactive Oversight as a way for non-experts to steer strong LLMs toward expert-level aligned outputs. It improves alignment in website requirement generation, while leaving code-level supervision and broader interaction studies for future work.

  • The framework enables non-experts to steer LLMs toward expert-level, intent-aligned outputs under weak supervision.
  • The evaluation shows significant alignment improvements over baselines on website requirement generation tasks.
  • Future work includes improving click-based interaction efficiency, evaluating broader real-user variation, and jointly training the tree updater.
  • The interaction agent acts as a glue layer between users and code agents for scalable oversight of code generation.
  • The study focuses on requirement-level oversight and does not fully investigate code-level supervision.

Impact Statement

The work targets settings where non-experts supervise strong models over extended interactions, using structured preference accumulation to improve intent alignment. Its deployment scope is limited by risks from early misunderstandings and the need for stronger safeguards in safety-critical domains.

  • As models handle more complex tasks, users often function as weak supervisors who struggle to convey intent through single-shot or unstructured feedback.
  • Structured interaction and accumulated preferences enable non-experts to more reliably steer strong models toward outcomes aligned with their intent.
  • Accumulated preferences may amplify early misunderstandings or biases, causing stable convergence on an incorrect objective.
  • The authors emphasize transparent interaction design and appropriate deployment safeguards rather than presenting a comprehensive AI safety solution.

A. Additional Meterials on Experiments Setting

The experiments evaluate PRD alignment using direct-generation and vanilla multi-turn baselines, modular rubric-based scoring, and validation of the LLM judge and user simulator.

  • Baselines: The study compares direct PRD generation through vibe coding with a vanilla multi-turn interaction baseline.
  • Baselines: The direct multi-turn baseline preserves simulated user interaction while removing the dialogue strategy component.
  • The two-stage method: Alignment evaluation decomposes each requirements document and its rubrics into five functional modules, then aggregates module-level scores.
  • The two-stage method: Rubrics are generated by an LLM and formatted as trees to support two-stage evaluation.
  • Validation: The LLM-judge validation found consistency among three popular LLMs across 80 evaluation tasks, while human annotators labeled only 0.5% of test-set rubrics unreasonable.
  • Validation: Human annotator agreement was 0.722, DeepSeek R1 agreement with humans was 0.677, and the SFT user simulator agreement was 0.707.

A.4. Details about RL training

The RL experiments use a GRPO variant tailored to multi-turn traces, compare reward designs and training settings, and illustrate generated PRDs and user interactions through figures and tables.

  • RL training: The RL setup uses learning rate 2e-6, batch size 8, clipping range [0.8, 1.2], one PPO epoch, and rollout size 4 unless otherwise specified.
  • RL training: The training algorithm is a GRPO variant designed for multi-turn traces with non-shared prefixes and small per-prompt groups.
  • Reward ablation: Table 5 presents an ablation of the reward design, reporting that the three reward types can each boost training separately.
  • Training curves: Figure 9 compares PPO epochs and rollout sizes using user reward only on the left and full reward on the right.
  • RL training: The advantage calculation uses terminal rewards, EOS masking, and batch-level return normalization.
  • Case studies: Figures 10 and 11 show real user interactions, generated PRD portions, oriented parts, and satisfied rubrics for website cases.

B.2. Website generation cases

The website-generation cases compare baseline and interaction-framework outputs across template-directory and Spanish-news tasks. The interaction framework is used to produce website requirements emphasizing detailed functionality, user experience, compatibility, and operational quality.

  • Website comparisons: The cases compare websites generated from baseline documents with websites generated from the interaction framework.Figure 12 labels the baseline cases on the left and interaction-framework cases on the right.
  • Case A: Websites for sharing templates: The template-directory case targets nontechnical users seeking attractive templates that can be previewed and quickly used on their own websites.The resulting PRD describes 300+ templates spanning multiple industries and aims to reduce technical barriers and website-building time.
  • Case A: Websites for sharing templates: The template-directory requirements include categorized and filtered discovery, immersive previews, guided onboarding, and one-click import and deployment.The deployment flow supports whole-site or single-page imports, dependency installation, and compatibility checks.
  • Case A: Websites for sharing templates: The template-directory PRD also specifies responsive device and browser support, WordPress ecosystem integration, differentiated free and premium tiers, and plugin compatibility.These requirements connect product reach, monetization, and implementation flexibility to the template experience.
  • Case B: Websites for Spanish news: The Spanish-news case targets readers in Peru and Latin America with politics, economics, society, and other coverage intended to establish a credible Spanish-language media brand.Its requirements include multi-channel browsing, live and video content, fact-checking, data journalism, and content-quality safeguards.
  • Case B: Websites for Spanish news: The Spanish-news requirements further specify editorial independence, proofreading, transparent corrections, high availability, multi-channel distribution, and real-time updates.These non-functional requirements address accuracy, credibility, stability, and timely publication.

B.4. Interaction progress case

The interaction-progress case shows how an initial requirement is decomposed into a tree, discussed through targeted questions, and converted into accumulated preferences that update the remaining plan. In the example, user feedback sharpens product positioning while removing an unnecessary market-analysis module.

  • Interaction progress: The system decomposes an initial user query into a hierarchical requirement tree and traverses it to select the next feature for discussion.The example begins with a news-site requirement tree and focuses first on product positioning.
  • Interaction progress: The interaction asks users to rank audience and content priorities, capturing both preference order and confidence.The example records the ranking [A >C >B] with confidence 0.8, followed by a question about data-driven versus investigative journalism.
  • Interaction progress: The accumulated preferences are summarized as product-positioning specifications for a professional primary audience, a broader secondary audience, and data-driven journalism.The resulting positioning emphasizes objective, in-depth information and quantitative, verifiable reporting.
  • Interaction progress: After interaction, the system updates the requirement tree by marking product positioning completed and retaining its audience, differentiation, value-proposition, and competitive-advantage elements.The completed node is incorporated into a revised plan while other modules remain unresolved.
  • Interaction progress: The updated plan deletes market analysis because of the user preference while retaining the business-model module and its revenue-stream features.The example explicitly shows advertising, subscription, licensing, and other revenue channels under business model.

C. Prompts in the Framework

The framework prompts define a structured requirements interview: initialize a hierarchical plan, discuss one feature through adaptive low-burden questions, update the plan, and generate specifications while respecting user scope and confidence. The prompts also prescribe simulation responses and structured evaluation inputs.

  • C.1. Interaction Model System Prompt: The interaction-model prompt assigns a senior product role and directs multi-round questioning to clarify requirements and produce an actionable feature specification.It focuses discussion on the current feature, its requirements, and previously stated user intent.
  • C.1. Interaction Model System Prompt: The interaction model requires topic boundaries, essential user decisions, consistency with stated needs, and a final detailed specification for the discussed feature.It allows direct decisions when requirements are already clear or the optimal choice is known.
  • C.1. Interaction Model System Prompt: The prompt adapts to DontCare responses and confidence scores, providing stronger guidance and explanation when the user is uncertain.Confidence is represented on a 0-to-1 scale.
  • C.1. Interaction Model System Prompt: The interaction reduces decision difficulty through module-by-module clarification, understandable explanations, dimension-based choices, and a metacognitive check on whether directions cover user concerns.Feature specifications are output in a fixed format covering overview, subfeatures, technical points, UI elements, and interaction logic when discussed.
  • C.2. Tree Initialization Prompt: The tree-initialization prompt organizes the interview around product overview, core functional modules, non-functional requirements, user experience, and business rules.Each module can contain nested submodules and fine-grained features in structured JSON.
  • C.3. Tree Updating Prompt: The tree-updating prompt permits adding, removing, modifying, or retaining features while preserving top-level titles, completed modules, logical structure, and similar overall complexity.It uses the original request, completed module, current plan, and unresolved modules as update context.
  • C.3. Tree Updating Prompt: The framework includes explicit handling for unresolved modules, revised full-plan output, and synthesis of feature discussions into a professional PRD.The document-generator prompt also requires strict scope boundaries and consistency with confirmed specifications.
Loading 2602.04210v2…