PaperlayerView exact revision on arXiv

Source-linked AI summary

InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training

Pengkai Wang, Pengwei Liu, Qi Zuo, Zhijie Sang, Congkai Xie, Hongxia Yang

arXiv:2510.15859v5cs.CLcs.AI

TL;DR

Open-ended medical dialogue has ambiguous, context-dependent feedback that is difficult to capture in a single reward and may require costly reward models. ORBIT trains with case-conditioned rubrics to guide incremental reinforcement learning, raising Qwen3-4B-Instruct’s HealthBench-Hard score from 7.0 to 27.5 with 2k samples. The paper also reports that rubric coverage can broaden while consultation quality remains strong.

  • Problem

    Open-ended medical dialogue involves ambiguous, context-dependent feedback that is difficult to summarize as a single scalar reward, while learned reward models are costly and can be vulnerable to reward hacking.

    Source: §1 · pp. 1–2

  • Method

    ORBIT uses expert rubric seeds and retrieval to generate case-tailored criteria that guide a general-purpose LLM judge in incremental reinforcement learning.

    Source: §1 · p. 2

  • Results

    7.0 to 27.5: ORBIT raises Qwen3-4B-Instruct’s HealthBench-Hard score using 2k training samples, achieving state-of-the-art performance among comparable-size open-source models.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    ORBIT’s analyses suggest that it learns case-conditioned clinical criteria rather than relying on evaluator-specific preferences or reusable rubric patterns.

    Source: §1 · p. 2

    ORBIT currently relies on a limited number of human-created rubric seeds.

    Source: §5 · p. 9

Abstract

from arXiv · show

Reinforcement learning (RL) has powered many recent breakthroughs in large language models (LLMs), especially for tasks where rewards can be computed automatically, such as code generation. However, it is less effective in open-ended medical dialogue, where feedback is ambiguous, context-dependent, and difficult to simply summarize into a single scalar signal-often requiring heavily supervised reward models and creating risks of reward hacking. Thus, we introduce ORBIT, an open-ended rubric-based incremental training framework tailored for critical medical dialogues. ORBIT integrates medical dialogue construction with dynamically generated case-conditioned rubrics that serve as adaptive guides for incremental RL. Unlike approaches that rely on external medical knowledge bases or handcrafted rules, ORBIT uses rubric-guided evaluation and can be implemented with general-purpose instruction-following LLMs, avoiding task-specific judge fine-tuning. With only 2k training samples, ORBIT raises Qwen3-4B-Instruct's HealthBench-Hard score from 7.0 to 27.5, achieving state-of-the-art performance among similarly sized open-source models while maintaining strong consultation quality as rubric coverage broadens.

1. Introduction

Open-ended medical dialogue makes reward specification difficult, while scalar rewards and learned reward models can be costly, fragile, and vulnerable to reward hacking. ORBIT uses case-tailored rubrics for incremental reinforcement learning and improves Qwen3-4B-Instruct’s HealthBench-Hard score with 2k training samples.

1. Introduction

  • Open-ended medical responses require case-dependent judgments about safety, empathy, and appropriateness, making a single precise reward difficult to define.Learned reward models are costly to construct, domain-dependent, fragile under distribution shift, and vulnerable to reward hacking when oversimplified.Source: §1 · p. 1
  • Rubrics make evaluation criteria explicit, while many medical LLMs still struggle on HealthBench-Hard, exposing a gap between QA-style optimization and realistic consultation.HealthBench uses expert-designed rubrics to assess clinical reasoning.Source: §1 · pp. 1–2
  • 7.0 to 27.5: ORBIT raises Qwen3-4B-Instruct’s HealthBench-Hard score using 2k training samples, achieving state-of-the-art performance among comparable-size open-source models.With larger rubric sets, ORBIT reaches 37.3, outperforming open-source models up to 8× its size.Source: §1 · p. 2
  • ORBIT uses expert-written rubric seeds and retrieval to create case-tailored criteria that a general-purpose LLM judge applies during incremental reinforcement learning.The framework is presented as replacing opaque scalar rewards and costly reward models with interpretable rubric-driven feedback.Source: §1 · p. 2

2. Related Work

Prior work has shifted open-ended LLM evaluation toward fine-grained rubrics, while reward-model approaches have expanded from preferences to rule-based and semantic evaluation. Medical LLM systems still face challenges with heterogeneous, context-dependent clinical reasoning.

Open-Ended Benchmarks

  • Rubric-based benchmarks assess multidimensional behavior with scenario-specific criteria; HealthBench highlights the difficulty of strong medical consultation performance under rigorous rubric evaluation.Examples include HealthBench, PaperBench, WildBench, AMEGA, and MultiChallenge.Source: §2 · p. 2

Reward Models in LLMs

  • Reward approaches progressed from preference signals to rule-based and semantic evaluators, but remain costly to build, hard to transfer across domains, and difficult to interpret at criterion level.The paper contrasts these approaches with rubric-guided RL using automatically generated context-sensitive criteria.Source: §2 · p. 2

LLMs for Health

  • Healthcare LLMs span applications including diagnosis support and documentation, yet most systems remain specialized and struggle to generalize to heterogeneous clinical reasoning.Research increasingly explores agentic systems with multi-step reasoning, tool use, and structured knowledge.Source: §2 · p. 2

3. ORBIT Framework

ORBIT turns realistic medical cases into dialogue-and-rubric training data, then uses case-specific rubric feedback to guide incremental reinforcement learning.

3. ORBIT Framework

  • ORBIT converts realistic clinical cases into dialogue-and-rubric training pairs through dialogue simulation, rubric generation, and rubric-guided reinforcement learning.The framework overview also identifies retrieval-augmented rubric generation and GRPO-based policy optimization.Source: §3 · p. 3, Figure 2 · p. 3

3.1. Medical Dialogue Construction

  • The dialogue-construction stage uses processed DoctorAgent-RL/MTMedDialog splits and adds ReMeDi dialogues for the large-scale training scenario.The authors use the resulting dialogues as query items for rubric evaluation and synthesize rubrics from them with in-context learning.Source: §3.1 · p. 3

3.2. Rubric Generator with In-Context Learning

  • The rubric generator embeds HealthBench cases and rubrics, retrieves and reranks relevant exemplars, then prompts a generative model to create case-specific evaluation checklists.It maintains both case–rubric pairs and a pool of unique rubrics for retrieval.Source: §3.2 · pp. 3–4
  • Pass@k-style filtering retains queries with intermediate performance and removes rubrics that are already frequently satisfied, concentrating training on informative supervision.The two filtering stages aim to retain solvable but challenging queries and discriminative rubrics.Source: §3.2 · p. 4

3.3. Rubric-Guided Reinforcement Learning

  • GRPO updates the policy from group-sampled rollouts using rubric-level rewards that positively weight desirable clinical behaviors and penalize unsafe or misleading ones.The reward formulation supports criterion-level credit assignment beyond matching conclusions.Source: §3.3 · p. 4
  • To stabilize training, ORBIT masks rollout groups with near-zero reward variance and periodically raises sampling temperature to restore exploration while retaining prior skills.Only valid queries contribute to the loss; staged entropy injection starts from the best prior checkpoint.Source: §3.3 · p. 4

4. Experiments

ORBIT’s experiments test performance on HealthBench-Hard and examine how data scale, filtering, rubric generation, and evaluator choice affect training and evaluation. The analyses report gains over inference scaling alone and probe judge reliability and rubric quality.

4.1. Experimental Setup

  • The experiments use 2,082 multi-turn consultations, larger 8k and 20k dialogue extensions, and non-Hard HealthBench rubrics as retrieval seeds; final benchmark scores use GPT-4.1.The core model is Qwen3-4B-Instruct-2507, and HealthBench-Hard is the 1k-case benchmark subset.Source: §4.1 · pp. 5–6

4.2. Quantitative Results

  • With 2k samples, InfiMed-ORBIT-4B raises its HealthBench-Hard score from 7.0 to 27.5, a +20.5-point gain.Using 8k samples raises the score to 33.6, and using 28k raises it to 37.3.Source: §4.2 · p. 6

4.3. Ablation Experiments

  • The ablations examine rubric generation and selection, evaluator reliability, pass@k filtering, dynamic sampling, and multi-stage training.These experiments characterize the contributions of individual ORBIT components.Source: §4.3 · p. 6

4.3.1. THE SELECTION OF EVALUATION MODEL AND RUBRIC GENERATION MODEL

  • GPT-OSS-120B tracks GPT-4.1 score improvements, while DeepSeek-R1 rubrics yield consistent downstream gains, motivating both choices for subsequent ablations.The study uses GPT-OSS-120B for evaluation and DeepSeek-R1 for rubric generation.Source: §4.3.1 · p. 6

4.3.2. EVALUATING DATA SCALABILITY AND TRAINING EFFICIENCY

  • InfiMed-ORBIT-4B improves monotonically from 2k to 28k training examples; moderate filtering preserves downstream performance while reducing runtime and response length.The authors also report that overly strict filtering creates an exploration–efficiency trade-off and that limited K = 8 rollouts may inadequately explore the policy space.Source: §4.3.2 · pp. 6, 8, Figure 4 · p. 7

4.4. Expanding Capability Boundaries through Rubric-Guided RL

  • At K = 40, ORBIT’s best-of-40 score distribution peaks above 0.85, versus approximately 0.65 for the baseline, with rubric pass-rate and hit-rate modes at 1.0.Increasing baseline inference sampling from K = 8 to K = 40 yields only marginal gains.Source: Figure 5 · p. 7, §4.4 · p. 8

4.5. Judge Reliability and Rubric Quality

  • Across 200 cases, the six-judge panel has case-level ICC(A, k) = 0.881; GPT-OSS-120B has the highest individual agreement with GPT-4.1 among local judges.The reported Krippendorff’s α = 0.999 reflects highly imbalanced binary rubric distributions, so the authors rely primarily on case-level ICC.Source: §4.5 · p. 8, Figure 6 · p. 9
  • Physicians rated 90.7% of generated criteria clinically meaningful, while RAG-grounded rubrics had 16-fold fewer near-duplicates than no-RAG rubrics.GPT-OSS-120B achieved 79.6% item-level agreement with physician annotations; the templatedness analysis links reduced rubric quality with lower downstream scores.Source: §4.5 · pp. 8–9

5. Conclusion

ORBIT uses rubric-guided reinforcement learning for open-ended medical dialogue and improves HealthBench-Hard performance with 2k training examples. The authors identify limited human-created rubric seeds as a remaining limitation.

5. Conclusion

  • With 2k training examples, ORBIT raises Qwen3-4B-Instruct’s HealthBench-Hard score from 7.0 to 27.5, setting a state of the art among comparable-size open-source models.The conclusion describes ORBIT as a scalable rubric-guided RL framework that decomposes alignment objectives into verifiable atomic rubrics.Source: §5 · p. 9
  • The authors recommend case-specific rubrics, evaluator signals stable across model families, and retaining sufficient reward variance after pruning.They report that expanding the candidate pool broadens case coverage, while difficulty-aware pruning improves sample efficiency.Source: §5 · p. 9
  • The authors identify reliance on a limited number of human-created rubric seeds as a limitation and propose grounding initial criteria in clinical practice guidelines as future work.Source: §5 · p. 9

Impact Statement

ORBIT is presented as a post-training research framework for assistive medical dialogue, not autonomous clinical deployment, and practical deployment requires expert oversight of rubric definitions and generated responses.

Impact Statement

  • ORBIT is intended for assistive medical dialogue rather than autonomous clinical deployment, with expert oversight required for initial rubric definitions and generated responses.Source: Impact Statement · p. 10

Reproducibility Statement

The appendix describes the rubric-generation pipeline, evaluator choices, and analyses of rubric quality. It reports that retrieval grounding reduces cross-case duplication while preserving within-case distinctness.

Reproducibility Statement

  • The Appendix provides training configurations, rubric-generation details, and the evaluation protocol to support faithful reproduction.Source: Reproducibility Statement · p. 10

A. Rubric Generator / A.2. Rubric Generation Prompt

  • For each case, a retrieval-augmented generator uses semantically related reference cases and rubric candidates to produce positive and negative clinical criteria; seed rubrics are synthesized rather than copied.The prompt requests multi-dimensional assessment, signed weights, and concise criteria.Source: A. Rubric Generator · p. 14, §A.2 · pp. 14, 16

A.1. System Prompt

  • The system prompt fixes rubric format and requires explicit attention to safety, escalation, context gathering, and factual correctness.Source: §A.1 · p. 14

A.3. Retrieval-Augmented Rubric Generation: Algorithm and Hyperparameters

  • The generation algorithm retrieves exemplar cases and rubric candidates, assembles an anti-copying prompt, generates criteria, and applies Pass@k refinement.The pipeline uses a two-stage retriever and outputs a case-specific rubric set.Source: §A.3 · p. 16

B. Evaluation Model Selection / B.1. Judge Agreement with GPT-4.1 on Fixed Rubric Cases

  • GPT-4.1 is the benchmark anchor, while GPT-OSS-120B is used for development because it agrees more closely with GPT-4.1 than notably lenient local evaluators.The fixed-case judge panel has case-level ICC(A, k) = 0.881, and GPT-OSS-120B achieves the highest local-judge agreement with GPT-4.1.Source: B. Evaluation Model Selection · p. 16, §B.1 · p. 17, Table 5 · p. 17

B.2. Judge choice during training

  • Stronger training-time judges consistently yield more effective policies, although even a self-judging 4B model provides a useful bootstrapping signal.Source: §B.2 · p. 17, Table 6 · p. 17

C. Rubric Generation Model

  • DeepSeek-R1 and Gemini-2.5-Pro produce the largest total-score improvements among tested rubric generators, and DeepSeek-R1 is selected as the default.The comparison holds prompt templates and the development scoring protocol constant.Source: C. Rubric Generation Model · p. 17, Table 7 · p. 18

C.1. Cross-case templatedness analysis

  • RAG-grounded rubrics show fewer cross-case near-duplicates and higher vocabulary diversity than Single-No-RAG rubrics, while within-case duplicate pairs exceed 0.85 at a rate of zero for both generators.The within-case mean pairwise similarities are 0.013 for Human-RAG and 0.014 for Single-No-RAG; the authors attribute the templatedness difference to cross-case patterns.Source: §C.1 · p. 18, Table 8 · p. 18, Figure 10 · p. 19

C.2. Seed-rubric source ablation

The ablation compares human-seed retrieval with automated rubric generation. Human-RAG is the low-cost default when expert seeds exist; a synthetic multi-LLM pipeline is a costly cold-start alternative.

C.2. Seed-rubric source ablation

  • 10.69 versus 17.89: single-model rubric generation lowers policy performance, while Multi-LLM-Syn-Contrast reaches 15.60 versus Human-RAG’s 15.34 on the matched 140-step subset.The synthetic pipeline uses four LLM generators, closed-model summarization, and pairwise contrastive filtering, but costs roughly 80× more than reusing HealthBench seeds.Source: §C.2 · pp. 19–20
  • Human-RAG achieves a score of 17.89 over 200 steps at roughly $50 per 1k rubrics, while Multi-LLM-Syn-Contrast scores 15.60 at 140 steps and costs about $4,000 per 1k rubrics.The figure compares HealthBench-Hard scores with estimated rubric-generation cost per 1k rubrics on a log axis.Source: §C.2 · p. 20, Figure 11 · p. 20
  • The authors recommend Human-RAG when expert seeds are available and Multi-LLM-Syn-Contrast for cold-start medical sub-domains without such seeds.The source ablation asks whether expert-written HealthBench seeds are necessary and compares retrieval-augmented human seeds with single-model and synthetic alternatives.Source: §C.2 · pp. 19–20

D. Evaluation of Smaller Medical Models

Small open-source medical models perform weakly on HealthBench-Hard despite medical specialization. The authors attribute this gap to training that does not capture the behaviors demanded by open-ended consultation.

D. Evaluation of Smaller Medical Models

  • Medical small-model baselines score weakly on HealthBench-Hard under GPT-4.1 evaluation, despite their medical specialization.The table reports these small medical-model baselines using the official benchmark evaluator.Source: D. Evaluation of Smaller Medical Models · p. 20, Table 10 · p. 21
  • Their structured question-answer and reasoning training does not capture multi-turn information gathering, safety-netting, and documentation behaviors required by HealthBench-Hard.The authors state that domain-focused pretraining or instruction tuning alone does not suffice for this open-ended, rubric-intensive consultation setting.Source: D. Evaluation of Smaller Medical Models · p. 20
  • The finding does not imply medical expertise is useless; domain expertise becomes an effective scaffold when paired with case-conditioned rubrics and RL.The rubrics translate broad medical proficiency into concrete behavioral objectives.Source: D. Evaluation of Smaller Medical Models · p. 20

E. Distributional Analysis of ORBIT Models

Across score and rubric-level distributions, ORBIT shifts outcomes toward higher scores and broader rubric satisfaction, while simply increasing baseline sampling yields only small gains. The accompanying appendices document data, baseline, and training configurations.

E. Distributional Analysis of ORBIT Models

F. More Experiment Details and Results

F.1. Datasets and Preprocessing

  • The 2k Core and 8k scalability splits serve as rubric-pipeline seed pools, with consultations truncated at assistant turns before rubrics are regenerated using RAG.The appendix also describes sample- and rubric-level difficulty filtering, dataset sizes, and context and response truncation limits.Source: §F.1 · p. 23

F.2. Baseline Models

  • Baseline documentation covers model size, release date, instruction-tuning style, and evaluation procedure; the RL and SFT setups are reported alongside training and infrastructure details.For MoE models, active and total parameter counts distinguish evaluation cost from overall capacity.Source: §F.2 · p. 24, §F.3 · p. 24

F.4. Filter Dynamics and Training Stability

Filtering increases reward-signal informativeness and limits response-length drift, while the broader results expose trade-offs, evaluation uncertainty, data-scaling patterns, and examples of rubric-guided behavior. The appendix also compares the semantic structure of HealthBench-Hard and HealthBench-Consensus rubrics.

F.4. Filter Dynamics and Training Stability

  • The strict [0, 0.25] rubric filter expands mean reward range from 2.23 to 4.30, a 93% increase; it also limits final mean response length to 1,794 tokens instead of the unfiltered 2,288.The unfiltered response length rises from about 560 tokens at step 50; entropy-guided restarts trigger near step 260 and increase rollout temperature by 10%.Source: §F.4 · pp. 24, 26, Figure 16 · p. 26

F.5. Evaluation Protocols and Configurations

  • Across three identical trials, InfiMed-ORBIT-4B has a HealthBench-Hard mean of 27.4, standard deviation 0.42, and range 26.8–27.7.The main evaluation uses GPT-4.1, with decoding settings specified by model lineage; local evaluators are used for development and ablations.Source: §F.5 · p. 27

F.6. Per-axis and Per-theme Improvement Breakdown

  • On the matched 200-case subset, ORBIT’s aggregate score rises from 8.0 to 19.1, driven mainly by completeness (+20.5) and context awareness (+11.8), with regressions in communication quality and instruction following.The complex responses theme remains at zero across evaluated models; the authors describe the safety-skewed rubric mix as a structural limitation.Source: §F.6 · pp. 27–28, Figure 17 · p. 28

F.7. Statistical Significance Analysis

  • On the matched 200-case subset, gains over the Instruct baseline are statistically significant, but ORBIT’s gain over the SFT baseline is only suggestive (p=0.057; confidence interval includes zero).The bootstrap resamples matched case IDs 1,000 times; significance testing on the full 1,000-case headline split is left to released per-case files and harness.Source: §F.7 · p. 28, Table 16 · p. 28

F.8.1. SFT+RL VS ZERO-RL

  • A moderate SFT warm start provides stable medical-response structure before RL, while direct RL from Qwen3-4B-Instruct can still improve with a sufficiently conservative learning rate.All variants use the same 500-step RL budget.Source: §F.8.1 · p. 29, Table 17 · p. 29

F.8.2. MORE DATA SCALING EXPERIMENTS

  • Increasing rubric-generated training cases from 2k to 8k and 28k yields monotonic total-score gains and broadens gains across most themes and axes.These scale-ablation results are single-run trajectories rather than selection-optimized upper bounds.Source: §F.8.2 · p. 29, Table 18 · p. 29

G. Case Studies

  • Case studies contrast baseline omissions with ORBIT’s postpartum risk screening and escalation, calibrated uncertainty about saffron tea, and completion of a coding request.The examples respectively emphasize safety triage, separating limited evidence from treatment claims, and satisfying a structured health-data task.Source: §G.1 · p. 29, §G.2 · p. 30, §G.3 · p. 30

H.1. Semantic-space comparison of HealthBench-Hard and HealthBench-Consensus

  • HealthBench-Consensus rubrics form tight, homogeneous clusters, whereas HealthBench-Hard rubrics are sparse and scattered, consistent with more variable constraints and reasoning demands.The authors note that the semantic separation makes straightforward memorization of Consensus few-shot examples an unlikely explanation for the observed improvements.Source: §H.1 · pp. 31–32, Figure 19 · p. 32

H.2. Contamination diagnostics between NoHard and Hard

The diagnostics find little evidence that NoHard causes instance-level contamination of Hard, while indicating substantial rubric-template reuse. The authors interpret the results as generalization to unseen prompts within a shared rubric framework, not to entirely new rubric formulations.

H.2. Contamination diagnostics between NoHard and Hard

  • The contamination analysis focuses mainly on prompts and combines prompt–rubric similarity with within-set redundancy checks to distinguish possible leakage from harmless template reuse.Source: §H.2 · p. 32
  • The analysis compares filtered NoHard items against Hard using character n-gram TF–IDF cosine similarity; SQ tracks prompt duplication, while SR tracks rubric similarity that may reflect templating.Source: §H.2 · p. 32
  • Prompt similarity concentrates at moderate levels and its thresholded contamination rate drops quickly, while rubric similarity has a heavier high-similarity tail consistent with templating.The analysis summarizes thresholded rates for thresholds τ ∈[0.5, 1.0].Source: §H.2 · p. 33
  • At prompt and rubric cutoffs of 0.90 and 0.80, respectively, only a small number of points exceed both, suggesting strong rubric similarity rarely coincides with near-duplicate prompts.Source: §H.2 · p. 33
  • Both splits show substantial within-set rubric redundancy, consistent with template reuse contributing to the increased rubric-similarity tail.Source: §H.2 · p. 33
  • The combined diagnostics suggest NoHard is unlikely to cause instance-level contamination of Hard, supporting generalization to unseen prompts within a shared rubric framework.The authors distinguish this from robustness to completely new rubric formulations.Source: §H.2 · p. 33

I. Negative Results and Failure Modes

The appendix reports failure modes involving stringent filtering, SFT overfitting, and rubric generation, then contrasts ORBIT with related alignment approaches. It also reports that a general-purpose 4B model can exceed specialized 32B medical baselines on HealthBench-Hard.

I. Negative Results and Failure Modes

  • Overly strict Pass@k filtering can restrict throughput to 9% of available instances and induce optimization gridlock on ultra-difficult tasks.The production framework instead uses a broader query band and adaptive rubric thresholds, retaining 67%–84% of the empirical distribution.Source: §I · p. 35
  • At η=10−5, the SFT initialization regressed to 20.3 after RL, as style overfitting collapsed response diversity and left low-information rollouts.More than 72% of rollouts converged to the same template; the authors adopted 10−7 as the standard SFT learning rate.Source: §I · p. 35
  • GPT-5-Chat produced less discriminative rubrics under the authors’ prompt because its safety post-training discouraged negative failure-mode criteria.The authors therefore adopted DeepSeek-R1 as their default rubric generator.Source: §I · p. 35

J. Extended Related Work

Loading 2510.15859v5…