PaperlayerView exact revision on arXiv

Source-linked AI summary

Astar: Learning to Propose Evolution Directions for Self-Evolving Industrial AI Systems

Jinxin Hu, Hao Deng, Haibo Xing, Lingyu Mu, Muyu Zou, Weiqin Yang, Sirui Chen, Bohao Wang, Zhezheng Hao, Hao Zhang, Zulong Chen, Shizhun Wang, Yu Zhang, Xiaoyi Zeng, Jiawei Chen

arXiv:2608.27287v1cs.IR

TL;DR

Proposing effective evolution directions remains a bottleneck because general-purpose LLMs lack scenario-specific experience and produce generic, misaligned suggestions. Astar learns from industrial iteration histories using corpus expansion and denoising, staged training, hierarchical hints, and surrogate evaluation. Deployed in Lazada advertising, Astar-8B achieved a 0.6786 single-proposal success rate and supported sustained automatic iteration with offline and online gains.

  • Problem

    Evolution-direction proposal remains difficult to automate because general-purpose LLMs lack scenario-specific experience, while historical training records provide sparse and noisy supervision.

  • Method

    Astar builds a denoised, pairwise-expanded evolutionary corpus and trains a specialized model with mid-training, SFT, RL, hierarchical hints, and a reward-model surrogate evaluator.

  • Results

    0.6786 single-proposal success rate: Astar-8B exceeded human experts at 0.3229 and the strongest general-purpose LLM at 0.3071 in real-execution evaluation.

  • Takeaways & Limitations

    Astar guided 20 consecutive iterations over two weeks, improving offline Hitrate@200 by 23.6%, with online relative lifts of 4.86% in GMV and 1.82% in advertising revenue.

  • Takeaways & Limitations

    Astar currently generates directions in a single forward pass; future multi-round reflection and multi-agent co-judgment are proposed to reduce trial-and-error costs.

Abstract

from arXiv · show

Modern AI systems advance through continuous iteration: a loop of proposing evolution directions, implementing code, training, and evaluation. While the latter three stages are increasingly automated, the starting point --- proposing effective evolution directions --- remains a critical bottleneck that still relies heavily on senior experts. In this work, we explore whether AI can take over this role. We find that general-purpose LLMs, even the advanced GPT-5.5, offer only generic and misaligned suggestions: the required expertise is accumulated through experience rather than explicitly codified, and thus hard to inject directly. To this end, we propose Astar, a training-based approach that learns a specialized evolution-guiding model from the abundant iteration histories of industrial systems. Realizing this idea, however, raises four challenges: sparse supervision, noisy data, a vast direction space, and prohibitively expensive verification. We address them along two fronts. On the data side, we design a pipeline that turns noisy historical commits into a large, clean evolutionary corpus via pairwise sample expansion and noise filtering. On the model side, we train the model through mid-training, SFT, and RL, guiding evolution direction generation with hierarchical hints and using the reward model in RL as a fast surrogate evaluator. Astar has been deployed in Alibaba's Lazada advertising system for evolution direction proposal. Astar-8B achieves a single-proposal success rate of 0.6786 in real-execution evaluation, far exceeding human experts (0.3229) and the strongest general-purpose LLM (0.3071). More importantly, Astar closes the loop and enables fully automatic iteration: it guided 20 consecutive iterations over two weeks, improving offline Hitrate@200 by 23.6%, while an online A/B test yielded relative lifts of 4.86% in GMV and 1.82% in advertising revenue.

1. Introduction

Astar treats evolution-direction proposal as the bottleneck in industrial AI iteration and learns this capability from historical system changes. It combines corpus construction, staged training, hierarchical guidance, and surrogate evaluation, achieving strong deployment results.

  • Motivation: Industrial AI iteration comprises proposing directions, implementing code, training models, and evaluating downstream performance.Evaluation determines whether proposals are retained or discarded, driving continual improvement.
  • Motivation: Proposal generation remains difficult to automate because it requires deep business, data, model, configuration, and training-dynamics expertise, while validation is high-stakes and costly.General-purpose LLMs produce generic, misaligned suggestions because scenario-specific expertise is accumulated through experience and difficult to codify.
  • Challenges: Historical commits provide abundant training signals by recording proposed changes and resulting performance shifts, but learning from them faces sparse supervision, noisy records, vast direction space, and expensive verification.The challenges span both evolutionary-corpus construction and model training.
  • Approach: Astar expands supervision by pairing arbitrary historical versions, filters irrelevant changes, and trains a specialized model through mid-training, SFT, and RL.Its model uses hierarchical hints to narrow generation progressively and a reward model as a fast surrogate evaluator for RL and candidate ranking.
  • Results: 0.6786 single-proposal success rate: Astar-8B exceeded human experts at 0.3229 and the strongest general-purpose LLM at 0.3071 in real-execution evaluation.Deployed on Alibaba’s Lazada advertising platform, Astar also guided 20 consecutive iterations and improved offline Hitrate@200 by 23.6%.

2. Evolutionary Corpus Construction

Astar constructs an evolutionary corpus from industrial experiment histories by expanding sparse supervision and filtering noisy code changes. Each training instance links system context to a denoised, structured evolutionary action with hierarchical guidance.

  • Data Expansion: Adjacent experiment pairs provide insufficient data and may capture incomplete micro-edits rather than meaningful evolutionary steps.This motivates pairing non-adjacent experiments in the historical sequence.
  • Data Expansion: Pairwise expansion combines any two experiments, compares their full loss curves, and labels which version performs better to determine the improvement direction.Pairing arbitrary experiments creates a quadratic number of training samples.
  • Execution Logic Filtering: A two-stage denoising pipeline removes unreachable code, syntax-level noise, engineering configuration changes, and edits lacking explicit evolutionary intent.It combines codebase-level cleaning, delta-level filtering, parameter whitelisting, and LLM semantic analysis.
  • Evolution Direction Representation: Hierarchical Hints classify modifications as coarse-grained direction, fine-grained modification action, and concrete modification plan, progressively narrowing the evolution search space.An Evolution Idea supplements the taxonomy with a concise explanation of exact code changes.
  • Context Construction: Each corpus instance combines Data Features and a natural-language Basic Information overview of the prior codebase with denoised evolutionary signals.The resulting format maps the initial system state to the intended evolutionary action.

3. Model Training

Astar uses a three-stage training pipeline to learn evolution strategies from historical AI-system experience, structure the vast direction space, and support exploration with fast reward-based feedback. The pipeline combines mid-training, supervised fine-tuning on successful directions, and reinforcement learning with a trained reward model, while verified outcomes feed back into continued training.

  • Training motivation: Training on evolutionary history helps Astar internalize experience and generalize it to novel AI-system scenarios.The approach avoids relying only on context-based memorization of past attempts.
  • Mid-training and structured generation: Hierarchical hints guide Astar from high-level strategies toward specific modifications, narrowing the evolution space and reducing generation uncertainty.They also cluster directions by category during training, reducing overfitting to specific optimization details.
  • Mid-training and structured generation: 112B tokens of evolutionary history are used for mid-training, aligning the model with the structured representation of evolution directions.Mid-training performs next-token prediction over the complete evolutionary history dataset.
  • Supervised fine-tuning: A combined loss metric uses final-step loss, mean loss over the final k steps, and standard deviation to measure convergence outcome and training stability.The mean term filters transient noise, the final-step term captures ultimate convergence, and the stability term penalizes oscillatory behavior.
  • Supervised fine-tuning: SFT retains 81.1M tokens of positive samples and fine-tunes exclusively on evolution directions that demonstrably improve performance.Positive samples satisfy s_cur < s_prev, while failed attempts are labeled negative.
  • Reinforcement learning: The reward model reaches AUC 0.8487 at 8B parameters, versus 0.6142 for human experts and 0.5997 for the best general-purpose LLM.It supplies a learned feedback signal for reinforcement-learning exploration when full training and deployment verification is expensive.
  • Continuous evolution: Verified outcomes are recycled into training, enabling Astar to co-evolve with the target system across consecutive iterations.The paper reports that a static model would become outdated as the system moves beyond its training states.

4. Experiments

Astar is evaluated against expert and general-purpose LLM baselines using surrogate and real-execution metrics, ablations, production deployment, and representative cases. The results show strong generation quality, benefits from each training and data-design choice, and improvements in industrial iteration outcomes.

  • Experimental Setup: RM@k and S@k evaluate candidate quality through reward-model predictions and actual loss decreases, respectively.RM@k is a cheaper surrogate, whereas S@k uses real-execution feedback.
  • Generation Quality Comparison: Astar-8B achieves S@1 of 0.6786, exceeding human experts at 0.3229 and GPT-5.5 at 0.3071.The smallest Astar-0.6B model also reaches S@1 of 0.5435.
  • Training Stage Ablations: RM@1 rises from 0.0380 for untrained Qwen3-8B to 0.6041 after mid-training, 0.6853 after SFT, and 0.7183 after RL.The ablation attributes successive gains to domain knowledge, alignment with effective directions, and exploration of novel directions.
  • Data-Design Ablations: Basic Information yields S@1 of 0.6786 versus 0.4151 for raw repositories, while removing Hierarchical Hints lowers reward AUC from 0.8487 to 0.7910.These ablations test the input abstraction and output-structure choices.
  • Online Experiments: Astar guided 20 consecutive iterations over two weeks, improving offline Hitrate@200 by 23.6% and online GMV by 4.86% and advertising revenue by 1.82%.The online A/B test also reports gains in Click and Order.
  • Case Studies: Representative cases target level-specific decoder experts and spectral denoising for the Muon optimizer.The examples illustrate targeted architectural and optimizer evolution directions.

5. Conclusion and Future Work

Astar learns evolution directions from AI systems’ iteration histories and addresses data and model-training challenges with targeted corpus and training designs. Future work proposes deliberative self-improvement and broader deployment across additional scenarios.

  • Conclusion: Astar uses pairwise data expansion, noise filtering, hierarchical hints, and a reward model to address four evolution-guidance challenges.It is trained through mid-training, SFT, and RL.
  • Conclusion: Astar-8B achieves S@1 of 0.6786 versus 0.3229 for human experts and 0.3071 for the strongest general-purpose LLM.In deployment, it improved offline Hitrate@200 by 23.6% and online GMV and advertising revenue by 4.86% and 1.82%.
  • Future Work: Future work proposes multi-round reflection, multi-agent co-judgment, and extension beyond e-commerce advertising to other scenarios.The proposed additional scenarios include content recommendation, foundation-model fine-tuning, and agent harnessing.

A.1. Raw Experiment Record

A raw experiment record is a complete snapshot of one experiment, combining code, training, metrics, and metadata fields.

  • Raw Experiment Record: Each raw record contains an engineering-code snapshot, training log and metrics, and experiment metadata.The full field structure is listed in Table 7.

A.2. Configuration Whitelisting Rules

The appendix provides complete classification rules for configuration parameters used in the evolutionary corpus.

  • Configuration Whitelisting Rules: Configuration parameters are organized according to the complete classification rules listed in Table 8.The passage identifies the table but does not enumerate its categories.

A.3. Syntax Normalization and Reachability Analysis

Astar normalizes code syntax to separate logical modifications from superficial edits and organizes purified changes with hierarchical evolution hints.

  • Syntax Normalization: AST normalization removes debugging statements, comments, and import-order differences before comparing Python files for logical changes.Files that fail to parse use a cleaned line-level comparison fallback.
  • Hierarchical Hints: The hierarchical hint schema classifies modifications from coarse-grained directions to fine-grained actions and concrete plans.The supplied table identifies the first- and second-level structure of Evolution-Intent Hierarchical Hints.

B. Training Details

Astar uses a three-stage training pipeline with documented hyperparameters spanning mid-training, supervised fine-tuning, reward-model training, and reinforcement learning.

  • Training Details: Astar’s training pipeline covers mid-training, SFT, reward-model training, and RL using GRPO.Table 10 specifies the hyperparameters for these stages.

C. Case Studies

Two representative case studies show Astar generating targeted, directly deployable evolution directions for a generative retrieval model.

  • Case Studies: The case studies begin from a generative retrieval model baseline and use hierarchical hints to produce evolution directions.The examples are designed to illustrate Astar’s ability to propose effective directions.
  • Case Studies: The two cases modify different components: model architecture in one case and the optimizer algorithm in the other.

C.1. Case 1: Level-Specialized Decoder Feed-Forward Network

The first case identifies interference from a shared decoder FFN across heterogeneous codebook levels and proposes level-specific experts with deterministic routing. The design preserves per-token computation while increasing parameters linearly, targeting specialized modeling without mutual interference.

  • Target System and Baseline: The GR decoder autoregressively generates L-level RQ-VAE codebook tokens, whose higher and lower levels represent coarse and fine item information.The baseline uses one SwiGLU FFN shared across all codebook levels.
  • Training Call Flow: The training call flow sends every codebook level through the same decoder stack and applies a per-level InfoNCE loss.The supplied implementation excerpts show teacher forcing, shared decoder inputs, logits computation, and per-level loss calculation.
  • Problem Analysis: Astar identifies a structural mismatch because heterogeneous token distributions across levels must share one FFN mapping, causing cross-level interference.The mismatch is between level-specific semantic distributions and the shared function f_θ applied to every level.
  • Proposed Evolution Direction: The proposed direction replaces the shared FFN with L experts, routing each level-ℓ token deterministically to expert ℓ.Routing depends only on the codebook level and does not use a learnable gating network.
  • Design Rationale: Dedicated experts specialize coarse- and fine-grained tokens while avoiding common MoE routing-loss and load-balancing issues.The routing signal is uniquely determined by level rather than by learned gating.
  • Design Rationale: Per-token forward computation and memory remain unchanged, while the parameter count grows linearly with the number of experts.The comparison reports unchanged O(d d_ff) per-token cost and parameters increasing from P to L·P.

C.2. Case 2: Spectral Denoising for the Muon Optimizer

This case identifies noise amplification in Muon’s Newton-Schulz orthogonalization and proposes MP-based spectral denoising to separate signal from noise before updates.

  • C.2. Case 2: Spectral Denoising for the Muon Optimizer: Muon’s Newton-Schulz orthogonalization drives all singular values to 1, equalizing update weights across gradient directions.The target system uses Muon for two-dimensional attention matrices and AdamW for one-dimensional parameters and heterogeneous layers.
  • C.2. Case 2: Spectral Denoising for the Muon Optimizer: Because signal directions have larger singular values than noise directions, undifferentiated normalization can amplify noise that was previously suppressed.The problem arises when noise singular values are raised to the same update weight as signal singular values.
  • C.2. Case 2: Spectral Denoising for the Muon Optimizer: Astar uses the Marchenko-Pastur noise upper bound as a deterministic criterion for signal/noise-separated orthogonalization.The criterion is based on the gradient matrix aspect ratio γ = m/n.
  • C.2. Case 2: Spectral Denoising for the Muon Optimizer: Directions above σth are treated as signal and normalized, while directions within the MP distribution are attenuated rather than normalized.This contrasts the baseline, which maps all singular values to 1, with MP spectral denoising.
  • C.2. Case 2: Spectral Denoising for the Muon Optimizer: The MP criterion requires no learnable parameters, extra loss term, gating, or load balancing, and adds no training burden.A single spectral separation is inserted before orthogonalization while preserving Muon’s signal-direction equalization.
Loading 2608.27287v1…