Source-linked AI summary
EvolveScaler: Synthesizing Information-Evolution Contexts via Executable State Machines and Natural-Language Rendering
Ziliang Zhao, Zenan Xu, Shuting Wang, Zhao Wang, Bowen Cao, Minda Hu, Lincheng Li, Pluto Zhou, Zhicheng Dou
TL;DR
Persistent interactions require models to track evolving information whose later updates can revise earlier records, but existing text-first synthesis leaves transition semantics and answer logic implicit. EvolveScaler defines these processes with executable specifications and simulators, then uses deterministic replay for supervision; its data supports challenging evaluation and improves performance across independently constructed benchmarks.
Problem
Information Evolution requires models to identify valid records, apply updates in order, and reconstruct query-relevant state, while existing evaluations rarely formalize these update semantics systematically.
Method
EvolveScaler uses human-authored operational specifications and LLM-synthesized executable simulators to generate natural-language histories, with deterministic replay producing answers and atomic checklists.
Results
Training an internal A3B model on 6,000 examples improved performance on all eight independently constructed out-of-distribution benchmarks, raising average score by 5.25 points.
Takeaways & Limitations
EvolveScaler provides both diagnostic evaluation of evolving-information reasoning and transferable training supervision beyond its task format.
Takeaways & Limitations
The framework focuses on discrete, programmatically specified state transitions; extending it to partially observed, continuous, and multimodal settings remains future work.
Abstract
from arXiv · showhide
In persistent interactions, long contexts may encode an evolving process rather than a fixed record: later events can revise or revoke earlier information, changing what remains valid and what conclusions follow. We call this setting information evolution (IE). Solving IE requires identifying valid records, applying updates in order, and reconstructing the query-relevant state from the event history. Existing text-first synthesis pipelines make such data difficult to verify because state transitions and answer logic remain implicit. We introduce EvolveScaler, a code-driven framework that defines information evolution before rendering it as natural language. Human-authored operational specifications define state transitions, record validity, difficulty controls, and executable answer logic; a strong LLM then synthesizes a self-contained simulator from each specification. Executing validated simulators produces natural-language multi-turn event histories, while deterministic replay computes reference answers and atomic checklists. We instantiate EvolveScaler with 117 task prototypes and 159 final-question operators across five difficulty levels spanning approximately 7 to 1,200 events per instance, yielding about 35,100 training examples and 585 validated evaluation instances. On the very_long tier, the strongest model reaches 59.3% avg@5, while six models score below 10%. Training an internal A3B model on 6,000 EvolveScaler examples improves performance over its base checkpoint on all eight independently constructed out-of-distribution benchmarks, with a 5.25-point average gain. These results show that code-driven IE synthesis provides both challenging evaluation and transferable training supervision.
1 Introduction
Information Evolution treats long interactions as ordered event histories whose updates can revise record validity and change query-relevant conclusions. EvolveScaler makes these dynamics executable before rendering them as natural language, enabling controlled synthesis and deterministic supervision.
- Problem: Existing evaluations rarely formalize update semantics as explicit, controllable task components across domains and difficulty levels.
- Method: EvolveScaler defines state transitions, record validity, difficulty controls, and answer logic in code before rendering histories as natural language.
- Scale: 117 task prototypes and 159 final-question operators support approximately 35,100 training examples and 585 validated evaluation examples across five tiers.
- Problem: Information Evolution requires reconstructing query-relevant state from ordered histories whose updates can revise or invalidate earlier information.
- Results: Deterministic answers support trustworthy evaluation across 14 LLMs and show reliability declining as trajectory scale and evolution complexity increase.
2 Related Work
Related work covers long-context retrieval and reasoning, multi-turn interaction, dialogue-state tracking, event-stream evaluation, synthetic data generation, and executable environments. EvolveScaler complements these lines by programmatically controlling event-system semantics and task difficulty.
- Long-context benchmarks: Long-context benchmarks test retrieval, integration, and reasoning over evidence distributed across extended inputs and varied context lengths.
- Multi-turn benchmarks: Multi-turn benchmarks examine instruction following, cross-turn coherence, compliance, multilingual interaction, and conversational preferences.
- State and memory: Dialogue-state and conversational-memory benchmarks model successive state changes or evaluate memory across extended and multi-session interactions.
- Event streams: MARS-Bench uses real sports event streams, whereas EvolveScaler encodes event systems programmatically to control trajectory scale, validity, noise, and question difficulty.
- Synthetic and executable data: LLM-generated synthetic data supports instruction and dialogue training, while programmatic datasets and simulators provide stronger structural guarantees through synthetic worlds or executable environments.
3 The EvolveScaler Framework
EvolveScaler defines information evolution as executable state transitions and then renders validated simulations into natural-language contexts with deterministic supervision. Its controls vary reconstructed-state questions and evolution difficulty across diverse task instances.
- Executable specifications: EvolveScaler formalizes each task with an operational specification defining state changes, record validity, difficulty controls, and answer computation.A strong LLM uses the specification to synthesize a self-contained simulator rather than directly authoring an unverifiable long-form sample.
- Deterministic supervision: Validated simulators generate event histories and questions, while deterministic replay computes reference answers and atomic checklists from the reconstructed state.The checklist must remain consistent with the reference answer, separating model-assisted synthesis from ground-truth computation.
- Validation: Program verification retains simulators only after successful execution across seeds, invariant preservation, deterministic replay, and checklist consistency.More than twenty annotators also audited a random sample of 117 instances and found zero errors.
- Framework overview: The framework renders executable trajectories into diverse natural-language contexts and automatically produces verifiable question–answer pairs for training and evaluation.The framework is illustrated as a pipeline from atomic task primitives to state-machine simulation, natural-language rendering, and answer generation.
- Controllable generation: Question operators select views of the reconstructed state, whereas difficulty tiers scale trajectory length and evolution complexity from approximately 7 to 1,200 events per sample.Difficulty controls include invalid-record rate, backfill distance, entity interleaving, and counterfactual depth; operators cover ranking, aggregation, audit, counterfactual, and localization tasks.
4 Experiments
EvolveScaler evaluates models across increasingly difficult trajectory-length tiers and uses held-out benchmarks to test whether continued training transfers beyond its task format. Results show sharp difficulty-dependent performance separation, broad gains from continued training, benefits from mixed-length supervision, and aggregation as the dominant semantic failure mode.
- Evaluation results: GPT-5.5-xhigh achieves the best overall performance, with a pass@5 of 82.7 and an avg@5 of 72.8.Performance declines as trajectory scale and evolution complexity increase across the five tiers.
- Evaluation results: The median avg@5 across models falls from 71.2 on very_short to 11.3 on very_long, with six models below 10 on very_long.The hardest tiers remain challenging even for strong models.
- Evaluation results: The avg@5 range widens from 59.0–81.7 on very_short to 4.8–59.3 on very_long, increasing separation among models.On very_long, GPT-5.5-xhigh scores 59.3 versus 9.6 for GLM-5.1.
- Training generalization: At 6,000 examples, continued training raises the eight-benchmark average from 33.17 to 38.42, a total gain of 5.25 points, and exceeds the base model on all eight benchmarks.The evaluated continued-training settings use 200, 2,000, and 6,000 EvolveScaler examples.
- Training generalization: A mixed set of 2,000 short-, medium-, and long-tier examples achieves the strongest performance on all eight benchmarks under a fixed 6,000-example budget.Single-tier settings exhibit benchmark-dependent trade-offs, while mixed trajectory scales provide complementary supervision.
- Error analysis: Net-Change Aggregation is the largest error category for every reported model, accounting for 46.2%–55.9% of labeled semantic failures.Retrieval & Counting contributes 16.8%–21.1%, while Ranking & Tie-Breaking remains prominent for the strongest model.
5 Conclusion
EvolveScaler synthesizes information-evolution data through executable simulators whose replayed states determine supervision. Its evaluation results show persistent difficulty for models and transfer to independently constructed benchmarks, while the current framework remains focused on discrete, programmatically specified transitions.
- Framework: EvolveScaler defines state transitions, record validity, difficulty controls, and answer logic in operational specifications, then uses executable simulators and deterministic replay for supervision.Natural-language event histories are generated from simulator execution, while reference answers and atomic checklists come from replayed program state.
- Resource: The resource contains 117 task prototypes and 159 final-question operators across five difficulty tiers.
- Evaluation: As trajectory scale and evolution complexity increase, the median model avg@5 score falls from 71.2 on very_short to 11.3 on very_long.The harder tiers also reveal substantially larger performance differences among models.
- Training: Continued training of an internal A3B model on 6,000 examples improves performance on all eight independently constructed out-of-distribution benchmarks.The average score rises by 5.25 points, while atomic checklists identify failures in record selection, net-change aggregation, and tie-aware ranking.
- Scope: The framework currently focuses on discrete, programmatically specified state transitions, leaving partially observed, continuous, and multimodal settings for future work.
A Task Domains
EvolveScaler’s task prototypes cover domains where information accumulates and mutates over time, with deliberately greater representation for process-intensive settings. The 117 prototypes are organized into 12 themes under five macro-groups.
- Task Domains: The 117 prototypes span 12 themes under five macro-groups and target registers where information accumulates or mutates over time.The distribution is deliberately uneven across domains.
- Task Domains: Process-intensive registers receive more prototypes because records are routinely filed, revised, and retracted.Registers dominated by stable accumulation receive fewer prototypes.
B The Final-Question Bank
The final-question bank organizes EvolveScaler’s query operators into seven families under four groups. Each family is represented by example operator names and a representative question.
- Final-Question Bank: The final-question bank contains 159 operators grouped into seven families under four groups.
- Final-Question Bank: Each family provides example operator names and a representative paraphrased question.Shares are calculated over 3,510 final questions randomly sampled from the training set.
C Checklist-Judge Template
The checklist-judge template evaluates responses against explicit and implicit checklist requirements using binary satisfaction indicators and an overall score. It requires exhaustive, strict, consistent, and objective review before assigning the score.
- Scoring: The judge assigns 1 only when every checklist item is satisfied and otherwise assigns 0; avg@5 averages five responses, while pass@5 records whether any response passes.
- Scoring Principle: The grader must treat every checklist requirement as equally important and assess requirements jointly when determining the final score.Multiple violations should lower the score, while satisfying all requirements should produce a higher score.
- Grading Procedure: The grading procedure analyzes explicit and implicit requirements, checks each requirement individually, and performs completeness, strictness, consistency, and objectivity checks.
- Output Format: The required output has exactly three parts: scoring rationale, requirement-satisfaction status list, and score.The status list uses one binary value per checklist requirement, and the score is an integer from 0 to 10.
- Inputs: The template includes dedicated fields for the scoring checklist and student answer that the judge evaluates.
D Worked Evolution Examples
The worked examples span different registers and seven final-question families, requiring answers to be recovered by replaying event traces and excluding invalid records.
- Three abridged samples cover all seven final-question families across different registers.
- Answers cannot be read from a single log line; they require replaying the trace and dropping invalid records.
- One example applies all seven families to the same trace, showing how one prototype yields distinct reasoning demands.
D.1 Example 1: an editorial pitch group (audit & exclusion)
The editorial-pitch example illustrates that only formally confirmed messages remain valid, while several realistic-looking records are excluded.
- Only formally confirmed messages count in the short-tier editorial-pitch sample.
- Old screenshots, hearsay, mock drafts, tests, and retracted messages are treated as invalid.
- The example is presented as a worked evolution example for an editorial pitch group.
D.2 Example 2: a warehouse shift board, answered seven ways
The warehouse trace keeps one evolving context while seven question families request different views of its replayed state, with validity changes affecting the answers.
- The medium-tier warehouse trace covers four crews, four SKU-locations, and five tracked metrics.
- +165 is the inbound-unit change after Wave 1, based on the valid receipt records.
- +255 is the inbound-unit change for the same question after Wave 2.
- The Wave 3 question returns 2 total valid exceptions, while the hidden replay state records 315 inbound, 175 outbound, 135 relocated, and zero pending discrepancy.
- Seven final questions are applied to the identical warehouse log, changing the requested view rather than the context.
D.3 Example 3: a clinical-trial follow-up log (profile and comparison)
The clinical-trial example uses one follow-up log for profile and comparison questions, with answers determined by valid records after excluding replayed or drilled events.
- The short-tier clinical-trial trace uses training drills and screenshot replays as sources of invalid records.
- The profile question asks for subject Han’s current metrics and latest valid record.
- Han’s valid trajectory has completed items 1, risk flags 1, pending re-review 0, waiting 0 minutes, and latest valid record V2-R1.
- The screenshot-replayed ECG entry V2-R2 is excluded from Han’s answer.
- The comparison question evaluates Chen and Zhou on completed items and risk flags.
- Zhou leads completed items 1 versus 0, while Chen leads risk flags 1 versus 0 and alone has an open pending re-review.