SAGE: Mitigating Long-Horizon Reasoning Biases via Topological Guidance

1 Introduction

• The paper asks how to characterize long-horizon failures under structural complexity and sparse rewards, and how to use that characterization to guide a unified framework.It notes that existing process supervision is expensive to scale, while heuristic approaches leave feasible-trajectory geometry implicit.Source: §1 · p. 2
• SCA attributes exploration bias to feasible support being diluted across unproductive branches, and compounding bias to local deviations accumulating when intermediate rewards are limited.It models reasoning as locally admissible transformations and analyzes how feasible support evolves in expanding spaces.Source: §1 · p. 2
• SAGE injects structural guidance during policy optimization through algebraic sparsification for exploration bias and hyperbolic guidance for compounding bias.The components are intended to reduce spurious branching and provide dense depth-wise signals, respectively.Source: §1 · p. 2
• Across 12 benchmarks and 7 model families, SAGE outperforms competitive baselines and achieves up to 8-fold improvement on the AC problem.The paper describes AC as an open real-world long-horizon reasoning task.Source: §1 · p. 2

2.1 Long-Horizon Reasoning

• Reasoning is modeled as a finite-horizon process over locally admissible symbolic manipulations, with reachable search volume growing as O(Ā^T) while successful trajectories occupy a vanishing fraction as depth increases.Meaningful feedback is often available only at the trajectory end, making credit assignment harder.Source: §2.1 · p. 3

2.2 Biases in Long-Horizon Reasoning

• Exploration bias arises when many locally admissible paths are not structurally extendable, while sparse terminal feedback lets small deviations accumulate into compounding bias.In the AC example, legal moves may lead to paths that preserve or amplify complexity, with failure appearing several steps later.Source: §2.2 · p. 3
• The problem seeks a policy that raises successful-trajectory probability under exponentially growing search and negligible initial success, without dense process labels or ground-truth paths.The setup assumes a pretrained policy and sparse terminal reward.Source: §2.2 · p. 3

3.1 SCA: Symbolic Closure Analysis

• SCA defines feasible trajectories through local admissibility, making the feasible region prefix-closed; in less formal domains, this region can approximate long-horizon extendability.Unlike likelihood-based closure, which retains trajectories that are easy to sample, SCA uses local admissibility; instantiations range from exact symbolic systems to semantic proxies.Source: §3.1 · p. 4, §3.2 · p. 4

3.2 Theoretical Analysis of Biases under SCA

• With sparse terminal-only rewards, locally inadmissible prefixes can persist and dominate trajectories; when successful trajectories are rare under the reference policy, KL-regularized optimization has limited leverage to shift away from it.Local deviations can go uncorrected until terminal evaluation.Source: §3.2 · pp. 4–5
• Because reachable space grows exponentially while admissible and successful trajectories occupy a small fraction, unconstrained rollouts concentrate outside F unless the base policy exponentially favors F.SCA identifies this volume mismatch as the structural origin of exploration bias.Source: §3.2 · p. 4

4.1 From SCA to SAGE

• Algebraic sparsification scores candidate operators by how much unresolved symbolic residual they address, promoting feasible-support concentration without inference-time pruning.The compatibility score is a soft bias, not a replacement for local admissibility.Source: §4.1 · p. 5
• Hyperbolic structural guidance supplies prefix-level feedback before terminal rewards arrive, and SAGE retains a nonzero advantage when terminal rewards are uniformly zero but guidance varies within the rollout group.The paper states that both potentials can be computed without successful trajectories or process labels, with no additional inference-time search or filtering.Source: §4.1 · pp. 5–6

4.2 SAGE

• For non-symbolic tasks, structure-guided sampling applies soft biases over a finite candidate set sampled from the old policy, rather than hard projection onto the feasible region.Normalization is over the candidate set, not the full vocabulary.Source: §4.2 · p. 6
• SAGE augments terminal outcomes with structural guidance and standardizes the resulting trajectory rewards into group-relative advantages, supplying dense feedback in low-resource regimes.The advantage uses the group mean and standard deviation, with ε added to the denominator.Source: §4.2 · p. 6
• When guidance scores feasible trajectories at least Δ above inadmissible ones, increasing λ exponentially suppresses inadmissible mass relative to feasible mass under the trajectory-level guided distribution.The bound scales the original inadmissible-to-feasible mass ratio by e^-λΔ.Source: §4.2 · p. 6

5 Experiment

• The evaluation spans 12 benchmarks, 7 backbone models, and 3 competitive baselines, with comparable rollout budgets, generation lengths, and decoding constraints.The benchmarks cover mathematical, free-form natural, and real-world long-horizon reasoning.Source: §5 · p. 6, §5.1 · p. 7

5.1 Experimental Settings

• The experiments use multiple model backbones, including Qwen, DeepSeek, Kimina-Prover, and Llama-3.3-70B-Instruct variants.The listed Qwen models range from 2B to 35B parameters.Source: §5.1 · pp. 6–7

5.2 Main Results

• Average accuracy rose from 27.24% to 42.11% (2B), 29.01% to 47.95% (9B), and 45.21% to 64.86% (35B).At 35B, SAGE’s 64.86% substantially exceeds Llama-3.3-70B-Instruct’s 38.48% at roughly half the parameters.Source: §5.2 · p. 7
• At 9B, SAGE raised MMLU-Pro average from 37.91% to 39.99%, BBH-H from 44.04% to 45.31%, and ARC-C from 39.73% to 42.04%.The passage also reports gains at 27B and 35B, leading among question-only post-training methods.Source: §5.2 · p. 7

5.3 Long-Horizon Reasoning

• AC Validity gains were +19.2% to +26.0%, while Lean-Verified gains exceeded +13% in every case and Qwen3 approached an 8-fold increase over base.Figure 3 compares SAGE with base models on AC Validity and Lean-Verified Proofs; its labels show absolute percentage-point improvements.Source: §5.3 · pp. 7–8, Figure 3 · p. 8

5.4 Component Ablation Analysis

• Removing either algebraic sparsification or hyperbolic guidance degrades Qwen3.5-9B performance, indicating complementary rather than redundant signals.Replacing hyperbolic guidance with Euclidean distance weakens performance, and shuffling target anchors substantially reduces accuracy and reward density.Source: §5.4 · p. 8
• SAGE remains stronger than GRPO-PRM, especially on AC, supporting target-aligned structural guidance rather than generic dense reward shaping alone.GRPO-PRM improves over GRPO, confirming that dense process feedback helps under sparse rewards.Source: §5.4 · pp. 8–9
• Removing either guidance component on AC reduces AC Validity, AC Path Solving, and Lean-Verified success; full SAGE has the strongest verified performance.The ablation tests when structural validity and long-horizon extension are jointly required.Source: §5.4 · p. 9

• Outcome-based RL improves LLM reasoning, but its effectiveness degrades as reasoning horizons grow and terminal feedback becomes sparse.Process-level verifiers, formal proofs, and step-wise reward models can mitigate this issue, while shifting bottlenecks to annotation cost and verifier coverage.Source: §6 · p. 9
• Exploration methods use entropy regularization, syntactic constraints, empirical consistency, and self-improvement, but typically treat exploration as sampling or reward shaping.The passage identifies how reasoning-space geometry shapes feasible long-horizon trajectories as an open question.Source: §6 · p. 9

7 Conclusion

• SAGE implements SCA’s requirements through algebraic sparsification and hyperbolic structural guidance, and improves over strong post-training baselines across 12 benchmarks and 7 backbones.On the open real-world AC task, the conclusion reports a nearly 8-fold improvement.Source: §7 · p. 9

A Formal Properties of Symbolic Closure Analysis

• SCA defines feasible trajectories through locally admissible transitions, while explicitly separating local rule consistency from ground-truth solution steps and terminal outcome labels.The task-specific symbolic interface specifies admissible operators and a local admissibility predicate.Source: A Formal Properties of Symbolic… · p. 13
• The feasible region is prefix-closed: a trajectory in F has every prefix in F, and a continuation retaining an invalid prefix remains outside F.Local admissibility does not guarantee terminal success; F may contain feasible but unsuccessful trajectories.Source: A Formal Properties of Symbolic… · p. 13
• Feasible-but-failing mass can remain large despite high local admissibility, so step-level validity alone is insufficient for long-horizon success.SAGE therefore provides depth-wise structural guidance to help feasible prefixes extend toward successful trajectories.Source: A Formal Properties of Symbolic… · p. 13
• SAGE uses soft training-time structural potentials to bias rollout sampling and reward shaping, while preserving direct inference with the trained policy.Hard validity masking is not required by the general framework and is not used at inference time.Source: A Formal Properties of Symbolic… · p. 13

B Proof of Theorem 3.5

• When successful trajectories are rare under the reference policy, terminal-only KL-regularized optimization has limited leverage to move probability mass away from reference-likely prefixes.The optimizer remains close to the reference when (e^Rmax/λ − 1)p ≪ 1; in the high-KL or weak-reward regime, the bound has the stated asymptotic scaling.Source: B Proof of Theorem 3.5 · p. 14

C.1 Setup

• The recovery guarantee is conditional on a sparse-residual model and does not guarantee that SAGE globally solves the reasoning task.When the unresolved residual is concentrated on a small number of operator-aligned directions, greedy projection can identify relevant structural directions for a soft compatibility score.Source: §C.1 · p. 15

D Soft Feasible-Support Concentration

• The concentration result is idealized and conditional on the structural potential separating feasible and infeasible trajectories; it is not a universal guarantee.It formalizes suppression of locally inadmissible rollout mass when that separation holds.Source: D Soft Feasible-Support Concentration · p. 16
• Under a relative structural margin, trajectory-level reweighting suppresses infeasible mass exponentially in guidance strength λ.The result is conditional, invariant to additive shifts of the structural potential, and does not require bad trajectories to have negative potential values.Source: D Soft Feasible-Support Concentration · p. 16

E Experimental Details

• Structural modules and probes are used only during post-training; at inference, the trained policy is used directly without evaluating them.Source: E Experimental Details · p. 17

E.1 Task-Specific Instantiation

• For AC, algebraic sparsification scores candidate moves against unresolved algebraic residuals and hyperbolic guidance measures progress toward the trivial presentation; other tasks use learned structural priors rather than exact SCA guarantees.For mathematical reasoning, the prompt-conditioned target anchor is not derived from gold traces, labels, or terminal correctness; for free-form reasoning, learned proxies have empirical fidelity to symbolic guarantees.Source: §E.1 · p. 17

E.2 Training-Time Rollout Sampling and Reward Construction

• During post-training, SAGE samples finite candidate reasoning steps and reweights them using structural potentials and length-normalized policy log-probability; inference uses the trained policy directly.Length normalization prevents the sampler from favoring shorter steps solely because of token-product probability effects.Source: §E.2 · pp. 17–18

E.3 State Encoder and Hyperbolic Embedding

• A frozen reference-model encoder and fixed projection produce Poincaré-ball representations for intermediate states and task anchors, whose distance supplies the hyperbolic guidance signal.The anchor is the canonical trivial presentation for AC and a prompt-conditioned structural anchor for mathematical and free-form tasks.Source: §E.3 · p. 18

E.4 Residual Representation and Algebraic Sparsification

• Algebraic sparsification scores whether a candidate's operator type acts on the unresolved residual, but for non-symbolic tasks this is a learned training-time prior, not a formal admissibility certificate.The residual probe is trained on rollout-local structural pseudo-labels, without gold answers, traces, terminal rewards, or test information.Source: §E.4 · p. 19

E.5 Meaning Clustering and Fixed-Subset Entropy Filtering

• Entropy filtering defines one fixed training subset before methods are trained; controlled comparisons use the same retained prompts, while evaluation uses complete benchmark test sets without filtering.The filter is computed from reference-policy rollouts and is not used at test time.Source: §E.5 · pp. 19–20

F Hyperparameter Ablation

• At Pass@64, SAGE reaches approximately 87% versus approximately 84% for GRPO, and the ablation reports robustness across sampling budgets, temperatures, and KL coefficients.SAGE with βKL = 10−3 outperforms the strongest GRPO variants in the KL sweep.Source: F Hyperparameter Ablation · p. 20, Figure 4 · p. 20

G Process Reward Model Baseline

• PRM-GRPO tests whether SAGE's gains can be explained by adding a learned dense reward during post-training.Its process model is trained on the same training rollouts as the RL baselines; process labels come from outcome-to-prefix credit assignment, with local symbolic-validity targets additionally used for AC.Source: G Process Reward Model Baseline · pp. 20–21
• PRM-GRPO augments GRPO's sparse terminal reward with a learned process reward while keeping rollout budgets, decoding constraints, KL coefficient, and optimization schedule the same.The PRM is used only for post-training reward shaping, not to filter test examples or guide inference.Source: G Process Reward Model Baseline · p. 21

H Additional Results

• Table 5 reports accuracy (%) on free-form natural reasoning benchmarks over 5 random seeds as mean with standard deviation.The best mean is bolded and the second-best underlined; the supplied passage gives no cell values.Source: Table 5 · p. 21

I Computation Resources

• SAGE used 4 NVIDIA A100 GPUs and 2 NVIDIA H200 GPUs, with guidance computation adding training-time overhead but no extra inference-time cost.Compute was mainly used for post-training rollouts, structural-potential computation, policy optimization, ablations, and evaluation.Source: I Computation Resources · p. 21

J Limitations

• Structural priors depend on the task interface: admissibility is exact in symbolic domains but approximate in mathematical and free-form tasks, where theoretical results are conditional.SAGE also adds training-time candidate scoring, residual probing, and hyperbolic-distance computation, while inference uses the trained policy directly.Source: J Limitations · p. 22

K Broader Impact

• The work may improve reliability in extended symbolic or semisymbolic reasoning and reduce dependence on dense human-written process supervision.The authors also identify possible dual-use risks when stronger reasoning is integrated into external tools or autonomous systems.Source: K Broader Impact · p. 22

L Safeguards

• SAGE is a research framework, not a deployment-ready safety mechanism; future extensions with tools or autonomous planning should add safeguards such as sandboxing and misuse monitoring.The released work does not include a new pretrained model, user-facing agent, scraped dataset, or deployment system.Source: L Safeguards · p. 22

M LLM usage

• LLM assistance was limited to polishing author-written text; the authors state that scientific ideas, methods, experiments, results, and conclusions were produced and verified by them.Source: M LLM usage · p. 22

4. Experimental result reproducibility

• The paper describes models, benchmarks, baselines, metrics, settings, and ablations, and provides an anonymized code repository for reproducing main results.It also reports open benchmarks and reproducible construction instructions for the Andrews–Curtis presentations.Source: §4 · p. 24, §5 · p. 25

7. Experiment statistical significance

• The paper reports five-seed mean-standard-deviation results for main benchmark tables, with variability measured across independent training and evaluation runs under the same conditions.Source: §7 · p. 25