Hesitation Has a Geometry: Entropy-Trained Hyperbolic Probes for Sparse Activation Steering

1 INTRODUCTION

• Hyperbolic space is proposed as a fit for mathematical reasoning because its tree-like branching structure embeds with lower distortion than in same-dimensional Euclidean space.The motivation is that reasoning branches at a small number of high-entropy tokens.Source: §1 · p. 1
• HEST uses the model’s next-token entropy both to choose which tokens to edit and to train a probe whose hyperbolic embedding supplies the steering direction.The method requires no correctness labels, contrastive prompts, language-model training, or additional sampled solutions.Source: §1 · pp. 2–3
• A fixed-length geodesic step lowers the Busemann readout by exactly √c t at every state, giving the step length a consistent interpretation.For the radius readout, the drop is exactly t when the step does not pass the origin.Source: §1 · p. 2
• In five of six model–dataset pairs, Busemann HEST improves greedy accuracy on MATH-500 and GSM8K by up to 1.8 points, while an every-token contrastive vector lowers accuracy in all six.HEST steers a median of 0.8 to 3.0% of solution tokens; gains are largest on problems where the greedy solution hesitates often.Source: §1 · p. 3

2 RELATED WORK

• HEST combines a state-dependent steering direction with a sparse gate, both derived from the model’s own entropy rather than labels.Related approaches include nonlinear-classifier gradients, vector fields, rotations, and confidence-gated edits.Source: §2 · p. 3
• Reasoning methods use high-entropy tokens for policy updates, branching, or interventions, while SEAL and FlowSteer classify reasoning steps and steer at step boundaries.HEST also selects by entropy but edits hidden states without keywords or correctness labels.Source: §2 · pp. 3–4
• Latent-space test-time scaling methods use multiple candidates or optimize per problem, whereas HEST edits a single greedy decode and is not evaluated against them as a baseline.Random embedding injection is described as complementary to HEST.Source: §2 · p. 4
• Hyperbolic language-model work includes hyperbolic layers, fine-tuning, and reasoning models, while HEST uses a probe to steer a Euclidean model without fine-tuning.Its probe follows HyperGuide’s ranking and triplet objective but trains on the model’s entropy rather than search-tree solution distances.Source: §2 · p. 4

3 METHOD

• Next-token entropy is computed from decoding logits, defines hesitation points when it exceeds θ, and both trains the probe and gates steering.This signal requires no annotation.Source: §3 · p. 4

3.1 A HYPERBOLIC PROBE TRAINED ON ENTROPY

• The probe maps hidden states into the Poincaré ball, where readouts are low for confident states and high for hesitant ones.The Busemann readout uses a learned ideal point; the radius readout measures distance to the origin.Source: §3.1 · pp. 4–5
• The probe is trained with entropy-based pairwise ranking and a triplet objective that keeps nearby states from one solution close; on held-out problems, the readouts correctly order most entropy pairs.Training uses states and entropies from greedy solutions to 100 problems without correctness labels; held-out pair ordering is 85–90% for Busemann and 82–89% for radius.Source: §3.1 · p. 5

3.2 STEERING ALONG A GEODESIC

• A normalized negative Riemannian gradient sets the steering direction, and the geodesic step has hyperbolic length t; the Busemann readout drops by √c t.The radius readout drops by t if the step does not pass the origin, and its direction differs less between hyperbolic and Euclidean probe spaces than the Busemann direction.Source: §3.2 · p. 5

3.3 MAPPING THE STEP BACK TO THE HIDDEN STATE

• HEST maps the desired probe-space step back to a hidden-state edit using damped Gauss–Newton iterations, with the edit clipped to at most the hidden-state norm.The realized distance and Busemann-readout drop match their targets within 10^-4 on average.Source: §3.3 · pp. 5–6

3.4 DECODING WITH THE ENTROPY GATE

• During greedy decoding, HEST recomputes the uncertain token’s forward pass with its edited state, retains edited keys and values in the cache, and edits no other position.A token is edited only when its entropy exceeds θ.Source: §3 · p. 4, §3.4 · p. 6

3.5 CHOICE OF HYPERPARAMETERS

• The method sets layer, entropy threshold, and step length; thresholds for other models are calibrated to steer about as many tokens per solution as on the development model.The development model uses θ = 1, and its selected step lengths are retained for the other two models without using their test-set labels.Source: §3.5 · p. 6

4.1 SETUP

• The evaluation uses three instruction-tuned models on MATH-500 and GSM8K, plus OlympiadBench for the development model, and compares HEST with greedy decoding and CAA.CAA uses the same probe-training states, layer, and threshold as HEST, but adds a fixed direction at every generated token.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

• On Qwen2.5-Math-1.5B-Instruct, Busemann HEST gains up to 1.80 points on MATH-500 at t = 0.5, but does not improve OlympiadBench at any step length.The evaluation also includes GSM8K; OlympiadBench is used for the development model.Source: Table 1 · p. 6, §4.1 · p. 6, §4.2 · p. 7
• HEST with the Busemann readout improves greedy accuracy in five of six MATH-500 and GSM8K model–dataset pairs, by 0.23 to 1.80 points; CAA lowers accuracy in all six.HEST steers a median of 0.8 to 3.0% of solution tokens.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 ANALYSIS OF THE MAIN RESULTS

• Problems with 0–10 hesitation points make up 71 to 81% of each setting; in five settings, greedy decoding solves 89 to 98% of them and HEST changes accuracy by −0.28 to +0.84 points.HEST breaks almost nothing in this group (b ≤ 0.03).Source: §4.3 · p. 7, Figure 2 · p. 8
• Among problems with at least 25 hesitation points, HEST gains 2.94 to 5.06 points in five of six settings, although this group is small.The figure groups problems by hesitation points in the greedy solution and shows greedy accuracy and HEST’s accuracy change.Source: §4.3 · p. 8, Figure 2 · p. 8
• In one illustrated solution, 3 of 189 greedy tokens exceed θ = 1, while HEST steers 12 of the 583 tokens in its own solution.The figure shows the two continuations from the first high-entropy token.Source: Figure 3 · p. 8

Where to steer

• Editing every token or only step boundaries does not improve MATH-500 accuracy; boundary edits lower it by 0.40 points, while only 8.2% of hesitation points fall at step starts.CAA uses a vector about five times larger than the HEST step and lowers accuracy by 4.20 points; the small all-token step stays close to neutral.Source: §5 · p. 9, Table 3 · p. 9

Which direction

• With HEST’s gate and step size, the Busemann direction fixes the largest fraction of changed wrong solutions: 15% on MATH-500 and 20% on GSM8K.A fixed direction matches HEST on MATH-500 but gains 0.08 points on GSM8K, versus HEST’s 0.53; reverse direction gains 0.40 and 0.15 points over greedy decoding.Source: §5 · p. 9, Table 3 · p. 9

Which geometry

• On Qwen2.5-Math-1.5B-Instruct, the Euclidean probe lowers accuracy by 0.80 points on MATH-500 and 0.15 on GSM8K, while HEST gains 1.80 and 0.53 points, respectively.The Euclidean probe predicts entropy at least as well as the hyperbolic probe and uses the same median edit size.Source: §5 · p. 9, Table 3 · p. 9

6 LIMITATIONS AND CONCLUSION

• Near greedy-decoding ceilings, such as Qwen2.5-Math-7B-Instruct on GSM8K, there is little room for HEST to improve.When solutions hesitate densely, as on many OlympiadBench problems, correct answers are fragile and HEST can break them.Source: §6 · p. 10
• HEST remains untested with other decoding methods, models larger than 8B parameters, and tasks beyond mathematical reasoning.Source: §6 · p. 10
• HEST steers only tokens above an entropy threshold and improves greedy accuracy on mathematical reasoning in most settings, while a fixed vector added at every token lowers it.On Qwen2.5-Math-1.5B-Instruct, the gain disappears with a Euclidean probe that ranks entropy at least as well, suggesting the gain depends on hyperbolic geometry.Source: §6 · p. 10

A COMPARISON WITH RELATED STEERING METHODS

• HEST selects steering positions using entropy alone and trains its direction probe with that same signal, unlike methods using contrastive or labeled data or other position rules.Table 4 compares methods by edit positions and whether they report mathematical-benchmark accuracy.Source: A COMPARISON WITH RELATED STEERING… · p. 14, Table 4 · p. 15

B.1 THE POINCARÉ BALL

• The Poincaré-ball boundary lies at infinite hyperbolic distance from interior points, and its metric preserves angles relative to the Euclidean metric.Source: §B.1 · p. 16

B.2 PROOF OF PART (I)

• The probe’s exponential-map construction follows the geodesic from an embedded state with the specified initial velocity for unit time.The proof identifies the chosen velocity as unit length in the hyperbolic metric before substituting the step into the exponential map.Source: §B.1 · p. 16, §B.2 · p. 16

B.3 PROOF OF PART (II): THE RADIUS READOUT

• A radius-readout descent step lowers the radius by exactly t when 0 < t ≤ r(z), so the step must not exceed the state’s radius.The reported experiment states that none of the evaluated hesitation states violated this condition.Source: §B.3 · p. 17

B.4 PROOF OF PART (III): THE BUSEMANN READOUT

• A Busemann steepest-descent step of length τ lowers its readout by exactly τ; in curvature magnitude c, a step of length t lowers b by √c t.Source: §B.4 · p. 17

B HYPERBOLIC GEOMETRY AND PROOF OF PROPOSITION 1

• In the Poincaré-ball geometry, a step follows a geodesic for a fixed hyperbolic distance, while the probe embedding is trained using hyperbolic distance and a readout defined on the ball.The text defines c > 0 as curvature magnitude, so sectional curvature is −c.Source: B HYPERBOLIC GEOMETRY AND PROOF OF… · p. 16, §B.5 · p. 17

B.5 REMARKS ON THE DIRECTION

• The Busemann descent direction varies by state in the ball, whereas the Euclidean linear counterpart has a constant gradient.The paper locates the geometric effect in the geodesic step and in the probe embedding and readout, not in a changed initial direction from the conformal metric.Source: §B.5 · pp. 17–18

C CLOSED-FORM JACOBIAN OF THE PROBE

• The update uses a closed-form probe Jacobian, and damping scaled to tr(JJ⊤)/n stabilizes the solve when JJ⊤ is ill-conditioned.The text gives a near-boundary example in which the exponential map’s radial derivative is much smaller than its tangential derivative.Source: C CLOSED-FORM JACOBIAN OF THE PROBE · p. 18

D PROBE TRAINING DETAILS

• For each model and dataset, probe data come from one greedy decode of 100 training problems, with recorded hidden states and next-token entropies aligned by generated position.Twenty problems are held out for checkpoint selection; ranking pairs and triplets are sampled for optimization, and the checkpoint is selected by held-out pair accuracy.Source: D PROBE TRAINING DETAILS · pp. 18–19, Table 5 · p. 19

E DECODING PROCEDURE

• HEST rolls back the cache before editing a state whose next-token entropy exceeds the threshold, then recomputes the forward pass with the edit.The edit is written into later-layer keys and values, so later tokens can attend to it; the procedure is causal and does not require future-generation information.Source: E DECODING PROCEDURE · pp. 19–20

F GRADING PROTOCOL

• Answers are extracted from several possible response formats and checked by string, numerical, and symbolic equivalence procedures.The protocol uses the same grading for all arms, and the unsteered model produces any answer restatement used for extraction.Source: F GRADING PROTOCOL · p. 20

G CALIBRATION OF THE GATE THRESHOLD

• Gate thresholds for new models are calibrated without labels by matching their per-solution counts of high-entropy tokens to a reference profile.Calibration uses unlabeled evaluation prompts and greedy solutions, matching counts per solution rather than rates per token because solution lengths differ.Source: G CALIBRATION OF THE GATE THRESHOLD · pp. 20–21, Table 6 · p. 21

H FULL RESULTS

• On Llama-3.1-8B-Instruct MATH-500, four of 17 fixed problems are cases where greedy decoding hits the length limit but HEST finishes correctly.Without those cases, 13 problems are fixed and 11 broken; on Qwen2.5-Math-7B-Instruct MATH-500, one of eight fixed problems is a greedy repetition case.Source: H FULL RESULTS · p. 21
• Across six settings, HEST fixes 88 problems and breaks 63 in the pooled sign test (p = 0.050), while the test treats settings as independent despite shared benchmark problems across models.In one MATH-500 setting, 23 outcomes change and HEST fixes 16 versus breaking 7 (p = 0.09); the per-setting tests have limited power because few outcomes change.Source: H FULL RESULTS · p. 22

I CAA BASELINE

• CAA lowers accuracy relative to greedy decoding in all six settings, including when HEST’s fallback is added.CAA’s vector norm is 0.18–0.33 relative to the median hidden-state norm across the listed model and benchmark pairs; a median single HEST edit on the development model is 0.06.Source: I CAA BASELINE · p. 22

J HESITATION GROUPS AND HESITATION DENSITY

• The densest hesitation group has the largest loss per problem in four reported Busemann settings, although those groups contain only 2 to 60 problems.The groups were chosen after the Table 10 results were known.Source: J HESITATION GROUPS AND HESITATION… · p. 23, Table 11 · p. 24

K HOW MANY TOKENS HEST STEERS

• The median fraction of tokens steered per solution ranges from 0.8 to 3.0% across settings.Capped Qwen2.5-Math-7B-Instruct decodes that repeat account for 62% of steered tokens on MATH-500 and 77% on GSM8K, inflating pooled fractions; the fallback replaces their answers with greedy answers.Source: K HOW MANY TOKENS HEST STEERS · p. 24, Table 12 · p. 24

L DIVERSITY OF THE STEERING DIRECTIONS

• On 400 development-model hesitation states, Busemann HEST edits have median pairwise cosine 0.21, with 32 principal components needed to explain half their variance.Source: L DIVERSITY OF THE STEERING DIRECTIONS · p. 24, Table 13 · p. 25

• The fallback changes every HEST variant’s accuracy by at most 0.40 points, leaving the ablation conclusions unchanged.Source: M ABLATION DETAILS · p. 25

Threshold and layer

• Lowering θ from 1 to ln 2 nearly triples the median steered-token fraction and lowers accuracy by 1.00 point; layers 9 and 25 gain 0.60 and 0.20 points, versus 1.80 at layer 19.Raising θ to ln 4 or ln 8 steers at most 0.5% of tokens, with only 11 and 4 problems changing outcome, respectively.Source: M ABLATION DETAILS · p. 25

Hesitation points and step boundaries

• Only 9.0% of step starts are hesitation points, and only 8.2% of hesitation points are step starts; hesitation points lie a median of 27 tokens after the last step boundary.The two variants therefore edit about as many tokens but at different positions.Source: M ABLATION DETAILS · p. 25

Fixed and broken solutions by direction

• The probe direction has the highest fix rate on both benchmarks; it is correct when the fixed direction is wrong on 39 problems, while the reverse comparison is 45 to 33.Against the fixed direction, the opposite outcome holds on 33 problems; against the reverse direction, it holds on 33.Source: M ABLATION DETAILS · p. 25

Euclidean probe construction

• The Euclidean limit uses a linear Busemann readout with a learned unit vector, whose gradient and probe-space step direction are the same at every state.The Euclidean probes retain the perceptron, entropy loss, training states, held-out problems, optimizer, and training steps; curvature and curvature-dependent operations change.Source: N EUCLIDEAN PROBES · pp. 25–26

Edit-size comparison

• On MATH-500, HEST gains 0.60 to 1.80 points at every tested step length, while the Euclidean Busemann probe loses 0.80 points at matched edit size.The median edits on test decodes are 0.066 for HEST and 0.061 for the Euclidean Busemann probe.Source: N EUCLIDEAN PROBES · pp. 26–27, Table 16 · p. 27

Probe quality and results

• The Euclidean probes order held-out entropy pairs at least as well as hyperbolic probes, so the steering-direction difference is not an entropy-prediction advantage.The Euclidean Busemann probe fixes 10.6% of changed greedy-wrong MATH-500 solutions, versus 15.4% for HEST; it breaks 9.3%, versus HEST’s 4.2%.Source: N EUCLIDEAN PROBES · pp. 26–27
• With the radius readout, the two geometries fix and break similar fractions on GSM8K and reach the same accuracy.For the Euclidean radius readout, the level sets are spheres around the origin.Source: N EUCLIDEAN PROBES · pp. 25, 27

How the states are drawn

• The Busemann visualization preserves each state’s exact readout using its norm and angle to the learned ideal point; the radius view uses norm and angle to the first principal direction.The probe space has 128 dimensions, and the displayed states are samples from 20 validation problems held out from probe training.Source: O PROBE STATES ON REAL DATA · p. 27, Figure 4 · p. 28, Figure 5 · p. 28

Comparison with the schematic

• For Busemann probes, states below 0.1 nats have median angles of 42° to 66°, versus 66° to 87° above the threshold; radius-probe median norms are 0.50 to 0.62 versus 0.65 to 0.84.For 90% of states, the norm lies between 0.39 and 0.82, and almost none are at the origin or near the boundary.Source: O PROBE STATES ON REAL DATA · pp. 27–28
• Across 14 probes, readout–entropy Spearman correlations range from 0.39 to 0.76, and confident and hesitant states overlap.The paper notes that these visualization differences do not affect HEST, which uses the readout gradient at each state and a fixed step length.Source: O PROBE STATES ON REAL DATA · p. 28

• The steered solution places the right angle at S but consistently swaps the labels for RS and RT, so its final ratio and answer remain correct despite those intermediate errors.The problem asks for sin T given sin R = 2/5; the diagram places the right angle at S and labels RT as length 5.Source: P THE EXAMPLE OF SECTION 4.3 IN FULL · p. 28
• Greedy decoding returns 1 and is wrong, while HEST with the Busemann readout returns 21/5 and is marked correct.Source: P THE EXAMPLE OF SECTION 4.3 IN FULL · p. 29
• The solution uses complementary angles to relate sin T to cos R, then evaluates cos R as RT/RS = √21/5.It sets sin R = ST/RS = 2/5 and derives the requested value as sin T = √21/5.Source: P THE EXAMPLE OF SECTION 4.3 IN FULL · pp. 29–30