Source-linked AI summary
MentorPulse: Refreshing Cross-Model Latent Guidance for Long-Form Generation
Ziwu Liu, Guozhong Li, Chen Qiu, Weiyang Kong, Panos Kalnis
TL;DR
Fixed latent guidance can become stale as long-form outputs grow, reducing the value of mentor–student generation. MentorPulse refreshes a capped slot memory from newly generated prefixes without resetting the student’s KV cache, and across thirteen datasets it closes 52.2% of the mentor–student gap, with gains concentrated on long outputs.
Problem
Existing cross-model latent guidance is computed before decoding and kept fixed, leaving its usefulness during long-form generation uncertain as the generated prefix changes.
Method
MentorPulse uses capped slot memory, incremental mentor prefill, gated cross-attention, and Windowed Refresh Training to update prefix-conditioned guidance without changing the student’s decoding cache.
Results
52.2% of the mentor–student gap is closed across thirteen datasets on the main pair, with the largest gains on long outputs and best performance across all eleven pairs from three model families.
Takeaways & Limitations
Guidance freshness is a design axis for long-form generation, with refresh intervals selected through measured cost and pair applicability screened by V64.
Takeaways & Limitations
The evidence covers eleven pairs, V64 is comparable only under one training recipe, and the check screens pairs rather than replacing evaluation.
Abstract
from arXiv · showhide
Cross-model latent guidance lets a frozen large mentor encode an input once and a frozen small student generate from the resulting signal. Existing methods keep this signal fixed, assuming it stays useful as the output grows; we show this fails in long-form generation. On multi-turn instruction following, static guidance pushes a 4B student's constraint satisfaction 2.5 points below its no-guidance baseline; a training-free refresh every 16 tokens changes only the memory content and restores a 2.0-point gain over that baseline. We propose MentorPulse to keep guidance fresh at practical cost: it compresses mentor states into a capped slot memory, incrementally processes newly generated tokens, and updates the memory that the student reads through gated cross-attention without resetting the student's KV cache. Windowed Refresh Training exposes the bridge to prefix-conditioned memory. Across thirteen datasets, MentorPulse closes 52.2% of the mentor-student gap on macro average, outperforming C2C, T2T, and equal-budget LoRA, with the largest gains on long outputs. It performs best on all eleven mentor-student pairs from three model families, with margins that narrow as the capability gap grows, and a lightweight read-pattern check predicts the gain before deployment. Measured costs identify refresh intervals that dominate text guidance on long outputs.
1 Introduction
Existing latent guidance is fixed before decoding, but it becomes stale during long-form generation and can fall below the student-only baseline. MentorPulse refreshes a capped memory from the generated prefix, recovering long-output performance at practical cost.
- Static prompt-only guidance can become stale as the generated prefix grows, pushing the student below its no-guidance baseline.
- Long-form generation offers the largest decoding savings from mentor–student separation, while fixed guidance is most likely to become outdated.
- Existing LLM-to-SLM, C2C, and Latent-Guided Reasoning interfaces keep transferred guidance fixed during student decoding.
- Training-free refresh replaces memory from the growing prefix while leaving the student’s weights and KV cache unchanged, restoring gains on affected long-output tasks.
- MentorPulse uses capped slot memory, incremental mentor processing, and gated cross-attention without resetting the student’s decoding history.
- 52.2% of the mentor–student gap is closed across thirteen datasets on the main pair, with advantage concentrated on long outputs.
- MentorPulse demonstrates generality across eleven pairs and three model families while using read-distribution variance V64 to screen pairs before deployment.
2 Related Work
Prior cross-model interfaces transfer latent or persistent information between large and small models, but MentorPulse refreshes prefix-conditioned latent state during decoding while preserving the student’s autoregressive process.
- LLM-to-SLM transfers a one-time prompt encoding, C2C fuses mentor and student KV caches, and Latent-Guided Reasoning transfers compact strategy vectors.
- Unlike distillation, LoRA, routing, cascading, and speculative decoding, MentorPulse supplies intermittent prefix-conditioned hidden state while the student generates.
3 Guidance Staleness in Long-Form Generation
The diagnostic separates guidance staleness from student capability limits: static memory loses alignment with the growing context, while prefix-conditioned refresh repairs affected long-output behavior but is too expensive without incremental processing.
- 3.1 Setup: The Qwen3.5-27B mentor prefills the input, compresses states into memory Z, and supplies them to the Qwen3.5-4B student through a gated cross-attention bridge.
- 3.1 Setup: Figure 2 measures constraint satisfaction by generation segment and probe accuracy for upcoming-window properties, with StaleGap(t) marking their difference.
- 3.1 Setup: The diagnostic contrasts Multi-IF, QMSum, and HelloBench long-form positives with BigCodeBench and ARC-Challenge controls.
- 3.2 Staleness and the Oracle: On Multi-IF, stale probe accuracy falls from 0.79 at t=0 to 0.47 at t=512, while fresh accuracy remains between 0.76 and 0.80.
- 3.2 Staleness and the Oracle: OracleSwap@16 changes only memory content and raises Multi-IF from 56.2 for OneShot to 60.7, or 2.0 above the S-only floor of 58.7.
- 3.2 Staleness and the Oracle: Refresh also improves QMSum and HelloBench by +2.7 and +3.1 over OneShot, while mismatched memory provides no repair.
- 3.2 Staleness and the Oracle: Refresh cannot fix capability failures, and short outputs ending before the first swap show no refresh effect.
- 3.2 Staleness and the Oracle: Full-prefix re-prefilling makes the oracle’s cost grow with generation, motivating a mechanism that processes only new tokens and preserves decoding history.
4 MentorPulse
MentorPulse maintains prefix-conditioned guidance through a capped slot memory and incremental mentor refreshes, while gated cross-attention and windowed training let the student consume changing memories without cache resets.
- MentorPulse keeps both backbones frozen, trains a pair-specific bridge, and replaces a side memory tensor at interval R rather than changing the student’s KV cache.
- 4.1 Slot Memory Constructor: The Slot Memory Constructor retains the newest Wtail=32 states, summarizes earlier positions into 32-token segments, and caps prompt segments at P=128.
- 4.1 Slot Memory Constructor: Figure 3 traces mentor prefill, slot construction, gated student reads, and every-R-token in-place refresh; purple components are trained and saturation denotes freshness.
- 4.1 Slot Memory Constructor: The slot design keeps transfer size bounded independently of input length and requires no cross-model token alignment.
- 4.2 Gated Cross-Attention Bridge: Gated cross-attention reads the current slots as keys and values, and replacing the side tensor leaves student self-attention states valid.
- 4.3 Versioned Incremental Refresh: Incremental refresh sends only newly generated tokens to the mentor, so each token is prefilled once and total mentor compute is independent of R.
- 4.3 Versioned Incremental Refresh: Refresh re-segments the previous tail, appends a fresh tail, and preserves existing summary slots while distinguishing prompt and generated-prefix summaries.
- 4.3 Versioned Incremental Refresh: At R=16, the amortized update is 10.6 MB with all 64 mentor layers and 2.7 MB with the top 16 layers.
5 Experimental Evaluation
Across thirteen datasets and eleven mentor–student pairs, MentorPulse consistently improves latent-guidance recovery, especially for long outputs, while refresh interval controls the quality–cost trade-off.
- Overall performance: 52.2% macro-average recovery on the main pair exceeds T2T, LoRA, and C2C.The corresponding baselines achieve 26.9%, 17.5%, and 10.9%, respectively.
- Length dependence: On long outputs, C2C falls 1.7 to 5.1 points below the student-only baseline on five open-ended datasets.The sign-flip pattern is concentrated in longer generations, whereas latent methods are closer on short selection tasks.
- Ablations: Removing refresh drops mean recovery on two long-output sets from 59.4% to −24.4%.Short-answer controls do not move because they finish before the first refresh boundary.
- Ablations: Windowed Refresh Training adds +8.3 and +3.1 recovery points beyond fresh memory alone on the reported long-output ablations.Fresh memory without refresh training contributes only +1.2 and +2.2 over the static row.
- Generality: MentorPulse performs best on all eleven mentor–student pairs across three model families.Margins are smallest on the most capability-separated pairs, and recovery falls monotonically as the parameter ratio grows within mentor groups.
- Applicability: Recovery tracks read-distribution variance V64 with Spearman ρ = 0.78 and p = 0.004.The indicator is intended to screen mentor–student pairs before deployment, but it is comparable only under one training recipe and does not replace evaluation.
- Refresh interval and cost: R=16 lies among the cost-efficient settings {8, 16, 32} while avoiding the synchronization overhead of more frequent refreshes.R≤8 is nearly flat in quality, whereas recovery declines from R=16 and is 12 to 28 points lower by R=64.
6 Conclusion
One-shot latent guidance becomes unreliable during long-form generation, while prefix-conditioned refresh repairs the degradation without changing the student. MentorPulse implements this repair efficiently and shows gains across datasets and model pairs, especially for long outputs.
- Static guidance falls below the student-only floor on long outputs, whereas a zero-training prefix-conditioned swap repairs it without changing the student’s weights or cache.
- MentorPulse keeps guidance fresh through versioned slot memory, incremental mentor prefill, and Windowed Refresh Training.
- Across thirteen datasets and eleven pairs, refresh remains the main source of long-output gains after ablation, while measured costs constrain the useful interval range.
- The gains require long outputs; short outputs never reach a refresh boundary.
A.1 Dataset Registry
The registry defines thirteen evaluation datasets, their metrics, subset sizes, and length profiles, with explicit protocols for short-answer and long-generation tasks. It also records contamination controls, tokenizer-based median lengths, seeds, and significance conventions.
- Thirteen evaluation datasets are identified by fixed letter codes, metrics, evaluated subset sizes, and long-generation membership.
- LiveCodeBench is restricted to post-July-2025 problems as a contamination guard, while QuALITY uses the public development split.
- Length profiles use median input and output token counts, with outputs measured from M-only generations and tokenizer variation below 4%.
- Short-answer datasets have 5–7-token medians, so MP@16 never reaches a refresh boundary and operates as a static bridge.
- Results use greedy no-think decoding, fixed judge and evaluator protocols, three seeds, and per-dataset generation caps.
- Reported numbers average at least three seeds, and differences below twice the pooled standard deviation are labeled comparable rather than better or worse.
B.1 Experimental Conditions
The experiments compare MentorPulse with static and refreshed text or latent guidance, LoRA, and C2C under matched training and evaluation conditions. Protocols distinguish representation format from refresh schedule and define cost-quality dominance without weighted scoring.
- Main-text comparisons include S-only, M-only, T2T, C2C, LoRA, and MP@16, while diagnosis and ablation sections add dedicated controls.
- Table 6 compares interfaces by frozen mentors, explicitly capped transferred state counts, and prefix-conditioned refresh, with speculative decoding treated as a discussion-only contrast.
- rT2T refreshes text guidance under the same mentor-conditioning path as MentorPulse, separating refresh schedule from representation format.
- The refreshed text baseline replaces prior hints, strips delimiters before scoring, and receives a best-effort 64- or 128-token hint budget.
- C2C fuses projected mentor input KV states into the student cache before decoding and matches the bridge’s trainable parameter count and learning-rate search.
- LoRA uses rank 64 with matched trainable parameters, shared data, training budget, learning-rate search, serving stack, decoding, and scoring scripts.
D.1 Training Data
MentorPulse training combines filtered long targets with short-answer data, prefix-conditioned refresh objectives, and a frozen student–mentor bridge. Its capped memory and hot-swapped refresh path preserve the student’s decoding cache while supporting incremental mentor processing.
- The training store contains 20,408 filtered long targets and 31,500 short-answer rows, totaling 51,908 examples, with mentor states stored in about 1.9 TB of bf16 data.
- Windowed Refresh Training samples refresh points and computes loss only on the following R-token window, using the prefix to form memory rather than supervision.
- An optional second training round uses bridged-student prefixes to reduce train–test prefix mismatch.
- The student and mentor backbones, embeddings, and heads remain frozen; training updates the bridge projector, corrections, embeddings, gates, and cross-attention blocks.
- Mentor states are compressed into position-aware segment slots plus 32 raw recent-state slots under a capped memory layout.
- Gated cross-attention reads the memory as a side tensor, initially behaving as an identity map when gates are zero.
- Refreshes process only newly generated tokens, replace the memory in place, and leave the student’s KV cache and generated text intact.
- Capability-limited pairs can remain near the student’s difficulty floor, where comparison methods recover no meaningful fraction of the gap.
F.3 Refresh-Interval Sweep
The refresh-interval sweep evaluates how inference-time refresh frequency affects MentorPulse quality, recovery, and cost across model pairs. It also compares latent refresh with text refresh and examines diagnostic indicators and accumulation behavior.
- Refresh-Interval Sweep: The sweep reuses one bridge checkpoint trained at R=16 while changing only the inference-time refresh interval.Smooth curves around R=16 show that no per-interval retraining was triggered.
- Refresh-Interval Sweep: Short-answer sets trigger no refresh and equal T2T cell by cell, as does the R→∞ limit.These identities reflect outputs ending before the first refresh boundary or the one-shot limit.
- Text-Guidance Comparison: 52.2% macro recovery at R=16 exceeds T2T’s 26.9% and rT2T@16’s 33.7%.The rT2T comparison uses the per-task best L′g at R=16.
- Significance: MP@16 exceeds rT2T@16 significantly only on B (+2.8) and G (+3.6); other triggering-task differences are within noise.The corresponding rT2T-over-T2T differences are significant only on B (+3.6) and G (+0.9).
- Read-Distribution Diagnostic V64: V64 correlates with MP@16 macro recovery across eleven pairs, with Spearman ρ = 0.78 (p = 0.004, n = 11).The metric is computed from early bridge-attention read distributions and is intended as a predeployment fit check.
G.1 Suite Registry and Inclusion Criteria
The registered diagnostic suite is designed to isolate guidance staleness from capability limits using disjoint, staleness-sensitive tasks and controlled refresh interventions. Supporting protocols define the condition matrix, stale-signal probes, and transfer-cost accounting.
- Suite Registry and Inclusion Criteria: The diagnostic suite is disjoint from both the thirteen benchmarks and the fuse_v3 training set, and is selected for expected staleness sensitivity.The suite targets tasks whose requirements shift as generation proceeds.
- Main Condition Matrix: OracleSwap@16 changes only memory freshness by re-prefilling the input and generated prefix, while leaving student weights, generated text, and KV cache untouched.The intervention performs no training and uses full mentor recomputation.
- Main Condition Matrix: On the three staleness-sensitive tasks, OracleSwap@16 improves over OneShot by +4.5, +2.7, and +3.1, while clearing the D-IF floor by +2.0.The sign flip is definite only on D-IF; the other two tasks move in the same direction below the significance threshold.
- StaleGap Probe Protocol: The stale probe’s readability falls from 0.79 to 0.47 over 512 generated tokens, while fresh readability remains within 0.76–0.80.The probes predict verifiable properties of the next fixed-length window from frozen mentor states independently of the bridge.
- Per-Request Cost Formulas: Incremental prefill processes each generated token once, so mentor compute is independent of R; decreasing R mainly increases synchronization.Text guidance instead repeatedly incurs mentor decoding, the most expensive per-token pipeline operation.
- Byte Accounting: The transfer accounting assumes a 150 Mbps cross-site link, 18 ms RTT, blocking synchronization, and k=16 transmitted layers.A measured refresh at R=16 takes 165 ms end to end under this deployment setup.
H.5 Cost Support Numbers
The cost analysis attributes MentorPulse’s long-output advantage to lower synchronization overhead than text refresh, while showing that network, synchronization, and mentor-billing assumptions materially affect the conclusion.
- Long-output requests: 52.9–105.1 s versus 7.3 s: GovReport accumulates substantially more blocking under rT2T than MP@16.The comparison concerns accumulated request-level blocking time.
- Cost structure: rT2T’s divergence comes from per-refresh decoding tax, whereas MentorPulse’s comes from transferring memory bytes more often.Text refresh requires the mentor to generate at every refresh; MentorPulse primarily incurs transfer overhead.
- Accounting sensitivity: On a same-site gigabit LAN, synchronization becomes negligible and the cost analysis extrapolates MentorPulse’s favorable region left to approximately R ≈2.Asynchronous prefetch reduces synchronization further, but the main figure uses conservative blocking accounting.
- Accounting sensitivity: Resident dedicated-mentor billing reverses the cost conclusion because idle time between refreshes exceeds the compute time billed.The paper reports this reversal explicitly rather than treating it as a minor accounting adjustment.
I.1 Per-Dataset Generality
MentorPulse generalizes across mentor–student pairs and datasets, with static guidance harming long-output columns while refreshed guidance preserves recovery; its benefits diminish near capability limits.
- Per-dataset coverage: Eleven pairs × thirteen datasets: Figure 9 reports per-dataset recovery for four methods across all mentor groups.The figure uses the recovery color scale from Figure 5 and marks two capability-limited B1 cells.
- Static guidance: All eleven pairs show harm from static C2C guidance in sign-flip columns G, H, K, L, and M.The repeated red columns indicate that this long-output harm is not confined to one mentor–student pair.
- MentorPulse recovery: MentorPulse keeps those columns favorable across mentor groups, though recovery saturation fades as the capacity gap grows.The MP panel remains purple across the sign-flip columns, with weaker saturation within each mentor group as the gap increases.
- Freshness diagnostic: 0.79 to 0.47: stale probe readability decays by depth 0.32, while fresh readability remains between 0.76 and 0.80.The widening StaleGap indicates that guidance drifts from the task described by the unchanged input.
- Trigger outlook: A learned refresh trigger remains future work; the reported trigger evidence is only a preliminary stage-2 pilot using offline refresh-value labels.The pilot compares cross-entropy under stale and fresh memory versions and is not the fixed-interval protocol used in the experiments.
- Applicability boundary: Refresh repairs staleness, not missing capability: in the capability-limited coding case, OneShot and MP@16 fail identically.Fresh hints restate a plan the student cannot execute better, matching the reported capability boundaries.