Compress What You See, Not What You Say: Anchored Context Distillation for Latent-Observation Software Engineering Agents

• Tool observations average 69% of CoderForge trajectory tokens, while discarded details may need to be retrieved again before an agent can act.Source: §1 · p. 1
• LOHA compresses older tool observations into soft tokens while keeping agent turns and recent observations in text for exact reference.This separates compact historical context from direct access to recent content.Source: §1 · p. 2
• ACD distills full-text base-model predictions into the latent view and anchors full-text predictions to the same base model, limiting behavioral drift.The experiments also report that better read-back alone does not ensure better agent behavior, and identify self-anchoring as an effective balance.Source: §1 · pp. 2–3
• K=3 reduces context per call by 43% for Qwen3-4B and 57% for SWE-Master-4B-RL, with resolve rates of 12.1% and 21.8% versus 14.5% and 27.5% for their uncompressed bases.Source: §1 · p. 2

2 APPROACH

• The approach combines a context layout for compressed observations and uncompressed text with training that enables latent reading while limiting behavioral change.Source: §2 · p. 3

2.1 PROBLEM SETUP

• An issue-resolution trajectory includes the system prompt, issue description, assistant turns, and tool observations; the LCLM pipeline encodes text as shorter sequences of continuous embeddings.At the default 16× ratio, n text tokens become approximately n/16 soft tokens, processed alongside ordinary text tokens.Source: §2.1 · p. 3

2.2 LATENT OBSERVATIONS, HARD ACTIONS

• LOHA keeps the system prompt, issue, and assistant turns in text, compresses older observation bodies, and retains the K most recent observations as raw text.Observations remain in their tool-response envelopes; compression does not guarantee verbatim reconstruction, and exact older content can be retrieved with existing tools.Source: §2.2 · p. 3

2.3 ANCHORED CONTEXT DISTILLATION

• ACD trains the base agent to read latent observations while regularizing ordinary-text predictions, using a low-rank reading patch initialized from the LCLM-to-base weight difference.Its two objectives match latent-view predictions to the base model’s full-text predictions and anchor full-text predictions to that same model.Source: §2.3 · pp. 3–4

2.3.1 THE STUDENT

• The student uses a frozen LCLM encoder, trainable adapter and low-rank decoder patch, and trainable normalization gains, with remaining decoder parameters frozen.At rank 64, the patch has 132M parameters and the adapter has 9.2M.Source: §2.3.1 · p. 4

2.3.2 TWO VIEWS, TWO TERMS

• The latent-view distillation and full-text anchor use the original base model as teacher, respectively supervising latent reading and preserving its full-text behavior.The anchor directly regularizes ordinary-text inputs, which use the same adapted decoder as latent inputs.Source: §2.3.2 · p. 5
• Read-back questions supervise literal content recovery, while the divergence estimate uses top-64 token probabilities plus a bucket for remaining probability mass.The coarsened divergence does not constrain the tail’s internal distribution and is a lower bound on full-vocabulary KL before numerical safeguards.Source: §2.3.2 · pp. 4–5

2.4 TRAINING AND INFERENCE

• Training is offline on recorded trajectory prefixes; at inference, a selected K mixes older latent observations with recent raw text, and context savings are smaller than the encoded-body reduction.Encoded observation bodies use approximately one soft token per 16 source tokens; context-level accounting includes text that remains uncompressed.Source: §2.4 · pp. 5–6

3.1 FULL-WINDOW RESULTS

• Qwen3 decoder input falls from 1.65M to 1.51M tokens per trajectory, while SWE-Master falls from 4.40M to 1.81M; their representation ratios are 2.24× and 2.35×.Qwen3 makes 60.9 versus 37.9 calls, while SWE-Master makes 81.5 versus 85.0 calls.Source: §3.1 · pp. 6–7
• At K=3, context per call falls 43% for Qwen3 and 57% for SWE-Master, while resolve rates are 12.1% and 21.8% versus 14.5% and 27.5% for their uncompressed bases.The respective full-window results are reported for Qwen3-4B-Instruct and SWE-Master-4B-RL.Source: §3.1 · p. 6, Table 1 · p. 7, Table 2 · p. 7

3.3 ABLATION STUDIES

• Raising K from 3 to 8 increases single-run resolve rates from 13.6% to 14.4% for Qwen3 and from 22.2% to 23.0% for SWE-Master, with higher decoder compute.Estimated decoder compute rises from 1.67 to 2.10 PF for Qwen3 and from 3.74 to 8.37 PF for SWE-Master; the trend is not strictly monotonic, as Qwen3 resolves 76 tasks at K=4.Source: §3.3 · p. 7

3.2 PERFORMANCE UNDER CONTEXT LIMITS

• At 32K, Hard-Last-3 resolves 42 instances versus 22 for adapted full text, 26 for masking, and 31 for the base; masking overflows less often, 23 versus 36 trajectories.The 32K comparison is on a 199-instance Qwen3 subset; paired McNemar tests support gains over adapted full text and masking (p=0.0002 and p=0.0025).Source: §3.2 · p. 7, Figure 3 · p. 8

3.3 ABLATION STUDIES

• The LCLM decoder exactly recites 20.4%, 15.6%, and 15.1% of 358 later-quoted literals at 4×, 8×, and 16× compression, respectively.The reported low fidelity even at 4× supports retaining recent source text for exact tool arguments.Source: §3.3 · p. 8
• With rank 256, the 30B model exactly recites 35/96 literals versus 26/96 for the self-anchored student, yet resolves 4/20 tasks versus 10/20 under either context policy.Teacher diagnostics also support retaining the base anchor while allowing a different distillation teacher.Source: §3.3 · p. 8
• On Qwen3, rank-64 initialization retains 17% of LCLM–base weight-difference energy and recovers 79% of the LCLM decoder’s structured read-back score, but still degrades agent behavior before ACD training.Source: §3.3 · p. 8

3.4 SERVING EFFICIENCY

• At concurrency 16 on one GPU, LOHA completes 63.4 instances per GPU-hour versus 33.4 for adapted full text and 58.1 for the base agent.Median call latency falls from 5.07 to 1.71 seconds; 90th-percentile latency falls from 45.6 to 7.17 seconds. Masking reaches 64.7 instances per GPU-hour.Source: Figure 3 · p. 8, §3.4 · p. 9

4 RELATED WORK

• Prior agent-history methods mask, summarize, fold, or selectively prune observations, while LOHA keeps older observations latent and recent content verbatim.The passage also contrasts prompt compressors and code-structure methods; masking serves as a direct comparison using the same recency window without latent history.Source: §4 · p. 9
• Soft-token compression and latent-memory systems address compressed representations or experience reuse, whereas this setting requires repeated in-trajectory tool use, including exact-text actions.Source: §4 · p. 9
• ACD combines context distillation with behavioral anchoring: the base model supplies both latent-view targets and the full-text reference, while a low-rank patch enables adaptation.The paper’s ablations distinguish the effects of the distillation teacher and behavioral anchor.Source: §4 · p. 9

5 CONCLUSION AND FUTURE WORKS

• LOHA and ACD combine compressed older observations with text for recent observations and agent turns, while ACD anchors full-text behavior to the base policy.The conclusion treats context representation and policy adaptation as a joint design problem.Source: §5 · p. 9
• Future work proposes allocating observation fidelity by anticipated tool use, constraining tool-use or trajectory outcomes, and training on the adapted agent’s own trajectories.These directions would connect compression more closely to how agents use context.Source: §5 · p. 9

A EXPERIMENTAL PROTOCOL

• The main comparison evaluates LOHA + ACD and compression baselines on Qwen3-4B-Instruct-2507 and SWE-Master-4B-RL using a common scaffold.Additional results and implementation and training details are reported in Appendices B and D.Source: A EXPERIMENTAL PROTOCOL · p. 13

A.1 BENCHMARK, AGENTS, AND DECODING

• Evaluation uses 499 SWE-bench Verified instances, excludes one image-failing case, and applies bounded runs with model-family-specific seed counts.The main evaluation has a 7,200-second wall-clock limit, with a 6,600-second alarm ending trajectories.Source: §A.1 · p. 13

A.2 CONDITIONS AND BASELINES

• Each family’s base anchors its ACD student, while compression baselines share a K=3 recent window and 128-character threshold; LOHA re-encodes observations per call.Comparisons distinguish adaptation from changes in context representation.Source: §A.2 · p. 13

A.3 SCAFFOLD CHANGES AND GRADING

• The scaffold adds bounded editor fallbacks for unmatched replacements, and grading reads complete official-harness logs to avoid truncation-related misclassification.The fallback normalizes whitespace before attempting a uniquely matched edit-distance replacement.Source: §A.3 · pp. 13–14

A.4 METRICS

• Resolve rate is the fraction of 499 instances passing the benchmark grader; behavioral measures include patches, loop and cap terminations, edits, calls, tool validity, and editor fallbacks.The ablation tables abbreviate resolved count as “res.”Source: §A.4 · p. 14
• Cost measures include decoder prompt length, trajectory decoder and encoder tokens, context representation ratio, estimated compute, and serving throughput and latency.Reading probes test exact recitation of literal fields and structured extraction from compressed tool outputs.Source: §A.4 · p. 14

B ADDITIONAL RESULTS

• The additional-results section supplements the two main result tables with paired comparisons, context-limit experiments, ablations, and serving measurements, usually on Qwen3.Source: B ADDITIONAL RESULTS · p. 14

B.1 FULL-WINDOW COMPARISONS

• Hard-Last-3 has paired p-values of 0.049 and 0.32 against the Qwen3 base; LOHA’s 16× observation ratio exceeds pruning baselines’ 1.02×–2.9×.LongCodeZip retains nearly every chunk during its coarse ranking stage on these observations.Source: §B.1 · p. 14
• SWE-Pruner and LongCodeZip each resolve 120 instances, compared with LOHA’s mean of 109, while their contexts remain 34–49K tokens per call versus LOHA’s 22.2K.Self-Prune resolves 109 instances; paired resolve-rate gaps against the SWE-Master base are 3.2 points for adapted full text and 5.7 for Hard-Last-3.Source: §B.1 · p. 15

B.2 CONTEXT-LIMIT EXPERIMENTS

• The experiment tests four Qwen3 conditions on the same 199-instance subset at 65,536- and 32,768-token limits, reserving 4,096 tokens for generation.Each capped condition runs once; the 262K column is the subset mean from full-window seeds.Source: §B.2 · p. 15
• At 32K, Hard-Last-3 resolves 24 instances only it solves versus four only by adapted full text (p=0.0002), and 21 versus five against masking (p=0.0025).Against the base, the paired counts are 19 versus eight (p=0.052).Source: §B.2 · p. 15
• Among trajectories stopped at the cap, four base runs, zero adapted full-text runs, one Hard-Last-3 run, and two masking runs still yield passing patches.Masking overflows less often than Hard-Last-3 but resolves fewer tasks.Source: §B.2 · p. 15

B.3 RECENCY ABLATION

• The diagnostic uses 20 instances drawn from those solved by one Qwen3 base run, while teacher and objective ablations use students trained on the first 3,514 trajectories.The anchor-ablation table varies the distillation source and patch rank, with plain-text and Hard-Last-3 evaluations.Source: §B.3 · p. 15, Table 6 · p. 16
• At rank 64, Hard-Last-3 and full text each resolve 11 tasks; at rank 256, each resolves 10, while fully latent observations yield only one or two resolved tasks.Hard-Last-3 produces 16 non-empty patches at either rank, compared with 14 and 15 for full text.Source: §B.3 · p. 16
• Fully latent agents average 18–26 edits and 94–103 model calls per trajectory, with eight or nine runs reaching the iteration cap; recent text addresses this failure mode on the diagnostic subset.Their trajectories include reconstructed edit strings, inaccurate test commands, and repeated views returning compressed observations.Source: §B.3 · p. 16

B.4 ANCHOR AND OBJECTIVE ABLATIONS

• Self-anchoring resolves 28 tasks in full text and 29 under Hard-Last-3, versus 15 and 14 for the 30B reference and none for the external SWE-Master reference.Increasing patch rank from 64 to 256 does not repair the 30B-reference behavior.Source: §B.4 · p. 16
• A rank-256 student with the 30B reference recites 35 of 96 literals exactly versus 26 for a self-anchored student, despite worse benchmark behavior.The self-anchored subset-trained model resolves 28 versus 31.0 tasks in full text and 29 versus 27.5 under Hard-Last-3 compared with the full-data model.Source: §B.4 · p. 16
• Six distillation variants resolve 8–12 tasks in full text and 6–10 under Hard-Last-3; reversing KL directions or using a larger teacher gives no consistent advantage.Fully latent evaluation resolves only 1–3 tasks in every variant, while retaining the base-model anchor avoids widespread collapse in most variants.Source: §B.4 · pp. 16–17, Table 7 · p. 17

B.5 SERVING MEASUREMENTS AND ENCODER COST

• At 16 concurrent trajectories, the serving cache accommodates about 20 Hard-Last-3 trajectories versus 11 adapted full-text trajectories; the latter exceeds capacity and has a long latency tail.The workload uses one RTX PRO 6000 GPU with 96 GB and throughput is measured through the 84th completion.Source: Table 8 · p. 17, §B.5 · p. 17

B.4 ANCHOR AND OBJECTIVE ABLATIONS

• Cross-entropy improves extraction from 17 to 24 fields and exact recitation from 22 to 28, but edits no files across eight plain-text tasks and emits 15 malformed tool calls.The complete objective solves five of those eight tasks.Source: §B.4 · p. 17

B.5 SERVING MEASUREMENTS AND ENCODER COST

• Hard-Last-3 prefix-cache shares are 92.5% for Qwen3 and 85.0% for SWE-Master, while the recency-ablation shares vary with workload.The 20-instance recency ablation reports 99.6–99.7% reuse for fully latent histories, 98.4–99.2% for full text, and 92.1–95.4% for Hard-Last-3.Source: §B.5 · p. 17
• Per-observation caching reduces encoder input from 1.87M to 25.7K tokens per trajectory, but estimated LOHA compute remains 2.12 PF versus 1.82 PF for the Qwen3 base.The reported throughput was measured using the current implementation, which re-encodes compressed observations at each call.Source: §B.5 · pp. 17–18

C.1 SAMPLING REPEATS

• The recency-window sweep uses one run per condition, with one final result per instance.The per-seed results table reports means and standard deviations computed from the main-evaluation seeds; additional runs provide reference results for the sweeps.Source: §C.2 · p. 18, §C.1 · p. 18

C.2 RECENCY-WINDOW SWEEPS

• Increasing K from 3 to 8 raises resolved counts from 68 to 72 for Qwen3 and from 111 to 115 for SWE-Master, while decoder compute rises from 1.67 to 2.10 PF and from 3.74 to 8.37 PF per trajectory.Larger windows generally improve task performance, but Qwen3 resolves 76 tasks at K=4, so results are not monotonic in K.Source: §C.2 · p. 18

D.1 CONTEXT RENDERING

• Compressed observation bodies are replaced by soft tokens inside their original tool-response envelopes, while the surrounding envelope and assistant turns remain text.At inference, Hard-Last-K selects older bodies for compression; bodies shorter than 128 characters remain text, and observation bodies change representation as they leave the window.Source: §D.1 · p. 19, §D.8 · p. 20

D.2 STUDENT PARAMETERS

• The student freezes the encoder and trains a 9.2M-parameter adapter plus a low-rank patch and decoder normalization gains; at rank 64, total trainable parameters are 141.5M.The rank-64 patch has 132M parameters, while all other decoder parameters remain frozen.Source: §D.2 · p. 19

D.3 LOSS AGGREGATION

• The loss combines latent-view distillation over assistant and read-back tokens with full-text anchoring over assistant tokens, weighted by λ=1 in reported runs.Targets align by assistant-token identity across views because rendered positions differ.Source: §D.3 · p. 19

D.4 CACHED TEACHER DISTRIBUTIONS

• Teacher top-64 token distributions are scored on full-text trajectories and cached before training; both KL terms use the cached tokens plus a tail bucket.The default uses forward KL, and coarsening supplies lower bounds on full-vocabulary divergences by the log-sum inequality.Source: §D.4 · pp. 19–20

D.5 READ-BACK SUPERVISION

• Latent-view read-back questions, asked once per four observations, target literal content such as code ranges, search lines, test failures, tracebacks, and exit codes.Answers are filtered to be absent from hard-token context and account for 3% of supervised positions; they receive next-token cross-entropy.Source: §D.5 · p. 20

D.6 DATA AND OPTIMIZATION

• Training uses the shortest successful CoderForge trajectory per task: 35,140 trajectories, 1.29B raw tokens, and 0.31B teacher-scored assistant positions.The models train for one epoch with AdamW, and cached targets avoid a teacher forward pass during optimization.Source: §D.6 · p. 20

D.7 BEHAVIORAL PROBES; D.8 INFERENCE AND CACHING

• Behavioral probes pair literal-recall and structured-extraction tests with a plain-text tool-calling probe, while Hard-Last-K causes decoder prefix-cache recomputation when observations leave the text window.The serving implementation re-encodes compressed observations at each call; encoder and decoder cache effects are measured separately.Source: §D.7 · p. 20, §D.8 · p. 21

E DIAGNOSTIC MEASUREMENTS; E.1 MEASUREMENT SETUP

• Figure 4 measures decoder projection changes and rank-64 energy, adapter-output norms and embedding locations, and per-layer latent-versus-text probe accuracy and norms.The setup compares the released 16× LCLM decoder with Qwen3-4B-Instruct-2507, using 48 real-observation windows at 4×, 8×, and 16× compression.Source: Figure 4 · p. 21, §E.1 · pp. 21–22

E.2 WEIGHT CHANGES AND READING INITIALIZATION

• Rank-64 initialization recovers 79% of the LCLM decoder’s structured read-back score but only 27% of its exact literal-recitation rate, and the untrained patch degrades agent behavior.Projection changes span 6–14% of Frobenius norm, while rank-64 truncation retains 17% of total change energy and a median 22% per matrix.Source: §E.2 · p. 22

E.3 LATENT AND TEXT REPRESENTATIONS

• The latent-versus-text probe reaches 98–100% accuracy through layer 35 even after norms converge; adapter-output norms are 26×, 62×, and 111× vocabulary norms at 4×, 8×, and 16× compression.Unit-normalized latent and vocabulary embeddings occupy separate t-SNE regions.Source: §E.3 · p. 22

E.4 GRADIENT AND TEACHER DIAGNOSTICS

• A direction mask retains 2,559 of 16,128 patch directions but recovers only eight of 48 structured fields, versus seven for the unpatched base and 22 for the full patch.Latent-read-back and full-text next-token gradients have correlations of 0.93 and 0.95 in two measurements; teacher disagreement concentrates on 15.5% of high-entropy positions.Source: §E.4 · p. 22

E.5 CACHED-DISTRIBUTION APPROXIMATION

• At M=32, cached forward-KL estimates are within 0.1% of exact KL, while reverse-KL estimates underestimate by 3.5–4.2%.The reverse-estimate gradient cosine is 0.99, and its gap approximately halves as M doubles.Source: §E.5 · p. 22

E.6 EXACT RECITATION

• Exact literal-recitation accuracy falls from 20.4% at 4× compression to 15.6% at 8× and 15.1% at 16×; no literal-type and length bucket with at least 10 examples reaches 90%.Composed pytest identifiers are never exact, and literals of at least 96 characters are almost never exact.Source: §E.6 · p. 22

F.1 SOFTWARE-ENGINEERING AGENTS; F.2 CONTEXT MANAGEMENT AND PRUNING

• LOHA retains the tool-interaction pattern while changing how accumulated observations are represented and how agents are trained to use them.For coding observations, it represents older bodies with soft tokens and keeps a recent text window, unlike methods that select or prune text spans.Source: §F.1 · p. 23, §F.2 · p. 23

F.3 PROMPT AND CODE COMPRESSION; F.4 LATENT COMPRESSION AND MEMORY

• LOHA separates recent exact text from compressed history for coding tasks where later edits or commands may need previously observed identifiers or code fragments.It builds on latent-compression interfaces and applies them to multi-turn tool observations within the current trajectory rather than experience reuse or coordination.Source: §F.3 · p. 23, §F.4 · p. 23

F.5 DISTILLATION AND BEHAVIORAL ANCHORING

• ACD pairs latent-view distillation with a distributional anchor toward the agent’s own base policy, giving the two objectives distinct roles in teaching compression and regularizing behavior.The method trains a low-rank patch initialized from the compressor’s weight difference.Source: §F.5 · p. 23