TextualRL: Textual Reinforcement Learning for Context Management

• Prior work uses execution feedback to adapt textual artifacts, leaving open how trajectories should be organized before feedback becomes a shared context update.Source: §1 · pp. 1–2
• Shared contexts create two challenges: uniform outcomes remove within-task contrast, and task-specific modifications may not apply across tasks.Source: §1 · p. 2
• TextualRL optimizes a frozen agent’s shared textual context using grouped rollout rewards, reviewed bounded edits, and held-out validation.Source: §1 · p. 2
• Across six benchmarks, TextualRL improves test accuracy over the no-skill baseline by 24.4 percentage points for GPT-5.5 and 15.3 for Qwen3.8-27B.Source: §1 · p. 2
• Groupwise Policy Critique contrasts mixed outcomes within tasks and uses cross-task evidence for preservation or repair analysis when outcomes are uniform.Source: §1 · p. 2

• Prompt optimization ranges from gradient-guided or demonstration-based instruction search to compiling modular pipelines and jointly optimizing instructions and demonstrations.Source: §2 · p. 3
• Context and skill methods derive reusable guidance from execution traces, failed trajectories, or successful and failed minibatches, often selecting bounded updates with validation.Source: §2 · p. 3
• TextualRL couples comparative same-task rollout critique with cross-task evidence review and held-out validation for updates to a shared frozen-agent context.Source: §2 · p. 3

3.1 Context-Space Policy Optimization

• TextualRL keeps target parameters fixed and optimizes shared instructions and procedures to maximize expected task reward.Each training task contributes an indexed group of K rollouts and their rewards; retaining same-task rollouts preserves comparisons for proposing updates.Source: §3.1 · p. 3

3.2 Groupwise Policy Critique

• Critique routes mixed-outcome tasks to within-task contrasts, all-success tasks to cross-task preservation, and all-failure tasks to cross-task repair hypotheses.Homogeneous routes require shared behavior or failure mechanisms across at least two distinct tasks; unsupported behaviors yield no edit, and failure alone does not verify a repair.Source: §3.2 · p. 4

3.3 Cross-Group Policy Update

• Cross-Group Policy Update reviews proposals against independently pooled evidence, forms a bounded textual patch, and selects the candidate using held-out validation rewards.The review can narrow or revise edits based on supporting, limiting, or contradictory observations; rejected candidates leave the context and reference score unchanged, while feedback informs later critique calls.Source: §3.3 · p. 5

4.1 Experimental Setups

• The evaluation covers six task benchmarks, seven baselines, and five frozen targets, with a separate context optimized for each target–benchmark pair.Unless otherwise specified, GPT-5.5 is the optimizer; the default setup uses four epochs and K = 4 rollouts per task.Source: §4.1 · pp. 6–7

4.3 Ablations

• Six-benchmark averages fall by 1.5 points without outcome routing, 2.5 with K = 1, 1.2 with intra-group-only review, 1.7 without no-edit cards, and 2.7 without cross-group review.Full TextualRL scores highest on every benchmark in both ablation groups.Source: Table 2 · p. 7, §4.3 · pp. 7–8

4.2 Main Results

• TextualRL has the highest six-benchmark average for all five targets, gaining 11.4–24.4 points over Base Model and 0.9–3.2 over SkillOpt.It outperforms all baselines in 29 of 30 target–benchmark settings and ties SkillOpt on GPT-5.5 with ALFWorld.Source: §4.2 · p. 7

4.4 Additional Experiments

• Across three Qwen3.8-27B runs, TextualRL averages 70.68% versus SkillOpt’s 67.43%, with a 3.18–3.36-point advantage across runs.TextualRL’s mean is higher on all six benchmarks; the largest mean gain is 8.27 points on LiveMath.Source: Table 3 · p. 8, §4.4 · p. 8
• On Qwen3.8-27B, increasing K from 1 to 4 raises the overall score from 68.4% to 71.0%; increasing K to 8 adds only 0.1 points.K = 4 captures most observed improvement with half the rollouts per task required by K = 8; Spreadsheet and DocVQA decrease at K = 8.Source: Table 4 · p. 8, §4.4 · p. 8
• GPT-5.5 harness averages are 80.1% with Codex and 78.6% with Claude Code; transferred skills slightly raise overall scores on both tested Qwen models without re-optimization.TextualRL is highest in nine of ten harness–benchmark settings, while transfer gains are not uniform across benchmarks.Source: Table 5 · p. 8, §4.4 · pp. 8–9, Table 6 · p. 9
• With Qwen3.8-27B as both optimizer and target, the overall score is 69.7%, versus 71.0% with GPT-5.5 optimizing; the Qwen optimizer gains 14.1 points over Base Model.GPT-5.5 scores higher on all six benchmarks in this comparison.Source: Table 7 · p. 9, §4.4 · p. 9
• Most test improvement occurs early, but validation gains do not consistently carry over: OfficeQA reaches 47.1% test after epoch one despite 62.5% validation.By epoch two, DocVQA rises from 88.5% to 93.9% and LiveMath from 34.4% to 54.4%; later test scores fluctuate.Source: Figure 3 · p. 9, §4.4 · pp. 9–10

• Across five target models and six benchmarks, TextualRL achieves the highest six-benchmark average for every target.Ablations and additional experiments support the effectiveness of both modules and the consistency of the improvements.Source: §5 · p. 10

• Corrective and preservation proposals are merged in separate branches, then combined with corrective edits prioritized unless they conflict with a supported success pattern.Source: A Method Implementation Details · p. 13
• Edits are applied deterministically while protected scaffold regions are preserved; dependency filtering removes affected edits that rely on unavailable information.The filter uses explicit patterns with limited negation handling and retains other valid edits in the same proposal.Source: A Method Implementation Details · p. 13
• Evidence cards are retained only when task and rollout references resolve to displayed trajectories and their supporting quotes appear in those traces.Repeated cards are consolidated; if no valid cards remain, ordinary proposal merging is used.Source: A Method Implementation Details · p. 13
• The step buffer screens exact or near-equivalent repetitions of rejected edit combinations; it blocks an edit family only after three rejected candidates and no accepted candidate.Source: A Method Implementation Details · p. 13

B.1 Data Splits and Metrics

• Benchmark scores are binary accuracy or success rate in percentages, and overall performance is the unweighted mean across six benchmarks.SpreadsheetBench requires every test case for a task to pass; DocVQA counts ANLS ≥0.999 as correct.Source: §B.1 · p. 13, Table 8 · p. 14
• For ALFWorld, the released validation manifest contains 18 environments, and update acceptance and checkpoint selection use those rather than the 140-environment pool described in the paper.Source: §B.1 · p. 13

B.2 Critique and Merge Batches

• Critique minibatches count distinct task blocks, while merge minibatches count proposal bundles; homogeneous critique routes group task blocks in batches of eight.Mixed tasks are analyzed separately with all unique sibling trajectories; a one-task homogeneous route is skipped.Source: §B.2 · p. 14
• Outcome routing uses the original K rollout outcomes before deduplication, which consolidates duplicate trajectories only within the same task and outcome label.Deduplication does not change K, the outcome category, or the number of distinct tasks supporting a proposal.Source: §B.2 · p. 14
• Hierarchical merging processes up to eight proposal bundles per call; singleton bundles pass to the next level without a merge call.Proposal-bundle batch sizes are separate from the edit budget for the candidate context.Source: §B.2 · p. 14
• The no-outcome-routing ablation pools trajectories into analyst minibatches of eight and removes the mixed-outcome prompt’s same-task and both-outcomes requirements.In the intra-group-cards-only ablation, each merged edit can use cards only from critique groups that contributed its source proposals.Source: §B.2 · p. 14

B.3 Generation and Evaluation

• Each target–benchmark pair uses K = 4 same-context training rollouts per task, and validation selects the context for test evaluation.Rollouts start with fresh histories; Spreadsheet uses independent workbook copies, OfficeQA new conversations, and ALFWorld reset environments.Source: §B.3 · p. 14

B.4 Checkpoint Selection

• The test context is the run’s highest-validation-score checkpoint, tracked separately from the context after the final update.Training uses temperature 0.7; validation and test inference use temperature 0, with top-p at the backend default.Source: §B.3 · p. 14, §B.4 · p. 15

B.5 Token Accounting

• Table 9’s token totals cover saved optimization steps, including steps after the selected checkpoint, but exclude subsequent unseen evaluation and other unrecorded usage.Inference text volume sums input and visible output text per task, counting repeated history at each decision and treating each token equally regardless of price or compute cost.Source: §B.5 · p. 15
• TextualRL records 2.05–3.24 times SkillOpt’s iteration tokens on five benchmarks, while Spreadsheet uses 24.3% fewer iteration tokens and 54.2% fewer inference text tokens.Inference-token increases on the other benchmarks are 6.6–86.6%.Source: §B.5 · p. 15, Table 9 · p. 15

Evidence Review Audit

• Review narrows or revises 37 of 436 edits with review metadata, and two assessments label 87.58% of sampled references as supporting a rule.The audit examines 113 edits and 153 references; 134 of 153 card statements are labeled as supporting the rule.Source: §B.6 · pp. 15–16, Table 10 · p. 16

Effect of reviewed updates

• Across three selected updates, review improves already beneficial proposals while clarifying rule conditions and exceptions in SearchQA, OfficeQA, and DocVQA.Examples include protecting proper names, restricting the ambiguous-constant exception, and specifying when document-specific Roman numerals apply.Source: §B.6 · p. 16, Figure 4 · p. 17

Examples of reviewed edits

• The reviewed SearchQA guidance preserves the full title “Dalai Lama” as a proper name, while the OfficeQA constant rule removes the ambiguous-constant fallback.The latter retains only a qualified exception for well-established, unambiguous external constants.Source: §B.6 · p. 17

Effect of reviewed updates

• 82.71 (+0.64 pp) and 47.09 (+1.16 pp) are reported for reviewed “New skill” variants relative to their respective no-review variants.The paired no-review scores are 82.07 and 45.93; the comparisons use unseen-test performance.Source: §B.6 · pp. 16–17, Table 11 · p. 16

C.1 Training Dynamics

• SearchQA and Spreadsheet gain most of their test performance in epoch 1, while ALFWorld’s test score rises across all four epochs.SearchQA moves from 73.6% to 81.4% in epoch 1 and finishes at 82.5%; Spreadsheet moves from 43.6% to 58.9% and finishes at 60.7%; ALFWorld rises from 50.8% to 84.3%, with validation reaching 100% at epoch 3. Epoch 0 is the base model, and gain timing and magnitude vary by benchmark.Source: §C.1 · p. 18, Figure 6 · p. 19

C.2 Optimized Contexts

• Optimized-context examples specify concrete operations and applicability conditions, such as verifying source values before arithmetic or checking inventory after a failed action.Other examples address static spreadsheet values and when a theorem supports a claimed bound.Source: §C.2 · p. 19

C.3 Optimizer Prompts

• The optimizer prompts use the editable context, edit budget, history, and task trajectories to produce a proposed patch and cross-group evidence cards.The method-specific prompt blocks define outcome-specific critique tasks and share an evidence-card instruction; the final block guides proposal review.Source: §C.3 · p. 20, §C.3.1 · p. 20, §C.3.2 · p. 21

C.3.1 Outcome Analysis

• Mixed-outcome critique seeks a within-task behavioral divergence; homogeneous groups instead require cross-task patterns before proposing preservation rules or cautious repair hypotheses.All-success proposals require an inference-time behavior shared by at least two tasks; all-failure repairs require the same observable failure mechanism in at least two tasks.Source: §C.3.1 · p. 20

C.3.2 Evidence and Review

• Cross-Group Policy Update reviews proposed edits against semantically relevant evidence from mixed, successful, and failed groups, retaining, narrowing, revising, or flagging proposals as insufficiently evidenced.Cards record salient observations even when no edit is proposed; correlated sibling rollouts are not independent task votes, and review metadata stays out of skill content.Source: §C.3.2 · p. 21

C.4 Complete Update Example

• In the SearchQA example, validation accepts the complete patch after accuracy rises from 69.0% to 77.0%.Three trajectories return Henrik Ibsen for the same declarative clue, while one returns True; the reviewed rule reserves Boolean answers for explicit verification questions.Source: §C.4 · p. 21, Figure 7 · p. 22