On the Fragility of Tasks under Composition: Implicit Budgeting in Language Models

• Across open models and Claude Opus 5, composing previously solved tasks causes accuracy drops larger than isolated-error rates predict, in math and language.Source: §1 · p. 2
• Reasoning tokens per task decrease approximately as a power law with composition size, despite no explicit per-task token constraint.Source: §1 · p. 2
• Fragility-aware batching reduces token use by 43% on average with no performance drop, grouping non-fragile tasks larger and fragile tasks smaller.Source: §1 · pp. 2–3
• Tasks with longer isolated reasoning are more fragile; this affected set is stable within each model but differs between models.Source: §1 · pp. 2–3

• The controlled setup varies the number of unchanged constituent tasks and measures each task’s accuracy relative to its isolated performance.Source: §2 · p. 3
• A composition combines independent tasks from one dataset in a single prompt, then splits the response into answers and grades each for correctness.Source: §2 · p. 3
• Accuracy estimates use five independent trials, with negative accuracy-drop values indicating degradation.Source: §2 · p. 3
• The eight datasets include six established benchmarks plus 1,000 symbolically verified randomized integrals and OneWordQA factual questions requiring minimal reasoning.Source: §2 · p. 3
• Evaluations cover seven open-weight Qwen3 and DeepSeek-R1-distilled models and Claude Opus 5, with medium reasoning effort.Source: §2 · p. 3

3 Composing tasks causes performance degradation

• The evaluation selects tasks solved in three independent isolated attempts, then concatenates them in compositions of 3, 6, 10, or 20 tasks.Accuracy at k = 1 comes from runs independent of those used to select tasks.Source: §3 · pp. 3–4
• Composing confidently solved tasks substantially reduces accuracy across models on GPQA-Diamond and most tested datasets, with little-reasoning tasks as exceptions.The figure evaluates tasks solved correctly in three out of three isolated attempts.Source: Figure 2 · p. 4, §3 · p. 4

3.1 Degradation as a misallocation of the implicit thinking budget

• Per-task thinking tokens generally shrink as composition size grows, while context limits are rarely reached in most datasets.Figure 4 measures the ratio of thinking tokens spent on a task in composition versus isolation; under 1% of responses hit the context limit in most datasets.Source: Figure 3 · p. 4, §3.1 · p. 5, Figure 4 · p. 5
• Tasks with longer isolated reasoning traces degrade more when token use falls, whereas shorter-trace tasks are largely unaffected and are termed fragile when they break.The token ratio follows an approximately shared power law in composition size, so longer-trace tasks do not systematically lose a larger fraction of their reasoning.Source: §3.1 · pp. 5–6, Figure 5 · p. 6

3.2 Fragile tasks are stable within a model

• The same tasks tend to remain fragile across composition sizes within a model, but different models largely disagree about which tasks degrade.Mean cross-model correlation at a fixed composition size is 0.24 ± 0.03, while within-model correlation across composition sizes is 0.81 ± 0.04.Source: §3.2 · p. 6

4 Exploiting implicit budgeting for efficient inference

• Fragility-aware batching assigns robust tasks to larger compositions and fragile tasks to smaller ones, using composition size to control implicit budgeting.The batching objective trades expected token savings against expected accuracy drop under a dataset-level degradation budget.Source: §4 · p. 7, §4.1 · p. 8

4.1 Fragility-aware batching

• 43% fewer tokens on average with accuracy constant (+0.6 pp) across models and datasets except AIME, where fragility-aware batching leaves nearly every task isolated.On OneWordQA and StrategyQA, it saves 68–74% of output tokens without accuracy loss; on five other datasets, it preserves accuracy while saving roughly 32%.Source: Table 1 · p. 7, §4.1 · p. 8

4.2 Predicting fragility on held-out tasks

• A linear predictor of task fragility limits accuracy loss to at most 0.5 pp on every dataset while saving 31.0% of tokens on average.Its simple embedding-based predictions work best on short, plain task descriptions such as Integrals, GSM8K, and OneWordQA; fragility is thus partly predictable from task properties.Source: §4.2 · pp. 8–9
• On Claude Opus 5, measured-fragility batching loses 0.4 pp while saving 24% of generated tokens, compared with random batching’s 0.9 pp loss and 18% savings.The linear predictor transfers less well in this setting, saving almost no tokens while losing 2.0 pp.Source: §4.2 · p. 9

5 Effects of post-training on compositional degradation

• At similar token ratios, RLVR loses substantially less accuracy than SFT and DPO, whose accuracy losses are similar.The figure plots token ratio against composition size on the left and accuracy drop against token ratio on the right.Source: §5 · p. 9, Figure 6 · p. 9
• Token counts shrink at similar rates across SFT, DPO, and RLVR, while RLVR’s fragile-task set overlaps less with those of SFT and DPO.Fragile-task degradation correlations are ρ = 0.69 ± 0.23 for RLVR versus SFT and ρ = 0.89 ± 0.26 for RLVR versus DPO.Source: §5 · p. 9
• RLVR post-training mitigates accuracy loss from implicit budgeting but does not eliminate the similar budget misallocation across stages.Source: §5 · p. 9

6 Related work

• Studies report composition-related accuracy losses, but their causes remain incompletely understood; related work also examines independent-error explanations and sensitivity to added context.The cited literature spans multi-task requirements, reasoning tasks, answer grading, and context changes.Source: §6 · p. 10
• Prior batching studies report efficiency gains, reasoning-token reductions, and accuracy gains in some settings.Source: §6 · p. 10
• Budget misallocation connects deeper-task sensitivity, self-compression, composition errors beyond independent error rates, and token savings from batching.The paper identifies reasoning-trace length in isolation as a key variable linking these findings and informing interventions.Source: §6 · p. 10

7 Discussion and conclusions

• Across the studied models, composition reduces per-task reasoning and disproportionately harms fragile tasks; fragility-based batching saves about 43% of tokens on average while preserving accuracy.The approach does not require access to model internal states and also applies to Claude Opus 5.Source: §7 · p. 10
• The experiments compose independent tasks of the same type, while prior work on irrelevant context, conversation history, or larger tasks suggests the budgeting effect may extend beyond that setting.The cited prior work reports shorter reasoning traces in those contexts [Rodionov et al., 2026].Source: §7 · p. 11
• For agent workflows, accumulated context may misbudget high-effort steps and possibly propagate errors; the paper suggests lean contexts and fresh-context subagents as interventions.Source: §7 · p. 11
• The authors suggest that post-training on composed tasks could help models allocate reasoning more effectively across a shared prompt.Source: §7 · p. 11
• The study uses independent, same-dataset compositions, and detailed reasoning-length analysis is restricted to open-weight models.The authors do not yet know what drives the reduction and identify mixed-domain, dependent, and proprietary-model settings as future work.Source: §7 · p. 11

• Accuracy drops after composition across datasets except OneWordQA and StrategyQA; R1-distilled models degrade most, with strong variation between datasets.Source: Figure 7 · p. 16
• Fewer than 1% of responses reach the context limit on most datasets, even at the largest evaluated composition size.Source: §A.1.1 · p. 16
• Parsing failures are rare except on AIME, where most occur because the model returns too few answers, itself a composition failure.Source: §A.1.1 · p. 16

• Per-task token ratios decrease approximately linearly with composition size on log-log axes for nearly all models and datasets, without a consistent reduction by isolated trace-length bin.Tasks are binned by isolated trace length as Low (≤20%), Medium (20–80%), and High (> 80%); the exception is R1 Qwen 7B on MATH-500, where the Low-bin ratio rises above 1 and the all-task fit is flat (m = −0.04, R2 = −0.43).Source: §A.1.2 · pp. 16–17
• Median power-law slopes m were −0.46 (Qwen3 4B Thinking), −0.56 (Qwen3 8B), −0.55 (Qwen3 14B), and −0.15 (Qwen3.8 27B).Source: Figure 8 · p. 17, Figure 9 · p. 18, Figure 10 · p. 18
• Median power-law slopes m were −0.68 (R1 Llama 8B), −0.62 (R1 Qwen 7B), and −0.61 (R1 Qwen3 8B).Source: Figure 12 · p. 19, Figure 13 · p. 20, Figure 14 · p. 20

• The analysis extends to six datasets spanning math, logical inference, and linguistics, selected because running it on all datasets would be prohibitively expensive.Source: §A.1.3 · p. 21
• Accuracy drops as relative thinking-token reduction grows, and tasks with long isolated traces usually degrade more than tasks with short traces across the analyzed models.Source: §A.1.3 · p. 21
• Abruptly forcing composition-sized budgets in isolation causes larger degradation than planned token reduction, but accuracy still falls monotonically as budgets shrink, especially for higher-reasoning tasks.Source: §A.1.3 · p. 25

A.1.4 Budgeting is slightly position dependent

• Models use fewer thinking tokens toward the end of compositions, and accuracy drops grow toward later positions in most models; Qwen3.8 27B shows neither positional bias nor changing accuracy.Qwen3 8B and its R1-distilled version share an architecture, but the distilled model reduces later-position tokens more, suggesting post-training influences budgeting; the authors note that high-effort tasks at the end may affect performance.Source: Figure 22 · p. 25, §A.1.4 · p. 26

A.1.5 Correlating accuracy drop across models and composition sizes

• Accuracy-drop correlations are substantially higher across composition sizes within the same model than across models, especially across model families, indicating that fragile tasks vary by model.The correlations are disattenuated Pearson correlations computed over tasks; the method uses independent passes to avoid shared isolated-accuracy measurement noise.Source: Figure 23 · p. 26, §A.2.3 · p. 29

A.1.6 Fragility-aware batching

• Across added datasets, fragility-aware batching keeps accuracy close to isolated baselines while using fewer tokens; batching raises accuracy by up to 6.7 points, while random batching drops R1 Qwen3 8B OneWordQA from 81.5% to 28.6%.On Integrals, the predictor-based variant outperforms the measured-fragility variant because limited observations make estimates noisy, while integral structure helps the predictor. Random batching saves tokens but loses accuracy, especially on reasoning-heavy datasets; the linear predictor partially recovers the fragility-aware gain. AIME is omitted because fragility-aware batching falls back to k = 1 throughout.Source: Table 2 · p. 27, §A.1.6 · p. 28

A.2.1 Measuring per-task thinking length

• Task-level thinking length is measured by assigning tokens after each task tag to that task until the next tag; models name every task in 99.5% of tagged traces, except GPQA-Diamond at 92.9%.A dummy task is appended because the final task’s trace has no following tag to mark its end reliably.Source: §A.2.1 · p. 28

A.2.4 Implementation details of the fragility-aware batching algorithm

• Fragility-aware batching selects composition sizes by cross-fitting estimated accuracy drops, then forms groups whose size does not exceed the smallest assigned upper bound among their tasks.The method chooses the cheapest exchange rate whose accuracy minus one standard error stays above the isolated-accuracy budget; forming groups can leave the realized mean size slightly below assigned sizes, making token savings conservative.Source: §A.2.4 · pp. 29–30

A.2.5 Predicting fragility from the task

• The fragility predictor uses task embeddings and a linear head whose predicted probability is constrained to decrease with composition size, with nested cross-fitting keeping evaluation data out of training.The monotonicity constraint matches the observed pattern that performance does not improve as composition size increases.Source: §A.2.5 · p. 31

A.2.2 Sampling parameters and context length

• The evaluation uses six public benchmarks—MMLU-Redux, GPQA-Diamond, GSM8K, MATH-500, AIME, and StrategyQA—and fewer than 1% of responses reach the context limit in most datasets.Open-weight models use temperature 0.6 and top-p 0.95, with no top-k cutoff or repetition penalty; Claude Opus 5 uses medium reasoning effort and a 200,000-token context window.Source: §A.2.2 · p. 29, §A.2.6 · p. 32

A.2.6 Further details on datasets

• GPQA options are shuffled once with a fixed seed, while multiple-choice prompts require answers as letters in question order.The prompt asks for a comma-separated list of exactly k letters.Source: §A.2.6 · pp. 33–34
• GPQA uses the MMLU-Redux prompt with an added instruction to re-check the final answer line after long reasoning.The added sentence addresses answer formatting, not the questions themselves.Source: §A.2.6 · p. 34
• OneWordQA and related short-answer prompts require numbered answer blocks; StrategyQA restricts answers to yes or no, while accepting true or false and ignoring case and punctuation.The math datasets use the same numbered-block contract, with their own problem and answer-format instructions.Source: §A.2.6 · pp. 34–35
• MATH-500 answers are wrapped in \boxed{...} because answers may contain spaces or commas and problems may specify their own required form.Reference answers follow that requested form, such as a common fraction.Source: §A.2.6 · p. 35
• For k ≤3, all three mathematical prompt examples show every answer-block line without an ellipsis..Source: §A.2.6 · p. 36