PaperlayerView exact revision on arXiv

Source-linked AI summary

Instruction Capacity: Behavior and Degradation of Language Models under Controlled Load

Hennadii Kutomanov

arXiv:2610.36095v1cs.LG

TL;DR

The paper addresses the lack of a common, uncertainty-bearing measure of how much instruction load language models can handle. It measures load with controlled word-replacement rules and models the probability of correct replacements across load factors. GPT-4o mini measurements yield regular laws for each load factor and for answer collapse.

  • Problem

    There is no quantity that expresses a model’s instruction-load limit in comparable load units with uncertainty.

    Source: §1 · p. 1, §2.2 · p. 13

  • Method

    The instrument applies independent single-token word-replacement rules and treats each occurrence as a Bernoulli trial with a computable reference outcome and known answer position and rule number.

    Source: §3 · p. 14, §7 · p. 65

  • Results

    The GPT-4o mini measurement found regular laws for each load factor and for answer collapse, with text length and word position acting as interacting constraints.

    Source: §7 · p. 65

  • Takeaways & Limitations

    The instruction capacity is one characteristic derived from degradation laws that provide a fuller description of model behavior under load.

    Source: §6.1 · p. 59

    Real instructions can jointly constrain the same text, so requirements are not independent in that setting.

    Source: §2.2 · p. 8

Abstract

from arXiv · show

The quality with which language models follow instructions falls as the load grows, yet there is still no quantity that expresses the limit of a model in units of load with an uncertainty. We propose an instrument in which the load consists of rules that replace single-token words in a text without meaning: such an instruction is atomic, the rules are independent by the construction of the task, and the outcome of every application of a rule is determined by a computable reference answer. Every occurrence of a word to be replaced becomes a Bernoulli trial with a known position in the answer and a known rule number. The statistical model first measures the shape of the dependence of the probability of a right replacement on the text length, the word position, the replacement density, the number of rules and the rule number, then describes it by smooth laws and restores a reference level that does not depend on the choice of the coding. The model yields the instruction capacity - the number of rules held at a given share of right replacements - with an uncertainty. A measurement of GPT-4o mini (2958 requests, 642,430 trials) gave regular laws for every load factor and for the collapse of the answer; the instruction capacity was about 37-38 rules at a share of right replacements of 0.95, and about 18-19 rules at the full replacement density. The code of the instrument, the measurement data and the passport are open, and every number of the passport is recomputed from the published points. The passport of a model serves as the calibration of a future meter of the instruction load of prompts.

1 Introduction

The paper addresses the lack of a comparable, uncertainty-bearing measure of instruction load by defining a controlled instrument and statistical model. It introduces instruction capacity and a GPT-4o mini passport covering degradation laws and answer collapse.

1 Introduction

  • Existing measures confound instruction count with task composition and difficulty, while no common quantity expresses a model’s load limit with uncertainty.A useful load unit requires atomic, independent instructions with computable outcomes and observable applications.Source: §1 · p. 1
  • The instrument uses single-token word-replacement rules as load units, with computable outcomes and each occurrence treated as a Bernoulli trial.The instrument checks answers line by line and distinguishes three kinds of failure.Source: §1 · p. 2
  • The statistical model measures load-axis effects by bins, fits smooth laws, and corrects approximation shifts to restore a coding-independent reference level.The correction addresses dependence on the choice of reference bins.Source: §1 · p. 2
  • Instruction capacity is the number of rules held at a specified share of right replacements, defined through an inverse problem with uncertainty.Its uncertainty uses the delta method and accounts for request-difficulty spread.Source: §1 · p. 2
  • The GPT-4o mini passport reports laws for text length, word position, replacement density, rule-list size and rule number, plus answer-collapse behavior and several capacity measures.The listed capacities are instruction, fatigue and guaranteed instruction capacities.Source: §1 · p. 2
  • The public instrument recomputes every passport number from the published points.The work presents degradation laws and their parameters and uncertainties for comparison across models and generations.Source: §1 · p. 2

2 Related Work

This section reviews prior work on language-model performance under instruction and context load, evaluating its heterogeneous measurement approaches against the need for a controlled instruction-load test. It identifies the need for a comparable capacity estimate with uncertainty, since existing approaches may confound instruction count with task difficulty or measure a different phenomenon.

2.1 History of research on language model behavior under controlled load

  • Compliance also degrades as requirements accumulate across dialogue turns and as answers grow longer; studies additionally find that long-context performance depends on where relevant information appears.Some long-answer studies also report answer degeneration or failure to finish at greater lengths.Source: §2.1 · pp. 3–5
  • Existing studies report declines with instruction count, input and answer length, instruction position and order, parallel tasks, and dialogue duration, using varied checks and outcome measures.They also examine limits in working-memory and long-context tasks.Source: §2.1 · p. 6

2.2 Evaluation of the testing methodology

  • A load meter should treat the experimenter-set number of instructions as load and return a value with uncertainty in those units.The authors derive nine criteria from this requirement, including atomicity, independence, computable checking, and resolution by instruction and answer position.Source: §2.2 · pp. 6–7
  • Form-instruction benchmarks offer programmatic checks and broad rule-count ranges, but shared-text constraints are not independent, vary in difficulty, and can make the task itself harder as rules accumulate.The observed decline therefore cannot isolate model limits from increased joint-task difficulty.Source: §2.2 · pp. 8–9
  • Composite-constraint benchmarks gain realism through complex and practical instructions, but subjective criteria require judges whose errors, wording sensitivity, and bias can distort measured compliance.The passage reports, among other examples, a judge-model score of 0.657 versus a programmatic score of 0.213 for ten instructions.Source: §2.2 · p. 9
  • Instruction-following benchmarks such as FollowBench, ComplexBench, SEQUOR, and AgentIF couple constraints or vary their difficulty, so constraint count does not isolate load.Their scales are limited or not experimenter-controlled, and evaluation may depend on a judge model whose version and prompt affect results.Source: §2.2 · pp. 9–10
  • Required-word tests, including IFScale [21] and Li et al. [23], provide atomic, computable checks and loads up to hundreds of words, but coherence and semantic fit couple difficulty to word count.IFScale spans 10–500 words, while Li et al. [23] use sets of up to 20; word position in the answer is not recorded, and the strongest models do not collapse by 500 words.Source: §2.2 · pp. 10–11
  • Synthetic memory and context probes use computable, independent trials and controlled loads, but measure information retrieval or retention with short answers rather than simultaneous instruction following.Their load varies input features such as context length or the number of updates and keys, not the number of instructions.Source: §2.2 · pp. 11–12
  • LongGenBench [43] places seven requirements in long answers and tracks fulfillment by answer position, but its requirements are coupled, heterogeneous, judge-checked, and tested in only two task variants.Qwen2-72B writes 94.3% of prescribed parts at 16 thousand tokens and 66.2% at 32 thousand; the two variants also change task content and part count.Source: §2.2 · pp. 12–13
  • Prior studies report fulfillment shares at selected loads, all-instruction success, or qualitative decline shapes, with uncertainty estimates inconsistent across methods.A model-maker needs one comparable parameter with uncertainty; the effective-length measure for long inputs is coarse and lacks an interval.Source: §2.2 · p. 13
  • No existing work provides a comparable instruction-load capacity with uncertainty, even though execution quality is known to decline as instruction count, input or answer length, and instruction position change.Harada et al. model the decline but do not derive the load sustainable at a given reliability; related outcomes can make uncertainty intervals too narrow by up to three times without correction [26].Source: §2.2 · pp. 13–14

3 The Instrument

The instrument measures instruction-following under controlled loads using independent word-replacement rules and programmatic answer checking. Its protocol fixes task construction and validation while allowing load conditions and statistical analyses to vary.

3 The Instrument

  • The instrument generates meaningless word-replacement tasks at experimenter-chosen loads and checks each application against a program-computed reference answer.Every occurrence to be replaced is scored as a trial, so degradation appears as a falling share of correct replacements as load grows.Source: §3 · pp. 14–15

3.1 The atomic instruction—word replacement

  • Each atomic rule replaces one word with another, and rules are independent, making rule count a conventional load unit within the instrument’s protocol and vocabulary.A rule does not require understanding the text, and its applications have one correct outcome.Source: §3.1 · p. 15

3.2 Construction of the vocabulary

  • The vocabulary standardizes tasks with single-token, monosyllabic singular nouns, while random word and rule sets help smooth residual differences between items.The current vocabulary contains 2220 words; for another tokenizer, the vocabulary is rebuilt using the same rules.Source: §3.2 · pp. 15–16

3.3 Construction of a task

  • A task’s load is set by (D, n, m), while three random seeds determine its specific rules, text, and rule order.The prompt is reproducible byte for byte, and repeated requests with different seeds sample different material at the same load.Source: §3.3 · pp. 16–17

3.4 Answer validation

  • Validation scores each replacement against a program-built reference and matches answers line by line, confining local alignment failures to one line.Unmatchable positions count as unsuccessful, and framing added before or after the text is excluded when line counts permit.Source: §3.4 · pp. 18–19

3.5 What the instrument measures

  • The instrument’s primary observations are trial outcomes indexed by rule number and answer position; its protocol is separate from the statistical model used to analyze them.The design supports varying load parameters, reproducing published tasks, and comparing models on identical tasks.Source: §3.5 · p. 19

4 The Statistical Model

The statistical model measures how correct-replacement probability changes with load and fits smooth laws to those dependencies, correcting for the arbitrary level introduced by bin coding. It defines instruction capacity as the load at which the share of right replacements reaches a chosen threshold, with rule-load capacity expressed as the number of rules whose cumulative cost fits within the reference-level reserve.

4 The Statistical Model

  • The statistical model estimates replacement accuracy across multidimensional load axes, accounting for binary outcomes, linked axes, shared request difficulty, and uneven coverage of load conditions.It first measures each axis's dependence from the data without imposing a functional form, then summarizes it with smooth laws for computing model characteristics.Source: §4 · pp. 20–21

4.1 The Bernoulli trial and the logit

  • Each trial is Bernoulli, and its right-replacement probability is modeled through an additive logit with a request-level random shift for shared task difficulty.The request shift is normally distributed; coefficients describe a median-difficulty request, while the population-wide share is obtained by averaging over the shift distribution.Source: §4.1 · p. 21

4.2 Variables and bins

  • The model uses logarithmic bins to measure axis profiles from data, and selects axes whose bin estimates are minimally correlated to separate their contributions.Equal-width bins poorly represent axes spanning orders of magnitude with sparse large-load data; logarithmic bins double in value at each boundary and widen where data are scarcer.Source: §4.2 · p. 22

4.3 Reference bins and the level

  • The model removes reference-bin indeterminacy by excluding all-one-outcome bins and estimating a coding-independent level from the fitted axis laws.The conventional free term can depend on the arbitrary reference-bin choice, and all-correct or all-wrong bins have no finite maximum-likelihood estimate; the model therefore has no free term, and the later level is determined from the laws.Source: §4.3 · p. 23

4.4 Approximation shift correction

  • The model first estimates each load-axis profile without prescribing its form, then fits a smooth law and shifts profiles to a shared minimum-load reference level.The reference level is the logit for a right replacement by the median-difficulty request at the smallest load on every axis; corrected axis contributions are comparable and sum to the load cost.Source: §4.4 · pp. 24–25

4.5 The notion of instruction capacity

  • Instruction capacity is the number of rules at which the share of correct replacements reaches quality q, with other load axes fixed at their minimum.It is a continuous quantity with uncertainty; the paper uses q = 0.95 for reliable execution and q = 0.5 for half of replacements being correct. IEC0.5 is independent of request-difficulty spread, whereas IEC0.95 depends on it.Source: §4.5 · pp. 25–26
  • Capacity along any load axis is the root where its cumulative price equals the reserve, and capacities on different axes have different units and do not add.Rule capacity expresses how many rules are held at once, while position capacity expresses how far into an answer they are applied.Source: §4.5 · pp. 26–27
  • Rule capacity is the largest rule count whose cumulative price fits the reserve, including a fraction of the last rule; the first rule has no price because it enters the reference level.Rule prices can differ, and the capacity is continuous.Source: §4.5 · p. 26
  • When the reserve remains unexhausted at the measured region's edge, smooth-law capacity is extrapolated, while bins provide only a lower bound at that edge.Capacity is computed from both corrected bins and smooth laws; their discrepancy measures the approximation quality.Source: §4.5 · p. 27
  • Capacity uncertainty is obtained by the delta method, and uncertainty in the shift variance σ2 does not affect IEC0.5 because its threshold derivative is zero at q = 0.5.The shift variance enters capacity uncertainty only through the threshold.Source: §4.5 · p. 27

5 The Passport of GPT-4o mini

The paper applies its instrument and statistical model to study GPT-4o mini under controlled load. The analysis identifies factors associated with gradual degradation and failures of the third kind, and quantifies their influence.

5 The Passport of GPT-4o mini

  • The GPT-4o mini measurement uses the instrument and statistical model to identify degradation factors and quantify their effects on instruction following.The analysis also examines the occurrence of failures of the third kind.Source: §5 · p. 27

5.1 The measurement

The measurement studies the gpt-4o-mini-2024-07-18 model with controlled generation settings and tasks constrained to fit its context and answer limits. It spans 2,958 requests and 642,430 replacement trials.

5.1 The measurement

  • The study measured gpt-4o-mini-2024-07-18, with a 128,000-token context window and a 16,384-token maximum answer length.Tasks were formed so the full correct answer fit within the answer-length limit.Source: §5.1 · p. 27
  • Temperature zero, a fixed seed of 20260907, top_p = 1, and max_tokens = 16,384 were used across requests.These settings were chosen to reduce answer variability unrelated to task content and avoid additional output-length restrictions.Source: §5.1 · p. 27
  • 2,958 requests and 642,430 replacement trials covered texts of 20–10,240 words and tasks with 1–256 rules.The measurement comprised 1,890 experimental points.Source: §5.1 · pp. 27–28

5.2 Choice of the axes of the predictor model

The authors select task and trial coordinates to characterize overall load and its variation across individual replacements, then express length and answer position jointly in the VS-PMR model. Comparisons of alternative parametrizations support the chosen axes while identifying a possible context-position extension.

5.2 Choice of the axes of the predictor model

  • Replacement density captures why equal rule counts can impose different loads: eight rules over eight distinct words yielded about half correct replacements, versus practically all over 64 words.Density is the share of distinct text words for which replacement rules are given.Source: §5.2 · p. 29
  • The task-level load coordinates are text length v, replacement density p, and rule count m; adding interactions improved fit but not forecasts at new points.Using original parameters D, n, and m worsened description of the data and prediction at points omitted from fitting.Source: §5.2 · p. 29
  • The chosen trial-level coordinates are rule number r and answer position s; event number g, rule-application number k, and excess background Q lacked independent stable contributions.The tested g, s, and k measures substantially overlapped, while Q mixed text-length and word-position contributions.Source: §5.2 · pp. 29–30
  • Context position c contained additional information but could replace neither text length v nor answer position s, so it remained a candidate for model extension.The context position counts tokens from the prompt beginning through the answer word, including the instruction and rule table.Source: §5.2 · p. 30
  • The selected five-axis coordinate system was the most complete among those studied while retaining low mutual correlation, and additional tested axes did not justify complicating it.The model is intended to characterize both overall task load and its heterogeneous action on individual trials.Source: §5.2 · pp. 30–31
  • The model combines text length and answer position in a two-dimensional grid because doubling text length reduced the logit by 1.4–1.8 near the answer beginning but had an effect approaching zero deep in the answer.The resulting VS-PMR form adds this joint grid term to the binned contributions of density, rule count, and rule number.Source: §5.2 · p. 31

5.3 Laws and observations

This section characterizes fitted GPT-4o mini load profiles with smooth laws, examining how text length and word position, replacement density, and rule count affect right-replacement probability. It distinguishes the constant logit decrease per doubling of rule-list size from the accelerating degradation at later rule positions.

5.3.1 Text length and word position

  • κS = −2.67 ± 0.31 and κV = −1.60 ± 0.43 logits per doubling; a smoothed minimum captures the shared positional decline and length-dependent plateau, transitioning near 274 words.The length transition estimate spans 146–517 words, and its width was not determined by the data, so it is not a sharp threshold.Source: §5.3.1 · pp. 33, 34, 36

5.3.2 Replacement density

  • At full replacement density, the loss relative to rare replacements reaches 9.35 ± 1.01 logits, about 78% of its limiting loss; it becomes noticeable from a few-percent density.The loss is about one logit at density 1/16 and continues growing through the measured range without reaching saturation.Source: §5.3.2 · pp. 39–40

5.3.3 Number of rules

  • κM = −0.43 ± 0.10 logits per doubling of rule count; all nine requests with 256 rules collapsed and were excluded from the fit.Across the fitted range, the coefficients decrease approximately linearly with log rule count, with no plateau.Source: §5.3.3 · p. 40
  • Each doubling of the rule list lowers the right-replacement logit by 0.43 ± 0.10, reaching about 2.8 logits at 65–128 rules.The corrected bins measure this contribution relative to a one-rule task, with other model terms unchanged.Source: §5.3.3 · pp. 42–43, Figure 8 · p. 42
  • The profiles measure dependence, not its cause: interference is a proposed explanation requiring separate tests, and separating axes relies on model additivity despite task-linked rule counts and positions.The number of rules is moderately linked to text length, replacement density, and rule number; rule position is least linked to other axes but is constrained by the number of rules.Source: §5.3.3 · p. 43, §5.3.4 · pp. 44–45

5.3.4 Rule number

  • Rules 65–128 are executed 1.05 ± 0.12 logits worse than the first rule, and each doubling increases the loss by about 1.7.The first eight rule-position bins lie between −0.01 and −0.04; degradation accelerates farther down the list.Source: §5.3.4 · pp. 43–45, Figure 10 · p. 45

5.4 Collapse of the answer

The answer-collapse analysis separates losing the text from individual wrong replacements and models both how often collapse occurs and where it occurs. For GPT-4o mini, collapse risk rises with text length and replacement density, while its location is associated with progress through the answer.

5.4 Collapse of the answer

  • In 995 of 2,958 requests, or 33.6%, GPT-4o mini lost the text through structural failure or truncation.Replacements after the loss, including those in missing lines, count as unsuccessful.Source: §5.4 · p. 45

5.4.1 Collapse rate

  • The collapse-rate model treats each request as one Bernoulli trial and describes text-loss probability using text length and replacement density.Requests excluded from the capacity model were included because they inform the collapse law.Source: §5.4.1 · p. 46
  • Each doubling of text length multiplies the odds of collapse by about 11, while density contributes about 1.7 logits at 1/16 and 12.3 logits at full density.The fitted length slope is 2.40 ± 0.19 logits per doubling; the main density increase occurs above 1/16.Source: §5.4.1 · p. 47
  • The odds of losing the text grow approximately with text length to the power 3.5 and rise steeply with replacement density above 1/16, but not with rule count at fixed length and density.The analysis distinguishes collapse of the answer from the instruction-capacity limit.Source: §5.4.1 · pp. 49–50

5.4.2 Place of the collapse

  • For requests that lost the text, the place model treats each replacement word as a trial indicating whether its line is at or after the loss point.The fit covered all 995 such requests and 365,955 replacement words.Source: §5.4.2 · p. 50
  • Beyond word number and passed text share, text length does not determine where collapse occurs; the authors infer that longer texts collapse more often chiefly because their answers run longer.The reported length-bin test gives p = 0.053.Source: §5.4.2 · p. 54
  • After about the 40th word, the odds that the text is already lost grow with word number; they also rise with replacement density and the share of text passed.The estimated corner is around 40 words, with a range from 22 to 70; it is not a sharp measured threshold.Source: §5.4.2 · pp. 51, 54, 55

5.5 The passport

The passport turns fitted load effects into capacities at specified replacement-success levels, with uncertainty. It reports instruction, fatigue, and full-density capacities for GPT-4o mini and summarizes the model’s load and collapse laws.

5.5.1 Instruction capacity

  • At the zero point, GPT-4o mini holds 37–38 rules at a 0.95 success share and 44–46 at 0.5; late-rule execution consumes most of the reserve.The bin and smooth-law estimates agree within their uncertainties; at the 37th rule, rule number costs about 11 logits versus about 2.2 for table size.Source: §5.5.1 · p. 56

5.5.2 Fatigue capacity

  • At 0.95 success, GPT-4o mini’s fatigue capacity is 3277 ± 571 answer words within the measured range; its 0.5 capacity, 11 890 ± 1839 words, is extrapolated beyond it.The fatigue law uses a shared slanted asymptote across text-length slices.Source: §5.5.2 · pp. 56–57

5.5.3 Guaranteed instruction capacity

  • At full replacement density, GPT-4o mini holds about 18–19 rules at 0.95 success and about 30 at 0.5, roughly half and a third fewer than at the zero point.Full density deducts a loss of about 9 logits; the remaining reserve is spent on the cheapest, earliest rules.Source: §5.5.3 · pp. 57–58

5.5.4 Summary of the passport

  • The passport combines text-length and word-position effects with replacement density, rule-list size, rule number, and a request-specific shift; it also gives separate laws for answer collapse.The capacities are computed from these fitted laws and collected alongside the collapse laws.Source: §5.5.4 · pp. 58–59

6 Outlook

The outlook proposes comparing model passports while recognizing that GPT-4o mini’s degradation laws may not generalize and that measurement faces practical constraints. It also outlines prompt weighing as a future use of a model’s calibrated passport.

6.1 Passports of existing models

  • GPT-4o mini’s degradation laws are not assumed universal; broader comparisons could reveal different dependencies or significant load axes across models.A passport catalog is proposed both as a basis for comparative study and as a tool for quantitative capability evaluation.Source: §6.1 · p. 59
  • Tokenizer availability and measurement cost are practical limits; a full measurement took 5 h 43 min and $6.03 for GPT-4o mini under the reported setup.The authors estimate that measurements can be reduced by at least an order of magnitude after initial laws are established, though a strategy remains to be developed.Source: §6.1 · pp. 59, 62, 63

6.2 Measuring prompts

  • Prompt weighing would infer a prompt’s load from its effect on an independent, calibrated replacement task, converting the change in logit into equivalent replacement rules.This approach assumes the prompt’s additional load affects replacement success like an increase in rule count; that representation remains to be validated experimentally.Source: §6.2 · p. 64
  • A model passport and prompt weight would put model capacity and prompt demand on a common scale, including the capacity reserve remaining for a task.The passport supplies the calibration; after full calibration, subsequent prompt measurements are expected to use only a small fraction of the initial resources.Source: §6.2 · p. 64

7 Conclusion

The conclusion presents a controlled replacement task and statistical model for measuring instruction capacity with uncertainty. GPT-4o mini shows regular load effects, while the paper frames broader model comparisons and prompt weighing as future work.

7 Conclusion

  • Single-token replacement rules create atomic, independent load units with computable outcomes; each replacement occurrence is a Bernoulli trial indexed by answer position and rule number.The resulting model measures load effects and restores a reference level independent of coding before expressing capacity in rules with uncertainty.Source: §7 · p. 65
  • GPT-4o mini’s measured load effects are regular: text length sets initial quality, word position governs its subsequent decline, and density and rule-list factors contribute distinct effects.Answer collapse follows separate laws tied to text load and answer progress, rather than the number of instructions held.Source: §7 · p. 65
  • The reported laws describe one model; the proposed next steps are comparing model passports and using them to calibrate a meter of prompt load.The prompt meter is intended to express arbitrary prompt load in the same units as model capacity.Source: §7 · p. 65

8 Code and data availability

The paper makes the instrument, measurement data, and passport available for reproducibility, while keeping the adaptive point-selection technology closed.

8 Code and data availability

  • The public repository contains task-generation, model-call and answer-validation code, passport-building tools, vocabulary materials, measurement records, checksums, and the GPT-4o mini passport.The code can rebuild the passport and verify the published measurement without calling the language model.Source: §8 · pp. 65–66
  • The adaptive technology that selected measurement points is undisclosed, but the passport is recomputed from published points regardless of who selected them.The code is under the MIT license, while the data and documents use CC BY 4.0.Source: §8 · p. 66

A Example of a request

The example request uses a 20-word text and two substitution rules; GPT-4o mini preserves the answer structure and makes all ten required replacements correctly.

A Example of a request

  • The example uses a 20-word text, two rules, order seed zero, and protocol version 0.1.The rules appear in the order of the choice.Source: A Example of a request · p. 70
  • The prompt requires replacing only listed words while preserving other words, order, spacing, and line breaks, and outputting only the rewritten text.Its rules are james -> hang and reign -> steal, applied to the supplied text.Source: A Example of a request · p. 70
  • All 10 substitutions are correct, with the answer’s structure preserved.The request has five occurrences of each of the two words to be replaced.Source: A Example of a request · p. 70

Related papers

Loading 2610.36095v1…