Source-linked AI summary
Grouping the Stochastic Machine: Precision, Not Capability, as the Frontier Metric for AI Systems
George Andrikopoulos
TL;DR
Capability benchmarks measure where outputs land, but not how tightly repeated outputs group around that target. This paper defines and tests deterministic grouping measurement, finding that one replicated gap moved from 0.200 unassisted to 1.000 with a single rule, while the method's interpretation requires caution about task scope and failure causes.
Problem
Capability benchmarks describe average performance but do not measure whether systems reproduce comparable outputs across identical requests.
Method
The paper measures grouping by repeatedly running fixed tasks with deterministic binary scoring, then uses failure consistency to distinguish rule-fixable from model- or sampling-level failures.
Results
0.200 unassisted versus 1.000 library performance replicated the recorded baseline, and one rule closed the measured gap completely.
Takeaways & Limitations
Grouping measurement can guide whether operators should adjust their discipline or change the model or sampling process.
Takeaways & Limitations
The method applies only to verifiable tasks, while confirming failure-cause homogeneity requires non-deterministic human or model judgment.
Abstract
from arXiv · showhide
Frontier language models are compared, marketed, and benchmarked on capability -- what their best or average output can achieve. I argue this measures the wrong axis. The models have saturated accuracy: their mean output lands on the target. What now separates one system from another in practice is precision: how tightly concentrated their outputs are around that target across repeated, identical requests. Borrowing the marksman's distinction, capability is where the average shot lands; reliability is the size of the group. I make three claims. First, precision, not capability, is the frontier differentiator between systems, and benchmark culture systematically fails to measure it, reporting central tendency rather than spread. Second, precision is measurable, cheaply and without circularity, by running a fixed suite of deterministically scored tasks many times at fixed temperature and computing the per-task consistency of outcomes -- no model-in-the-loop grader required. Third, the measurement is not merely descriptive but decision-guiding: it separates consistent failures (a tight group off-centre, correctable by the operating discipline of Paper 1 -- a sight adjustment) from scattered failures (a wide group, correctable only by changing the model or its sampling -- a rifle problem). I define a grouping metric, specify a harness, and show how tracking a human-AI pair's grouping over time yields the compounding signal that Paper 1's field study requires. A first real run, since replicated, illustrates both the method and its most important limit: one measured gap was closed completely by a single rule (0/5 -> 5/5), while a suite of tasks authored from the rules themselves found no value, because a frontier model already embodies explicit good practice -- establishing that a discipline's worth is found by measurement on real work, not constructed from its own rulebook.
1 The wrong axis
Frontier models have converged on capability and accuracy, but systems now differ most in precision: how tightly their outputs group around the target across repeated requests. Capability benchmarks report best or average performance while remaining silent on the consistency practitioners need in production.
- The wrong axis: Capability language describes where the average shot lands, but not whether a model will repeat the result the next time it is asked.Benchmarks emphasize scores, reasoning depth, context length, and impressive best outputs.
- The wrong axis: Accuracy measures closeness of the mean to the target, whereas precision measures the tightness of the grouping; the two axes are independent.A system can be accurate and imprecise or precise and inaccurate.
- The wrong axis: For production engineering, a model that is almost never wild can be more valuable than one that is correct nine times in ten but fails unpredictably on the tenth.The leading models’ best outputs are excellent and their average output is on target, making group width the differentiator.
- The wrong axis: Precision is the frontier metric because the tail is the group, while the field advertises capability-at-best and users experience capability-in-expectation.This reframes Paper 1’s rule to judge the tail rather than the average.
2 Why skills presuppose precision: the zeroing argument
Skills presuppose a precision floor: they can zero a consistent, tight group onto the target but cannot correct scattered errors. This yields a division of labour in which the model supplies precision, the operator supplies accuracy, and measurement determines whether to write a skill or change the model or sampling.
- The zeroing argument: A tight group landing off-centre can be zeroed onto the target, whereas a scattered group has no consistent error to correct.The marksmanship analogy distinguishes an adjustable systematic miss from irreducible variation across shots.
- The zeroing argument: Paper 1’s skills presuppose a precision floor because corrections against scattered errors do not recur in the same place and therefore fail to generalise.Versioned skills, the error loop, and “adjust the sights, not the memory” move a grouping onto the target only when the error is repeatable.
- Division of labour: The model’s job is precision—a tight group—while the operator’s job through skills is accuracy: zeroing that group onto the target.This is the paper’s explicit division of labour between system consistency and operating discipline.
- Decision rule: Before adding guardrails, measure whether a failure is consistent: write a skill for a tight off-centre group, but change the model or sampling temperature for a scattered group.The measurement in §4 is intended to distinguish these cases.
3 Related work and the precise claim
Prior work supplies repeated-sampling consistency measures, deliberate use of output variance, and evidence that instruction libraries exist at scale. Reliability research also shows that larger, instruction-tuned models can improve average performance while becoming less reliable.
- Related work: Repeated-sampling evaluation already measures consistency through the probability that all k samples succeed, alongside pass@k’s any-success criterion.The passage identifies this all-succeed probability as a severe consistency measure.
- Related work: Self-consistency deliberately exploits output variance, while persistent instruction libraries are documented across 401 open-source repositories.The repository study establishes that such libraries exist at scale, but not what they are worth.
- Related work: Scaling and instruction-tuning raised average performance while lowering reliability across several model families.Larger, shaped-up models declined fewer questions but more often returned confident, plausible, wrong answers.
4 Measuring the group
Grouping is measured by repeatedly running fixed, deterministically scored tasks and distinguishing consistent outcomes from scattered ones through pass rates and failure-cause homogeneity. A real run found a perfectly grouped but imperfectly accurate model, and one versioned rule closed its systematic miss completely.
- Measurement harness: Precision is measured without a model-in-the-loop grader by repeating fixed tasks with binary outcomes such as compilation, tests, linting, or typechecking.These signals are cheap, repeatable, and avoid circularity in the measurement.
- Decision procedure: The pass rate screens, but homogeneous failure causes decide whether a failure is a tight off-centre group correctable by one rule or genuinely scattered.Failing samples must be read before assigning the failure to either row.
- Real run: Accuracy 0.833, grouping 1.000, worst-case 0.000: the six-task run produced five decisive passes and one decisive fail, with zero scatter.The single miss failed 5/5 for one identifiable parse-validation cause, making it systematic rather than scattered.
5 Two groupings: the rifle and the zeroed rifle
The harness separates a model’s intrinsic grouping from the tighter grouping achieved when operating skills are loaded. Their gap measures the operating discipline’s value for day-to-day work.
- 5 Two groupings: the rifle and the zeroed rifle: Raw-model grouping measures the rifle’s intrinsic property for model selection, alongside cost-per-completed-task.This configuration uses no skills.
- 5 Two groupings: the rifle and the zeroed rifle: Pair grouping measures the zeroed rifle’s performance and governs day-to-day work.This configuration loads the operating skills.
- 5 Two groupings: the rifle and the zeroed rifle: The gap between raw-model and pair grouping quantifies how much the operating discipline tightens the group.Tracking the pair’s grouping over time on a fixed suite extends this measurement into ongoing evaluation.
6 Found, not authored: the limit of a constructed suite
The constructed suite failed to demonstrate the library’s broad value: authoring one task per rule selected for behaviors the model already handled, producing a near-zero raw→pair delta. The result bounds the library’s role to a small, measurement-fed set of patches for residual model-specific gaps.
- Constructed-suite limit: Five authored tasks, one per major rule, failed to produce a meaningful raw→pair delta because they selected error classes the model already handled.The attempted suite therefore measured baseline competence rather than the library’s marginal contribution.
- Constructed-suite limit: A discipline’s value cannot be established by authoring tasks from its own rulebook, because the resulting delta is structurally near zero.A capable model has internalised most explicit good practice, so constructed tasks do not expose the library’s marginal contribution.
- Operational implication: A comprehensive rulebook’s marginal value trends toward zero on a sufficiently capable model, leaving residual model-specific gaps as the targets for a small, live, measurement-fed patch set.The rules must be re-measured as the current model changes.
7 Cost, confounds, and honest limits
The harness has honest limits: repeated evaluation is costly, scorer defects can systematically bias results, and uncontrolled provider defaults constrain cross-run temperature claims. Deterministic scoring also cannot replace human or model judgment when confirming whether failures share a cause.
- Cost: N× cost keeps the suite small and runs periodic rather than continuous.Suggested uses are a quarterly precision review or a preselection gate before adopting a model.
- Scoring confounds: Scorer bias is the dominant threat, with identified defects systematically biased against the discipline.Examples include penalizing legitimate .expect() usage and rejecting disciplined documentation because a template’s crate name could not resolve.
- Temperature: Within each back-to-back run-pair, the raw→ruled delta is valid because model, frozen task, scorer, and sampling configuration are held constant.Absolute p_i values are not anchored to a known temperature, and cross-pair comparisons are weaker because provider defaults may differ.
- Interpretive limits: Cause-homogeneity confirmation requires human or model judgment, so the screening signal is objective but the confirming test is not.When screening and confirmation disagree, the disagreement should be recorded rather than silently resolved.
8 Conclusion
The paper concludes that precision—not capability—is the frontier metric for AI systems: repeated deterministic trials reveal how tightly outcomes group and whether misses are fixable through operator skill or require changing the model or sampling.
- 8 Conclusion: Precision, measured by counting decisive outcomes over repeated deterministic trials, is presented as the frontier metric for evaluating AI systems.The conclusion contrasts precision with capability and frames grouping as the relevant practical distinction.
- 8 Conclusion: The grouping count does more than rank systems: it indicates whether a miss is the operator’s to fix with skill or the rifle’s to fix through system changes.This makes the measurement operationally useful for diagnosing the source of failure.