PaperlayerView exact revision on arXiv

Source-linked AI summary

Sensitivity or Criterion? Measuring and Moving When Language Models Act

Sixun Dong, Yebowen Hu, Ming Yin, Yiran Chen, Fei Liu, Chen Chen

arXiv:2610.15415v1cs.CLcs.AIcs.LG

TL;DR

A single accuracy cannot distinguish whether an agent fails to recognize a warranted action or applies the wrong decision criterion, and binary-label benchmarks do not provide a shared criterion boundary. FLIPACT computes evidence and answers by rule, then measures sensitivity and criterion separately. Across models, sensitivity rises with size within families, while criterion shifts through steering and distillation without significant sensitivity change in the reported distillation results.

  • Problem

    A single accuracy mixes failures to recognize warranted action with reluctance or over-readiness to act, and criteria estimated from binary labels depend on each benchmark’s example mix.

    Source: §1 · pp. 1–2

  • Method

    FLIPACT uses annotated tool properties and a rule-computed evidence scale with a fixed zero boundary, estimating sensitivity from the Yes-rate slope and criterion from its half-rate crossing.

    Source: §1 · p. 2

  • Results

    Across 44 open models, sensitivity rises with size within a family and its ranking agrees with external d′ on three of four benchmarks; steering and distillation shift criterion separately from sensitivity.

    Source: §1 · p. 2

  • Takeaways & Limitations

    When-to-act evaluation is more informative when it reports sensitivity and criterion alongside accuracy, because models with the same accuracy can need opposite fixes.

    Source: §6 · p. 9

    For 9 of 21 models, the Yes-rate never reaches one half, so τ is the fitted curve’s crossing rather than an observed-data crossing.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Before an agent acts, it must decide whether action is warranted. A single accuracy cannot distinguish failure to recognize a warranted action from reluctance to act. Signal-detection theory separates sensitivity from criterion, but criteria estimated from authored binary labels depend on each benchmark's example mix. We introduce FlipAct, a diagnostic of tool use in physical states. A rule over annotated tool properties computes each answer and a graded evidence score, fixing the Yes/No boundary at zero across models. Questions form 2-by-2 quads that hold required knowledge fixed while the state flips. The slope of the fitted Yes-rate curve measures sensitivity; its crossing at one half measures criterion. Across 44 open models, sensitivity rises with size within a family, and its ranking agrees with three of four external when-to-act benchmarks. Models respond to evidence sign rather than magnitude, accepting actions that leave the state no better about as often as helpful ones. When Yes means act, most models are conservative across rephrasings; in some, this partly reflects a preference for No. Criterion is not determined by size. Activation steering shifts criterion with weights fixed; distillation from a model's own steered copy puts the shift into its weights without a significant sensitivity change. The distilled shift transfers to six external benchmarks, including two without Yes/No answers, while criterion-free AUC is preserved on the five benchmarks where it is available. Accuracy improves where models start conservative and declines or remains within error where they start permissive. Project page: https://sixundong.com/projects/flipact

1 INTRODUCTION

A single accuracy cannot distinguish whether an agent misses warranted actions or applies an overly conservative or permissive criterion, and criteria from binary-label benchmarks lack a shared boundary. FLIPACT measures both against a rule-computed evidence scale, and the paper reports that sensitivity and criterion behave differently across models and interventions.

Motivation

  • Accuracy alone mixes failures of sensitivity with failures of criterion, while binary-label benchmarks yield criteria tied to each benchmark’s example mix.Models with similar balanced accuracy can show opposite tool-calling behavior, as gemma-2-9b and Phi-3.5-mini do on When2Call.Source: §1 · pp. 1–2

Approach

  • FLIPACT computes evidence for acting from annotated tool properties, fixes the Yes/No boundary at zero, and estimates sensitivity from the Yes-rate slope and criterion from its half-rate crossing.Its 2 × 2 quads hold required knowledge fixed while flipping the state that determines whether the action is warranted.Source: §1 · p. 2

Main findings

  • Across 44 open models, sensitivity rises with size within a family and its ranking agrees with external d′ on three of four when-to-act benchmarks.Models respond to evidence sign rather than magnitude, accepting actions that leave the state no better about as often as helpful ones; most are conservative when Yes means act.Source: §1 · p. 2

Interventions

  • Activation steering shifts criterion with weights fixed, and distillation from a steered copy puts the shift into weights without significant sensitivity change.The shift transfers to benchmarks without Yes/No answers, while supervised training on computed labels raises sensitivity and moves criterion with it.Source: §1 · p. 2

2 RELATED WORK

Prior work evaluates when-to-act decisions across tool calling, clarification, abstention, and over-refusal, while signal-detection research separates sensitivity from criterion. FLIPACT distinguishes itself with a shared rule-computed evidence scale and state flips that hold required knowledge fixed.

When-to-act evaluation

  • The paper positions tool calling, clarification, abstention, and over-refusal as related settings for deciding whether to act.The cited benchmark families include tool calling, clarification, and abstention; the broader comparison covers over-refusal as well.Source: §2 · pp. 2–3
  • Most existing when-to-act benchmarks use authored binary ground truth and a single performance rate, without per-question graded evidence or state flips that hold required knowledge fixed.The paper reuses four such benchmarks as external tests and measures transfer on six benchmarks.Source: §2 · p. 3

Signal detection and language models

  • Prior signal-detection work fits factual-answering criteria on each model’s confidence scale or reads call offsets from internal features without separately measuring sensitivity.FLIPACT instead uses rule-computed evidence shared across questions and models, holds required knowledge fixed, and moves criterion into weights.Source: §2 · p. 3

3 THE FLIPACT DIAGNOSTIC

FLIPACT separates sensitivity from criterion, constructs matched questions whose state determines whether an action is warranted, and computes answers and evidence scores from annotated tool properties. Its automated build is checked against independent recomputation and annotation perturbations.

3.1 PRELIMINARIES: SIGNAL DETECTION

  • Signal-detection theory defines sensitivity d′ from the separation of signal and noise evidence distributions and criterion c from the threshold’s position relative to their midpoint.For when-to-act decisions, hits are warranted actions and false alarms are unwarranted actions; FLIPACT maps sensitivity to the Yes-rate slope and criterion to its half-rate crossing.Source: §3.1 · p. 3

3.2 SAME KNOWLEDGE, FLIPPED STATE

  • Each quad crosses two tools with states that flip the answer, keeping the required tool and state knowledge fixed while the state determines whether action is warranted.The main set has 1,880 questions, including 840 in complete quads and 1,256 in state flips; situations contain equal numbers of Yes and No questions.Source: §3.2 · pp. 3–4

3.3 GROUND TRUTH AND EVIDENCE BY RULE

  • A rule computes evidence ∆U from how a tool’s direction and magnitude change the state’s distance from the goal, then derives the answer and question type.Tool properties are annotated by language models; unmet prerequisites receive a conventional ∆U of −2 per prerequisite, and ∆U takes seven values from −3 to 3.Source: §3.3 · p. 4

3.4 AN AUTOMATED BUILD

  • The automated pipeline builds questions and labels from annotated tool records; independent recomputation finds zero ground-truth mismatches, and annotation perturbations preserve a and τ rankings.The pipeline screens tool-property annotations, pairs tools by opposing direction, and generates states, answers, question types, and text by rule; perturbations move criteria toward acting rather than producing conservative criteria.Source: §3.4 · pp. 4–5

4 MEASURING SENSITIVITY AND CRITERION

FLIPACT fits Yes-rates against a shared evidence score to estimate sensitivity and criterion. Across models, sensitivity tracks size within families and external sensitivity measures, while criterion varies by model and is usually conservative when Yes means act.

4 MEASURING SENSITIVITY AND CRITERION

  • The study fits 44 open models and compares their coordinates with four external benchmarks spanning different tool-use decisions.A logistic fit maps Yes-rate over FLIPACT’s computed evidence score; its slope a is sensitivity and its half-crossing τ is criterion.Source: §4 · p. 5, §4.1 · p. 5

4.1 THE FIT AND THE RESPONSE PROFILE

  • Yes-rates respond to evidence sign rather than magnitude, with actions at ∆U = 0 accepted about as often as helpful actions.The jump from ∆U = −1 to 0 is significant in 19 of 21 well-determined models, while rates are flat within either side; ∆U = 0 questions are slight overshoots.Source: §4.1 · p. 5, Figure 4 · p. 5

4.2 SENSITIVITY GROWS WITH SIZE AND AGREES WITH EXTERNAL BENCHMARKS

  • Sensitivity rises with size within families and agrees with external d′ on three of four when-to-act benchmarks.The When2Tool relation is not significant.Source: §4.2 · pp. 5–6

4.3 CRITERION: A PER-MODEL SETTING, CONSERVATIVE BY DEFAULT

  • Most models are conservative across rephrasings, and for some, a preference for answering No partly accounts for that criterion.The criterion also carries across question types: models that accept more destructive actions tend to accept more slight overshoots, though most accept overshoots more readily.Source: §4.3 · p. 6
  • Criterion varies more by model than size: same-size Mistral-Small-24B releases fall on opposite sides of zero, while pooled criterion-size association vanishes.Within some families criterion drifts toward zero as models grow, partly because sensitivity rises.Source: §4.3 · pp. 6–7
  • FLIPACT criterion does not consistently predict criterion on external benchmarks: correlations are near zero on three, but positive on BFCL.The BFCL correlation is r = +0.51 and rests on the three models with the lowest τ.Source: §4.3 · p. 7

5 MOVING THE CRITERION

The paper tests whether criterion can shift independently of sensitivity, through inference-time steering and distillation, and whether those shifts transfer. It also tests whether training on computed labels changes sensitivity.

5.1 STEERING MOVES THE CRITERION WITH THE WEIGHTS FIXED

  • Steering shifts When2Call criterion toward acting on all five models, while accuracy changes by at most four points.Across sixteen models, eleven move significantly toward acting and four pass both controls; results are small when starting near zero and fail a held-out split for permissive phi-4.Source: §5.1 · p. 7

5.2 DISTILLING FROM A STEERED COPY PUTS THE CRITERION INTO THE WEIGHTS

  • Distillation moves criterion toward acting on all four students without significant sensitivity change; held-out accuracy rises one to five points from conservative starting points.The random direction moves criterion at most a fifth as far; sensitivity equivalence is established for two students, while the test is inconclusive for two.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 THE DISTILLED CRITERION MOVES THE SAME WAY ON SIX EXTERNAL BENCHMARKS

  • On all 23 scored model–benchmark pairs, distillation moves criterion toward acting in every seed, while criterion-free AUC stays unchanged on five benchmarks.Every random-direction control moves less than its paired student; AUC changes are equivalent to zero within ±0.02.Source: Figure 7 · p. 8, §5.3 · p. 9, Table 4 · p. 9
  • Criterion distillation moves c farther than a Yes-bias teacher on When2Call and BFCL-miss-func for Qwen3-8B and gemma-3-12b, including formats without Yes answers.Across four Yes/No benchmarks, the distilled model moves c as far or farther than the Yes-bias student.Source: §5.3 · p. 9
  • Where students start conservative, balanced accuracy rises; where they start permissive, it falls or stays within error, with one roughly ten-point decline.Steering toward acting lowers balanced accuracy on 12 of 25 model–benchmark pairs; choosing direction and strength from 50 labeled examples per benchmark leaves only Mistral-24B on When2Call worse.Source: §5.3 · p. 9

5.4 LABELS MOVE SENSITIVITY, AND THE CRITERION WITH IT

  • Sensitivity learning raises a by about 0.5 to 1.2 and held-out accuracy by 17 to 24 points, while also moving criterion close to zero.Its sensitivity gain stays on FLIPACT; external d′ and c changes vary across students and are matched by the shuffled-label control.Source: §5.4 · p. 9

6 CONCLUSION

The conclusion frames acting as a measurement problem: sensitivity and criterion can call for different fixes, and criterion shifts depend on the target benchmark. The appendix describes how FLIPACT constructs rule-derived questions from annotated tool properties.

6 CONCLUSION

  • Criterion can shift at inference or in the weights, with the target benchmark setting the useful direction; label training raises sensitivity on FLIPACT.Reversed wordings separate answer-word preference from the rest of the criterion shift.Source: §6 · p. 9

ETHICS STATEMENT

A.1 ANNOTATION SCHEMA

  • Each tool record describes one tool on one axis with a single direction; out-of-vocabulary annotations are treated as parse failures.A tool acting on two axes receives one record per axis.Source: §A.1 · p. 16, Table 5 · p. 16

A.2 FROM A TOOL TO ITS QUESTIONS

  • Question states are computed from the tool and goal: solution tools close the deficit, destructive tools widen it, and redundant tools overshoot.Pairs share an axis and target, act in opposite directions, and differ in magnitude by at most one band; prerequisite and already-at-goal questions are supplementary.Source: §A.2 · p. 16
  • Questions use state adverbs for state magnitude; prerequisite wording appears whether supplies are present or absent, so it carries no answer information.The adverbs are slightly, very, and dangerously.Source: §A.2 · p. 16

A.3 A WORKED QUAD

  • In the worked thermal example, cooling an already cold person and warming an already hot person are destructive; an ice bath can also overshoot a slight deficit.The ice bath points toward the goal but has magnitude two, making the overshoot a No despite the correct direction.Source: §A.3 · p. 17, Table 6 · p. 17

A.4 THE BUILD

  • Six filtering stages leave 149,840 candidate entities, and screening yields 1,638 tool records from 72,446 entities in physical-tool domains.Pairing produces 1,422 tool pairs and 6,176 questions; collision removal and selection leave 1,880 main-set questions.Source: §A.4 · p. 17

A.5 DATA QUALITY

The appendix checks whether FLIPACT answers can be inferred from wording, whether the set matches its rule, and whether annotation noise alters model conclusions. It also reports measurement reliability and response patterns.

A.5 DATA QUALITY

  • A lexical probe reaches AUC 0.498 ± 0.005 on held-out situations, and independently recomputing answers yields zero mismatches.The probe’s result lies within its 0.485 to 0.524 range under permuted labels; labels are exactly balanced in each of 180 situations.Source: §A.5 · pp. 17–18, Table 7 · p. 18
  • A held-out GLM-4-32B sample matches 100% of is-tool judgments and 92.0% of axis judgments; all magnitude judgments fall within one band.It matches the recorded magnitude band exactly for 67.3% of tools.Source: §A.5 · p. 18

A.6 ROBUSTNESS TO ANNOTATION NOISE

  • At observed annotation-noise rates, both model rankings remain above 0.98 and every criterion moves toward acting, so noise does not produce conservative criteria.Noise lowers sensitivity by two to four percent; refitting on records with agreed magnitude bands yields ρ(a) = 0.994 and ρ(τ) = 0.995.Source: §A.6 · p. 19, Table 8 · p. 19

B PER-MODEL COORDINATES AND MEASUREMENT CHECKS

  • Situation-cluster split-half reliability is 0.906 for sensitivity and 0.973 for criterion.When tool records rather than situation clusters are split, reliability is 0.959 and 0.991, respectively.Source: §B.2 · p. 23

B.2 RELIABILITY AND RESPONSE PROFILE

  • Yes-rate changes significantly from evidence −1 to 0 in 19 of 21 models, but from 0 to +1 in none; models largely respond to evidence sign, not magnitude.At zero evidence, the slight-overshoot Yes-rate lies nearer the hit rate than the false-alarm rate in most models, and its interval includes the hit rate in 20 of 21.Source: §B.2 · p. 23, Table 12 · p. 24
  • A step model fits held-out answers almost as well as the logistic; a step between −1 and 0 fits best in 20 of 21 models.The logistic improves median held-out fit by 0.010 nats per question, reliably in 3 of 21; nine models’ smoothed observed Yes-rate never reaches one half.Source: §B.2 · pp. 23–24
  • Models answer Yes less often when a tool is unnecessary or lacks its prerequisite than on matched slight-overshoot questions.The median Yes-rate reductions are 0.27 for already-at-goal states and 0.43 for missing supplies.Source: §B.2 · p. 24, Table 13 · p. 24

B.3 A SIGNAL-DETECTION READOUT

  • Signal-detection sensitivity gives the same model ranking as the logistic fit, while τ closely tracks c/d′ (r = 0.999).The absolute c ranking is less close to τ (ρ = 0.77) because c is not divided by sensitivity.Source: §B.3 · p. 24

B.4 CONVERGENCE WITH EXTERNAL BENCHMARKS

Sensitivity converges with external measures on several benchmarks, including criterion-free AUC comparisons, while criterion correlations are less stable. The supplementary analyses show how these patterns vary with model capability, generation, and reasoning mode.

B.4 CONVERGENCE WITH EXTERNAL BENCHMARKS

  • At panel cutoffs of 0.25, 0.30, and 0.35, sensitivity’s correlation with When2Call d′ stays positive and significant, but criterion correlation is not stable.The criterion correlation is negative and significant only at 0.25, so the authors draw no direction from it.Source: §B.4 · p. 25, Table 16 · p. 25
  • On QuestBench and When2Call, FLIPACT sensitivity correlates with criterion-free AUC at r = +0.70 and +0.81, respectively.Because this AUC has no criterion term, the convergence is not dependent on an external measure that includes criterion.Source: §B.4 · p. 25

B.5 SENSITIVITY AND GENERAL CAPABILITY

  • With model size partialed out, FLIPACT sensitivity correlates with external d′ at 0.62–0.64 on three benchmarks; with MMLU partialed out, the correlations are 0.52–0.55.The MMLU analysis covers 15 models and has p values between 0.04 and 0.06; FLIPACT d′ correlates with sensitivity at 0.96.Source: §B.5 · pp. 25–26, Table 17 · p. 26

B.6 GENERATIONS WITHIN THE QWEN AND GEMMA LINES

  • Across Qwen and Gemma generations at comparable sizes, sensitivity rises at small and middle sizes but changes little at the largest sizes, while criterion is not set by generation.The passage notes that criterion stays put from Qwen3-8B to Qwen3.5-9B, but its remaining comparison is truncated.Source: §B.6 · p. 26, Table 18 · p. 27

B.7 REASONING MODE

  • Reasoning significantly raises sensitivity only in models of 4B parameters or fewer, while it moves criterion toward zero in several larger Qwen models.The criterion moves significantly toward zero on Qwen3.5-4B, Qwen3-8B, Qwen3.5-9B, and Qwen3.5-27B; several models lack a usable fit in one mode.Source: §B.7 · p. 27, Table 19 · p. 27
  • For Qwen3.5-27B, reasoning changes a warranted hotplate response from No to Yes while the harmful hot-state response stays No.In the paired results, reasoning off answers Yes in both the very-cold and very-hot cases, whereas reasoning on answers Yes only when the food is very cold.Source: §B.7 · pp. 27–28
  • For Qwen3.5-4B, reasoning works through the distinction between the two states of a rivet-gun question, rather than merely shifting responses toward Yes.The example trace reasons about the tool’s effect on the state; the paired answers are No for the harmful, very-jammed state and Yes for the helpful, very-loose state.Source: §B.7 · pp. 27–28

C THE CRITERION UNDER REVERSED QUESTION POLARITY

Reversed question polarity helps separate an evidence threshold from a preference for the answer word. The additional steering analyses test criterion shifts across held-out data and external benchmarks, with sensitivity-stability checks and controls.

C THE CRITERION UNDER REVERSED QUESTION POLARITY

  • The reversed-wording design asks the same content under opposite polarities, then recodes answers so an answer-word preference can be distinguished from an evidence threshold.The decomposition writes the canonical and reversed criteria as τwise = τd + b and τunwise = τd − b, where b is positive for a No preference.Source: C THE CRITERION UNDER REVERSED… · p. 28, Table 20 · p. 29
  • For three Qwen3.5 models and GLM-4-32B, canonical conservatism reflects a No preference: b is +3 to +4 while τd is within a unit of zero.Seven models retain criterion sign across all reversed wordings; the other models that stay conservative do so on τd, with b between −0.97 and +0.58 for the sign-stable group.Source: C THE CRITERION UNDER REVERSED… · pp. 28–29
  • Across five same-polarity rephrasings, the conservative criterion stays positive on 15 or all 16 models and ranks models much as the canonical wording does.Sensitivity rank correlation (ρ(a)) is 0.87–0.94 for the five rephrasings, Answer No or Yes, and letter options with Yes first, and 0.58 for letter options with No first.Source: C THE CRITERION UNDER REVERSED… · p. 29

D.1 STEERING

  • On held-out FLIPACT data, steering moves Qwen3.5-9B and Qwen3.5-27B criterion toward zero, keeps d′ within a few hundredths, and raises accuracy by about two points.The direction and accuracy-selected strength are fitted on one half and evaluated on the other.Source: §D.1 · p. 30, Table 23 · p. 31
  • Four models retain a held-out When2Call criterion shift within 0.02 of the full-set shift, and every split lowers criterion and raises call rate.Phi-4 starts permissive and shows no reliable shift on the held-out half.Source: §D.1 · p. 31, Table 24 · p. 31
  • Eleven of sixteen models move significantly toward acting under steering on When2Call, but only four pass both controls and respond monotonically in steering strength.Those four shift at least three times as much as the largest control shift, with at least half the shift retained after refusal, sycophancy, and engagement directions are projected out.Source: Table 22 · p. 30, §D.1 · pp. 31–32

D.2 STEERING ON FURTHER BENCHMARKS

  • On further benchmarks, steering significantly shifts criterion toward acting on every scorable benchmark for Qwen3-8B and Mistral-24B, but on only one of four for Qwen3.5-27B.Two Mistral-24B benchmarks have no strength with stable d′, and Qwen3.5-27B’s response is nonmonotone on When2Tool and BFCL-miss-func.Source: §D.2 · p. 32, Table 25 · p. 32

D.3 TRAINING

The training analyses compare criterion distillation with sensitivity learning, using a shared data split and significance rule. Distillation shifts criterion toward acting and improves accuracy when models start conservative, while answer-only training on computed labels can fail to preserve a usable fit.

D.3 TRAINING

  • The training schemes split data by connected components, keeping tool records, situations, and state flips out of both halves, and assess changes across three seeds.A change is significant when it is at least twice the largest paired situation-cluster bootstrap error among the seeds.Source: §D.3 · p. 32
  • Criterion distillation uses a steered copy of the student as teacher, with a direction derived from its own Yes/No activations and a forward-KL loss.The method’s direction uses no labels; the passage reports non-Qwen students’ MMLU changes between −0.004 and +0.006.Source: §D.3 · p. 32
  • No matched random direction shifts criterion toward acting with sensitivity intact, despite some matched directions perturbing answers as much as the criterion direction.Other matched random directions collapse a model to one answer before reaching the teacher’s perturbation level.Source: §D.3 · p. 33, Table 27 · p. 33
  • Criterion distillation transfers most of the teacher’s criterion-associated logit shift into student weights: students reproduce 0.88 to 1.04 of the teacher’s shift in Σ.The teacher moved only in τ, whereas the student differs in Σ and not in D; random-direction teachers produce much smaller Σ shifts.Source: §D.3 · p. 34
  • Answer-only sensitivity learning on computed answers loses a usable criterion on seven of twelve runs, through one-answer collapse, loss of slope, or high criterion uncertainty.The answer-shuffled control removes the link between state and answer; free-form output and some external readouts also fail for Mistral-24B.Source: §D.3 · p. 34
  • Distillation raises balanced accuracy on external benchmarks where students start conservative, but where they start permissive, accuracy falls by up to four points or remains within error, with one larger exception.For Mistral-24B on When2Tool, accuracy falls by about ten points because the hit rate is already at ceiling and the shift adds false alarms.Source: §D.3 · p. 34

D.4 A CRITERION-FREE READOUT OF THE EXTERNAL BENCHMARKS

The external readout tests whether distillation changes criterion without changing criterion-free sensitivity, while further analyses separate answer preference from readiness to act and examine benchmark-specific steering. Qualitative pairs illustrate how criterion shifts can change answers while preserving the ordering of warranted and unwarranted cases.

D.4 A CRITERION-FREE READOUT OF THE EXTERNAL BENCHMARKS

  • Distillation leaves criterion-free AUC unchanged for every student on five benchmarks with answer log-probabilities.AUC is computed from answer-token log-probability margins, which yield a full ROC without a criterion term; BFCL-miss-func lacks this readout.Source: §D.4 · p. 34, Table 32 · p. 36
  • On Qwen3-8B, the d′ change tracks criterion movement along an unequal-variance ROC, while the external benchmark comparison also reports criterion-free AUC.The supplied excerpt gives a QuestBench d′ rise of 0.72 and an ROC slope of 1.42, but does not state the corresponding AUC value.Source: §D.4 · pp. 34–35, Table 33 · p. 36

D.5 ANSWER PREFERENCE AND A YES-BIAS TEACHER

  • On all four students, most of the act-rate shift flips with wording polarity, indicating that distillation mostly shifts Yes preference on FLIPACT’s binary readout.Qwen3-8B and Mistral-24B also show a smaller shared shift; random-direction controls move neither part.Source: §D.5 · pp. 36–37, Table 34 · p. 37
  • On two benchmarks without a Yes answer, criterion distillation shifts c further toward acting than a Yes-bias student in every seed on both tested students.The comparison covers Qwen3-8B and gemma-3-12b; BFCL-miss-func is the stronger test because its reply is free-form.Source: §D.5 · p. 37, Table 35 · p. 37

D.6 CHOOSING THE STEERING SIGN AND STRENGTH PER BENCHMARK

  • The benchmark-specific steering rule uses 50 labeled examples to choose sign and strength while retaining only strengths that keep d′ within a quarter of its unsteered value.It pushes nothing when the criterion is smaller than its bootstrap error or no eligible strength brings c closest to zero.Source: §D.6 · pp. 37–38
  • The per-benchmark steering push repairs every pair degraded by the fixed push but degrades one pair, Mistral-24B on When2Call.On that pair the criterion moves toward zero while d′ falls 17%, within the rule’s allowed quarter-loss; five pairs improve, or eight with 100 labeled examples.Source: §D.6 · p. 38, Table 36 · p. 38

D.7 QUALITATIVE EXAMPLES

  • On 354 held-out FLIPACT state flips, distilled margins rise in every flip and seed, while warranted cases remain above destructive cases about as often as before.Among base-model No answers, students switch to Yes on 220 of 286 warranted questions and 137 of 311 destructive questions; the control switches on none.Source: §D.7 · p. 39
  • On the 288 QuestBench pairs, both margins rise after distillation in every pair and seed, while the fully specified problem remains above its underspecified twin about as often as before.The example illustrates margin shifts without a corresponding change in pair ordering.Source: §D.7 · p. 40

E A MODEL THAT RETURNS A PROBABILITY

The paper evaluates whether a probability-returning decision model shows the same FLIPACT patterns as token models. JEV exhibits a conservative criterion and evidence-sign step, but its scope is limited to one closed model and one API version.

E A MODEL THAT RETURNS A PROBABILITY

  • JEV returns the probability that a statement is true rather than generating text, and its probability for “using the tool is wise” is read as P(act).It is evaluated descriptively rather than included in the token-model comparisons because its input differs from their prompt.Source: E A MODEL THAT RETURNS A PROBABILITY · p. 40
  • JEV’s criterion is conservative and excludes zero, while mean P(act) steps between evidence scores −1 and 0 and is flat on either side.Its sensitivity is close to the most sensitive token-model entries, and its probability tracks evidence magnitude only barely better than its ordered verbal ratings.Source: E A MODEL THAT RETURNS A PROBABILITY · p. 40
  • On FLIPACT, JEV has the same coordinates under statement and use-or-avoid readouts, but its When2Call criterion shifts from −1.15 to +0.02 when ask is offered as a third option.The comparison indicates that a criterion read from two options is not constant when a third response option exists.Source: E A MODEL THAT RETURNS A PROBABILITY · p. 40, Table 38 · p. 40
  • The result is limited to one closed model under one API version, and binarizing its probability introduces a threshold absent from token models.A single model cannot stand in for the 44 evaluated open models.Source: E A MODEL THAT RETURNS A PROBABILITY · p. 40

F WHEN-TO-ACT BENCHMARKS, STEERING AND EFFICIENT INFERENCE

The paper situates its diagnostic among benchmarks for tool calling, clarification, abstention, and refusal, and connects acting decisions to adaptive computation. Related systems use thresholds or conditional routing to decide when to spend more computation.

F WHEN-TO-ACT BENCHMARKS, STEERING AND EFFICIENT INFERENCE

  • Activation steering adds a direction read from the residual stream, with refusal as its best-studied target and sycophancy as a related disposition.The paper projects refusal and sycophancy directions out of its criterion direction as a control on the steering result.Source: F WHEN-TO-ACT BENCHMARKS, STEERING AND… · p. 41
  • LoHi requests high-resolution video frames only when answer entropy exceeds a threshold, sending about a third of questions to a second pass at nearly always-requested accuracy.The paper identifies this threshold as a criterion-like decision about spending computation.Source: F WHEN-TO-ACT BENCHMARKS, STEERING AND… · p. 41
  • MMTok allocates visual tokens according to question difficulty, while MAI passes a small model’s reasoning to a larger one only when needed.The paper notes an example in which a small vision-language model says a fan cools a cold user yet still calls it suitable.Source: F WHEN-TO-ACT BENCHMARKS, STEERING AND… · p. 41
  • Twelve when-to-act benchmarks vary in how they construct ground truth, match controlled negatives, grade evidence, and decompose sensitivity from criterion.They span tool-call necessity, minimally perturbed action pairs, abstention, clarification, over-refusal, and physical affordance.Source: Table 39 · p. 41, F WHEN-TO-ACT BENCHMARKS, STEERING AND… · p. 41

G LIMITATIONS AND OUTLOOK

The paper identifies limits in its instrument, measurement, and interventions, then outlines extensions to richer tool properties and multi-step scenarios. Its tested questions use one physical domain and template, and it evaluates only open checkpoints up to 70B parameters.

G LIMITATIONS AND OUTLOOK

  • FLIPACT labels are rule-computed from language-model-annotated tool properties without human observers, and all questions use one template in one physical domain.Side effects are omitted from the rule, and other templates and domains remain untested.Source: G LIMITATIONS AND OUTLOOK · p. 41
  • The model sample contains only open checkpoints of at most 70B parameters, and several measurement relations rely on just three to five models per family.Whether the inconsistent relation between FLIPACT criterion and benchmark criterion reflects limited power, benchmark interfaces, or the construct remains open.Source: G LIMITATIONS AND OUTLOOK · pp. 41–42
  • Sensitivity learning raises FLIPACT sensitivity but does not transfer stably to external benchmarks, where changes differ in sign across students and match the shuffled-label control.The paper reports that FLIPACT labels appear to add little that sharpens decisions to call tools elsewhere.Source: G LIMITATIONS AND OUTLOOK · p. 42
  • The proposed extensions include side effects, interacting tool properties, new axes and domains, and richer state-transition graphs for multi-step agent scenarios.The paper suggests these scenarios may also provide what sensitivity learning needs to transfer beyond FLIPACT.Source: G LIMITATIONS AND OUTLOOK · p. 42

Related papers

Loading 2610.15415v1…