PaperlayerView exact revision on arXiv

Source-linked AI summary

HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment

Kenan Alkiek, Moontae Lee, David Jurgens, V. G. Vinod Vydiswaran

arXiv:2609.38006v1cs.AI

TL;DR

Local language models offer privacy, latency, and cost benefits, but small models struggle with hard queries and cloud escalation gives up local-only advantages. HARISSA uses prefill and answer hidden states to allocate computation and defer predicted-wrong answers; it approaches chain-of-thought accuracy at lower latency and outperforms cascades at equal latency.

  • Problem

    Local-only deployments must decide which hard queries merit extra computation and which answers should be deferred rather than risking a wrong delivery.

    Source: §1 · p. 2

  • Method

    HARISSA fine-tunes a model so its prefill and answer states predict correctness, then uses one cost-ordered cascade to skip predicted failures and defer predicted-wrong answers.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    HARISSA is within one accuracy point of chain-of-thought at 2.7 times lower latency on device, beats FrugalGPT and Self-REF at equal server latency, and leaves fewer wrong answers at equal deferral in five of six pairs.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The model’s own hidden states can guide both local computation allocation and safe answer delivery without adding a second model or generated tokens.

    Source: §1 · p. 2

Abstract

from arXiv · show

Running a language model locally offers advantages in privacy, latency, and cost, but local hardware fits only small models, which are less capable than frontier models. The usual remedy for a hard query, escalating it to a cloud model, gives up the privacy and cost advantages of running locally. A deployment that stays local faces two decisions for hard queries instead. First, it can spend more computation on a query, e.g., reasoning before answering, which raises accuracy at a cost in latency, so it must decide which queries are worth the extra computation (efficiency). Second, some queries are beyond the local model, and delivering a wrong answer is worse than deferring the query to a human in the loop, so it must decide which answers are safe to deliver (safety). We show that both decisions can be made from the model's own hidden states. The prefill state, computed before any token is generated, predicts whether the model will answer correctly, and the answer state, at the end of the generated answer, predicts whether that answer is correct. HARISSA fine-tunes the model so that both states predict correctness, then makes both decisions with one policy that cascades through the ways of answering from cheapest to most expensive, skipping a way the prefill state predicts will fail and deferring the query when the answer it stops with is predicted wrong. On a device running a single model, HARISSA is within one accuracy point of chain-of-thought at 2.7 times lower latency. On a server holding four sizes of one model, HARISSA is more accurate than the FrugalGPT and Self-REF cascades at the same latency, and at the same deferral rate the answer state leaves fewer wrong answers than the standard confidence signals in five of six task and setting pairs.

1 INTRODUCTION

Local deployment protects privacy and offers cost, latency, offline, and energy advantages, but small models struggle with complex queries and cloud escalation sacrifices local-only benefits. HARISSA uses hidden states to allocate local computation and defer answers predicted wrong.

1 INTRODUCTION

  • Local models avoid cloud API costs and network latency, work offline, use less energy per query, and keep data from third parties.Local processing may also matter in domains where data transfer is restricted or confidentiality is required.Source: §1 · pp. 1–2
  • Small local models can fail on complex queries requiring specialized knowledge or multi-step reasoning, while cloud escalation gives up local-only deployment.Local deployments can instead spend more computation through reasoning, multiple samples, retrieval, or larger model sizes, but extra computation raises accuracy at a latency cost.Source: §1 · p. 2
  • A local-only system must decide which queries merit extra computation and which answers are safe to deliver, because wrong answers may be worse than deferral.The model’s prefill state predicts whether it will answer correctly, while its final generated-answer state predicts whether that answer is correct.Source: §1 · p. 2
  • On a device, HARISSA is within one accuracy point of chain-of-thought at 2.7 times lower latency, averaged over three tasks; it also beats two cascades at equal latency on a server.At the same deferral rate, answer probability leaves fewer wrong answers than standard confidence signals in five of six task and setting pairs.Source: §1 · p. 2
  • HARISSA fine-tunes a model to predict correctness from prefill and answer states, then skips predicted failures and defers answers predicted wrong.The policy cascades through answering methods from cheapest to most expensive, and the states are computed as part of answering without a second model or generated tokens.Source: §1 · p. 2

2 RELATED WORK

Prior approaches separately address model routing, computation allocation and deferral, or correctness prediction from hidden states. HARISSA combines prefill and answer-state correctness estimates in one policy for both computation and answer delivery.

Routing and cascading to a larger model

  • Model routing and cascades choose among models, but rely on a larger model for queries the small model cannot answer; HARISSA can skip predicted failures and defer instead.Routers choose before a model runs, whereas cascades typically score the small model’s generated answer before escalating.Source: §2 · p. 3

Deciding how much computation one model spends, and whether to answer

  • Methods that allocate computation and methods that defer answers make separate decisions, whereas HARISSA uses one policy to trade latency and deferral.Existing approaches include spending extra computation on predicted-hard queries and withholding low-confidence answers for human review.Source: §2 · p. 3

Estimating correctness from a model’s own hidden states

  • Prior hidden-state methods estimate correctness before or after generation, while HARISSA fine-tunes one model so both states predict correctness and one policy acts on both.A pre-generation estimate can skip a predicted failure; a post-generation estimate can withhold a predicted-wrong answer.Source: §2 · p. 3

3 METHOD

HARISSA applies a cost-ordered cascade to locally available answering actions, using correctness estimates from prefill and answer states to select actions and decide whether to deliver or defer. Its objective prices wrong answers, latency, and deferrals.

Problem setting

  • The cascade’s actions are ordered by cost and end by delivering one answer or deferring the query to a person.Device actions are direct and chain-of-thought decoding; server actions are model sizes 2B, 4B, 9B, and 27B.Source: §3 · p. 4
  • The policy objective combines delivered errors, latency, and deferrals, with λ pricing a second of latency and µ pricing a deferral in units of one delivered wrong answer.The error and deferral indicators are defined per query, as is its latency.Source: §3 · p. 4

Linear probes

  • The prefill state is the last prompt-token state, and the answer state is the last generated-token state; logistic-regression probes produce calibrated correctness probabilities from them.The prefill probe estimates whether an action would answer correctly, while the answer probe estimates whether its generated answer is correct; probes add no measurable latency.Source: §3 · p. 4

Fine-tuning

  • Fine-tuning teaches hidden states to predict whether each action and each generated answer is correct, using graded answers from the deployment’s training split.Linear heads on prefill and answer states provide the training signal; the heads are trained together with equal-weight binary cross-entropy.Source: §3 · p. 4

The HARISSA policy

  • HARISSA first uses a prefill probability to skip an action predicted to fail, then uses the answer probability to stop, continue, deliver, or defer.For the last action, it generates an answer and stops because no further action remains.Source: §3 · p. 4

Thresholds

  • At each decision, thresholds govern skipping an action, falling through to another action, and deferring rather than delivering an answer.One skip threshold and one fall-through threshold apply across actions; the deferral threshold equals the fall-through threshold.Source: §3 · pp. 4–5

4 EXPERIMENTAL SETUP

The experiments evaluate HARISSA on three benchmarks in device and server local-only settings. Comparisons include direct answering, reasoning, cascades, and alternative confidence signals, with models and thresholds selected using separate data splits.

Datasets and models

  • Experiments use MedQA, MedMCQA, and sixteen multiple-choice BBH subtasks in device and server settings.The device uses Qwen 3.5 4B with direct and think actions; the server uses Qwen 3.5 sizes 2B, 4B, 9B, and 27B with thinking off.Source: §4 · p. 5

Baselines

  • Device baselines include direct, chain-of-thought, self-consistency, FrugalGPT, and Self-REF; server comparisons include FrugalGPT and Self-REF.Deferral signals are compared on the same delivered answers using sequence likelihood, logit margin, and Self-REF confidence.Source: §4 · p. 5

Metrics and protocol

  • Models and probes are fit on training data, calibration and thresholds use development data, and reported numbers are test-split results.Accuracy, wrong answers delivered, deferral share, and mean latency are reported; deferral is off except in Section 5.3.Source: §4 · p. 5

5 RESULTS

HARISSA uses hidden-state probes to allocate computation and defer answers predicted to be wrong. It performs competitively with prior cascades across device and server settings, while its answer probability leaves fewer wrong answers at a fixed deferral rate in five of six task-setting pairs.

5 RESULTS

  • At the same latency, HARISSA is more accurate than FrugalGPT and Self-REF on device and server; at a fixed deferral rate, it leaves fewer wrong answers than standard confidence signals.The answer probability is the deferral signal; it leaves the fewest wrong answers in five of six task-setting pairs.Source: §5 · p. 5, Figure 1 · p. 6, Figure 2 · p. 7

5.1 SINGLE MODEL ON DEVICE

  • Averaged over three device tasks, HARISSA is 0.8 accuracy points below chain-of-thought at 2.7 times lower latency.Figure 1 plots accuracy against mean latency per query, with every query answered.Source: §5.1 · p. 5, Figure 1 · p. 6
  • On MedQA, HARISSA matches FrugalGPT’s accuracy while thinking on 29% of queries versus FrugalGPT’s 75%.Across device tasks, HARISSA is at least as accurate as FrugalGPT and Self-REF at every latency reached by its sweep.Source: §5.1 · pp. 6–7
  • On Gemma 4 and Cogito, HARISSA stays within one accuracy point of chain-of-thought averaged over three tasks, with savings depending on model and task.On Gemma 4, it is 0.7 points above chain-of-thought at 1.1 times lower latency; on Cogito MedQA it thinks on every query and matches chain-of-thought.Source: §5.1 · p. 7

5.2 SERVER CASCADE

  • On server MedQA, HARISSA is 12 points more accurate than FrugalGPT and Self-REF at similar latency; its sweep also stays above theirs across tasks.The server cascade chooses among 2B, 4B, 9B, and 27B sizes; against the 27B size alone, HARISSA is 6.6 points less accurate at 2.6 times lower latency on MedQA.Source: §5.2 · pp. 7–8, Figure 2 · p. 7

5.3 DEFERRING TO THE PERSON

  • At 10% deferred, answer probability leaves the fewest wrong answers in five of six task-setting pairs.On device MedQA, deferring 8.7% of queries removes 32% of otherwise delivered wrong answers, nearly four times the share removed by random deferral.Source: §5.3 · p. 8

6 ANALYSIS

The analysis finds that answer-state probes better predict correctness than prefill probes, while prefill signals reflect both question difficulty and the model’s own knowledge. The cost-sensitive cascade outperforms routing in the reported comparisons, and fine-tuning mainly improves the probes rather than changing answers.

6.1 WHAT DOES EACH HIDDEN STATE KNOW ABOUT THE ANSWER?

  • On device, the answer probe reaches 0.83 AUROC versus 0.63 for prefill on MedQA, and 0.92 versus 0.81 on BBH; server sizes show the same ordering.The question-only classifier is below the prefill probe on every task; prefill reflects both question difficulty and what the model knows.Source: §6.1 · pp. 8–9

6.2 CAN THE PROBES ROUTE INSTEAD OF CASCADE?

  • The answer router gains 1.3 points on MedQA and 1.1 on BBH, loses 1.1 on MedMCQA, and costs about twice the 27B size’s latency.Table 2 reports server test accuracy versus mean latency, using the 27B size on every query as the reference.Source: Table 2 · p. 9, §6.2 · p. 9

6.1 WHAT DOES EACH HIDDEN STATE KNOW ABOUT THE ANSWER?

  • Unlike a question-only classifier, the prefill state reflects the model’s own processing of the question, including its difficulty and what the model knows.The answer state additionally reflects the answer generated, which can provide a correctness signal unavailable before decoding.Source: §6.1 · pp. 8–9
  • The cost sweep sets the device skip threshold to 0 on every task, while the server skips the 2B size on most queries; removing that skip costs 6.3 accuracy points on MedQA at similar latency.Without the answer check, the device policy is less accurate on every task and slower on MedQA and BBH; the server comparison is against generating at 2B for every query.Source: §6.1 · p. 9
  • Fine-tuning raises the answer probe by 0.12 AUROC on device MedQA while each action’s accuracy moves by at most 1.7 points in either setting.The frozen-probe policy is 2.5 points less accurate on device MedQA, 5.5 on MedMCQA, and 2.6 on BBH.Source: §6.1 · p. 9

7 CONCLUSION

HARISSA uses a small model’s hidden states to allocate computation and decide whether to deliver an answer or defer it. It approaches chain-of-thought accuracy at lower device latency and beats two server cascades at the same latency.

7 CONCLUSION

  • On device, HARISSA is within one accuracy point of chain-of-thought at 2.7 times lower latency; on server, it is more accurate than FrugalGPT and Self-REF at the same latency.At the same deferral rate, the answer probability leaves fewer wrong answers than standard confidence signals in five of six task and setting pairs.Source: §7 · p. 10

AI USE STATEMENT

The authors report using generative AI for drafting, implementation, and parts of the analysis and experiment process, while excluding data generation and claim formulation or proof.

AI USE STATEMENT

  • Generative AI tools helped draft and edit the text, review structure and prose, and implement the method and reporting and figure scripts.The tools also helped in part with data analysis, result interpretation, and experiment design.Source: AI USE STATEMENT · p. 10
  • The authors state that they did not use generative AI to generate data, formulate or prove claims, or clean datasets.Source: AI USE STATEMENT · p. 10
  • The authors say they checked every number against tables printed by their code, reviewed AI-assisted text, and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A DATASETS AND PROMPTS

The appendix specifies the evaluation tasks, prompts, decoding settings, fine-tuning procedure, and probe configuration. It reports that BBH evaluation uses sixteen multiple-choice subtasks and describes the prompts and model-state probes used.

A DATASETS AND PROMPTS

  • Table 3 reports split sizes, and the BBH evaluation uses sixteen multiple-choice subtasks spanning tracking, deduction, interpretation, and reasoning tasks.Source: Table 3 · p. 15, A DATASETS AND PROMPTS · p. 15
  • Every action uses a zero-shot prompt with one reasoning hint per task; the medical-task hint requests step-by-step analysis of the scenario and answer choices.Source: A DATASETS AND PROMPTS · p. 15
  • Models decode at temperature 0.6 with a 10,000-token response cap, including thinking; on device, direct and think differ only in the chat-template thinking flag.Source: A DATASETS AND PROMPTS · p. 15
  • The prompt contains a question with lettered options and requires JSON output with a reasoning field and a final-answer letter.Source: A DATASETS AND PROMPTS · p. 15

B FINE-TUNING AND PROBE DETAILS

  • LoRA fine-tuning trains answer and per-action prefill heads on correctness labels from stored answers, then discards the heads before fitting probes.The appendix specifies separate fine-tuning for each server size and one forward pass over the prompt and stored answer.Source: B FINE-TUNING AND PROBE DETAILS · p. 15
  • Each size uses a fixed probed layer, and its probe is an L2-regularized logistic regression calibrated by isotonic regression on the development split.The layer is selected per size, with Gemma 4 probed at its final layer and Cogito at layer 32.Source: B FINE-TUNING AND PROBE DETAILS · p. 15

C COST MODEL

The cost model estimates query latency from per-size prefill time and generated tokens, using measured rates. It also accounts for probe and skipped-action costs.

C COST MODEL

  • Query latency combines per-size prefill time with generated tokens multiplied by the per-token generation rate.Both rates were measured per model size on one NVIDIA A40 with vLLM, BF16, and concurrency one.Source: C COST MODEL · p. 15
  • Each probe costs one dot product on a state computed while the model answers.Source: C COST MODEL · p. 16
  • A skipped action costs its prefill pass and probe: 0.08 s for device direct and 0.23 s for the server’s 2B, 4B, and 9B sizes together.The costs are measured by subtracting the last action’s latency from the policy latency when every earlier action is skipped.Source: C COST MODEL · p. 16
  • Table 5 reports mean input tokens, mean generated tokens per query, and each action’s standalone latency on the Qwen 3.5 test split.The 4B action is the device model’s direct action, so the corresponding columns agree.Source: Table 5 · p. 16

D FULL RESULTS IN BOTH SETTINGS

The full-results tables cover ablations, delivered-action shares, and selected accuracy–latency operating points. Thresholds are selected using different latency prices for device and server settings.

D FULL RESULTS IN BOTH SETTINGS

  • Table 6 compares accuracy and mean latency for ablations, individual model sizes, and HARISSA without fine-tuning at selected thresholds.Thresholds are selected at 10 s per accuracy point on device and 0.5 s on server; the server policy without the answer check is the prefill router of Table 2.Source: Table 6 · p. 16
  • Table 7 reports the share of queries answered by each cascade action at the selected thresholds.Source: Table 7 · p. 17
  • Table 8 reports HARISSA’s selected accuracy–latency point at each latency price, with the chosen skip threshold in parentheses.The reported prices are 10 s per accuracy point on device and 0.5 s on server.Source: Table 8 · p. 17

E REPLICATION ON OTHER MODEL FAMILIES

The supplementary results extend device experiments to two other model families, describe deferral at the selected threshold, and report fine-tuning’s effects on probes and answer accuracy.

E REPLICATION ON OTHER MODEL FAMILIES

  • Device experiments are repeated on Gemma 4 E4B and Cogito v1 8B using the same tasks, splits, actions, recipe, probes, threshold selection, and cost model.Table 9 reports selected-threshold accuracy and latency, HARISSA’s share of queries thought on, and fine-tuned probe AUROC averaged over the two actions.Source: E REPLICATION ON OTHER MODEL FAMILIES · p. 17, Table 9 · p. 17

F DEFERRAL

  • Table 10 reports deferred-query share and wrong answers delivered, with and without deferral, at the selected fall-through threshold.The deferral threshold is set equal to the fall-through threshold, so the answer probability that discards an answer earlier also withholds it at the last action.Source: F DEFERRAL · p. 18, Table 10 · p. 18

G FINE-TUNING’S EFFECT ON THE PROBES AND THE ANSWERS

Loading 2609.38006v1…