PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidence-Grounded SLM Behavioral Prediction Architecture for Future Assistance Use

Adeyemi Adeseye, Aisvarya Adeseye, Hannu Tenhunen, Jouni Isoaho

arXiv:2610.32193v1cs.AI

TL;DR

Reliable local SLM reasoning over low-level sequential interaction events remains underexplored, despite the importance of anticipating assistance use in high-stakes settings. ELBP is a four-layer local SLM architecture combining deterministic interaction processing, dual-view representation, evidence-grounded prediction, and verification with abstention. B2 reached full-set F1 of 0.80; B3 achieved selective F1 of 0.84 at 0.91 coverage, with evidence support of 0.96.

  • Problem

    Reliable local SLM reasoning over low-level sequential interaction events remains underexplored, despite the importance of anticipating assistance use in high-stakes settings.

    Source: §I · p. 1

  • Method

    ELBP is a four-layer local SLM architecture combining deterministic interaction processing, dual-view representation, evidence-grounded prediction, and verification with abstention.

    Source: §I · p. 1, §III · p. 2

  • Results

    B2 reached full-set F1 of 0.80; B3 achieved selective F1 of 0.84 at 0.91 coverage, with evidence support of 0.96.

    Source: B. Architectural Benchmark Comparison · p. 5

  • Takeaways & Limitations

    Structured behavioral representation can improve prediction over raw interaction logs, while verification improves reliability.

    Source: §VI · p. 6

    The study evaluates only one behavioral domain and prediction target: assistance-seeking.

    Source: §VI · p. 6

Abstract

from arXiv · show

Users often exhibit changes in interaction behavior before explicitly seeking assistance, enabling systems to anticipate future assistance use. This is especially important in high-stakes environments, where delayed assistance can have operational, financial, or safety consequences. However, conventional Large Language Models (LLMs) are often unsuitable for continuous on-device behavioral prediction because of their computational and memory demands. Behavioral patterns are also highly individual and device-specific, motivating local processing that preserves privacy and data sovereignty. Small Language Models (SLMs) offer a promising alternative, but their ability to reason reliably over low-level sequential interaction events remains underexplored. This study introduces Evidence-Grounded Local Behavioral Prediction (ELBP), a four-layer architecture combining deterministic interaction processing, dual-view behavioral representation, SLM-based prediction, and evidence verification with abstention. Three instruction-tuned models of 3B parameters or fewer were evaluated using raw, structured, dual-view, and complete ELBP configurations. The best-performing model, Qwen2.5-3B with ELBP, achieved a full-set F1-score of 0.80 and PR-AUC of 0.84, representing a 15.9\% relative improvement over raw-event prompting and 17.6\% over XGBoost, while full-set ECE decreased by 43.2\% relative to raw-event prompting. With evidence verification and abstention, selective F1 increased to 0.84 at 91\% coverage and evidence support reached 96\%. These results show that structured behavioral representation and evidence-verified selective prediction can improve the reliability of behavioral forecasting in SLMs.

I. INTRODUCTION

Interaction changes may precede assistance-seeking, but reliable local prediction from low-level sequential events remains underexplored. This study evaluates a four-layer local SLM architecture for forecasting future assistance use.

I. INTRODUCTION

  • Interaction changes may precede assistance-seeking, which matters in high-stakes settings where delayed assistance can have operational, financial, or safety consequences.Prior studies report that initial problem-solving attempts and hint-use patterns can predict later assistance-seeking, and sequential interaction traces can relate earlier actions to later help-seeking.Source: §I · p. 1
  • SLMs may support local behavioral prediction, but their reliable reasoning over long, low-level sequential interaction events remains underexplored.Conventional LLMs are often impractical for continuous on-device use because of computational and memory demands; local processing also supports privacy and data sovereignty.Source: §I · p. 1
  • The study develops and evaluates ELBP, a local SLM architecture for predicting future assistance use from sequential interaction traces.Its four layers combine deterministic behavioral processing, dual-view interaction representation, evidence-grounded prediction, and post-prediction verification with abstention.Source: §I · p. 1

II. RELATED WORK

Prior work examines help-seeking prediction, LLM-based behavior modeling, and local inference concerns, but not local SLMs predicting near-term behavior from low-level interaction events. ELBP combines local SLM prediction with structured representations and evidence grounding, while deterministic preprocessing converts event logs into ordered behavioral observations.

II. RELATED WORK

  • Earlier studies used conventional models, engineered features, and LLMs for assistance prediction or behavioral modeling, but did not examine local SLMs on low-level events for short-horizon prediction.Prior work also found that summarized interaction histories can affect LLM decisions, while local deployment raises computation, memory, latency, and privacy concerns.Source: §II · pp. 1–2

III. EVIDENCE-GROUNDED LOCAL BEHAVIORAL PREDICTION ARCHITECTURE

  • ELBP combines local SLMs, structured behavioral representations, evidence grounding, and abstention to predict near-term assistance-seeking behavior.The architecture separates deterministic interaction processing and representation from SLM-based prediction and evidence verification.Source: §II · p. 2, §III · p. 2

A. Layer 1: Interaction Processing

  • Layer 1 deterministically orders and groups events, calculates temporal intervals, normalizes labels, handles missing values, and derives task-level outcomes.Its purpose is to turn low-level logs into reliable behavioral observations while preserving temporal ordering and provenance.Source: A. Layer 1: Interaction Processing · p. 2

B. Layer 2: Behavior Representation

Layer 2 represents each interaction history through chronological evidence and a structured behavioral summary. The chronological view preserves event order and identifiers so later predictions can be tied to specific observations.

B. Layer 2: Behavior Representation

  • Layer 2 produces a dual-view representation that combines chronological evidence with a structured behavioral summary.The design assumes the views are complementary: one preserves event order, while the other exposes higher-level patterns that an SLM would otherwise calculate.Source: B. Layer 2: Behavior Representation · p. 2

1) Chronological Evidence Representation:

  • Chronological evidence assigns unique identifiers to observations, preserving trajectories and transitions that aggregate statistics may lose.Examples include repeated failures, increasing hesitation, recovery after difficulty, and transitions from stable to unstable interaction behavior.Source: 1) Chronological Evidence Representation · p. 2

2) Structured Behavioral Summary:

ELBP supplies a deterministic behavioral summary alongside the chronological evidence for local SLM prediction. The model returns a prediction with confidence, evidence identifiers, and a justification so its basis can be inspected.

2) Structured Behavioral Summary:

  • The structured summary exposes higher-level behavioral patterns through deterministic variables such as performance, response duration, revisions, retries, and prior assistance-seeking.It abstracts the event sequence and reduces low-level computation required of an SLM with limited reasoning capacity and context budgets.Source: 2) Structured Behavioral Summary · p. 2

C. Layer 3: Local SLM Behavior Prediction

  • Layer 3 uses a local SLM to predict a predefined near-term outcome from both views and return a class, confidence, cited evidence identifiers, and justification.Requiring evidence identifiers makes each prediction auditable by connecting the output to the supplied behavioral trace.Source: C. Layer 3: Local SLM Behavior… · p. 3
  • The prediction layer uses a fixed prompt and does not require model fine-tuning, enabling multiple local SLMs to be evaluated under identical prompting conditions.The implementation uses llama.cpp, though other inference frameworks may be used depending on the model and hardware.Source: C. Layer 3: Local SLM Behavior… · p. 3

D. Layer 4: Evidence Verification and Decision

Layer 4 checks whether predictions are structurally valid and whether their justifications are supported by cited interaction evidence. It combines deterministic checks with focused SLM verification before the final decision.

D. Layer 4: Evidence Verification and Decision

1) Structural Verification:

  • Structural verification checks schema compliance, label validity, confidence range, required evidence identifiers, and whether cited identifiers occur in the interaction history.These checks detect malformed outputs and fabricated evidence references without another SLM call.Source: 1) Structural Verification · p. 3

2) Semantic Evidence Verification:

  • Deterministic identifier checks do not establish that cited evidence supports the interpretation; a focused SLM verifier classifies support as SUPPORTED, NOT SUPPORTED, or UNCERTAIN.The verifier assesses the justification and does not generate a new prediction.Source: 2) Semantic Evidence Verification · p. 4

3) Selective Final Decision:

The final decision combines verification outcomes and model confidence through deterministic rules. The development-set confidence threshold is selected to maximize selective F1 while retaining at least 90% coverage, then fixed for testing.

3) Selective Final Decision:

IV. EXPERIMENTAL SETUP, BENCHMARKS, AND EVALUATION

The evaluation uses learner interaction histories from EdNet-KT4 and a future voluntary-assistance label defined over subsequent tasks. Models and benchmark configurations are compared under consistent local inference and learner-disjoint testing conditions.

A. Testbed and Gold Standard

  • EdNet-KT4 contains 131,441,538 interaction records from 297,915 learners; 70% of learners were sampled with seed 42, retaining their complete histories.Deterministically derived fields include correctness, duration, revisions, reversals, recent correctness, and prior assistance use.Source: A. Testbed and Gold Standard · p. 4
  • A positive instance means a learner voluntarily checks an explanation or lecture within the five tasks after a prediction point; system-initiated assistance is excluded.Prediction points occur at task ends, and the preceding five tasks provide the interaction history.Source: A. Testbed and Gold Standard · p. 4
  • The learner-disjoint training, development, and test split is 70/15/15, preventing histories from the same learner appearing across partitions.Source: A. Testbed and Gold Standard · p. 4

B. Local SLM Configuration

  • Three instruction-tuned models—Llama-3.2-1B-Instruct, Gemma-2-2B-IT, and Qwen2.5-3B-Instruct—run locally with comparable 4-bit quantization and no fine-tuning.The same task instruction, schema, decoding settings, and experimental instances are used across models.Source: B. Local SLM Configuration · p. 4

C. Benchmark Configurations

  • B0 uses raw events, B1 a behavioral summary, B2 both views, and B3 adds evidence attribution, verification, and abstention; XGBoost uses B1 variables as a non-SLM baseline.The same learner-disjoint test partition is retained for final evaluation.Source: C. Benchmark Configurations · p. 4

D. Evaluation Metrics

Evaluation covers classification performance, calibration, evidence support, coverage, and local deployment efficiency. The metrics distinguish complete-test-set performance from selective performance after abstention.

D. Evaluation Metrics

  • F1 is reported alongside Precision and Recall, while PR-AUC measures performance across classification thresholds and suits imbalanced positive events.Source: D. Evaluation Metrics · p. 4
  • Expected Calibration Error (ECE) measures calibration using confidence bins and their empirical accuracy and mean confidence.Source: D. Evaluation Metrics · pp. 4–5
  • Evidence Support and Coverage measure ELBP reliability, with coverage counting accepted predictions relative to accepted and abstained predictions.Selective F1 is calculated only on non-abstained predictions.Source: D. Evaluation Metrics · p. 5
  • Local Deployment Efficiency is reported using inference latency, throughput in tokens per second, and peak VRAM usage.Source: D. Evaluation Metrics · p. 5

V. RESULTS

ELBP improves full-set prediction across all three local SLMs, with Qwen2.5-3B achieving the strongest reported results. Evidence verification and abstention further reduce calibration error on retained predictions.

A. Overall Prediction Performance

B. Architectural Benchmark Comparison

For Qwen2.5-3B, structured behavioral summaries and chronological evidence improve prediction over raw-event reasoning, while verification and abstention raise selective performance. Ablations show contributions from both behavioral views and a performance–coverage trade-off.

B. Architectural Benchmark Comparison

C. Ablation Study

  • Removing the structured summary reduced full-set F1 from 0.80 to 0.74 and selective F1 from 0.84 to 0.77; removing chronological evidence reduced them to 0.77 and 0.80, respectively.Both ablations remained above the XGBoost full-set F1 of 0.68 when the summary was removed.Source: C. Ablation Study · p. 5, Table III · p. 6
  • Verification increases coverage from 0.91 to 0.93 and ECE from 0.051 to 0.064; removing abstention entirely restores coverage to 1.00, with full-set F1 of 0.80.The difference between full-set F1 of 0.80 and selective F1 of 0.84 quantifies the performance–coverage trade-off introduced by selective prediction.Source: C. Ablation Study · p. 6

D. Local Deployment Efficiency

Qwen2.5-3B provides the highest selective performance among the evaluated ELBP models, but at higher computational cost. The study’s conclusion is limited to one behavioral domain and prediction target.

D. Local Deployment Efficiency

  • 0.84 selective F1 at 0.91 coverage was Qwen2.5-3B’s highest ELBP result, with 1.87 s latency and 2.8 GB peak VRAM.Source: D. Local Deployment Efficiency · p. 6
  • XGBoost averaged 0.03 s per instance but had lower predictive performance; among local SLMs, Llama-3.2-1B was the lowest-latency option and Qwen2.5-3B the highest-performance, higher-cost option.Source: D. Local Deployment Efficiency · p. 6

VI. CONCLUSION

  • ELBP combines deterministic interaction processing, dual-view behavioral representation, evidence-grounded prediction, and post-prediction verification with abstention.Source: §VI · p. 6
  • The evaluation covers a single behavioral domain and prediction target; future work will test generalization to additional domains and outcomes.Source: §VI · p. 6

Related papers

Loading 2610.32193v1…