PaperlayerView exact revision on arXiv

Source-linked AI summary

Human--AI Interactions as First-Class Data: A Formal Interaction Model and Runtime for Explainable AI Collaboration

Michela Taufer, Kin Ng, Ian Lumsden, Aashish Pandey, Rajasi Gore, Angelina Ju, Sukanya Moudgalya, Ewa Deelman, Sai Swaminathan

arXiv:2610.40006v1cs.AIcs.HC

TL;DR

Transient prompt-response records omit the computational state and user actions needed to understand human–AI collaboration. The paper introduces HAIDL, a structured interaction model, and HAIDL-X, a JupyterLab runtime that captures and stores interactions. A classroom case study uses interaction and execution data to derive behavioral patterns, while reported system measurements characterize the persistence cost.

Abstract

from arXiv · show

AI assistants are increasingly embedded in computational notebooks, scientific workflows, software engineering tools, and learning systems, yet human-AI interactions are often treated as transient prompt-response exchanges rather than durable data products, hindering reproducibility, explainability, replay, behavioral analysis, and adaptive AI support. We argue that these interactions form a new data class requiring formal representation and introduce the Human-AI Interaction Description Language (HAIDL), a formal interaction model with a Backus-Naur Form-inspired concrete syntax. HAIDL encodes each interaction as a structured object capturing computational context, user request, prompt payload, AI response, feedback, problem context, provenance, and model metadata. We integrate this vision into HAIDL-X, a middleware for JupyterLab that instantiates HAIDL by capturing user-bot state, building contextualized prompts, interfacing with AI providers, collecting feedback, and storing interactions in a repository. We add a learning layer over HAIDL that maps interaction objects and notebook execution traces into reproducible measures of user behavior, including engagement, performance, help-seeking, error-oriented use, and learning personas. HAIDL-X provides a foundation for treating human-AI collaboration as structured, queryable, replayable, analyzable data.

I. INTRODUCTION

Prompt-response logs cannot explain how AI assistance relates to computational state, user actions, or task context. The paper proposes HAIDL as a structured interaction representation, instantiates it in JupyterLab, and uses it with execution evidence to derive behavioral knowledge.

I. INTRODUCTION

  • Existing logs omit the computational state and downstream actions needed to distinguish independent progress from silent struggle or debugging from answer extraction.The paper identifies this as a barrier to reconstructing why assistance failed, misled, or went unused.Source: §I · p. 1
  • HAIDL represents each exchange as a structured object spanning the motivating state, contextualized prompt, AI response, user assessment, and provenance.Its BNF-inspired grammar covers seven semantic components and separates observed data from inferred knowledge.Source: §I · p. 2
  • HAIDL-X captures notebook context, builds contextualized prompts, communicates with AI providers, collects feedback, and persists interaction objects.The paper also measures the system cost of turning transient exchanges into persistent records.Source: §I · p. 2
  • Composing HAIDL objects with execution evidence can recover request roles, assignment-level use trajectories, and collaboration patterns, including among students with no recorded interactions.This analysis distinguishes students progressing independently from those stuck without seeking help.Source: §I · p. 2

II. HUMAN–AI INTERACTIONS AS FIRST-CLASS DATA

  • An interaction is modeled as a state-changing episode: a request arises from computational state, AI guidance is assessed or acted on, and the resulting state begins the next exchange.Ordered interactions form trajectories that support analysis of strategies, learning behaviors, and AI effectiveness over time.Source: §II · p. 2

III. HUMAN–AI INTERACTION DESCRIPTION LANGUAGE

HAIDL formalizes human–AI collaboration as provider- and runtime-independent structured data. Its design principles, seven-component interaction object, and recursive grammar preserve the lifecycle details needed to query and recompose interactions.

A. Design Principles

  • HAIDL combines completeness, modularity, explainability, extensibility, and composability in a provider- and runtime-independent formal language.The language uses a BNF-inspired concrete syntax and semantics relating interaction constructs to computational state.Source: §III · p. 2, A. Design Principles · pp. 2–3

B. Interaction Objects

  • A single interaction object groups record metadata, user request, prompt payload, AI response, feedback, problem context, and model metadata.Problem context matters because the same observable exchange can mean different things for different tasks.Source: B. Interaction Objects · p. 3

C. Recursive Grammar

  • The recursive grammar decomposes interactions into components that separately represent computational context, prompts, responses, feedback, task context, and model provenance.Explicit component semantics let these fields be queried independently and recomposed.Source: C. Recursive Grammar · pp. 3–4

D. From Interaction Objects to Interaction Expressions

HAIDL composes single interactions into temporally ordered trajectories linked by observable execution states. A separate derivation relation turns relationships among recorded observations into inspectable higher-level knowledge.

D. From Interaction Objects to Interaction Expressions

  • Structural composition builds interaction objects, while temporal composition links them through states, timestamps, users, and problem contexts into trajectories.ExecutionState supplies observable computational links between episodes.Source: D. From Interaction Objects to… · p. 4

E. From Syntax to Knowledge

  • The derivation relation maps interaction-data expressions to knowledge, such as error recovery or patterns across successive interactions, without conflating inference with grammar syntax.The meaning of a derived pattern depends on its rule and domain, including repeated help seeking, validation, or progress following assistance.Source: E. From Syntax to Knowledge · p. 4

IV. HAIDL-X: AN AI MW FOR SCIENTIFIC COMPUTING

HAIDL-X realizes HAIDL as a layered JupyterLab middleware that captures and persists structured interaction records during computational work. Its capture pipeline separates the interaction model from provider-specific execution.

IV. HAIDL-X: AN AI MW FOR SCIENTIFIC COMPUTING

  • HAIDL-X is a layered middleware for JupyterLab that captures each interaction as a structured, persistent record during execution.The architecture separates user interaction, capture, AI communication, representation, and storage so layers can evolve independently around one interaction model.Source: §IV · p. 4

A. Software Architecture

  • The six layers include a notebook extension, context-capturing client, backend, HAIDL interaction layer, provider abstraction, and repository.The repository stores provenance, timestamps, user assessments, and model information for replay, querying, and behavioral analytics.Source: A. Software Architecture · p. 4

B. Capture Pipeline, Repository, and Provider Independence

V. DEPLOYMENT AND DATA COLLECTION

The classroom deployment paired HAIDL-X interaction records with notebook execution evidence to study authentic student computational work. The data were aligned by student–assignment pair while preserving interaction order and assigned access condition for later derivation.

A. Classroom Deployment

  • HAIDL-X was integrated into JupyterLab and used during regular assignments by 35 students across four assignments, yielding 140 student–assignment pairs.Assignments ranged from mathematical and text-processing tasks to distributed PySpark work.Source: A. Classroom Deployment · p. 5

B. Experimental Design

  • The crossover design assigned HAIDL-X access to different assignments for Groups A and B, while retaining cases where actual use differed from assigned access.The student–assignment pair, rather than the student alone, is the primary analysis unit.Source: B. Experimental Design · p. 5

C. Data Sources

  • HAIDL-X interaction records included available notebook context, prompts, responses, timestamps, model and latency metadata, and explicit feedback.These records support AI-use measures and semantic interaction events.Source: C. Data Sources · p. 5
  • Notebook traces capture computation even without AI use, while HAIDL objects capture requested assistance; neither source alone provides the combined derivation view.Execution success is an observable proxy for progress, not proof of a logically correct solution.Source: C. Data Sources · p. 5

D. Data Preparation and Alignment

  • The analysis record preserves notebook execution, HAIDL interaction objects, and assigned access separately, composing them only through explicit derivation rules.Records are organized by student–assignment pair, with interactions ordered by timestamp and linked to notebook cells when identifiers are available.Source: D. Data Preparation and Alignment · p. 5

VI. FROM INTERACTION DATA TO KNOWLEDGE

  • HAIDL’s knowledge layer composes interaction primitives with domain-specific observations to derive information not present in any single interaction record.The classroom learning scenario illustrates this composition rather than limiting HAIDL to learning applications.Source: §VI · p. 5

A. From Observations to Derived Knowledge

HAIDL distinguishes captured observations from knowledge derived by relating interaction objects to computational state. It derives event roles from individual exchanges and trajectories from events connected over time.

A. From Observations to Derived Knowledge

  • InteractionKnowledge is derived from observations rather than stored as an additional label, with ErrorRecovery and RepeatedHelp illustrating relations to execution evidence and progress.An error output is observed data; using AI assistance in response to it is a derived event.Source: A. From Observations to Derived… · p. 6

B. Deriving Interaction Events

  • Interaction events classify an exchange as ErrorRecovery, StuckNextStep, ConceptualHelp, or Validation using interaction components and notebook state.These categories describe HAIDL-X’s observable workflow role, not the user’s underlying intent, and are intended to be derivable and reproducible.Source: B. Deriving Interaction Events · p. 6

C. Composing Events into Interaction Trajectories

  • Timestamp-ordered interactions form trajectories from which HAIDL derives constructs such as RepeatedHelp and NoProgress across subsequent computational states.RepeatedHelp involves help-seeking after no progress; NoProgress connects an interaction to execution without observable progress.Source: C. Composing Events into Interaction… · p. 6

D. Composing Interaction and Domain Knowledge

The learning layer combines notebook-derived performance and engagement measures with HAIDL interaction measures. It uses performance and engagement levels to form relative learning profiles.

D. Composing Interaction and Domain Knowledge

  • Performance and engagement measures use notebook execution evidence, with successful execution treated as a proxy rather than proof of semantic correctness.Engagement is the proportion of assignment code cells touched; performance is based on correctly solved cells.Source: D. Composing Interaction and Domain… · p. 6
  • The measures produce four relative learning profiles: HPHE, HPLE, LPHE, and LPLE, crossing high or low performance with high or low engagement.The abbreviations correspond respectively to high-performance/high-engagement, high-performance/low-engagement, low-performance/high-engagement, and low-performance/low-engagement.Source: D. Composing Interaction and Domain… · p. 6
  • HAIDL interaction objects supply complementary AI-use measures, including the number of interactions for a student–assignment pair.These measures remain traceable to notebook observations or HAIDL components.Source: D. Composing Interaction and Domain… · p. 6

E. Deriving Behavioral States

The behavioral-state layer combines exposure status, learning profile, and interaction trajectories to interpret behavior for each student–assignment pair. These states are reproducible interpretations, not diagnoses or intrinsic traits.

E. Deriving Behavioral States

  • Exposure status distinguishes assigned users who used HAIDL-X, assigned non-users, controls without access or use, and users who crossed their assigned condition.Assigned access and actual use are separately represented.Source: E. Deriving Behavioral States · p. 6
  • Behavioral interpretations include self-sufficient students, disengaged strugglers, scaffolded learners, exploratory over-users, and cross-condition users.Expected non-users with high performance are interpreted as self-sufficient, while low performance supports a disengaged-struggler interpretation; productive help-seeking or sustained engagement can indicate scaffolded learning among expected users with low performance.Source: E. Deriving Behavioral States · p. 7
  • These behavioral states are reproducible interpretations of observable behavior for a student–assignment pair, not diagnoses or intrinsic student traits.Source: E. Deriving Behavioral States · p. 7

F. What the Language Reveals

HAIDL’s compositional representation links observations, interactions, events, and trajectories to behavioral knowledge through inspectable derivation rules. In the classroom case, combining interaction records with notebook and experimental evidence distinguishes behaviors that logs or traces alone leave ambiguous.

F. What the Language Reveals

  • HAIDL composes observations into interactions, events, trajectories, and behavioral knowledge, while distinguishing stored interaction data from inference.The same compositional language can express domain-specific knowledge for education, scientific workflows, software debugging, and other collaboration settings.Source: F. What the Language Reveals · p. 7

VII. CASE STUDY: DERIVING INTERACTION KNOWLEDGE

  • The case study applied the framework to 140 student–assignment pairs, combining notebook execution data, interaction logs, and access conditions to distinguish behavior ambiguous in either source alone.The analysis tests the value of first-class interaction data, not HAIDL-X’s causal effect on performance.Source: §VII · p. 7

A. Interaction Behavior and Learning Profiles

  • Interaction roles varied by assignment: requests included next-step help, validation, conceptual help, and error recovery, distinctions hidden by prompt counts.HAIDL links requests to execution context to capture these differences.Source: A. Interaction Behavior and Learning… · p. 7
  • Spark assignments surfaced 8 LPHE pairs in Assignment 07 and 6 in Assignment 08, representing engaged struggle distinct from low activity and low performance.Across 140 pairs, interaction count correlated only weakly with successfully executed cells (r = 0.12) and touched cells (r = 0.12).Source: A. Interaction Behavior and Learning… · p. 7

B. Exposure and Derived Behavioral States

  • Students with access but no interactions and users outside their assigned-access condition show why availability, observed use, and help-seeking must be distinguished.The study identifies 37 access-without-interaction pairs and 26 cross-condition pairs; the latter were analyzed separately.Source: B. Exposure and Derived Behavioral… · pp. 7–8

C. What the Derived Knowledge Reveals

  • Combining interaction data, notebook evidence, and experimental context yields behavioral knowledge that none of those sources provides alone.Each behavioral state traces to specific HAIDL fields, notebook observations, and derivation rules; these results do not show that HAIDL-X improves learning.Source: C. What the Derived Knowledge Reveals · p. 8

VIII. COST OF FIRST-CLASS INTERACTION CAPTURE

This section measures the runtime and storage costs of preserving interactions, rather than comparing providers or model quality, and finds low capture, persistence, and retrieval costs.

A. Capture Overhead

  • Capture overhead (Tserver − TLLM) grows steadily with concurrency but stays within 25 ms across providers and configurations.Tserver covers the full request callback; TLLM measures the model invocation alone.Source: A. Capture Overhead · p. 8, Figure 3 · p. 9

B. Persistence Cost

C. Retrieval Cost

  • Three retrieval queries have similar low cost, with median latency around 0.4 ms and P95 below 0.8 ms.The queries retrieve a student's assignment interactions, interactions containing ErrorOutput, and temporally ordered interactions.Source: C. Retrieval Cost · p. 9, Table III · p. 9

D. Discussion

  • HAIDL-X trades measurable systems cost for persistent interaction data supporting queries, reconstruction, and knowledge derivation.Preserving computational state, context, provenance, and temporal structure supports knowledge that is otherwise hard to reconstruct.Source: D. Discussion · p. 9

IX. RELATED WORK

Related work spans programming assistants, provenance and observability, and interaction-data standards, but the reviewed representations lack HAIDL's combination of human-side context and derivation semantics. The conclusion frames HAIDL as a data model for reproducible, evidence-traceable interpretation of collaboration.

A. AI Assistants and Programming Support

  • Programming-support analyses infer usage from conversation logs [7], IDE telemetry, and corpora, but do not link requests to motivating computational state; HAIDL couples request, response, context, and assessment.Source: A. AI Assistants and Programming Support · p. 9

B. Provenance and AI Observability

  • Provenance and observability systems cover processes, agent prompts, or model-invocation telemetry,, [5],, while HAIDL adds user computational state, assessment, and derivation semantics.Source: B. Provenance and AI Observability · p. 9

C. Interaction Data Standards and Formal Models

X. CONCLUSION

  • HAIDL objects and notebook execution context support evidence-traceable interpretation of behaviors such as error recovery, next-step assistance, conceptual help, and validation, without claiming causal learning effects.Source: §X · p. 10
Loading 2610.40006v1…