PaperlayerView exact revision on arXiv

Source-linked AI summary

Sensor-Language-Action Models

Yuekai Xu, Zitao Shuai, Yuzhe Yang

arXiv:2610.08244v1cs.AI

TL;DR

Existing sensor models largely focus on perception, while actions remain separately modeled through task-specific labels. This paper connects sensors, language, and actions in SLA and presents OpenSLA, which outperforms state-of-the-art methods across diverse real-world tasks and generalizes to unseen actions and cohorts.

  • Problem

    Sensor models largely focus on understanding observations, while actions are often modeled separately through task-specific, fixed labels.

    Source: §1 · p. 1

  • Method

    SLA connects sensor observations, natural-language context, and actions through a shared interface, with a benchmark, multi-faceted language supervision, and the OpenSLA model.

    Source: §1 · p. 2

  • Results

    OpenSLA achieves superior performance over state-of-the-art methods across diverse real-world tasks and shows language-guided evidence grounding and zero-shot generalization to unseen actions and cohorts.

    Source: §1 · p. 3

  • Takeaways & Limitations

    SLA provides a foundation for general sensor models that jointly reason about observations, language, and actions.

    Source: §7 · p. 13

    OpenSLA has not undergone clinical validation and is not intended for direct clinical decision-making; deployment would require prospective validation, safety evaluation, and regulatory review.

    Source: §7 · p. 13

Abstract

from arXiv · show

Sensors are useful not only for understanding the world but also for deciding what to do next. Existing sensor models however largely stop at perception: they recognize states or predict outcomes, leaving actions modeled separately through task-specific and often closed label spaces. We introduce Sensor-Language-Action (SLA) modeling, a framework that connects multimodal sensor observations, natural language, and actions within a unified model. SLA uses language as a semantic interface between sensing and acting, allowing heterogeneous actions to be represented, predicted, and explained while remaining grounded in the underlying sensor evidence. We build a large-scale SLA benchmark consisting of datasets that span more than 116,000 individuals, 79 sensor modalities, and 60 action groups, together with a multi-faceted captioning pipeline that aligns user context, sensor dynamics, and action evidence. Building on this framework, we present OpenSLA, a unified SLA model for hierarchical action prediction, state understanding, and action explanation. Extensive experiments on real-world tasks in clinical prediction, operating rooms, and metabolic health verify its superior performance over the state-of-the-art. OpenSLA also demonstrates intriguing capabilities including language-guided evidence grounding and zero-shot generalization to unseen actions and cohorts.

1. Introduction

Existing sensor models largely separate perception from action, leaving no common mechanism to connect heterogeneous sensor evidence, context, and decisions. The paper introduces SLA and OpenSLA to bridge them, with a large benchmark, multifaceted language supervision, and reported advantages over state-of-the-art methods.

1. Introduction

  • Existing sensor models often stop at state recognition or outcome prediction, while downstream decisions use separate task-specific action spaces without a shared way to connect evidence, context, and actions.Language-based sensor models have enabled richer descriptions, but these advances also largely stop before action.Source: §1 · p. 1
  • SLA connects sensor observations, natural-language context, and actions through a common multimodal interface, and OpenSLA instantiates this framework for action prediction, state understanding, and action explanation.The framework uses language to relate sensing and acting.Source: §1 · p. 2
  • The study formulates Sensor-Language-Action (SLA) modeling as a framework unifying sensor-action problems through a multimodal language interface and builds a benchmark spanning more than 116,000 individuals, 79 sensor modalities, 5 tasks, and 60 action groups.Source: §1 · p. 2
  • A hierarchical captioning pipeline aligns local sensor dynamics, cross-signal relations, individual context, and action-relevant evidence as language supervision for OpenSLA.The model jointly supports hierarchical action prediction, state understanding, and action explanation.Source: §1 · p. 2
  • OpenSLA outperforms state-of-the-art methods and supports language-guided evidence grounding and zero-shot generalization to unseen actions and cohorts.Evaluations span real-world tasks in clinical prediction, operating rooms, and metabolic health.Source: §1 · p. 3

2. Related Work

Prior sensor-action prediction, sensor foundation models, and sensor-language models address parts of the problem, but do not jointly connect sensor observations, context, and action-relevant semantics. This work positions language as a flexible interface for that connection.

Sensor-Action Modeling

  • Sensor-action methods map observations to action labels, sometimes conditioning on text, but are less effective and flexible in incorporating heterogeneous individual context and other decision-relevant information.The paper addresses this limitation by introducing language as a flexible interface for sensor-action modeling.Source: §2 · p. 3

Sensor and Time-Series Foundation Models

  • Sensor and time-series foundation models perform forecasting, classification, and representation learning, yet largely focus on understanding input states rather than translating sensor dynamics into action-relevant semantics.OpenSLA jointly models sensor observations, language context, and actions through a shared interface.Source: §2 · p. 3

Sensor-Language Modeling

  • Sensor-language models align sensor representations with text or enable language-based interaction, but their data and pipelines are primarily designed for sensor-text alignment rather than action-relevant semantics.The paper introduces multifaceted captioning to capture a broader spectrum of information for SLA modeling.Source: §2 · p. 3

3. Sensor-Language-Action Dataset Construction

The paper constructs a sensor-action testbed across clinical, operating-room, and daily-life metabolic settings, with shared action hierarchies and captions that combine sensor evidence with individual context. The resulting benchmark covers 79 modalities, 116,000 individuals, and 60 actions.

Data Regime

  • The testbed combines six development datasets across clinical ICU prediction, operating-room decisions, and daily-life CGM, reserving MIMIC-IV as an external clinical cohort.Development data include MC-MED, MIMIC-III, MOVER, VitalDB, MetaboNet, and PEDAP; individual-disjoint splits are used for each development dataset.Source: §3 · pp. 3–4

Multi-faceted Captioning Pipeline

  • Captions cover channel-level observations, cross-channel relationships, individual context, and action-conditioned evidence explaining recorded decisions.The observational, relational, global, and action captions respectively describe signal features, complementary measurements, patient context, and evidence relevant to the action label.Source: Figure 2 · p. 4, §3 · pp. 4–5

Unified Multi-level Action Space

  • Actions are organized into a shared three-level hierarchy: whether to act, the action category, and the fine-grained action label.Clinical and operating-room actions include medications, procedures, evaluations, and treatment adjustments; CGM actions correspond to insulin bolus delivery.Source: §3 · p. 4

Dataset Construction

  • The benchmark spans 79 sensor modalities, 116K individuals, and 60 unique actions across three real-world settings.The study also introduces structured action supervision.Source: §3 · p. 5

4. Unifying Sensor-Action Tasks through a Language Interface

The section motivates jointly modeling sensor observations, language, and actions, then presents OpenSLA as a model for action prediction and sensor-grounded descriptions. Its design combines hierarchical sensor representations with action- and evidence-conditioned language generation.

4.1. Sensor-Language-Action Modeling

  • Joint action-language fine-tuning performs best among the studied paradigms, motivating a framework that predicts both actions and language from sensor observations and context.The comparison includes zero-shot multimodal LLMs, VLA-style action-only training, and SensorLM-style sensor-language alignment.Source: §4.1 · p. 5
  • Direct LoRA adaptation achieves the strongest performance on most action-prediction tasks among the tested alignment strategies.The alternatives are LLaVA-style adapter training and Flamingo-style cross-attention injection.Source: §4.1 · p. 5, Figure 4 · p. 6

4.2. OpenSLA: Learning Sensor Dynamics through a Language Interface

  • OpenSLA jointly optimizes structured action prediction and autoregressive language generation within a shared representation.Its components are an LLM backbone with action-prediction heads, a hierarchical sensor encoder, and a multimodal fusion decoder.Source: §4.2 · p. 6
  • The hierarchical sensor encoder processes channels independently, then compresses sensor tokens across resolutions while retaining fine-grained patterns and sample-level information.A separate global-query pathway attends to the original token sequence to capture sample-level semantics.Source: §4.2 · p. 6
  • The fusion decoder conditions caption generation on predicted actions and retrieved sensor evidence to retain fine-grained grounding.It uses parallel cross-attention branches over the action profile and sensor evidence, with learnable gates.Source: §4.2 · pp. 6–7
  • Training combines action-prediction losses with autoregressive caption-generation cross-entropy, while adapting the LLM with LoRA and jointly optimizing the encoder and decoder.The remaining pretrained modules are kept frozen.Source: §4.2 · p. 7

5. Experiment & Results

Across clinical, operating-room, and CGM tasks, OpenSLA shows strong action-prediction performance, with particularly large gains in clinical and operating-room settings. It also produces accurate caption-based state estimates and action-evidence descriptions, with qualitative evidence of sensor grounding and a held-out-action representation near related antibiotics.

Action explanation

  • In a representative case, OpenSLA produces more precise and physiologically grounded descriptions than zero-shot multimodal LLMs, with closer agreement to the observed sensor state.The figure contrasts generated captions with their underlying sensor observations.Source: Figure 5 · p. 9, §5 · p. 10

Action prediction

  • OpenSLA achieves strong action-prediction performance across domains and action granularities, with particularly large gains in clinical and operating-room settings.Sensor-only prediction is most challenging in the operating-room setting; CGM category prediction remains challenging when decisions depend on unavailable future carbohydrate intake.Source: Table 3 · p. 8, Table 4 · p. 8, §5 · p. 9

State understanding

  • OpenSLA yields the most accurate caption-based physiological state estimates among compared methods, with gains most pronounced in the clinical setting.The estimates are evaluated without training an additional regression head, and clinical prediction requires integrating complementary information across sensor modalities.Source: §5 · pp. 9–10, Table 5 · p. 9

Action explanation

  • OpenSLA achieves the strongest overall agreement with reference action-evidence captions across domains.The comparison uses BERTScore F1 for semantic agreement.Source: Table 6 · p. 10, §5 · p. 10

Generalization

  • The learned representation places the held-out action Imipenem near semantically related antibiotics in a zero-shot generalization visualization.The figure also depicts zero-shot prediction of Imipenem.Source: Figure 6 · p. 10

6. Generalization and Predictive Structure in SLA Representations

OpenSLA's learned representations support transfer to unseen actions and clinical cohorts, while revealing continuous action structure and information predictive of future sensor states.

Continuous action structure

  • Retrieved insulin actions varied monotonically along interpolations between representations, indicating continuous action structure.The figure examines nearest real-sample actions along the interpolation trajectory.Source: Figure 7 · p. 11, §6 · p. 11

Generalization to unseen actions

  • OpenSLA achieved the highest balanced accuracy on zero-shot prediction of the held-out action Imipenem.Prediction matched sample representations to textual action embeddings by similarity.Source: §6 · p. 11

Future sensor states

  • Frozen OpenSLA representations supported accurate prediction of future glucose mean, minimum, maximum, and slope from 2-hour CGM windows.Targets covered the subsequent 2 hours and were evaluated with individual-grouped nested cross-validation.Source: §6 · p. 11, Figure 7 · p. 11

Transfer to unseen datasets

  • Without adaptation, OpenSLA outperformed baselines on both action necessity and category prediction in the external MIMIC-IV cohort.The transfer evaluation used a shared action vocabulary.Source: Table 7 · p. 11, §6 · p. 11

Model ablations

  • The full model performed strongest overall; removing language supervision caused the largest degradation, while removing action conditioning or the fusion decoder further reduced performance.The ablation compares action prediction across necessity and category tasks.Source: Figure 8 · p. 12, §6 · p. 12, Table 9 · p. 12

Caption-facet ablations

  • Action captions produced the strongest overall caption-facet results, improving necessity and category AUROC on both MOVER and VitalDB relative to supervision without that facet.Global captions improved results on MOVER, but gains were less consistent on VitalDB.Source: §6 · p. 12, Table 8 · p. 12

7. Discussion

The discussion presents SLA as a foundation for jointly reasoning over sensing, language, and action, while identifying clinical validation and robustness to real-world variation as unresolved boundaries.

Discussion

  • OpenSLA has not undergone clinical validation and is not intended for direct clinical decision-making; deployment requires prospective validation, safety evaluation, and regulatory review.These requirements define a boundary on present clinical use.Source: §7 · p. 13
  • Current models are trained separately by domain, and the authors identify a cross-domain generalist with shared heterogeneous-action representations as future work.The current domains define action spaces independently.Source: §7 · p. 13
  • Experiments assume fixed sensor modalities and input channels, while real-world devices and acquisition settings vary; the conclusion frames SLA as a foundation for joint sensing-language-action reasoning.Robust handling of sensor heterogeneity remains future work.Source: §7 · p. 13

A.1. Dataset Splits and Statistics

The appendix describes a multi-domain dataset collection, its evaluation pools, and broad sensor coverage across the development cohorts.

Dataset scale

  • The seven datasets together cover 116K individuals, and the six development datasets provide more than 2.3 million training windows.MIMIC-IV is held out as an external transfer cohort.Source: §A.1 · p. 18, Table 11 · p. 18

Dataset splits

  • Action-evaluation pools include training, validation, and test splits after quality filtering; test windows are balanced for action necessity, including 1,000 MIMIC-IV windows.The same balancing strategy is applied to external evaluation.Source: Table 10 · p. 18, §A.1 · p. 18

Sensor coverage

  • Development inputs include waveform and numerical measurements, spanning modalities such as ECG, EEG, CGM, vital signs, laboratory values, and insulin delivery.The authors report dataset-specific quality control and harmonization, with near-complete coverage for several cohorts and sensor types.Source: §A.1 · p. 18

A.2. Sensor-Action Sample Construction

The appendix specifies how pre-decision samples and action targets are built, how evidence is rendered into complementary captions, and how OpenSLA encodes, predicts, and generates from sensor data.

A.2. Sensor-Action Sample Construction

  • Samples pair historical sensor windows and individual context with action targets; clinical/OR may have multiple category and label targets, while CGM predicts bolus occurrence and one of four dose bins.Windows span 30 minutes for clinical care, 5 minutes for OR care, and 2 hours for CGM (24 five-minute slots).Source: §A.2 · pp. 18–20

B. Multi-Faceted Caption Construction

  • Caption targets are built by extracting findings from sensor windows and available context, then selecting and rendering them as four complementary facets.Intermediate records retain measurement identity, numerical summary, temporal information, and source.Source: B. Multi-Faceted Caption Construction · p. 20

B.1. Observational Caption Construction

  • Observational captions summarize numerical measurements and waveform channels, reporting limited availability when signal quality is insufficient.CGM summaries cover glucose level, range, trend, and latest reading, with additional variability descriptions when applicable.Source: §B.1 · p. 20

B.2. Relational Caption Construction

  • Relational captions describe supported temporal trends and historical event timing, while cross-channel statements use predefined physiologically related pairs subject to quality rules.The pipeline distinguishes directional change from persistent or relatively unchanged states.Source: §B.2 · pp. 20–21

B.3. Global Caption Construction

  • Global captions combine individual context with selected physiological findings, ranking supported candidate states or using predefined CGM combinations.When evidence does not sufficiently support a state description, the caption reports this.Source: §B.3 · p. 21

B.4. Action Caption Construction

  • Action captions select pre-decision findings relevant to an action interpretation and present them as evidence associations, not causal explanations or treatment recommendations.Recorded action categories guide selection where applicable.Source: §B.4 · p. 21

C.1. OpenSLA Implementation

  • Separate waveform and numerical encoders feed a hierarchical memory, while necessity, category, and fine-grained heads predict actions from the final hidden state.Candidate fine-grained actions are scored by cosine similarity within each category.Source: §C.1 · pp. 21–22
  • The multimodal decoder attends to predicted action profiles and retrieved sensor evidence to refine caption states; training combines action losses with weighted language-generation loss.Signal-related caption tokens receive higher loss weight.Source: §C.1 · p. 22

C.2. Baseline and Ablation Details

This section specifies the baselines, training and evaluation protocols, ablations, and unseen-action test used to assess OpenSLA.

C.2. Baseline and Ablation Details

  • ConvTrans and MMIM are action-prediction baselines, while Chronos-2 [2], Moirai, and GluFormer are frozen sensor-only models.MMIM combines sensor recordings and language context without caption generation; the sensor-only models use pooled pretrained representations and linear prediction heads.Source: §C.2 · p. 22
  • GPT-5.6-Luna and DeepSeek-V3.2 [3] predict actions through code-assisted inspection of pre-decision measurements, without adaptation or probability-based scoring.OpenTSLM [15] and SensorLM [43] are also evaluated as sensor-language baselines; their sensor encoders and fusion components are adapted or retained as described.Source: §C.2 · pp. 22–23
  • Ablations remove caption supervision, the residual fusion decoder, or predicted-action inputs, and compare LoRA with frozen-backbone projector and cross-attention alternatives.The alternatives retain sensor encoders, action heads, and caption supervision, with task-specific modules trainable except the pretrained DINO encoder.Source: §C.2 · p. 23

C.3. Training Details

  • Training configurations and effective batch sizes, including gradient accumulation, are summarized for OpenSLA and adapted sensor-language baselines.The training table’s scope is identified, but these passages do not provide its configuration values.Source: §C.3 · p. 23, Table 15 · p. 23

D.1. Action Prediction

  • Action prediction is evaluated at necessity, category, and fine-grained levels against held-out annotations, with fine-grained labels assessed within positive-category cohorts.Clinical and operating-room tasks use all three levels; CGM necessity denotes bolus intervention and its categories are dose bins, without a separate fine-grained task. Binary thresholds maximize validation BAcc, and AUROC and BAcc are reported.Source: §D.1 · p. 24

D.2. Caption-Based Evaluation

  • Caption-based state understanding uses MAE against reference statistics, while action explanations are scored by mean per-sample BERTScore F1 multiplied by 100.State errors use the reference unit and include only reference-eligible samples with valid extracted predictions; explanation scoring uses distilbert-base-uncased.Source: §D.2 · p. 24

D.3. Unseen Action Classification

  • The unseen-action test targets Imipenem/Cilastatin, a fine-grained antibiotic label absent from supervised training although its antibiotic category was seen.OpenSLA scores the descriptor through token embeddings and a trained projection; model parameters remain fixed, with no additional training on the unseen label.Source: §D.3 · p. 24

D.4. External-Cohort Evaluation

This section describes cross-cohort evaluation and additional analyses of adaptation and unseen-action generalization. Direct LoRA adaptation and OpenSLA-H are reported as strongest in their respective comparisons.

D.4. External-Cohort Evaluation

  • Clinical models are evaluated on waveform-linked MIMIC-IV samples without adaptation, measuring necessity and category AUROC and BAcc as transfer to another cohort.The evaluation cohort comprises 1,000 windows from 87 individuals, balanced between action-positive and action-negative samples.Source: §D.4 · pp. 24–25

E. Additional Results

  • Additional analyses cover sensor-language adaptation, caption supervision, generalization, and information retained in learned representations.These passages name the analysis areas without reporting their findings.Source: E. Additional Results · p. 25

E.1. Sensor–Language Adaptation

  • Direct LoRA achieves the highest necessity and category AUROC on both MOVER and VitalDB among LoRA, frozen-backbone projector, and cross-attention alternatives.Table 17 reports necessity and category AUROC (%) on operating-room datasets.Source: §E.1 · p. 26, Table 17 · p. 26

E.2. Unseen Action Generalization

  • For the unseen Imipenem label, OpenSLA-H achieves the highest F1 scores and balanced accuracies while scoring its textual representation without additional training.The label is excluded from supervised action training; Table 18 reports unseen-action prediction results.Source: §E.2 · p. 26, Table 18 · p. 26

E.3. Future-State Readout

This section examines future-state readouts through prediction, sensor-evidence sensitivity, and runtime tests, alongside language templates for describing sensor signals, context, states, and action-related evidence. The templates distinguish direct support from indirect, uncertain, or unavailable evidence.

E.3. Future-State Readout

  • OpenSLA variants have lower MAE than listed baselines for next-two-hour mean, minimum, and maximum glucose, while slope errors round to 0.39 for OpenSLA, OpenTSLM-SP, and SensorLM.These are trained ridge-readout results from frozen representations, not zero-shot forecasts; evaluation used nested five-fold cross-validation.Source: §E.3 · p. 26, Table 19 · p. 27

E.4. Contribution of Sensor Inputs

  • Including waveform and numerical sensor inputs improves necessity and category AUROC on both datasets versus a context-only variant.The variant retains the same pre-decision text context, hierarchical action heads, supervision text captions, and action-conditioned fusion decoder.Source: Table 20 · p. 27, §E.4 · p. 27

E.5. Controlled Sensor Substitution

  • With patient context fixed, substituting a higher-ABP sensor recording changes the caption: the Pleth rate proxy rises from 76.1 to 82.1/min and a high-rate ECG pattern is added, while cardiac/rhythm concern remains unchanged.The controlled comparison shows caption responsiveness to sensor changes while the patient-context component remains unchanged.Source: Figure 9 · p. 27, §E.5 · pp. 27–28

E.6. Hierarchical Memory Efficiency

  • Hierarchical Memory reduces token counts and average forward latency in the Clinical and OR domains, but yields little runtime benefit for CGM because short sequences make memory-processing overhead offset compression savings.The comparison measures batch-size-1 forward latency for sensor encoders, backbone, and necessity/category heads on the action prediction task.Source: Table 21 · p. 28, §E.6 · p. 28

F.2. Template pool

  • Signal templates describe numeric and waveform findings, temporal trends, basal delivery, insulin or carbohydrate on board, and recent events or changes.They include language for aligned or conflicting cross-signal evidence and for empty temporal windows.Source: §F.2 · pp. 34–35
  • Context and state templates summarize available patient context, selected glycaemic states, or limited direct pre-decision evidence.When no state is selected, the template attributes this to limited direct evidence.Source: §F.2 · p. 35
  • Treatment-evidence templates separate relevant support from indirect, mixed, incomplete, or absent action-specific evidence, avoiding unsupported immediate-action claims.They also allow recorded treatment to be retained as background when no label-specific rationale is supported.Source: §F.2 · p. 36
  • Diagnostic templates distinguish documented indications from inferred ones and allow for broad context or no identifiable pre-decision indication.They also state when the visible window does not establish an immediate treatment response.Source: §F.2 · pp. 36–37

Related papers

Loading 2610.08244v1…