Towards Automated Clinical Behavioral Coding with Large Language Models: A Case study Using BOSCC recordings of Children

Introduction

• BOSCC uses a clinician-led, 12-minute play assessment and 20 rubric-scored items; manual coding requires expertise and time, limiting scalability.Items receive ordinal scores from 0–5, with higher scores indicating greater severity of atypical behavior.Source: §1 · p. 1
• Automating BOSCC coding could make behavioral-change measurement more efficient and reliable, addressing the time demands and potential human errors of manual scoring.The paper motivates automation as a way to scale BOSCC coding for research.Source: §1 · p. 1
• Prior LLM studies examined behavioral coding and autism-related interactions, but their reliability for BOSCC coding remained unclear.This study addresses that gap by evaluating general-purpose LLMs on the sequence of questions in each item's scoring rubric.Source: §1 · p. 1
• The best zero-shot balanced accuracy exceeds chance by 19.3 percentage points across two speech-related BOSCC items and 163 recordings.The study evaluates whether LLMs can reproduce expert-assigned scores.Source: §1 · p. 1
• Diarization improves 12 of 14 comparisons, while audio input and few-shot prompting yield inconsistent gains.The study examines ASR transcripts, speaker-role diarization, audio input, and few-shot prompting for structured behavioral coding.Source: §1 · p. 1
• LLM performance varies across scoring decisions, shows no consistent diagnosis-group pattern, and is challenged by BOSCC criteria and ASR-related errors.An analysis of model predictions identifies difficulty applying the coding criteria and errors associated with automatic speech recognition.Source: §1 · p. 1

Dataset

• The dataset comprised 163 BOSCC-PSYF recordings from children grouped as typically developing, autistic, or developmentally delayed, with each recording divided into two independently coded six-minute segments.The recordings lasted about 12 minutes and included semi-structured play and conversational interactions between child and examiner.Source: §2 · p. 1

Selected BOSCC items

• The study selected Verbal Exchanges on a Topic (VET), which measures topic-related back-and-forth exchanges with at least two child turns, and Stereotyped and Echoed Speech (SES), which assesses echoed, repetitive, or stereotyped verbal behavior.Source: §2 · p. 2

Coding procedure

• Scores for VET and SES range from 0 to 5; trained coders who met agreement criteria applied item-specific decision trees to each six-minute segment.Coders completed consensus-based training and demonstrated the required agreement criterion on three consecutive videos before coding independently.Source: §2 · p. 2

3.1. BOSCC Decision Tree Scoring Formulation

• LLMs answered BOSCC binary questions independently, with responses linked through decision trees to determine terminal codes and item scores.VET has six questions and seven terminal nodes, including distinct 3a and 3b nodes; SES has five questions and six nodes. A segment receives the correct terminal node only when every question on its ground-truth path is answered correctly.Source: §3.1 · p. 2

3.2. Modality Configurations

• The study evaluated transcript, diarized transcript, diarized transcript with audio, and audio-only conditions using speech inputs constructed from recordings.Transcript-based inputs used Parakeet ASR; diarized transcripts added child/examiner labels from Xu et al. and intelligibility labels from ChildVox [21]. Audio conditions used twelve consecutive 30-second clips per six-minute recording; audio-only clips were mono-channel at 16 kHz.Source: §3.2 · pp. 2–3

3.3. Prompt Construction

• Question-specific prompts standardized BOSCC instructions, coding guidance, two response options, and JSON output across LLMs.Transcript conditions used one prompt per six-minute segment, while audio inferences on clips were aggregated to one response. Zero-shot prompts had no corpus examples; demonstration prompts included balanced examples for both labels.Source: §3.3 · p. 3

Models and Inference

• Four models were evaluated: three Qwen models and audio-capable Gemma 4 12B-IT, with text-only models using ASR transcripts and speaker labels.Non-audio models were tested on transcript and diarized-transcript inputs; Gemma was tested under all four modality conditions.Source: §4 · p. 3
• Models supporting thinking mode were tested with thinking enabled and disabled, and each segment-question pair was run five times with majority voting.Inference used vLLM; Qwen3.8 used FP-8 quantization, other models used bfloat16, and thinking runs had separate reasoning and final-answer token limits.Source: §4 · p. 3

Evaluation Protocol

• The primary experiment compared zero-shot models on 326 segments, scoring terminal codes as correct only when every binary answer on the reference path matched.Balanced accuracy was the unweighted mean recall across seven VET and six SES codes, with 95% bootstrap percentile intervals from 10,000 ground-truth-stratified resamples.Source: §4 · p. 3
• Using the strongest zero-shot model, Gemma 4 12B-IT, follow-up analyses tested two- and four-shot prompting, audio modalities, diagnosis-group matching, question-level performance, and error evidence.The modality comparison contrasted diarized transcripts with audio-augmented and audio-only conditions; prediction behavior was analyzed under diarized transcripts.Source: §4 · p. 3

5.1. Zero-shot Results

• Diarization improves zero-shot performance, especially for VET; Gemma 4 12B-IT performs strongest on average, but remains insufficient for standalone BOSCC scoring.VET and SES perform similarly with transcripts alone, while VET performs better under diarization; thinking mode has inconsistent effects.Source: §5.1 · pp. 3–4

5.2. Ablations

• Few-shot prompting and added audio provide no consistent benefit, while audio alone performs substantially worse than diarized transcripts for both items.Few-shot performance decreases in all settings except four-shot VET; audio added to diarized transcripts slightly improves VET but decreases SES performance.Source: §5.2 · p. 4, Table 3 · p. 4

5.3. Analysis of Scoring Decisions

• Performance varies more by BOSCC item and code than diagnosis group, with no consistent relative performance trend across diagnosis groups.Figure 2 reports exact item-code matching by diagnosis for codes with at least 75 segments.Source: §5.3 · p. 4, Figure 2 · p. 4
• Q3 yields the lowest balanced accuracy for both VET and SES, but the model errs in opposite directions on the two questions.For VET Q3, it predicts NO in 21 of 22 cases and misses all reference YES cases; for SES Q3, it overpredicts YES and misclassifies 21 of 28 reference NO cases, resulting in below-chance balanced accuracy.Source: §5.3 · p. 4, Table 4 · p. 4
• Manual inspection found errors from inconsistent application of BOSCC criteria and ambiguous or inaccurate ASR evidence.For VET Q3, the model sometimes required a more explicit link between successive questions than the criteria specify; for SES Q3, it treated ordinary task-related speech as atypical. An ASR transcription of “A cookie!” as “I’m cookie” was treated as unusual self-reference.Source: §5.3 · p. 4

6. CONCLUSION

• Qwen and Gemma captured meaningful zero-shot signal on two speech-related BOSCC items, but remained insufficient for standalone use.The evaluation asked models to answer the items’ linked decision questions.Source: §6 · p. 4
• Diarized transcripts provided the most consistent gains, while demonstrations and audio yielded no consistent benefit across items.Source: §6 · p. 4
• Among well-supported codes, performance showed no consistent pattern by diagnosis group; model evidence revealed inconsistent criterion application and sensitivity to ambiguous speech evidence.Source: §6 · p. 4
• Findings are limited to two items, one in-house dataset, and single-coder annotations; future work should improve acoustic integration and ASR and evaluate additional BOSCC items and datasets.Source: §6 · p. 4

• IRB Study No. 20160509 approved data collection and analysis; parents or legal guardians provided written consent, and authorized personnel accessed recordings under the approved protocol.Source: §7 · p. 5