Source-linked AI summary
Tellimation: Making Narrative Gaps Visible in Children's Storytelling with Just-in-Time Animation
Vincent Cavez, Marielle Zheng, Momin Siddiqui, Abbie L Olszewski, Srirangaraj Setlur, Maneesh Agrawala, Hari Subramonyam
TL;DR
Children with language difficulties may omit details and relationships that make picture-based stories coherent, while direct prompts can narrow what they attend to. Tellimation animates omitted or misdescribed scene elements to make gaps visible without supplying words. Children addressed more system-identified gaps with animations than without them.
Problem
Children with language difficulties often leave scene details and relationships unexpressed, and direct questions can narrow what they attend to and say.
Source: §1 · p. 2Method
Tellimation detects narrative gaps and selects and parameterizes animations that draw attention to them without supplying the child’s words.
Source: §1 · p. 2, §4.1 · p. 7Results
46% of system-identified gaps were addressed with animations, against 13% without.
Source: §1 · p. 2Takeaways & Limitations
Attentional animation can provide a promising complement to verbal prompting for child-led storytelling.
Source: §1 · p. 2The children’s study was a single session, so it does not establish whether gains carry over to unscaffolded storytelling after the novelty wears off.
Source: §7.4 · p. 20
Abstract
from arXiv · showhide
Illustrated scenes are used to prompt children's stories, yet children with language difficulties often omit the details, actions, and relationships that make a story coherent. We introduce Tellimation, which animates scene elements a child has omitted or misdescribed, drawing attention to a narrative gap without saying what to tell. Its design space covers eight kinds of gaps (who is in the scene, how many, what they are like, what they are doing, where, when, how they relate, and what lies beyond the picture, such as speech and thoughts), instantiated through 20 parameterized animations grounded in classical animation principles. A real-time pipeline detects discrepancies between utterance and scene, then selects and parameterizes an animation from the scene's narrative potential and the child's history. Adults interpreted most animations without instruction (N=120); children (N=12) resolved 46% of the gaps the system identified with animations, against 13% without.
1 Introduction
Children with language difficulties may leave important scene details and relationships out of their stories, while direct questions can constrain what they attend to and say. Tellimation uses animations to make narrative gaps visible without supplying the words, and its design space and system support this form of scaffolding.
1 Introduction
- Direct questions can narrow a child’s attention and hand initiative to the system, a concern for at-home practice aimed at increasingly independent storytelling.Intervention practice emphasizes providing only as much support as needed and gradually fading scaffolds.Source: §1 · p. 2
- Tellimation animates omitted or misdescribed scene elements to draw attention to a narrative gap without asking a direct question or supplying the answer.The child still has to find words for the detail.Source: §1 · p. 2
- The design space organizes gaps into eight kinds and provides 20 animations that target each kind in discrepancy and opportunity forms.The system detects gaps in a child’s narration and selects and parameterizes an animation in real time.Source: §1 · p. 2
- The approach’s three contributions are a visual-question design space, a real-time detection and animation system, and evidence that animations can be interpreted without instruction and prompt children to address more gaps.Children address gaps in their own words.Source: §1 · pp. 2–3
2 Related Work
Prior work offers narrative interventions and assessment frameworks, interactive storytelling technologies, and animation as a communication medium. Tellimation differs by responding visually to a child’s ongoing narration, using animation to direct attention to a narrative element rather than generate story content or act chiefly as a conversational partner.
2.1 Narrative Scaffolding and Assessment
- Narrative interventions use modeling, prompting, and guided practice, but typically rely on a trained adult to identify what is missing and decide how to scaffold it.MISL supplies an element-level vocabulary for identifying opportunities, and automated narrative analysis makes in-interaction assessment possible.Source: §2.1 · p. 3
2.2 Interactive Technologies for Children’s Storytelling
- Interactive storytelling systems often help children produce content, while evidence about their language-learning effects is mixed and engagement does not establish learning.Mathemyths [103] found comparable mathematical-language development with LLM and human partners; Storyfier [67] increased engagement but had poorer target-vocabulary recall than a non-AI baseline.Source: §2.2 · p. 3
- Tellimation responds to the child’s current narrative with visual feedback, rather than generating story content or acting primarily as a conversational partner.AACessTalk [14] scaffolds a parent-child conversation, while OSOS generates personalized storybooks for children to read.Source: §2.2 · p. 3
2.3 Animation as Attentional Communication
- Tellimation starts from a narrative element needing attention and selects a visual transformation, drawing on familiar comic conventions and animation principles.Unlike motion vocabularies whose meanings users learn, its transformations are selected to make a semantic objective perceptible; animations are localized, short, and single-purpose.Source: §2.3 · p. 4
3 Tellimation Design Space
Tellimation turns narrative gaps into visual questions: it classifies what information the child may need to notice, identifies whether the gap is a discrepancy or an unexpressed opportunity, and renders it with a parameterized animation. The animations draw on familiar visual conventions while making the gap salient without prescribing the child’s words.
3 Tellimation Design Space
- The design space asks what kind of gap is present, whether it is a discrepancy or opportunity, and how to make it perceptible through 20 animations.Its design was guided by implicit scaffolding, legibility without instruction, local targeting, and parameterization.Source: §3 · p. 4, §3.3 · p. 5
3.1 Eight Kinds of Narrative Gaps
- Eight gap kinds group shortfalls by what the child needs to notice, from identity and count to action, time, relations, and discourse.Gaps the illustration cannot address, such as pronunciation errors, fall outside Tellimation’s scope.Source: §3.1 · pp. 4–5
3.2 Two Forms of Gap: Discrepancies and Opportunities
- A discrepancy is a clear conflict with the scene or the child’s preceding utterance; an opportunity is unexpressed story content consistent with the illustration.Opportunities use the 15 MISL elements as enrichment objectives, not errors; plausible interpretations and creative elaborations are left untouched.Source: §3.2 · p. 5
3.3 Twenty Animations
- The 20 animations use familiar conventions to target particular gaps, including motion lines for actions, spotlights for identity, and bubbles for speech or thought.Some transform existing scene information, while others add temporary visual representations; the implementation exposes 47 adjustable parameters that tune visual realization and intensity to context and interaction history.Source: §3.3 · pp. 5–6
4 User Experience
Tellimation presents one illustrated panel at a time and uses animations to draw attention to errors or unexpressed story details while leaving children to narrate in their own words.
4.1 Setup
- Tellimation plays one of 20 animations on a relevant scene element when it detects a storytelling opportunity; otherwise, the scene remains unchanged.Children select a story, narrate through a single push-to-talk control, and can advance once the scene is sufficiently described.Source: §4 · p. 6, §4.1 · p. 7
4.2 Error Correction
- When Alex calls a running conductor “walking,” Motion Lines exaggerate his movement so Alex can reconsider and revise the description without being told the answer.If the error persists, the cue intensifies and can switch to another animation; cues fade as Alex describes actions correctly.Source: §4.2 · p. 7
4.3 Enriching the Story
- Figures 3 and 4 organize the 20 animations across eight narrative-gap kinds, pairing example errors and opportunities with animation behaviors and sample frames.The figures cover Identity, Count, Property, Action, Space, Time, Relation, and Discourse.Source: Figure 3 · p. 8, Figure 4 · p. 9
- The design space adapts cartoon conventions such as emanata, color pop, and motion lines into animations for story scenes.Figure 5 pairs six familiar visual conventions with the Tellimation animations they inspired.Source: Figure 5 · p. 10
- For an accurate but incomplete description, Emanation makes the boy’s sadness perceptible without supplying the word, leaving Alex to decide how to elaborate.Tellimation points toward unexpressed properties, internal states, plans, or causal relations; as Alex produces content independently, those cues fade and other opportunities can surface.Source: §4.3 · p. 10
5 System Architecture
Tellimation prepares layered scene assets and narrative metadata offline, then processes each utterance to select, parameterize, and play a correction or enrichment animation.
5.1 Scene Preparation
- Scenes are prepared as a background plus transparent entity layers, allowing runtime animations to target individual entities, groups, or the whole scene without regenerating the illustration.Generated scene descriptions support discrepancy detection, while scene-afforded MISL annotations constrain enrichment to content the illustration supports; outputs are checked and corrected by hand.Source: §5.1 · pp. 11–12
- The preparation pipeline generates consistent entity and background imagery, isolates entities into aligned transparent layers, and annotates scenes with visible details and narrative possibilities.The study used 20 scenes, with 100 additional scenes for pipeline-validation and interpretability studies.Source: §5.1 · pp. 11–12
5.2 Live Interaction Loop
- Each live cycle transcribes the utterance, updates the child profile, generates correction and enrichment candidates, selects and parameterizes a cue, then runs it until the child speaks again.A full cycle takes a median of 5.2 seconds; selection, validation, and dispatch take less than a second.Source: §5.2 · p. 12, Figure 9 · p. 12
- The profile tracks MISL production and prior cue responses; conservative correction checks flag clear conflicts, while enrichment rules select scene-supported elements the child has rarely produced.When both paths produce candidates, correction takes priority; target and animation compatibility are validated before a cue is sent.Source: §5.2 · pp. 12–13
- Salience levels tune animation parameters over time: persistent errors strengthen cues, independently produced content fades them, and a fully intensified cue can switch to another animation.The scene is restored frame by frame during playback, and profile updates are recorded whether or not an animation played.Source: §5.2 · pp. 13–14
6 Evaluation
Tellimation was evaluated in three stages: whether the pipeline chose appropriate scaffolding acts, whether adults could interpret the animations, and how children used them in storytelling. The studies found generally favorable pipeline ratings, mostly interpretable animations, and more gap resolutions with animations than without.
6.1 Pipeline Validation
- 7 of 200 decisions (3.5%) involved a misrecognized gap; among 50 corrected utterances, 6 produced spurious corrections, and reviewers observed no false negatives.The remaining low-rated decisions had rationales that named the discrepancy present in the utterance; the lowest-rated cases often concerned entities absent from the picture.Source: §6.1 · pp. 14–15
- 3.78 out of 5 was the overall pipeline-decision mean rating, and two thirds of ratings were at least 4; suggestions averaged 3.96 versus 3.60 for corrections.The validation used 200 pipeline decisions and yielded 799 ratings from 80 participants.Source: §6.1 · p. 14, Table 2 · p. 15
6.2 Animation Interpretability
- 49.6% overall animation-interpretation accuracy exceeded the 20% chance level, and 22 of 25 animations significantly exceeded chance.Action and Property were the best-interpreted categories, while Identity and Time were the lowest, though both remained above chance.Source: §6.2 · p. 15
- After redesign, pooled accuracy rose from 23.2% to 31.1%; Missing Piece and Peek significantly exceeded chance, while Silhouette remained below chance.The three redesigned animations used a silhouette, a jigsaw-shaped opening for an absent entity, and a hinged-flap reveal; Peek's difficulty was concentrated when the occluder was small.Source: §6.2 · pp. 16–17
6.3 Storytelling with Children
- With animations, children addressed 197 of 425 gaps followed by another utterance in the same scene (46%), versus 34 of 265 (13%) in control.Ten of eleven children completing both conditions resolved a larger share of gaps with animations; the remaining child resolved one gap in each condition.Source: §6.3 · p. 17
- 42% of animations were interrupted before appearing; for one child, 29 of 39 interruptions coincided with turns arriving faster than the pipeline could respond.The authors conclude that latency and display timing relative to children's turns matter because an animation arriving after a child starts speaking is lost.Source: §6.3 · p. 17
7 Discussion
With the same pipeline in both conditions, children addressed more than three times the share of identified gaps when animations were shown, using their own words. The discussion distinguishes attention direction from message interpretation, offers design guidance, and notes limits on the scenes and evaluations studied.
7.1 Tellimation as Attentional Scaffold
- Children addressed more than three times the share of identified gaps with animations than without, and did so in their own words.The pipeline ran in both conditions; animations did not supply words.Source: §7.1 · p. 17
- A Spotlight cue redirected attention beyond its target, while Silhouette errors often still identified the intended entity, separating attentional affordance from interpretability.81% of Silhouette errors identified the right entity, although it did not exceed chance in the forced-choice task.Source: §7.1 · p. 18
- Corrections were interpreted more accurately than suggestions (53.3% vs. 46.0%), but enrichment decisions were rated more natural than corrections (3.96 vs. 3.60).The redesigns and lower-rated decision types also centered on absent entities, making absence perceptible a central design challenge.Source: §7.1 · p. 18
- Logs suggest that changing the animation’s angle may work better than increasing intensity when a cue is not taken up, but the pipeline does not yet do this deliberately.This is a hypothesis to test, supported by productive sequences that addressed the same gap with different animations.Source: §7.1 · p. 18
7.2 Design Guidelines for Guided Storytelling Animations
- The guidelines favor transforming visible content and making absence perceptible through changes to what is present, with parameters bounded by scene geometry.Examples include a hole where an entity should be and a lifted occluder; Peek was rarely read correctly when its occluder was small.Source: §7.2 · pp. 18–19
- Escalation should preserve intent while varying intensity, then change angle if needed; corrections should be conservative, enrichment generous, and words left to the child.The scaffold should also recede step by step once the child supplies the element.Source: §7.2 · p. 19
7.3 Broader Utility across Clinic, Classroom, and At Home
- Tellimation could support clinic progress monitoring, classroom summaries of recurring gaps, and adjustable support at home, but its scope is bounded by what each scene affords.The scenes, story arcs, and panel affordances were authored and hand-verified; the authors propose a workflow for educators or clinicians to author them.Source: §7.3 · pp. 19–20
7.4 Limitations
- Evaluation used adults as proxies and a single session per child, while indivisible scene layers, human verification, and English-only prompts constrain the current system.The authors identify longitudinal evaluation, gaze tracking, articulated characters, and other languages as relevant next steps.Source: §7.4 · p. 20
8 Conclusion
Tellimation makes narrative gaps visible by animating scene elements children omit or misdescribe, without supplying words. Its real-time pipeline selects and adjusts animations, and the reported studies found adult interpretability and increased child uptake of identified gaps.
8 Conclusion
- In a within-subjects study with 12 children, animations more than tripled the share of identified gaps addressed, in the children’s own words.The pipeline detected gaps in each utterance and adjusted animation intensity to the child’s history in a median of 5.2 seconds.Source: §8 · p. 20
- Tellimation organizes narrative gaps into eight kinds and two forms, with 20 animations selected and parameterized by a real-time pipeline.The animations make omitted or misdescribed scene elements perceptible without supplying words.Source: §8 · p. 20
- Adult raters judged pipeline decisions appropriate, and 22 of 25 animations were interpreted above chance without instruction; redesign improved the other three.The authors conclude that directing attention can serve as a scaffold in its own right.Source: §8 · p. 20