"Just Like This": Manner Deixis at the Graphics Interface

1 Introduction

• Manner deixis describes how an action is performed: a hand demonstration can supply what “open it like this” leaves unspecified.Unlike speech and pointing, which identify an action and its referents, the movement conveys how the action should unfold.Source: §1 · p. 2
• The technique combines speech, pointing, and demonstrated motion to author 3D behavior, using motion as evidence for a mechanism’s axis, pivot, range, and pace.The resulting joint can be revised with another sentence or demonstration.Source: §1 · p. 2, Figure 2 · p. 2
• Reference highlights, captured-path feedback, and an animated preview expose what was selected, received, and inferred; a confidence gate can ask for another demonstration.The system can accept the watched hand movement as an answer within the same scene-editing conversation.Source: §1 · p. 2
• The implementation includes a desktop hand-tracker setup and two authoring scenes, but only the castle scene was evaluated comparatively.The castle supports articulated mechanisms and part-level edits; the lighting atelier supports placement and hand-drawn trajectories.Source: §1 · p. 2
• 89% of trials were participant-declared complete with the system versus 47% with speech alone, and nine of twelve participants preferred it.The comparison tested complete input configurations; adjusted tests did not establish differences in accuracy, trial duration, or word count.Source: §1 · p. 2
• The study compared three complete input configurations in a within-subjects design with twelve participants, and its findings characterize the bundles rather than isolate each added channel.The system added span and motion to pointing; the contribution passage says the effectiveness of separate design components remains to be tested.Source: §1 · pp. 2–3

2.1 Speech, reference, and manner deixis

• Earlier multimodal systems combine speech with spatial reference, while manner deixis distinguishes pointing to where from demonstrating how an action is performed.The paper frames the open interaction question as turning that composite instruction into behavior that remains inspectable and revisable.Source: §2.1 · p. 3

2.2 Hand input for modeling and animation

• Prior hand-input systems already use gestures for modeling and animation; this work emphasizes conversational use of a demonstrated mechanism, with speech and pointing supplying context.The paper also notes that articulation estimation from observed motion is established and does not claim a new estimation algorithm.Source: §2.2 · p. 3

2.3 Language and hand input in generative authoring

• Related generative-authoring systems combine language and hand input, and the paper distinguishes its focus as editable articulated behavior addressable through later speech or motion.When more evidence is needed, the system can request a demonstration, capture it without additional speech, and return an animated preview.Source: §2.3 · pp. 3–4

2.4 Grounding and demonstration requests

• Clarification and demonstration requests have precedents in interface and robot systems; here a continuous hand movement answers a question about 3D behavior.A captured-path ribbon shows the received movement, while an animated preview shows the inferred mechanism.Source: §2.4 · p. 4

2.5 Contribution in relation to prior work

• The contribution is the connection of speech with spatial reference, demonstrated motion abstracted into editable behavior, and further demonstration requests during clarification.The paper presents the interaction technique as the contribution, not the linguistic concept or the individual sensing, fitting, and clarification components.Source: §2.5 · p. 4

3 Our system

• Users speak, point, and demonstrate motion to author a 3D scene and its behaviors.They interact at an ordinary display with a hand tracker, with the scene positioned above it.Source: §3 · p. 4

3.1 Design principles

• Speech supplies categories and explicit values, pointing supplies reference, and span and motion supply spatial and temporal evidence.The design prioritizes explicit spoken values when they conflict with magnitudes inferred from hand input, and exposes interpretation through highlights, paths, and previews.Source: §3.1 · p. 4

3.2 Example interaction

• A user can point to a gate, request a door, demonstrate a slow opening arc, inspect the preview, and then ask to slow that same hinge.The captured path and translucent door preview make the inferred opening inspectable before approval.Source: §3.2 · p. 4

3.3 Architecture

• Unity handles sensing, rendering, and execution, while a Python service handles audio, speech recognition, multimodal fusion, and dialogue state.The service forwards validated, confidence-gated commands to Unity, which executes them and returns status for later turns.Source: §3.3 · p. 4

3.4 Sensing pointing and hand movement

• Pointing and span events are streamed alongside buffered hand motion, while ordered hover episodes preserve the sequence of references within a turn.Dwell contributes contextual evidence to target interpretation rather than deterministically resolving the target; an eight-second ring buffer lets motion be retrieved after speech ends.Source: §3.4 · p. 5

3.5 Speech-anchored fusion

• The fusion window is anchored to the utterance and padded around its speech interval, retaining reference evidence even when transcription finishes later.The assembled prompt combines the transcript, ordered pointing episodes, optional motion summary, scene table, and any pending question or on-screen options.Source: §3.5 · p. 5

3.6 From demonstration to mechanism

• Motion is summarized as a polyline, movement statistics, and a fitted articulation block reporting geometry, travel, and pace.The block includes net sweep versus total travel, clip duration, and straightness for deciding whether to attempt an arc fit.Source: §3.6 · p. 5
• A nonzero-winding movement plane supplies a candidate hinge axis, while a circle fit estimates the pivot and range.The estimated pivot and axis are reported relative to the object as well as in world coordinates.Source: §3.6 · pp. 5–6
• Figure 4 traces the path from demonstrated opening shape and pace to fitted hinge parameters, opening mode, and later pace refinement.Its geometry and angle–time plots are illustrative.Source: Figure 4 · p. 6
• Net sweep distinguishes a monotone one-way demonstration from one complete out-and-back excursion, whose travel is about twice its one-way range.Repeated cycles and exact retracing complicate that distinction; large travel relative to net sweep alone does not establish range or cycle count.Source: §3.6 · p. 6
• Pace is modeled independently of geometry, allowing a later spoken “slower” to retime a mechanism built with the hands.Period conversions use mean travel per unit time; easing and period limits can prevent an exact pace match.Source: §3.6 · p. 6
• Speech and scene context guide geometry-based readings, so the same fitted hinge parameters can apply across different articulated objects.The motion-to-behavior mappings are organized by mechanism rather than by object name.Source: §3.6 · p. 6, Table 1 · p. 7

3.7 Grounding and clarification

• Contact highlights and names a part, a ribbon shows the captured hand path, and an animated preview shows the inferred behavior.Ghost-preview approval commits an object; rejection reopens the request, while correction revises it.Source: §3.7 · p. 7
• A confidence threshold turns uncertain commands into clarification questions, with candidate references or alternative magnitude previews shown to the user.Watch mode captures a requested demonstration without requiring additional speech and fuses it with the original request.Source: §3.7 · p. 7

3.8 Scene editing and behavior execution

• Props retain referenceable identifiers and live state, enabling later commands to move, restyle, retime, or rebuild them.The castle doorway maintains a spatial relationship: resizing a door adapts the surrounding opening within structural limits.Source: §3.8 · pp. 7–8

3.9 Implementation and response timing

• Across 973 retained task turns, the logged fusion interval had a 4.01-second median and an 8.20-second ninetieth percentile.Condition medians were 3.62 seconds for speech alone, 3.85 seconds with pointing, and 4.33 seconds with all channels; the interval is not end-to-end latency.Source: §3.9 · p. 8

4.2 Apparatus and design

• Twelve participants completed a within-subjects comparison of speech alone, speech plus pointing, and speech, pointing, span, and motion.All conditions used the same backend, feedback, and clarification machinery; the comparison does not isolate a motion-only effect.Source: §4 · p. 8, §4.1 · p. 8, §4.2 · p. 9

4.3 Pilot study

• Condition order and target sets were counterbalanced across participants, with each participant completing all three conditions and sets.Task order was fixed; two pilot participants, excluded from the analyses, helped establish the task interval and session procedure.Source: §4.2 · p. 9, §4.3 · p. 9, §4.4 · p. 9

4.2 Apparatus and design

• Door, birds, and flag tasks tested articulated mechanisms, combined trajectory and local motion, and wind-like cloth motion.Targets were replayable translucent animations without verbal descriptions, and each task had a nominal three-minute limit.Source: §4.2 · p. 9

4.5.1 Task outcomes (RQ1).

• Task success was recorded as finished, ran out of time, or gave up; completion meant participant-declared satisfaction and was distinct from target accuracy.Duration and word counts were interpreted alongside completion because early abandonment could reduce either measure.Source: §4.5.1 · p. 10
• Target accuracy covered eight categorical matches and six continuous parameter differences, including mechanism, motion mode, path, and pace.Retained final results included unfinished trials; smaller continuous differences indicated closer agreement with the target.Source: §4.5.1 · p. 10

4.5.3 Clarification responses (RQ3).

• Channel logs recorded pointing, span, and motion evidence, while clarification summaries described requested modalities and evidence in the next turn.These are descriptive measures, not causal channel-to-parameter attribution or measures of clarification request resolution.Source: §4 · p. 8, §4.5.2 · p. 10, §4.5.3 · p. 10

4.5.4 Workload and preference.

• Workload used the raw, unweighted mean of six NASA-TLX subscales, with arm and hand tiredness reported separately.CSI factors and custom experience items were supplementary descriptive ratings; overall preference was recorded as a forced choice.Source: §4.5.4 · pp. 10–11

4.6 Analysis

• Completion was the primary comparative outcome; duration, word count, workload, accuracy, and preference analyses were exploratory.Channel and clarification findings were descriptive, and the study was not preregistered.Source: §4.6 · p. 11

5.1.1 Completion and trial outcomes.

• 89% of trials were completed with the system versus 47% with speech alone; completion was 69% with speech and pointing, and the speech-alone contrast was significant after correction.Overtime outcomes were 42%, 25%, and 8%, and abandoned trials were 11%, 6%, and 3%, respectively.Source: Figure 6 · p. 11, §5.1.1 · pp. 11–12

5.1.2 Trial duration and words spoken.

• Mean recorded duration was 151 s with speech, 133 s with pointing, and 120 s with the system; neither duration nor word-count test detected differences.These measures include unfinished trials and do not represent time or words required to reach a successful result; door-task means were 174 s and 116 s for speech and the system.Source: Figure 8 · p. 12, §5.1.2 · p. 12, Figure 10 · p. 12

5.1.3 Target accuracy.

• The system had the highest observed match rate on six of eight categorical parameters and tied on two, but no individual accuracy measure separated conditions after Holm adjustment.The Wilson intervals describe individual condition proportions, not paired differences.Source: §5.1.3 · pp. 12–13, Figure 11 · p. 13

5.2.1 Logged channel evidence; 5.3 Clarification responses (RQ3)

• 81% of full-system turns included pointing, while 32% included span and 21% motion; nonverbal evidence appeared in 93% of followed clarifications.Clarification counts describe reply channels, not whether the request was resolved.Source: §5.2.1 · p. 13, §5.3 · p. 14

5.2.2 Recorded authoring exchanges.

• Recorded exchanges show movement combined with a request, supplied as a clarification reply, and reused to refine an existing door behavior.The selected excerpts illustrate interaction patterns, not their prevalence.Source: §5.2.2 · p. 13

5.2.3 Participant accounts of channel use.

• Participants described hand input for conveying measurements, placement, and fine motion, while preferences for task-suited channels varied across participants.Of eleven asked what they could not express, five reported none; remaining responses included wingbeat motion, size, graphical changes, or feedback.Source: §5.2.3 · pp. 13–14

5.4.1 Perceived workload.

• Physical demand averaged 2.9, 18.8, and 23.3 for speech, pointing, and the system, respectively, and differed after correction; raw TLX did not.The authors describe a physical cost associated with hand input over the short study tasks.Source: §5.4.1 · p. 14

5.4.2 Creativity support and experience; 5.4.3 Rankings and overall preference.

• No ranking question detected condition differences; nine participants chose the system in the final forced choice, which does not establish pairwise preference advantage.The two CSI factors and eight additional experience ratings appear only in supplementary summaries, with no significance claims based on them.Source: §5.4.3 · p. 14, §5.4.2 · p. 14

6.1 Motion as conversational content

• Demonstration lets spatial and temporal detail enter the conversation without requiring users to name every parameter; completion was higher than with speech alone.The authors call for separate analysis to establish which channel supplies each parameter.Source: §6.1 · p. 14

6.2 Editable behavior after the demonstration

• An axis, pivot, range, and period make demonstrated behavior addressable for later instructions, including requests to change an existing mechanism.In P7’s example, verbal direction and demonstrated motion address the same door; comparing parameter-based refinement with editing recorded motion remains future work.Source: §6.2 · pp. 14–15

6.3 Demonstration as a clarification response

• 93% of replies to system clarifications with a subsequent turn in the same trial included hand evidence, showing movement can participate in the answer.The authors note that P11’s door-opening demonstration illustrates this use.Source: §6.3 · p. 15

6.4 Demonstration in future agent architectures

• Demonstrations could connect perception, language, and action, with an inspectable, revisable behavior serving as a shared reference for later instructions.The authors suggest agents should request the form of evidence suited to an unresolved detail, with alternatives to demonstration when another channel better suits the user.Source: §6.4 · pp. 15–16
• Future agents could retain a demonstration’s target, spatial frame, and constraints so later instructions can refer to the earlier intent.The authors also propose preserving movement’s alignment with speech and its communicative role, and exposing proposed behavior for correction before consequential actions.Source: §6.4 · p. 16

6.5 Scope and limitations

• The evaluation was a small comparison of cumulative input configurations, not conventional keyframing or isolated motion effects; accuracy and efficiency gains were not established.Completion reflects participant satisfaction rather than verified target accuracy.Source: §6.5 · p. 16
• The evidence covers three short castle tasks, mainly occasional 3D-tool users, and English interaction in a quiet room; the supported vocabulary also excludes several behavior types.Generalization to expert workflows, open-ended authoring, other languages, or users with different motor abilities requires evaluation.Source: §6.5 · pp. 16–17

6.6 Future work

• Future work could separate the contributions of sizing, motion, and clarification, compare the system with established animation tools, and study the interaction over extended use.The authors also propose evaluating VR or autostereoscopic displays and richer feedback for mechanism limits.Source: §6.6 · p. 17

7 Conclusion

• Speech, pointing, and demonstrated motion jointly specify behavior that remains editable after the demonstration ends.A preview gives users behavior to inspect and discuss, while clarification can invite another demonstration.Source: §7 · p. 17
• Across door, bird, and flag tasks, participants reported finishing 89% of trials with our system versus 47% with speech alone.Nine of twelve participants preferred our system; it had the highest observed match rates on six categorical accuracy measures and tied on two.Source: §7 · p. 17
• Demonstration can connect perception, language, and action around human intent, with behavior visible and revisable as the conversation continues.The authors present this as a broader direction for future agents and interfaces.Source: §7 · p. 17

A Participant background

• All twelve main-study participants reported comfort conversing in English.Ten selected “completely” and two selected “mostly.”Source: A Participant background · p. 19
• Ten participants selected “completely” comfortable with English, and two selected “mostly.”Source: A Participant background · p. 19
• The passage refers readers to Table 2 and Section 4.1 for participant backgrounds and the sample summary.Source: A Participant background · p. 19

B Detailed system architecture

• Python sends the model named parts, live scene states, and hand summaries; meshes remain in Unity for execution.Source: B Detailed system architecture · p. 19
• Model calls can delay decisions without blocking rendering.Source: B Detailed system architecture · p. 19
• Mode flags and execution priorities arbitrate hand input, while deictic events convey the active mode for interpretation.Source: B Detailed system architecture · p. 19

• Speech recognition (gpt-transcribe) converts participant speech into text.Source: C AI models · p. 19
• Multimodal reasoning (gpt-5.4-mini) combines text with scene and hand evidence to produce commands and replies.Source: C AI models · p. 19
• Speech synthesis (gpt-4o-mini-tts) converts written replies into spoken audio; voice detection and hand tracking run locally.Models and settings were held constant across conditions and sessions.Source: C AI models · p. 19

D Code and supplementary materials

• The review supplement contains accuracy summaries, participant-level tests, questionnaire results, scoring definitions, interaction traces, and an implementation audit.The current code snapshot is not a verified archival build of every study session; source code, prompts, configuration, and tasks will be shared on GitHub.Source: D Code and supplementary materials · p. 19
• The authoring loop interprets requests, shows their meaning, and refines the scene, with speech, touch answers, or watched demonstrations as turn entry points.Source: D Code and supplementary materials · p. 20
• Unity handles hands, scene execution, and feedback, while Python combines speech, scene context, and motion for interpretation and dialogue.Hosted services provide transcription, reasoning, and speech; arrows indicate bidirectional message flow.Source: Figure 13 · p. 20
• Table 2 groups participant backgrounds by demographics, technology experience, and communication habits; entries are self-reports, not observed behavior.Ages are bands, and pilots P1 and P2 are excluded.Source: Table 2 · p. 21, D Code and supplementary materials · p. 21