Source-linked AI summary
RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching
Ryo Suzuki, Rubaiat Habib Kazi, Li-Yi Wei, Stephen DiVerdi, Wilmot Li, Daniel Leithinger
TL;DR
AR sketching tools can embed drawings in the real world, but their content is generally static. RealitySketch lets users sketch graphics, bind them to physical objects, and visualize real-world movement; the paper demonstrates applications across several domains and reports positive preliminary feedback alongside prototype limitations.
Problem
Existing AR sketching tools embed digital drawings in the real world, but their content is static and does not respond to user actions or real-world phenomena.
Source: INTRODUCTION · p. 1, Sketching Interfaces for VR/AR · p. 2Method
RealitySketch uses real-time sketching to track physical entities, parameterize them, bind those variables to graphic properties, and visualize movement without predefined programs or configurations.
Source: INTRODUCTION · p. 2Results
The system demonstrates applications in physics experiments, mathematical explorations, sports and exercise, and improvised tangible interfaces, with participants expressing enthusiasm about educational and sports uses.
Source: INTRODUCTION · p. 2, Insights and Feedback · pp. 10–11Takeaways & Limitations
Interactive graphics bound to physical objects can respond to their movement, supporting in-situ visualizations of physical phenomena and tangible interactions.
Source: INTRODUCTION · p. 2, CONCLUSION · p. 11The prototype supports only simple constraints and spatial parameterizations, limiting possible visualizations and excluding other variables such as shape, orientation, color, and texture.
Source: Parameterization and Visualization… · p. 11
Abstract
from arXiv · showhide
We present RealitySketch, an augmented reality interface for sketching interactive graphics and visualizations. In recent years, an increasing number of AR sketching tools enable users to draw and embed sketches in the real world. However, with the current tools, sketched contents are inherently static, floating in mid air without responding to the real world. This paper introduces a new way to embed dynamic and responsive graphics in the real world. In RealitySketch, the user draws graphical elements on a mobile AR screen and binds them with physical objects in real-time and improvisational ways, so that the sketched elements dynamically move with the corresponding physical motion. The user can also quickly visualize and analyze real-world phenomena through responsive graph plots or interactive visualizations. This paper contributes to a set of interaction techniques that enable capturing, parameterizing, and visualizing real-world motion without pre-defined programs and configurations. Finally, we demonstrate our tool with several application scenarios, including physics education, sports training, and in-situ tangible interfaces.
INTRODUCTION
Dynamic sketching has a rich history, but AR sketching tools largely leave embedded content static. RealitySketch lets users create graphics that respond to physical objects through an improvisational workflow, and demonstrates applications across several domains.
Sketching Interfaces
- Dynamic sketching already supports animation authoring, education, and data visualization, while AR sketching tools embed drawings but leave them static.The paper builds on a broader vocabulary of dynamic sketching techniques developed for real-time interactive content.Source: INTRODUCTION · p. 1
INTRODUCTION
- RealitySketch lets users sketch interactive AR graphics and bind them to physical objects so they respond as the real world changes.Users draw on a phone or tablet screen; the system supports real-time creation and sharing.Source: INTRODUCTION · p. 2
- Its workflow tracks a real-world entity, parameterizes it with drawn lines or arcs, binds variables to graphic properties, and visualizes movement.These interactions are performed in real time without predefined programs or configurations.Source: INTRODUCTION · p. 2
- Applications span physics experiments, mathematical explorations, sports and exercise assistants, and improvised tangible UI elements.A preliminary evaluation and expert interview inform discussion of benefits and limitations.Source: INTRODUCTION · p. 2
- The paper contributes a system, interaction techniques for parameterizing the real world and binding it to graphics, and application scenarios for in-situ visualization.The scenarios include capturing physical phenomena for classroom experiments and interactive concept explanations.Source: INTRODUCTION · p. 2
Sketching Interfaces for VR/AR
AR sketching tools commonly support static drawings or noninteractive animation, while dynamic sketching has largely been explored in screen-based settings. RealitySketch targets graphics that are both embedded in the real world and responsive to it, with improvisational authoring intended to let end users adapt visualizations across contexts.
Sketching Interfaces for VR/AR
- Most existing AR sketching tools support static drawings; DoodleLens adds flip-book animation but not interaction, while video prototyping and paper-sketch tools also keep sketches static.The design space contrasts embedded versus separated graphics and responsive versus static behavior.Source: Sketching Interfaces for VR/AR · p. 2, Figure 2 · p. 3
- Responsive sketching can use tangible objects and human motion as inputs, but mapping virtual content to physical references often requires advance configuration.Real-time, improvisational specification is presented as a way to preserve natural workflows and support creation and communication.Source: Sketching Interfaces for VR/AR · p. 2
Sketching Dynamic and Responsive Graphics
- Dynamic sketching has been explored extensively, but most interfaces are screen-based; ChalkTalk AR supports dynamic graphics without embedding them in or integrating them with the real world.RealitySketch explores the embedded-and-responsive region of the design space.Source: Sketching Dynamic and Responsive… · p. 3, Figure 2 · p. 3
Embedded Data and Concept Visualization
- Embedded visualizations couple data to physical referents, and AR/MR can present phenomena such as wind, optical paths, and electromagnetic flow in the real world.Such interfaces can let viewers observe hidden phenomena directly rather than imagine them.Source: Embedded Data and Concept Visualization · p. 3
- Many existing embedded interactive systems are pre-programmed for specific applications, limiting nontechnical users’ ability to adapt them or create custom experiences.The paper proposes real-time, improvisational authoring for end users across application domains.Source: Embedded Data and Concept Visualization · p. 3
Object Manipulation through Spatial Tablet Interactions
Embedded, responsive graphics interact with the environment, physical phenomena, or users’ tangible and gestural actions. Their behavior can be predefined, user-defined, or combined, with the appropriate balance depending on the application and user needs.
Object Manipulation through Spatial Tablet Interactions
- Spatial tablet interfaces manipulate virtual and physical objects; RealitySketch instead uses physical objects as inputs for sketched responsive graphics rather than programming those objects.This extends tablet-based AR interaction with real-world objects toward dynamic sketching.Source: Object Manipulation through Spatial… · p. 3
Definition of Embedded and Responsive Graphics
- Embedded graphics are spatially integrated within the real world, while responsive graphics change or animate based on interaction with it.The real world includes the environment, physical phenomena, and users’ tangible or gestural interactions.Source: Definition of Embedded and Responsive… · p. 3
- The paper identifies real-time sketching as a largely unexplored way to create interactive experiences with embedded, responsive graphics.It uses examples to explore interaction approaches and the broader design space.Source: Definition of Embedded and Responsive… · p. 4
Pre-defined Behavior vs User-defined Behavior
- Predefined behavior specifies a sketch’s behavior in advance, whereas user-defined behavior lets users determine movement and animation on the fly.A physics simulation can make a drawn object fall and bounce; user-defined constraints and parameter bindings can specify pendulum visualization.Source: Pre-defined Behavior vs User-defined… · p. 4
- Predefined behavior can produce complex, expressive animations but is application-specific; user-defined building blocks can generalize, though complex behaviors are challenging to create in real time.The appropriate balance depends on the application and user needs.Source: Pre-defined Behavior vs User-defined… · pp. 4–5
Focus of This Paper
The paper focuses on user-defined input specification and mapping for embedded graphics, while using predefined visual outputs such as graph plots. It explores how users can sketch spatial inputs and map them to visual outputs for spontaneous, interactive exploration.
Focus of This Paper
- The system focuses on user-defined input specification and mapping, while relying on predefined outputs such as graph plots.The stated research questions concern the scope and visual annotation of inputs, and on-the-fly mapping from inputs to graphical elements.Source: Focus of This Paper · p. 5
- The paper explores parameterizing the real world through sketching, focusing on continuous spatial inputs such as position, distance, and angle.Other visual variables, including shape, size, and type, and gestural interactions are outside the paper’s scope.Source: Focus of This Paper · p. 5
- For output, the system uses dynamic sketched elements, responsive graph plots, and motion-recording visual effects to analyze and interact with spatial variables.The input, output, and mapping choices can vary across application domains and user needs.Source: Focus of This Paper · p. 5
REALITYSKETCH: SYSTEM AND DESIGN
RealitySketch provides a workflow for creating responsive AR graphics by tracking objects, parameterizing them through sketches, binding those parameters to graphics, and visualizing motion. It operates over detected real-world surfaces and supports both live and internally recorded scenes.
Overview
- Users track an object, sketch parameters, bind them to graphic properties, and visualize real-world dynamics through plots or recording and playback.The workflow uses spatial variables as dynamic inputs to graphics.Source: Overview · p. 5, Figure 4 · p. 5
Basic Setup
- The mobile AR interface overlays finger- or pen-drawn sketches on the camera view, with interactions based on detected horizontal or vertical surfaces.ARKit and SceneKit support surface detection and object placement.Source: Basic Setup · p. 5
- For recorded scenes, RealitySketch stores camera and surface metadata to reconstruct the 3D scene for sketching, annotation, and visualization; this feature does not work with conventional videos.Users can control the recorded scene’s timeline in the internal player.Source: Basic Setup · pp. 5–6
Object Tracking
RealitySketch tracks selected objects and uses sketched line segments to parameterize physical quantities that change with tracked motion. Its line-based interactions also support angles, areas, and constraints between segments.
Object Tracking
- The system selects and tracks objects through color matching, while also supporting body joints through ARKit 3 human-motion tracking.Color tracking works best with distinct, solid colors; it converts detected screen positions to 3D coordinates by ray casting onto a detected surface and ran at 20–30 FPS on an iPad Pro 11-inch 2018.Source: Figure 6 · p. 6, Object Tracking · p. 6
Parameterization using Line Segments
- A sketched line segment defines the distance between two surface points, and attaching an endpoint to a tracked object makes its length change as the object moves.The system displays line-segment values with labels.Source: Parameterization using Line Segments · p. 6, Figure 7 · p. 6
- When a line connects two tracked objects, both endpoints attach to objects and the segment captures the distance between them.ARKit’s measuring capability supplies physical distance values for sketched line parameters.Source: Parameterization using Line Segments · p. 6
- Drawing between two lines can define a dynamic angle or an enclosed area, while near-perpendicular lines can receive a perpendicular constraint.The system distinguishes an angle from a distance using the drawn line’s endpoints and their proximity to existing lines.Source: Parameterization using Line Segments · p. 6
Naming Variables
Users name sketched parameters and bind them to graphics to make graphical properties respond to real-world changes. They can also plot parameter values over time or against another variable.
Naming Variables
- Users assign names such as x, y, angle, or length to existing parameters, then reuse those names for input-output mapping.The chosen name appears in the parameter label.Source: Naming Variables · p. 7, Figure 9 · p. 7
Make Graphics Responsive based on Parameter Binding
- A static segment’s parameter can be bound to a named dynamic parameter, or to a function such as angle/2, creating a constraint that updates as the source changes.The system also supports fixed-distance, fixed-direction annotations attached to moving objects to portray hidden forces; its current implementation parses basic arithmetic, trigonometric functions, and polynomial expressions.Source: Make Graphics Responsive based on… · p. 7, Figure 10 · p. 7, Figure 11 · p. 7
Visualize Motion with Responsive Graph Plots
- Binding a variable to a graph plots its time series, while assigning a variable to the x-axis enables plotting its relationship with another parameter.Multiple variables can be plotted together; the paper also illustrates visualizing a rolling-circle point’s path as a cycloid.Source: Visualize Motion with Responsive Graph… · p. 7, Figure 12 · p. 7, Figure 13 · p. 7
Visualize Motion by Recording and Replaying
RealitySketch records tracked-object positions and replays their trajectories as 3D visualizations. The paper illustrates these capabilities with physics experiments that make object motion and measured quantities visible.
Visualize Motion by Recording and Replaying
- Recording stores tracked-object positions frame by frame, and playback visualizes the trajectory with 3D spheres placed at recorded world positions.Users can adjust visualization density; a 3 cm minimum-distance default limits the number of rendered objects to avoid slow performance.Source: Visualize Motion by Recording and… · pp. 7–8
Augmented Physics Experiments
- In a pulley experiment, tracking the person’s hand and the load lets students compare their travel distances and see that the load moves a shorter distance than the person pulls.Responsive annotations, such as a force vector represented by a red line, can move with tracked positions.Source: Augmented Physics Experiments · p. 8, Figure 15 · p. 8
- For a ball thrown vertically by a moving person, recording its trajectory shows that it continues moving horizontally; comparing height graphs illustrates that its vertical trajectory is the same.The setup tracks ball height relative to the ground using perpendicular ground and vertical lines.Source: Augmented Physics Experiments · p. 8, Figure 16 · p. 8
Interactive and Explorable Concept Explanation
RealitySketch supports interactive explanations of mathematical and physical concepts through sketches tied to real-world motion. It also offers ways to visualize sports movement and check posture during exercise.
Interactive and Explorable Concept Explanation
- Tracking both gears’ rotating angles visualizes their motion, including how the larger gear rotates more slowly than the smaller one.The tool also illustrates triangle properties, sine-curve generation, and a rolling point’s cycloid trajectory.Source: Interactive and Explorable Concept… · pp. 8–9
- Teachers and students can use the visualizations for classroom presentations, collaborative exploration, or remote learning from recorded experiments.A teacher can share a visualization on a large display or share a recorded experiment for students to explore on their own devices.Source: Interactive and Explorable Concept… · p. 9
Improvised Visualization for Sports and Exercises
- For sports practice, RealitySketch can visualize ball trajectories and compare multiple attempts to help users analyze what works.Stroboscopic effects can show a ball’s path.Source: Improvised Visualization for Sports… · p. 9
- Sketched guide lines and measured body angles help users check exercise form, while recording supports later review and rehabilitation.The tool supports posture analysis in settings including yoga, cycling, and weight lifting, and can be used for co-located or remote instruction.Source: Improvised Visualization for Sports… · p. 9, Figure 19 · p. 9, Figure 18 · p. 9
In-situ Tangible User Interfaces
RealitySketch maps tracked parameters to virtual properties, enabling improvised tangible controllers. A preliminary usability study found that participants completed tasks unaided and rated the workflow and engagement positively.
In-situ Tangible User Interfaces
- Users can turn tracked parameters into improvised controllers, including toggle buttons, joysticks, 2D sliders, and dial sliders.The examples map object distance to model size, an angle to virtual-object rotation, and a tracked parameter to the number of virtual trees.Source: In-situ Tangible User Interfaces · pp. 9–10, Figure 20 · p. 9, Figure 21 · p. 9
Method
- The usability study asked six university participants to perform pendulum-motion and thrown-ball trajectory tasks after a system demonstration.Sessions took approximately 30–45 minutes.Source: Method · p. 10
Results
- All six participants completed the tasks unaided; workflow averaged 5.83 out of 7 and engagement 6.83 out of 7.Participants also described the interactions positively.Source: Results · p. 10, Method · p. 10
- Participants saw potential for classroom learning and engagement, and suggested applications including engineering visualization, sports analysis, and remote medical consultation.They also cited interactive concept exploration and design discussions as possible uses.Source: Results · p. 10
Suggestions for Future Improvements
Participants identified missing editing and media-analysis features, while expert feedback highlighted potential educational and sports uses. Instructors also wanted a way to compare experimental observations with theoretical or simulated data.
Suggestions for Future Improvements
- Participants requested undo and delete operations, easier drawing through snapping, and support for analyzing pre-existing online videos.The prototype did not support analysis of pre-existing videos.Source: Suggestions for Future Improvements · p. 10
Expert Review
- Eight instructors reviewed the application scenarios: six taught math or physics, one taught yoga, and one coached baseball.The review used face-to-face or remote discussions lasting about an hour.Source: Expert Review · p. 10
Insights and Feedback
- Teachers saw potential for learning physics with commodity devices and for exploring mathematical ideas through real-world experiments.One example discussed was manipulating four objects to explore a quadrilateral midpoint theorem.Source: Insights and Feedback · pp. 10–11
- An instructor wanted the system to compare experimental observations with theoretical or simulated data, which it did not then show.The suggested comparison included how a ball falls under different conditions.Source: Insights and Feedback · p. 11
- Sports and exercise instructors saw posture-analysis value for novice learners, though sketching could feel tedious for everyday use by experts.They also considered sketching engaging for end users.Source: Insights and Feedback · p. 11
LIMITATIONS AND FUTURE WORK
The prototype has limits in tracking robustness, constraints, parameterization, and interaction setup, while its preliminary evaluation needs longer-term study. Future work includes broader interaction configurations and applications; the conclusion presents RealitySketch as a tool for responsive AR graphics.
More General and Robust Object Tracking
- Simple color matching becomes unstable with non-solid or similarly colored objects; tested detection algorithms were more robust in some circumstances but ran at 0.1–5 FPS on mobile phones and required training.The authors anticipate that computer-vision and camera-hardware advances may provide more robust, general, and fast tracking methods.Source: More General and Robust Object Tracking · p. 11
Spatial Interactions for Augmented Storytelling
- Mobile AR interactions are limited to the tablet’s 2D screen; HMDs or projection mapping could support spatial and whole-body interaction, including sketching directly on world surfaces.The authors identify interactive storytelling, interaction prototyping, and augmented presentations as possible applications of alternative configurations.Source: Spatial Interactions for Augmented… · p. 11
Parameterization and Visualization beyond Physical Motion
- Simple constraints limit possible visualizations, and current parameterizations cover spatial properties such as length, angle, and area rather than broader real-world variables.Future directions include other visual variables, sensor-driven inputs, 3D outputs, broader mappings, and combining pre-defined procedures with user-drawn sketches.Source: Parameterization and Visualization… · p. 11
Sketching with HMD-based Augmented and Mixed Reality
- The prototype uses mobile AR partly for its wide availability, but the authors want to extend it to HMD-based setups and explore sketching on physical surfaces or using a tablet with an HMD.They suggest in-situ tangible interaction could complement or replace mid-air gestures in HMDs.Source: Sketching with HMD-based Augmented and… · p. 11
Deployment to Actual Environments
- The preliminary evaluation is limited in scale, and the authors call for an in-depth, long-term deployment in introductory physics classes to study student engagement and learning activities..Source: Deployment to Actual Environments · p. 11
CONCLUSION
- The conclusion presents RealitySketch as a tool for sketching interactive AR graphics that respond to physical-object motion, demonstrated through a range of applications.The authors hope it encourages further exploration of embedded, responsive sketching in AR and MR.Source: CONCLUSION · p. 11