Source-linked AI summary
Collaboratively Eliciting Gestures for Geospatial Data Exploration on an MSE with Tangibles and Styluses
Karen Penaranda Valdivia, Nujaimah Ahmed, Aswah Butt, Mashrufa Orchi, Roozbeh Manshaei, Sarah Hoyos-Hoyos, Emmanuel Kyeremeh, Gabby Resch, Robert McLeman, Jamy Li, Ali Mazalek
TL;DR
The paper addresses the limited understanding of how tangibles and styluses can support collaborative geospatial visualization in multi-surface environments. It conducts a co-designed gesture elicitation study across 16 tasks, producing a taxonomy of user-defined interactions and thematic findings about modality use and collaboration. Participants generally preferred uni-modal interactions, while tangible and stylus inputs complemented one another for compound tasks.
Problem
Research has limited evidence on using tangibles and styluses collaboratively for geospatial data exploration, despite challenges with multi-touch interactions in MSEs.
Method
Thirty-six participants in novice, expert, and mixed pairs proposed interactions for 16 co-designed geospatial visualization tasks using active tangibles, a stylus, or both.
Results
Participants generally preferred uni-modal interactions, with tangible-only most frequent and stylus-only a close second; the study produced 166 unique interactions.
Takeaways & Limitations
Tangibles were suited to direct map-layer manipulation, styluses to annotations and freehand selections, and combined modalities to some compound tasks.
Takeaways & Limitations
The study’s 16 referents offered a low-level workflow with limited compound tasks, restricting the use of multi-modal interactions for complex tasks.
Abstract
from arXiv · showhide
Large tabletop displays and multi-surface environments offer potential for enhancing visual data exploration and collaborative work with geospatial datasets. These systems typically rely on multi-touch interactions, which can pose challenges when the multi-touch sensors misrepresent transitory movements as control inputs, leading to interruptions. Active tangibles and styluses offer an alternative to multi-touch interactions in MSEs, and have shown the potential to facilitate sense-making around large datasets. However, further research is needed to better understand how these modalities can be effectively leveraged for interacting with geospatial data visualizations. To address this, a gesture elicitation study was conducted in which users suggested interactions for 16 geospatial data visualization tasks, presented as a realistic collaborative workflow co-designed with geography and migration researchers. The study produced a taxonomy of user-defined gestures using tangibles and styluses for engaging with geospatial data, along with a thematic analysis of users' experiences with visualization tasks and interaction techniques.
I. INTRODUCTION
This study addresses how active tangibles and styluses can support collaborative geospatial data exploration in tabletop multi-surface environments. Using a co-designed elicitation study, it examines interaction proposals across expertise levels and identifies preferred modality patterns.
- Motivation: Multi-touch remains common in MSEs, but its technical and two-dimensional constraints can interrupt workflows and limit physical collaboration.Tangibles provide physical directness and can support broader observation of collaborative gestures.
- Study aims: The study investigated how tangibles and styluses should support collaborative geospatial exploration and how expertise influences gesture proposals.The research questions covered manipulation, collaborative workflow, and participant expertise.
- Study design: Thirty-six participants in novice, expert, and mixed pairs proposed interactions for 16 geospatial visualization tasks using tangibles, a stylus, or both.The workflow and referents were co-designed with migration, geography, and HCI collaborators.
- Findings: Across expertise levels, participants preferred uni-modal interactions, with the single tangible most popular for data exploration and direct manipulation and the stylus commonly used for annotations and selections.Multi-modal inputs complemented one another for compound tasks, often combining stylus selections with tangible confirmation or activation.
- Findings: Mixed-expertise pairs produced more matched and simpler proposals than novice or expert pairs, while the study yielded a gesture set of 166 unique interactions.The authors connect these patterns to cross-expertise dialogue and contribute design knowledge about uni-modal, bi-manual, and multi-modal interactions.
II. RELATED WORK
The related work section situates collaborative geospatial interaction within MSEs and tangible user interfaces, emphasizing their relevance to spatial information work.
- Scope: MSEs and large displays have been studied as environments for collaborative work, while TUIs can help organize spatial information in data visualizations.The section reviews geospatial TUIs, collaborative exploration, elicitation methodology, and expertise.
- Illustrative context: Figure 1 presents net migration as a background map layer with precipitation, gold mining, and conflict overlaid as front layers in Ghana.The visualization illustrates layered geospatial data relevant to the paper’s migration context.
A. Geospatial Tangible User Interfaces
Geospatial tangible user interfaces use physical objects to manipulate large-scale digital maps, but prior work leaves important gaps around migration data, styluses, and multi-screen settings.
- Motivation: GIS advances provide extensive geospatial data features, but increasing complexity can raise mental effort and cognitive overload for interdisciplinary users.The paper motivates geovisualizations that are comprehensible and easy to use.
- Geospatial TUIs: GTUIs use physical objects to manipulate large-scale digital maps and can support face-to-face collaboration, communication, and spatial-data organization.Tangibles have also been reported to facilitate filtering and help multi-expertise users understand spatial data.
- Styluses: Styluses can provide more precise input than fingers and support annotations that compensate for absent multi-touch and some advanced GIS features.These capabilities may improve the intuitiveness and fluidity of tangible interactions.
- Research gap: Existing GTUI research has focused largely on urban logistics, crisis management, and traffic simulations rather than migration studies.The authors report no prior work explicitly studying map-based migration exploration with both tangibles and styluses.
- Research gap: Prior systems generally do not examine tangible dials with styluses or combined tabletop and wall displays.Many earlier studies focus on a single tabletop display.
B. Collaborative Exploration of Large Datasets with Tangibles and MSEs
Collaborative MSEs provide shared physical and visual space for large-dataset exploration, while elicitation studies reveal user-defined gestures and can examine expertise-related differences.
- MSE collaboration: Wall and tabletop MSEs support individual and collaborative visual analytics, physical-interaction equity, and co-located problem-solving.Spatial information can also contribute conceptual and trans-disciplinary clarity in societal problem contexts.
- Multi-surface environments: Adding vertical wall displays offers more collaborative interaction space, exposes more information simultaneously, and can reduce virtual navigation demands.These benefits may help users create mental maps with less cognitive overload.
- Elicitation methodology: Elicitation studies present referents describing system outcomes and ask participants to propose symbols, which researchers group into signs and consensus sets.This method captures the most agreed-upon interaction for each referent.
- Elicitation methodology: The elicitation methodology is suited to discovering interaction techniques, tendencies, and preferences that may not appear during initial design.It is also suitable for uncovering consensus in proposed multi-modal interactions such as tangible-plus-stylus input.
- Expertise: Elicitation studies can investigate how expertise shapes interaction proposals, an issue relevant to migration research with varied geospatial-modelling familiarity.The authors identify a lack of prior elicitation studies explicitly comparing participant expertise and performance.
III. STUDY DESIGN
The study used a flexible multi-surface setup combining vertical visualization displays, a tabletop interaction surface, active tangibles, and a stylus. Participants could move around the environment while collaboratively exploring geospatial data.
- Apparatus and Input Modalities: The MSE combined vertical wall displays for visualizations with a tabletop display for interactions.The setup also included referent displays, two participants, one observer, two cameras, one stylus, and three tangibles.
- Apparatus and Input Modalities: Each pair received three pentagon-shaped active tangible dials and one pen-like stylus.The dials used rotary encoders and touchscreen displays, while their physical cases were designed to explore how affordances might influence gestures.
- Apparatus and Input Modalities: The stylus supported annotating, drawing, tapping, and circling or lassoing, while multi-touch was disabled to encourage collaboration and prevent workflow disruptions.Its precision was intended to mitigate limitations associated with the lack of multi-touch input.
- Participants: Thirty-six participants were organized into 18 expert-expert, mixed, and novice-novice pairs.Participants varied in geospatial modelling knowledge and could sit, stand, or walk around the tabletop and vertical screens.
C. Referents
The referents modeled a realistic, collaboratively designed workflow for exploring map-based visualizations and traditional interface controls. Participants proposed and documented tangible- and stylus-based interactions, which coders later standardized into unique signs.
- C. Referents: A multidisciplinary team co-designed 16 task-oriented referents using Ghanaian migration, precipitation, gold-mining, and conflict data from 1985 to 2010.The referents covered both direct map-data manipulation and interactions with menus or controls.
- C. Referents: Some referents varied time periods or data-layer selections to reduce repetitive tasks while preserving a realistic geospatial workflow.Two screens presented referent descriptions and visualizations, with selected referents showing alternate descriptions or map views.
- C. Referents: The study procedures combined priming, production, and partners activities with pre-task and post-task questionnaires.These activities were intended to mitigate legacy biases in elicitation studies and encourage collaborative proposal generation.
- C. Referents: For each referent, paired participants brainstormed interactions, then each wrote two proposals, with at least one proposal incorporating a tangible.Participants focused on different wall displays while role-playing migration researchers.
- C. Referents: Three coders reviewed proposals against video and drawings, then grouped them into unique signs by modality, manipulated data attributes, and interaction outcome.The results section reports maximum consensus, consensus-distinct ratio, and statistical analyses.
A. MC and CDR Analysis
The analysis used Morris’ maximum consensus and consensus-distinct ratio to characterize agreement when participants supplied multiple proposals per referent. Uni-modal interactions showed stronger consensus than multi-modal interactions, while tangible-only and stylus-only patterns were broadly similar.
- A. MC and CDR Analysis: 1152 proposals were grouped into 166 unique interactions or signs.The gesture coding framework and illustrative examples were summarized in Table II.
- A. MC and CDR Analysis: Morris’ MC and CDR metrics were used because standard agreement scores do not account for multiple proposals per referent.Higher MC can indicate stronger agreement on a primary interaction, while higher CDR reflects a greater proportion of interactions with some consensus.
- A. MC and CDR Analysis: 18.6% mean MC and 0.212 mean CDR were observed across all referents and modalities.These values used a consensus threshold of 2 for CDR.
- A. MC and CDR Analysis: 1 tangible only and 1 stylus only produced similar mean MC values, 14.9% versus 14.7%, while stylus-only CDR was higher, 27.5% versus 23.7%.The comparison concerns interactions using one tangible only versus one stylus only.
- A. MC and CDR Analysis: Multi-modal interactions had lower mean consensus than uni-modal interactions: tangible(s) + stylus had MC 1.2% and CDR 7.4%, while ≥2 tangibles had MC 1.6% and CDR 5.7%.The authors note that multi-modal interactions were less frequent, which limits interpretation of their lower consensus values.
1) Effect of Pair Type:
Pair composition affected proposal matching and interaction length. Mixed pairs generally produced more matched and shorter proposals than expert or novice pairs, although some participant-level matching differences did not reach significance.
- 1) Effect of Pair Type:: 15.5, p < 0.001: proposal matching differed significantly by pair type, with mixed-pair proposals more likely to match another participant’s proposal.Non-matched proposal counts were 200 for mixed, 168 for expert, and 146 for novice pairs.
- 1) Effect of Pair Type:: F(2, 1149) = 32, p < 0.001: proposal length differed significantly by pair type.Mixed-pair proposals had the fewest steps, followed by expert and novice proposals.
- 1) Effect of Pair Type:: Mixed-pair proposals averaged 2.65 steps, compared with 2.82 for expert pairs and 3.39 for novice pairs.The corresponding standard deviations were 1.13, 1.15, and 1.66.
- 1) Effect of Pair Type:: Participant-level matched proposals were higher for mixed pairs, M = 16.7, than expert pairs, M = 14, and novice pairs, M = 12.2, but the difference did not reach significance.The reported participant-level ANOVA was marginally significant, F(2, 33) = 3.1, p = 0.57.
- 1) Effect of Pair Type:: Participant-level proposal length differed significantly, F(2, 33) = 6.0, p = 0.006, with mixed pairs producing shorter interactions than uniform pairs.Follow-up tests found fewer Lasso motions in mixed pairs and more Rotate, Press, and Place motions in novice pairs.
2) Effect of Expertise:
Expertise generally did not significantly affect proposal modality, matching, or participant-level proposal counts and lengths, while mixed-expertise pairs produced more matched and simpler interactions than uniform pairs.
- Effect of Expertise: User knowledge did not significantly affect proposal modality or proposal matching.Proposal modality was not significant, χ(2)(3) = 5.5, p = 0.14; proposal matching was also not significant, χ(2)(1) = 1.5, p = 0.21.
- Effect of Expertise: Participant-level comparisons found no significant expertise differences in matched proposals, mean proposal length, or modality distribution.The tests reported were t(32) = 0.78, p = 0.44; t(28) = −1.5, p = 0.14; and χ(2)(3) = 0.30, p = 0.96, respectively.
- Effect of Expertise: The '1 tangible only' modality had the highest MC across 9 of 16 referents, followed by '1 stylus only' across 7 of 16 referents, with a tie at referent 9.These modality preferences were summarized in Table IV and Table V.
3) Participants Monitored Each Other’s Activity Within Personal Spaces and Often Provided Feedback:
Participants monitored partners’ work and exchanged feedback, but communication patterns differed by expertise pairing. Tangibles and styluses also supported visible, shared negotiation of geospatial interactions across collaborators.
- Participants Monitored Each Other’s Activity Within Personal Spaces and Often Provided Feedback: Participants frequently observed each other’s work and offered feedback or modifications while documenting proposals.This monitoring occurred within participants’ personal workspaces during elicitation.
- Participants Monitored Each Other’s Activity Within Personal Spaces and Often Provided Feedback: Expert pairs often interrupted one another with “what if” questions, focused on failure modes, and sometimes redirected proposals toward new referents.Facilitators occasionally had to refocus experts on the current task.
- Participants Monitored Each Other’s Activity Within Personal Spaces and Often Provided Feedback: Novice pairs generally communicated fluidly and supportively, while mixed pairs showed more passive exchanges and occasional hesitation caused by knowledge imbalances.In some mixed pairs, experts encouraged timid novices to contribute or reacted negatively to repeated requests for reassurance.
- Participants Monitored Each Other’s Activity Within Personal Spaces and Often Provided Feedback: Novices proposed less planar gestures, including throwing tangibles between tabletop and wall displays, whereas expert and mixed pairs mostly suggested planar gestures.These proposals were less constrained by legacy biases or physical restrictions than suggestions from expert or mixed pairs.
- Participants Monitored Each Other’s Activity Within Personal Spaces and Often Provided Feedback: After priming, participants often used tangibles for cartographic interactions and styluses for system-perspective interactions.Participants also used tangible colors to represent distinct map layers and moved layers between tabletop and wall displays.
- Participants Monitored Each Other’s Activity Within Personal Spaces and Often Provided Feedback: Tangibles and styluses provided visible gesture representations that supported communication and conflict-resolution among collaborators with mixed expertise and professions.The shared tabletop workspace combined novices’ creativity with experts’ insights on feasibility and utility.
VI. LIMITATIONS
The study’s conclusions are constrained by a predominantly young participant pool and relatively low-level workflow, while future work targets broader samples and more complex modelling tasks.
- Over 90% of participants were under 40 and recruitment predominantly came from a migration institute at an undergraduate-focused university.
- Future work will broaden participant age and discipline and evaluate interactions in a working prototype with higher-level modelling tasks.
- The 16 referents offered limited compound tasks, which may explain why multi-modal interactions were used less often for complex tasks.
- The study’s potential applicability beyond migration studies remains a proposed direction for informing collaborative tangible systems supporting mixed-methods researchers.