Source-linked AI summary
Intelligent Interaction Techniques (IIxT) - Proposal
Brad A. Myers
TL;DR
Many AI interfaces place chat alongside conventional GUIs rather than integrating AI into their interaction techniques. The paper proposes intelligent interaction techniques and concludes they could improve usability, effectiveness, accessibility, and consistency, while raising substantial design and infrastructure challenges.
Problem
Most current AI interfaces use chat alongside conventional GUIs, while AI remains separate from the interaction techniques themselves.
Source: §1 · p. 2Method
The paper proposes using today’s AI to make reusable interaction techniques intelligent, with examples spanning selection, widgets, search, and accessibility.
Source: §1 · p. 2, §4.1 · p. 4, §4.2 · p. 5Results
The paper concludes that low-level intelligent techniques could provide better usability, effectiveness, and accessibility within and across applications, with consistent behavior available by default.
Source: §6 · p. 7Takeaways & Limitations
Building these techniques requires addressing design, architecture, security, privacy, economic, and business challenges.
Source: §6 · p. 7Users may not be able to opt out of IIxTs, giving them higher security and privacy requirements than intelligent apps and potentially increasing platform-vendor power.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Interaction techniques (IxTs) are the low-level, reusable components out of which user interfaces are designed, including menus, scroll bars, text input fields, and also copy-paste, text-entry, and selecting objects. The IxTs for graphical user interfaces (GUIs) were well established in the 1980s, with relatively minor additions and tweaks for smartphones in the 2000s. Most of today's AI user interfaces involve a chat window, which is an excellent interaction for some tasks, but is generally considered separate from the GUI IxTs. I argue for making the IxTs themselves more intelligent, so users can freely mix modalities, even within the same interaction. This will require research into new IxTs, and also into the infrastructure that will enable these intelligent IxTs (IIxTs) to be built. There are also significant security, privacy and economic implications to this vision.
1 Introduction
Interaction techniques are reusable interface components, distinct from whole applications. The paper proposes integrating newer AI into these low-level techniques rather than keeping AI separate in chat interfaces.
1 Introduction
- Interaction techniques include widgets and actions such as menus, scrollbars, copy-and-paste, drag-and-drop, and text editing; applications are composed of many such techniques.The techniques are generally reusable across applications and often collected in toolkits or frameworks.Source: §1 · pp. 1–2
- Intelligent interaction techniques use AI, making previously deterministic techniques probabilistic and therefore potentially incorrect.The paper defines the concept somewhat vaguely and cites [20, section 18.2].Source: §1 · p. 2
- Today’s AI applications are mostly chat interfaces alongside conventional GUIs, rather than AI integrated into the interaction techniques themselves.The paper notes that interaction-technique research exists, but older commercial “smart” menus were unsuccessful.Source: §1 · p. 2
- IIxTs bring AI into reusable, composable low-level interaction techniques, a direction the author argues is worth revisiting with today’s AI.The paper distinguishes this from AI interfaces that use conventional interaction techniques alongside chat.Source: §1 · p. 2
2 Why GUIs are Still Relevant
The paper argues that GUIs remain useful alongside natural language because different tasks and users benefit from different ways of finding, understanding, and acting on information.
2 Why GUIs are Still Relevant
- Some tasks are easier in natural language, but others are not, so the paper argues for augmenting interaction techniques rather than replacing them.The cited direct-manipulation properties include visible object representations and rapid, incremental, reversible operations [23].Source: §2 · p. 2
- With many items, users may switch between visual browsing and text search, then use a list to act on selected objects.The author describes preferring folder navigation for files, textual search for email, and list-based deletion for messages.Source: §2 · p. 3
- Menus can help users who do not know available actions or their names by displaying options and teaching task-specific vocabulary.Source: §2 · p. 3
- Visual representations can support understanding that words alone do not, and visual search can operate in parallel while attention is elsewhere.Visual browsing is especially useful when distinguishing features are visual; language production and consumption are sequential.Source: §2 · p. 3
3 Historical Examples of Intelligent IxTs
The paper points to earlier intelligent interaction techniques in speech, text editing, image selection, and XR, while noting that this section offers examples rather than a comprehensive review.
3 Historical Examples of Intelligent IxTs
- Put-That-There demonstrated how speech recognition could enhance pointing, including multiple references within one utterance.The author knows of no current system supporting temporal information in selections to match objects with parts of an utterance.Source: §3 · p. 3
- Text-editing intelligence dates at least to Teitelman’s 1966 Do What I Mean spelling correction system.Auto-correct, auto-complete, and auto-fill developed over time; LLMs now enable longer completions and more sophisticated editing [31].Source: §3 · pp. 3–4
- Image recognition in Photoshop can help select and extract parts of photographs, such as a person’s face or body apart from the background.The author classifies this as an interaction technique because it can be available across applications needing image selection or editing.Source: §3 · p. 4
- XR environments need interaction techniques for 3D objects because conventional 2D GUI widgets do not work well there.Image, gesture, and speech recognition are often used to improve interface accuracy and success.Source: §3 · p. 4
4 Proposed Examples of IIxTs
The proposed examples extend intelligence across interaction techniques, from selecting and searching to widgets, cross-application operations, and accessibility.
4 Proposed Examples of IIxTs
- The paper presents examples rather than a comprehensive survey, arguing that virtually all interaction techniques could potentially be made more intelligent.Source: §4 · p. 4
4.1 Intelligent Referencing
- Users could combine pointing with natural-language descriptions for semantic snapping, multiple selections, selection correction, and restoring prior selection context.Examples include selecting an iris, all phone numbers in a paragraph, or returning to where a selection task left off.Source: §4.1 · pp. 4–5
- Selection infrastructure and language should record temporal information so users can refer to multiple items in one utterance or actions from the past.The section also proposes referring to items without operating on them, including a “wiggling” technique that avoids triggering hover or click actions.Source: §4.1 · p. 5
- Search could extend beyond text to semantic properties, visual features, and associations such as creation time, location, or related items.The paper notes that graphical search had been difficult to provide and suggests AI may make it easier.Source: §4.1 · p. 5
4.2 Other Intelligent Widgets
- Semantic menus could expose relevant options, shortcuts, and icons, while intelligent customization could help users personalize commands without making them harder to use.The paper contrasts this with unsuccessful Windows XP smart menus and toolbar customization removed with Office 2007 [20, section 9.5].Source: §4.2 · p. 5
- Intelligent navigation could combine manual scrolling with jumps, bookmarks, return-to-position actions, and coordinated scrolling across documents.The paper also proposes generalized intelligent undo, including selective undo expressed in natural language, and an always-available agent with an autonomy control.Source: §4.2 · pp. 5–6
4.3 Intelligent Operations Across Applications
- Cross-application techniques could make copy-and-paste and drag-and-drop smarter, transform pasted content, and coordinate operations across independent apps.Examples include pasting as bullet points, BibTeX citations, or translated text, and grouping travel reservations from separate applications.Source: §4.3 · p. 6
4.4 Intelligent Accessibility
- Low-level intelligent interaction techniques could improve accessibility, including natural-language interaction available across interfaces for people who can type or speak.The paper notes that major-platform IxTs often have accessibility hooks, while web IxTs such as menus typically do not.Source: §4.4 · p. 6
5 Architectural considerations for IIxTs
Cross-application intelligent interactions require application and user information that current architectures do not readily provide, creating architectural, security, and privacy barriers. Addressing those barriers is necessary for intelligent agents and is complicated by business interests and the difficulty of providing sufficient access without overwhelming users.
5 Architectural considerations for IIxTs
- Most interaction techniques are provided in libraries and frameworks, but many proposed forms of intelligence require application and user knowledge, creating architectural, security, and privacy barriers.The text-entry keyboard example illustrates the privacy constraint: it may not be allowed to know much about the application.Source: §5 · p. 7
- IIxTs are likely to come from platform vendors, but users may not be able to opt out, raising security and privacy requirements and potentially increasing vendor power.Source: §5 · p. 7
- Cross-application agents need application information that is not available today; APIs and screen scraping offer limited access, while full access requires a new application architecture.The passage notes that companies may want to keep data proprietary and that providing sufficient access without overwhelming users is difficult.Source: §5 · p. 7
- Even if an intelligent agent is built, it will need substantial access to application data, so privacy, security, and architectural challenges will still need to be addressed.Source: §5 · p. 7
6 Conclusions
Providing intelligent interactions in low-level toolkits and frameworks could make them easier for developers to include and give users consistent access across applications. Significant design, architecture, security, privacy, economic, and business challenges remain.
6 Conclusions
- Low-level toolkits and frameworks could make intelligent interactions easier to include and provide consistent, ubiquitous usability, effectiveness, and accessibility across applications, though substantial challenges remain.Source: §6 · p. 7