Source-linked AI summary
Revisiting Active Perception
Ruzena Bajcsy, Yiannis Aloimonos, John K. Tsotsos
TL;DR
The paper addresses how complete artificial agents can perceive by actively choosing and controlling sensing according to goals and beliefs. It presents a selective history organized around active-perception components and concludes that full perception requires an active agent, while noting persistent research and evaluation limits.
Problem
Complete artificial perception requires agents to connect beliefs, goals, sensing choices, and actions rather than rely only on fixed perception processes.
Method
The paper offers a selective historical perspective organized by decomposing active perception into constituent computations, actions, and seminal research.
Results
Full perception requires an active agent that knows why it senses, chooses what to perceive, and determines how, when, and where.
Takeaways & Limitations
Earlier active-perception contributions remain relevant, and modern computational tools may support their renewed use in robotic perception systems.
Takeaways & Limitations
The paper notes that selecting which visual field to sense receives little attention compared with blanket sensing of the full environment.
Abstract
from arXiv · showhide
Despite the recent successes in robotics, artificial intelligence and computer vision, a complete artificial agent necessarily must include active perception. A multitude of ideas and methods for how to accomplish this have already appeared in the past, their broader utility perhaps impeded by insufficient computational power or costly hardware. The history of these ideas, perhaps selective due to our perspectives, is presented with the goal of organizing the past literature and highlighting the seminal contributions. We argue that those contributions are as relevant today as they were decades ago and, with the state of modern computational tools, are poised to find new life in the robotic perception systems of the next decade.
1.0 Introduction
Active perception links belief-driven goals to deliberate sensing and action. This paper organizes decades of computational work around the components and argues that complete perception requires an active agent.
- Introduction: Active perception sets a goal from a current world belief and initiates actions intended to achieve it.
- Introduction: The historical perspective spans computational active-perception methods across visual and other sensing modalities, while emphasizing visual perception.
- Introduction: The paper distinguishes active perception from ordinary control because feedback uses processed sensory data and a priori world models.
- Introduction: The paper decomposes active perception into constituent computations and actions, linking each element to seminal research.
- Introduction: An active perceiver knows why it senses, then chooses what to perceive and determines how, when, and where.
- Introduction: Active perception includes dynamic external and internal behavior modulation, with resource constraints making sensing and processing choices necessary.
2.0 Why does an agent need to choose what to sense?
Choosing what to sense is part of purposive perception: an agent uses goals and models to select information and may act to reveal otherwise unavailable views. The section contrasts progress in attention with limited attention to selecting visual fields.
- Why does an agent need to choose what to sense?: Active perception combines perception and action around goals, requiring an organizing framework for goals and behavioral control.
- Why does an agent need to choose what to sense?: In the salad example, model-guided top-down attention searches for a knife, while movement or scene manipulation can expose an occluded knife.
- Why does an agent need to choose what to sense?: Methods for selecting what to process have made significant progress, but many open issues remain.
- Why does an agent need to choose what to sense?: Active camera control can fixate or track targets, overcome occlusion, expand the visual field, compensate for nonuniform processing, increase resolution, and alter depth of interest.
3.0 How does an agent choose what to sense?
Active perception selects what to process or sense to manage computational complexity, using attention, hypotheses, and interpretation-guided sensing. The literature distinguishes selecting image content from selecting which visual field to acquire.
- 3.0 How does an agent choose what to sense?: Early vision systems used guided processing, including oriented-line detection, attentional selection, hypothesis-model focus, and temporal prediction.Kelly’s face-recognition system guided later processing from reduced-resolution edge information; VISIONS and later work extended selection across model components and time.
- 3.0 How does an agent choose what to sense?: The complexity of selecting spatial relationships among scene objects can grow exponentially, increasing the importance of selective behavior.Garvey’s indirect search extended selection beyond image locations to spatial relationships and scene context.
- 3.0 How does an agent choose what to sense?: Attention limits visual processing because computational resources remain constrained even as hardware improves.The paper argues that current successes solve only small instances within reasonable error bounds, preserving the utility of attentional mechanisms.
- 3.0 How does an agent choose what to sense?: Tsotsos’s hypothesis sieve progressively narrows interpretation during active image sequences, making active vision less complex than passive vision under certain conditions.Active vision is not always more efficient, but the derived constraints can help an agent choose between active and passive strategies.
- 3.0 How does an agent choose what to sense?: Active perception includes selecting which visual field to sense, a problem receiving less attention than choosing what to process within an image.The paper contrasts growing progress in visual attention with continued preference for blanket environmental sensing, while reporting improvements from combining viewpoint control with attention.
4.0 How does an agent control how sensing occurs?
Controlling how sensing occurs requires mechanically aligning the agent, configuring sensor geometry and internal settings, and adapting perception to the task. Historical systems implemented these ideas through movable cameras, focus control, foveation, touch, and feedback control.
- 4.0 How does an agent control how sensing occurs?: Sensing begins with mechanical alignment, sensor-geometry configuration, and adaptation of the perception mechanism to the task.Alignment includes positioning the agent in the sensory field and setting internal sensor variables such as focus and light levels.
- 4.0 How does an agent control how sensing occurs?: The Stanford Hand-Eye project used pan, tilt, focus, and neutral-density control, while the Stanford Cart acquired multiple views through a movable camera.The Cart separated one-meter movements by ten to fifteen minutes of image processing and route planning.
- 4.0 How does an agent control how sensing occurs?: In 1979, the Stanford Cart crossed a chair-filled room without human intervention in about five hours.Its multiple-view mechanism enabled depth computation without moving the cart during image acquisition.
- 4.0 How does an agent control how sensing occurs?: Foveated cameras actively scan scenes to overcome the limitations of space-variant retinas, extending active control into the sensor’s spatial sampling pattern.Sandini and Tagliasco demonstrated computational benefits from borrowing characteristics of human vision.
- 4.0 How does an agent control how sensing occurs?: Active perception combines vision with tactile sensing because visual information is limited, with exploratory touch used to characterize surfaces and objects.The cited work includes Shape from Touch, Active Touch, and Feeling by Grasping; contextual expectations constrain geometry and manipulation functionality.
5.0 How does an agent determine when and where to sense?
Choosing when and where to sense depends on the agent’s position, orientation, task, and target-specific viewing geometry. Active systems use these variables to select viewpoints and to guide exploration toward physical and geometric properties.
- 5.0 How does an agent determine when and where to sense?: An agent must determine its body position and orientation before choosing where to sense, represented by a six-dimensional pose vector.The representation contains three position coordinates and three orientation angles relative to an arbitrary coordinate origin and gravity reference.
- 5.0 How does an agent determine when and where to sense?: After selecting a sensing target, the agent chooses sensor position, viewing angle, and distance to capture task-relevant aspects.The distance to the target is represented by r in the paper’s formulation.
- 5.0 How does an agent determine when and where to sense?: Gaze control requires predictive control structures that account for agent pose and binocular movements such as vergence.Brown’s work compared several control strategies and addressed how agent position and pose could be combined.
- 5.0 How does an agent determine when and where to sense?: The appropriate control strategy depends on the current goal, including tasks such as picking up objects, inserting pieces, or decomposing clutter.The described system combined a laser range finder with a manipulator carrying two-fingered and suction grippers.
- 5.0 How does an agent determine when and where to sense?: Exploration control can deliver material, geometric, and kinematic attributes, yielding a flow from active perception through exploration to perceptual learning.The cited procedures characterized hardness, brittleness, compliance, elasticity, shape, size, and degrees of freedom.
6.0 Empirical Issues
The paper revisits active perception as a controllable, selective perceptual process embedded in an agent's action and learning cycle. It argues that active perception remains necessary because standard datasets and passive pipelines cannot fully address context-dependent, real-time interaction with the world.
- Empirical limitations: Standard computer-vision pipelines and benchmark datasets support progress, but fixed datasets can cause research to plateau before the underlying problem is solved.The Middlebury Stereo Dataset is presented as an example where algorithms were tailored to benchmark images and performance leveled off.
- Active perception: The active-perception pipeline targets a robot's perceptual-motor loop in a live environment rather than a simulated world.Evaluation is framed around performance on the tasks the active system is intended to accomplish or on its components.
- Empirical limitations: Learning action-perception systems is difficult because actions alter the input statistics, making the environment's relevant probability distribution action dependent.The resulting hidden dependencies, redundancies, and symmetries are difficult or potentially impossible to compute directly.
- Active perception: Active perception couples sensing to action, allowing agents to select what, when, where, and how to perceive in context.The paper distinguishes active systems by their external motor actions and internal parameter changes, rather than decision-making alone.
- Research outlook: The authors argue that earlier active-perception ideas remain relevant because insufficient computational power and costly hardware previously constrained their broader use.They expect modern computational tools to support renewed use in robotic perception systems.
- Design requirements: The paper identifies active and flexible sensing, exploratory information gain, selective real-time acquisition, and learning as four necessary ingredients.Within these constraints, an agent can describe its world using perceptual and actionable primitives.