Source-linked AI summary
RoboBrain: Large-Scale Knowledge Engine for Robots
Ashutosh Saxena, Ashesh Jain, Ozan Sener, Aditya Jami, Dipendra K. Misra, Hema S. Koppula
TL;DR
Robots need fine-grained knowledge that spans physical interactions, multiple modalities, and heterogeneous sources, but existing human-oriented systems do not provide a unified representation. RoboBrain addresses this gap with a continuously updated graph and query interface that integrates such knowledge, and demonstrations show improved language grounding and path planning alongside applications in perception.
Problem
Robots require joint knowledge for language grounding, perception, planning, and control across multiple modalities, while existing human-oriented knowledge systems provide limited physical-world detail.
Method
RoboBrain builds a continuously updated graph that integrates Internet knowledge, robot interactions, and research-project representations, with a Robot Query Library for robot access.
Results
RoboBrain supports natural-language grounding, perception, and planning, and shared knowledge improves existing language-grounding and path-planning algorithms.
Takeaways & Limitations
RoboBrain provides a collaborative knowledge layer through which robots and research projects can share and query multimodal representations for diverse tasks.
Takeaways & Limitations
Some low-degree graph nodes come from Internet sources and represent abstract concepts that are not directly related to the physical world.
Abstract
from arXiv · showhide
In this paper we introduce a knowledge engine, which learns and shares knowledge representations, for robots to carry out a variety of tasks. Building such an engine brings with it the challenge of dealing with multiple data modalities including symbols, natural language, haptic senses, robot trajectories, visual features and many others. The \textit{knowledge} stored in the engine comes from multiple sources including physical interactions that robots have while performing tasks (perception, planning and control), knowledge bases from the Internet and learned representations from several robotics research groups. We discuss various technical aspects and associated challenges such as modeling the correctness of knowledge, inferring latent information and formulating different robotic tasks as queries to the knowledge engine. We describe the system architecture and how it supports different mechanisms for users and robots to interact with the engine. Finally, we demonstrate its use in three important research areas: grounding natural language, perception, and planning, which are the key building blocks for many robotic tasks. This knowledge engine is a collaborative effort and we call it RoboBrain.
I. INTRODUCTION
RoboBrain addresses the open problem of representing and retrieving fine-grained knowledge across modalities and robotics tasks. It integrates knowledge from robot interactions, Internet data, and research projects to support language grounding, perception, planning, and control.
- Motivation: Human-oriented knowledge systems lack the physical and task-specific details robots need for perception, planning, control, and language understanding.Examples include grounding symbols in physical entities, locating objects, and inferring grasping and manipulation plans.
- Approach: RoboBrain learns and shares knowledge representations from robot interactions, natural language, visual data, and multiple robotics research projects.Its representation includes symbols, language, visual or shape features, haptic properties, and other modalities.
- Approach: A graph structure connects heterogeneous knowledge, enabling traversals that retrieve semantic information and functional constraints for specific tasks.Examples include alternative grasps for an object and spatial constraints such as keeping a bottle on a table.
- System: The system is designed for concurrent distributed updates and retrieval through a large-scale cloud architecture and robot-facing query mechanisms.The paper introduces the Robot Query Library for interacting with RoboBrain.
- Applications: Applications cover natural-language grounding, perception, and planning, with experiments showing that shared knowledge improves language-grounding and path-planning algorithms.The system is presented as-a-service so researchers can use its algorithms and representations.
II. RELATED WORK
RoboBrain extends existing knowledge bases and robot-learning systems by combining Internet knowledge with physical-world representations in a continuously updated graph. It also provides a knowledge layer through which robots can share and query heterogeneous representations.
- Knowledge bases: Traditional knowledge bases are largely text-centric, human-oriented, and insufficiently detailed for robotic manipulation and navigation.RoboBrain combines Internet sources with physical-world knowledge from project partners.
- Robot learning: RoboBrain stores perception knowledge such as point clouds, grasping features, images, and videos, connecting it to human-understandable concepts.The system complements prior work in object labeling, scene understanding, localization, planning, and affordances.
- Robot learning: It stores planning parameters learned by previous robotics methods so robots can query those parameters for tasks such as manipulation and path planning.Related applications include baking, folding, assembling furniture, and preparing pancakes.
- Robot platforms: Unlike cloud platforms focused on data storage or coordination, RoboBrain provides a knowledge-representation layer over shared robot data.The comparison distinguishes RoboBrain from platforms such as RoboEarth and KIVA systems.
- Representation: Unlike Open-Ease’s template-based formal statements, RoboBrain represents unconstrained robotic concepts as graph nodes, including grasp features, trajectory parameters, and visual data.This representation supports knowledge from partner projects and Internet sources.
- Knowledge maintenance: RoboBrain maintains beliefs about knowledge correctness, resolves ambiguity using graph context, and incorporates crowd-sourcing feedback as supervision.Context helps distinguish meanings such as a tree from an industrial plant, while feedback can update beliefs and graph structure.
IV. KNOWLEDGE ENGINE: FORMAL DEFINITION
RoboBrain represents heterogeneous robotic knowledge as a directed, belief-annotated graph and continually updates it through acquisition and inference operations. The formalism supports graph restructuring, including node splitting and merging, as new information arrives.
- Formal graph: RoboBrain models knowledge as a directed graph whose nodes store multimodal concepts and whose typed edges represent relations.Nodes may contain images, text, videos, haptic data, affordances, features, and parameters; edges have types from a defined set.
- Formal graph: Each node and edge carries a feature-vector representation and a scalar belief representing the estimated accuracy of its information.Feature vectors depend on local graph connections, while beliefs quantify information correctness.
- A. Creating the Graph: Graph creation alternates between knowledge acquisition from feeds and inference that updates graph structure using aggregated data.Feeds are sets of edges from Internet crawlers or partner projects and are first incorporated through union operations.
- A. Creating the Graph: A feed is unioned with the existing graph to produce an intermediate graph before inference applies structural updates.The feed contains N typed edges, which are combined with the existing edge set.
- A. Creating the Graph: Inference can add or delete nodes and edges and can merge or split nodes after new knowledge is incorporated.These operations update the graph to reflect inferred relationships and revised concept assignments.
- A. Creating the Graph: A split divides one node into two and reallocates its incident edges, whereas a merge combines two nodes while updating their connected edges.When “sitting human can use a mug” is added, the Cup node is split into Cup and Mug, which are then merged with the corresponding Mug node.
- A. Creating the Graph: The graph includes examples of multiple node types and edge types, with complete lists deferred to the code documentation.The tables serve as examples rather than exhaustive inventories.
V. SYSTEM ARCHITECTURE
RoboBrain uses four interconnected layers to transform heterogeneous data into a structured knowledge graph and to support robot and user interaction. The architecture combines acquisition, parsing, storage, inference, and access mechanisms.
- System architecture: The architecture comprises knowledge acquisition, knowledge parser, knowledge storage, and knowledge inference layers.These layers process unstructured multimodal knowledge into the structured RoboBrain graph.
- Knowledge acquisition: The acquisition layer interfaces with multimodal sources, including partner projects, existing knowledge bases, and unstructured sources such as Wikipedia.Examples of crawled sources include Freebase, ImageNet, and WordNet.
- Knowledge parser: The parser converts acquired data into a consistent storage format and attaches metadata such as timestamps and source versions.It also preserves a pointer to the original source for knowledge bases that change over time.
- Knowledge storage: The storage layer uses RoboBrain-KB, a NoSQL document database cluster, for parsed feeds, user feedback, and machine-learning parameters.Large media such as images, videos, and 3D point clouds are offloaded to distributed storage.
- Knowledge inference: The inference layer routes new and updated feeds through a replicated distributed queue to machine-learning plugins that populate and update the graph database.Plugins include inference algorithms and graph builders, alongside algorithms operating over the entire graph.
- Interaction mechanisms: RoboBrain supports robot and user interaction through a Robot Query Library, public APIs, and a content-delivery network.The APIs support online learning mechanisms such as crowdsourcing, while the CDN reduces end-user latency.
VI. ROBOT QUERY LIBRARY (RQL)
The Robot Query Language lets robots retrieve and process subgraphs through graph templates, traversals, and operations over returned results. It supports both direct relation queries and ranking paths by belief.
- RQL overview: RQL provides retrieval functions and programming constructs for complex traversals of the RoboBrain graph.A representative task is finding possible ways for humans to use a cup by traversing from Human to Cup.
- Graph retrieval functions: Graph retrieval matches subgraphs against a template containing source node, directed edge, and destination node variables.The fetch operation instantiates the template variables and returns the matching values.
- Graph retrieval functions: The fetch query can retrieve objects connected to Human by a CanUse edge and return the corresponding nodes.This expresses relation-based retrieval without requiring a fixed destination node.
- Path queries: RQL can retrieve all paths from Human to Cup and sort them in decreasing order by their belief values.The paths function returns the list, and SortBy applies the belief-based ordering.
B. Programming construct functions
RQL programming constructs process retrieved subgraphs using functional operations such as map and filter. They support affordance queries and retrieval of parameters shared across one or more nodes.
- Programming constructs: Programming constructs process subgraphs returned by fetch using functional operations including map, filter, and find.These constructs extend graph retrieval with computation over returned lists.
- Programming constructs: The affordances query maps an affordance-retrieval function over every object that Human can use.The objects function first returns usable objects, then map applies affordances to each one.
- Parameter queries: The parameters query retrieves parameters associated with a single node through a HasParameters edge.It defines parameters for a node by fetching its connected parameter representation.
- Parameter queries: The joint-parameters query filters parameters whose parent count is two and whose parameter representation belongs to the second node's parameter set.The filter and Len operations identify parameters shared by both input nodes and not by another node.
VII. APPLICATIONS
RoboBrain is used as-a-service for anticipation, language grounding, and path planning by letting robots query representations learned by different projects. In anticipation, robots retrieve affordances, trajectories, and learned parameters to rank likely future human actions.
- Applications: RoboBrain supports anticipation, natural-language grounding, and path planning as-a-service across perception, language, and trajectory modalities.The applications demonstrate robots querying representations learned by partner projects.
- Applications: RoboBrain lets robots use representations learned by different partner projects without independently rebuilding the underlying task-specific knowledge.The service is intended to address multiple robotics applications through shared representations.
- 1) Anticipating human actions:: Robots anticipate human actions by querying RoboBrain for activity, affordance, and trajectory parameters, then generating and ranking possible future activities.The stored knowledge includes learned parameters, object affordances, and human trajectories from prior work.
- 1) Anticipating human actions:: The anticipation workflow first retrieves object affordances, then queries trajectory parameters for each affordance before applying learned parameters to predict future actions.RoboBrain also stores the learned parameters and supports queries for independent and joint parameters.
2) Grounding natural language:
Grounding a natural-language instruction requires identifying objects and actions, reasoning about their affordances, and checking whether generated actions are satisfiable in the environment. RoboBrain supplies knowledge for these satisfiability checks instead of relying only on manually defined preconditions.
- 2) Grounding natural language:: Natural-language grounding requires an action sequence that satisfies the instruction while identifying relevant objects and their environmental affordances.The robot must discover both the instruction’s action verbs and objects before grounding it.
- 2) Grounding natural language:: RoboBrain is queried to check whether generated actions are satisfiable in the given environment.For example, pouring water on a book should be rejected as unsatisfiable.
- 2) Grounding natural language:: The RoboBrain-based workflow replaces manually defined action preconditions with graph knowledge used during grounding.The earlier approach manually specified many conditions, such as constraints on squeezing a syrup bottle.
3) Path planning using RoboBrain:
RoboBrain supports trajectory planning by sharing task-relevant object attributes and learned planning parameters across research projects. For fragile objects, a robot can retrieve these representations to produce trajectories that reflect user preferences.
- 3) Path planning using RoboBrain:: Desirable robot trajectories must be geometrically valid and satisfy user preferences, such as moving sharp objects away from nearby humans.These preferences are commonly represented through cost functions modeling the environment and user requirements.
- 3) Path planning using RoboBrain:: RoboBrain stores and retrieves planning parameters learned by different research groups, whose cost functions were previously not shared.This provides a service for robots to access prior planning representations.
- 3) Path planning using RoboBrain:: For an egg carton, the robot queries RoboBrain for object attributes and trajectory parameters, then uses them to plan a path suited to fragile objects.The example reflects the preference to move eggs slowly and close to the table surface.
- 3) Path planning using RoboBrain:: Sharing planning knowledge lets RoboBrain connect representations from multiple research groups and support robotic applications through a common query interface.The section presents knowledge sharing as a mechanism for improving existing robotic applications.
1) Sharing knowledge from the Internet:
RoboBrain combines Internet-derived attributes and learned robotic representations so robots can select useful knowledge for planning and language grounding. The experiments report better planning with RoboBrain’s broader attribute coverage and better grounding when selecting among learned algorithms.
- 1) Sharing knowledge from the Internet:: A single Internet source may not cover the real-world situations required by robotic tasks, motivating aggregation from multiple sources.Internet knowledge has supported path planning, natural-language understanding, and object retrieval, but coverage can remain incomplete.
- 1) Sharing knowledge from the Internet:: RoboBrain connects attributes from multiple Internet sources to objects, reducing reliance on manually defining many attributes for trajectory planning.Manual attribute definition is challenging and time-consuming because objects have many relevant properties.
- 1) Sharing knowledge from the Internet:: Planning performance is best with RoboBrain attributes compared with using no attributes or attributes from OpenCyc alone.The graph retrieves the combined attributes through a single RQL query.
- 2) Sharing learned representations:: For language grounding, RoboBrain retrieves candidate algorithms and parameters, combines their likelihoods with prior correctness beliefs, and selects the highest-scoring representation.This enables choosing among multiple learned representations for a command and environment.
- 2) Sharing learned representations:: RoboBrain-based representation selection achieves better performance than the individual grounding algorithms.The comparison is reported for the learned-representation sharing experiment in Table III.
VIII. DISCUSSION AND CONCLUSION
RoboBrain connects knowledge from independent sources into a denser graph, increasing cross-project connectivity. Its experiments show that sharing these representations improves path planning and natural language grounding, while abstract Internet concepts remain weakly connected to the physical world.
- 44347 nodes and 98465 edges comprise the RoboBrain graph, which increases the average degree per node by 0.8 over independent sources.
- About fifteen thousand nodes have degree one, mostly from Internet sources such as Wikipedia and WordNet.
- RoboBrain connects projects by reducing nodes with degree 1 and 2 while increasing nodes with degree 3 or more.
- These degree-one nodes mainly represent abstract concepts that are not directly related to the physical world.
- Experiments show that sharing knowledge through RoboBrain improves existing path planning and natural language grounding algorithms.