A hybrid pipeline for dynamic ontology-based semantic mapping

1. Introduction

• SLAM builds a map from sensory data while estimating the robot’s position, but the paper describes it as insufficient for deeper understanding of entities and their relations.Perception and semantic fusion combine detections with the map and robot pose to create a semantic representation.Source: §1 · p. 1
• Prior knowledge can support interpretation of sensory data through stored object classifications and relations, represented with probabilistic models or graph structures.Examples given include knowledge graphs and semantic scene graphs.Source: §1 · p. 2
• Existing semantic-mapping systems often lack reasoning or keep object positions and knowledge bases static, motivating structural knowledge representation and dynamic updates.The stated need is for a representation that supports higher-level reasoning and updates from sensory data.Source: §1 · p. 2
• The proposed pipeline combines an external calibrated camera and homography with object detection, tracking, prior ontology knowledge, and real-time ontology updates.Its stated goal is to build a dynamic semantic map.Source: §1 · p. 2
• The contributions include image-to-world coordinate transformation and dynamic semantic spatial relations for reasoning about robot–object proximity.The pipeline uses an external PoE camera for observations, pose estimation, and object localization.Source: §1 · p. 2

2. Related work

• The review frames semantic mapping as an active research area for dynamic scene understanding and environment representation.The cited literature includes systems with semantic maps, scene graphs, and ontologies.Source: §2 · p. 2

2.1. On semantic mapping

• Semantic mapping studies cover online exploration, active mapping, planning, and navigation, with approaches differing in map dimensionality, sensors, and use of prior knowledge.The cited examples include a 3D mapping and navigation architecture [16], horticultural mapping, semantic planning, and the IntelliMove framework.Source: §2.1 · p. 3
• A related 2D method uses a monocular camera and bounding boxes for object localization, then combines scene information with wheel odometry for large-scale mapping.The paper notes that both methods use a monocular camera and aim to create 2D semantic maps.Source: §2.1 · pp. 3–4

2.2. On prior knowledge and graph representation

• Prior-knowledge approaches include embodied scene-graph generation and ontology-based reasoning that filters and post-processes scene-graph relations.The scene-graph approach uses RGB and depth images and includes navigation; the ontology-based method reports quantitative and qualitative improvements in generated scene graphs.Source: §2.2 · p. 4
• The 3D Dynamic Scene Graph represents spatial concepts and relations in layered abstraction, while Kimera builds the graph from visual data [23].The cited work reports evaluation on real-life datasets and photorealistic simulations.Source: §2.2 · pp. 4–5
• A robotics-related ontology later converted to a knowledge graph provides the basis that heavily influenced the paper’s ontology.That ontology covers objects and robot actions, with properties, and was evaluated for consistency and queried using a graph-query language.Source: §2.2 · p. 5

3. Material and Methods

• The laboratory mapping system uses an external PoE camera, trained YOLOv11s detection, calibration, undistortion, and homography to estimate object coordinates and robot pose.Ontology updates are triggered by object movement, addition or removal, or the robot approaching an object.Source: §3 · p. 5

3.1. The ontology

• The ontology has 35 classes, including physical entities such as laboratory objects and abstract entities representing observations.The paper selects an ontology for hierarchical readability and convenient modification by Python scripts; Fig. 1 shows part of its structure.Source: §3.1 · p. 5, Figure 1 · p. 6
• Observation properties store pose, confidence, and time as history, while separate current-coordinate and confidence properties on physical entities are updated dynamically.The ontology’s observes relation links an observation to a physical entity; no individuals are created by hand.Source: §3.1 · pp. 5–6

3.2.1 Camera identity

• A wall-mounted Tapo C501GW PoE/IP camera provides the scene stream for NAO localization, object detection, and pixel-to-real-coordinate conversion.The robot’s own camera was not adequate for the application, mainly because of its hardware limitations.Source: §3.2.1 · p. 6

3.2.2 Fixing distortion

• Checkerboard images estimate the camera matrix and distortion coefficients; an optimized projection matrix is then used to undistort each incoming frame.The checkerboard is 9×6, and the camera settings are stored for live-frame undistortion.Source: §3.2.2 · p. 7

3.2.3 Real world coordinates estimation

• A 180×240 cm test region supplies four corresponding corner pairs for homography; saved calibration maps detected pixel locations to world coordinates at runtime.The homography coefficients are computed from the four selected point pairs.Source: §3.2.3 · pp. 7–8

3.2.4 Camera calibration process

• The calibration workflow undistorts the camera feed, selects four real-world reference points, estimates and saves the homography, then converts detected pixels during runtime.Source: §3.2.4 · p. 9

• Object detection could identify the robot but did not provide sufficiently accurate position and orientation, motivating use of an ArUco marker.Source: §3.3 · p. 9
• The marker provides more stable detection with less jitter than a typical YOLO bounding box, is more durable to small lighting changes, and gives unique identification without training.It is mounted horizontally on the robot’s head.Source: §3.3 · p. 10
• The system averages the marker’s four corner coordinates to find its center, then applies homography to obtain the resulting point.Source: §3.3 · p. 10

• YOLOv11s was selected for accuracy and response-time balance after YOLOv4-tiny failed to consistently detect smaller and distant objects.The target was 10 object classes.Source: §3.4 · p. 10
• Training used 5,000 images from three datasets and 400 additional images captured with the external camera; the model was validated with 40 images.Source: §3.4 · p. 10
• A mAP of 99.5% was achieved with over 90% confidence factor for the 10 classes of objects.Source: §3.4 · p. 11
• The bounding-box bottom midpoint is treated as the object’s ground-contact point, then homography transforms it into real-world coordinates.Source: §3.4 · p. 11

3.5. Correcting estimated real world coordinates

• Objects averaged 5.3 cm x-axis and 17.8 cm y-axis error; NAO errors averaged 17.7 cm on x and 28.1 cm on y.The authors attribute the large errors to the oblique camera view, perspective compression, residual lens distortion, and the NAO marker’s position on its head.Source: §3.5 · pp. 11–12, Table 1 · p. 12, Table 2 · p. 12
• To meet the 3 cm MAE target, separate linear-regression models corrected x- and y-coordinates for objects and the NAO robot.The NAO used different models because its position was estimated differently from objects.Source: §3.5 · p. 12, Figure 7 · p. 13
• Object-coordinate models achieved approximately 2.8 cm MAE, while NAO models achieved approximately 1.6 cm MAE; leave-one-out errors were comparable to calibration errors.The object model had R2=0.9936; the NAO models had R2=0.9982.Source: §3.5 · pp. 13–14
• On non-seen data, all models maintained MAE <3 cm; object y-axis MAE fell to 2.4 cm.The test included newly produced instances and objects absent from the initial data set, including scissors and a backpack.Source: Table 5 · p. 14, Table 6 · p. 15, §3.5 · p. 15

3.6. Updating the ontology

• The ontology receives new object observations every 5 seconds, while coordinates update only after a position change exceeds about 5 cm.The threshold filters small detection fluctuations.Source: §3.6 · pp. 15–16
• Same-class objects are represented as separate individuals and matched across observations by nearest neighbor, but the system cannot distinguish a replacement object from the same object moving.Missing objects are removed only after they fail to reappear in several consecutive detections.Source: §3.6 · p. 16
• The task-level is_near_to relation uses a 10 cm distance threshold and is removed after the changed situation persists for three consecutive frames.Distance is computed from the robot’s and object’s current_X/current_Y coordinates.Source: Figure 8 · p. 16, §3.6 · p. 17

3.7. Running System

• The running system’s algorithm is accompanied by a flowchart in Fig. 9.The figure is identified as the hybrid pipeline for dynamic semantic mapping.Source: §3.7 · p. 17, Figure 9 · p. 17

Algorithm for running system

• For each detection, the system transforms image coordinates to real-world coordinates, applies regression correction, and creates an ontology observation.Source: Algorithm for running system · p. 17
• The loop updates ontology individuals, removes objects after enough frames with fewer detections, computes is_near_to, and saves the ontology.Source: Algorithm for running system · pp. 17–18
• Each active RTSP-stream frame is undistorted, then used to estimate robot pose from its ArUco marker and detect objects with YOLO V11s.Source: Algorithm for running system · pp. 17–18

4. Conclusion

• The external calibrated camera uses homography for geometric mapping and localization, with linear regression correcting estimated real-world coordinates.Source: §4 · p. 18
• YOLOv11s provides object detection and tracking, while the ontology stores prior knowledge and updates from camera observations in real time.Source: §4 · p. 18
• Dynamic updates to “is_near_to” enable continuous reasoning about proximity between the robot and other scene objects.Source: §4 · p. 18
• Integration into the task-planning and seamless-navigation robotic application is identified as future work.Source: §4 · p. 18