StructVIO : Visual-inertial Odometry with Structural Regularity of Man-made Environments

I. INTRODUCTION

• StructVIO models scenes as Atlanta worlds, detecting multiple local Manhattan worlds on-the-fly and refining their headings as observations arrive.A local world can be detected from a single horizontal line direction, supporting irregular triangular or polygonal scenes.Source: §I · p. 1
• Horizontal structural lines constrain heading, while vertical lines can make sensor roll and pitch observable even without a detected Manhattan world.The heading constraint prevents orientation error from growing in the local area, according to the paper.Source: §I · p. 1
• The method combines a minimum-parameter line representation with local-world heading state variables and automatic, on-the-fly recognition of structural lines and local worlds.When no structural line is detected, the method works like a point-based system; vertical lines may still aid estimation when no Manhattan world is detected.Source: §I · pp. 1–2

II. RELATED WORK

• Line features complement point features with extra measurements, particularly in texture-poor indoor scenes, but are harder to initialize and estimate.The paper notes that adopting lines can sometimes perform worse than using points alone, so line-related estimation issues need careful handling.Source: §II · p. 2
• Prior systems often used line features without prior orientation or orientation without line measurements; methods combining both assumed three directions and required vanishing-point initialization.StructVIO instead permits multiple local headings, refines them gradually, and does not require vanishing points at initialization.Source: §II · p. 2

III. STRUCTURAL LINES AND ATLANTA WORLDS

• An Atlanta world superposes local Manhattan worlds with different horizontal orientations; a local world need not be a box world with three perpendicular axes.Lines are anchored to the local coordinate system where first detected, and their intersection points use an inverse-depth representation (θ, ρ). Fig. 1 illustrates examples with two and three local headings.Source: §III · pp. 2–3, Figure 1 · p. 3

IV. SYSTEM OVERVIEW

• Each detected local Manhattan-world heading is added to the state and gradually refined as new observations arrive.The estimator uses an Atlanta-world model with multiple local worlds.Source: §IV · pp. 4–5
• The EKF recursively estimates the state and covariance, while point and line features are estimated outside the filter and used to derive geometric constraints among IMU poses.The state includes the IMU pose, velocity, and gyroscope and accelerometer biases.Source: §IV · p. 4
• IMU integration predicts the state, and the error-state model propagates covariance using the state-transition and IMU-noise matrices.The method uses Runge–Kutta integration and approximates within-step angular velocity and acceleration from measurements at adjacent time steps.Source: §IV · p. 4
• Structural-line measurements include each local Manhattan world’s horizontal direction, allowing the filter to estimate headings for multiple box worlds.Lines are decoupled from state estimation to reduce the number of state variables.Source: §IV · p. 5
• When a tracked structural line disappears, the estimator waits for several frames before updating with its accumulated measurements to accommodate occasional tracking failure.An update also occurs when the number of poses exceeds the maximum allowed, using features visible in selected poses.Source: §IV · p. 5
• New images append the current IMU pose to the state, while new Manhattan worlds trigger addition of their headings.Historical poses are removed when their number exceeds the maximum allowed.Source: §IV · pp. 4–5

A. Line detection & tracking

• The IMU-predicted camera pose projects each 3D structural line into the new image, where nearby detected segments are searched as candidates.3D–2D tracking uses camera motion to narrow the correspondence search and helps handle occasional tracking loss.Source: A. Line detection & tracking · p. 5
• Candidate segments are screened by positional and directional consistency, then matched using image appearance at multiple points rather than only the segment midpoint.This change from midpoint-only matching is intended to improve tracking robustness.Source: A. Line detection & tracking · p. 6
• The tracker samples evenly spaced points along each line, accepts ZNCC matches above 0.6, and selects the candidate with the most corresponding points.The paper reports that this two-phase tracking method was effective in its tests.Source: A. Line detection & tracking · p. 6

B. Recognition of structural line segments

• Vanishing points for the three Manhattan-world directions are computed from the camera orientation, with horizontal-direction points depending on the local-world heading.The vertical direction can be used even before any local Manhattan world is detected.Source: B. Recognition of structural line… · p. 6
• A segment is recognized as structural when a ray from a vanishing point to its midpoint meets the set closeness and directional-consistency thresholds.If multiple vanishing points pass the checks, the best-consistency point determines the associated Manhattan world; unmatched segments are excluded from the estimator.Source: B. Recognition of structural line… · p. 6

C. Initialization of structural lines

• Initialization selects long segments that are not close to already initialized segments, reducing redundant line initialization and distributing selected lines across the image.Segments are processed in decreasing length order, removing nearby candidates after each selection.Source: C. Initialization of structural lines · p. 6
• Vertical lines use the world-aligned frame or a virtual Manhattan world with φ0 = 0, while horizontal lines use the frame aligned with their local Manhattan world.The angular parameter θ is determined from the direction from the camera center to the line on the local parameter-space XY plane.Source: C. Initialization of structural lines · p. 6
• The line’s angular parameter is initialized from the local heading, while inverse depth receives a preset value and large uncertainty.The angular uncertainty accounts for line-detection, heading, and camera-orientation uncertainty; inverse-depth uncertainty is set to 5 to cover distances from 0.2 meter to infinity.Source: C. Initialization of structural lines · pp. 6–7

D. Triangulation of structural lines with prior knowledge

• Using only visible views in the state can yield small motion parallax and inaccurate triangulation, while using all views can be costly and rely on obsolete poses.The method maintains a prior distribution for each structural line to address this trade-off.Source: D. Triangulation of structural lines… · p. 7
• The line objective combines signed reprojection distances over visible views with a prior distribution over line geometry.The prior mean and covariance can come from initialization or historical measurements; line-detection noise is represented by a standard deviation.Source: D. Triangulation of structural lines… · p. 7
• Gauss–Newton solves the nonlinear triangulation problem in 3–5 iterations, even when the initial estimate is far from the true value.The passage attributes this to linearity from inverse-depth parameterization.Source: D. Triangulation of structural lines… · p. 7
• After triangulation, the method updates each structural line’s range from the line’s intersections with back-projection rays through the segment endpoints.The endpoints are taken from the last visible view.Source: D. Triangulation of structural lines… · p. 7

E. Handling dropped measurements of long line tracks

• Measurements outside the sliding window are often discarded, risking inaccurate line estimates, while keeping long-lived features in the state can make the state dimension too large.The dropped measurements may still carry information about line geometry.Source: E. Handling dropped measurements of… · p. 7
• The method converts dropped measurements into a prior distribution about line geometry to facilitate future updates, and it can also be applied to point features.The line formulation is described in detail.Source: E. Handling dropped measurements of… · p. 7
• When poses are removed, the prior mean is recomputed from dropped measurements by minimizing an objective with Gauss–Newton, and the new covariance is derived from the final normal equation.The covariance is updated before the next triangulation.Source: E. Handling dropped measurements of… · p. 7
• If a line’s starting frame is removed, the method reanchors it to a remaining pose and transforms its parameters and covariance into the new frame.The coordinate transformation’s Jacobian is used to update the covariance.Source: E. Handling dropped measurements of… · p. 7
• In the experiments (Section VI-B0c), adopting information accumulation in the VIO implementation reduces RMSE error to 60% of its original value.Source: E. Handling dropped measurements of… · p. 7

F. Detecting and Merging Manhattan worlds

• With an IMU, gravity makes the vertical direction observable; detection begins after vertical lines are identified and computes the ground-plane vanishing line.The approach uses vanishing points from parallel-line groups to determine world orientation.Source: F. Detecting and Merging Manhattan… · pp. 7–8
• For each candidate world, the detector extends an unidentified line to the horizontal vanishing line, estimates one horizontal direction, then derives the other from the known vertical.It counts line segments consistent with both vanishing points and repeats the search up to a maximum number of iterations.Source: F. Detecting and Merging Manhattan… · p. 8
• The largest consistent-line cluster is accepted only if it exceeds four lines, existing horizontal structural lines, and the minimum angular separation from existing worlds.The minimum separation is set to 5° in the implementation.Source: F. Detecting and Merging Manhattan… · p. 8
• When two world orientations become closer than the threshold after EKF updates, the newer world is removed and its anchored structural lines are moved to the remaining world.The covariance is adjusted during merging.Source: F. Detecting and Merging Manhattan… · p. 8

G. Outlier rejection

• Outlier rejection first applies a 95% Chi-squared gate before EKF updates, then discards lines whose reprojection error exceeds about 4 pixels after retriangulation.The authors describe the two-phase procedure as more robust against outliers than Chi-squared gating alone.Source: G. Outlier rejection · p. 8

A. Benchmark tests

• StructVIO is evaluated on EuRoC machine-hall and VICON-room sequences, with ground truth from a laser tracker and motion-capture system, respectively.The comparison uses OKVIS and VINS with their default parameters; VINS loop closing is disabled for odometry evaluation.Source: A. Benchmark tests · p. 8
• The benchmark scenes are small and richly textured, conditions in which point-only methods are expected to work well, yet the tests still show gains from structural regularity.The authors caution that the EuRoC scenes are relatively small and that the structural-information pattern requires further testing in large indoor scenes.Source: A. Benchmark tests · pp. 8–9
• StructVIO performs better than OKVIS and VINS on all EuRoC sequences except V01 02 Medium, where its RMSE is slightly larger than VINS’s, and has the lowest relative positional errors.Figure 6 reports average, minimum, and maximum relative-position errors; the feature-count comparison finds clearer gains in the machine hall than in the cluttered VICON rooms.Source: A. Benchmark tests · pp. 8–9, Figure 6 · p. 9

• The evaluation uses three buildings with irregular structures and challenging conditions including low texture, lighting changes, distant features, and rapid camera motion.For long trajectories, ArUco-based reference trajectories were validated against Vicon at about 2.8 cm accuracy.Source: B. Large Indoor Scenes · pp. 9–10
• 0.957%/0.559% versus 0.956%/0.570% average/median drift errors show that adding unconstrained lines does not improve accuracy much over points alone.The authors suggest general lines' greater degrees of freedom and lower distinguishability may sometimes negatively affect VIO.Source: B. Large Indoor Scenes · p. 10
• 0.292% mean drift error, down from 0.956%, with structural lines under the Atlanta world assumption; StructVIO achieves the best performance among tested methods.The comparison covers extensive tests in large indoor scenes.Source: B. Large Indoor Scenes · p. 10, Table II · p. 11
• The tests compare point-only, point-plus-line, and StructVIO against Tango, OKVIS, and VINS, reporting positional RMSE and maximum error in Table II.The table marks failed runs and ranks the first- and second-best results.Source: B. Large Indoor Scenes · p. 10, Table II · p. 11

C. Information accumulation

• Disabling information accumulation increased StructVIO drift error by about 150%, supporting its use to retain historical measurements for long feature tracks.The method applies information accumulation to both point and line features.Source: C. Information accumulation · p. 11
• 287% average RMSE increase on the irregular Soft and Mech scenes when Manhattan-world modeling replaces the Atlanta-world assumption.Those scenes contain oblique corridors or curvy structures.Source: D. Atlanta World v.s. Manhattan World · pp. 11–12, Table III · p. 12

VII. DISCUSSION

• The discussion focuses on feature-based VIO; the authors describe an extreme failure case when only slant walls and insufficient texture are present.The system treats slant lines as outliers and then acts as point-only VIO; the paper also notes its implementation runs at about 15 frames per second on an i7 laptop without computational-time optimization.Source: §VII · p. 12

VIII. CONCLUSION

• StructVIO outperforms the tested state-of-the-art visual-inertial odometry methods on benchmark datasets and challenging real-world tests with a Project Tango tablet.The conclusion says the tests include low texture, poor lighting, and fast camera motion.Source: §VIII · p. 12