PaperlayerView exact revision on arXiv

Source-linked AI summary

SAGE: A hybrid vision-language and geometry framework for work-zone pathway prediction under temporary traffic control

Jiaxi Liu, Keshu Wu, Weizhe Tang, Bofeng Cao, Sikai Chen, Xiaopeng Li, Steven T. Parker, Bin Ran

arXiv:2610.31835v1cs.CVcs.RO

TL;DR

Geometric predictors can offer plausible work-zone paths without identifying the path consistent with temporary control, while VLM trajectories may not reliably follow their own scene interpretation. SAGE ranks intact geometric candidates using VLM trajectory preference and checks them against MUTCD-derived constraints. On ROADWork validation episodes, it reduces displacement errors and hard-constraint violations relative to the geometric ranker.

  • Problem

    Geometric selection lacks an interpretation of temporary control, while a VLM can identify the hazard side yet produce a trajectory that ends there.

    Source: §1 · p. 3

  • Method

    SAGE combines VLM trajectory preference over intact geometric candidates with a structured control reading and MUTCD-derived predicates that support a bounded intervention.

    Source: §1 · p. 3, §3.5 · p. 9

  • Results

    8.3% lower ADE, 12.0% lower FDE, and 37% fewer hard-constraint-violating paths versus the geometric ranker on 2,306 ROADWork validation episodes.

    Source: §7 · p. 23

  • Takeaways & Limitations

    The trajectory opinion improves all tested generator–selector combinations, especially when pools contain device-guided alternatives and selection is weak.

    Source: §7 · p. 23

    Evaluation compares predicted paths with human demonstrations in open loop, so it does not show how selected paths interact with other road users over time.

    Source: §6.3 · p. 22

Abstract

from arXiv · show

Temporary traffic control redirects vehicles through work zones, requiring automated vehicles to connect an interpretation of the visible devices to a continuous passing path. Geometric predictors can provide several plausible paths, yet selecting the one consistent with the temporary control remains difficult. We propose SAGE (Semantic-to-Action Grounded Execution), a read-decide-execute-check framework that separates path construction, path preference, and control compliance within a shared candidate space. The central idea is to use a vision-language model's (VLM's) trajectory as a preference over intact geometric candidates. A role-decoupled generator constructs the candidates, and rank fusion combines each candidate's geometric score with its agreement with the VLM trajectory. Supervision structured around the Manual on Uniform Traffic Control Devices (MUTCD) trains SAGE-VLM to state the lane status and hazard side before its passage decision and trajectory. The predicted hazard side enters image-space predicates derived from MUTCD provisions, which support a final intervention bounded in score and displacement. On 2,306 real-world ROADWork validation episodes, the candidates include a path within 10 px of the human demonstration in 87% of episodes, yet selection regret accounts for 53% of the geometric ranker's error. SAGE reduces average and final displacement errors by 8.3% and 12.0% relative to the ranker, mostly in the far field, and hard-constraint violations by 37%. The trajectory opinion improves all four tested generator-selector combinations. SAGE is comparable to state-of-the-art VLM modules in accuracy, with a faster-responding VLM, and outperforms four other adapted driving VLMs in displacement error and safety measures. Project page: https://hhjj233.github.io/SAGE-page/.

1. Introduction

Temporary-control-aware prediction requires an AV to infer a continuous passing path from visible work-zone devices, but geometric ranking and VLM trajectories each leave a gap between path plausibility and control compliance. SAGE connects geometric candidates, VLM preference, and MUTCD-derived checks in a shared candidate space.

1. Introduction

  • AVs must interpret temporary control and produce a continuous, spatially precise passing path; work-zone crashes killed 850 people in the U.S. in 2024.Prior work addresses work-zone effects and perception, but the vehicle must still determine the continuous path through the encountered layout.Source: §1 · pp. 1–2
  • 87.4% of validation episodes contain a candidate below 10 px ADE, yet selection regret accounts for 53% of the geometric ranker’s error.Geometric selection can still choose a smooth, plausible path toward the coned-off side.Source: §1 · p. 3
  • A VLM can identify the hazard side while producing a trajectory that ends there: 2.25% of its directly executed paths do so.SAGE-VLM names the described hazard side in 92.7% of cases in which it names one.Source: §1 · p. 3
  • SAGE uses a VLM trajectory to rank intact geometric candidates, while a structured reading supplies input to checks derived from MUTCD provisions.Its role-decoupled generator, geometric ranker, and trajectory-as-opinion fusion separate candidate construction from path preference and control compliance.Source: §1 · p. 3
  • 8.3% lower ADE, 12.0% lower FDE, and 37% fewer hard-constraint violations versus the geometric ranker on 2,306 ROADWork validation episodes.The trajectory opinion also improves all four tested generator–selector combinations; SAGE is comparable to OneVL in accuracy with a faster-responding VLM and outperforms four adapted driving VLMs in displacement error and safety measures.Source: §1 · p. 3

2. Problem Formulation

The problem formulation treats prediction as choosing one future image-space pathway from a work-zone image and observed path prefix. It separates candidate generation, preference, and control compliance, with accuracy, control consistency, bounded semantic influence, and availability as desiderata.

2.1. Task definition

  • A prediction instance maps a work-zone image and observed path prefix to one future pathway, evaluated against the human driver’s subsequent path.ADE and FDE measure displacement in source-image pixels; physical-unit metrics are added in a device-anchored ground frame.Source: §2.1 · p. 4

2.2. Problem statement

  • The pathway should be accurate and control-consistent: stay out of the work space, follow channelization, and end away from the redirected-from side.SAGE checks candidates against predicates derived from MUTCD provisions.Source: §2.2 · p. 4
  • Semantic influence is bounded: the VLM trajectory only ranks candidates, output coordinates remain geometric, and intervention is bounded in score gap and mean displacement.If a foundation-model output is unavailable or rejected, SAGE returns the geometric ranker’s choice.Source: §2.2 · pp. 4–5
  • Generation constructs alternatives reflecting visible free space and device roles; SAGE’s RD-YNet supplies free-space and TTC-conditioned branches.Each branch contributes half of the candidate pool.Source: §2.2 · p. 5
  • Preference ranks candidates by geometric cost and by distance to the VLM’s complete trajectory, fusing their within-pool ranks with a fixed weight.Source: §2.2 · p. 5
  • Compliance uses the VLM’s lane status, hazard side, and passage decision; MUTCD-derived predicates use the hazard side to check candidates.A bounded intervention can replace a violating choice with a nearby compliant candidate.Source: §2.2 · p. 5
  • Generation determines which alternatives exist, while preference and compliance determine which is selected; changing the scorer cannot alter a fixed pool’s oracle capacity.Source: §2.2 · p. 5

3. SAGE: A Read–Decide–Execute–Check Framework

SAGE separates pathway construction, preference, and control checks within a shared candidate pool, with an independent geometric pipeline available as a fallback. It combines role-based alternatives, a structured VLM reading and trajectory opinion, and MUTCD-derived bounded checks.

3.1. Framework overview

  • SAGE decouples reading, deciding, execution, and checking around a shared candidate pool, while the independent generator and ranker preserve an output if VLM output is unusable.The VLM reads lane status and hazard side, then supplies a passage decision and trajectory; MUTCD-derived predicates use the hazard-side reading in the bounded check.Source: §3.1 · p. 5

3.2. Role-decoupled candidate generation

  • RD-YNet combines 60 free-space candidates with 60 TTC-conditioned candidates, preserving alternatives from both branches before receiving the VLM’s opinion or reading.The free-space branch uses drivable and off-road maps; the TTC-conditioned branch uses all six role maps and can respond to temporary devices and guidance cues.Source: §3.2 · p. 7

3.3. Reading and deciding with SAGE-VLM

  • SAGE-VLM is trained to report lane status and hazard side before its passage decision and trajectory, making the structured reading available before the path is complete.Its target also includes a scene description and goal; labels for status, hazard side, and maneuver are derived from ROADWork tags, descriptions, and demonstrated paths.Source: §3.3 · p. 8

3.4. Candidate selection with geometry and trajectory opinion

  • TAO fuses geometric and full-trajectory opinion ranks, while a held-out training-data sweep admits a source only if its optimum weight is nonzero.The fixed weight is 0.3, and selection returns to the geometric choice when no usable opinion exists; the decision-anchor ablation compresses the trajectory but discards within-maneuver shape.Source: §3.4 · p. 9

3.5. Standard-grounded checks and bounded intervention

  • MUTCD-derived predicates check endpoint side, restricted-space intrusion, and channelization-boundary crossing; Z0 may replace a flagged choice only within score-gap and displacement bounds.The predicates are image-space proxies, not a determination of regulatory compliance; provisions dependent on engineering judgment, depth, speed, or timing are not instantiated.Source: §3.5 · pp. 9–10

4. Experimental Setup

The experimental setup uses ROADWork validation episodes to assess pathway accuracy, temporary-control consistency, and safety across geometric, VLM, and combined methods. It also describes the implementation and timing conditions.

4.1. Dataset

  • 2,306 validation episodes with parsable OneVL trajectories form the evaluation set, drawn from real-world ROADWork videos recorded in 16 U.S. cities.Each episode has five observed and 15 future waypoints; trajectories are evaluated in source-image pixels after scaling from the 448 × 256 working grid.Source: §4.1 · pp. 11–12

4.2. Evaluation metrics

  • ADE and FDE are the primary displacement metrics, with errors also averaged over near, middle, and far waypoint segments.Two diagnostics distinguish recent-motion departures from candidate-pool limitations and selection regret; turning is defined by a 76.8 px lateral band.Source: §4.2 · p. 12
  • A device-anchored flat-ground pinhole model expresses pathway errors in traffic-engineering units for 996 episodes with at least eight eligible drums, using median fits for other recordings.The evaluation also reports row-matched lateral error, lane-level miss rate, and collision rate; safety scoring uses no-collision, drivable-area, channelization, progress, and curvature terms on human-filtered episodes.Source: §4.2 · p. 12
  • Control consistency includes hazard-side share, restricted-space intrusion, channelization-boundary crossing, and choices flagged by Γ.Hazard-side share is computed on 1,295 described-side episodes whose demonstrations end elsewhere; uncertainty uses 95% bootstrap intervals over 187 recordings, and an annotator reviews all 93 episodes changed by Z0 alongside matched unchanged episodes.Source: §4.2 · p. 13

4.3. Baselines

  • Baselines test geometry without language, direct VLM/VLA trajectory outputs, and SAGE’s components on a shared candidate pool.The direct-output group includes adapted driving models and a zero-shot general-purpose VLM; SAGE ablations separate the geometric layer, trajectory opinion, and bounded intervention.Source: §4.3 · p. 13

4.4. Implementation details

  • The two RD-YNet branches train for 20 epochs, while SAGE-VLM uses three epochs of LoRA fine-tuning on 17,260 examples with epoch selection on a held-out development subset.At batch size 1 on an RTX 6000 Ada, the reading arrives after about 0.7 s and the complete output after about 3 s; the geometric layer does not wait for either.Source: §4.4 · pp. 13–14

5. Results

SAGE improves work-zone path selection and control consistency by combining VLM trajectory opinions with geometric candidates and a bounded intervention. Results show gains in displacement error and hard-constraint compliance, with benefits clearest when temporary control redirects the path.

5.1. Main results

  • SAGE lowers ADE by 8.3% and FDE by 12.0% and reduces hard-violation paths by 37% versus its geometric ranker; on 137 safety-critical frames, violations fall from 34.3% to 24.1%.SAGE has the compared methods’ best safety measures and accuracy at the strongest-baseline level.Source: §5.1 · p. 14, Figure 5 · p. 15
  • The last five waypoints account for 86% of ADE reduction, consistent with turning paths departing recent motion only in the far third of the horizon in more than 90% of cases.Over the first waypoints, SAGE variants and constant-velocity extrapolation have nearly the same error.Source: §5.1 · p. 14, Figure 5 · p. 15
  • Against OneVL executed directly, SAGE has ADE of 11.79 vs. 12.74 px and FDE of 30.49 vs. 29.02 px, with fewer hard violations (1.73% vs. 2.83%).SAGE also outperforms four ROADWork-adapted driving VLMs in ADE, FDE, safety score, and hard violations.Source: §5.1 · p. 14

5.3. Reading and bounded intervention

  • SAGE-VLM agrees with human hazard-side descriptions in 92.7% of cases when it names a side; Z0 lowers flagged choices from 13.1% to 9.1% and hard violations on flagged choices from 16.9% to 12.4%.Z0 changes 93 of 2,306 selections, all within the displacement bound.Source: §5.3 · pp. 16–17, Table 5 · p. 17
  • Among 88 changed cases with definite human judgments, visible contact judgments fall from 25 to 13 and corridor-inconsistency judgments from 24 to 13.The anonymized review assessed paths changed by Z0.Source: §5.3 · p. 17, Table 6 · p. 17

5.4. Efficiency

  • Each RD-YNet branch generates 60 candidates in about 0.26 s; SAGE-VLM completes its output in 3.06 s versus 3.75 s for OneVL on the same network and code.SAGE-VLM states its reading and decision after 0.71 s, before its trajectory; OneVL’s released code takes about 30 s per frame.Source: §5.4 · p. 18, §5.3 · p. 18, Table 7 · p. 20

5.5. Qualitative analysis

  • Qualitative cases show SAGE-VLM selecting missed close candidates and Z0 replacing paths entering restricted-space or channelization proxies; in one ground-frame case, clearance rises from 0.48 m overlap to 2.35 m.The traced Denver example’s selected candidate is closer to the demonstration than the SAGE-VLM path itself (2.21 vs. 4.65 px).Source: Figure 7 · p. 18, §5.5 · p. 20

6. Discussion

SAGE assigns geometry, VLM preferences, and standards-grounded checks distinct roles in work-zone path selection. The discussion connects these roles to planner evaluation, traffic-control practice, oversight, and limits of the current evaluation.

6.1. Implications for automated driving in work zones

  • The VLM opinion helps most when a planner has good alternatives but selects weakly, especially in the far field where devices redirect the path.The discussion recommends measuring selection regret and reporting far-field errors separately; a standards-based reading also gives a well-tuned planner information it cannot provide on its own.Source: §6.1 · pp. 20, 22
  • MUTCD-structured supervision restores a stated control reading to a driving VLA while keeping its trajectory ADE at 12.15 against 12.74 px.The same supervision teaches a general-purpose checkpoint to read the described side with 79.9% agreement.Source: §6.1 · p. 22

6.2. Implications for traffic control practice and oversight

  • The opinion gains most where the path departs laterally from its heading (3.81 px), and gains 1.10–1.41 px where channelizing devices or work activity are visible.These layouts are proposed as candidates for evaluating MUTCD uniformity measures with AV perception in the loop.Source: §6.2 · p. 22
  • Current WZDx lane-status feeds could supplement or supervise the on-board reading, while disagreements could flag work-zone layouts changed after publication.The discussion notes that TTC is modified as work progresses.Source: §6.2 · p. 22
  • SAGE records each frame’s VLM reading and decision, evaluated predicates, and decision source, giving agencies and developers an account that can help diagnose failures.The discussion connects this audit trail to MUTCD crash-record monitoring and a reported restriction of AV operation after a temporary-control misreading.Source: §6.2 · p. 22

6.3. Limitations and future research

  • The evaluation is open loop, and image-space predicates cannot check provisions requiring depth or speed, such as buffer and taper lengths or flagger control.Future directions include closed-loop testing, time-resolved ground truth, faster VLM outputs, agency feeds, and broader jurisdictions and settings.Source: §6.3 · p. 22

7. Conclusion

SAGE separates pathway construction, preference, and control compliance around a shared candidate space, using VLM readings and trajectories to guide intact geometric candidates. In ROADWork validation, it improves displacement errors and constraint compliance over the geometric ranker, with gains concentrated in the far field.

7. Conclusion

  • SAGE uses a role-decoupled generator, geometric ranker, MUTCD-supervised VLM opinion, and bounded predicate-based intervention, while keeping every output an intact geometric candidate.The VLM reads lane status and hazard side, states a passage decision, and contributes its trajectory as a preference over candidates subject to a training-split admission test.Source: §7 · p. 23
  • 8.3% and 12.0%: SAGE reduces ADE and FDE relative to the geometric ranker, and reduces evaluated hard-constraint violations by 37%.Across 2,306 ROADWork validation episodes, a candidate within 10 px of the human demonstration appears in 87% of episodes, while selection regret accounts for 53% of selected-path error.Source: §7 · p. 23
  • The trajectory opinion improves every tested generator-selector combination, with the largest gains where the pool contains device-guided alternatives and selection is weak.The gain concentrates in the far horizon; the framework also achieves accuracy comparable to the released ROADWork OneVL checkpoint, responds about one-fifth sooner with the same backbone, and outperforms four adapted driving VLMs in displacement error and safety measures.Source: §7 · p. 23
  • The proposed division of labor pairs geometric action-space construction with a control-reading VLM and standards-bounded intervention; the evaluation remains open loop on real-world driving data.Future work is described as closed-loop evaluation, faster valid-time VLM outputs, and evaluation on additional work-zone datasets.Source: §7 · p. 23

CRediT authorship contribution statement

The authors divide conceptual, technical, and writing responsibilities across the team, with multiple contributors sharing supervision and manuscript review.

CRediT authorship contribution statement

Loading 2610.31835v1…