PaperlayerView exact revision on arXiv

Source-linked AI summary

NavGPT-3: Harnessing Context in a Hierarchical Navigation Runtime

Gengze Zhou, Yicong Hong, Jiazhao Zhang, Xunyi Zhao, Jian Zhou, Zixing Lei, Zun Wang, Chongyang Zhao, Xionghui Chen, Stephen Gould, Anton van den Hengel, Qi Wu

arXiv:2610.10787v1cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

Language-model reasoning can support long-horizon embodied decisions, but physical control must respond faster than step-by-step reasoning allows. NavGPT-3 connects a reasoning model and an action policy through a harness and OS-like runtime. The complete system achieves strong benchmark results, including human-level RxR-CE success and path fidelity.

  • Problem

    Language models can reason over long horizons, but physical execution must respond to a world that continues changing during inference, and action policies can drift from instructions over long rollouts.

    Source: §1 · p. 2

  • Method

    NavGPT-3 combines context, spatial tools, persistent route state, and execution evidence in a harness, while an OS-like runtime schedules reasoning, acting, and monitoring threads and controls motion authority.

    Source: §1 · p. 2

  • Results

    NavGPT-3 sets the state of the art on R2R-CE and matches human followers on RxR-CE in success (90.43 vs. 90.4 SR) and path fidelity (78.47 vs. 77.7 nDTW).

    Source: §1 · p. 2

  • Takeaways & Limitations

    The complete embodied interface combines route inspection and revision with thread control, while VLA delegation provides frequent action updates with fewer reasoning turns.

    Source: §5 · p. 23

Abstract

from arXiv · show

Language models trained with long-horizon agentic reinforcement learning can generalize knowledge through reasoning, express precise actions, and pursue goals over many steps, raising the ceiling on what an embodied agent can understand and decide. Physical interaction, however, remains the domain of action policies, which provide dense, low-latency control. We present NavGPT-3, a harness that connects the two models, with an OS-like runtime built above it: reasoning, acting, and monitoring run as threads with their own context, tools, and permissions, while the runtime schedules them and decides which thread controls the robot's motion, so that the robot can react to sudden real-world events through interruption and thread switching. Beneath it, our action policy NavGPT VLA, trained on 19.28M examples, allocates visual tokens using codec allocation, in proportion to scene change; its 8B model alone reaches 74.51 SR on R2R-CE and leads RxR-CE with 78.19 SR. With the complete harness, NavGPT-3 sets the state of the art on R2R-CE (81.51 SR) and, for the first time, brings an autonomous agent to human level: on RxR-CE it matches human followers in success (90.43 vs. 90.4 SR) and path fidelity (78.47 vs. 77.7 nDTW) at 1 min 22 s per episode, versus roughly 3 min for a human. We comprehensively ablate the harness design and the interaction between the two models, showing how tools and the action policy shape the path from language-model reasoning to physical control: when NavGPT VLA executes the route, the reasoning loop shortens and the system's minimum reaction time falls from 3-19 s per language-model decision to 0.5-1 s per action-policy step (1-2 Hz). These results show that designing this embodied interface is central to connecting frontier language-model intelligence with low-level physical control. We will release all models, code, and evaluation records.

1 INTRODUCTION

NavGPT-3 connects language-model reasoning to fast physical control through a navigation harness and an OS-like runtime. Its VLA and runtime deliver strong navigation results while supporting more frequent action updates than language-model decisions alone.

1 INTRODUCTION

  • Navigation execution runs on the world’s clock: observations and motion continue during model inference, while hazards may require stopping before reasoning finishes.The action policy provides control but is weaker at following instructions over long rollouts, so it must return evidence for the reasoner to check.Source: §1 · p. 2
  • The harness combines context management, spatial tools, persistent route state, and execution evidence; its runtime schedules reasoning, execution, and monitoring threads and controls motion authority.Returned route images, feedback, a map, and visited places let the Planner inspect and revise delegated rollouts, while monitoring can interrupt motion without waiting for reasoning.Source: §1 · p. 2
  • NavGPT VLA predicts eight waypoints and allocates visual tokens by scene change, using a full-reference I-frame for each view’s first observation and P-frames for later observations.The paper adapts codec allocation because the visual context is bounded.Source: §1 · p. 2
  • 81.51 SR on R2R-CE establishes NavGPT-3 as state of the art, while on RxR-CE it matches human followers in success (90.43 vs. 90.4 SR) and path fidelity (78.47 vs. 77.7 nDTW).The standalone 8B NavGPT VLA leads published methods on RxR-CE with 78.19 SR.Source: §1 · p. 2

2 RELATED WORK

Prior VLN work explores language-model reasoning, learned action policies, and hierarchical planning, while NavGPT-3 connects a reasoning model and VLA policy at the system level. Its visual-context strategy allocates tokens according to scene change.

Agentic Vision-and-Language Navigation with LLMs

  • NavGPT began testing how much navigation an LLM can perform zero-shot from textual observations; later work adds maps, discussion, open-vocabulary grounding, or direct code-based embodiment control.NavGPT-2 jointly emphasizes language-model reasoning and a faster, more accurate policy, but its reasoning and actions cannot be perfectly aligned.Source: §2 · pp. 2–3

Language models over learned policies

  • Hierarchical systems pair planning models with learned controllers, while NavGPT-3 uses returned rollout evidence to revise instruction-following routes (Chiang et al., 2024; Huang et al., 2026b; Liu et al., 2026).Related systems already include graph memory, backtracking, verification, and recovery; NavGPT-3’s Planner can inspect a delegated route, return to a visited place, repair locally, and decide completion.Source: §2 · p. 3

Visual context in navigation policies

  • Navigation policies compress long visual histories by merging or pruning tokens or distributing a fixed budget across views, whereas NavGPT VLA allocates tokens in proportion to scene change (Zhang et al., 2025b; Wei et al., 2025b; Qwen Team, 2026b; Tang et al., 2026; An et al., 2026).Source: §2 · p. 3

3 METHOD: HARNESS, RUNTIME, AND VLA CO-DESIGN

NAVGPT-3 co-designs a context-and-tool harness with an OS-like runtime that coordinates the Planner, VLA, and environment. Its VLA provides optional high-frequency execution beneath this interface.

3.1.1 HARNESS AND RUNTIME ABSTRACTION

  • The harness builds model context, maintains spatial state, validates and executes permitted calls, and returns evidence; the runtime manages thread lifecycles, permissions, and motion authority.The Planner chooses the next step and task completion, while the VLA can provide high-frequency execution.Source: §3.1.1 · p. 3

3.1.2 CONTEXT, SPATIAL TOOLS, AND EXECUTION RETURNS

  • Observation, delegation, spatial revision, annotation, and termination tools return route evidence that lets the Planner inspect and revise execution while retaining final task-stop authority.A VLA stop ends one operation; the Planner decides when the task itself is complete.Source: §3.1.2 · p. 4

3.1.3 EXECUTION COORDINATION AND MOTION AUTHORITY

  • Threads run asynchronously, but runtime event priorities transfer or revoke motion authority; protective events can revoke it while inference continues, requiring fresh state and authorization to resume.Route review can pause execution for correction and then return authority to the VLA.Source: §3.1.3 · p. 4, Figure 2 · p. 4

3.2 PART II: NAVGPT VLA

  • NavGPT VLA is a shared navigation policy trained across instruction, point-goal, object-goal, tracking, and driving tasks, plus embodied question answering and visual grounding.The Planner can delegate navigation instructions to the policy.Source: §3.2 · pp. 4–5

3.2.1 TRAJECTORY POLICY AND VISUAL-CONTEXT ALLOCATION

  • The VLA predicts eight cumulative agent-frame waypoints from retained multi-view history, commits the first, and shifts tokens toward changing views when the visual budget is constrained.Its codec allocation scores later frames by thumbnail pixel difference, alongside recency and view; when budget reaches capacity, it coincides with positional allocation.Source: §3.2.1 · p. 5

3.2.2 TRAINING DATA TAXONOMY AND MIXTURE

  • The training taxonomy spans capability, embodiment and observation interface, conditions, and visual-language understanding; the mixture contains 83 sources and approximately 19.28M effective records per epoch.Grounding data pair language queries with image boxes or points, covering named objects and landmarks not specified by waypoint supervision alone.Source: §3.2.2 · pp. 6–7

3.2.3 DATA PROCESSING AND TRAINING

  • Navigation data become instruction, ordered images, and eight-waypoint targets, while QA and grounding examples use next-token supervision without trajectory targets.The models are initialized from Qwen3-VL-Instruct and trained for one epoch on the complete mixture.Source: §3.2.3 · pp. 8–9

4 EXPERIMENTS

The experiments evaluate NavGPT-3’s harness, VLA collaboration, standalone policy, and real-robot runtime, including navigation results and tests of tools, visual context, supervision, codec allocation, data scaling, and interruption handling. Qualitative cases examine how the Planner responds to VLA execution problems and incomplete progress through backtracking, correction, alternate paths, and renewed delegation.

4.1 HARNESS DEVELOPMENT THROUGH RECURSIVE SELF-IMPROVEMENT

  • 81% development SR came from combining VLA execution with spatial tools; offline recursive self-improvement searched executable harness revisions using THINK, CRITIC, EXPERIMENT, DISTILL, and REFLECT.The search used success rate on a fixed 100-episode training-split subset, and the selected interface remained fixed during evaluation, without online Planner or VLA weight updates.Source: §4.1 · p. 9, Figure 5 · p. 10

4.2 HARNESSING AND VLA ABLATION

  • Mapping and Backtrack raise success to 83 and 73 SR while cutting tokens by 41% and 33%; VLA delegation helps Opus but lowers Astra’s success despite fewer Planner turns.With VLA execution, Opus success rises from 73 to 76–81 SR and cost falls from $0.99 to $0.35–0.40 per episode; Astra turns fall from 10.71 to 3.37 while success drops to 75 SR. The VLA also lowers reaction time from 3–19 s per Planner decision to 0.78 s per step on average.Source: Table 3 · p. 11, §4.2 · pp. 11–12

4.3 VISUAL PROMPT ORGANIZATION

  • Across five visual-prompt versions, success stays within 76–78% for Astra and 73–77% for Opus, while absolute bearings use the fewest tokens for both models.The most expensive version differs by model: compact glyphs for Astra and the grounded reference for Opus.Source: Figure 6 · p. 12, §4.3 · p. 12

4.4 COMMUNICATION FREQUENCY AND COST

  • 83 SR at checks every 16 steps, versus 78 SR at 32 steps, raises SPL from 67.51 to 77.36 and cuts NE from 3.82 to 2.01 m, while tokens reach 2,319.5k per episode.More frequent checks improve the route at steep cost because each check rereads accumulated route context.Source: Table 5 · p. 13, §4.4 · p. 13

4.5 OVERALL NAVIGATION PERFORMANCE

  • 81.51 SR on R2R-CE and 90.43 SR with 78.47 nDTW on RxR-CE establish NavGPT-3’s state-of-the-art and human-level results, matching human followers’ 90.4 SR and 77.7 nDTW.On object-goal navigation, the runtime’s gains appear in success on MP3D and path efficiency on HM3D v2; on embodied question answering, accuracy is ahead on both datasets at comparable normalized steps and is level with the strongest prior system.Source: Table 6 · p. 13, §4.5 · pp. 13–14, Table 7 · p. 14

4.6 CODEC ACROSS TRAINING SCOPES; 4.7 MODEL AND DATA SCALING

  • With cross-embodiment data, codec allocation improves SR and SPL on both benchmarks at both model sizes; scaling data from 1.70M to 19.28M records raises R2R-CE and RxR-CE SR.At 4B and 8B, respectively, R2R-CE SR rises from 62.6 to 72.54 and from 65.3 to 74.51, while RxR-CE SR rises from 68.7 to 76.77 and from 70.5 to 78.19. The data effect outweighs doubling model parameters; codec gains are strongest where visual diversity and image-budget competition make change-aware allocation useful.Source: §4.6 · pp. 14–15, §4.7 · pp. 15–16, Figure 7 · p. 15

4.8 REAL-WORLD DEPLOYMENT; 4.9 QUALITATIVE RESULTS

  • In a matched Unitree Go2 comparison, overlapping reasoning with priority monitoring raised SR from 73.3 to 83.3 versus sequential queued handling and cut p95 halt latency from 1840 to 180 ms.A sequence shows a revised path through an opening and illustrates pause–revise–continue. Selected successful episodes also show recovery from overshoot, a dead end, or stalled execution, and renewed delegation.Source: §4.8 · p. 16, §4.8.2 · p. 17, §4.9 · p. 18

4.9 QUALITATIVE RESULTS

  • On R2R-CE, the Planner retraced 8.54 m to node 1 and verified a 1.2 m correction after a VLA overshoot, and recovered from a blocked aisle via the other aisle.In the blocked-aisle case, it stopped beyond the table beside the lamp.Source: Figure 10 · p. 19, Figure 12 · p. 21
  • On RxR-CE, the Planner returned from a dead-end nook and used relative motion to complete omitted route clauses at a gray cabinet and an open doorway.The doorway had a rising staircase.Source: Figure 11 · p. 20
  • On RxR-CE, after unsuccessful local correction left the stair clause unfinished, the Planner delegated the full instruction again and verified the upstairs bathroom.The second rollout reached the bathroom before the Planner stopped.Source: Figure 13 · p. 22

5 CONCLUSION

NAVGPT-3 combines a navigation harness with a runtime that coordinates thread control and motion authority. Its full system matches human followers on RxR-CE, while the harness studies connect tools, visual presentation, and VLA delegation to navigation behavior.

5 CONCLUSION

  • The complete system matches human followers on RxR-CE under continuous rather than graph-based control, while the runtime coordinates harness capabilities through thread control and motion authority.The harness manages context and navigation state, exposes spatial tools, and returns execution evidence for route repair and explicit task completion.Source: §5 · p. 23
  • Harness studies show that spatial tools support Planner navigation, visual presentation affects processing cost, and VLA delegation provides frequent action updates with fewer reasoning turns.The harness provides an interface for reasoning to inspect, revise, and control extended physical execution.Source: §5 · p. 23
  • The authors used Claude Code and Codex to help implement experiments, create figures, and draft and edit parts of the paper, then checked the assisted work against experiment outputs.They report testing AI-assisted code and taking responsibility for the final content.Source: AI USE STATEMENT · p. 24

A NAVIGATION INTERFACE IMPLEMENTATION

The appendix describes how the navigation harness validates tool use, maintains route records, and divides state and decisions across system components. It also explains how runtime threads retain context and how motion authority transfers between operations.

A NAVIGATION INTERFACE IMPLEMENTATION

  • The harness validates each tool call before execution, records routes and visited places, and prevents the Planner from adding tools or directly changing navigation state.The full configuration provides the capabilities through eight tools.Source: A NAVIGATION INTERFACE IMPLEMENTATION · p. 30

A.1 DIVISION OF RESPONSIBILITIES

  • System components own distinct state and decisions, while the main task thread invokes Planner reasoning, VLA execution, and other tools as its current operation.Planner context draws on observations, route records, formatted tool results, and provider-held conversation history.Source: §A.1 · p. 30, Figure 14 · p. 30

A.2 CAPABILITIES AND EXECUTION

  • A VLA delegation runs until a stop signal or step or episode limit; a VLA stop ends only that delegation, not the episode.The observed map distinguishes unknown from free space, and numbered nodes provide stable return targets; the configuration lacks primitive motion tools, arbitrary past-image recall, and an object detector.Source: §A.2 · pp. 30–31

B RUNTIME STATE AND TIMING

  • The appendix separates harness interfaces visible to models from thread control performed by the runtime above them.Shared state combines harness instructions and spatial records, environmental observations, and runtime records of threads and running operations.Source: B RUNTIME STATE AND TIMING · p. 31, §B.1 · p. 31

B.1 LOGICAL THREADS AND SHARED STATE

  • Each thread has its own responsibility, working history, lifecycle status, and permitted shared-state reads and proposed writes; shared spatial history records routes and places.The runtime tracks tool-call status separately, and handoffs revoke the prior operation’s motion permission while preserving thread contexts; outdated commands do not regain validity on resumption.Source: §B.1 · p. 31

B.2 SYNCHRONOUS IMPLEMENTATION

The runtime’s sequential interface constructs context, validates and executes permitted calls, and returns state and evidence. Its broader design adds permission-checked motion, asynchronous review, and prioritized handoffs, with a simpler simulator implementation used for ablations.

B.2 SYNCHRONOUS IMPLEMENTATION

  • At each sequential handoff, the harness builds context, validates and executes a permitted tool call, then returns updated state and execution evidence.The runtime controls thread execution and motion authority, while the environment executes physical motion; a VLA rollout may span multiple observations and motion steps between Planner calls.Source: §B.2 · p. 31

B.3 CONTEXT CONSTRUCTION AND STATE UPDATES

  • Each thread’s context is assembled at invocation from state and tools it may access, together with delivered events; proposed updates are checked against write permissions and target-state rules.Replaying requests exactly would also require a record of hosted-provider conversation history.Source: §B.3 · p. 31

B.4 MOTION AUTHORITY AND PROTECTIVE CONTROL

  • Motion commands require current permission, matching permission version, and satisfied preconditions; protective stops can revoke permission without waiting for the Planner.Recovery requires clearing the stop, refreshing state, and granting new permission; only the Planner decides goal completion.Source: §B.4 · p. 32

B.5 SCHEDULING AND INTERACTION FREQUENCY

  • The runtime allows activities to update on different periods or events, so the Planner can review a fixed state copy while VLA execution continues and interrupt it for revision or termination.A VLA step predicts eight waypoints but executes the first; discarded predictions and failed retries do not count as steps.Source: §B.5 · p. 32

B.6 PRIORITY AND HANDOFF POLICY

  • Protective stops and timeouts outrank episode limits, operation completion, and route reviews; stale events are discarded, and a correction waits for the current motion step to finish.The simulator ablations use a simpler implementation with version checks, completion precedence, one pending review, and VLA-history continuation; independent safety-monitor preemption is measured on Go2.Source: §B.6 · p. 32

B.7 INTERACTION-FREQUENCY EVALUATION

The frequency comparison holds the Planner, VLA, prompts, tools, route-summary format, and resumption procedure constant while varying scheduled checks. Table 12 describes component permissions at handoff.

B.7 INTERACTION-FREQUENCY EVALUATION

  • All check frequencies use the same Planner, VLA, prompt, tools, route-summary format, and history-preserving resumption procedure; checks may let the route continue without recovery.Blocked-move review requests are delivered at scheduled checks or when the VLA stops, not as extra event-triggered checks; cost includes model calls and waiting time.Source: §B.7 · pp. 32–33
  • Table 12 summarizes what each component may do when execution is interrupted.Source: Table 12 · p. 33

C VISUAL-CONTEXT ALLOCATION DETAILS

This section specifies the positional and codec allocation rules, their evaluation settings, and a synthetic check of allocator behavior.

C VISUAL-CONTEXT ALLOCATION DETAILS

  • Codec adds an RGB-change multiplier to the positional allocator, whose time weights favor recent observations; the change score also reflects camera motion and lighting, not learned semantic importance.Images are resized to their assigned pixel budgets.Source: C VISUAL-CONTEXT ALLOCATION DETAILS · p. 33
  • The allocator starts each image at bmin, distributes remaining tokens by positive joint weights, caps additions at bmax, and reallocates overflow among uncapped images.Final integer remainders go in descending weight order; ties use a fixed sort order, and excess budget is unused when every image reaches bmax.Source: C VISUAL-CONTEXT ALLOCATION DETAILS · p. 33
  • Evaluation uses Bvis = 3072, γ = 2, camera weights (2, 1, 0.5, 1), and at most 16 observations; allocation is budget-limited from four observations onward.The first and current observations are always retained, with intermediate observations sampled uniformly without replacement.Source: C VISUAL-CONTEXT ALLOCATION DETAILS · p. 33
  • All integer bounds and allocation totals hold in the synthetic check, and four saturated rows produce identical allocations under both rules.The diagnostic is a synthetic calculation, not a navigation result.Source: C VISUAL-CONTEXT ALLOCATION DETAILS · pp. 33–34, Table 13 · p. 34

D.1 EVALUATION SETTINGS

The supplied evaluation material reports human-comparable RxR-CE performance and notes that benchmark goal distances and success labels are reserved for offline evaluation.

D.1 EVALUATION SETTINGS

  • Benchmark goal distances and success labels are available only to the offline evaluator.The supplied table caption identifies Table 14 as harness evaluation settings but gives no setting values.Source: §D.1 · p. 34, Table 14 · p. 34

D.2 HUMAN PERFORMANCE ON RXR-CE

  • 90.43 SR and 78.47 nDTW match English human followers’ 90.4 SR and 77.7 nDTW on RxR-CE despite continuous motion’s added control burden.Human demonstrations use discrete graph edges, abstracting away low-level movement and obstacle avoidance.Source: §D.2 · p. 34

D.3 CROSS-TASK EVALUATION

These passages delimit cross-task reporting and clarify the scaffold, accounting, and visual-prompt conditions used in evaluation.

D.3 CROSS-TASK EVALUATION

  • Per-category MP3D results are omitted because proportional sampling leaves the rarest object-goal categories with about two episodes each.The passage also notes that prior EXPRESS-Bench scores used a now-deprecated GPT-4o-mini judge.Source: §D.3 · p. 34

D.4 PLANNER SCAFFOLDS AND ACCOUNTING

  • NAVGPT-3 replaces scaffold task prompts with its own navigation prompt and tools, while scaffolds still assemble conversations and manage context windows.Cross-Planner comparisons concern complete Planner–scaffold combinations, not underlying models in isolation.Source: §D.4 · pp. 34–35
  • Tokens sum processed input and output, turns count Planner responses between environment feedback, and cost is hosted inference charge per attempted episode.VLA latency is reported separately; inference timing excludes communication and harness overhead, and action-update intervals exclude interrupt and braking time.Source: §D.4 · p. 35

D.5 VISUAL PROMPT VARIANTS

  • Visual-prompt variants retain the same tools, frames, route keyframes, pose readings, annotations, geometry, and tool behavior while changing map or label presentation.Labels remain outside scene content, so variants do not change occlusion.Source: §D.5 · p. 35

E.1 R2R ABLATION SUBSET

The ablation subset is compared with the full R2R-CE split on selected episode attributes, while the text cautions against treating that comparison as full representativeness or protocol equivalence.

E.1 R2R ABLATION SUBSET

  • 100 ablation episodes match the public OpenNav subset and R2R-CE v1.3 Val-Unseen on listed episode attributes; Table 15 summarizes subset coverage and distributions.The compared attributes include instruction, scene, start position, reference path, and geodesic distance.Source: §E.1 · p. 35, Table 15 · p. 35
  • The reported task-complexity proxies are interpretable but do not provide a complete measure of semantic complexity.Path distance and directional mentions have small mean differences, while subset instructions are shorter and have narrower length distribution.Source: §E.1 · pp. 35–36
  • 10 of 11 scenes are covered, and scene-frequency total-variation distance is 0.097; the observed imbalance is typical of random subsets but does not establish representativeness.The subset is curated, and one scene accounts for 3% of it versus 9.63% of the full split.Source: §E.1 · p. 36
  • The public OpenNav and v1.3 files differ in initial headings for all 100 episodes, which are fixed for evaluation; full-split results remain the benchmark-comparison basis.The subset comparison establishes similarity on listed attributes, not full execution-protocol or performance equivalence.Source: §E.1 · p. 36

E.2 MATCHED OBJECTNAV EVALUATION

The matched ObjectNav evaluation compares NAVGPT-3 and Qwen-RobotNav 8B on controlled, identical episode conditions. In EXPRESS judge validation, GPT-6 Astra agrees with human labels more often than GPT-4o-mini, though item-level pairing is unavailable.

E.2 MATCHED OBJECTNAV EVALUATION

  • ∆SR vs. full was −1.0 on MP3D and +1.2 on HM3D v2 for Qwen-RobotNav 8B on the selected 500-episode subsets.The passage says these deltas indicate how closely the selected episodes track the full split; NAVGPT-3 has no full-split run.Source: §E.2 · p. 36, Table 16 · p. 36
  • Table 16 compares Qwen-RobotNav 8B and NAVGPT-3 on identical episodes, scene versions, sensing, motion budgets, collision handling, and stop/success rules.It is a system comparison; NAVGPT-3 additionally uses its hosted Planner.Source: §E.2 · p. 36, Table 16 · p. 36

E.3 EXPRESS JUDGE VALIDATION

  • Without item-level pairing, the exact McNemar p-value for the judge comparison lies between 0.002 and 0.031.This caveat applies to open-answer EXPRESS scoring; HM-EQA and MT-HM3D multiple-choice accuracy uses the reference option.Source: §E.3 · p. 36
  • On 50 human-labeled answers, GPT-6 Astra agreed with human labels on 46 versus 36 for GPT-4o-mini, a 20.0-point difference.The totals permit an exact McNemar p-value between 0.002 and 0.031, depending on how the judges’ errors overlap.Source: Table 17 · p. 36, §E.3 · p. 36
Loading 2610.10787v1…