PaperlayerView exact revision on arXiv

Source-linked AI summary

TinyFlow: Breaking the Warping Bottleneck for Edge Optical Flow

Xiao'ao Song, Chahat Deep Singh

arXiv:2610.33338v1cs.ROcs.CV

TL;DR

Optical flow networks with iterative refinement are difficult to deploy on edge accelerators because warping requires flow-dependent resampling. TinyFlow replaces warping with fixed-window correlation while retaining coarse-to-fine refinement, and reports better benchmark performance than EdgeFlowNet and NanoFlowNet alongside low trajectory error on seven of eleven visual-odometry sequences.

  • Problem

    Accurate iterative optical-flow refinement uses warping that requires flow-dependent resampling and memory access, complicating deployment on edge accelerators.

    Source: §I · p. 1, B. The Problem with Dynamic Resampling · p. 2

  • Method

    TinyFlow is a 1.26M-parameter warp-free network that retains iterative refinement using fixed-window correlation and compiles as a static INT8 graph.

    Source: §I · p. 1

  • Results

    TinyFlow outperforms EdgeFlowNet and NanoFlowNet on Sintel Clean, Sintel Final, and FlyingChairs in both precisions, and has the lowest trajectory error on seven of eleven flow-only visual-odometry sequences.

    Source: C. Optical Flow Benchmark Evaluation · p. 5, §V · p. 8

  • Takeaways & Limitations

    The same INT8 model supports flow-only visual odometry on TartanAir and UZH-FPV while retaining iterative refinement in an edge-deployable network.

    Source: §I · p. 1, §V · p. 8

Abstract

from arXiv · show

High-performance optical flow networks are difficult to deploy on resource-constrained edge accelerators because their iterative refinement warps features by the current flow, which relies on flow-dependent resampling and data-dependent memory access. We present TinyFlow, a 1.26M-parameter warp-free optical flow network that preserves iterative coarse-to-fine refinement by replacing the warp with a fixed-window correlation over constant offsets, so the whole network compiles as a single static INT8 graph. Quantization-aware fine-tuning, warp-teacher distillation, and model souping reduce the Float32-to-INT8 penalty to below 0.2 end-point error (EPE) on MPI-Sintel. TinyFlow outperforms EdgeFlowNet and NanoFlowNet on Sintel Clean, Sintel Final, and FlyingChairs in both Float32 and INT8, while using 54% fewer parameters than EdgeFlowNet, and our INT8 model achieves 6.29/7.13 EPE on Sintel Clean/Final. In flow-only monocular visual odometry on TartanAir and UZH-FPV, the same INT8 model yields the lowest trajectory error on seven of eleven sequences.

I. INTRODUCTION

TinyFlow targets the gap between accurate iterative optical flow and edge models that omit refinement, by replacing flow-dependent warping with fixed-window correlation. The paper combines this design with quantization-aware training and reports benchmark gains over EdgeFlowNet and NanoFlowNet.

I. INTRODUCTION

  • Deployable edge estimators omit iterative refinement, while warping-based refinement requires memory access patterns that static accelerator graphs cannot execute.The paper attributes the accuracy gap of deployable networks to this omission rather than to their smaller size.Source: §I · p. 1
  • TinyFlow retains iterative refinement without warping, using fixed-window correlation so the network can compile as a static INT8 graph.The approach carries flow dependence in feature values rather than memory addresses.Source: §I · pp. 1–2
  • TinyFlow outperforms EdgeFlowNet and NanoFlowNet on Sintel and FlyingChairs in Float32 and INT8, using 54% fewer parameters than EdgeFlowNet.Quantization-aware fine-tuning, warp-teacher distillation, and model souping reduce the Float32-to-INT8 penalty from over 1.0 to below 0.2 EPE on Sintel.Source: §I · p. 2

II. RELATED WORKS

  • Earlier lightweight edge models remove both warping and iteration, whereas TinyFlow re-expresses refinement in a form suited to static-dataflow INT8 accelerators.The related-work discussion distinguishes these models from approaches targeting larger embedded GPUs, which retain flow-dependent resampling or other unsupported operations.Source: §II · p. 2
  • TinyFlow combines quantization-aware fine-tuning, distillation from a warp-based teacher, and model souping for deployment.Checkpoint selection uses quantized rather than float accuracy.Source: §II · p. 2

III. TINYFLOW FRAMEWORK

TinyFlow estimates pixelwise displacement from consecutive RGB frames at three scales, refining from coarse to fine. Its deployment goal is to preserve this estimation process without dynamic resampling in the forward pass.

A. Problem Formulation

  • Flow is the pixelwise displacement from a location in the first frame to its corresponding location in the second, estimated at scales 16, 8, and 4.Estimation begins at the coarsest scale, and each finer scale refines the preceding result.Source: A. Problem Formulation · p. 2

B. The Problem with Dynamic Resampling

  • Warping aligns second-frame features to the first by sampling at coordinates shifted by the current flow, which generally requires bilinear resampling.The passage identifies grid sample as a common implementation.Source: B. The Problem with Dynamic Resampling · p. 2
  • Edge toolchains may reject flow-dependent sampling because its memory reads depend on input data and its access pattern is not fixed at compile time.Unsupported warping can prevent compilation or send execution to the host CPU, losing accelerator throughput.Source: B. The Problem with Dynamic Resampling · p. 2

C. Multi-Scale Feature Representation

TinyFlow uses a shared lightweight encoder to build features for its multi-scale flow estimation. Before matching, it projects features to keep correlation values bounded in a quantization-friendly form.

C. Multi-Scale Feature Representation

  • The shared encoder builds a multi-scale feature pyramid using depthwise-separable convolutions with pointwise nonlinearities to keep parameter count small and quantize well.The architecture caption describes three flow heads with the same architecture but separate weights.Source: C. Multi-Scale Feature Representation · p. 2, Figure 2 · p. 3
  • A learnable projector replaces feature normalization before correlation, which the paper describes as less hardware-friendly because it requires channel reduction and nonlinear, data-dependent normalization.The projector holds correlation values in a bounded range.Source: C. Multi-Scale Feature Representation · p. 2
  • The projector’s 1×1 convolution starts as an identity, while batch normalization and fixed scaling let the operations fuse into an integer-legal operation.The paper computes correlations after this projection unless stated otherwise.Source: C. Multi-Scale Feature Representation · p. 3

D. Warp-Free Matching via Fixed-Window Correlation

TinyFlow matches projected features over fixed offsets rather than warping them by the current flow. It computes axis marginals and keeps the search window centered at zero, relying on coarse-to-fine refinement to keep the true displacement within range.

D. Warp-Free Matching via Fixed-Window Correlation

  • TinyFlow correlates the two feature maps directly over a fixed window of offsets, avoiding flow-dependent resampling.The comparison uses an inner product over channels.Source: D. Warp-Free Matching via Fixed-Window… · p. 3
  • The implementation computes two axis marginals, giving each location 2(2r_s+1) correlation channels instead of (2r_s+1)^2.The update block learns to combine the marginals, and each offset is constant at compile time.Source: D. Warp-Free Matching via Fixed-Window… · p. 3

1) Relationship to Warping:

  • For the full window, warping adds no matching evidence beyond the fixed volume; it only recenters the search window on the current estimate.For the computed axis marginals, the same holds when the orthogonal residual displacement is small relative to feature smoothness along that axis.Source: 1) Relationship to Warping · p. 3
  • The fixed window stays centered at zero, with radii chosen large enough to contain the true displacement; coarse-to-fine refinement is designed to support the axis-marginal condition.The passage relates one offset at scale s to s pixels at full resolution and states the containment condition as ∥u_full(x)∥∞≤r_s s.Source: 1) Relationship to Warping · p. 3

2) Capture Range:

TinyFlow’s fixed-window marginals capture residual motion at each scale, while coarse-to-fine updates refine the flow without recentering the correlation window.

2) Capture Range:

  • With radii r16=8, r8=6, and r4=3, the capture ranges are ±128, ±48, and ±12 pixels per axis, respectively.At each scale, the window contains the true displacement when both components are within its range; the axis marginals also require features to vary slowly across the orthogonal axis over the residual displacement.Source: 2) Capture Range · p. 3

E. Coarse-to-Fine Warp-Free Flow Estimation

  • The coarsest-scale head regresses initial flow from concatenated features and cost volume, with a scaled tanh bounding output to ±d16 pixels.The final scale-4 estimate is bilinearly upsampled to full resolution for evaluation.Source: E. Coarse-to-Fine Warp-Free Flow… · pp. 3–4
  • At scales 8 and 4, flow starts from the upsampled coarser estimate, scaled by two to preserve finer-grid pixel units, then undergoes weight-tied iterative updates.The update block runs for four steps during training.Source: E. Coarse-to-Fine Warp-Free Flow… · p. 4
  • Each update adds a bounded correction to the running flow; the flow enters as a feature channel, while the cost volume is computed once per scale and never recentered.Per-step limits are ±ds pixels, with d16=4, d8=2, and d4=1; bounded steps keep flow-tensor ranges known for 8-bit representation.Source: E. Coarse-to-Fine Warp-Free Flow… · p. 4

F. Edge-Compilable Correlation

The correlation volume is implemented with fixed-weight layers that avoid input-dependent memory addresses. Ground-truth flow is supervised across scales using a robust Charbonnier loss.

F. Edge-Compilable Correlation

  • Fixed-offset shifts, feature replication, elementwise products, and grouped channel reductions implement the correlation marginals as ordinary Conv and Mul layers without input-dependent addresses.The offset-major layout is essential: a depthwise channel-major shift would make the grouped reduction combine values across offsets rather than compute each offset’s inner product.Source: F. Edge-Compilable Correlation · p. 4

G. Training Objective

  • At each scale, ground-truth flow is bilinearly downsampled and divided by the scale to express it in that grid’s pixel units.The comparison uses the final estimate at scales 8 and 4.Source: G. Training Objective · p. 4

1) Robust Supervision:

  • The data loss penalizes each flow component with a Charbonnier function that behaves like L1 on large errors and remains smooth at zero.The supplied parameters are ϵ=10^-3 and γ=0.45.Source: 1) Robust Supervision · p. 4

2) Deep Supervision:

Training supervises every recurrent estimate, weighting later iterates more heavily, and applies edge-aware smoothness regularization to the finest flow.

2) Deep Supervision:

  • Every iterate of the weight-tied update block receives supervision, with later iterates weighted more heavily.This gives each update step its own gradient.Source: 2) Deep Supervision · p. 4

3) Edge-Aware Smoothness:

  • Edge-aware smoothness favors smooth flow in homogeneous regions while relaxing the penalty across image edges.The edge weights use finite differences of the quarter-resolution grayscale first frame, with β=10.Source: 3) Edge-Aware Smoothness · p. 4
  • The total loss uses λs=1 at all scales, μ8=0.05, μ4=0.1, and ν=0.1; fine-tuning adds a distillation term.The supplied passage does not specify the individual terms associated with these weights.Source: 3) Edge-Aware Smoothness · p. 4

H. Quantization-Aware Training (QAT)

TinyFlow uses quantization-aware fine-tuning, a warp-based teacher, and checkpoint averaging and selection to reduce INT8 accuracy loss. The reported evaluation uses Sintel only for testing, not training, fine-tuning, or calibration.

H. Quantization-Aware Training (QAT)

  • Quantization-aware fine-tuning inserts fake-quantization at compiler quantization points, using per-channel symmetric INT8 weights and per-tensor asymmetric UINT8 activations.Gradients pass through fake-quantization with a straight-through estimator.Source: H. Quantization-Aware Training (QAT) · p. 4
  • A more accurate but undeployable warped TinyFlow teacher guides the quantized network alongside ground truth through quarter-resolution flow distillation.The distillation weight ranges from 0.25 to 0.5 across fine-tuning stages.Source: H. Quantization-Aware Training (QAT) · p. 4
  • The quantization penalty falls below +0.2 EPE, with Sintel Clean EPE increasing from 6.13 to 6.29 and Final from 6.95 to 7.13.Checkpoint selection and averaging use INT8 accuracy because float and quantized accuracy rank checkpoints differently.Source: H. Quantization-Aware Training (QAT) · p. 5

A. Training Setup

B. Training Mixture, Model Souping, and Inference

TinyFlow combines training-data mixing, checkpoint averaging, and additional inference iterations to improve accuracy. It leads deployable models across benchmarks and motion ranges, while iterative refinement costs latency on the tested accelerator.

B. Training Mixture, Model Souping, and Inference

  • Mixing FlyingThings3D into FlyingChairs fine-tuning lowers deploy-iteration Sintel Clean EPE from 6.58 to 6.13; averaging three checkpoints yields 6.13 Clean and 6.95 Final EPE.Deploying eight rather than four iterations per scale lowers Clean EPE by about 0.5 without changing Final; beyond eight, Clean gains are marginal and Final degrades.Source: B. Training Mixture, Model Souping,… · p. 5

C. Optical Flow Benchmark Evaluation

  • TinyFlow offers the best accuracy-size trade-off among deployable models, with second-best overall >3 px results behind NeuFlow v2.Its >3 px rates are 20.6% in F32 and 21.1% in INT8.Source: C. Optical Flow Benchmark Evaluation · p. 5
  • On Sintel Final, TinyFlow beats other deployable models across motion ranges except below 10 px, with its margin over EdgeFlowNet reaching 14.4 EPE beyond 128 px in both precisions.On Clean, it leads every deployable model below 40 px and from 80 px upward, while EdgeFlowNet retains a 0.51 EPE edge in the 40–80 px bin.Source: C. Optical Flow Benchmark Evaluation · p. 5
  • TinyFlow preserves coherent flow boundaries in the qualitative examples; EdgeFlowNet fragments large motion and NanoFlowNet produces noisy fields.On market 6, TinyFlow reaches 2.68 EPE against EdgeFlowNet’s 6.79; across the four Final rows, it has lower EPE than both deployable models.Source: C. Optical Flow Benchmark Evaluation · p. 6, Figure 3 · p. 6
  • On the Raspberry Pi 5 Hailo-8, TinyFlow runs at 10.5 FPS with 5.76 EPE, compared with EdgeFlowNet at 64 FPS and 6.38 EPE and NanoFlowNet at 190 FPS and 7.50 EPE.This 100-pair Sintel Clean sample shows iterative refinement improving accuracy at the cost of latency while TinyFlow remains smaller than EdgeFlowNet.Source: C. Optical Flow Benchmark Evaluation · p. 6

D. Visual Odometry Evaluation

The INT8 flow-only VO evaluation tests whether optical-flow model rankings carry over to camera-motion estimation on synthetic and real trajectories. TinyFlow leads overall ATE on seven of eleven sequences, but EdgeFlowNet performs better on the two indoor flights.

D. Visual Odometry Evaluation

  • In the four plotted trajectories, TinyFlow follows loops and sharp turns more closely, while EdgeFlowNet drifts on some return legs and NanoFlowNet leaves the path.The panels show one UZH-FPV flight and three TartanAir trajectories against ground truth; the black dot marks each start.Source: Figure 4 · p. 7, D. Visual Odometry Evaluation · pp. 7–8
  • On TartanAir, TinyFlow has the lowest ATE on six of eight trajectories and the lowest mean across all three metrics: 2.46 m, 0.28° and 23.9°.EdgeFlowNet’s corresponding means are 2.56 m, 0.29° and 26.1°; NanoFlowNet drifts 2.7× as far as TinyFlow.Source: D. Visual Odometry Evaluation · pp. 6–7, Table III · p. 7
  • TinyFlow has the lowest ATE on seven of eleven TartanAir and UZH-FPV sequences; EdgeFlowNet leads on four and NanoFlowNet on none.The evaluation uses INT8 deployable models and the same five-point RANSAC pose estimator, with translation scale taken from ground truth.Source: D. Visual Odometry Evaluation · pp. 6–7, Table III · p. 7
  • On the real UZH-FPV flights, TinyFlow leads all three metrics on outdoor forward 1, while EdgeFlowNet has lower mean ATE across the two indoor flights.On outdoor forward 1, TinyFlow’s ATE is 2.04 m against EdgeFlowNet’s 2.41 m; across the indoor flights, mean ATE is 5.32 m versus 4.25 m.Source: D. Visual Odometry Evaluation · pp. 6–7, Table III · p. 7

E. Robustness to Illumination Change

TinyFlow remains comparatively accurate when illumination changes are applied to the second frame, and its conclusion reports advantages over EdgeFlowNet across several optical-flow benchmarks. The conclusion also reports that TinyFlow leads NanoFlowNet across motion magnitudes and has the lowest ATE on seven of eleven VO sequences.

E. Robustness to Illumination Change

  • Across eight illumination conditions, TinyFlow stays within 5.92 to 8.99 EPE, while EdgeFlowNet rises to 27 to 61 and NanoFlowNet to 18 to 90.The tests apply gamma, brightness scaling or an additive offset to the second frame of a fixed 209-pair Sintel Clean subset; the authors attribute robustness to tanh-bounded update heads.Source: E. Robustness to Illumination Change · p. 8

V. CONCLUSION

  • TinyFlow outperforms EdgeFlowNet on Sintel Clean, Sintel Final and FlyingChairs in both precisions with 54% fewer parameters, and leads NanoFlowNet at every motion magnitude.TinyFlow has the lowest ATE on seven of eleven TartanAir and UZH-FPV sequences.Source: §V · p. 8
Loading 2610.33338v1…