PaperlayerView exact revision on arXiv

Source-linked AI summary

RV565: Cross-Layer Co-Design for Frame-Independent Video Playback on MicroPython-Class ESP32-S3 Systems

Ayushman Bhattacharya, Anwesha Chakraborty

arXiv:2610.35935v1cs.ARcs.MMcs.PFeess.IV

TL;DR

Video playback on SPI-attached microcontrollers must complete frame fetch, decode, transform, and transfer before deadlines, while devices continue operating-system work. RV565 combines a frame-independent, display-native container with a dual-core decode pipeline; it sustains 24 FPS with zero misses on content the serial path cannot schedule.

  • Problem

    Video playback on SPI-attached microcontrollers must complete frame fetch, decode, transform, and transfer before deadlines while devices continue operating-system work.

    Source: §1 · p. 1

  • Method

    RV565 combines a frame-independent, display-native container with a dual-core decode pipeline that removes inflation from the critical path.

    Source: §1 · p. 2

  • Results

    24 FPS with zero misses on content the serial path cannot schedule, with frame time reduced to display transfer within 1.9% of the wire-time bound.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Independent Deflate frames make decode cost linear in stored compressed length, allowing schedulability to be decided before playback.

    Source: §8 · p. 13

    Evaluation covers two windows of one film on one board and resolution, and reports no image-quality metric.

    Source: §7 · p. 12

Abstract

from arXiv · show

Video on SPI-attached microcontroller displays is limited less by decoding than by where each byte lives and which stages run serially. We present RV565, a decode framework for frame-independent video whose decoded representation is the panel's native RGB565 format, and evaluate it on an ESP32-S3 with two ten-second windows of an openly licensed film. A serial path - inflate, then scale into internal-memory stripes overlapped with SPI direct-memory-access (DMA) transfers - meets the 41.7 ms deadline on the lighter window but misses all 7,200 frames of the heavier one. RV565's dual-core pipeline inflates the next frame on the second core during the transfer and sustains 24 FPS with zero misses and 10.3 ms of slack on both windows; frame time then equals the display transfer, which runs at 98.1% of the 40 MHz wire-time bound. Because frames are independent, inflation time is linear in the compressed length stored in the container (pooled r^2 = 0.9987): a model fitted on one window predicts the other within 0.04 ms and identifies from metadata alone which window the serial path cannot schedule. We show why Deflate cannot exploit co-located temporal redundancy in RGB565 frames larger than 128x128 pixels and measure the price: a shared Deflate stream changes size by at most 1.1%, whereas an explicit XOR delta would save 46-80%. Inside OreoOS, a MicroPython application operating system, firmware inflation alone reaches 12.2 FPS; stage timing attributes the 80.4 ms frame time to serialized filesystem input, external-RAM placement, and blocking presentation, with 0.8 ms of interpreter overhead. Firmware, measurements, and analysis scripts are released.

1 Introduction

Microcontroller video must move each frame through storage, decoding, transformation, and display before a deadline while leaving time for operating-system work. RV565 combines host-side resampling and quantisation, independent Deflate frames, and a dual-core pipeline to address those constraints.

1 Introduction

  • SPI display transfer consumes a substantial, predictable part of each frame period, even as an interactive device must continue polling buttons and servicing operating-system work.A complete frame must be fetched, decoded, transformed to the panel’s pixel layout, and transferred before its presentation deadline.Source: §1 · p. 1
  • Existing video approaches trade off decode complexity, inter-frame dependence, and distance between decoded and panel formats, each mapping to constrained device resources.Device decode consumes CPU cycles, inter-frame state consumes scarce internal SRAM, and format conversion adds a full-frame memory pass.Source: §1 · pp. 1–2
  • RV565 uses host-side resampling and colour quantisation, intra-only frames, and lossless Deflate payloads that decode directly to panel-native RGB565.Its framework includes a validated container, dual-core pipeline, absolute-deadline scheduler, and metadata-based cost model; it uses the standard tinfl inflater rather than proposing a new entropy coder.Source: §1 · p. 2
  • RV565’s dual-core pipeline sustains 24 FPS with zero misses on content the serial path cannot schedule, while bringing frame time within 1.9% of the wire-time bound.The framework also predicts schedulability from container metadata and measures the cost of frame independence.Source: §1 · p. 2

2 RGB565 as the Design Starting Point

RV565 stores frames as panel-ready, big-endian RGB565, making the decoded layout deterministic and avoiding an intermediate colour-format conversion. This choice bounds fidelity: Deflate preserves encoded bytes, but resampling and RGB565 quantisation can lose information relative to the source.

2 RGB565 as the Design Starting Point

  • RGB565 uses 16 bits per pixel: five for red, six for green, and five for blue.The additional green bit reflects comparatively greater human sensitivity to luminance detail in that spectral portion.Source: §2 · p. 2
  • RV565 stores RGB565 values in the panel’s expected big-endian byte order, so decoded frames need no intermediate RGB888 or YUV conversion.Each frame contains W × H pixels and occupies 2WH bytes before compression.Source: §2 · p. 2
  • Deflate is lossless for encoded RGB565 bytes, but resampling and RGB565 quantisation can lose fidelity relative to the original source.RV565 does not claim that 16-bit colour is visually lossless relative to the source.Source: §2 · p. 2

3 From a Gallery Feature to a Systems Problem

A Gallery video feature became a systems problem when frame work blocked continuous playback and delayed button events. Subsequent prototypes exposed successive bottlenecks in Python processing, filesystem input, and the boundary between standalone firmware and the operating system.

3 From a Gallery Feature to a Systems Problem

  • The initial Gallery player occupied the interpreter for so long that playback advanced visibly and button events waited behind media work.The problem became moving a complete frame through storage, memory, conversion, and display before its deadline while returning to the operating-system loop.Source: §3 · pp. 2–3

3.1 Moving pixels out of Python

  • A native scaler removed nested Python loops, exposing repeated frame-sized filesystem reads; RV565 v4 then used independent palette-and-index blocks in PSRAM to remove per-frame reads.The per-frame palette imposed an image-fidelity ceiling, and blocks avoided requiring one large contiguous PSRAM allocation.Source: §3.1 · p. 3

3.2 Separating the hardware limit from the OS cost

  • The standalone v6 experiment inflated frames before transferring scaled output through two internal-memory stripes, while the integrated firmware-inflation path reached 12.170 FPS with Gallery’s existing input, scaling, and display path.Serial-path measurements showed the CPU idle during much of each transfer, motivating the dual-core pipeline; the prototype history was qualitative, while v6 paths were evaluated quantitatively.Source: §3.2 · p. 3

4 Background and Previous Work

RV565 brings together display-native output, independent compression, external memory, and DMA-backed transfers under interactive MicroPython constraints. The background contrasts these design choices with prior approaches and the ESP32-S3’s resource architecture.

4 Background and Previous Work

  • RV565 combines display-native pixels, independent compression, external memory, and DMA-backed transfers within an interactive MicroPython system.The comparison is organised around decode location, inter-frame dependence, and distance between decoded and panel formats.Source: §4 · p. 3

4.1 Display-native playback

  • Display-native playback simplifies device execution but trades storage capacity and input bandwidth for that simplicity; RV565 compresses independent frames while retaining display-native decoded output.Microchip’s AN1415 describes RGB565 video playback from removable storage.Source: §4.1 · p. 3

4.2 Transform codecs and embedded decoders

  • JPEG and MJPEG decoders can output RGB565, but they perform entropy decoding, inverse transforms, and colour conversion on the device; RV565 moves transform and quantisation to the host.TJpgDec and Espressif’s ESP NEW JPEG library are examples of embedded JPEG decoders; Espressif’s multimedia framework supports MJPEG and H.264 playback.Source: §4.2 · p. 3

4.3 Inter-frame codecs and decode-cost prediction

  • Dependent-frame codecs such as H.264/AVC achieve high compression through reference-picture prediction, while RV565 makes decode cost a linear function of stored payload length.Prior work predicted per-frame complexity to schedule dependent streams on media processors; RV565’s intra-only frames expose a measured cost from container metadata.Source: §4.3 · pp. 3–4

4.4 Lossless compression and independent frames

  • Independent frames reset the Deflate dictionary at each boundary, bounding corruption, enabling direct seeking, and avoiding reconstruction of a dependency chain after a skipped frame.RV565 uses Deflate because the target has a ROM decoder and its entropy stage yields smaller payloads relevant to filesystem input.Source: §4.4 · p. 4

4.5 ESP32-S3 memory and display I/O

  • The ESP32-S3’s cores, SRAM, PSRAM, cached flash, and shared DMA do not by themselves provide a video architecture; decoded data and work must be partitioned so display transfer is not starved.ESP-IDF provides queued SPI transactions backed by DMA.Source: §4.5 · p. 4

5 The RV565 Method

RV565 moves expensive image preparation to the host and keeps device-side work bounded, using independent, display-native frames. Its architecture separates inflation, scaling, and DMA transfer so they can run on distinct execution resources.

5.1 Design goals

  • RV565 moves resize and format decisions to the host, leaving the device to parse, inflate, scale, and transfer display-native pixels.The design targets predictable per-frame work, bounded device memory, and responsive controls, assuming substantially more compute is available offline.Source: §5.1 · p. 4, §5 · p. 4

5.2 Why frame independence

  • Each RV565 v6 payload is a complete frame, enabling indexed seeking and local recovery without decoding earlier frames.After one O(N) scan builds the offset table, lookup is O(1); a damaged payload cannot change the dictionary used by later frames.Source: §5.2 · p. 4
  • Table 1 organizes relevant video and image representations by inter-frame state and dominant per-frame device work.Inter-frame state means decoder state carried between frames; device work means the dominant per-frame operation on the decoding processor.Source: Table 1 · p. 5
  • For RGB565 frames larger than 128×128 pixels, Deflate cannot reference the co-located pixel in the previous frame because it lies beyond the 32,768-byte back-reference limit.For the evaluated 180×135 frames, that distance is 48,600 bytes; exploiting this temporal redundancy requires explicit prediction and adds reference state and catch-up decoding after seeking.Source: §5.2 · p. 5

5.3 End-to-end architecture

  • The architecture overlaps scaling of one stripe with GDMA transfer while, in the dual-core configuration, the second core inflates the next frame into another buffer.Decoded frames and two 24-row output stripes reside in internal SRAM; the serial configuration instead inflates and transfers on one core.Source: §5.3 · p. 5, Figure 2 · p. 6

5.4 Container format

RV565 v6 stores each frame as a length-prefixed zlib-wrapped Deflate payload following a fixed header. Host and device checks validate the file and its decoded frames before presentation.

5.4 Container format

  • The 12-byte header stores the format identifier and version, dimensions, frame rate, flags, and frame count; each frame has a little-endian 32-bit compressed length and a zlib-wrapped Deflate payload.Inflation must produce exactly w × h × 2 bytes in RGB565 big-endian order; the flags byte is currently zero.Source: §5.4 · p. 6
  • A single scan of frame lengths builds an offset table, allowing seeks to jump directly to a frame without decoding earlier payloads.Gallery builds its table lazily at first seek; standalone firmware builds pointer and length tables when the clip opens.Source: §5.4 · p. 6
  • Known compressed lengths also provide metadata for predicting frame decode cost.The paper uses this relationship in its cost model.Source: §5.4 · p. 6

5.5 Validation and failure recovery

  • Before transfer, the host ingest tool checks every frame's metadata, decoded length, payload coverage, and trailing data, then checks the encoded artifact against media-budget and free-space limits.Malformed or interrupted conversions are rejected before they become Gallery input.Source: §5.5 · p. 6
  • Device validation enforces panel dimensions, a 1–30 FPS range, nonzero in-bounds frame counts and lengths, complete tables, no trailing bytes, and exactly 2wh output bytes per frame.Gallery currently fails closed if retrying a frame after rewind also fails; skipping a corrupt payload and resuming at the next boundary is not implemented.Source: §5.5 · p. 6
  • The board schematic documents a 40 MHz write-only ST7789 interface and a complete video-and-interaction test path; GPIO21 and GPIO33 are reserved for future input-latency markers.The panel uses four-wire write-only SPI without MISO, and its module includes a 3.3V regulator for the 5V supply.Source: Figure 3 · p. 7, §5.5 · p. 7

5.6 Host encoding

Host encoding prepares each requested window as individually compressed RGB565 frames, moving resampling and image conversion off the device.

5.6 Host encoding

  • FFmpeg resamples, Lanczos-resizes, crops, and converts each frame to RGB565BE, then compresses frames independently in zlib streams at level 1.Level 1 deliberately favors fast compression over minimum file size; encoder complexity does not affect device timing.Source: §5.6 · p. 8

5.7 Serial decode and stripe pipeline

The serial path reuses DMA-capable buffers and alternates scaling and transfer across two internal-memory stripes. A dual-core variant moves inflation of the next frame onto a worker core during the current frame's transfer.

5.7 Serial decode and stripe pipeline

  • The serial path inflates a frame into a reused buffer, then scales it into alternating 24-row internal-memory stripes while GDMA transfers the other stripe.Ten stripes cover a frame, and the chip select remains asserted across the panel write command and all stripes.Source: §5.7 · p. 8
  • For a 320×240 RGB565 frame on a 40 MHz single-lane bus, the wire-time bound limits full-screen presentation to 32.55 FPS regardless of decoding.Playback meets its target only when the upper tail of work time, not merely its mean, fits within the frame period.Source: §5.7 · p. 8

5.8 Dual-core overlapped decode

  • The dual-core path requests frame k + 1 on core 1 while core 0 scales and transfers frame k, then waits for inflation to finish before swapping buffers.Inflation leaves the critical path when it is shorter than the transfer; the approach uses a second decoded-frame buffer and a 24 KiB worker stack, and speculative prefetch can become stale if frames are skipped.Source: §5.8 · p. 8

5.9 OreoOS integration

At the measured OreoOS revision, Gallery owns playback policy while firmware only inflates frames; Gallery bounds catch-up work instead of decoding a burst when playback falls behind.

5.9 OreoOS integration

  • At the measured revision, Gallery reads length-prefixed payloads into reused buffers, firmware inflates them, then native code scales and blocking SPI writes present them serially from external RAM.The firmware module validates the decoded length without creating a temporary Python bytes object.Source: §5.9 · p. 8

5.10 Scheduling and interaction semantics

  • The standalone player advances absolute deadlines by one period, waits for the remaining time, and skips overdue frame indices without decoding them.The period is P = 1/f for clip rate f.Source: §5.10 · pp. 8–9
  • Gallery decodes at most one frame per update tick and discards delay beyond one period, so playback slows rather than skipping source frames.This bounds the time before the next input poll.Source: §5.10 · p. 9
  • Soft button callbacks run after the current native call and sample the pin then, so retention of taps shorter than a native frame operation depends on timing.The paper makes no input-latency or tap-retention claim.Source: §5.10 · p. 9

6 Evaluation

Across two ten-second windows, the dual-core path meets every deadline at 24 FPS, while the serial and integrated paths expose scheduling and integration limits.

6.1 Workloads and methodology

  • The evaluation uses two preselected ten-second windows of Big Buck Bunny and ten independently reset 720-frame runs per standalone configuration.Timing separates inflation, cumulative stripe scaling, DMA wait, and LCD phases; records are retained until the timed window ends.Source: §6.1 · p. 9

6.3 Overlap removes decode from the critical path

  • On W90, the 31.312 ms display transfer runs at 98.1% of wire time; overlapping inflation reduces frame work by 22.1% on W90 and 25.1% on W360.Inflation occupies 28–34% of the transfer it is hidden behind, leaving 10.3 ms of headroom per frame at 24 FPS.Source: §6.3 · p. 10

6.4 Decode cost is predictable from the container

  • Pooled inflation time is linear in payload length (r^2 = 0.9987), and a W90-fitted model predicts W360 inflation with 0.040 ms mean error.Because payload length is stored in the container, the threshold predicts no W90 frames and all W360 frames late on the serial path; measurements agree across all 480 source frames.Source: §6.4 · p. 10

6.2 Results

  • The dual-core path meets every deadline on both windows with 10.331 ms worst-frame slack; the serial path misses every W360 deadline, and integrated playback reaches 12.170 FPS.Table 4 reports ten pooled 720-frame runs for standalone configurations and one 720-frame run for the integrated configuration.Source: §6.2 · p. 10, Table 4 · p. 11

6.5 The cost of frame independence

  • A shared Deflate stream changes payload size by at most 1.1%, while XOR deltas save 80.5% on W90 and 46.0% on W360.Deflate cannot reach co-located pixels in the previous 48,600-byte frame; independence trades a 1.9–5.1× larger stream for O(1) seeking, local error recovery, and length-predictable decode cost.Source: §6.5 · p. 11

6.6 Anatomy of the integration gap

  • Gallery’s filesystem read takes 27.631 ms, scaling 7.641 ms, and blocking presentation 33.317 ms, while OS and interpreter residual averages 0.83 ms.Removing filesystem input alone would still leave about 52.8 ms of serial work; meeting 24 FPS inside OreoOS requires memory-resident input, SRAM stripes, scaling overlapped with DMA, and second-core inflation.Source: §6.6 · pp. 11–12

6.7 Memory

  • The dual-core standalone path uses 156,096 bytes of internal SRAM, 74,280 bytes more than the serial path’s 81,816 bytes.The difference comprises a second decoded frame, a 24 KiB worker stack, queues, and a task control block; integrated playback memory was not sampled during playback.Source: §6.7 · p. 12, Table 7 · p. 12

7 Limitations and Threats to Validity

The evaluation and deployment comparisons are bounded by a narrow test configuration and several unmeasured or uncontrolled factors. RV565 is currently targeted at silent, small RGB565 displays.

7 Limitations and Threats to Validity

  • The model and independence measurements were validated on two windows of one film, one board, and one resolution and rate; beyond-range dual-core behavior is extrapolated.The evaluation reports no image-quality metric; loss comes from resampling and RGB565 quantisation.Source: §7 · p. 12
  • No codec baseline is reported; fair comparison requires matched decoded dimensions, rate, clip range, quality, and complete resource accounting.The integrated row also differs from standalone rows in input path, memory placement, scaling, presentation, and OS use.Source: §7 · p. 12
  • The integrated path lacks the dual-core pipeline and input-latency measurement; stage timing, not controlled removal, separates its costs.The configured CPU clock is reported rather than measured.Source: §7 · pp. 12–13
  • RV565 is currently silent, RGB565-specific, and intended for small displays; streaming, synchronised audio, other processors and buses, and quality-matched baselines remain future work.A fused dual-core firmware path for OreoOS and ingest-time schedulability checking are also future work.Source: §7 · p. 13

8 Conclusion

RV565’s dual-core pipeline sustains playback on content the serial path cannot schedule, with display transfer setting frame time. The conclusion also identifies limits of independent Deflate and bottlenecks that remain inside OreoOS.

8 Conclusion

  • 24 FPS, zero misses, and 10.3 ms slack on content the serial path cannot schedule; display transfer bounds frame time at 98.1% of its wire-time limit.Source: §8 · p. 13
  • Independent Deflate frames make decode cost linear in stored compressed length and enable metadata-only schedulability decisions, but offer little size advantage over shared Deflate above 128×128 pixels.Explicit XOR deltas would save 46–80%.Source: §8 · p. 13
  • In OreoOS, stage-resolved measurements attribute remaining frame cost to serialized filesystem input, external-memory placement, and blocking presentation rather than interpreter overhead.The conclusion identifies a fused firmware operation as the next step.Source: §8 · p. 13
Loading 2610.35935v1…