PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Computers

Mingchen Zhuge, Changsheng Zhao, Haozhe Liu, Zijian Zhou, Shuming Liu, Wenyi Wang, Ernie Chang, Gael Le Lan, Junjie Fei, Wenxuan Zhang, Yasheng Sun, Zhipeng Cai, Zechun Liu, Yunyang Xiong, Yining Yang, Yuandong Tian, Yangyang Shi, Vikas Chandra, Jürgen Schmidhuber

arXiv:2604.06425v2cs.LGcs.AI

TL;DR

The paper asks whether elementary computer-like primitives can be learned from collected I/O traces without instrumented program state. It models Neural Computers as video systems that roll out interface frames from prompts, pixels, and actions, finding early I/O alignment and short-horizon control but unresolved reuse and symbolic stability. The authors use these results to motivate a roadmap toward Completely Neural Computers.

  • Problem

    The paper asks whether a neural system can learn computation, memory, and I/O as a unified runtime state from collected I/O traces without instrumented program state.

  • Method

    The authors build video-based CLI and GUI Neural Computer prototypes that model interface frames from prompts, pixels, and synchronized user actions.

  • Results

    Current NCs exhibit early runtime primitives, especially I/O alignment and short-horizon control, while symbolic reliability and stable capability reuse remain unresolved.

  • Takeaways & Limitations

    The results support a staged roadmap toward a Completely Neural Computer as a fully learned computer with unified compute, memory, and interfaces.

  • Takeaways & Limitations

    CNC-level reliability likely requires architectural and training ingredients beyond scaling current video generators.

Abstract

from arXiv · show

We propose a new frontier: Neural Computers (NCs) that unify computation, memory, and I/O of traditional computers in a learned runtime state. Our long-term goal is the Completely Neural Computer (CNC): the mature, general-purpose realization of this emerging machine form, with stable execution, explicit reprogramming, and durable capability reuse. As an initial step, we study whether elementary NC primitives can be learned solely from collected I/O traces, without instrumented program state. Concretely, we instantiate NCs as video models that roll out screen frames from instructions, pixels, and user actions (when available) in CLI and GUI settings. We show that NCs can acquire elementary interface primitives, especially I/O alignment and short-horizon control, while routine reuse, controlled updates, and symbolic stability remain challenging. We outline a roadmap toward CNCs, to establish a new computing paradigm beyond today's agents and conventional computers.

2 KAUST (

Neural Computers unify computation, memory, and I/O within a learned runtime state, representing a proposed machine form beyond modular computer stacks.

  • Neural Computers unify computation, memory, and I/O within a learned runtime state.
  • The Completely Neural Computer is envisioned as a mature, general-purpose realization of this machine form.
  • The runtime state functions as neural compute, working memory, and the interface substrate.
  • The proposed architecture shifts from a modular hardware stack to a neural latent stack.

1 Introduction

The paper defines Neural Computers, implements video-based CLI and GUI prototypes, and evaluates whether they can learn elementary runtime primitives from interface traces. The prototypes show I/O alignment and short-horizon control, while reuse, symbolic stability, and governance remain challenging.

  • A Neural Computer unifies computation, memory, and I/O in a learned runtime state.
  • The paper instantiates NCs as video models for command-line and graphical desktop interaction.
  • NCCLIGen learns basic command-line workflows and often maintains terminal alignment, but symbolic stability remains limited.
  • NCGUIWorld evaluates data quality, cursor supervision, action injection, and action encoding using fidelity, responsiveness, and cursor-accuracy measurements.
  • Current NCs learn elementary runtime primitives, especially I/O alignment and short-horizon control.
  • The CNC target requires stable execution, explicit reprogramming, durable capability reuse, and runtime governance.
  • The paper contributes video-based CLI and GUI prototypes, synchronized text-action-frame data, ablation studies, and a CNC roadmap.

2 Preliminaries

The paper frames NCs as learned latent-state systems that replace separated computer subsystems with an update-and-render loop. Video-based prototypes use observations and action conditions to predict future interface frames.

  • Conventional computers separate computation, memory, and I/O across stored-program and operating-system/application stacks.
  • An NC folds these roles into one learned latent state through an update-and-render loop.
  • The prototypes use screen frames as observations and actions as time-indexed conditioning inputs.
  • The NC updates runtime state from the current observation and conditioning input, then predicts or samples the next observation.
  • The persistent latent state carries executable context, while the update function integrates inputs and the decoder renders the next frame.
  • The framework draws on neural execution, differentiable memory, and neural-programming architectures while adding interface-specific conditioning and action modules.

3 Implementation of Neural Computers

The implementation evaluates Neural Computers as interface video models across terminal and GUI settings, emphasizing rendering fidelity, conditioning, action control, and symbolic probes. Results show strong short-horizon interface behavior, while symbolic stability and reliable learning from noisy data remain limited.

  • Terminal rendering: At practical 13 px font sizes, the NC preserves readable terminal rendering, while extreme small fonts cause localized blurring despite strong global metrics.The 13 px CLIGen font remains visually sharp, whereas 6 px text exhibits noticeable blurring.
  • Training behavior: PSNR/SSIM plateau around 25k steps on CLIGen (Clean), with later optimization yielding little or slightly unstable perceptual improvement.Remaining errors are associated with rendering glitches and rapid screen changes that disrupt temporal alignment.
  • Terminal rendering: Character accuracy rises from 0.03 at initialization to 0.54 at 60k steps, while exact-line accuracy reaches 0.31, demonstrating measurable terminal interface fidelity.Most gains occur within the first 40k steps, followed by smaller refinements.
  • Symbolic probes: Reprompting raises CLIGen (Clean) arithmetic accuracy from 4% to 83% without changing the NC backbone or adding reinforcement learning, primarily indicating steerability rather than native arithmetic.The authors interpret the gain as faithful rendering of conditioned symbolic content, not clean evidence of internal computation.
  • GUI control: GUI results show that explicit visual supervision reaches 98.7% cursor accuracy, while aligned goal-directed data outperforms larger random exploration and deeper action fusion improves responsiveness.API-like meta-actions also outperform raw event-stream encoding under the same injection mode.

4 Position: Toward Completely Neural Computers

Neural Computers unify computation, memory, and I/O in a learned runtime state, pointing toward Completely Neural Computers with stable execution, explicit reprogramming, and reusable capabilities. Current prototypes demonstrate early interface primitives, but reliable programmability, symbolic stability, long-horizon behavior, and runtime governance remain unresolved.

  • Current prototypes show measurable interface fidelity, especially I/O alignment and short-horizon control, but stable reuse and general-purpose execution remain out of reach.CLI and GUI experiments provide evidence for early runtime primitives rather than proximity to CNC-level maturity.
  • NCs use learned programming-language semantics in which natural-language instructions, examples, and constraints can function as executable specifications.This shifts development toward curating, specifying, and verifying inputs rather than relying only on explicitly authored code.
  • A CNC requires Turing completeness, universal programmability, behavior consistency unless explicitly reprogrammed, and NC-specific architectural and programming-language advantages.These conditions define the mature form of an NC.
  • Behavior consistency requires separating ordinary execution from explicit updates so installed capabilities are not silently modified by routine inputs.This operational requirement motivates explicit programming interfaces and governance mechanisms.
  • Unlike AI agents that rely on external execution substrates, NCs aim to internalize executable state, computation, memory, and interfaces within one learned runtime.Agents retain a separation between the learned model and the runtime that stores state, applies updates, and enforces contracts.
  • Video models provide a pragmatic prototype substrate, but inconsistent symbolic and algorithmic reasoning suggests CNC-level reliability needs ingredients beyond scaling current generators.The paper presents machine-native architectures with discrete operations, compositional structures, and verifiable computation as a conjectured direction.

5 Conclusion

Neural Computers suggest a machine form in which one latent runtime state acts as the computer itself. Early progress is concentrated in I/O alignment and short-horizon control, while reuse, consistency, symbolic reliability, and governance remain open prerequisites for usable runtimes.

  • Neural Computers use a single latent runtime state to drive pixels, text, and actions while subsuming functions handled by operating systems and interfaces.
  • The main demonstrated primitives are I/O alignment and short-horizon control; stable reuse, symbolic reliability, and runtime governance remain unresolved.
  • Progress toward CNCs depends on making reuse, consistency, and governance sustained and testable rather than merely exhibiting prototype behavior.

A Explorations: Alternative Data Sources and Online Interaction

Alternative data sources could expand interface experience, but web-scale crawling introduces governance, quality, and systems burdens. Online interaction offers a scalable, auditable template, yet current infrastructure bottlenecks led the study to prioritize controlled video data.

  • Web-scale screen videos combine uncontrolled content, heterogeneous interfaces, and inconsistent capture quality, making reliable localization, OCR, and temporal alignment difficult.
  • The explored pipelines remain future-facing because rights, privacy, filtering, alignment, communication, debugging, and safety requirements constrain practical scaling.

A.1 Web video extraction

Web video extraction was investigated as a potential data source, while agentic online collection was prototyped as a sandboxed, auditable alternative. Both approaches exposed substantial data-governance, alignment, infrastructure, and safety costs, so the main experiments used controlled video data.

  • Web video extraction: Web crawling used OCR and layout detectors to locate terminal regions, estimate text and timestamps, and extract related clips.
  • Web video extraction: Web videos require heavy filtering and sanitization because uncontrolled content, cross-platform domain shift, and capture artifacts degrade OCR and temporal alignment.
  • Web video extraction: Privacy, copyright, rights-cleared sourcing, redaction, and paid multimodal filtering made web crawling’s cost–quality trade-off unfavorable for the study.
  • Online environment interaction: The online pipeline used a sandboxed environment plane and a control plane to collect synchronized renders, structured state, and action traces for storage and model updates.
  • Online environment interaction: The environment exposed deterministic step/reset interactions with multimodal observations and structured actions, enabling auditable rollouts and decoupled collection and updating.
  • Online environment interaction: Closed-loop interaction could induce a curriculum, expose rare failures, target data collection, and scale experience beyond static demonstrations.
  • Online environment interaction: Cross-cluster communication latency and asynchronous debugging complexity were the dominant end-to-end bottlenecks, so controlled video data was prioritized.

B Datasets: Collection and Evaluation Protocols

The datasets combine public recordings, scripted replays, and controlled GUI captures into synchronized, privacy-filtered episodes. Structured metadata supports alignment and evaluation but is withheld from the video models as state input.

  • Collection: Data collection uses public terminal recordings, scripted terminal replays, and a controlled desktop-capture rig across CLIGen and GUIWorld.The pipeline emphasizes rights-respecting acquisition, privacy filtering, and temporal alignment of frames, actions, and text when available.
  • Collection: CLIGen (General) replays public asciinema archives, while CLIGen (Clean) executes deterministic VHS scripts in isolated environments.The sources preserve terminal appearance or provide controlled exercises such as package installs, REPLs, and log filters.
  • Collection: GUIWorld pairs RGB footage with low-latency pointer and keyboard logs, with optional accessibility cues recorded for analysis but excluded from model inputs.The capture rig supplies the visual and interaction streams used for GUI modeling.
  • Preprocessing: All modalities share a common clock; events are aligned to nearby frames, drift is corrected, and misaligned, privacy-sensitive, or repeated-frame clips are removed.Privacy filters redact sensitive terminal strings and GUI regions likely to contain private content.
  • Packaging: Runs are windowed into fixed-length overlapping episodes containing frames, buffers or GUI metadata, serialized actions, source tools, and environment metadata.Structured fields support alignment and evaluation, while dataloaders reconstruct batches without exposing those fields as video-model state inputs.
  • Packaging: Each CLIGen (General) clip includes three aligned description tiers and metadata sufficient to rebuild terminal geometry, environment, and source information.These fields make each clip self-contained and provide prompts at different levels of detail.

B.2 OCR evaluation protocol (CLIGen Clean)

The CLIGen Clean protocol evaluates generated terminal videos against reference renderings in OCR-derived text space. Character fidelity and position-sensitive line reconstruction are averaged over sampled frames and then videos.

  • Sampling: OCR metrics compare each generated video with its paired reference using up to K=5 sampled frames.Only clip IDs with both generated and reference videos are retained, and the frame count is truncated to the shorter video.
  • OCR preprocessing: Each sampled frame is converted to normalized OCR lines while preserving case and punctuation and removing empty lines.Tesseract output is whitespace-normalized so commands, paths, and symbols remain visible.
  • Character accuracy: Character accuracy measures normalized Levenshtein similarity between concatenated ground-truth and generated frame text.The denominator uses max(|s|, 1), and empty-string cases are assigned explicit scores.
  • Exact-line accuracy: Exact-line accuracy scores only ground-truth lines reproduced at the same position in the generated frame.Empty-list cases receive explicit values, and misplaced lines are not counted as correct.
  • Aggregation: Frame scores are averaged across sampled pairs to produce per-video scores, then averaged across videos for aggregate reporting.Together, the metrics stress fine-grained text fidelity and line-ordered terminal-state reconstruction.

B.3 Evaluation metrics and protocol (GUIWorld)

GUIWorld reports global video fidelity and action-driven fidelity after interaction events. The protocol standardizes paired-video preprocessing and evaluates post-action windows using logged timestamps.

  • Global metrics: Global metrics FVDall, SSIMall, and LPIPSall compare paired generated and ground-truth videos after standardized decoding, subsampling, and resizing.The protocol uses RGB frames and fixed preprocessing before computing the three global measures.
  • Action-driven metrics: Action-driven metrics SSIM+15, LPIPS+15, and FVD+15 evaluate interface fidelity in windows following interaction events.These metrics focus on visual behavior after actions rather than over the entire rollout.
  • Global metrics: The global protocol decodes videos at fps=3, size=256, and max_seconds=5 by default before averaging SSIM and LPIPS over frames.FVD uses an r3d18 prelogit embedding space with fixed-length clips.
  • Action-driven metrics: For each logged action, the protocol skips the action frame and evaluates the next k=15 valid frames, deduplicating indices chronologically.Clips without logged actions or valid post-action windows are excluded from +15 metrics.

D.1 Action schema

GUIWorld represents interaction as structured, temporally aligned action features and injects them into video models through several conditioning modes. The design uses short lagged histories and attention restrictions to connect frames with plausible preceding operations.

  • Action schema: NCGUIWorld encodes cursor movements and key presses as a structured action stream with absolute coordinates, button transitions, scroll deltas, keyboard events, and drag or hold state.Typed characters and shortcut chords are represented separately, while extended interactions become labeled segments.
  • Action schema: The meta-action encoder uses S=2 action slots per frame, padding empty slots with type 0 and attaching parameters to each action type.The schema includes typed actions such as mouse clicks and keyboard input.
  • Conditioning: Raw and meta-action encoders produce per-frame features that are temporally windowed and aligned before injection as external, contextual, internal, or residual conditioning.These modes respectively alter VAE latents, mix tokens, add transformer cross-attention, or apply hidden-state deltas.
  • Temporal alignment: Latent-aligned action embeddings summarize short, lagged histories of cursor and keyboard events around each anchor frame.The window accounts for GUI response delay and is reused across injection modes.
  • Temporal attention: Contextual attention restricts frame-to-frame connections locally, lets frames attend only to recent actions, and prevents actions from attending to earlier frames.The structured mask allows action tokens to see later frames only after their effects could occur, while action-to-action attention is diagonal.
  • Design insights: Short lagged windows and local attention bands address bursty, high-dimensional action streams by linking each frame to operations that could plausibly have produced its pixels.The paper also treats cursor rendering as an explicit first-class conditioning signal rather than relying only on global diffusion loss.
  • Cursor representation: Cursor trajectories are aligned to sampled video frames through the same letterbox mapping as RGB data and stored as normalized time-stacked trajectories.The resulting mouse_trajectories tensor is reused for rendering and action encoding.

D.5 Training signals and encoder design

The GUIWorld design combines cursor-reference supervision, continuous mouse encoding, and temporally aligned action features to improve interface control. These signals constrain cursor placement while preserving learned dynamics for the rest of the frame.

  • Reference supervision: Cursor references anchor the initial desktop frame and supervise only the cursor region at later timesteps.At t=0 the full frame is used; afterward, cursor foregrounds and masks provide localized reference conditions.
  • Action encoding: The Fourier mouse encoder converts normalized trajectories into per-frame embeddings and aggregates them with lag-aware, stride-aligned windows.Coordinates are clamped, rescaled, projected into Fourier features, and mapped through an MLP before aggregation.
  • Training signals: 98.7% cursor accuracy is reached when explicit cursor rendering and masked patch loss supplement position-only objectives.Position-only losses produce low hit rates and jittery arrows despite plausible videos.
  • Encoder and injection design: GUIWorld compares raw-action and meta-action encoders across external, contextual, internal, and residual conditioning schemes.The comparison emphasizes typed, API-like action schemas with explicit lag handling and evaluates multiple injection modes.
  • Temporal alignment: Temporal contrastive alignment matches frame and action features at lag-adjusted timesteps, optionally including mouse features.An InfoNCE-style objective brings same-timestep pairs together, while a future head trains actions to anticipate later visual consequences.

E.1 CLIGen Visualizations

The CLIGen visualization appendix presents both broad real-world terminal dynamics and controlled REPL probes. Together, the samples expose action-to-pixel causality, terminal rendering behavior, and the distinction between computation and answer-conditioned rendering.

  • CLIGen (General): CLIGen (General) samples cover ANSI repainting, syntax-highlighted editing, shell outputs, scrolling jobs, full-screen TUIs, and progress dashboards.The examples emphasize changing terminal geometry, colors, cursor edits, and dynamic status displays.
  • CLIGen (Clean) REPL: CLIGen (Clean) REPL samples use scripted, well-paced traces with explicit actions such as Sleep, Type, Enter, arrow keys, and Hide.This format makes action-to-pixel causality easy to inspect.
  • CLIGen (Clean) math: CLIGen (Clean) math samples compare rollouts and reprompting cases to separate native computation from answer-conditioned rendering.The visualization set includes both standard math prompts and reprompting probes.

E.2 GUIWorld Visualizations

The GUIWorld visualization appendix organizes rollout examples around trace-conditioned generations and detailed CLI sample fragments. The figures emphasize state drift, low-level deviations, numeric and UI fidelity, and precise interaction semantics.

  • GUIWorld rollout comparisons: GUIWorld CUA pages contrast Ground Truth trajectories with generations conditioned on the first frame and action sequence.Natural-language rationale and structured actions such as clicks, drags, and typing are overlaid when available.
  • Failure and stress cases: Figures 31–33 emphasize compounding low-level deviations, while Figures 34–36 focus on numeric/UI fidelity and interaction semantics.Figures 37–39 add stress cases where correctness depends on precise field edits and page state.
Loading 2604.06425v2…