PaperlayerView exact revision on arXiv

Source-linked AI summary

pico-type: A 1.5M-Parameter Byte-Level Multi-Head Content Classifier

Gautam Kishore

arXiv:2608.14658v1cs.LGcs.AIcs.CLcs.IR

TL;DR

Real-time content classification needs a compact alternative to heavyweight models. pico-type predicts seven content properties directly from raw bytes in one pass, achieving 60.3% code-language accuracy and 98.2% text-language accuracy on reported benchmarks while supporting on-device deployment.

  • Problem

    Existing content classifiers do not adequately combine broad multi-property coverage with the millisecond-scale efficiency required for continuous on-device use.

  • Method

    pico-type uses a shared byte-level trunk with seven Matryoshka-style heads to jointly classify content properties from raw UTF-8 bytes in one forward pass.

  • Results

    60.3% code language accuracy on The Heap and 98.2% text language accuracy on Wikipedia demonstrate strong performance across the reported content properties.

  • Takeaways & Limitations

    The approximately 1.5M-parameter model is suitable for real-time on-device deployment without a tokenizer, pretrained embeddings, or GPU hardware.

  • Takeaways & Limitations

    Seven code languages remain below 50% accuracy because of insufficient training samples, and Malay text detection reaches 70% accuracy.

Abstract

from arXiv · show

We introduce pico-type, a byte-level multi-head content classifier with approximately 1.5 million parameters that simultaneously predicts seven content properties from raw UTF-8 bytes in a single forward pass. Operating directly at the byte level -- no tokenizer, no subword vocabulary, no pretrained embeddings -- pico-type classifies coarse type (12 classes), modality (8), subtype (24), code language (62), text language (30), file MIME type (90), and risk flags (6-label multi-label: API keys, JWTs, passwords, emails, phone numbers, SSH keys). The architecture combines a learned byte embedding, three convolutional blocks with growing receptive fields, two bidirectional attention layers with rotary position encodings, and a statistical pooling layer feeding seven Matryoshka-style classification heads. Four tiered variants (tiny/small/base/pro) share the same trunk with sliced representations from 16 to 576 dimensions, yielding ONNX exports under 210 KB and CPU inference under 10 ms. Trained on a mixture of synthetic templates and real-world data (8709 GitHub code samples, 5000 Wikipedia articles), pico-type achieves 60.3 percent code language accuracy on The Heap benchmark (24 languages) and 98.2 percent text language accuracy on Wikipedia (30 languages) -- improvements of +57 and +79 percentage points respectively over the synthetic-only baseline. Format-based heads (coarse, modality, subtype, file_mime, risk) maintain 100 percent accuracy on synthetic benchmarks. The model, code, and pretrained weights are released under Apache 2.0.

1 Introduction

Content classification supports clipboard, file, security, and developer workflows, but existing rule-based and neural approaches trade generality for deployment cost. pico-type addresses this gap with a tiny, on-device byte-level classifier that jointly predicts multiple content properties in one forward pass.

  • Motivation: Content classification underpins clipboard managers, file browsers, security scanners, and developer tools that identify formats, secrets, and language.These applications need to distinguish code, configuration, prose, and other content types.
  • Limitations: Smallest distilled LLMs exceed 100 million parameters, require gigabytes of storage, and incur CPU inference latencies of hundreds of milliseconds or more.These costs make continuous, millisecond-scale on-device classification impractical.
  • Approach: pico-type fills the gap with a tiny multi-head classifier that predicts multiple content properties from raw bytes in one forward pass without GPU, tokenizer, or network dependence.The design targets entirely on-device operation for real-time use cases.
  • Contributions: Raw UTF-8 bytes enable tokenizer-free support for all languages and binary formats, while seven Matryoshka-style heads jointly predict content type, modality, subtype, languages, MIME, and risk flags.The shared trunk produces coarse type, modality, subtype, code language, text language, file MIME, and risk outputs.

2 Related Work

Related work situates pico-type at the intersection of tokenizer-free byte modeling, compact neural classification, Matryoshka representations, and multi-head learning. It distinguishes pico-type from existing content tools by combining type, language, format, and security-risk prediction in one forward pass.

  • Byte-level modeling: Byte-level models motivate tokenizer-free representations and lightweight Conv1D alternatives to full Transformers for classification.ByT5 matched subword BART on discriminative tasks, while earlier character-embedding convolutional networks were effective for sentence classification.
  • Compact classification models: 1.5M parameters make pico-type 10–50× smaller than DistilBERT, ALBERT, and MobileBERT, while SqueezeBERT remains 3.8M and uses pretrained subword embeddings.The cited compact models range from 12M to 66M parameters, and SqueezeBERT’s pretrained subword embeddings limit language coverage.
  • Matryoshka representations: Four sliced representations from a shared 576-dimensional pooled vector apply Matryoshka representation learning to tier-specific multi-head classification.The design enables one model to serve deployment scenarios with varying capacity constraints through multiple representation granularities.
  • Multi-task and multi-head classification: Pico-type adopts a shared-trunk, specialized-head philosophy like Polymorph but trains all heads jointly from scratch without distillation.Polymorph introduced per-head distillation for efficient multi-label classifiers; pico-type instead uses synthetic data for joint training.
  • Content classification tools: Pico-type uniquely combines type, language, format, and security-risk prediction in a single forward pass, beyond tools focused on limited patterns, file signatures, or one property.ClipGate and PasteBot use rule-based matching, file identification tools use magic-byte databases, and guesslang detects only code language.

3 Model Architecture

pico-type processes raw UTF-8 bytes through learned embedding, multi-scale convolutions, bidirectional attention, statistical pooling, and seven gated classification heads. Tier-specific linears reuse a shared pooled representation to support four inference variants.

  • Architecture: The model processes raw UTF-8 byte sequences through byte embedding, convolutional extraction, bidirectional attention, statistical pooling, and multi-head classification.These are five learned stages in the overall architecture.
  • Byte representation: Each byte maps to a 96-dimensional learned vector, with no tokenization, stemming, subword processing, or pretrained initialization.The embedding table has shape E ∈R256×96 and is learned from scratch.
  • Convolutional feature extraction: Three Conv1D blocks use kernel sizes 3, 5, and 7 to capture multi-scale byte patterns with increasing receptive fields.The design captures character n-gram patterns without the quadratic complexity of full self-attention over long sequences.
  • Bidirectional attention: Two bidirectional self-attention blocks use pre-norm layer normalization, fused QKV projections, RoPE with θ = 500,000, and 4 heads of dimension 48.Each block also includes a two-layer GELU MLP with 4× hidden-dimension expansion.
  • Pooling and classification heads: The pooled vector concatenates mean, maximum, and standard deviation statistics, while seven heads use tier-specific slices of the shared 576-dimensional representation.Tiers use dimensions 16, 64, 192, and 576; heads are gated by coarse type, and risk detection is multi-label.

4 Training

Training combines balanced synthetic coverage with real-world code and text data, using a 50:50 mixed dataset to improve language detection while preserving the other heads. Samples are raw UTF-8 bytes up to 1,024 bytes, trained with weighted multi-task losses and a two-phase schedule.

  • Real-world data: 8,709 code samples span 62 programming languages, while 5,000 Wikipedia articles cover all 30 target text languages.Real code covers 52 languages; 10 rare languages are supplemented with synthetic templates.
  • Data representation: Each sample uses raw UTF-8 bytes truncated to 1,024 bytes, with labels derived from file extensions or article language tags.Evaluation uses separate sources: The Heap for code and a held-out Wikipedia split for text.
  • Mixed training: 50:50 real-to-synthetic batching preserves format-based heads while exposing language heads to real-world distributions.Each batch contains 50% real code or text samples and 50% synthetic samples covering all seven heads.
  • Synthetic data: Synthetic generation provides balanced coverage across all 12 coarse buckets and seven heads using language-specific templates, word lists, magic bytes, and secret patterns.Code templates cover all 62 languages; binary generators include 22 formats, and secret generators produce patterns for six risk categories.
  • Optimization: Per-head loss weights are 3.0 for coarse, 2.0 for modality, 1.5 for code_lang and text_lang, and 1.0 for subtype, file_mime, and risk.Single-label heads use gated cross-entropy, while risk uses binary cross-entropy.
  • Optimization: 6,700 training steps comprise 1,700 synthetic-only pretraining steps followed by 5,000 mixed real-plus-synthetic steps.Training uses AdamW, 100-step linear warmup, cosine decay, gradient clipping at norm 1.0, float32 precision, and batch size 16.

5 Experiments

Experiments show strong real-world language detection alongside near-perfect synthetic performance on structurally deterministic format and risk tasks. The shared trunk also yields comparable latency across model tiers.

  • Real-world language detection: 60.3% overall code-language accuracy on The Heap improved over the synthetic-only v0.1 baseline of 3.0%.The benchmark covers 1,200 files across 24 languages; 16 languages reach at least 76% accuracy.
  • Real-world language detection: 98.2% overall text-language accuracy on Wikipedia improved over the synthetic-only v0.1 baseline of 19.0%.The evaluation uses 1,500 articles across all 30 supported languages.
  • Synthetic format and risk benchmarks: 100% accuracy was achieved by coarse type, modality, and file MIME classification on 500 held-out synthetic samples.These heads exploit deterministic byte-level signatures such as magic numbers, file headers, and syntactic markers.
  • Synthetic format and risk benchmarks: 93.8% subtype accuracy and 100.0% mean average precision for the risk head were achieved across 24 format types and 6 label types, respectively.Subtype errors were limited to structurally similar pairs such as YAML/TOML and Markdown/reST.
  • Inference performance: Approximately 18 ms warm latency was measured for every tier at batch size 1 and sequence length 256 bytes.The shared trunk dominates runtime, so tier dimension has little effect on speed.

6 Deployment

pico-type is packaged for practical, dependency-light deployment through ONNX Runtime and multiple user-facing interfaces. Its ecosystem includes Python, web, MCP, Rust, and offline browser-based access paths.

  • Deployment: ONNX Runtime enables inference without a PyTorch dependency, supporting practical deployment across multiple platforms.Exported models are used directly at inference time.
  • Deployment: The Python CLI accepts stdin, file paths, macOS clipboard contents, or direct text, returning JSON with all seven classification results.It automatically locates the packaged ONNX model or uses a configured path.
  • Deployment: A Gradio web interface provides seven tabbed result panels, tier selection, and examples spanning all content types.The interface is hosted as a Hugging Face Space.
  • Deployment: An MCP server exposes classify and classify_file tools over stdio for Claude Desktop, Cursor, and compatible VSCode extensions.The tools use the Model Context Protocol.
  • Deployment: A Rust CLI uses the ort crate to provide identical functionality without Python, while a scaffolded Manifest V3 extension supports fully offline clipboard and text-selection classification.The extension uses a local HTTP inference server.

7 Limitations

pico-type is limited by sparse data for some languages, a 1,024-byte input cap, and the need for retraining to add classes or languages. Apple Silicon training also faces an MPS-specific memory constraint that does not affect inference.

  • Data coverage: 0% accuracy is reported for SQL, Julia, and Elixir, while several other code languages remain weak because of insufficient training samples.Perl reaches 50%, Haskell 22%, Scala 6%, JavaScript 2%, and Clojure 2%; Malay text detection reaches 70%.
  • Sequence length: 1,024 bytes is the maximum sequence length, so longer inputs are truncated and may lose distinguishing features.The limit prioritizes inference speed and memory efficiency.
  • Model updates: Retraining is required to add new classes or languages because the model is frozen after training.Future work will explore LoRA-based head adaptation for user-customizable classification.
  • MPS training constraints: Approximately 2,000 contiguous steps is the Apple Silicon training limit caused by MPS graph cache memory growth.This framework limitation does not affect inference on any platform.

8 Conclusion and Future Work

pico-type is a 1.5M-parameter byte-level classifier covering seven content properties without tokenizers, pretrained embeddings, or GPUs. Future work targets broader rare-language data, customizable heads, and teacher distillation, while the code and weights are released under Apache 2.0.

  • Conclusion: 1.5M parameters define pico-type as a byte-level multi-head classifier for seven content properties.The properties are coarse type, modality, subtype, code language, text language, file MIME, and risk flags.
  • Conclusion: No tokenizer, pretrained embeddings, or GPU hardware are required for inference.The model is intended for on-device deployment in CLIs, browser extensions, MCP servers, and Rust binaries.
  • Conclusion: Approximately 9 MB of single-file FP32 ONNX storage and approximately 18 ms CPU inference support on-device deployment.These figures are reported for the deployed model configuration.
  • Future Work: Future work includes expanding data for eight underperforming code languages and one text language, adding per-head LoRA adapters, and distilling teacher models.The stated targets are above 90% for all code languages and above 70% for text languages currently below that threshold.
  • Release: All code, model weights, and deployment configurations are released under Apache 2.0.The project provides GitHub, HuggingFace, PyPI, and interactive-demo access points.

A Per-Class Performance Details

The subtype head’s per-class performance is detailed through precision, recall, and F1, revealing variation despite near-perfect overall accuracy. Classes with fewer than five support samples are excluded from the table.

  • Subtype head: Table 8 reports per-class precision, recall, and F1 for the subtype head.The subtype head shows the most interesting performance variation among heads with near-perfect accuracy.
  • Subtype head: The subtype head exhibits notable per-class performance variation despite near-perfect accuracy across heads.
  • Subtype head: Classes with fewer than 5 support samples are omitted from the per-class subtype metrics.
Loading 2608.14658v1…