PaperlayerView exact revision on arXiv

Source-linked AI summary

Falcon Perception

Aviraj Bevli, Sofian Chaybouti, Yasser Dahou, Hakim Hacid, Ngoc Dung Huynh, Phuc H. Le Khac, Sanath Narayan, Wamiq Reyaz Para, Ankit Singh

arXiv:2603.27365v1cs.CV

TL;DR

Modular encoder–decoder systems raise whether dense perception requires separate visual encoding and task decoding. Falcon Perception tests a unified early-fusion Transformer, compares favorably with state-of-the-art systems, and extends the recipe to compact OCR.

  • Problem

    Dense perception commonly separates visual feature extraction from task decoding, leaving whether this encoder–decoder split is necessary as an open question.

  • Method

    Falcon Perception uses a single dense Transformer that processes image patches and text tokens in a shared parameter space with hybrid bidirectional-image and causal-prediction attention.

  • Results

    Falcon Perception compares favorably with state-of-the-art perception systems, while FalconOCR achieves 80.3% average accuracy on olmOCR with 300M parameters.

  • Takeaways & Limitations

    A single early-fusion Transformer with an appropriate interface is presented as viable for dense perception, particularly for OCR, spatial constraints, relations, and long-context outputs.

  • Takeaways & Limitations

    The single-stack autoregressive design requires more expensive training and can decode more slowly than fully parallel DETR-like models.

Abstract

from arXiv · show

Perception-centric systems are typically implemented with a modular encoder-decoder pipeline: a vision backbone for feature extraction and a separate decoder (or late-fusion module) for task prediction. This raises a central question: is this architectural separation essential or can a single early-fusion stack do both perception and task modeling at scale? We introduce Falcon Perception, a unified dense Transformer that processes image patches and text tokens in a shared parameter space from the first layer, using a hybrid attention pattern (bidirectional among image tokens, causal for prediction tokens) to combine global visual context with autoregressive, variable-length instance generation. To keep dense outputs practical, Falcon Perception retains a lightweight token interface and decodes continuous spatial outputs with specialized heads, enabling parallel high-resolution mask prediction. Our design promotes simplicity: we keep a single scalable backbone and shift complexity toward data and training signals, adding only small heads where outputs are continuous and dense. On SA-Co, Falcon Perception improves mask quality to 68.0 Macro-F$_1$ compared to 62.3 of SAM3. We also introduce PBench, a benchmark targeting compositional prompts (OCR, spatial constraints, relations) and dense long-context regimes, where the model shows better gains. Finally, we extend the same early-fusion recipe to Falcon OCR: a compact 300M-parameter model which attains 80.3% on olmOCR and 88.64 on OmniDocBench.

1. Introduction

Falcon Perception asks whether dense grounding requires an encoder–decoder split and proposes a unified early-fusion Transformer instead. It combines shared visual-language processing with lightweight interfaces, PBench, and an OCR extension.

  • Motivation: Modular encoder–decoder systems separate visual feature extraction from task decoding, adding task-specific mechanisms and limiting vision–text interaction.The cited motivation identifies modality fusion, query matching, and post-processing as sources of complexity.
  • Contribution: Falcon Perception is a unified dense Transformer for promptable segmentation and language-grounded perception.Given an image and prompt, it determines concept presence, identifies referred instances, and predicts pixel masks.
  • Architecture: A hybrid attention mask gives image tokens bidirectional global context while text and task tokens attend causally within the same weights.This replaces the encoder–decoder split with encoder-like visual processing and decoder-like language generation.
  • Evaluation: PBench targets compositional prompts and dense long-context evaluation beyond saturated referring-expression benchmarks.The benchmark focuses on capabilities such as OCR, spatial constraints, and relations.
  • Interface: Falcon Perception uses a Chain-of-Perception sequence that predicts coordinates, size, and segmentation for each instance.The ordering resolves position and size before mask generation, supporting fast dense inference.
  • Extensions: Falcon-OCR extends the early-fusion recipe to text-heavy vision tasks with a compact 300M-parameter model.The paper positions this extension alongside favorable segmentation and OCR results.

2. Architecture

Falcon Perception uses one early-fusion Transformer for image patches, text, and autoregressive task tokens, with specialized spatial and segmentation heads. Its interface combines structured instance generation with high-resolution mask decoding and supports compositional prompts.

  • Unified backbone: Image patches and text embeddings are concatenated into one sequence processed by a shared dense Transformer.The architecture uses the same weights for visual and language representations.
  • Unified output interface: The model serializes each object as coordinates, size, and mask, using the Chain-of-Perception order c_k → s_k → m_k.This coarse-to-fine sequence resolves spatial ambiguity before pixel-level prediction.
  • Attention: Hybrid masking lets image tokens attend bidirectionally to image tokens while text and task tokens attend causally to the visual prefix.This makes one stack function as both a bidirectional vision encoder and an autoregressive language decoder.
  • Specialized heads: Fourier features encode continuous coordinates in a higher-dimensional frequency space before projection into the Transformer.The design replaces coarse coordinate tokenization and supports finer spatial detail.
  • Specialized heads: High-resolution features are produced through upsampling and cross-attention, then combined with the segmentation-token state by a dot product to form masks.The high-resolution image guides boundaries while backbone visual features provide semantic information.
  • Behavior: Falcon Perception remains stable as prompts become more compositional, while SAM 3 can fail on OCR-driven and higher-level constraints.The comparison concerns queries involving OCR, spatial constraints, and relations.

3. PBench: Perception Benchmark

PBench separates compositional perception capabilities and stresses long-context generation in crowded scenes. It reports per-level profiles and performance as instance counts increase, rather than relying only on saturated aggregate referring benchmarks.

  • Motivation: Existing referring-expression benchmarks exceed 80% accuracy and conflate distinct challenges such as prompt complexity and spatial ambiguity.These limitations make model ranking and capability diagnosis difficult.
  • Complexity Levels: PBench assigns samples to Levels 0–4 that isolate recognition, attribute binding, OCR disambiguation, spatial constraints, and relations.Prompts avoid cross-level cues so each level targets one dominant capability.
  • Complexity Levels: Per-level scores provide a capability profile that reveals where a model fails, rather than reducing performance to one scalar.The benchmark reports an overall average alongside level-specific results.
  • Prompt progression: Figure 3 keeps the image fixed while progressively adding specificity from object class to attributes, OCR, spatial layout, and fine-grained relations.The same scene therefore exposes capability changes attributable to the prompt.
  • Dense regime: Dense evaluation varies the number of instances per prompt, reaching approximately K ≈ 600 in long-context settings.This tests long-sequence generation, object-not-found behavior, duplication, and drift as instance count grows.

4. Training details

Falcon Perception is trained through staged distillation, large-scale perception data, and design choices for dense autoregressive prediction. The recipe combines curated multimodal supervision with optimization and regularization strategies evaluated through targeted ablations.

  • Training initialization: Multi-teacher distillation initializes the unified backbone using DINOv3 and SigLIP2 to combine local visual and language-aligned features.DINOv3 supports segmentation features, while SigLIP2 contributes language-aligned features for open-vocabulary understanding.
  • Data curation: The perception dataset contains 54M images, 195M positive expressions, 488M negative expressions, and 570M masks.Data are assembled through clustering, VLM listing, negative mining, ensemble consensus, and human verification.
  • Sequence format: The training sequence uses explicit <present> and <absent> tokens before localization outputs, requiring an existence decision before coordinates, sizes, and masks.The serialized format supports multiple expressions and variable numbers of instances in a unified autoregressive sequence.
  • Optimization: Muon lowers training losses for language, coordinate, and size heads versus AdamW, with improved MacroF1 on PBench and SaCo.The optimizer comparison targets learning-rate balance between the pretrained backbone and initialized perception modules.
  • Feature regularization: Gram loss improves both benchmarks by preserving the structural integrity of visual features relative to training without this regularization.The loss matches the student patch-feature correlation matrix to that of a frozen teacher model.
  • Training recipe: Approximately 685 Gigatokens train the perception task in stages that first build scene-level context and then specialize the model for independent query inference.Training maintains a 1:1 positive-to-negative ratio; Stage 1 uses 450 GT for in-context listing and Stage 2 uses 225 GT for task alignment.

5. Results

Falcon Perception performs strongly on mask quality and compositional perception, with gains increasing on OCR, spatial, relational, and dense tasks. Performance also depends strongly on spatial resolution, while presence calibration remains a key limitation of the autoregressive formulation.

  • SA-Co: 68.0 Macro F1 versus 62.3 for SAM3 shows Falcon Perception’s stronger average mask quality on SA-Co.It improves F1 on most splits, including Food&Drink, Sports, and Attributes.
  • Limitations: Presence calibration remains weaker than SAM3, with average MCC 0.64 versus 0.82 because the autoregressive model lacks an inherent empty-class mechanism.The reported cgF1 collapse without explicit negative modeling motivates targeted negative sampling and future reinforcement learning.
  • PBench: +21.9 on PBench Level 3, alongside +9.2 on Level 1 and +13.4 on Level 2, shows larger gains on compositional capabilities.PBench gains become substantial as prompts require attributes, OCR, and spatial understanding.
  • Resolution: 61.0% Dense micro-F1 at 1024^2 versus 3.9% at 448^2 marks a resolution-driven phase transition for crowded-scene perception.Presence detection remains stronger than mask drawing at low resolution, with IL_MCC 0.34 versus pmF1 23.6.
  • Resolution: +1.6 pmF1 from Fixed Resizing, reaching 63.9 versus 62.3 for Adaptive Resizing on SA-Co, confirms the benefit of upscaling inputs.The gain reaches +3.7 pmF1 on the Crowded split, where small and densely clustered objects must be resolved.
  • Inference: Sampling improves SA-Co cgF1 from 34.7 at the deterministic baseline to 54.3 at Pass@8, a +19.6-point absolute increase.The largest reported gain is on Wiki-Common, where cgF1 rises from 19.3 to 45.0.

6. OCR Extension

FalconOCR extends Falcon Perception’s early-fusion architecture to document understanding as a compact two-stage system, combining layout detection with text recognition. It achieves competitive benchmark performance, while showing weaknesses on degraded scans, tiny text, and some structural matching cases.

  • Architecture: FalconOCR repurposes a unified dense Transformer for OCR while using a two-stage pipeline that separates layout detection from text recognition.A layout detector produces regions that the recognizer processes, enabling cropped-region inference and parallel batching.
  • olmOCR: 300M parameters yield 80.3% average accuracy on olmOCR, matching or exceeding several substantially larger systems.FalconOCR matches Gemini 3 Pro at 80.2%, PaddleOCR VL 1.5 at 79.3%, DeepSeek OCR v2 at 78.8%, and GPT 5.2 at 69.8%.
  • olmOCR: 87.1% on Multi-Column layouts and 90.3% on Tables demonstrate particularly strong performance on spatial document structure.The model also reaches 94.0% on Headers/Footers and 99.5% on Base documents.
  • olmOCR: 43.5% on OldScan and 78.5% on TinyText expose weaknesses with degraded historical documents and low-resolution small text.The authors connect these gaps to insufficient degraded-scan representation and limited pixel resolution for cropped tiny-text regions.
  • OmniDocBench: 88.64 Overall on OmniDocBench places FalconOCR ahead of several larger systems, while its 84.6 TEDS trails leading table-structure models.The paper attributes part of the TEDS gap to non-unique HTML representations and limited structural canonicalization.
  • Efficiency: FalconOCR occupies a favorable efficiency–accuracy point with 300M parameters, approximately 6000 tokens/second, and 2.8 images/second.It ranks top-3 on olmOCR and top-4 on OmniDocBench according to the paper’s comparison.

7. Discussion

Falcon Perception pursues a simpler dense-perception design: one early-fusion Transformer backbone, with complexity shifted toward training signals and sequence interfaces. The discussion identifies higher training cost and slower decoding as trade-offs, while pointing to data, context, and post-training as future improvement areas.

  • Design philosophy: Falcon Perception uses a single early-fusion Transformer backbone and shifts complexity toward training signals and the sequence interface.The design keeps small heads only for continuous, dense outputs.
  • Supported regimes: The model targets dense perception settings involving OCR, spatial constraints, relations, and long-context outputs.
  • Limitations: Training is more expensive and decoding can be slower than with fully parallel DETR-like models.
  • Future directions: Future gains are expected from better data mixtures, longer-context training for dense scenes, and post-training reinforcement learning.The stated post-training goal is improving selection of the right prediction from the model’s own distribution.

Contributions

The contributions section assigns responsibilities across architecture, perception training, distillation, data, evaluation, OCR, infrastructure, integrations, and leadership.

  • Architecture and perception: Architecture design was led by Yasser Dahou, Phúc H. Lê Khắc, Sanath Narayan, and Sofian Chaybouti.
  • Architecture and perception: Perception training and ablations were conducted by Sanath Narayan and Yasser Dahou.
  • Data and evaluation: Distillation, data, and evaluation and benchmark work involved the contributors listed in the report.The passage separately attributes distillation, data, and evaluation responsibilities across the named authors.
  • OCR and infrastructure: OCR training, codebase infrastructure, integrations, and leadership were assigned to the contributors listed in the report.

A. Metrics

The metrics evaluate localization, image-level presence classification, and their combination, using thresholded mask matching and both micro- and per-sample macro-aggregation.

  • Localization: Localization uses Hungarian matching to maximize total IoU between predicted and ground-truth masks.
  • Localization: At each IoU threshold, matched predictions meeting the threshold are true positives, while unmatched predictions and ground-truth masks yield false positives and false negatives.
  • Localization: Localization performance is summarized with micro-F1 computed from counts accumulated across positive datapoints.
  • Thresholding: Evaluation averages localization scores over 10 IoU thresholds from 0.50 through 0.95 and may also report per-threshold values.
  • Classification: Image-level classification treats each datapoint as object-present or object-absent based on whether any mask is predicted, then uses Matthews Correlation Coefficient.
  • Aggregation: Macro-F1 averages per-sample scores, assigning 1 to true negatives and 0 to false positives and false negatives to emphasize per-image stability.

B. Architecture Study

The architecture study examines how depth and width allocate capacity in a shared visual-autoregressive Transformer, including fixed-depth width scaling and iso-compute trade-offs. At 20 layers, performance improves through width 768 before slightly regressing at 896.

  • Motivation: The study tests depth–width capacity allocation because shared visual and autoregressive representations may create modality crowding.The concern is that channel bandwidth could bottleneck when both modalities compete for representational capacity.
  • Study design: Width scaling is evaluated at fixed depth L=20, alongside an iso-compute depth–width trade-off study.
  • Experimental control: Learning-rate transfer uses µP while keeping other hyperparameters identical across width configurations.The reference configuration is 20L / dmodel=768.
  • Width scaling: +6.8 MCC and +5.8 Macro F1 accrue as width increases from dmodel=384 to dmodel=768 at 20 layers.The reported endpoints are MCC = 46.4, F1 = 55.7 at 384 and MCC = 53.2, F1 = 61.5 at 768.
  • Width scaling: Increasing width to dmodel=896 slightly regresses performance, indicating saturation around dmodel ≈768 at 20 layers.At width 896, the reported MCC is 52.4.

B.2 Depth–Width Trade-off at Fixed Compute

At a fixed compute budget, balanced depth–width configurations outperform both deep-narrow and shallow-wide extremes, indicating that depth and width provide complementary capacity for dense prediction.

  • The balanced configurations outperform the deepest-narrowest 72L / 384 and shallowest-widest 14L / 896 variants by 2–4 MCC points.
  • The 54L / 448 model underperforms 72L / 384 despite its wider hidden state, placing that depth–width combination in an unfavorable operating regime.
  • Under iso-compute conditions, moderate configurations are optimal because representational diversity from depth and channel capacity from width are independently necessary.
  • The explored optimum is 42L / 512, while 20L / 768 offers a competitive shallower alternative when inference latency matters.

C. Qualitative Analysis

The qualitative analyses examine how Falcon Perception’s early-fused image features and upsampling affect instance masks, including comparisons with SAM 3 across varied prompts.

  • Early-fused image features distinguish multiple objects before upsampling, including cups, notebooks, pens, chopsticks, food items, and cups.
  • AnyUp upsampling improves mask quality over direct bilinear upsampling and refines instance boundaries as the factor increases.
  • Falcon Perception and SAM 3 produce different qualitative errors across prompts requiring visual text recognition, complex expressions, and dense predictions.
  • SAM 3 is not reliably suited to levels 2–4 because it was not trained for those prompt types, limiting the directness of that comparison.

C.4 Effect of Sampling

Sampling changes Falcon Perception’s predictions, while qualitative comparisons illustrate behavior across prompt complexity and dense instance counts.

  • Effect of Sampling: Nondeterministic sampling at higher temperature improves some predictions and identifies smaller or harder instances than greedy decoding.
  • Prompted Perception: Falcon Perception avoids several false positives shown for SAM 3 on prompts involving OCR, spatial relations, and object interactions.
  • Prompted Perception: Falcon Perception correctly identifies 168 wine bottles and a specified direction sign where SAM 3 predicts masks for each bottle or sign.
  • Dense Splits: For dense splits exceeding 200 instances, Falcon Perception scales autoregressively without the fixed query-token limitation described for SAM 3.
  • OCR Examples: The qualitative materials also include OCR outputs for formulae, tables, and handwriting.

D. Qualitative Analysis - OCR

FalconOCR qualitative examples cover challenging document inputs and show structured-text outputs across scientific, formulaic, tabular, and handwritten content.

  • FalconOCR handles challenging real-world images containing formulae, tables, handwriting, and varied text semantics while producing structured text.
  • The examples include arXiv papers and scientific formulae.
  • Additional examples vary equation structure and table structure, including sparse tables and dense tables with multi-column headers.
Loading 2603.27365v1…