PaperlayerView exact revision on arXiv

Source-linked AI summary

ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation

Anindya Mondal, Sauradip Nag, Anjan Dutta

arXiv:2606.23835v1cs.CVeess.IV

TL;DR

Unified models struggle to support fine-grained counting and generation together because existing approaches rely on task-specific pipelines and limited spatial grounding. ABACUS addresses this with spatially grounded counting, boundary-aware optimization, and cycle-consistent self-reward, achieving state-of-the-art performance across seven benchmarks with one 3B-parameter model.

  • Problem

    Existing unified vision-language models struggle with fine-grained instance-level counting and with jointly bridging visual understanding and generation.

  • Method

    ABACUS combines density-aware adaptive zooming, attention-derived objectness maps, boundary-aware GRPO, and cycle-consistent self-reward in one unified model.

  • Results

    ABACUS achieves state-of-the-art results across seven benchmarks spanning counting, count-faithful generation, and count reasoning, surpassing specialists and larger generalist models.

  • Takeaways & Limitations

    The results support jointly optimizing count understanding and generation as mutually reinforcing objectives for spatial awareness.

  • Takeaways & Limitations

    ABACUS’s current count-balanced training mixture does not cover specialized domains such as medical cell counting, satellite vehicles, or industrial defects.

Abstract

from arXiv · show

ABACUS is a unified vision-language model that handles object counting, crowd counting, referring-expression counting, and count-faithful image generation without any benchmark-specific training required. Our model is built on existing 3B-parameter unified foundation model and is adapted for object localization tasks using three key innovations: density-aware adaptive zooming with objectness maps for spatial grounding; a boundary-aware count policy via GRPO to eliminate crop-boundary errors; and a cycle-consistent GRPO strategy where the understanding branch self-critiques generated outputs, closing the understanding-generation gap without any external annotations. ABACUS achieves state-of-the-art results across seven benchmarks, outperforming both task-specific specialists and larger generalist models.

1. Introduction

ABACUS unifies object, crowd, and referring-expression counting with count-faithful image generation in a single zero-shot vision-language model. It addresses the persistent understanding–generation synergy gap through adaptive spatial grounding, boundary-aware counting, and cycle-consistent reinforcement learning.

  • Motivation: Existing counting and count-conditioned generation methods use task-specific architectures, losses, and training pipelines, while unified models still exhibit a synergy gap between understanding and generation [5].Models may count four apples correctly yet fail to generate exactly four, and dense-scene counting remains unreliable [61].
  • ABACUS: ABACUS jointly performs object counting, crowd counting, referring-expression counting, and count-faithful image generation in one unified model with zero-shot generalization.It is presented as the first unified VLM covering these counting and generation tasks without benchmark-specific training.
  • Method: Density-aware adaptive zooming uses MLLM attention-derived objectness maps to recursively partition dense images into manageable regions for spatially grounded counting.The method targets the documented difficulty of off-the-shelf MLLMs in parsing dense visual scenes [61].
  • Method: Boundary-aware count policy via GRPO removes overcounting and undercounting artifacts introduced at crop boundaries, while cycle-consistent GRPO uses the frozen understanding branch to score generated images.Count-deviation and aesthetic rewards update only the generation branch, closing the understanding–generation gap without external annotations.
  • Results: ABACUS sets a new state of the art across seven benchmarks spanning object counting, crowd counting, referring-expression counting, and count-faithful generation.The contribution summary explicitly reports coverage across these four evaluation areas.

2. Related work

Prior work spans class-specific and class-agnostic counting, multimodal vision-language models, and counting-aware image generation, but existing generation pipelines keep counting separate from synthesis. ABACUS instead unifies generation, counting, and verification within one model without external critics or annotations.

  • Counting methods: Counting methods include class-specific predictors for fixed categories and class-agnostic approaches guided by visual exemplars or text prompts, with most requiring dense point-level supervision.Class-specific methods use detection or density-map regression for categories including persons, vehicles, and cells [30] [64] [69] [54] [43] [46] [39].
  • Count-faithful generation: Text-to-image diffusion models achieve only 25–28% exact-match on count-conditioned benchmarks, while prior methods add external counting signals through losses, denoising-loop heads, or cross-attention manipulation.These approaches leave the counting module architecturally disjoint from the generator [57] [14].
  • Unified generation and verification: Recent systems use external VLM critics for iterative correction, but ABACUS unifies the generator, counter, and verifier so understanding directly rewards generation without external critics, planners, or annotations.The limitation of critic-based correction is dependence on critic reliability [53].
  • Vision-language models: Early VLMs such as CLIP [63] and BLIP [40] aligned vision and text through contrastive pretraining, while later MLLMs [7] [38] strengthened reasoning and generation for downstream tasks.Related work also adapted VLMs for text-promptable object counting [33] [61].

3. Preliminaries

ABACUS uses Group Relative Policy Optimisation (GRPO) as a unified post-training mechanism for understanding and generation. GRPO computes rollout-relative advantages without a value network and regularizes policy updates against a frozen reference policy.

  • Group Relative Policy Optimisation: GRPO serves both ABACUS understanding and generation branches as a unified post-training mechanism [68].
  • Group Relative Policy Optimisation: GRPO removes PPO’s value network by computing advantages relative to groups of policy rollouts, reducing memory and compute overhead.
  • Group Relative Policy Optimisation: The policy maximizes a surrogate objective with a KL penalty against a frozen reference policy, whose regularization strength is controlled by β > 0.
  • Density-aware adaptive zooming: Density-aware adaptive zooming recursively partitions dense regions into 2×2 sub-regions until resolution γ, while sparse regions use one pass before aggregation.

4. Method

ABACUS equips one unified vision-language model with count-accurate image generation and precise object understanding. Its method combines density-aware zooming, objectness-guided localization, boundary-aware GRPO counting, and cycle-consistent generation training.

  • Framework Overview: ABACUS gives a single unified VLM both count-accurate image generation and precise object counting.The framework targets unified count understanding and generation rather than separate models.
  • Density-aware Adaptive Zooming: Density-aware adaptive zooming recursively partitions dense images into sparser regions, whose local counts are aggregated into a global count.Each sub-image is queried independently, improving reliability where local object density is lower.
  • Infusing Objectness in MLLM: Objectness maps extracted from language-model attention provide spatial evidence for object localization and steer counting toward instance-aware reasoning.Per-head isolation and learned affine alignment identify object peaks, while Gaussian-smoothed point supervision regularizes the predicted map.
  • Boundary-Aware Count Policy: Boundary-aware GRPO classifies objects as interior, edge, or boundary and trains consistent quadrant counts, preventing crop-line double-counting from adaptive zooming.Objects exactly on crop lines are assigned to one quadrant randomly, while nested rewards enforce local and global consistency.
  • Generation via Understanding: Cycle-consistent GRPO trains generation with rewards combining understanding-branch count deviation and aesthetic quality while keeping the counter frozen.Only the generation-branch LoRA is updated, preventing reward hacking and corruption of the branch that evaluates generated counts.
  • Training Strategy: Training proceeds in stages: LoRA-based understanding finetuning with objectness regularization, GRPO post-training on curated samples, and subsequent connector training.The first stage uses 2M densely annotated images and 50K curated samples, while the connector and generation branch remain frozen during understanding training.

5. Experiments

ABACUS delivers strong unified performance across object, crowd, referring-expression counting, and count-faithful generation, outperforming specialists and larger unified models without benchmark-specific training. Ablations attribute these gains to objectness-guided adaptive zooming, boundary-aware counting, and cycle-consistent GRPO.

  • Object and Crowd Counting: ABACUS outperforms specialist and VLM counters across FSC-147, CARPK, and ShanghaiTech, including 78.59/14.75 MAE on ShanghaiTech-A/B versus CountGD++’s 116.0/28.0.It achieves 5.71 validation and 5.03 test MAE on FSC-147, 8.41 MAE on CARPK, and gains 5× and 3× over UniLIP-3B on FSC-147 and CARPK.
  • Referring Expression Counting: On REC-8K, text-only ABACUS achieves MAE 7.67 and RMSE 15.84, surpassing fine-tuned GDINO and matching GrREC while lowering RMSE from 19.79.The evaluation covers 3,153 pairs without architectural modification or benchmark-specific training.
  • Qualitative Comparison: Qualitative comparisons show ABACUS tracking ground-truth counts across density regimes and producing exact or near-exact counts with natural spatial arrangements.Baselines exhibit dense-scene overcounting, out-of-distribution failures, systematic undercounting, overcounting, rigid layouts, or mode collapse.
  • Ablation Studies: Removing objectness regularization causes the largest ablation degradation, increasing MAE by 3.92 and widening the overlap-heavy/light MAE gap from 2.31 to 6.18.Density-aware adaptive zooming avoids single-pass failures on dense scenes and fixed-grid double-counting at tile boundaries while remaining within 1.2× of single-pass inference.
  • Ablation Studies: Cycle-consistent GRPO raises CoCoCount exact-match beyond SFT’s 45% and open-loop GRPO’s 17-point improvement by a further 9 points through co-adaptation of understanding and generation.The cycle generates images, self-counts them, compares counts with prompts, and updates generation using progressively more informative rewards.

6. Limitations and future work

ABACUS struggles on low-resolution or heavily compressed images because its patch-based spatial tokens cannot resolve individual objects, although a single 3B-parameter model achieves state-of-the-art results across seven diverse benchmarks. Super-resolution preprocessing is proposed to extend the method to degraded settings.

  • Low-resolution and degraded inputs: ABACUS’s counting pipeline requires sufficient resolution because InternViT’s 14×14 patch encoding may not distinguish individual instances in degraded images.The limitation arises when visual tokens are too coarse for the objectness map to resolve individual objects.
  • Low-resolution and degraded inputs: Super-resolution preprocessing could extend ABACUS to low-resolution and heavily compressed settings.
  • Single-model generality vs. domain adaptation: A single 3B-parameter ABACUS model achieves state-of-the-art results across seven benchmarks spanning diverse visual domains.

7. Conclusion

ABACUS unifies count-aware image understanding and count-faithful generation in one architecture, using spatial grounding, boundary-aware counting, and cycle-consistent self-reward. The conclusion argues that jointly optimizing understanding and generation produces mutually reinforcing spatial awareness.

  • Conclusion: ABACUS unifies count-aware image understanding and count-faithful generation within a single architecture, combining objectness-based spatial grounding, boundary-aware GRPO counting, and cycle-consistent self-reward.The objectness map uses MHSA head decomposition and point supervision; the boundary-aware policy uses nested rewards, while cycle-consistent self-reward links understanding and generation.
  • Conclusion: Jointly optimizing count understanding and generation yields emergent spatial awareness that neither specialist alone can achieve, motivating extension of cycle-consistent self-reward beyond counting.The proposed extension targets other spatial reasoning tasks where the same model produces and verifies its outputs.
  • Count understanding gallery: ABACUS obtains exact or near-exact counts across sparse and dense scenes in FSC-147, CARPK, and ShanghaiTech using text-only prompts.The gallery spans counts from 1 to 261 across diverse categories and count ranges.
  • Count generation gallery: ABACUS generates images with the exact requested count across diverse prompts while maintaining naturalistic spatial arrangement and high aesthetic quality.This gallery demonstrates count-faithful generation across varied prompts.

A. Additional Ablations · A.1. Counting Readout: Autoregressive vs. Objectness Peak

The appendix ablates the counting readout and boundary-aware count policy using the main paper’s FSC-147 validation protocol. For counting readout, ABACUS compares autoregressive text generation with direct objectness-peak counting from the same model without extra parameters or training.

  • A. Additional Ablations: The ablations use FSC-147 validation MAE/RMSE under the main ABACUS training setup unless otherwise stated.Variants share the same LoRA adapter, training data, and hyperparameters as the main model.
  • A. Additional Ablations: The appendix evaluates two component-level choices: counting readout and decomposition of the boundary-aware count policy.These ablations were deferred from the main paper.
  • A.1. Counting Readout: Autoregressive vs. Objectness Peak: ABACUS counts objects either by thresholding its binarised objectness map and counting connected components or by generating the count as text tokens.The former is the objectness-peak readout, while the latter is autoregressive.
  • A.1. Counting Readout: Autoregressive vs. Objectness Peak: The autoregressive readout is ABACUS’s default counting method.
  • A.1. Counting Readout: Autoregressive vs. Objectness Peak: The two readouts are compared from the same model in Tab. 8, with no additional parameters or training.
  • A.1. Counting Readout: Autoregressive vs. Objectness Peak: Objectness-peak counting is competitive on dense images containing ≥50 ground-truth objects.The supplied passage specifies this dense-image regime but does not provide the corresponding metric values.

A.2. Boundary-Aware Count Policy: Reward Decomposition

The boundary-aware count policy uses GRPO reward shaping to address double-counting and missed objects at quadrant boundaries. Ablations show that reinforcement learning and boundary arbitration are especially important for dense images.

  • Reward Decomposition: Removing the reward that arbitrates split-object ownership causes the largest degradation, increasing MAE by 1.57.This arbitration prevents objects crossing quadrant boundaries from being double-counted or missed.
  • Reward Decomposition: Without GRPO, SFT-only training increases MAE by 2.48, confirming that reinforcement-learning reward shaping is necessary.The policy decomposes training into nested rewards, including per-quadrant local accuracy (∆q).
  • Reward Decomposition: Removing the reward that enforces coherent aggregation increases MAE by 0.64, while local quadrant counts remain accurate.On dense-activated images, the boundary policy improves MAE by 3.74 without changing sparse-image performance.

B. Implementation Details

ABACUS is instantiated on UniLIP-3B with a selectively trainable multimodal architecture, while most vision, diffusion, decoding, and query components remain frozen. Training uses LoRA-based adaptation followed by sequential boundary-aware and generation GRPO post-training on 8× A100 80GB GPUs.

  • Architecture: ABACUS builds on UniLIP-3B [74], combining InternViT [18], Qwen2, SANA, and DC-AE [16] through a multimodal connector with N=256 learnable queries.The cross-modal projector and output head are trained jointly, while the visual encoder, diffusion transformer, pixel decoder, and query bank remain frozen.
  • Optimization: LoRA adapts Qwen2 attention and feed-forward projections with rank r=32 and α=64, adding ∼48M trainable parameters.The adapters are applied to WQ, WK, WV, WO, Wup, and Wdown projections.
  • Optimization: Training uses AdamW with 2K-step warmup, cosine decay from 2×10−5 to 2×10−6, norm-1.0 clipping, and sequential 2K-step boundary-aware and 5K-step generation GRPO.The model is trained in bfloat16 mixed precision for ∼44 hours on 8× A100 80GB GPUs.

C. Human Evaluation

The human evaluation assesses generated images for count accuracy, aesthetic quality, prompt alignment, and overall preference using 30 annotators and 60 stratified prompts. Results compare seven methods, with consolidated aesthetic, alignment, and preference scores reported in Table 10.

  • Evaluation Setup: Human evaluation uses 30 annotators and 60 stratified prompts spanning CoCoCount, T2I-CompBench, and GenEval across the full count range.The prompt set contains 25 CoCoCount prompts, 25 T2I-CompBench counting prompts, and 10 GenEval prompts.
  • Comparison and Reporting: Seven methods are compared, with scores averaged across applicable prompt groups and preference win rates measured against a 20% random baseline.For T2I-CompBench, only human scores are reported because YOLOv9 detection is not applicable to the open-vocabulary setting.
  • Evaluation Criteria: Images are rated on Count Accuracy, Aesthetic Quality, and Prompt Alignment using 0–4 Likert scales, alongside an overall preference judgment.Count Accuracy measures exact quantity, Aesthetic Quality covers visual craftsmanship, and Prompt Alignment measures textual fidelity excluding object count.
  • Comparison and Reporting: Table 10 consolidates Aesthetic Quality, Prompt Alignment, and Overall Preference results, with Likert scores normalized to 0–100.Preference is reported as win rate percentage, while Count Accuracy is reported in the main paper.
Loading 2606.23835v1…