VisionPsy-Nano: Improving Accuracy, Efficiency, and Reliability in On-Device Vision-Language Models

1. Introduction

• 62.3 normalized average over 17 benchmarks makes VPsy the highest-scoring openly released ∼0.5B model, +7.4 over its base at identical architecture and token budget.VPsy-Flash scores 61.4 with a 6.1 s Pixel 9 warm TTFT, compared with 138 s at the default policy (23×).Source: Figure 1 · p. 1, §1 · p. 2
• At 512² resolution, 0.5B checkpoints emit 256–1,088 visual tokens and warm TTFT ranges from seconds to over two minutes on the same phone.Checkpoints within five accuracy points can span a 9× TTFT range, while short-answer benchmarks and fixed question sets miss behavioral reliability issues.Source: §1 · p. 1
• A teacher VLM stress-tests the student and maintains a weakness catalogue that directs repair stages beyond generic supervision.The paper reports that failure-targeted supervision beats size-matched generic supervision across all four capability areas.Source: §1 · p. 2
• The post-training recipe adds no parameters and preserves base serving cost, while two visual-token policies produce VPsy and VPsy-Flash.At 512², VPsy uses the default 1,088-token visual prefix; Flash uses 64 tokens under a bounded native-resolution policy.Source: §1 · p. 2
• Evaluation covers 17 benchmarks, warm TTFT on four phones, and long-form and adversarial stress tests; alignment repairs a doom-loop regression and improves safety pass rates in seven of eight categories.The doom-loop repair is reported in Table 5 and safety results in Figure 5.Source: §1 · p. 2

Edge-efficient vision-language models

• Efficient VLM work addresses compact backbones and visual-token cost, while PACT [8], ATP-LLaVA [57], and Delta-LLaVA reduce or adapt token use.The paper targets this complementary efficiency bottleneck on the public nanoVLM implementation.Source: §2 · p. 2

Failure-guided adaptation and retention

• Failure-guided adaptation includes teacher diagnoses [56], multimodal evaluation signals [53], and failure-informed data synthesis [16], while retention methods address forgetting [6] [38].The paper combines failure mining and teacher-guided discovery with gold-anchored correction and treats retention as a constraint.Source: §2 · p. 2

Model merging for capability composition

• TIES-Merging [54] resolves sign disagreement and averages aligned task-vector components; the paper uses it to consolidate specialists without changing serving cost.The related-work discussion situates this among model averaging [51], task vectors [14], and multimodal merging [47].Source: §2 · pp. 2–3

3. Post-Training Recipe

• Every stage after the first constrains retention, defined as preserving earlier-stage capabilities and measured by accuracy changes on untargeted capability areas.The narrowed training distributions can degrade untargeted capabilities within a fraction of an epoch at 460M parameters.Source: §3 · p. 3

3.1. Failure Analysis Framework

• Passive mining finds verified errors in existing labeled data, while active discovery probes residual weaknesses and feeds failures back into a persistent catalogue.The framework runs against θ2, after broad and capability-targeted supervision, and its diagnosis guides repairs in Stages 3–5.Source: §3.1 · pp. 3–4, §3.2 · p. 4
• The active loop samples toward failing data sources, has a teacher write probes, grades answers with the image in view, and revises the weakness catalogue from failures.Successive rounds rewrote seeded hypotheses and found categories not covered by seeds, including hallucination, refusals, and gibberish arithmetic.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2. Training Stages

• Stage 1 broadens multimodal coverage with image-question-answer data and token-level cross-entropy before targeted intervention.Its corpus is filtered to single-image samples and has chain-of-thought segments stripped.Source: §3.2 · p. 4
• Stage 2 strengthens deployment capabilities with D1 ∪ D2 while rehearsing D1, producing θ2 as the residual-diagnosis reference and merge base.D2 supplements underrepresented OCR, document and chart understanding, formula recognition, spatial reasoning, and multimodal instruction following.Source: §3.2 · p. 4
• Stage 3 creates six capability specialists from diagnosed weaknesses using gold-anchored teacher-generated examples that pass repeated answer verification.The targeted mixtures use replay buffers and InfoSFT weighting to focus loss on informative tokens while supporting the later merge.Source: §3.2 · pp. 4–5
• Stage 4 merges specialist task vectors relative to θ2 with TIES-Merging [54], while Stage 5 uses preference alignment for doom-loop termination and safety.Stage 5 mines clean-versus-loop rollouts and uses capability replay; safety alignment uses curated targets and preference pairs with benign boundary cases.Source: §3.2 · p. 5
• Stage 4’s merged checkpoint recovers each specialist’s advantage and is stronger overall than any single specialist or naive averaging, while incurring a small perception trade-off.Specialists are selected on held-out validation splits, and λ is set to favor capability areas most in need of improvement; neither is tuned on benchmarks.Source: §3.2 · p. 5

• VPsy-Flash executes the full five-stage recipe under a reduced visual-token budget without architectural changes, added parameters, or runtime token-compression heuristics.Source: §4 · p. 5
• VPsy-Flash trains every stage with bounded native-resolution preprocessing, while VPsy retains the default full-resolution policy.Source: §4 · pp. 5–6
• The policy floors short sides at 512 pixels, caps long sides at 2048, and snaps intermediate images to exact 512-pixel tile multiples.Source: §4 · p. 6
• Each view yields 64 embeddings, so Nvis = 64 · (1 + Nlocal) directly determines visual-prefix length, prefill latency, and memory footprint.Source: §4 · p. 6

5.1. Accuracy

• On 16 of 17 benchmarks, VPsy outperforms LFM2.5-VL-450M and attains cohort-highest Avgnorm; its 7.4-point gain over nanoVLM isolates post-training.Source: §5.1 · p. 6

5.1.1. Ablation Study

• Avgnorm rose from 54.9 to 60.0 with broad SFT, gained +1.3 to 61.3 with capability SFT, then +1.0 to 62.3 after specialist fine-tuning and merging.Source: §5.1.1 · p. 6, Table 2 · p. 7
• The best failure-guided specialist improves document understanding and holds other axes, while the best size-matched random specialist leaves every axis flat or slightly degraded.Source: Table 3 · p. 7, §5.1.1 · p. 7
• TIES leads naive averaging and the best single specialist on three of four axes at unchanged inference cost, though it trades a small perception dip.Source: Table 3 · p. 7, §5.1.1 · p. 7
• Training on D2 alone degrades every untargeted axis; full Stage 2 holds them within one point of θ1, while single-stage D1∪D2 trails staged θ2.Source: Table 3 · p. 7, §5.1.1 · p. 7

5.2. On-Device Efficiency

• 6.1 s is VPsy-Flash’s warm TTFT on Pixel 9 versus 138.3 s for nanoVLM, a 23× speedup; it is fastest on every tested phone.Source: Table 4 · p. 7, §5.2 · p. 8

5.3. Behavioral Reliability

• Stage-5 alignment repairs doom-loop rates from 4.3% and 1.52% at the respective θ4 checkpoints to 0.25% and 0.51%.Source: Table 5 · p. 7, §5.3 · p. 8
• VPsy improves over nanoVLM on seven of eight safety categories, including illegal activity (44%→67%), PII leakage (67%→81%), and misinformation (64%→81%).Source: Figure 5 · p. 8, §5.3 · p. 8

• The recipe is evaluated only in the 0.5B on-device regime and on a single base architecture; transfer to larger models or other backbones remains future work.Source: §6 · p. 8
• Failure discovery is bounded by the teacher, so failure modes it cannot detect remain outside the weakness catalogue.Source: §6 · p. 8
• Safety results cover a fixed suite of eight vulnerability categories and nine attack strategies, making them a controlled probe rather than a guarantee of real-world robustness.Source: §6 · p. 8
• Evaluation is automatic over 17 public benchmarks, with conclusions checked across three judges to mitigate metric and judge sensitivity.Source: §6 · p. 8

7. Conclusion

• VPsy reaches a 62.3 normalized average across 17 benchmarks, while VPsy-Flash scores 61.4 with 6.1 s Pixel 9 warm TTFT, 23× below the default policy.Both are independently trained, deployable ∼0.5B checkpoints; the conclusion reports the highest normalized average among openly released ∼0.5B models for VPsy.Source: §7 · pp. 8–9
• The recipe uses post-training alone, adds no parameters, and repairs failure modes—including a doom-loop regression—that all 17 benchmarks are structurally unable to observe.The active teacher–student loop uncovers failures beyond human priors.Source: §7 · pp. 8–9

A.2. Loop mechanics

• The planner feeds the teacher the image, reference Q&A, and current weakness catalogue, then requests probes tagged with expected answers, capability categories, personas, and weaknesses.Reusing the catalogue directs probes toward known soft spots rather than generic exploration.Source: §A.2 · p. 12
• The judge grades each image’s probe answers for verdict, severity, category, and rationale, while ungraded outputs are excluded from reported rates and reflection.The rubric treats persistent repetition as a severe failure, while character or tokenization slips and empty or filler answers receive separate labels.Source: §A.2 · p. 12
• At each round’s end, the teacher rewrites the catalogue using the 40 latest failures, mapping them to 19 fixed labels and recomputing cluster evidence from the probe log.The rewritten catalogue informs the next round’s planner.Source: §A.2 · p. 12

A.3. Seed catalogue

• The seed catalogue starts from error analysis across capability areas and models, manual inspection of failures, and generic sub-billion-VLM failure modes corroborated by inspection.Unlike a fixed red-teaming rubric, the initial seed catalogue can be re-ranked, rewritten, or abandoned by the teacher.Source: §A.3 · p. 13

A.4. How the loop refined the seeds

• The teacher rewrote all 13 seed hypotheses and probe-template sets, escalated two to maximum severity, and effectively abandoned unsupported geometry and multi-step math.The refined descriptions identify observed mechanisms and inform the K = 6 weakness-targeted mixtures; the passage also names the largest-evidence clusters as chart reading, numeric reasoning, spatial–OCR integration, digit OCR, and doom-loop termination.Source: §A.4 · p. 13
• The refined catalogue distinguishes concrete failures such as conversational chart phrasing, numeric operations over correctly read values, dense-numeral OCR, and lost row or column associations.Other listed weaknesses include multi-span document synthesis, spatial–OCR integration, figure and diagram QA, and fine-grained recognition.Source: §A.4 · p. 13

A.5. Novel weaknesses discovered by the teacher

• Among graded probes, 78.3% were fail or partial (12,430 fail, 1,511 partial, 3,870 pass); the 337 ungraded probes were excluded from every reported rate.Conditioning the planner on the catalogue tilted sampling toward sources the student was currently failing, unlike a uniform sampler over the same pool.Source: §A.5 · p. 14
• The first novel cluster appeared in round 1, half of the 24 were catalogued by round 41, and all 24 by round 51; no new cluster appeared across 10,872 later graded probes through round 229.The catalogue therefore saturated within a few dozen rounds in this run.Source: §A.5 · p. 14

B.1. Quantization and Accuracy Retention

• Q4 K M reduces complete VLM footprint from 1,666 to 393 MB while reducing normalized score by 0.24 points for VPsy and 0.53 for VPsy-Flash.IQ4 variants use 346–348 MB while remaining within one normalized point of their FP32 references; quantization outcomes depend on scheme, not storage footprint alone.Source: §B.1 · p. 15, Table 9 · p. 15

B.3. Visual-Token Footprint

• At 512², VPsy-Flash uses a 64-token global view, a 17× smaller visual prefix than VPsy and nanoVLM and 4× smaller than Qwen3.5.At 2048×1024, VPsy and VPsy-Flash converge to the same 576-token layout; nanoVLM and SmolVLM2 use 1,088 tokens for a 512² image versus 576 for the larger image.Source: §B.3 · pp. 15–16, Table 11 · p. 15

B.4. Per-Backend TTFT Profiling

• On Android, CPU execution can match or outperform GPU TTFT on Pixel 9 and often S25 Ultra, while GPU execution systematically dominates on iPhone and typically leads on Galaxy S23.The passage attributes Android variation to sequential-stage overhead, kernel launches and synchronization, and weaker quantized matrix-multiplication kernels on Mali GPUs.Source: §B.4 · p. 16
• At 2048×1024, VPsy-Flash has 1.35×–2.23× TTFT speedups over nanoVLM, 1.51×–2.08× over SmolVLM2, and 1.58×–2.54× over LFM2.5-VL.The speedup gap narrows at this resolution as visual-token budgets converge.Source: §B.4 · p. 16

B.5. Peak Memory

• At 512², VPsy-Flash lowers peak memory by 24%–39% on Android CPU and GPU paths and by 41%–65% on iPhone 15 relative to nanoVLM and SmolVLM2.At 2048×1024 the advantage over full-resolution baselines narrows, while VPsy-Flash remains below Qwen3.5 in six of eight device-backend pairs.Source: §B.5 · p. 16

B.6. Autoregressive Decode Throughput

• At 512², VPsy-Flash provides 1.1×–1.8× higher decode throughput than nanoVLM and SmolVLM2 and 1.3×–2.1× than Qwen3.5 on each device’s faster lane.Across both resolutions, it exceeds nanoVLM and SmolVLM2 in 13 of 16 device-backend configurations and Qwen3.5 in 15 of 16.Source: §B.6 · p. 16, Table 14 · p. 18

B.7. End-to-End Latency

• In free-form generation, TTLT captures vision encoding, projection, prefill, and decoding while excluding model loading; models decode naturally up to a 4,096-token cap.VPsy-Flash’s median response is 35.9 tokens versus 29.1 for LFM2.5-VL-450M, while nanoVLM and SmolVLM2 average 136.4 and 254.4 tokens; the passage attributes its efficiency advantage to the compressed visual footprint and prefill bottleneck.Source: §B.7 · p. 17

C.1. Robustness to the choice of judge.

• VPsy leads LFM2.5-VL-450M by 2.7, 2.9, and 2.8 normalized points under Qwen3.6-27B, GPT-4o-mini, and GPT-5-chat, respectively, and wins 16 of 17 benchmarks under each judge.Absolute scores vary across judges, particularly on MM-IFEval and MMVet; the table rescored eight judge-dependent benchmarks and retained the other nine unchanged in Avgnorm.Source: Table 15 · p. 18, §C.1 · p. 18
• The judge adjustment applies to five benchmarks where the protocol departs from conventional scoring, and the judge raises every model’s score on all five.The largest average gains are for SmolVLM2-500M and the base checkpoint, at 6.6 and 6.7 points, versus 5.0 for the two authors’ models.Source: §C.1 · p. 18

C.2. Impact of Post-Training under the Bounded Native-Resolution Policy

• The LLM judge rescues every model on all five judged benchmarks, with the largest score corrections going to baselines rather than the authors’ models.Across three independent judges, absolute scores shift but the ordering and aggregate margin do not.Source: Table 16 · p. 19
• The Zero-shot baseline applies VPsy-Flash preprocessing to the full-resolution checkpoint only at inference, while Full Recipe evaluates the checkpoint adapted under the compressed budget.The section motivates adaptation by arguing inference-only bounding induces a severe distribution shift.Source: §C.2 · pp. 18–19
• 61.4 overall normalized average with the Full Recipe, versus 60.1 for Zero-shot inference-time compression, supports post-training under the bounded visual-token budget.The full recipe also improves ChartQA by 8.0 points, OCRBench by 29 points, MME by 79 points, and POPE by 4.5 points over Zero-shot.Source: §C.2 · p. 19

D.1. Evaluation Protocol

• All main-paper entries use one VLMEvalKit harness with shared prompts, configuration, and metric implementations, while reporting each benchmark’s official metric.Five free-form-answer benchmarks use an LLM judge because exact match can conflate reading accuracy with surface-form compliance; three judges preserve the ordering and aggregate margin despite score shifts.Source: §D.1 · pp. 19–20

D.2. Training Details

• Stage 1 fine-tunes on decontaminated Nemotron VQA data, and Stage 2 adds FineVision and MM-IFEngine instructions while retaining the Stage-1 data mix.Training images near-duplicate to evaluation images are removed; multi-page examples use a page-selection heuristic and chain-of-thought segments are stripped.Source: §D.2 · p. 20
• Stage 3 adds InfoSFT with replay, Stage 4 merges six specialists using TIES-Merging, and Stage 5 aligns behavior with an MPO objective combining DPO, BCO, and chosen-response SFT.The final stage proceeds through doom-loop stabilization and safety alignment.Source: §D.2 · p. 20

E.1. Qualitative Comparisons

• VPsy and VPsy-Flash provide accurate, grounded responses in qualitative comparisons, while sub-billion baselines frequently hallucinate details or miss spatial and counting tasks.The examples indicate VPsy-Flash retains VPsy’s reasoning capabilities despite substantial visual-prefix compression.Source: §E.1 · p. 21

E.2. Failure Cases and Limitations

• All evaluated sub-billion models struggle with complex mathematical and geometric reasoning, fine-grained symbol detection, and conditional arithmetic over dense charts.Baselines often produce verbose, nonsensical reasoning when failing, whereas VPsy and VPsy-Flash typically fail concisely; the paper notes specialized multi-step capabilities generally benefit from larger-model capacity.Source: §E.2 · p. 21

E.3. Live On-Device Deployment

• In the room-identification example, VPsy and VPsy-Flash answer “Museum room,” while nanoVLM and LFM2.5-VL give different responses.The deployment screenshots are a qualitative demonstration on a specific high-resolution chart, not the standardized warm-run evaluation.Source: §E.3 · pp. 22, 24
• The accompanying failure examples show sub-billion models struggling with multi-step geometry, fine-grained symbols, and conditional data extraction from dense charts.The figure notes that baseline reasoning can be verbose and hallucinated, while the authors’ models tend to fail more concisely.Source: Figure 8 · p. 23
• 0.9s on iPhone 15 and 9.5s on Pixel 9 show VPsy-Flash reducing first-token delays across devices, including an over 12× speedup over its full-token counterpart on Pixel 9.The screenshots report 13.7s for default-policy nanoVLM on iPhone and 115.8s for the base model on Pixel 9; Samsung Galaxy S25 Ultra delay falls from up to 54.3s to 3.2s.Source: §E.3 · p. 24, Figure 9 · p. 25, Figure 10 · p. 25