PaperlayerView exact revision on arXiv

Source-linked AI summary

VectraYX-Vision-1B: A Sub-2B Spanish/LATAM Cybersecurity Vision-Language Model with Structured Visual Reasoning and Native Tool Use

Juan S. Santillana

arXiv:2608.08477v1cs.CL

TL;DR

Security analysts lack Spanish/LATAM cybersecurity VLMs that support specialized imagery and air-gapped deployment. This paper introduces VectraYX-Vision-1B, but preliminary valid checkpoints achieve near-zero visual-grounding scores, indicating insufficient vision SFT.

  • Problem

    Existing assistance is text-only or general-purpose, while specialized Spanish/LATAM cybersecurity VLMs suitable for air-gapped environments remain limited.

  • Method

    The paper couples a frozen SigLIP encoder with a Spanish/LATAM cybersecurity decoder and introduces NoPE/RoPE ablations for visual-token attention.

  • Results

    Near-zero B6/B7 scores from valid checkpoints indicate that the current vision SFT volume is insufficient for visual grounding.

  • Takeaways & Limitations

    The released system and ablation designs provide a functional platform for investigating visual grounding and periodic NoPE layers after remediation.

  • Takeaways & Limitations

    Early phase-4b/4c metrics are invalid because a checkpoint-loader bug left the LLM backbone randomly initialized.

Abstract

from arXiv · show

We present VectraYX-Vision-1B, a sub-2B vision-language model (VLM) for Spanish/LATAM cybersecurity imagery, coupling a frozen SigLIP-so400m encoder to a 1.04B Spanish/LATAM security decoder via an MLP. To our knowledge, it is the first sub-2B VLM specialized for cyber UI (IDA, Ghidra, Wireshark, Nmap, Metasploit, Volatility) that answers in Spanish, emits structured reasoning via native <|think|> tokens, invokes tools via Model Context Protocol (<|tool_call|>), and exports to llama.cpp's LLaVA mmproj format for air-gapped deployment. We report a negative preliminary visual-grounding result: despite fully functional pipelines, the current vision SFT (400-1900 steps, ~16M tokens) yields near-zero B6 scores (0.08 tool-identification), ignoring image content. We specify remediation (longer SFT, >=60% replay, lower LR) and expose a checkpoint-loader bug (unstripped llm. prefix) masquerading as training collapse. Crucially, we introduce a 3-variant ablation matrix (V0: NoPE-every-4, V1: all-RoPE, V2: NoPE+learned 2D) to study if periodic no-positional-encoding (NoPE) layers help or hurt attention over the 729-token visual block. Code, configs, and weights are released to establish priority on this architectural question. We provide B1-B5 for the text backbone, text controls, preliminary B6/B7 scores, wall times, GGUF efficiency on CPU, and a corpus of 14,596 QA pairs across 10 domains. We open-source all models and trajectories: jsantillana/vectrayx-1b, jsantillana/vectrayx-vision-1b, and jsantillana/vectrayx-vision-1b-checks.

1 Introduction

VectraYX-Vision-1B is a sub-2B Spanish/LATAM cybersecurity VLM designed to interpret security screenshots, reason visually, invoke tools, and run offline. The paper reports a negative preliminary visual-grounding result while releasing the system, corpus, benchmarks, and an ablatable NoPE×vision hypothesis.

  • Motivation: Security analysts rely on screenshots from IDA, Splunk, Wireshark, Volatility, Nmap, and Metasploit, but dominant machine assistance remains text-only or general-purpose VLMs.The motivation is that text-only LLMs cannot see the analyst’s screen.
  • System: VectraYX-Vision-1B couples a frozen SigLIP-so400m encoder and MLP projector to a 1.04B Spanish/LATAM cybersecurity decoder.It targets security screenshots and is designed for defensive or authorized-offensive use within organizational security perimeters.
  • Deliverable: The released system emits structured visual reasoning with <|think|> tokens, invokes MCP tools through <|tool_call|> tokens, and runs offline under llama.cpp.The artifact includes model.gguf and a LLaVA-compatible mmproj.gguf.
  • Preliminary result: 0.08 best B6 tool-identification on 50 held-out screenshots indicates that ≈16 M vision-SFT tokens across 400–1,900 steps did not produce visual grounding.A text-only backbone control confirmed the benchmark was not answerable from text alone.
  • Evaluation resources: 14,596 QA pairs across ten domains support two released benchmarks: B6_vision for screenshot tool identification and correctness, and B7_think for thinking-chain presence and quality.The corpus combines synthetic IDA/Ghidra/terminal images with real tool screenshots in a dual English/Spanish pipeline.

2 Related Work

VectraYX-Vision-1B follows the established encoder–projector–decoder VLM recipe while specializing the backbone, language, security imagery, and native reasoning/tool use. Its design is motivated by prior work on visual reasoning, domain-specialized VLMs, text-dense GUI understanding, and positional encoding.

  • General-purpose VLMs: The model follows the LLaVA-style pretrained vision encoder, lightweight projector, and LLM decoder recipe, while using a domain- and language-specific backbone trained from scratch.Unlike general 7B+ backbones, its decoder targets Spanish/LATAM cybersecurity.
  • Architectural and capability distinctions: Periodic NoPE layers, native reasoning and tool-call tokens, and security-focused training distinguish VectraYX from prior released VLMs.The model interleaves NoPE layers with RoPE elsewhere and emits <|think|> and <|tool_call|> tokens as trained capabilities rather than prompt conventions.
  • Visual encoders: SigLIP provides the visual encoder, producing 729 patch tokens without a CLS token for direct insertion into the decoder sequence.The choice follows SigLIP’s reported advantage over CLIP at matched compute and its adoption in recent strong VLMs.
  • Specialized and text-dense imagery: Prior work on Visual CoT, medical VLMs, and GUI understanding motivates grounded reasoning and domain specialization for text-dense cybersecurity imagery.Security-tool screenshots depend on exact tokens such as hex addresses, CVE IDs, opcodes, packet fields, and IOCs; no prior VLM is identified as targeting security-tool screenshots.
  • Cybersecurity and small-model foundations: VectraYX extends small-model and text-only cybersecurity research into a sub-2B, Spanish/LATAM-first multimodal system for offline inference.Its design inherits compute-optimal over-training and complements English-centric text-only security models and benchmarks.

3 Architecture

VectraYX-Vision-1B uses a three-stage LLaVA-style architecture that maps a frozen SigLIP patch grid into a 2048-dimensional VectraYX-1B decoder sequence. Its periodic NoPE design, structured special tokens, and split GGUF export support 2D visual processing, tool-use traces, and llama.cpp deployment.

  • System architecture: The stack combines a frozen visual encoder, trainable projector, and from-scratch language decoder in a three-stage LLaVA-style pipeline.Projected visual tokens replace the <image> placeholder and are consumed by the decoder like text tokens.
  • Visual-token insertion: Visual tokens occupy 729 contiguous ordinary sequence positions, so RoPE imposes absolute left-to-right phases on the intrinsically 2D patch grid while NoPE removes that ordering.The tokenizer expands one <image> placeholder into 729 positions, and projected tokens are scattered into them in place at constant sequence length.
  • Structured tool use: Structured traces use <|think|> blocks and, when appropriate, <|tool_call|> JSON payloads consumed by an MCP server.The model can describe screenshot evidence before answering and invoking tools.
  • Deployment: The implementation exports separate mmproj.gguf and model.gguf artifacts, with visual weights isolated from the decoder and native llama.cpp-compatible decoder tensor conventions.The decoder checkpoint contains no visual weights, and the visual artifact contains the SigLIP tower plus projector.

4 Training Curriculum

The vision extension adds a three-stage Phase 4 curriculum: projector-only alignment, joint visual instruction tuning with text replay, and joint reasoning/tool training with lower learning rates. It starts from the Phase 3 checkpoint, preserves text behavior through replay, and trains on approximately 16M vision-phase tokens.

  • Curriculum initialization: The curriculum begins from the Phase 3 checkpoint, whose language training establishes native <|tool_call|> and <|think|> behavior before image training.Phase 3 used ≈6B tokens and a mixture including tool-SFT, curated cybersecurity, reasoning, Spanish replay, code replay, and math replay.
  • Phase 4a — alignment: Phase 4a trains only the 13.1M-parameter projector while SigLIP and the 1B backbone remain frozen, mapping visual features into the decoder’s token-embedding space.The phase uses a 10^-3 learning rate and one epoch.
  • Phase 4b — instruct: Phase 4b trains the projector and backbone on a 60% vision / 40% text-replay mixture to reduce catastrophic forgetting in tool use and conversation.SigLIP remains frozen; replay includes conversational data, CVE Q&A, and tool traces.
  • Phase 4c — think + tools: Phase 4c trains on 50% vision, 25% tool-replay, and 25% text-replay, adding image-conditioned <|think|> traces and <|tool_call|> completions.The phase uses a 10^-5 learning rate and two epochs to refine reasoning and tool behavior without destabilizing grounding.
  • Executed schedule: ≈16M tokens were processed across Phase 4: 228 steps in 4a, 819 in 4b, and 546 in 4c on 2×A100-40GB DDP.The corresponding runtimes were 24, 43, and 63 minutes, respectively.

5 Multimodal Corpus

The multimodal corpus contains 14,596 QA pairs spanning ten cybersecurity domains, built through bilingual English/Spanish generation and paired with reproducible synthetic or real tool imagery. Its offense/soc-heavy distribution supports realistic workflows, while small LATAM coverage and replay records constrain interpretation of low-resource performance.

  • Corpus composition: 14,596 multimodal QA pairs span ten cybersecurity domains, with records storing messages, image paths, descriptions, template hints, domains, and source provenance.A null image_path identifies text-only replay records.
  • Corpus composition: The corpus uses parallel raw_en and raw_es tracks combining English security sources, Spanish-native content, and machine translations.Pairs are generated per source chunk and retain source_file provenance.
  • Image sources: Synthetic renders cover IDA, Ghidra, Nmap, Volatility, hashcat, Metasploit, and Wireshark, while a smaller real-screenshot set grounds the synthetic distribution.The released renderer enables reproducible image generation without the private screenshot set.
  • Image sources: Exact on-screen tokens—including addresses, opcodes, CVE ids, and ports—serve as ground truth because correctness depends on technical-token matching.This design distinguishes the benchmark from natural-image visual question answering.
  • Generation pipeline: ≈$3.74 USD generated 6,077 factual/technical pairs over 2,029 chunks, using ≈2.77M input and 1.52M output tokens.The reported subset covers arch, asm, debugging, reverse engineering, and ICPC content.
  • Distribution and limitations: 45% of the corpus is offense/soc content, whereas the 120-pair LATAM domain is explicitly under-powered for per-domain claims and intended only to probe regional grounding.Text-only replay records are interleaved according to phase-ratio settings.

6 Evaluation Design

The evaluation combines inherited text/tool benchmarks B1–B5 with new visual-grounding and structured-reasoning axes B6_vision and B7_think. It defines controlled baselines, reproducibility safeguards, and explicitly labels the current single-seed and pending-rerun limitations.

  • Text and tool competence: B1–B5 measure CVE recall, threat classification, command completion, tool selection, and conversational quality, with no significant B4/B5 regression required after vision training.The acceptance criterion compares B4/B5 against the phase-3 backbone while reporting results before and after the vision phase.
  • Visual evaluation: B6_vision evaluates 50 held-out security-tool screenshots using tool_identification and answer_correctness, separating visual recognition from generic cybersecurity knowledge.tool_identification accepts canonical aliases, while answer_correctness checks the key technical fact demonstrated by each image.
  • Structured reasoning: B7_think reuses B6 images to measure think_present and evidence-grounded chain_quality in native <|think|> blocks rather than rewarding long reasoning chains.chain_quality counts distinct on-screen references and saturates at four references; non-native baselines receive think_present=0 when they do not open a think block.
  • Baselines: The comparison set includes Phi-4-multimodal (4.5B), a 1B text-only backbone, and GPT-4o as an explicitly labeled capability ceiling rather than a peer baseline.All comparisons are zero-shot on B6/B7 unless noted, and the text-only ablation tests how much performance is answerable from the question alone.
  • Statistical protocol and limitations: Headline results target N=4 seeds with mean±std and 10K-resample paired bootstrap intervals, but every phase is single-run because phase-2 requires ≈2 weeks on 2×A100-40GB.B6/B7 headline numbers remain pending a remediated phase-4b/4c rerun, and single-seed results are labeled accordingly.

7 Results

All training phases completed, but preliminary vision evaluation found near-zero visual grounding despite valid pipelines, while a loader bug invalidated earlier checkpoints. The model nevertheless demonstrates compact CPU deployment, and the released V0/V1/V2 ablations await remediated vision runs.

  • Training completion: Phase 2 reached 192,000 steps (≈50.3 B tokens), phase 3 reached 20,000 steps (≈6 B tokens), and phase 4 completed all sub-stages with ≈16 M tokens.Phase 4 required ≈2.2 h wall time on 2×A100-40GB.
  • Vision results: 0.08 tool-id and 0.06 answer_correctness: the best valid vision run, v3B, remained near zero after 1,900 steps on 50 synthetic items.All VectraYX-Vision-1B multimodal scores were near zero, indicating fluent Spanish generation without reliable image-content use.
  • Measurement validity: The checkpoint-loader bug failed to strip the llm. prefix, leaving affected inference backbones randomly initialized although training updated the correct parameters.After the fix, emb_norm≈630 was verified before and after runs, making v3B and its SFT continuation valid measurements.
  • Baselines and controls: 0.940 B6 tool-id and 0.940 B7 chain quality: GPT-4o established the frontier ceiling, while the text-only control scored 0.02 B6 tool-id and 0.00 B6 correct.GPT-4o’s chain-quality score used the fallback leading-paragraph regime because it lacks <|think|>; the control was numerically indistinguishable from the with-image 4b-sft step-400 row.
  • Deployment and ablations: ≈2.2 GB F16 plus ≈0.8–0.9 GB for mmproj.gguf: the combined artifact fits comfortably in <4 GB, with ≈150 tokens/s CPU throughput and under 10 s load time.The decoder is ≈0.6–0.7 GB in 4-bit quantization, and the figures represent on-premise deployment without a dedicated accelerator.

8 Discussion: Ablations and the NoPE×Vision Question

The discussion identifies text-only, replay, and NoPE×vision ablations as necessary controls, motivated by a mismatch between 1D RoPE ordering and the visual block’s 2D patch structure. It also flags visual prompt injection and tool-call over-triggering as required future safety evaluations rather than established claims.

  • Required ablations: Text-only B6/B7, replay-% in 4b, and NoPE×vision are necessary controls for testing the paper’s core claims.The text-only baseline tests whether B6 measures vision, replay tests preservation of B4 tool use, and NoPE×vision tests the introduced architectural novelty.
  • NoPE×vision question: 729 visual tokens form a contiguous block whose 2D patches receive arbitrary 1D RoPE ordering on RoPE layers, potentially injecting noise into visual self-attention.NoPE layers avoid imposing this sequence order, creating the central NoPE×vision question.
  • NoPE×vision question: A clean NoPE×visual-block measurement, including a negative result, would inform future VLMs built from NoPE-family backbones.The interaction is expected to recur as NoPE-based text decoders are adapted into VLMs through standard visual-token injection.
  • Safety limitations: Visual prompt injection and over-triggering tool calls are required future evaluations because the model consumes attacker-controlled images and emits tool calls.The paper explicitly scopes these as future work and makes no safety claim here.
  • Dual-use posture: ≈45% of the corpus is offense+SOC, making the model’s dual-use implications an issue requiring an explicit release position.The stated release posture emphasizes interpretive analysis rather than exploit-code synthesis.

9 Limitations

The study reports substantial limitations spanning evaluation validity, data realism and translation, training and modeling choices, deployment safety, reproducibility, and independence. Several safeguards and ablations remain incomplete, including checkpoint correction, expert auditing, frozen-versus-unfrozen encoder comparison, multi-seed evaluation, and external replication.

  • Evaluation validity: An early checkpoint-loader bug left the LLM backbone randomly initialized during phase-4b/4c evaluation, although all four training phases completed without divergence.The bug affected v1/v2 runs at 200–819 steps because the loader failed to strip the llm. prefix; the intact-backbone embedding norm was ≈630 versus ≈164.
  • Evaluation validity: The largely LLM-generated QA corpus risks answer leakage and factual ground-truth errors, and no ≥200-item expert audit with inter-annotator agreement has yet been completed.A text-only-backbone baseline is mandatory to detect leakage-driven B6 inflation.
  • Data limitations: Synthetic renders may not transfer to real cybersecurity screenshots, while machine translation may distort Spanish security terminology and the LATAM domain has only 120 pairs.Real interfaces contain chrome, fonts, themes, and layout noise absent from many renders; the real-screenshot gap is unquantified.
  • Training and modeling: Supervised fine-tuning without preference optimization leaves helpfulness, refusal calibration, and reasoning quality unconstrained, while imitation-learned <|think|> chains may be unfaithful.The model uses no RLHF or preference tuning.
  • Training and modeling: A frozen SigLIP may cap visual grounding for out-of-distribution security imagery; the current system also supports only single-image, short single-turn examples.The frozen-versus-unfrozen ablation is planned but unrun, and multi-image reasoning and long analyst sessions remain out of scope.
  • Deployment and reproducibility: Safety risks, single-seed uncertainty, and author-controlled benchmark design remain unresolved, including unmeasured visual prompt injection and tool-call over-triggering.Every reported number is single-run; the observed B5 0.62–0.69 band comes from per-checkpoint trajectories, while external replication and independent evaluation are still needed.

10 Conclusion

The conclusion presents VectraYX-Vision-1B as a sub-2B Spanish/LATAM cybersecurity VLM with native reasoning, tool use, and offline llama.cpp deployment. Its end-to-end pipeline is functional, but valid phase-4 checkpoints produce near-zero B6/B7 scores, motivating further training, audits, and multimodal evaluation.

  • Contributions: VectraYX-Vision-1B combines security-screenshot reasoning, native <|tool_call|> MCP invocation, and llama.cpp export for fully offline, air-gapped deployment.The system is described as a sub-2B Spanish/LATAM cybersecurity vision–language model with a clean GGUF-splittable architecture.
  • Results: Near-zero B6/B7 scores from the valid phase-4 checkpoints show that the functional pipeline has not yet delivered effective multimodal performance.The reported checkpoints are v3B step 1,900 and a 4b-sft continuation at step 400.
  • Future work: Future work includes multi-image and multi-turn sessions, visual prompt-injection and tool-call-over-triggering safety audits, faithful-reasoning preference optimization, and a learned-2D NoPE×vision study.The learned-2D variant is intended to separate NoPE effects from explicit spatial signals on the visual block.
Loading 2608.08477v1…