Source-linked AI summary
Cost-Aware Vision--Language Model Arbitration for Fabric Structure Recognition A Deployable Multi-Agent System
Chenwei Wang, Haochen Li, Shuk Ching Tang, Misbah Iqbal, Carman Lee, Elif Ozden-Yenigun
TL;DR
Fabric-structure recognition must distinguish visually ambiguous categories, while CNNs and VLMs present different cost and reliability trade-offs. TAHA routes routine inputs through a hierarchical CNN cascade and selectively invokes a taxonomy-constrained VLM arbiter. On the 14-class benchmark, it improves hard-class accuracy over a flat ConvNeXt-Tiny baseline while supporting local CPU inference and lower API expenditure than VLM-only recognition.
Problem
Visually ambiguous fabric categories challenge CNN classifiers, while VLMs cost more per image and can be unstable on specialist domains.
Source: §I · p. 1Method
TAHA uses a hierarchical CNN cascade for routine classification, selectively routes ambiguous inputs to a top-3 taxonomy-constrained VLM arbiter, and distills the cascade for edge deployment.
Source: §I · p. 1Results
+17.6 pp on the four hard classes over flat ConvNeXt-Tiny; TAHA reaches 93.94% Top-1 versus 32.7% for open-14way Qwen2.5-VL.
Source: B. Accuracy versus Cost · p. 4Takeaways & Limitations
The system runs in under 100 ms on commodity CPU for 72% of inputs handled locally, with about 72% lower API expenditure than VLM-only zero-shot.
Source: §VI · p. 6
Abstract
from arXiv · showhide
Recognizing a fabric's structure is a prerequisite for translating textile-specific material information into structured digital form for downstream supply-chain systems. Pure CNN classifiers are cost-efficient but fail on visually ambiguous categories; vision--language models (VLMs) generalize more broadly but cost much more per image and are unstable on specialist domains. We present a multi-agent system in which a CNN cascade handles the easy majority and a VLM is invoked only as a selective arbiter, constrained to a top-3 taxonomy-consistent choice. The fabric taxonomy performs as a constraint for the whole recognition process to increase the accuracy and reduce the VLM calls. Meanwhile, the CNN cascade is distilled to a small parameter size to reduce the inference time and meet the needs of practical deployment. On a newly curated 14-class benchmark, a flat ConvNeXt-Tiny baseline reaches $90.45\,\%$ top-1 and $76.9\,\%$ on the four hardest classes; \method's hierarchical cascade reaches $93.94\,\%$ top-1 and $94.50\,\%$ hard ($+17.6$\,pp). Tightening the VLM trigger from $60\,\%$ to $<\!10\,\%$ cuts API cost by ${\sim}90\,\%$ with no measurable accuracy loss. CPU inference is $\le\!93$\,ms without a VLM call ($9.3$\,ms distilled). Each prediction carries a machine-readable reasoning record, offered as an entry point for future supply-chain documentation.
I. INTRODUCTION
Fabric recognition is difficult in a small, visually ambiguous minority, while VLMs are costly and unstable on specialist domains. TAHA combines a hierarchical CNN cascade with selective, taxonomy-constrained VLM arbitration and a distilled edge model.
I. INTRODUCTION
- The design targets a small ambiguous minority where strong CNNs are unreliable, while keeping routine recognition cheap and compatible with commodity-CPU latency budgets.VLMs cost more per image than on-device CNNs and can be unstable on fine-grained specialist domains.Source: §I · p. 1
- 90.45% overall accuracy but 76.9% on the four hardest categories marks the gap for fine-tuned ConvNeXt-Tiny on the 14-class benchmark.The hard-class scores range from 65.5% for Ribbed Poplin to 83.8% for Plain Weave.Source: §I · p. 1
- TAHA pairs a hierarchical CNN cascade with selective VLM arbitration constrained to a top-3 taxonomy-consistent choice, then distills the cascade to a 5.4-MB MobileNet variant.The VLM is intended to trigger only on genuinely ambiguous inputs.Source: §I · pp. 1–2
- The paper proposes a cost-aware multi-agent architecture and an empirical cost–latency–accuracy analysis across cloud VLM-only, on-device CNN-only, and selective-hybrid configurations.It also presents a reference implementation with a FastAPI service, Docker image, on-device MobileNet, and machine-parsable reasoning chains.Source: §I · p. 2
II. RELATED WORK
The paper situates TAHA among fabric-specific deep learning, cost-aware cascades, and industrial vision systems. Its fabric-recognition context emphasizes texture and periodic micro-structure rather than object shape.
II. RELATED WORK
- TAHA is positioned at the intersection of fabric-specific deep learning, cost-aware cascade-and-defer systems, and industrial computer vision with auditability requirements.Limited-supervision recognition and constrained multimodal inference provide additional technical ingredients for the design.Source: §II · p. 2
A. Fabric structure recognition
- Classical fabric pipelines combined Gabor or LBP descriptors with SVMs, while modern fabric-specific deep learning includes FabricNet.The paper notes that earlier works do not couple CNNs with multi-source evidence or VLMs.Source: A. Fabric structure recognition · p. 2
- Texture and material benchmarks establish periodic micro-structure, rather than object shape, as central to this texture-dominated recognition task.The tension between designed and learned evidence also appears in other texture-dominated sensing modalities.Source: A. Fabric structure recognition · p. 2
B. Cost-aware AI deployment
TAHA adapts cascade-and-defer routing to textile recognition with taxonomy-filtered candidates and confusion-aware triggering. It also distinguishes structure recognition from defect inspection and uses taxonomy-constrained staging for limited, imbalanced data.
B. Cost-aware AI deployment
- Selective VLM arbitration extends cascade-and-defer routing with taxonomy-filtered candidate construction and confusion-aware triggering for textile inputs.The broader routing literature connects abstention signals to selective classification, expert deferral, confidence calibration, and out-of-distribution scoring.Source: B. Cost-aware AI deployment · p. 2
C. Industrial computer vision and quality control
- Textile defect-detection systems target surface defects rather than weave or knit structure, and lack a taxonomy constraint or VLM component.TAHA's reasoning-chain output provides a structured, machine-readable record of the kind envisaged by supply-chain documentation initiatives.Source: C. Industrial computer vision and… · p. 2
D. Recognition under limited and imbalanced supervision
- FabricFlow-14 has 200–417 images per class, and TAHA adopts taxonomy-constrained staging before delegating residual ambiguity to an external arbiter.Prior approaches also include feature augmentation, multi-view aggregation, semi-supervision, generative augmentation, and open-set rejection.Source: D. Recognition under limited and… · p. 2
E. Constrained multimodal inference and on-device reliability
TAHA uses a domain schema to constrain multimodal inference and a confidence router to decide when a local model should defer. Its methodology combines a CNN cascade, selective cloud arbitration, reconciliation, and distillation for edge deployment.
E. Constrained multimodal inference and on-device reliability
- The top-3 taxonomy filter encodes a structured prior over specialist labels, while the confidence router addresses when a local model should decline to answer.The paper frames these as constrained multimodal inference and on-device reliability choices.Source: E. Constrained multimodal inference… · pp. 2–3
III. METHODOLOGY
- The orchestrator comprises a cost-aware CNN cascade, a confidence router for a constrained cloud VLM arbiter, a reconciler, and an on-device distillation step.The reconciler produces a structured, machine-readable decision record.Source: §III · p. 3
A. Cost-Aware Cascade
- The two-stage ConvNeXt-Tiny cascade separates Knit, Woven, and Others before classifying 14 sub-categories, with ONNX export and INT8 quantization for CPU inference.It runs in 93 ms on an Intel Xeon CPU; the backbone has 28.6 M parameters and reaches 90.45% Top-1 on FabricFlow-14.Source: A. Cost-Aware Cascade · p. 3
B. Confidence Router
The confidence router combines dominance, separation, and confusion-pair information to identify uncertain inputs for VLM arbitration. Its threshold separates groups with different CNN accuracy and leaves most inputs for classification without a VLM call.
B. Confidence Router
- c(x) fuses dominance confidence, top-two score separation, and a confusion-pair penalty into one router score.The score is computed for each image.Source: B. Confidence Router · p. 3
- ∼72 % of inputs satisfy c(x)≥0.55 and are classified without any VLM call.Source: B. Confidence Router · p. 3
- The dominance and separation terms capture absolute uncertainty and ambiguity between the top two candidates, respectively.A dominance-only router under-triggers on confident confusion pairs, while a separation-only router over-triggers on uniform distributions without a competitive neighbour.Source: B. Confidence Router · p. 3
- 66.7 % of inputs below the 0.55 threshold are correct versus 96.0 % above it; the weighting maximizes VLM-trigger recall of CNN errors at a fixed 5 % trigger budget.The weighting was selected by grid search on the held-out validation split.Source: B. Confidence Router · p. 3
C. Taxonomy-Filtered VLM Arbiter
The arbiter is called for low-confidence or closely competing candidates and chooses among a taxonomy-filtered top three with textile-specific cues. A reconciler combines or routes the CNN and VLM outputs and records each agent’s contribution.
C. Taxonomy-Filtered VLM Arbiter
- When c(x) < 0.55 or a known confusion pair has margin < 0.15, the orchestrator triggers the VLM arbiter.It receives the image, top-3 taxonomy-consistent candidates, textile-specific visual cues, and a JSON schema requiring one choice and a justification.Source: C. Taxonomy-Filtered VLM Arbiter · p. 3
- Fine-grained ≥10-way VLM zero-shot is unstable, whereas 3-way taxonomy-grounded selection is cheaper and more accurate [23], consistent with broader evaluations [24],.Source: C. Taxonomy-Filtered VLM Arbiter · p. 3
D. Reconciliation and Output
- On top-1 agreement, the reconciler blends scores 0.7 PCNN + 0.3 PVLM; it accepts a VLM override only for the CNN’s second choice with self-reported confidence >0.8.Other conflicts are flagged for human review, and each decision produces a machine-readable reasoning chain listing each agent’s action, observation, and contribution.Source: D. Reconciliation and Output · p. 3
E. On-Device Distillation
Distillation produces a smaller MobileNet-V3-Large student with faster measured CPU inference than its ConvNeXt-Tiny teacher. The evaluation is on FabricFlow-14, a 14-class benchmark with a stratified test set.
E. On-Device Distillation
- 2.1× measured CPU end-to-end speed-up (19.8→9.3 ms) and 6.8× parameter compression result from distilling the 28.6 M-param ConvNeXt-Tiny teacher into a 4.2-M MobileNet-V3-Large student.Stage-1 ternary retention is 97.00 →95.46 (−1.54 pp); Stage-2 Knit is 92.96→94.13 (+1.17 pp); Stage-2 Woven is 88.63 →87.65 (−0.98 pp).Source: E. On-Device Distillation · p. 3
- The deployment architecture routes a factory-floor camera through an on-device CNN cascade and confidence router to a constrained 3-way cloud VLM arbiter, with each classification recorded as a JSON reasoning chain.Source: Figure 1 · p. 4
A. Dataset and Baselines
- FabricFlow-14 contains 5,719 studio images across 14 fabric-structure classes, with a stratified-by-class-and-source test set of 1,550 images.The reported comparisons include a flat ConvNeXt-Tiny baseline, zero-shot VLMs, and TAHA variants.Source: A. Dataset and Baselines · p. 4, §IV · p. 4
B. Accuracy versus Cost
TAHA’s hierarchical architecture outperforms the flat-CNN baseline, with its largest gains on the four hardest classes. The reported comparisons also frame deployment cost in terms of VLM use and latency.
B. Accuracy versus Cost
- +3.49 pp Top-1 and +17.6 pp on the four hard classes over flat ConvNeXt-Tiny, TAHA’s reported accuracy gains.Table I covers the FabricFlow-14 test set and defines “Hard” as the mean accuracy on four named classes.Source: Table I · p. 4, B. Accuracy versus Cost · p. 4
- Optional Fusion and VLM layers extend the cascade’s Top-1 without degrading it; their Top-1 gain is only +0.06 pp on studio-quality FabricFlow-14.The paper attributes the architecture’s larger overall gains to the taxonomy-aware hierarchy rather than a more powerful single classifier.Source: B. Accuracy versus Cost · p. 4
- Latency is reported as the median over 250 samples; Table IV compares TAHA-noVLM with the CNN cascade across six input regimes using added rescued errors and per-error cost delta.The Table II caption says non-VLM rows use a single Intel Xeon CPU without a GPU and that the VLM path includes network and Claude inference round-trip.Source: Table II · p. 5, Table IV · p. 5
- Table III reports cloud-VLM call rate rather than absolute price, which depends on provider and throughput.The caption defines call rate as the share of inputs sent to the VLM and notes the VLM-only and edge-only rows use a different sample.Source: Table III · p. 5
C. Latency and On-Device Feasibility
The distilled student supports local inference, while ambiguous cases are forwarded asynchronously to a cloud VLM. Deployment operating points trade VLM call volume against hard-class accuracy and can be chosen to match operating constraints.
C. Latency and On-Device Feasibility
- The distilled student runs at ~18 ms/image without internet for at least 72% of inputs; ambiguous cases go asynchronously to a cloud VLM.The paper presents this hybrid as matching the bandwidth and latency constraints of typical Industry 4.0 production lines.Source: C. Latency and On-Device Feasibility · p. 5
D. Deployment Efficiency and Operating Points
- At 1 M images/month, TAHA delivers the highest accuracy at ~29% of the cloud-VLM call volume of an all-VLM deployment.The deployment comparison reports call volume rather than absolute price because marginal serving cost depends on the selected provider, hardware, and throughput.Source: D. Deployment Efficiency and Operating… · p. 5
- Tightening the trigger from 0.55 to 0.40 reduces cloud-VLM calls from 28% to under 10% with no measurable hard-class accuracy loss.Relaxing the threshold to 0.70 pushes calls above 50% and recovers about 0.5 pp on the hardest classes.Source: D. Deployment Efficiency and Operating… · p. 5
- The call-rate/accuracy curve lets deployers choose an operating point for their latency, throughput, and connectivity constraints.The reported trade-off uses the share of inputs escalated to the VLM rather than provider-specific absolute cost.Source: D. Deployment Efficiency and Operating… · p. 5
E. Conditional Robustness Under Input Degradation
The multi-agent fusion’s value depends on the input corruption: it helps under some degradations and underperforms the cascade under others. The paper also describes a containerized implementation with review and provider-configuration features.
E. Conditional Robustness Under Input Degradation
- A six-corruption sweep on the same n = 1,550 test set found that fusion’s marginal value depends on the corruption family.The sweep follows standard corruption-robustness protocols.Source: E. Conditional Robustness Under Input… · p. 5
- Fusion adds net value under Gaussian noise, reduced brightness, and partial occlusion, but underperforms the cascade under motion blur and heavy JPEG compression.The paper attributes the latter pattern to spurious high-frequency directional energy misread by Gabor/FFT heuristics and recommends gating fusion by camera conditions.Source: E. Conditional Robustness Under Input… · p. 5
V. DEPLOYABLE REFERENCE IMPLEMENTATION
- The reference implementation includes FastAPI endpoints, a Docker bundle of at most 800 MB requiring no GPU, a Gradio review UI, and a swappable VLM-provider adapter.The UI visualizes the reasoning chain and offers a toggle for VLM arbitration; the adapter permits provider changes without modifying the rest of the pipeline.Source: §V · p. 5
VI. CONCLUSION
The paper concludes with an industrially deployable, selective VLM-as-Arbiter system for fabric structure recognition. It reports hard-class accuracy, cost, local CPU latency, and structured reasoning records as deployment-relevant outcomes.
VI. CONCLUSION
- The system achieves higher hard-class accuracy than VLM-only zero-shot at ~72% lower API expenditure.It runs in under 100 ms on commodity CPU for the 72% of inputs handled locally and offers a 5.4-MB distilled student for edge deployment.Source: §VI · pp. 5–6
- Every classification yields a structured reasoning chain that could feed future supply-chain documentation, although no specific compliance regime was tested.The code, Docker image, and a curated fraction of the FabricFlow-14 test set are slated for release on publication.Source: §VI · p. 6