PaperlayerView exact revision on arXiv

Source-linked AI summary

ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation

Yangyi Chen, Xingyao Wang, Manling Li, Derek Hoiem, Heng Ji

arXiv:2311.13258v1cs.CVcs.CLcs.LG

TL;DR

VLMs have limited ability to capture visual structure, including relations among objects. ViStruct uses code-based representations and curriculum learning to train across structural levels, with a dataset suite built partly from captions. The authors report consistent improvements on visual structure-prediction tasks.

  • Problem

    Existing VLMs have limitations in capturing visual structural information, including relations among objects.

    Source: §1 · p. 1

  • Method

    ViStruct represents visual structures in code and uses a curriculum pyramid to train VLMs from foundational concepts toward complex events.

    Source: §1 · p. 2

  • Results

    ViStruct consistently outperforms baselines on three structure-prediction tasks, with significant and consistent improvements reported across structural levels.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The framework is presented as a step toward multi-granularity semantic parsing of visual data.

    Source: §6 · p. 8

    The base model’s constrained capability makes direct code generation infeasible and reduces downstream performance.

    Source: Limitations and Future Work · p. 9

Abstract

from arXiv · show

State-of-the-art vision-language models (VLMs) still have limited performance in structural knowledge extraction, such as relations between objects. In this work, we present ViStruct, a training framework to learn VLMs for effective visual structural knowledge extraction. Two novel designs are incorporated. First, we propose to leverage the inherent structure of programming language to depict visual structural information. This approach enables explicit and consistent representation of visual structural information of multiple granularities, such as concepts, relations, and events, in a well-organized structured format. Second, we introduce curriculum-based learning for VLMs to progressively comprehend visual structures, from fundamental visual concepts to intricate event structures. Our intuition is that lower-level knowledge may contribute to complex visual structure understanding. Furthermore, we compile and release a collection of datasets tailored for visual structural knowledge extraction. We adopt a weakly-supervised approach to directly generate visual event structures from captions for ViStruct training, capitalizing on abundant image-caption pairs from the web. In experiments, we evaluate ViStruct on visual structure prediction tasks, demonstrating its effectiveness in improving the understanding of visual structures. The code is public at \url{https://github.com/Yangyi-Chen/vi-struct}.

1 Introduction

ViStruct addresses VLMs’ limited capture of visual structures with unified code-vision representations and curriculum-guided learning. It also introduces a multi-level dataset suite and reports gains on structure-prediction tasks.

1 Introduction

  • Code-vision representations encode visual concepts, attributes, relations, and events in a structured format, addressing VLMs’ difficulty capturing visual structure.The framework aims to capture structures at each granularity, rather than relying on representations focused on primary objects.Source: §1 · pp. 1–2
  • A curriculum pyramid progressively trains VLMs from foundational concepts toward complex visual events, with replay buffers preserving lower-level knowledge.The approach targets comprehensive understanding across granularities, unlike pretraining on a mixture of tasks.Source: §1 · p. 2
  • ViStruct Suite combines existing labeled datasets with large-scale unlabeled image-caption pairs and uses weak supervision to create visual event labels from captions.Concepts from different datasets are aligned to WordNet and FrameNet synsets with minimal manual assistance.Source: §1 · p. 2
  • ViStruct consistently outperforms baselines on visual relation detection, scene graph classification, and situation recognition.Code representation and curriculum learning also show consistent gains on downstream tasks and zero-shot visual structure prediction.Source: §1 · p. 2
  • The framework targets multi-granularity visual parsing, capturing hierarchy among visual knowledge levels for compositional understanding of overall structure.The paper presents this as a first approach to multi-granularity visual structured knowledge extraction via code-vision representation learning.Source: §1 · p. 2

2 Related Work

Related work traces VLMs from detector-dependent systems toward unified pretraining, while noting persistent shortcomings in structural understanding. It also distinguishes ViStruct’s task-level, multimodal curriculum from prior curriculum learning focused on ordering examples for a single task.

2.1 Vision-Language Models (VLMs) for Visual Knowledge Extraction

  • VLMs progressed from detector-based visual representations to approaches eliminating detectors and then unified pretraining, but concerns remain about object-focused representations and structural extraction.Prior programming-language approaches to vision also relied on pretrained object detectors and lacked perception ability.Source: §2.1 · pp. 2–3

2.2 Curriculum Learning

  • Prior curriculum learning generally orders examples for a single task, whereas ViStruct proposes task-level, multifaceted, multimodal curriculum learning for visual structural knowledge.Its curriculum pyramid has five knowledge-acquisition levels, progressing from basic to advanced structures represented in programming language.Source: §2.2 · p. 3, Figure 2 · p. 3

3 ViStruct

ViStruct generates multi-granularity visual structures from images using code-vision representations, curriculum-guided training, a dedicated dataset suite, and continued pretraining (p13).

3.1 Code-Vision Representation

  • Python class definitions unify visual concepts, relations, and events with their semantic roles and definitions, drawing on WordNet and FrameNet.Concept classes include locations and attributes; events include visual concepts as participants.Source: §3.1 · pp. 3–4

3.2 Curriculum-Based Training for Knowledge Extraction with Replay Buffer

  • Training progresses through concept recognition, object grounding, attribute detection, object relations, and events, while replayed earlier-stage samples help prevent forgetting.The replay buffer stores randomly sampled data from earlier curriculum stages.Source: §3.2 · pp. 4–6
  • The curriculum can be extended with new structural information at different levels, enabled by flexible code-vision representations.The authors state this adaptability is intended to improve overall multimodal knowledge acquisition.Source: §3.2 · p. 5

3.3 ViStruct Suite Construction

  • ViStruct Suite combines labeled and unlabeled datasets, including 10,450 object categories, 80.6M grounding categories, and 14.6M attribute annotations.Visual event training data is weakly generated by parsing large-scale image-caption pairs and aligning event and argument definitions with FrameNet.Source: §3.3 · p. 6

3.4 Model Training

  • ViStruct continues pretraining a VLM, using a warm-up stage and optimizing loss only on masked tokens; the final two curriculum levels receive three training epochs each.The initial three levels receive one epoch each, and OFA-base is the backbone used in this work.Source: §3.4 · p. 6

4 Experiment

ViStruct is evaluated on visual relation detection, scene graph classification, and situation recognition, outperforming baselines across these structure-prediction tasks. Results also attribute gains to diverse training relations, broad concept coverage, curriculum learning, and caption-derived event structures.

4 Experiment

  • Evaluation covers relation detection, scene graph classification, and situation recognition.The experiments use Visual Genome for the first two tasks and SWiG for situation recognition.Source: §4 · p. 6, §4.2 · p. 7, §4.3 · p. 7

4.1 Visual Relation Detection

  • ViStruct improves recall for both common and long-tail relations, with diverse relation exposure helping prevent overfitting to frequent types.The relation-detection comparison is reported in Table 3.Source: §4.1 · p. 7, Table 3 · p. 7

4.2 Scene Graph Classification

  • ViStruct outperforms baselines by a larger margin than in relation detection, consistent with its concept coverage and curriculum matching scene-graph classification needs.The comparison is reported in Table 4.Source: §4.2 · p. 7, Table 4 · p. 7

4.3 Situation Recognition

  • ViStruct improves activity and participant detection, with caption-derived event structures increasing coverage of infrequent events and visual concepts.The paper also reports curriculum-learning gains for situation recognition; results are shown in Table 5.Source: §4.3 · pp. 7–8, Table 5 · p. 8

5 Further Analysis

Curriculum training progressively improves relation-detection metrics, especially for long-tail relations, and code-vision representation yields substantially better zero-shot performance than natural-language training. These results support the framework's curriculum and representation choices for visual structure prediction.

5.1 Curriculum Learning

  • As curriculum training progresses, both evaluation metrics rise and gradually surpass ViStruct-Mix, especially for long-tail relations.The authors interpret this as evidence of curriculum benefits for generalization in low-resource settings.Source: §5.1 · p. 8

5.2 Zero-shot Capability

  • 2.4x better on average, code-vision representation achieves significantly better zero-shot visual relation detection performance than ViStruct-Text.The authors attribute the zero-shot ability to the unified format used in pretraining and downstream tasks.Source: §5.2 · p. 8, Table 6 · p. 8

6 Conclusion

ViStruct represents visual structures at multiple granularities using programming-language structure and uses curriculum learning to guide acquisition across levels. The paper reports significant and consistent improvements on structure-prediction tasks.

6 Conclusion

  • Programming-language structure provides a unified representation for visual structural knowledge at different levels.The framework is presented as a step toward multi-granularity semantic parsing of visual data.Source: §6 · p. 8
  • A curriculum-learning pyramid uses linguistic cues to guide models in acquiring visual structural knowledge at each level.The approach is intended to capture interdependencies among granularities.Source: §6 · pp. 8–9
  • ViStruct yields significant and consistent improvements on structure-prediction tasks across levels.Source: §6 · p. 9

Limitations and Future Work

The framework focuses on programming-language representation for visual structures, but the base model constrains direct code generation and reduces downstream performance. The authors suggest using more capable vision-language models as a potential solution.

Limitations and Future Work

A Adaptation of Downstream Tasks

Downstream evaluations train models on the training dataset, select models using validation-set performance indicators, and evaluate the selected models on the test dataset.

A Adaptation of Downstream Tasks

A.1 Visual Relation Detection

The downstream tasks use masked code prompts to predict relations, object categories, events, and event participants from images, with task-specific inputs and targets.

A.1 Visual Relation Detection

  • Given annotated object boxes and categories, ViStruct predicts the visual relation for each object pair, using a mask prompt and a single-token relation verbalizer.For the horse-and-grass example, the target relation is “walk_on”.Source: §A.1 · p. 15

A.2 Scene Graph Classification

  • Scene graph classification jointly predicts object categories and visual relations from bounding boxes, treating recognition and relation detection as a multitask problem.The object-category prompt includes the annotated location.Source: §A.2 · p. 15

A.3 Situation Recognition

  • Situation recognition separates visual-event prediction from event-role prediction, which uses a ground-truth or predicted event and a mask prompt to predict its participant.For example, the event may be “eat” and the masked “ingestor” argument is predicted as an object.Source: §A.3 · p. 15

B Visual Structure Learning

ViStruct organizes visual structures across semantic levels and teaches them progressively; a warm-up stage introduces code syntax and visual concepts before structural learning.

B Visual Structure Learning

  • Figure 6 depicts focusing optimization as prioritizing generated code’s semantic content while disregarding its syntactic structure.Source: Figure 6 · p. 16
  • The curriculum progresses from visual concepts through grounding, attributes, and relations to event understanding, and the ViStruct Suite assembles datasets spanning these structural levels.The dataset statistics are listed in Table 7.Source: B Visual Structure Learning · pp. 15–16, Table 7 · p. 16

C Warm-up Pretraining

  • Before multimodal structural learning, a one-epoch warm-up combines code data, code-formulated WordNet and FrameNet class definitions, and all ViStruct Suite datasets.The warm-up targets fundamental code syntax structures and visual concepts with their definitions.Source: C Warm-up Pretraining · p. 16

D Focusing Optimization

ViStruct limits loss computation to generated code’s semantic content; its suite combines labeled structural datasets with weakly supervised visual-event data from captions.

D Focusing Optimization

  • Focusing optimization computes loss only on generated code’s semantic content, since syntax is redundant and quickly learned during warm-up.The authors say this directs models toward structural knowledge and accelerates training.Source: D Focusing Optimization · p. 16

E Details of ViStruct Suite

  • The suite uses ImageNet-21K for object recognition, COCO, OpenImages, and Object365 for grounding, LSA for attributes, and OpenImages and Visual Genome for relations.ImageNet-21K retains 10,450 categories with 100 samples each after excluding classes with fewer than 500 samples; labeled datasets use training splits to avoid contamination.Source: E Details of ViStruct Suite · p. 16
  • For visual-event understanding, weakly supervised training data are generated by semantically parsing captions paired with images from COCO, SBU Captions, and CC3M.An off-the-shelf semantic role labeling model performs the parsing, with FrameNet used for event and argument definitions.Source: E Details of ViStruct Suite · p. 16
  • Categories across datasets are aligned to WordNet and FrameNet synsets using NLTK automatic mapping, with manual adjustments when multiple categories map to one synset.Source: E Details of ViStruct Suite · p. 16

Related papers

Loading 2311.13258v1…