PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Faithful and Efficient Semantic Communication: An Ontological Approach

Yixiao Feng, Yueting Wang, Yining Wang, Han Han, Bo Zhang

arXiv:2608.25422v1cs.IT

TL;DR

Multi-view VQA must handle redundant, ambiguous, and inconsistent scene graphs produced by heterogeneous VLMs. ODSC uses a shared ontology to reduce transmission and align, reconstruct, and verify semantic information at the receiver. It reduces SI size by up to 87.1% while improving answering accuracy by 4.5% over complete scene-graph transmission and MVA by up to 16.0% over SI filtering approaches.

  • Problem

    Heterogeneous VLMs produce multi-view scene graphs that are redundant, ambiguous, and inconsistent, complicating semantic communication for multi-view VQA.

  • Method

    ODSC shares an ontology-based knowledge base defining synonyms, inference rules, and consistency constraints for transmission reduction and receiver-side alignment, recovery, and verification.

  • Results

    Compared with transmitting complete scene graphs, ODSC reduces SI data size by up to 87.1% and improves answering accuracy by 4.5%, while improving MVA by up to 16.0% over SI filtering approaches.

  • Takeaways & Limitations

    ODSC provides faithful and efficient semantic communication for multi-view VQA by using shared ontology knowledge to recover omitted information and verify views.

Abstract

from arXiv · show

In this paper, an ontology-driven semantic communication (ODSC) framework is proposed for multi-view visual question answering (VQA) tasks. In the considered framework, multiple transmitters observe a scene, extract the semantic information (SI) with vision-language models (VLMs), and transmit the scene graphs to a receiver. Due to the completeness, heterogeneity, and uninterpretability of the VLMs, the extracted scene graphs are redundant, ambiguous, and inconsistent. To solve these problems, the transmitters and the receiver share an ontology-based knowledge base that predefines synonyms, inference rules, and consistency constraints. For each transmitter, the proposed ODSC framework removes the partial scene graph that can be inferred based on the inference rules. For the receiver, the proposed framework aligns the SI of different views based on the synonyms and detects the inconsistency among the views based on the constraints. A metric of multi-view VQA accuracy (MVA) is defined to evaluate the proposed framework. Simulation results show that, compared with transmitting the complete scene graphs, the proposed framework reduces the data size of the SI by up to 87.1% while improving the answering accuracy by 4.5%. Moreover, the proposed framework yields up to a 16.0% improvement in terms of the MVA compared with the SI filtering approaches.

I. INTRODUCTION

The paper proposes ODSC for multi-view VQA, where heterogeneous VLMs generate scene graphs that may be redundant, ambiguous, and inconsistent. Shared ontology knowledge enables efficient transmission and receiver-side semantic alignment and verification.

  • System setting: Multi-view VQA uses transmitters observing a common scene from different views and extracting scene graphs with heterogeneous VLMs.The scene graphs are transmitted to a receiver for answering questions about the scene.
  • Proposed framework: ODSC shares an ontology-based knowledge base defining synonyms, inference rules, and consistency constraints between transmitters and receiver.Transmitters remove inferable partial scene graphs, and the receiver can recover the removed information.
  • Proposed framework: At the receiver, ODSC aligns expressions across views, reconstructs removed scene-graph information, and verifies cross-view consistency before answering questions.The receiver uses the shared ontology for disambiguation, recovery, and conflict detection.
  • Semantic information extraction: Scene graphs encode object categories, attributes, and inter-object relations as semantic subject–predicate–object triples.Each transmitter extracts objects and their bounding boxes, while bounding boxes support cross-view object association but are not part of the SI.
  • Challenges: Complete scene graphs can contain inferable triples, while heterogeneous extractors use different entity expressions and may introduce erroneous triples.These properties create redundancy, ambiguity, and inconsistency in multi-view semantic information.

B. Transmission Model

The transmission model selects reduced scene graphs for transmission over orthogonal wireless resources. Shared ontology provisioning is performed in advance, and transmission delay decreases as the transmitted semantic-information size decreases for a fixed channel gain.

  • Transmitter transmission: Each transmitter selects a subset G′_k of its scene graph G_k for transmission and serializes it with the bounding-box set B_k into a bit stream.The selection method is defined in the ODSC framework, while bounding boxes support receiver-side processing.
  • Wireless links: Transmitters use orthogonal wireless resources, so their links to the receiver are modeled separately without mutual interference.This separates the transmission channels of the individual transmitters.
  • Ontology provisioning: The shared ontology is deployed at transmitters and receiver before transmission, so offline ontology provisioning is excluded from communication overhead.The ontology is assumed to be available in advance at both sides of the system.
  • Transmission delay: For a given channel gain h_k, transmission delay decreases as the transmitted data size L_k(G′_k) decreases.The model relates semantic-information size to transmission delay under the channel formulation.

C. Semantic Information Reconstruction

The receiver deserializes each bit stream and reconstructs each transmitter’s semantic information using the shared ontology. It then jointly verifies overlapping views and uses confidence-based arbitration to resolve category conflicts before answering questions.

  • Semantic reconstruction: The receiver deserializes each received bit stream to obtain the selected scene graph G′_k and bounding-box set B_k.These reconstructed inputs are processed using the shared ontology.
  • Semantic reconstruction: The receiver reconstructs each transmitter’s semantic information through a semantic reconstruction function using the received data and shared ontology T.The resulting reconstructed scene graph represents the transmitter’s view after semantic recovery.
  • Consistency verification: Overlapping views provide evidence for verification, and a joint consistency verifier identifies violations for each transmitter’s view.The verifier examines all views jointly rather than evaluating each view in isolation.
  • Confidence-based arbitration: When views conflict on an object category, the receiver replaces the low-confidence description with the aligned category from the conflicting view having the highest confidence.Violations within a single view are retained as verification outputs.

D. Multi-View VQA Accuracy

The MVA metric evaluates scene-level multi-view VQA by aggregating whether the receiver’s answers match ground truth across questions. The framework aims to minimize transmission delay while maximizing MVA.

  • MVA records whether each receiver answer matches the corresponding ground-truth answer across the scene’s questions.The receiver answers every question using semantic information from all transmitters.
  • Single-view accuracy is also assessed to isolate the effect of each proposed mechanism on individual transmitters.
  • Reducing the number of transmitted triples decreases transmission delay, while the stated objective is to maximize MVA.

III. ONTOLOGY-DRIVEN SEMANTIC COMMUNICATION FRAMEWORK

The ODSC framework addresses redundancy, ambiguity, and inconsistency in scene-graph semantic information using a shared ontology. It removes inferable information at transmitters and reconstructs and verifies information at the receiver.

  • ODSC uses a shared ontology-based knowledge base to address redundancy, ambiguity, and inconsistency in semantic information.The knowledge base predefines synonyms, inference rules, and consistency constraints.
  • The framework performs lossless redundancy elimination by removing scene-graph information that the receiver can recover through the shared ontology.

A. Shared Ontology

The shared ontology is an offline-deployed knowledge base containing synonym mappings, inference rules, and consistency constraints. It standardizes expressions, supports semantic recovery, and identifies conflicting information across views.

  • The shared ontology contains synonym mappings, inference rules, and consistency constraints and is deployed before transmission.
  • Synonym mappings convert extractor expressions to unique ontology concepts, enabling consistent interpretation across transmitters.For example, “mug” maps to the ontology concept “cup.”
  • Inference rules derive semantic triples from premises, including inverse, symmetric, relation-hierarchy, and spatial rules.Applying the rules until no new triple appears produces the closure ΓT(X).
  • The same inference rules identify removable triples at transmitters and recover those triples at the receiver.
  • Consistency constraints specify forbidden combinations of categories, attributes, or relations, such as assigning both “open” and “closed” to one object.Violations are treated as semantic extraction errors.

B. Semantic Redundancy Elimination

The transmitter-side semantic filter partitions scene-graph triples, preserves unmapped information, and removes mapped triples only when ontology-based inference can recover them. The ODSC framework then serializes the retained graph for transmission.

  • B. Semantic Redundancy Elimination: Each transmitter applies ODSC to its scene graph and removes only triples recoverable through the ontology’s inference rules.
  • B. Semantic Redundancy Elimination: The filter partitions each graph into mapped and unmapped triples using the shared synonym mappings.
  • B. Semantic Redundancy Elimination: Unmapped triples are transmitted unchanged because the ontology provides no reliable alignment or inference-based recovery for them.
  • B. Semantic Redundancy Elimination: Mapped triples are replaced by ontology counterparts for filtering, while retained triples are transmitted in extractor-produced form.
  • B. Semantic Redundancy Elimination: The removal process tests triples one by one under a recoverability condition based on a depth-bounded closure with maximum inference depth D.D is a design parameter.
  • B. Semantic Redundancy Elimination: The algorithm initializes each graph, orders mapped triples with rule conclusions before premises, removes recoverable triples, and transmits the serialized retained graph.
  • B. Semantic Redundancy Elimination: The selected and original scene graphs preserve the same ontology-supported facts, making redundancy elimination semantically lossless.

C. Semantic Disambiguation and Recovery

The receiver resolves view-specific expression ambiguity through conservative ontology-based synonym alignment, then reconstructs omitted scene-graph triples using shared inference rules. This recovery restores inferable information but cannot correct incorrectly extracted triples.

  • Semantic disambiguation: Synonym mappings express received semantic information using shared ontology concepts before views are compared.Only uniquely matched, manually validated entity concepts are aligned; relations require exact matches.
  • Semantic disambiguation: The receiver conservatively aligns expressions to avoid propagating errors through subsequent inference.Unaligned triples remain in the set U_k, while relation alignment accepts only exact matches.
  • Semantic recovery: The receiver recovers removed triples from mapped triples and received bounding boxes by applying the shared inference rules.The reconstruction function η implements this recovery step.
  • Semantic recovery: Every removed mapped triple is recovered and represented with ontology concepts, but recovery cannot repair incorrectly extracted triples.Without disambiguation, synonym pairs would be incorrectly reported as conflicts during direct comparison.

D. Semantic Consistency Verification

Semantic consistency verification checks reconstructed scene graphs against ontology constraints and compares overlapping views. Conflicts are recorded and arbitrated using agreement-based view confidence, but constraint validation does not identify which conflicting description is erroneous.

  • Verification design: The verifier combines per-view constraint validation with cross-view comparison to detect semantic extraction errors.The reconstructed graphs may contain object, attribute, and relation errors from the VLM extractors.
  • Validation: Validation flags forbidden category, attribute, and relation combinations, including mutually exclusive values and invalid relation endpoints.For example, assigning “wearing” to a tree violates a constraint requiring the subject to be a person.
  • Comparison: Cross-view comparison associates objects using bounding-box IoU and then compares their normalized categories under ontology synonyms and inference rules.Each object is associated with at most one object from another view, and θ sets the association threshold.
  • Conflict arbitration: The receiver defines view confidence as the fraction of associated object pairs judged consistent, without requiring ground truth.For a marked object, the description from the highest-confidence conflicting view is accepted when its confidence exceeds the current view’s confidence.
  • Verification limits: Constraint violations within a single view are retained as verification outputs because the constraints do not determine which conflicting triple is erroneous.Validation identifies violations but does not modify the reconstructed scene graph.

IV. SIMULATION RESULTS AND ANALYSIS

Simulations on the GQA validation set evaluate ODSC for data reduction, answering accuracy, erroneous-object verification, transmission delay, and multi-view VQA accuracy. Across these measures, ODSC reconstructs and verifies semantic information while reducing communication and computation costs.

  • Framework example: The proposed framework reconstructs a common scene from three transmitters by inferring removed triples, mapping synonyms, and handling a mistaken object description.The receiver maps “automobile” to “car” and uses the shared ontology to reconstruct the observed scene and answer questions correctly.
  • Data size: 60.2% of triples are removed from ground-truth scene graphs while all answers remain identical to baseline a).This result isolates ontology-based redundancy removal when the scene graphs are correct.
  • Answering accuracy: 11.7%, 12.7%, and 22.9% average answering-accuracy improvements are obtained over baselines a), b), and c), respectively.The evaluation covers 8,067 questions supported by complete scene graphs, isolating transmission-scheme effects from extraction quality.
  • Verification: 81.3% erroneous-object recall is reached with seven transmitters, increasing from 37.5% with two transmitters.ODSC verification counts conflicts with every view, whereas baseline d) remains below 46% and stops growing beyond four transmitters.
  • Transmission delay: At 10 dB SNR, average transmission delay falls from 26.4 ms to 4.2 ms versus baseline a), with up to 87.1% reduction for one transmitter.ODSC has the lowest transmission delay at every SNR because redundant semantic information is removed.
  • Multi-view VQA accuracy: ODSC MVA rises from 52.8% with one transmitter to 64.8% with seven and is highest for every transmitter count.With five transmitters, ODSC reaches 63.0% MVA, exceeding baseline a)’s 62.0% with seven transmitters.
  • Computation: The framework processes each scene in 5.9 ms using 2.0 × 10^4 logical operations and no neural network.Compared with baseline c), it adds 3.0 ms per scene while achieving a 16.0% MVA improvement; baseline b) requires 21.9 s per scene and 2.0×10^11 floating point operations.

V. CONCLUSION

The ODSC framework uses a shared ontology to reduce and verify semantic information for multi-view VQA, addressing redundancy, ambiguity, and inconsistency from heterogeneous VLMs. Simulations show reduced SI size, improved answering accuracy, and higher MVA than SI filtering approaches.

  • The ODSC framework removes transmitter-side scene-graph information recoverable through shared ontology inference rules.At the receiver, it aligns different views, identifies semantic inconsistencies, and answers questions using received and verified SI.
  • The framework addresses semantic redundancy, ambiguity, and inconsistency in scene graphs extracted by heterogeneous VLMs.
  • 87.1% reduction in SI data size and 4.5% improvement in answering accuracy are achieved versus transmitting complete scene graphs.
  • 16.0% gain in MVA over SI filtering approaches is achieved by the proposed framework.
Loading 2608.25422v1…