Joint Video and Text Parsing for Understanding Events and Answering Queries

1.1 Motivation and objective

• Video understanding seeks to recognize objects, scenes, and events, but low resolution, deformation, occlusion, and hidden intentions or goals make video-only interpretation challenging.Accompanying text ranges from screenplays with substantial overlap to surveillance descriptions with partial overlap and news captions with less overlap.Source: §1.1 · pp. 1–2
• A parse graph represents entities and their relations, composing spatial structure, temporal actions and events, and causal relations between events and object statuses.This explicit semantic representation is proposed as an alternative to bag-of-words approaches and supports text generation and query answering.Source: §1.1 · p. 2
• Joint parsing must match corresponding video and text entities and relations when they overlap, including cases with multiple possible matches.The text may identify a person by name while video detects that person visually.Source: §1.1 · p. 2
• When video and text overlap little, joint parsing must infer additional information to form a coherent interpretation and resolve conflicts caused by parsing errors or inaccurate input.The section gives waiting at a food truck and buying food as an example of connecting information across modalities.Source: §1.1 · p. 2

1.2 Overview of our work

• The joint parsing objective is to optimize the posterior probability of the joint, video, and text parse graphs given the input video and text.The framework focuses on surveillance videos with narrative descriptions but can be extended to other video-text data.Source: §1.2 · p. 2
• The system has a video parser, a text semantic parser, and a joint inference module that combines their parse graphs under conditional probabilities and an S/T/C-AOG prior.The video parser uses the S/T/C-AOG, while the text parser builds on the Stanford Lexicalized Parser to extract entities, events, fluent changes, and relations.Source: §1.2 · pp. 2–3
• Joint inference uses matching to resolve coreference, deduction to fill gaps between modalities, and revision to address conflicts.Matching merges overlapping entities and relations from the video and text parses.Source: §1.2 · pp. 2–3
• In a low-overlap example, joint inference links people waiting in a courtyard to a food truck and infers event types and resulting fluent changes.The example contrasts video that does not recognize the people’s target with text that mentions the food truck.Source: §1.2 · p. 3
• The S/T/C-AOG encodes hierarchical compositions of objects, scenes, and events and guides both video parsing and joint inference.Its spatial, temporal, and causal components model object and scene decompositions, event decompositions, and causal changes in fluents; parse graphs select configurations from this structure.Source: §1.2 · p. 3

1.3 Evaluation

• The evaluation compares joint parses with ground-truth parses using precision and recall, and separately measures accuracy on who, what, when, where, and why queries.The ground truth is constructed by merging parses of human descriptions; query answers are compared with answers from human subjects.Source: §1.3 · p. 3
• Joint parse graphs provide more accurate and comprehensive scene interpretations than video or text parse graphs, and additional text improves them with diminishing returns.The evaluation uses two surveillance datasets and text descriptions supplied at three levels of detail.Source: §1.3 · p. 3
• The illustrated examples contrast a parking-lot scene with significant video-text overlap and a courtyard scene with no overlap.The figures show video, text, and joint parse graphs for each scene.Source: Figure 3 · p. 4, Figure 4 · p. 4

1.4.1 Bag-of-words based approaches

• Bag-of-words methods represent text and visual content as unordered sets, omitting compositional structure and typically supporting coarse tasks such as classification or retrieval.The paper instead aims to produce a full interpretation with spatial, temporal, and causal parsing.Source: §1.4.1 · p. 5

1.4.2 Parsing based approaches

• The authors build on spatial, temporal, and causal image/video parsing approaches,,,,,,,,,,,,,,,,,, and semantic text parsing methods,,.Text semantic parsing is typically built on a syntactic parser such as the Stanford Lexicalized Parser.Source: §1.4.2 · p. 5
• Hobbs and Mulkar-Mehta’s preliminary news-broadcast framework is identified as the closest related work; this system processes raw video and text, models uncertainty probabilistically, and is evaluated experimentally.Liu et al. focus on spatial relations, whereas this work studies spatial-temporal-causal parsing of objects, scenes, and events.Source: §1.4.2 · p. 5

• The work is presented as the first to jointly parse video and text for more accurate and comprehensive results, beyond traditional bag-of-words approaches.Source: §1.5 · p. 5
• The joint S/T/C-AOG crosses spatial, temporal, and causal dimensions, enabling information to propagate between them during parsing.Source: §1.5 · p. 5
• The paper proposes parse-graph comparison and storyline query-answering evaluation, including who, what, when, where, and why questions beyond classification or detection.Source: §1.5 · pp. 5–6

• Sections 2–5 introduce the knowledge representations, video and text parsing approaches, and joint inference module.Source: §1.6 · p. 6
• Section 6 presents natural-language query answering using joint parsing.Source: §1.6 · p. 6
• Sections 7–8 present the experiments and conclude the paper.Source: §1.6 · p. 6

2.1 Ontology

• The ontology taxonomizes objects, events, and changing fluents, with relation taxonomies supporting semantic-distance computation.Objects include physical entities, events include actions and activities, and fluents are object attributes that can change over time.Source: §2.1 · p. 6

2.2 Knowledge representation by And-Or graphs

• The S/T/C-AOG extends constituency grammar with And-node decompositions and relations, and Or-node alternative configurations, to model objects, scenes, events, and perceptual causal effects.Its energy terms define a probabilistic model of parse graphs.Source: §2.2 · p. 6

2.2.1 Spatial And-Or graph

• The S-AOG represents spatial object and scene compositions with And-nodes, alternatives with Or-nodes, and indivisible objects with leaves.For example, a table can be composed of a top and legs, while its style can vary.Source: §2.2.1 · p. 7

2.2.2 Temporal And-Or graph

• The T-AOG decomposes events into temporally ordered sub-events and atomic actions defined by initiator and target properties and interactions.A buying event, for instance, may involve waiting followed by ordering, or ordering without waiting.Source: §2.2.2 · p. 7

2.2.3 Causal And-Or graph

• The C-AOG represents event and fluent leaves, composing conditions and sub-events into causes while allowing alternative explanations for a fluent change.A door may open after someone unlocks and pushes it, or through an alternative cause such as an automatic mechanism.Source: §2.2.3 · p. 7

2.2.4 Joint S/T/C-AOG

• The joint graph connects temporal actions to spatial agents, patients, and locations, and causal events and fluents, enabling information propagation across graph types.It also supports more coherent interpretations for queries about who, what, when, where, and why.Source: §2.2.4 · pp. 7, 9

2.3 Parse graphs

• Parse graphs realize AOG choices as labeled entity-relation graphs with spatial and temporal annotations, allow incomplete interpretations, and support text generation, querying, and RDF representation.The framework infers parse graphs by maximizing posterior probability.Source: §2.3 · p. 9

2.4 Probabilistic Modeling

• Or-node configuration and And-node relation energies define parse-graph probabilities; missing parts incur a penalty, and the normalized prior contributes a factor to the joint posterior.The total energy includes spatial, temporal, causal, and cross-AOG relation terms.Source: §2.4 · pp. 9–10

3 VIDEO PARSING

• The video parser optimizes an energy combining the S/T/C-AOG prior and video likelihood, retaining low-energy candidate parse graphs because video parsing is uncertain.High-energy video parses are pruned because, under the stated superset assumption, they would likely yield low joint posterior probability.Source: §3 · p. 10

3.1 Spatial parsing

• Spatial parsing composes frame-level visual entities into candidate objects under the S-AOG, prunes high-energy compositions, and samples spatial parse graphs using Metropolis-Hastings.The sampler adds compatible candidate objects or randomly prunes the current graph.Source: §3.1 · p. 10

3.2 Temporal parsing

• Temporal parsing processes frames sequentially with an Earley-style state set, using prediction, scanning, and completion while pruning high-energy derivations.After all frames, the state sets yield candidate parses of the whole video.Source: §3.2 · p. 10

3.3 Causal parsing

• Causal parsing uses events before each detected fluent change to parse the relevant C-AOG subgraph, then combines spatial, temporal, and causal graphs into a video parse graph.Figure 10 illustrates a candidate parse graph for the example video.Source: §3.3 · p. 10, Figure 10 · p. 11

4 TEXT PARSING

• The text parser produces one parse graph under an unambiguous-text assumption and can also produce multiple probabilistic parses when text is ambiguous.It targets narrative-event entities, fluent changes, and spatial, temporal, functional, and causal relations.Source: §4 · p. 11

4.1 Text Filtering

• Text filtering identifies names, times, organizations, and locations, and groups compound noun phrases such as “food truck” into single words.The described implementation uses named-entity recognition and a noun-phrase chunker.Source: §4.1 · p. 11

4.2 Part-of-Speech Tagging and Dependency Inference

• POS tagging and dependency parsing label grammatical categories and identify word-to-word relations, providing cues for extracting information from text.The described implementation uses the Stanford Lexicalized Parser and Penn Treebank tags.Source: §4.2 · p. 11

4.3 Dependencies Filtering

• Dependencies are mapped to ontology entity types and top-level supertypes; combining these types with dependency context helps distinguish semantic relations such as an agent-event link from an occupation attribute.POS tags can disambiguate “open” as a fluent state or an action.Source: §4.3 · pp. 11–12

4.4 Parse Graph Generation

• An attribute grammar parses mapped dependencies using their context to infer semantic relations, then propagates attribute values upward to generate the text parse graph.The grammar also extracts spatial, temporal, causal, and functional relations from function words; semantic context helps resolve words with multiple meanings.Source: §4.4 · pp. 12–13

5.1 Challenges in Joint Inference

• Joint inference must resolve coreference despite differing entity types or annotations, add useful missing information, and detect and correct conflicts.For example, the same food truck may be labeled “Vehicle” in the video parse and “Food Truck” in the text parse; “Has Food” can be deduced from buying food.Source: §5.1 · pp. 13–14
• Matching merges corresponding subgraphs, deduction adds implicit information, and revision resolves conflicts between the video and text parses.The inference procedure iteratively changes the graph using these three operator types to maximize the joint posterior probability.Source: §5.1 · pp. 14–15, §5.3 · p. 16

5.2 Relating the three parse graphs

• The conditional probability relates the video and text parses to the joint parse, modeling their generation without assuming the two parses are independent.For tractable inference, the model assumes each entity’s or relation’s inclusion and change is independent of those of other entities and relations.Source: §5.2 · p. 15
• The annotation penalty combines semantic, temporal, and spatial distances with heuristic weights that could instead be learned from annotated data.Semantic distance is large for disjoint types; temporal and spatial distances use annotation time differences and Euclidean distance, respectively.Source: §5.2 · pp. 15–16

5.3 Joint inference algorithm

• Starting from the union of the input parses, the algorithm applies matching, revision, then deduction with follow-up matching, choosing changes by joint posterior probability.For surveillance parses with a few tens of nodes and edges, it uses depth-first search with pruning; it selects the highest-posterior parse triple across candidate video parses.Source: §5.3 · p. 16
• Under the open-world assumption, deduction is canceled when the entropy of mutually exclusive candidate subgraphs exceeds a threshold.This leaves the system agnostic when several inferred possibilities remain similarly plausible.Source: §5.3 · p. 16

6.1 Text query parsing

• Natural-language questions are parsed into SPARQL queries, with wh-words specifying the query objective and the remaining text defining its conditions.The generated query is evaluated against the joint parse graph, and the resulting values are processed into answers.Source: §6.1 · p. 17

6.2 Computing answer probabilities from multiple interpretations

• Across multiple interpretations, answer probabilities combine posterior probabilities of parses that entail each answer, allowing competing answers to be ranked.Answers from different interpretations are matched by semantic types and spatial-temporal annotations before their probabilities are combined.Source: §6.2 · p. 17

6.3 Text query GUI tool

• The GUI returns query results with relevant video segments and, for event answers, generated text descriptions; responses typically take a few seconds.The displayed video segment includes overlays around the event’s agent and patient.Source: §6.3 · pp. 17–18

7.1 Datasets

• The hallway and courtyard surveillance datasets support controlled studies of how text detail and overlap with video affect joint parsing.The videos were divided into fifteen clips per dataset; five subjects described them at three detail levels, including information inferred but not directly visible.Source: §7.1 · p. 18, Figure 20 · p. 18

7.2 Qualitative results

• Video and text parses share prominent events but contribute different details, while joint inference can deduce information absent from both.Video parses often include visible low-level details; text parses can add inferred information or visible content missed by the video parser.Source: §7.2 · pp. 18–19

7.2.1 Text generation

• Joint parses can be verbalized for non-experts, and a larger share of their content comes from text parsing as input descriptions grow.A sentence planner selects content and creates functional descriptions; a sentence realizer uses HPSG to produce sentences.Source: §7.2.1 · p. 19

7.3.1 Comparison against ground truth

• Joint parse graphs have higher recall and F-score than same-level video and text parses, with level-1 text already near full-description performance.All precision scores are close to 1; adding more text further improves joint parses, but with diminishing returns.Source: Figure 21 · p. 20, §7.3.1 · p. 20
• Courtyard recall is lower amid complex events and divergent descriptions; stricter consensus ground truth raises recall, often above 0.9.At the highest consensus, most parse graphs exceed 0.9 recall; hallway video parses are the exception without large recall improvement.Source: §7.3.1 · p. 21
• Increasing deduction first boosts recall without reducing precision, but further deduction sharply lowers precision and may add only false information.The reported behavior comes from varying the deduction degree on a subset of courtyard videos containing complex events.Source: Figure 23 · p. 21, §7.3.1 · p. 21

7.3.2 Evaluation based on query answering

• Joint-parse query answering clearly outperforms video- and text-parse answering, with the largest advantage on why queries and a large advantage on where queries.The authors attribute the why-query advantage to causal relations often being inferred in joint inference, and the where-query advantage to deduced location information.Source: Figure 24 · p. 22, §7.3.2 · pp. 22–23

8 CONCLUSION AND DISCUSSION

• The framework represents spatial, temporal, and causal information in joint video-text parse graphs based on an S/T/C-AOG.Its joint inference module matches, deduces, and revises the video and text parse graphs.Source: §8 · p. 23
• The joint parse graph is used to answer natural-language who, what, when, where, and why queries.The paper reports satisfactory empirical evaluation using ground-truth comparison and query-answering accuracy.Source: §8 · p. 23
• Future work targets parser quality and efficiency, automatic AOG learning, and testing on more challenging video-text data beyond surveillance descriptions.The authors identify news reports as an example of data with more varied content, a larger video-text gap, and greater background-knowledge needs.Source: §8 · p. 23