PaperlayerView exact revision on arXiv

Source-linked AI summary

AVA-Encoder: Towards Agent-Native Video Representation Learning

Chuyue Li, Jinpeng Yu, Haozhe Wang, Tian Xueyun, Zhijing Zhang, Bingnan Li, Shuqi Gu, Kan Ren, Jiaming Liu, Ruihua Hua

arXiv:2608.12313v1cs.CVcs.CL

TL;DR

Creative agents lack a faithful, agent-operable representation for learning filmmaking knowledge from high-quality human films. AVA-Encoder converts videos into text-centered knowledge graphs, reconstructs them with a fixed decoder, and uses reconstruction feedback for policy and graph refinement. It achieves a 20.7-percentage-point Overall reconstruction improvement over the strongest external baseline, while its pseudo-trained policy uses 74.3% fewer system-prompt tokens than the human-tuned policy.

  • Problem

    Creative agents lack a structured video representation that is both faithful to film content and usable for agentic reasoning and manipulation.

  • Method

    AVA-Encoder maps videos into hierarchical text-centered knowledge graphs with linked assets, reconstructs them through a fixed decoder, and applies textual-gradient policy training and graph refinement.

  • Results

    20.7 percentage points: AVA-Encoder improves Overall reconstruction over the strongest external baseline, while its pseudo-trained policy uses 74.3% fewer system-prompt tokens than the human-tuned policy.

  • Takeaways & Limitations

    The framework provides an agent-native film representation supporting structured learning, graph-based linked editing, and continued self-improvement.

Abstract

from arXiv · show

Creative agents still lack an effective way to learn from high-quality human films, limiting their ability to produce cinematic-grade videos. A key challenge is the absence of a structured video representation that is both faithful to film content and directly usable for agentic reasoning and manipulation. To address the challenge, we propose the Agentic Video Auto-Encoder (AVA-Encoder), a framework for learning agent-native video representations via agentic auto-encoding. AVA-Encoder transforms a video into a knowledge graph (KG) representation and then reconstructs it back into video. Its hierarchy and state nodes store structured text, while a linked asset layer holds generated images, audio, and video. Typed edges preserve the relations between these text descriptions and assets in a form that agents can easily understand, query, and edit. The video reconstruction differences drive a textual-gradient optimization framework, which expresses evaluation feedback as natural-language update directions for Data-Independent Encoding Policy Pseudo-Training in the outer loop and optional Data-Dependent KG Representation Refinement in the test-time inner loop. Extensive experiments show that AVA-Encoder improves by 20.7 percentage points over the strongest external baseline. In the controlled policy-only setting, its pseudo-trained shot-level Agentic Video Encoder policy also outperforms a carefully human-tuned policy while using 74.3% fewer system-prompt tokens. We release the complete AVA-Encoder framework, a reliable agentic video reconstruction benchmark, and the first dataset of high-quality film KG representations.

1. Introduction

AVA-Encoder addresses the gap between complex cinematic content and agent-operable representations by encoding films as structured knowledge graphs and reconstructing them through agentic auto-encoding. Its hierarchical representation and textual-gradient optimization target faithful, editable film understanding, while experiments report gains over baselines and human-tuned policies.

  • Creative agents lack reliable planning across scripts, characters, shots, and audiovisual elements, partly because high-quality complete creation records are scarce.
  • Films combine tightly linked story, character, visual, temporal, and audio information, whereas agents operate through structured text, code, plans, and graphs.
  • An effective representation must be agent-understandable, easy to reason over and edit, and faithful to cinematic information.
  • AVA-Encoder uses a Story–Event–Shot hierarchy, state nodes, linked multimodal assets, and typed edges to preserve cross-modal and cross-level relations.Film-, shot-, and keyframe-level understanding passes higher-level context to finer levels to reduce information loss.
  • AVA-Encoder encodes films into knowledge graphs, reconstructs them, and converts reconstruction errors into signals for shared-policy and input-specific representation refinement.

2. Related Work

Existing video systems support generation, editing, and remixing, but current representations remain poorly suited to learning faithful, editable filmmaking knowledge. AVA-Encoder uses film-centric knowledge graphs to connect agent operation with preserved film structure and linked editing.

  • Recent LLM/VLM-based systems explore video generation, editing, and remixing through planning and tool orchestration.
  • Current systems are limited by foundation models and a shortage of high-quality records of complete video-creation processes.
  • AVA-Encoder turns films into structured graphs recording entities, events, and links, supporting agent learning, self-improvement signals, and linked updates across scripts, characters, and shots.
  • Existing representations span low-level features, textual descriptions, and structured long-video forms such as sparse memories and hierarchical summaries.
  • Low-level and textual formats hinder agent manipulation or lose visual details, while existing graphs target understanding tasks and do not ensure information preservation for visual creation.

3. Task Definition

AVA-Encoder formulates filmmaking as auto-encoding through a text-centered knowledge graph between a video encoder and fixed decoder. Reconstruction residuals support separate shared-policy training and optional input-specific graph refinement.

  • AVA-Encoder reframes filmmaking as auto-encoding over a structured, agent-native intermediate representation.
  • The closed loop maps videos to graphs, reconstructs them, and converts reconstruction residuals into gated policy pseudo-training and graph refinement.
  • The Agentic Video Encoder maps a continuous video into an explicit knowledge graph using film-, shot-, and keyframe-level policies.
  • The bottleneck is a text-centered graph with Story–Event–Shot hierarchy, state descriptions, linked assets, and preserved cross-shot dependencies.
  • A fixed two-stage decoder renders new reference keyframes and shot videos from graph text and linked assets, without storing or passing source frames to generation models.
  • The outer stage learns a shared shot-level policy before deployment, whereas the optional inner stage refines only the current input’s graph; the variables are never updated simultaneously.

4. Method: AVA-Encoder

AVA-Encoder converts videos into text-centered knowledge graphs with linked multimodal assets, then reconstructs them to optimize agent-native representations. Its multi-level encoder, typed graph structure, and dual textual-gradient loops support faithful, editable representations and policy improvement.

  • Agentic Video Encoder: The multi-level encoder processes films, shots, and keyframes coarse-to-fine, passing global context and recurring-entity identities to finer stages.Its policy separates film-level narrative understanding, shot-level dynamics, and keyframe composition while injecting hierarchical and registry context.
  • Knowledge-Graph Representation: The knowledge graph stores Story–Event–Shot hierarchy and Character, Scene, Object, Style, Camera, and Audio states as text, while generated media remains in a linked asset layer.Typed edges preserve hierarchy, temporal order, semantic relations, and links between descriptions and assets.
  • Knowledge-Graph Representation: Typed graph edges support one-hop lookup and linked subgraph editing, allowing local text or asset changes to update connected shots consistently.The representation is designed for direct agent reading, querying, and editing rather than storing raw multimodal data in graph nodes.
  • Framework: AVA-Encoder maps videos into structured knowledge graphs and reconstructs them, using reconstruction quality to refine representations and encoding policies.The framework applies textual-gradient optimization to both shared policies and input-specific graph representations.
  • Dual-Loop Textual-Gradient Evolution: The outer loop pseudo-trains a shared shot-level encoding policy, while the optional inner loop refines an individual input’s KG after the policy is frozen.The two loops update different variables, are not nested, and may be enabled independently.
  • Evaluation: 45.8% Overall reconstruction accuracy was achieved by the pseudo-trained policy versus 44.4% for a human-tuned policy, with 74.3% fewer system-prompt tokens.With optional input-specific refinement, the complete configuration reached 49.0%.

5. Experiments

The experiments evaluate reconstruction fidelity, optimization-stage contributions, graph-based editing, and downstream reuse. AVA-Encoder achieves strong reconstruction results, benefits from both optimization loops and acceptance gates, supports linked edits, and improves downstream generation systems.

  • Experiment Setup: The benchmark evaluates four reconstruction directions using identical test videos and a shared fixed video-generation decoder.The directions are direct Video, direct Keyframe, Video Back-Captioning, and Keyframe Back-Captioning; Overall is their unweighted mean.
  • RQ1: Reconstruction Fidelity: AVA-Encoder outperforms every baseline in all four comparison directions.It improves over the strongest baseline by 21.1 points on Video, 34.2 on KF, 13.9 on V-BC, and 11.6 on KF-BC.
  • RQ2: Contribution of the Two Optimization Stages: The hierarchical policy-only configuration reaches 45.8 Overall versus 27.5 for single-level understanding, while pseudo-training exceeds the human-tuned policy by 1.4 points.The pseudo-trained policy uses 8,052 rather than 31,336 system-prompt tokens, a 74.3% reduction.
  • RQ2: Contribution of the Two Optimization Stages: 49.0 Overall with both optimization loops exceeds 45.8 with policy pseudo-training only, 45.4 with KG refinement only, and 42.4 with neither.Together, the two loops add 6.6 points, or 15.6% relative, over removing both.
  • RQ2: Contribution of the Two Optimization Stages: Acceptance gates raise performance from 43.5 without gates to 49.0 with both gates.The anti-degradation gate prevents declines in non-target dimensions, while the anti-forgetting gate prevents regression on previously pseudo-trained clips.
  • RQ3: Knowledge-Graph Operability: The knowledge graph propagates character and visual-treatment edits through linked shots while preserving unrelated assets and scene structure.The graph editor traces affected dependencies and re-renders only linked assets.
  • RQ4: Downstream Reuse: A single textual injection of the complete AVA-Encoder representation improves Overall for every evaluated downstream story-video generation framework.No framework-specific representation adapter or prompt tuning is introduced.

6. Conclusion

AVA-Encoder connects agent-based video editing with structured knowledge graphs and agentic auto-encoding, using reconstruction differences to guide textual-gradient evolution. It achieves stronger reconstruction and policy performance while reducing prompt-token use.

  • AVA-Encoder connects complex film content with agent-based editing through structured knowledge graphs and agentic auto-encoding.
  • 20.7 percentage points separate AVA-Encoder’s Overall reconstruction score from the strongest external baseline, 49.0% versus 28.3%.
  • 45.8% versus 44.4% is the controlled policy-only comparison between the pseudo-trained and independently human-tuned shot-level policies.
  • 74.3% fewer system-prompt tokens are used by the pseudo-trained policy than by the independently human-tuned policy.
  • Reconstruction differences become natural-language feedback for asset- and policy-level revision through fact-level correction records.

B. Reconstruction Evaluation Metrics

The reconstruction benchmark evaluates representation faithfulness through four comparison directions and shared film dimensions, without introducing a separate evaluation task.

  • The benchmark evaluates reconstruction through Video, Keyframe, Video back-captioning, and Keyframe back-captioning directions.
  • The four directions resolve reconstruction quality across shared film dimensions using distinct direct-comparison and back-captioning procedures.
  • A frozen multimodal evaluator constructs or extracts source-grounded facts and assigns categorical judgments to reconstructions.

B.1. Reconstruction Directions and Evaluation Dimensions

The evaluation combines direct and blind back-captioning comparisons across video shots and keyframes, scoring reconstruction fidelity through atomic facts and applicable dimensions.

  • Reconstruction Directions: Video and Keyframe use direct source-to-reconstruction comparisons, while V-BC and KF-BC compare independently generated descriptions.
  • Reconstruction Directions: Back-captioning separates source and reconstruction observations so source information cannot enter the reconstruction description before scoring.
  • Evaluation Dimensions: Video directions score eight dimensions, whereas Keyframe directions score seven because Audio is not applicable.
  • Fact-Level Scoring: Atomic facts receive scores of 1 for matches, 0.5 for partial matches, and 0 for mismatches or absent facts.
  • Fact-Level Scoring: Direct dimension scores average fact scores and cap the result when a critical fact is incorrect or missing.
  • Blind Back-Captioning Scoring: Back-captioning uses matched-fact rate with an additional contradiction penalty; omitted facts contribute neither matches nor contradictions.
  • Aggregation: Overall is the unweighted mean of the four direction scores, averaged through facts, dimensions, shots, and cases.

B.6. Alignment with Human Evaluation

AVA-Encoder validates reconstruction with blinded human comparisons and uses frozen QA banks to derive optimization rewards and diagnostic textual gradients. The automatic evaluator agrees with human rankings on 97.3% of blinded trials.

  • Human evaluation: 97.3% human-evaluation agreement was measured across 710 of 730 blinded triples comparing automatic rankings with expert preferences.The statistic is ranking agreement, not inter-annotator agreement or evaluation of the pseudo-training reward.
  • Human evaluation: Each human-evaluation trial contains a ground-truth keyframe or shot video and two source-hidden generated candidates.Annotators select the candidate closer to ground truth, while the automatic evaluator independently ranks both candidates.
  • Signal separation: R_eval is reserved for final four-direction benchmark reporting, whereas R_reward diagnoses failed facts and verifies candidate updates inside optimization loops.The two signals are both derived from source–reconstruction differences but serve different methodological roles.
  • QA-based reward: Frozen QA banks decompose each selected keyframe or shot into approximately 30 binary questions testing observable facts.The same ground-truth-derived questions evaluate current and candidate reconstructions, yielding more precise rewards and correctable errors.
  • QA-based reward: The QA residual is lower-is-better, while R_reward is its complementary higher-is-better pass rate used for optimization.Video QA spans eight reconstruction dimensions and verifies generated audio through its transcript.
  • Textual gradients: Failed QA tuples are mapped into correction records that form policy-level textual gradients for Data-Independent Encoding Policy Pseudo-Training.In keyframe refinement, direct image differences provide the modification direction while QA supplies candidate verification.

D.3. Data-Independent Encoding Policy Acceptance Gate

The policy acceptance gate accepts updates only when they improve current-video QA reward while limiting visual degradation and historical replay regressions. These checks compare equal-shot aggregates against the incumbent and inherited policy.

  • Historical replay: Historical replay evaluates candidate and inherited policies on the same sampled retained shots from preceding pseudo-training videos.Replay is evaluated only when the sampled replay memory is nonempty; the first video bypasses this average.
  • Acceptance conditions: Candidate policies must improve equal-shot current-video R_reward by more than 0.02.This threshold equals an absolute two-percentage-point reward improvement on the [0, 1] scale.
  • Acceptance conditions: Candidate policies may decrease equal-shot visual reward by no more than 0.03.The visual-reward condition limits degradation by three percentage points.
  • Acceptance conditions: Candidate policies may decrease sampled historical replay reward by no more than 0.05 relative to the inherited policy.The inherited shot-level policy is frozen at the start of training on the current video.

D.4. Data-Independent Encoding Policy Pseudo-Training Procedure

Data-Independent Encoding Policy Pseudo-Training sequentially updates a shared shot-level policy across six videos using frozen QA banks, textual gradients, candidate reconstruction, and acceptance gates. The evaluation setup standardizes generators, source-pixel exclusion, and representation budgets.

  • Pseudo-training procedure: Pseudo-training processes six videos sequentially and evaluates exactly three candidate shot-level updates per video.Accepted candidates become incumbents within the video, and the resulting policy passes to the next video.
  • Pseudo-training procedure: Each update converts failed QA questions into a textual gradient, generates a candidate prompt, reruns reconstruction and QA, and applies the acceptance gate.The QA bank for each selected shot is source-derived and frozen throughout the procedure.
  • Shared evaluation: The comparison retains each baseline system’s representation outputs but maps them to shared fixed image and video generators.The external comparison includes VideoAnalyzer, Storyboard Studio, and soap2soap.
  • Source-pixel exclusion: No source-video pixels are passed to generation models; reconstructed keyframes are newly generated from text-centered descriptions.Extracted frames and contact sheets remain analysis inputs or evaluation ground truth rather than generation inputs.
  • Representation budget: All systems use at most five generated reference keyframes and 1200 prompt tokens per shot at the representation-to-decoder interface.These budgets apply uniformly across the four systems and exclude source material available during representation construction and evaluation.
  • Decoder robustness: A decoder-robustness pilot found Overall reconstruction variation within 1.5 percentage points across HappyHorse and Seedance 2.0.The pilot regenerated 20 shots per representation system while holding assets, prompts, and evaluator fixed.

G. Dataset and Reproducibility Details

The released dataset and reproducibility materials cover the Film KG dataset, an 18-video evaluation collection, a six-video pseudo-training stream, and a held-out evaluation split.

  • Dataset and reproducibility: The supplementary dataset details describe the Film KG release, 18-video evaluation collection, six-video pseudo-training stream, and held-out evaluation split.

G.1. Released Film KG Dataset

The released Film KG dataset contains tens of thousands of shots represented as structured text derived from high-quality human-made film content. It omits multimodal assets while supporting external rendering and linked editing.

  • The dataset contains structured-text Film KG representations for tens of thousands of shots from high-quality human-made film content.It includes fine-grained descriptions of scripts, characters, scenes, objects, shots, and keyframes, including Oscar-caliber short films and human-produced AI videos.
  • The released dataset excludes image, audio, and video assets, which users can render through their own generation APIs or AVA-Encoder.
  • Graph-linked editing propagates node updates to dependent video content while allowing structured text to be queried without rendering video.

G.2. Pseudo-Training and Evaluation Video Collections

The study uses separate open-source video collections for pseudo-training and held-out evaluation, with varied evaluation content and fixed models. Evaluation spans annotated shots, selected keyframes, and multiple comparison directions.

  • The pseudo-training collection contains six videos assembled from publicly available open-source data.The listed clips include sequences from The Big Bang Theory, The Truman Show, Friends, Zootopia, Harry Potter, and The Pursuit of Happyness.
  • Video and keyframe reconstruction comparisons are reported by dimensions including character, scene, position, motion, audio, style, camera, and narrative.
  • The held-out evaluation collection contains 18 videos assembled from publicly available open-source data.Examples include Spirited Away, Zombie Cleaner, Harry Potter, Joker, Kung Fu Panda, and Genshin Impact sequences.
  • The evaluation collection includes 129 annotated shots and 246 selected keyframes across varied content and forms.Shot and keyframe counts can differ across representation systems because each retains its own segmentation and keyframe-selection policy.
  • The six pseudo-training videos and 18 evaluation videos do not overlap at the clip level.Each pseudo-training video evaluates three candidate policy updates, while all understanding, revision, QA, image-generation, and video-generation models remain fixed.

H. Fine-Grained Reconstruction Results

Fine-grained reconstruction evaluation reports keyframe-direction scores by applicable dimensions, with audio excluded from the keyframe direction. Scores are higher when reconstruction better matches the evaluated evidence.

  • The reported fine-grained values are dimension-level results from the R_eval framework rather than R_reward.The row means correspond to the keyframe entries reported in the main evaluation table.
  • Keyframe reconstruction is evaluated across seven applicable dimensions, with Audio marked N/A and higher scores indicating better performance.

I. Ablation Definitions and Effect Sizes

The ablations isolate AVA-Encoder’s hierarchical representation, optimization loops, and acceptance gates, while the graph schema supports structured storage, relation tracking, and targeted editing. Qualitative comparisons cover Zombie Cleaner and Kung Fu Panda.

  • I. Ablation Definitions and Effect Sizes: The complete AVA-Encoder configuration combines hierarchical understanding, both optimization loops, and anti-degradation and anti-forgetting gates.
  • I. Ablation Definitions and Effect Sizes: 20.7 percentage points separate the complete AVA-Encoder score of 49.0 from the strongest external baseline, soap2soap at 28.3.The complete configuration also improves over the no-loop configuration from 42.4 to 49.0.
  • J.1. Stored Schema: The graph contains nine structured-text node categories plus graph-addressable keyframe records and a linked multimodal asset layer.Text nodes cover story, event, shot, and six shot-specific state types; assets include generated images, audio, and rendered videos.
  • J.1. Stored Schema: Source-pixel exclusion forces reconstruction-critical information through structured text and relations rather than copied source-video pixels.This makes the graph queryable, editable at selected nodes, and reusable for generating new multimodal assets.
  • J.1. Stored Schema: Eleven explicit relation types preserve hierarchy, temporal order, entity recurrence, speech attribution, and other cross-modal dependencies.
  • J.1. Stored Schema: Identity edits propagate across an entity’s states and dependent keyframes, while content and parameter edits regenerate only affected prompts and assets.Unrelated subgraphs remain unchanged when edits are localized.
  • J.1. Stored Schema: Qualitative reconstruction comparisons use row-wise displays for Zombie Cleaner and Kung Fu Panda across GT, AVA-Encoder, and baseline systems.

K. Additional Qualitative Results

The section presents qualitative reconstruction comparisons and a graph-topology editing procedure that preserves film dependencies while updating only affected content.

  • K.1. Reconstruction Comparisons: Figures 6–11 compare GT, AVA-Encoder, soap2soap, VideoAnalyzer, Storyboard Studio, and naive Agentic Video Encoder reconstructions across six source videos.The comparisons use a shared generation setting and consistent row ordering.
  • K.2. Graph-Topology Editing: The film knowledge graph exposes story–event–shot hierarchy, state nodes, typed dependencies, and linked assets for inspection and editing.The visualization includes temporal, semantic, and asset-reference relations rather than a flat shot list.
  • K.2. Graph-Topology Editing: Graph-topology editing determines the affected subgraph through material closure, semantic consistency checks, and hierarchical updates.These stages collect regenerated assets, add one-hop contextual dependencies, and revise containing event and story summaries.
  • K.2. Graph-Topology Editing: The affected-node traversal suppresses repeated visits and costs O(|N_G|+|E_G|) time and O(|N_G|) memory.The complete affected set combines material, semantic, and hierarchical dependencies.
  • K.2. Graph-Topology Editing: Edits retain the original graph as an overlay while applying dependency-aware actions such as prompt recompilation, state revision, and asset rendering.Propagation differs by edit type, including identity replacement, content rewrite, visual treatment, and parameter edits.

L. Downstream Story-Video Evaluation Detail

The section details a reference-free story-video rubric and the prompt protocols used to evaluate reconstruction consistency across visual, character, and audiovisual dimensions.

  • L. Downstream Story-Video Evaluation Detail: The downstream rubric evaluates generated story videos without source-film similarity, using Character, Plot, Camera, Style, and Audiovisual Quality dimensions.Sub-dimension grades A–D map to 4–1; Overall is the unweighted mean of the five dimensions.
  • L. Downstream Story-Video Evaluation Detail: The “Asset refs.” condition supplies the complete AVA-Encoder representation once as text before generation, with a basic request to refer to it.The condition is described as system-independent.
  • L. Downstream Story-Video Evaluation Detail: Character checkpoints cover core identity, appearance, body proportions, expression, gaze, pose, clothing, and hairstyle or headwear.The protocol specifies evidence-based verdicts of match, partial, mismatch, or absent and delegates scoring downstream.
  • L. Downstream Story-Video Evaluation Detail: Reconstruction consistency prompts compare a reference/GT shot with an AI reconstruction while evaluating separate scene and character dimensions.The protocol first decomposes Video 1 into atomic visual checkpoints before inspecting Video 2.
  • L. Downstream Story-Video Evaluation Detail: Scene checkpoints cover appearance, furnishings, spatial layout, shot layers, and audiovisual style, with five to nine total items and limited supplementary details.The protocol requires binary, directly observable facts and includes negative questions to counter yes bias.
  • L. Downstream Story-Video Evaluation Detail: The mandatory self-check list verifies dialogue, synchronization, micro-expressions, spatial areas, exact species features, and mechanical actions before generation.It also requires screen-coordinate descriptions and distinguishes dialogue content from lip-motion descriptions.
Loading 2608.12313v1…