Source-linked AI summary
FoldingAgent: Inferring Parametric Origami Procedures from Demonstration Videos
Maya Moriya, Sigal Raab, Yael Vinker, Tali Dekel
TL;DR
Origami procedures are often available as unstructured videos, while computational systems typically require structured representations. FoldingAgent converts such videos into executable Pureland folding programs using sequential VLM reasoning, specialized tools, simulation, verification, and rollback. The framework successfully reconstructs procedures on PurelandFold, while remaining limited by occlusion, compound simultaneous actions, and its Pureland scope.
Problem
A semantic gap separates human-oriented origami videos from the structured representations required by computational origami systems.
Method
FoldingAgent combines a pretrained VLM with a parametric Pureland action space, simulator, verification tools, visual tools, and rollback-based sequential replanning.
Results
The framework successfully transforms unstructured origami demonstrations into executable, physically plausible folding procedures on the PurelandFold benchmark.
Takeaways & Limitations
The framework takes a step toward bridging intuitive visual demonstrations and structured computational origami representations.
Takeaways & Limitations
Performance remains constrained by heavily occluded folds, compound simultaneous actions, and the paper’s focus on Pureland Origami.
Abstract
from arXiv · showhide
We present FoldingAgent, an agentic framework for inferring explicit parametric folding programs directly from origami demonstration videos. Our framework leverages the reasoning power of a pre-trained Vision-Language Model (VLM) equipped with a suite of specialized tools that enable the agent to simulate geometric transitions, verify physical plausibility, retrieve and compare visual content, and evaluate its own predictions. To translate visual content into folding programs, we define a parametric space that consists of the paper's geometry and a set of parametric folding actions. Unlike models that predict static crease patterns, our agent operates sequentially and possesses the ability to re-plan its actions, effectively mitigating the compounding errors inherent in multi-step folding. Our approach takes a step toward closing the gap between human origami knowledge, which is primarily shared through unstructured visual demonstrations, and computational methods, which typically rely on structured, parametric representations such as a crease pattern or an executable parametric plan. We evaluate our approach on PurelandFold, a newly curated benchmark of diverse Pureland origami videos with ground-truth geometry and action labels. Our results demonstrate that by combining VLM reasoning with a set of specialized tools and physical simulation, we can successfully transform unstructured visual demonstrations into executable, physically plausible folding procedures.
1 Introduction
FoldingAgent addresses the gap between procedural origami demonstrations and structured computational representations by inferring executable folding programs from video. It combines a parametric Pureland action space with sequential VLM reasoning, specialized tools, simulation, verification, and replanning.
- Motivation: Origami knowledge is commonly communicated through ordered visual instructions, whereas computational methods typically require static crease patterns or other explicit structured representations.This mismatch creates a semantic gap between human folding language and machine-oriented representations.
- Task: The paper introduces a task that translates demonstration videos into procedural programs containing paper geometry, layered structure, and actions between successive states.The target representation captures both the evolving paper state and the action transforming one state into the next.
- Approach: FoldingAgent operates sequentially across keyframes, using a pretrained VLM and tools for geometric simulation, physical verification, visual comparison, self-evaluation, and rollback.Replanning is intended to mitigate long-range error compounding in multi-step folding.
- Task: Pureland Origami restricts folds to flat folds along single creases and provides primitive actions including fold, unfold, rotate, and flip.The simulator maintains geometry, layer structure, and crease patterns while executing these parameterized operations.
- Contributions: The paper contributes a procedural folding-program task, a parametric Pureland action space and simulator, an agentic tool-using framework, and the PurelandFold dataset.PurelandFold contains diverse videos with labeled geometry and folding actions.
2 Related Work
Prior computational origami work generally assumes structured geometry, while vision-based methods recover limited representations from controlled or isolated observations. FoldingAgent instead targets executable procedural reconstruction directly from unstructured instructional videos.
- Geometry-based Computation: Most computational origami methods analyze, simulate, design, or synthesize from explicit crease patterns or folded-state graphs.These approaches demonstrate the value of geometric models but assume the relevant representation is already available.
- Geometry-based Computation: Existing geometry-based methods cannot ingest the unstructured visual demonstrations through which origami knowledge is naturally shared.This leaves unresolved how computational representations are obtained from practice.
- Vision-based Reconstruction and Reasoning: Vision-based reconstruction studies have addressed illustrations, controlled state classification, line labels, 3D interpretations, guidance, physical states, and isolated crease prediction.These settings differ in observation conditions or target representation from procedural video reconstruction.
- Novelty: FoldingAgent is described as the first method to translate unstructured video keyframes into executable procedural programs supporting the broader Pureland action vocabulary.The stated vocabulary includes fold, unfold, rotate, and flip rather than only isolated actions.
- VLM Reasoning for Inverse Vision Tasks: Learn2Fold generates folding sequences from a given final crease pattern, whereas FoldingAgent reconstructs procedures directly from instructional videos without additional parametric information.The comparison distinguishes structured-input sequence generation from zero-shot video-based reconstruction.
3 Preliminaries
FOLD is a JSON-based interchange format for computational origami that represents geometry, crease labels, and face ordering. FoldingAgent adopts and extends it because flat-folded states require explicit relationships among overlapping faces.
- FOLD Representation: FOLD represents origami surfaces using vertices, edges, faces, and per-edge labels such as mountain, valley, boundary, or flat.It is designed as a standard interchange format for computational origami models and tools.
- FOLD Representation: FOLD adds faceOrders to encode pairwise above/below relationships between faces in a folded configuration.This captures layer ordering that ordinary mesh representations do not fully specify.
- FOLD Representation: The paper adopts and extends the FOLD representation for its computational origami states.
4 Method
The method represents Pureland folding procedures as executable sequences of geometric states and composable actions, reconstructed from video keyframes. A pretrained VLM interacts with simulation, visualization, verification, and rollback tools to produce consistent sequential reconstructions.
- 4 Method: Video keyframes are mapped to parametric paper states and transition actions, with a simulator executing actions or short action sequences between states.The recovered procedure contains states {S1, ..., SK} and corresponding Pureland actions {a2, ..., aK}.
- 4 Method: Pureland Origami restricts the task to flat folds along straight creases plus unfold, rotate, and flip, providing a compact yet expressive representation space.Intermediate states remain flat-folded layouts of polygonal faces, simplifying representation and simulation.
- 4 Method: Each state is a planar graph of vertices, faces, and labeled edges, extended with per-face orientation and explicit layer ordering for visual state tracking and layer recomputation.Orientation distinguishes front- and back-side-up faces, while layers organize coplanar faces from bottom to top.
- 4 Method: The action vocabulary comprises add_vertex, fold, unfold, rotate, and flip, whose minimal composability expresses non-vertex creases and paper reorientation.A non-vertex crease can be represented by composing add_vertex with fold, while rotate or flip can precede a fold.
- 4 Method: A deterministic controller dispatches agent tool calls to the simulator, renderer, and critic while maintaining simulator state across calls.The agent changes states only through controller-mediated tool calls.
- 4 Method: The agent sequentially proposes simulator actions, inspects frames and rendered states, invokes a visual critic, and rolls back to earlier checkpoints when transitions diverge.The critic compares source and target photos with rendered simulator states; rollback addresses errors that propagate through early transitions.
5 Results
FoldingAgent is evaluated qualitatively and quantitatively on PurelandFold, including long, occluded sequences and recovery from critic-approved errors. Its full framework outperforms ablations across geometry and pattern metrics, while requiring substantial iterative computation.
- Qualitative Results: The method reconstructs long procedures with rotations, folds, unfolds, flips, and precise vertex definitions, including sequences of 13 and 14 keyframes.Qualitative examples also include challenging hand occlusions, accumulating layers, and difficult lighting conditions.
- Qualitative Results: The agent recovers from a critic-approved incorrect fold by detecting later divergence, rolling back to an earlier checkpoint, and reconstructing the remaining sequence with additional tool calls.In the illustrated case, the corrected decision persists through the final fold despite the critic’s 88% success rate.
- Dataset: PurelandFold contains 27 diverse sequences annotated with keyframe-level folding actions and geometric states, including topology, geometry, orientations, and layer structure.The dataset was self-captured from curated Pureland instructions to evaluate procedural fidelity on real-world demonstrations.
- Ablations: 8% of VLM-CP outputs compile successfully, while the proposed state and action representation improves compilation, topology, and geometry, and adding the critic further improves CPD.The comparison isolates contributions from the representation, executable actions, and critic validation before evaluating the complete agentic framework.
- Ablations: The full agentic framework improves all reported metrics, yielding more precise topology, geometry, and crease patterns than the ablation variants.Intermediate performance declines with sequence progress for all methods, but the full method has a steadier slope and an increasing gap as complexity grows.
- Ablations: 84% and 80% of 600 user-study judgments preferred the full method over the two variants on image-to-render similarity.The study covered a subset of 10 sequences jointly completed by all variants within the tool-call budget.
6 Conclusion, Limitations, and Future Work
FoldingAgent translates unstructured origami demonstrations into executable procedural programs while addressing compounding errors in multi-step folding. Its scope remains limited by occlusion, compound simultaneous actions, and the Pureland action space.
- Conclusion: FoldingAgent translates unstructured origami demonstrations into executable procedural folding programs.
- Conclusion: Parameterized actions and simulation, verification, and rollback tools help mitigate compounding errors and long-range dependencies.
- Limitations: Occluded folds remain difficult to identify, whether obscured by the demonstrator or accumulating paper layers.
- Limitations: Compound simultaneous actions challenge the framework because they violate its standard sequential action-space assumptions.
- Future Work: Extending beyond Pureland origami requires a broader action space and may require stronger visual-language reasoning.
- Future Work: The framework is presented as a step toward applications in robotic manipulation, interactive instructional systems, and automated generative design.
A Full Results
The project page provides full reconstruction results and agent traces for a wide range of folding sequences. Figure 10 documents the interface for inspecting reconstructed geometries and detailed agent interactions.
- Full Results: The results page presents a wide range of folding sequences together with the full agent interaction for each sample.
B Implementation Details
The implementation combines a function-calling controller, simulator state management, parameterized geometric actions, visual verification, and recovery mechanisms. These components support sequential reconstruction while exposing practical runtime and token-budget constraints.
- Agent and Controller: Gemini 3.1 Pro Preview powers both the main agent and visual critic through typed JSON function calls dispatched by a controller.
- Execution Limits: Each sequence allows up to 300 iterations with a maximum thinking budget of 8,192 tokens per iteration.
- Execution Limits: Running times range from 9 to 96 minutes, averaging 31 minutes with a median of 20 minutes across folding sequences.
- Recovery: Thinking-token exhaustion can terminate runs with MAX_TOKENS, prompting rollback to the beginning of the current folding step.
- Agent and Controller: The controller validates tool calls against JSON schemas, dispatches valid calls, serializes outputs, and returns validation errors to the agent.
- Context Management: Each folding step starts from a text-only summary of previous steps while retaining the full current-step interaction history.
- Recovery: Checkpoint restoration replaces simulator state and increments a persistent restore counter used for logging and retry decisions.
- Parameterized Actions: Rotate and flip are explicitly modeled because demonstrators may reorient or turn over the paper between frames.
C Dataset Construction Details
PurelandFold is constructed from manually annotated instructional videos and supported by prompts, critic specifications, and backbone generalization experiments. The appendix records the dataset and evaluation materials used by the framework.
- Dataset Construction: PurelandFold videos were recorded with an iPhone 12 under ring-light and daylight illumination against black Bristol board.
- Dataset Construction: Keyframes representing geometric transitions were manually extracted, and corresponding folding actions were annotated.
- Prompts: The system prompt specifies the reconstruction task, simulator state schema, available actions, and expected checkpoint behavior.
- Prompts: Step-specific prompts initialize the first frame and then provide later targets, remaining-frame counts, and compact history logs.
- Prompts: The critic prompt defines four-image comparison, geometric inspection criteria, fold anchors, verdicts, and structured JSON output.
- Generalization Across Backbones: FoldingAgent substantially improves the plain VLM-CP agent across GPT-5.5, Claude-Opus-4.8, and Gemini-3.1-Pro-Preview on five sampled sequences.
- Generalization Across Backbones: Table 2 reports the study of generalization across backbones.
E.2 Additional Ablation: In Context Examples
The ablation compares zero-shot prompting with few-shot prompting using three external transition examples, finding that zero-shot performs better on five evaluation sequences.
- Zero-shot prompting outperforms few-shot prompting on the same subset of 5 sequences.The few-shot context contains three transitions of varying difficulty selected outside the evaluation set.
- The authors attribute few-shot underperformance to the breadth of the visual state space and diversity of valid action sequences.A small fixed example set cannot cover this space and biases the VLM toward particular observed states and actions.
- Few-shot performance is sensitive to the particular examples selected without improving generalization.
E.3 Detailed Metrics
The paper evaluates generated origami states through compilation validity, structural and geometric similarity, constraint satisfaction, final-state similarity, and crease-pattern dissimilarity.
- Compilation Validity checks whether each generated geometry is syntactically valid, geometrically foldable, non-self-intersecting, and associated with a definite folded state.The score is binary, and the Flat-Folder compiler is used for compilability.
- When compilation fails, GS, CS, and FFS receive penalty values of 0.2, 0.2, and 0.3, respectively.
- Topological Structure Similarity compares vertices, edge connectivity, face relationships, and crease-type distributions between generated and ground-truth patterns.
- Geometric Similarity compares normalized 3D point positions, crease dihedral-angle distributions, and bounding-box aspect ratios.
- Final Folded State combines Hausdorff distance with layering-consistency relationships between generated and reference faces.Its per-state score is averaged over completed folding sequences.
- Crease-Pattern Dissimilarity sums edge- and vertex-matching costs computed from geometric graph bipartite matchings.Edge costs use crease length, orientation, spatial distance, and type; vertex costs use degree and position.
F.2 Comparison with VIGA
The comparison evaluates VIGA for single-keyframe reconstruction and sequential scene editing, finding that its general-purpose scene representation cannot preserve origami geometry and layering requirements.
- VIGA is evaluated both on final-keyframe reconstruction and on sequential editing across folding keyframes.These settings test single-image geometry recovery and maintenance of geometry and accumulated layers over time.
- VIGA outputs violate origami requirements including a single square sheet, straight edges, and non-intersecting faces.The experiments use meshes, default configuration, 100 iterations, GPT-5, and a memory window of 12.
- After 96 iterations, single-keyframe reconstruction resembles the outer shape but lacks internal structure and fails to converge.
- Sequential editing discards internal structure and flattens the reconstruction after a few steps.The failure occurs as deeper layering and additional crease structure must be maintained.
G User Study - Screenshot Visualization
The user-study setup presents video keyframes alongside rendered geometric alternatives, while the reconstruction interface combines legal folding operations, visual feedback, checkpoints, and critic verification.
- User Study: Participants select which of two geometric representations more closely matches an instructional origami video keyframe.The study presents the keyframe alongside a pair of rendered geometries.
- Geometric Representation: The representation stores 2D vertices, counter-clockwise faces, labeled edges, face orientations, and bottom-to-top layer order.Edges distinguish boundary, flat, mountain, and valley types; layers group coplanar faces into continuous planes.
- Visual Feedback: The agent views photos, motion filmstrips, simulator state, and rendered diagrams to infer and inspect each transition.Rendered diagrams use side colors, solid active folds, and dashed crease marks as visual feedback.
- Verification and Recovery: Checkpoints let the agent save and restore states, while the critic compares real photos with rendered diagrams and returns match, mismatch, or divergence verdicts.The critic checks silhouette, proportions, fold positions, layer order, anchors, and contact topology; its verdict determines whether to save or retry.
- Verification and Recovery: The workflow uses checkpoint overviews and rollback to locate the first divergence and resume from the last known-good frame.The protocol also recommends fresh assumptions after repeated mismatches.
- Geometry Sensitivity: Small geometric offsets, asymmetries, gaps, and non-midpoint vertices are treated as meaningful features rather than errors to remove.The guidance warns that substituting an existing nearby vertex can change flap proportions and produce a mismatch.
- Failure Handling: After two mismatches, the agent should stop fine-tuning parameters and reconsider the action class, source checkpoint, or geometric assumptions.
- Simulation Protocol: The simulator permits only legal fold, unfold, rotate, and flip operations applied to the current paper state.A transition may include one fold preceded by one or two vertex additions, or a single unfold, rotate, or flip.