Source-linked AI summary
Git4Data: Database-Native Version Control for AI Agents
Hongshen Gou, Zuyu Zhang, Yuze Sun, Peng Xu, Feng Tian, Long Wang, Jianguo Wang
TL;DR
Agentic data workflows need a unified, database-native version-control layer for isolated and auditable exploration of parallel relational-data states. Git4Data provides SQL extensions for snapshot, branch, diff, and merge using immutable storage and MVCC. It runs up to an order of magnitude faster than DoltDB on BranchBench, while resource governance and richer merge semantics remain open problems.
Problem
Agentic workflows require isolated, reproducible, and auditable parallel data states, but source-code VCS does not scale to large datasets and relational databases rarely provide native branching, comparison, and merging.
Method
Git4Data treats a database as a repository and a table as a versioned object, exposing snapshot/tag, branch, row-level diff, and three-way merge with conflict policies through SQL.
Results
Up to an order of magnitude faster than DoltDB on BranchBench while sustaining 1,000 concurrently branching agents.
Takeaways & Limitations
Git4Data demonstrates a database-native approach to efficient versioning for agentic relational-data workflows.
Takeaways & Limitations
Resource governance for competing speculative branches and richer-than-row-level merge semantics remain open problems.
Abstract
from arXiv · showhide
Large Language Model (LLM) agents increasingly explore many candidate states of relational data in parallel, each of which should remain isolated, reproducible, and auditable, preferably through the same SQL interface used for ordinary data work. Existing tools support this requirement only partially: source-code version control does not scale to large datasets, whereas relational databases manage large data efficiently but rarely expose native branching, comparison, and merging. We present Git4Data, a database-native version-control layer for agentic workflows. Git4Data treats a database as a repository and a table as a versioned object, exposing Git-style operations (snapshot/tag, branch, diff, and merge with explicit conflict-resolution policies) through SQL extensions. Implemented in MatrixOne, a cloud-native relational database, Git4Data leverages immutable object storage and MVCC to make the cost of these operations proportional to the size of the change rather than the size of the data. On the BranchBench agentic branching workloads, Git4Data outperforms DoltDB by up to an order of magnitude. Overall, we believe this work sheds light on how relational databases can better support AI agents through efficient versioning.
1 INTRODUCTION
Git4Data addresses the need for isolated, reproducible, auditable branching in agentic data workflows by bringing Git-style version control into relational databases through SQL. Its MatrixOne implementation uses metadata-only cloning and delta-oriented operations, achieving major performance gains over conventional approaches and DoltDB.
- Motivation: LLM agents explore many relational-data states in parallel, requiring isolation, reproducibility, auditing, row-level diffs, validated merges, and rollback without copying base data.These requirements motivate database-native version-control primitives for agentic workflows.
- Motivation: Source-code version control is prohibitively slow for large datasets because diff and merge load the entire dataset and do not scale to billions of cloud-stored records.Relational databases scale data processing but generally lack explicit branching, comparison, and merging.
- Contribution: Git4Data exposes snapshot/tag, branch, row-level diff, and merge as SQL statements, treating the database as a repository and each table as a versioned object.Merges are three-way and conflict-aware, with explicit row-level conflict policies.
- Implementation: Metadata-only cloning and delta-based diff and merge make version-control costs proportional to changes rather than the full dataset in MatrixOne.The design relies on immutable append-only objects and MVCC.
- Evaluation: Up to an order of magnitude faster than DoltDB on BranchBench while sustaining 1,000 concurrently branching agents.Git4Data also reports a 0.2 s clone time for a 100 GB table and orders-of-magnitude improvements over SQL equivalents for diff and merge.
2 GIT4DATA
Git4Data maps Git’s snapshot, branch, diff, and merge vocabulary onto relational database constructs and exposes these operations through SQL. Its semantics support isolated writable branches, relational-content comparison, and three-way conflict-aware merging, while immutable storage makes costs depend on changes rather than whole-table size.
- Git4Data vocabulary: Git4Data treats a database as a repository and each table as a versioned object, mapping snapshots, branches, diffs, and merges onto database constructs.A snapshot names an immutable state; a branch evolves independently; diff reports disagreements; merge integrates accepted changes under a policy.
- Design decisions: SQL-based version-control operations inherit database transactions, authentication, and access control, allowing engineers and agents to use the ordinary data-work interface.The operations are defined at the SQL level and are not tied to one engine.
- Design decisions: A table version is an unordered multiset of rows, with a primary key providing stable row identity when present.This makes comparison semantics independent of physical layout and row order.
- Storage model: Immutable append-only storage makes snapshots metadata-only, so branching copies metadata and version differences are confined to post-divergence appended units.This supports operation costs proportional to changes rather than whole-table size.
- Branch: Cloned tables inherit a snapshot’s schema and data but then evolve independently, providing isolated states for speculative agent changes.The running workflow branches from sn1, advances the original to sn2 and the clone to sn3, then merges into sn4.
- Diff: Diff compares two snapshots as unordered record multisets and reports rows whose signed counts do not cancel.The SQL equivalent aggregates unioned positive and negative row counts by record values.
- Merge: Three-way merge preserves non-overlapping changes and applies FAIL, SKIP, or ACCEPT when both branches modify the same row.The target is the live table version, while the source may be any snapshot.
- Conflict resolution: With a primary key, conflicts require independent modifications to the same key; without one, Git4Data falls back to multiset reasoning and physical row identifiers.Identical changes cancel, while one-sided changes apply automatically.
3 DESIGN AND IMPLEMENTATION
Git4Data’s design maps version-control operations onto transactional relational-database primitives and implements them in MatrixOne using immutable objects, MVCC, and delta-based processing. Clone, diff, and merge operate on metadata and changed objects while preserving snapshot history and handling row-level conflicts.
- 3.1 Key Requirements: Git4Data requires append-only data, explicit deletion marks, and MVCC to support efficient, transactional version control.These requirements confine differences to appended units, expose deletions without full scans, and make merges atomic.
- 3.2 MatrixOne: MatrixOne stores immutable columnar objects and tombstones in cloud object storage, with metadata identifying the objects belonging to each table version.MVCC timestamps and metadata directories provide point-in-time table snapshots.
- 3.3 Clone: Cloning a table from a snapshot copies only object metadata, allowing branches and tags to retain shared data without copying table contents.Snapshot-aware garbage collection preserves objects referenced by named snapshots.
- 3.4 Diff: Git4Data computes diffs by scanning only the object deltas from two versions and aggregating inserts, updates, and deletions using primary-key or row-level matching.With primary keys, repeated operations on one key collapse into a single logical operation; without them, matching uses full values and physical row identifiers.
- 3.5 Merge: Three-way merge aggregates branch deltas against a common base, treating genuine same-key changes as conflicts while recognizing compaction-induced row relocation as non-conflicting.If lineage is unavailable, MatrixOne uses an empty base; shared objects can still be skipped during aggregation.
4 EXPERIMENTAL EVALUATION
Git4Data is evaluated from individual version-control operations through end-to-end BranchBench workloads, including concurrency and data-size scaling. Across these settings, built-in metadata- and delta-based operations substantially outperform SQL alternatives and DoltDB, while whole-table workflows remain sensitive to data size.
- Version-Control Operations: Built-in diff and merge are substantially faster than equivalent SQL implementations because they scan changed objects rather than entire tables.Primary keys further help by collapsing multiple operations on the same key; no-primary-key cases may require additional tuple lookups.
- Version-Control Operations: In collaborative tests with four engineers and 10% primary-key overlap, built-in diff and merge remain orders of magnitude faster than SQL alternatives, even for one million updates.Conflicting updates are resolved with ACCEPT.
- BranchBench: Across four BranchBench workflows, Git4Data is up to 18.5× faster than DoltDB because snapshot-and-delta operations scale with change size rather than table size.Runtime variance remains under 3.5 s.
- BranchBench: With 1,000 concurrent agents at Scale Factor 100, Git4Data completes in 400 s while DoltDB does not finish within two hours.The remaining cost is dominated by shared compute and I/O from concurrent branch-local executions.
- BranchBench: At tenfold larger data size, branch-local workflows slow by at most 3.2× or remain essentially flat, but failure reproduction grows about 13× when repairs rewrite the entire table.The 1,000-agent simulation finishes in 600 s at Scale Factor 1,000 versus 400 s at Scale Factor 100.
5 RELATED WORK
Prior systems provide pieces of data versioning, but gaps remain in record-level branching, comparison, merging, and agent-scale parallel exploration. Git4Data is motivated by combining database-native operations with row-oriented relational semantics for agentic workloads.
- Git and extensions such as Git LFS and DVC support source code or dataset pointers, but not record-level diff or conflict resolution.
- Data-lake systems provide snapshot lineage, branches, tags, and merges, but generally identify differences at object or table granularity rather than by row.Disjoint row updates within one table can therefore collide, with conflicts resolved by retaining one side’s file.
- Database snapshots and PITR preserve named past versions, while writable clones support parallel development, but these mechanisms do not provide the full branch-and-merge model.PostgreSQL reflink copying can also require exclusive source access, whereas Snowflake, Supabase, and Neon provide zero-copy or online branching capabilities.
- Agentic workloads require isolated branches for speculative updates, evaluation, rollback, comparison, merging, and pruning across parallel hypothetical data states.BranchBench models these branch lifecycle operations, branch-local SQL, cross-branch comparison, and pruning as workload dimensions.
6 LESSONS AND FUTURE WORK
Git4Data suggests that database-native version control can be implemented as a thin layer over transactional storage, while exposing open challenges in semantics, resource governance, schema evolution, and retention.
- Append-only data, deletion marks, and transactions can provide the mechanisms for database version control without modifying the storage layer.The implementation interprets existing transactional storage capabilities through a version-management layer.
- Primary-key reconciliation gives relational versioning cleaner semantics and more than an order-of-magnitude speedup over traditional value-based matching.Diff and merge still assume compatible schemas, leaving schema evolution on long-lived branches unresolved.
- Once branching is metadata-only, shared computation and I/O—not branch creation—become the dominant scalability cost for concurrent agent workloads.This shifts the open problem toward resource governance for shared infrastructure.
- Future work includes resource governance, richer merge semantics, schema evolution, and retention policies for sustained branching.The paper specifically identifies scheduling, quotas, cell- or semantic-level conflict resolution, joint schema versioning, and auditability-versus-storage-growth trade-offs.