PaperlayerView exact revision on arXiv

Source-linked AI summary

Consistency-Driven Co-Evolution for Self-Supervised Cross-Representation Learning

Xuehang Guo, Pengyuan Li, Tom Hope, Tirthankar Ghosal, Manling Li, Qingyun Wang

arXiv:2608.04926v1cs.LGcs.AIcs.CL

TL;DR

Cross-representation learning across charts, tables, and code faces ambiguous one-to-many mappings, costly supervision, and task-specific optimization. CoCoEvolve uses consistency-driven co-evolution and evaluation across the representation cycle, improving performance across training-time, test-time, and out-of-domain settings.

  • Problem

    Cross-representation learning suffers from ambiguous one-to-many mappings, costly correspondence annotation, and optimization signals that may not generalize across directions or representations.

  • Method

    CoCoEvolve defines one-to-one correspondences and uses annotation-free cross-representation agreement for co-evolution across the chart-table-code cycle at training and test time.

  • Results

    CoCoEvolve improves cross-representation understanding across code, chart, and table evaluations, with up to ↑37.91% gains on the non-overlapping test set.

  • Takeaways & Limitations

    Cross-modal consistency can provide a scalable, reliable supervision signal for annotation-free structured multimodal reasoning.

  • Takeaways & Limitations

    The framework currently uses small model architectures and static chart images, leaving larger models and dynamic visualizations untested.

Abstract

from arXiv · show

As chart images, tabular data, and visualization code play increasingly important roles across diverse domains, cross-representation understanding across these modalities poses fundamental challenges for AI systems: the relationships across representations are inherently \textit{one-to-many}, supervision is ambiguous and costly, and model optimization lacks a principled signal that is both direction-adaptive and representation-generalizable beyond task-specific objectives. We introduce CoCoEvolve to improve consistency across chart, table, and code representations. Instead of treating cross-representation mapping as a one-to-many problem, we define explicit one-to-one correspondences and optimize models using agreement between representations, without additional annotations. During training, CoCoEvolve@Train performs co-evolution across the chart-table-code cycle, while CoCoEvolve@Test applies the same consistency objective at inference time for test-time co-optimization. We also present CoCoEvolve@Eval, an evaluation suite covering all six cross-representation tasks. Across four benchmarks, CoCoEvolve improves performance in both training-time and test-time settings. Our project page: https://xhguo7.github.io/CoCoEvolve/.

1 INTRODUCTION

Cross-representation understanding across chart images, tables, and visualization code is hindered by inherently one-to-many relationships, ambiguous and costly supervision, and optimization signals that may not generalize across directions or representations. CoCoEvolve addresses these challenges with precise one-to-one correspondences, annotation-free consistency-based co-evolution, and unified evaluation across the representation cycle.

  • Motivation: Chart images, tables, and visualization code express the same underlying information through visual, structured, and programmatic representations.Chart images encode data through axes, marks, and layouts; tables provide structured data; visualization code specifies rendering.
  • Challenges: One-to-many relationships make cross-representation supervision ambiguous and expensive, because each chart may correspond to multiple valid tables and rendering programs.Annotating the large space of potential correspondences is costly to produce at scale.
  • Challenges: Existing benchmarks’ unconstrained one-to-one ground truths poorly address these mappings, motivating precise one-to-one correspondence constraints.Accurately labeled cross-representation correspondences remain scarce and poorly defined.
  • Challenges: Fixed chart-to-table or chart-to-code training can degrade performance on trained tasks, reversed directions, and unseen representation combinations.The introduction identifies table-to-chart and table-to-code as examples of reversed-direction and unseen-combination challenges.
  • CoCoEvolve: CoCoEvolve uses cross-representation agreement as an annotation-free, task-, direction-, and representation-agnostic optimization signal through co-evolution at training and test time.The framework jointly defines precise one-to-one correspondences and enables annotation-free optimization at both train and test time.
  • CoCoEvolve: CoCoEvolve also provides a systematic, unified evaluation suite covering all tasks in the chart-table-code cycle.The cycle comprises six cross-representation tasks.

2 RELATED WORK

Prior work advances individual edges of the chart-table-code cycle and develops cycle consistency as an annotation-free self-supervision signal. However, existing methods are constrained by single representations or domain-specific applications, motivating a unified multimodal framework.

  • Cross-Representation Learning: Recent work advances chart-to-table extraction and chart-to-code generation as individual edges of the chart-table-code representation cycle.Chart-to-table methods recover structured tabular data from chart images, while chart-to-code methods target the visualization program underlying a chart.
  • Self-Supervised Cycle Learning: Cycle consistency enables annotation-free learning through round-trip reconstruction, extending from vision to language and other vision settings.CycleGAN introduced this self-supervision signal for unpaired cross-domain translation in vision.
  • Self-Supervised Cycle Learning: Existing approaches rely on single representations poorly suited to structured cross-modal reasoning or remain limited to specific domains.These limitations motivate unifying cycle consistency and co-training into an annotation-free paradigm for multimodal cross-representation learning.

3 METHOD

CoCoEvolve frames chart, table, and code understanding as a six-task representation cycle, then co-optimizes chart-to-table and table-to-code models using consistency-driven, annotation-free rewards. Its method also supports optional teacher guidance and evaluates outputs with deterministic and fine-grained learned judging criteria.

  • Representation Cycle: The method defines six cross-representation tasks over V ⇔ T, T ⇔ C, and V ⇔ C, connected by chart-to-table, table-to-code, and code-to-chart mappings.The mappings form the cycle h ◦ gψ ◦ fθ : V → T → C → V, with h deterministically rendering code into a chart image.
  • Constraint Conditioning: Auxiliary descriptive constraints condition table-to-code mappings to resolve ambiguity from inherently one-to-many cross-representation relationships.Each constraint si aligns ti and ci with respect to vi, while gψ(· | s) represents the conditioned mapping family.
  • Co-Evolve Training: CoCoEvolve jointly trains Mθ and Mψ on unannotated chart images, coupling model sampling and rewards through hierarchical consistency signals.Code consistency uses execution success and embedding similarity; visual consistency uses four metrics gated on code execution, while format reward enforces well-formed outputs.
  • Hierarchical Reward: Shared agreement rewards are additionally grounded against the original chart, preventing both models from colluding on mutually consistent but semantically incorrect outputs.The reward compares the models’ rendered charts with each other and independently with the original input chart.
  • Test-Time Optimization and Evaluation: At test time, CoCoEvolve optimizes both models using annotation-free signals, while optional teacher guidance can transition toward self-supervised co-evolution through linear annealing.The method also provides rule-as-judge, LLM-as-judge, and MLLM-as-judge evaluation using deterministic or independently weighted fine-grained dimensions across chart, code, table, and constraint outputs.

4 EXPERIMENTS

Experiments evaluate CoCoEvolve across six cyclic cross-representation tasks, including out-of-domain and multi-domain settings. Results show consistent gains across representations, supervision configurations, domains, and reinforcement-learning algorithms.

  • Experimental Setup: Evaluation covers six cyclic chart-table-code tasks, with additional out-of-domain complexity levels and multi-domain test sets.The evaluation uses ChartCoder, ChartMimic, ChartNet, and Chart2Code-derived settings.
  • Main Results: CoCoEvolve@Train+@test achieves up to 100.00% sandbox execution success, with gains up to ∆=↑17.48% in rule-as-judge and ∆=26.57% in LLM-as-judge evaluations.Chart evaluation also reports gains of ∆≥6.79% and ∆≥3.33% across rule-as-judge and MLLM-as-judge evaluations.
  • Teacher Module Ablation: Enabling the teacher module only at test time improves all six cross-representation tasks, yielding gains of ∆≥2.54% across representations.Enabling the module during training alone is less stable, while enabling it during both training and test time provides more balanced overall performance.
  • Generalizability: CoCoEvolve generalizes across domains, improving performance by up to ↑35.68% on ChartNet and by ∆≥2.41% on ChartCoder’s non-overlapping test set.The models are finetuned on 10,298 instances adapted exclusively from ChartCoder.
  • RL Algorithm Generalizability: CoCoEvolve improves both Mθ and Mψ across all six tasks regardless of whether GRPO, DAPO, or GSPO is used.Reported gains reach ↑18.71% in execution, ↑6.96% in code, and ↑12.15% in chart evaluations.

5 CONCLUSIONS

CoCoEvolve is a consistency-driven co-evolution framework for structured multimodal reasoning that uses self-evolving cross-representation supervision. It demonstrates cross-modal consistency as a scalable alternative to annotation-heavy supervision pipelines.

  • Conclusions: CoCoEvolve enables self-evolving training in which improvements by one model provide stronger supervision to the other, progressively refining both.The framework is designed for cross-representation structured multimodal reasoning.
  • Conclusions: The work shows that cross-modal consistency can provide scalable, reliable supervision for structured reasoning tasks without relying on annotation-heavy pipelines.This positions consistency as an alternative supervision signal for cross-modal learning.

A LIMITATIONS … B.2 CAN FIXED SUPERVISION RELIABLY GUIDES CROSS-REPRESENTATION LEARNING?

The paper identifies limitations involving model scale, assumed one-to-one annotations, and fixed supervision for cross-representation learning. Preliminary studies test whether these supervision strategies reliably optimize trained tasks and generalize across directions and representation combinations.

  • A LIMITATIONS: CoCoEvolve currently uses small-size model architectures, leaving broader MLLM and LLM families, including models around 480B parameters, for future study.The authors seek to examine how architectural choices interact with co-evolution dynamics.
  • B.1 CAN ASSUMED ONE-TO-ONE ANNOTATIONS RELIABLY SUPERVISE MODEL LEARNING?: The study asks whether assumed one-to-one annotations can reliably supervise learning despite chart-table-code mappings being inherently one-to-many.This question targets the mismatch between benchmark annotations and the underlying representation relationships.
  • B.1 CAN ASSUMED ONE-TO-ONE ANNOTATIONS RELIABLY SUPERVISE MODEL LEARNING?: Qwen3-VL-4B and Qwen3-4B were SFT-finetuned for 1 epoch on 30,000 ChartNet instances and evaluated on 542 held-out non-overlapping instances.The evaluation compares both base and SFT-finetuned models on the same benchmark.
  • B.1 CAN ASSUMED ONE-TO-ONE ANNOTATIONS RELIABLY SUPERVISE MODEL LEARNING?: SFT improved chart-to-code generation by ∆=↑1.14% in rule-as-judge code evaluation and ∆=↑ 4.57% in LLM-as-judge code evaluation, but degraded other evaluations.The reported degradation affects code, chart, and table evaluation.
  • B.2 CAN FIXED SUPERVISION RELIABLY GUIDES CROSS-REPRESENTATION LEARNING?: The second question asks whether fixed chart-to-table and chart-to-code supervision generalizes beyond trained tasks to held-out tasks, directions, and representation combinations.The study specifically tests robustness on unseen instances and transfer across the broader task space.
  • B.2 CAN FIXED SUPERVISION RELIABLY GUIDES CROSS-REPRESENTATION LEARNING?: Because ground-truth tables were unavailable, Gemini-3-Pro generated reference tables and constraints through outcome-grounded filtering for both training and evaluation.These generated tables support the preliminary fixed-supervision study.
  • B.2 CAN FIXED SUPERVISION RELIABLY GUIDES CROSS-REPRESENTATION LEARNING?: ∆ up to ↓25.19% shows fixed supervision failed to reliably optimize models even on trained tasks across the six-task evaluation.The tasks cover trained directions V ⇔T and V ⇔C, reversed directions T ⇔V and C ⇔ V, and unseen combinations T ⇔C.

B.3 HOW DO MODELS FAIL IN CROSS-REPRESENTATION UNDERSTANDING? … C.2 TRAINING DATASET

The preliminary analysis finds that cross-representation errors propagate across tables, code, and charts, producing both low fidelity and poor agreement. The paper then constructs datasets through sandbox verification, staged filtering, and sampling for training and evaluation.

  • B.3 HOW DO MODELS FAIL IN CROSS-REPRESENTATION UNDERSTANDING?: Models exhibit correlated failures across tables, visualization code, and rendered charts when they misunderstand a chart.Errors include incorrect numerical values, missing titles and labels, and incomplete information extraction across representations.
  • B.3 HOW DO MODELS FAIL IN CROSS-REPRESENTATION UNDERSTANDING?: Cross-representation failures involve both low within-representation fidelity and poor agreement among representations.The analysis concludes that independent optimization and fixed supervision signals are insufficient for robust understanding.
  • B.3 HOW DO MODELS FAIL IN CROSS-REPRESENTATION UNDERSTANDING?: CoCoEvolve addresses these failures by explicitly encouraging agreement and rewarding mutually consistent generations across representations.The framework aims to improve fidelity within each representation and alignment across representations through consistency-driven co-evolution.
  • C DATASET CONSTRUCTION: Training and evaluation datasets adapt existing chart benchmarks to the cross-representation learning framework, reformat instances into representation-cycle instructions, and verify them through sandbox execution.Table 3 summarizes the resulting data statistics.
  • C.1 SANDBOX ENVIRONMENT: Visualization code is executed in isolated Docker containers pre-installed with common data-visualization and scientific-computing packages.A fixed timeout is used for reproducibility and safety across dataset construction, training, and evaluation.
  • C.2 TRAINING DATASET: The training dataset is built exclusively from ChartCoder through sandbox-executability, token-length, and outcome-grounded verification filters.The filtering pipeline retains executable instances, enforces input and code length thresholds, and checks whether generated tables and constraints improve downstream code generation.
  • C.2 TRAINING DATASET: 10,298 instances form the final training set, while 542 remaining non-overlapping instances form the in-domain ChartCoder evaluation set.Both subsets are randomly sampled after filtering.

C.3 EVALUATION DATASETS · D COCOEVOLVE@TRAIN: CONSISTENCY-DRIVEN CO-EVOLUTION · E IMPLEMENTATION DETAILS

The paper evaluates CoCoEvolve on four non-overlapping benchmark-derived datasets filtered and sampled through the same pipeline as training. Its training objective uses paired clipped-ratio surrogates with KL penalties and cross-model advantage estimation, while implementation details are summarized in Tabs. 4–6.

  • C.3 EVALUATION DATASETS: Evaluation datasets are constructed from four benchmarks using the training-data filtering pipeline and contain no instances overlapping the training set.Filtering includes sandbox executability verification, token-length filtering, reference-table and constraint generation, and outcome-grounded verification.
  • C.3 EVALUATION DATASETS: The evaluation sets are randomly sampled and adapted to cross-representation learning instructions after filtering.Sampling configurations and token-length thresholds are summarized in §E.1 and Tab. 6.
  • C.3 EVALUATION DATASETS: ChartCoder provides a 542-instance in-domain evaluation set adapted to cross-representation understanding tasks.The set is non-overlapping with the training data.
  • C.3 EVALUATION DATASETS: ChartNet provides a 542-instance out-of-domain evaluation set spanning different domains and visualization packages.Examples include health and finance domains and matplotlib and plotly packages.
  • C.3 EVALUATION DATASETS: ChartMimic provides 1084 adapted evaluation instances comprising 542 in-domain cross-representation tasks and 542 out-of-domain chart modification instances.The passage describes both components as adapted to the paper’s cross-representation understanding tasks.
  • D COCOEVOLVE@TRAIN: CONSISTENCY-DRIVEN CO-EVOLUTION: The CoCoEvolve@Train co-evolution objective uses token-level clipped-ratio surrogate losses with KL penalties and paired reward functionals.The surrogates and reward functionals are defined in Eqs. 8 and 7, respectively.
  • D COCOEVOLVE@TRAIN: CONSISTENCY-DRIVEN CO-EVOLUTION: The πθ advantage averages across Kψ children of each πθ rollout, while the πψ advantage is computed within Kψ siblings sharing the same πθ parent.This assignment makes πθ credit reflect expected cross-model quality rather than a single πψ draw.
  • E IMPLEMENTATION DETAILS: Tabs. 4–6 summarize the paper’s core symbols, notations, and hyperparameter settings to support reproducibility and customized adaptation.The implementation section further explains key design choices.

E.1 EXPERIMENT CONFIGURATION · E.2 COMPUTATION OVERHEAD

The experiment configuration separates task mappings, model identities, and training policies while using asymmetric reward weights to reflect the distinct roles of the two models. Computation overhead is measured across GPU use, API cost, and wall-clock time, with consistency-driven training and evaluation reducing costs or time in reported settings.

  • E.1 EXPERIMENT CONFIGURATION: CoCoEvolve distinguishes task mappings fθ and gψ, model identities Mθ and Mψ, and training policies πθ and πψ.Mθ is multimodal, Mψ is a language model, and either the same or different base models may instantiate them.
  • E.1 EXPERIMENT CONFIGURATION: Mθ receives stronger individual grounding emphasis because it generates code, tables, and constraints from charts, whereas Mψ relies more heavily on cross-model consistency.The reported configuration assigns Mθ λθ = 0.7 and an individual grounding weight, while Mψ is primarily grounded through cross-model consistency.
  • E.1 EXPERIMENT CONFIGURATION: Asymmetric reward weights also act as an anti-collusion mechanism against mutually consistent but semantically incorrect outputs.The design is intended to prevent reward hacking and training crashes by strengthening individual grounding relative to cross-model consistency for Mθ.
  • E.1 EXPERIMENT CONFIGURATION: The weighting design uses ω(π) for intra-component reward contributions, λ(π) for per-model reward combinations, and γ for composite evaluation scores.These coefficient families distinguish reward sub-weights, asymmetric training rewards, and multidimensional evaluation aggregation.
  • E.2 COMPUTATION OVERHEAD: Training configurations use H100 (96GB) GPUs with no API cost because rewards rely on deterministic sandbox execution and lightweight embedding models.Compared with the Baseline, CoCoEvolve@Train (M disabled) reduces wall-clock time through a more efficient co-evolution objective.
  • E.2 COMPUTATION OVERHEAD: 71h (no annealing) and 72h (with annealing) are the reported training times when enabling M.The passage states that enabling M further reduces training time to these values.
  • E.2 COMPUTATION OVERHEAD: $0.8637 per sample is the baseline evaluation API cost, which CoCoEvolve reduces through test-time consistency-driven candidate selection.All evaluation configurations use H100 (96GB) GPUs, and wall-clock time follows the same downward trend across methods.

F COCOEVOLVE@EVAL: EVALUATION SUITE FOR MULTIDIMENSIONAL ASSESSMENT … G ANALYSIS ON CROSS-REPRESENTATION FAILURES

CoCoEvolve@Eval is a systematic suite spanning 27 metrics, three complementary judge types, and all six cross-representation tasks. It evaluates charts, code, tables, and constraints through multidimensional rule-based, LLM, and MLLM criteria rather than single broad scores.

  • F COCOEVOLVE@EVAL: EVALUATION SUITE FOR MULTIDIMENSIONAL ASSESSMENT: CoCoEvolve@Eval combines rule-based judge, LLM-as-Judge, and MLLM-as-Judge approaches across 27 metrics covering all six cross-representation tasks.The suite is designed to address limitations of existing LLM-as-Judge approaches.
  • F.1.1 RULE-AS-JUDGE CHART EVALUATION: Rule-based chart evaluation combines structural Fssim, semantic Fclip, perceptual Fdino, and textual Focr accuracy into the final score Fchart.These dimensions respectively target structural similarity, semantic alignment, fine-grained visual fidelity, and OCR-based text fidelity.
  • F.1.2 RULE-AS-JUDGE CODE EVALUATION: Rule-based code evaluation measures executability separately with Fexec and combines Fcodebleu, Fast, Fcosine, Fcodebert, and Funixcoder into Fcode.The combined dimensions cover code quality, AST structure, lexical similarity, contextual similarity, and semantic correspondence.
  • F.1.3 RULE-AS-JUDGE TABLE EVALUATION: Rule-based table evaluation combines schema Fschema and value Fvalue accuracy into Ftable using column-level and cell-level F1.Schema accuracy compares columns, while value accuracy evaluates row-column-value triples.
  • F.1.4 RULE-AS-JUDGE CONSTRAINT EVALUATION: Rule-based constraint evaluation combines semantic Fssem = Fs and lexical Frouge accuracy into Fconstraint.The semantic score uses sentence-level cosine similarity, whereas the lexical score uses ROUGE-L.
  • F.2 LLM-AS-JUDGE EVALUATION: LLM-as-Judge decomposes visualization code quality into five independently scored, non-overlapping dimensions and combines them into Jcode.The dimensions include data correctness, chart type accuracy, structural fidelity, visual accuracy, and style-related criteria.
  • F.3 MLLM-AS-JUDGE EVALUATION: MLLM-as-Judge scores each dimension on a 0–5 integer scale normalized to [0, 100], replacing unreliable single wide-range judgments with fine-grained assessments.For charts, the five dimensions include chart type, data, text, style, and visual accuracy; the final score is Jchart.
  • F.3.2 MLLM-AS-JUDGE TABLE EVALUATION: MLLM-as-Judge evaluates tables and constraints across five dimensions each, producing Jtable and Jconstraint from schema/data/coverage or fidelity/completeness/clarity criteria.Table evaluation uses schema, data, coverage, row alignment, and format dimensions, while constraint evaluation uses visual fidelity, completeness, structural clarity, trend, and style dimensions.

G.1 QUANTITATIVE ANALYSIS ON CROSS-REPRESENTATION FAILURES

The five-dimensional evaluation analyzes cross-representation failures for the baseline, CoCoEvolve@Train, and CoCoEvolve@Train+@Test, finding substantial gains over the non-finetuned base model driven mainly by fewer data-related errors.

  • Quantitative analysis on cross-representation failures: Fig. 10 reports cross-representation failure distributions using five-dimensional evaluation for the baseline, CoCoEvolve@Train, and CoCoEvolve@Train+@Test.The evaluation is defined in §3.5.
  • Quantitative analysis on cross-representation failures: ↑35.10% overall improvement is achieved by CoCoEvolve over the non-finetuned base model.The passage describes this as an improvement of up to ∆=↑35.10%.
  • Quantitative analysis on cross-representation failures: CoCoEvolve’s gain is primarily accompanied by notable reductions in data-related errors across different representations.The passage notes that other error categories remain.

G.2 QUALITATIVE ANALYSIS ON CROSS-REPRESENTATION FAILURES

Qualitative inspection shows that larger models generally understand cross-representation relationships better than smaller models, whose data-recognition errors can propagate across representations. Even in a simpler labeled-chart case, larger models retain errors in data range and color encoding that create inconsistencies.

  • Overall comparison: Larger models generally demonstrate stronger cross-representation understanding than smaller-scale models.The analysis is based on manual inspection of model outputs and evaluation of their performance.
  • Larger-model failures: Explicit value labels can mitigate inaccurate data extraction in a relatively simple bar chart, but Claude-4.6-Sonnet still exhibits style-related failures.The passage identifies the labeled-chart setting as reducing the negative impact of inaccurate extraction while noting remaining style-related issues.
  • Small-model failures: Qwen3-4B-VL produces data-recognition errors that propagate into inaccurate extraction, incomplete information transfer, and cross-representation inconsistencies.The failure example omits constraints for clarity and attributes the accumulating errors to the model’s small size.
  • Larger-model failures: Claude-4.6-Sonnet still makes errors in data range and color encoding, causing visualization inaccuracies and cross-representation inconsistencies.These residual errors occur despite the chart’s explicit value labels supporting accurate data recognition and extraction.
Loading 2608.04926v1…