Source-linked AI summary
Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents
Srishti Palani, Vidya Setlur
TL;DR
Curated references are costly, cannot exhaustively represent valid CVA outputs, and are often unavailable in production. LEXARA-RF evaluates responses through reference-free verification using the prompt, data, and model output. It aligns comparably to reference-based methods, exceeds surface-similarity baselines, and localizes structural failures with high accuracy.
Problem
CVA outputs are open-ended multimodal responses, while curated references are costly to create, cannot cover all valid outputs, and are often unavailable in production.
Source: §1 · p. 1Method
LEXARA-RF uses 13 verification metrics grounded in visualization design theory and conversational principles to assess outputs from the prompt and underlying data rather than references.
Source: §1 · p. 1Results
LEXARA-RF aligns with expert judgment comparably to reference-based methods, exceeds surface-similarity NLG baselines, and localizes structurally grounded failures at ≥90% accuracy.
Source: §5 · p. 4Takeaways & Limitations
LEXARA-RF can complement reference-based evaluation where per-test-case ground-truth authoring is prohibitive.
Source: §5 · p. 4Validation is limited to Vega-Lite-style specifications, English utterances and judge prompts, and the Lexara corpus with inherited metric weights.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Conversational visual analytics (CVA) agents powered by large language models generate visualizations and natural-language explanations from open-ended queries. Evaluating these multimodal outputs is challenging: curated reference benchmarks are costly to author, cannot comprehensively capture the space of valid responses, and are unavailable in production. Building on the Lexara evaluation framework, we introduce Lexara-RF, a reference-free set of metrics that scores CVA outputs using only the prompt, data, and model response. We reformulate evaluation as verification: 13 metrics operationalize visualization design theory and Gricean cooperative principles as computable consistency, intent-alignment, and design validity checks. On a human-rated corpus of CVA test-cases, Lexara-RF achieves alignment comparable to reference-based formulations, outperforms surface-similarity NLG baselines, and localizes structurally grounded failures with high accuracy.
1 INTRODUCTION
CVA agents produce data-grounded visualizations and explanations, but their open-ended outputs are difficult to evaluate with curated references. LEXARA-RF instead verifies outputs against the prompt and data, offering reference-free metrics and evidence of alignment with human judgments.
1 INTRODUCTION
- CVA evaluation must assess data fidelity, encoding effectiveness, and user-intent alignment across multimodal responses.Responses pair visualizations with natural-language explanations.Source: §1 · p. 1
- Reference benchmarks cannot cover open-ended valid visualizations, are costly to maintain, and are often unavailable for production evaluation.A query can admit valid charts that differ in type, encoding, or aggregation, risking false negatives when compared with a single reference.Source: §1 · p. 1
- LEXARA-RF recasts evaluation as verification of conditions derived from visualization design theory and conversational principles, using the prompt and underlying data rather than references.It contributes 13 reference-free metrics spanning expressiveness, effectiveness, and conversational alignment.Source: §1 · p. 1
- On a human-rated corpus of 120 responses, LEXARA-RF showed parity with reference-based metrics at this sample size and higher alignment than BLEU, ROUGE-L, and BERTScore.The metrics also localized structurally grounded failures while surfacing semantically ambiguous ones as distributed signals.Source: §1 · p. 1
2 RELATED WORK
Prior work provides foundations for visualization design, reference-based evaluation, and conversational quality, but leaves a gap in computable, reference-free evaluation for multimodal CVA. LEXARA-RF applies design constraints and conversational principles as verification metrics grounded in data and context.
2 RELATED WORK
- LEXARA-RF builds on three themes: visualization design theory, automated visualization evaluation, and conversational quality in analytical dialogue.The related work distinguishes visualization expressiveness from effectiveness and situates CVA within analytical dialogue evaluation.Source: §2 · p. 1, §2.1 · p. 1, §2.3 · pp. 1–2
2.1 Visualization Design Theory
- Visualization design research distinguishes expressiveness from effectiveness and informs LEXARA-RF’s reference-free checks of design constraints.Unlike systems such as Show Me and Draco that generate visualizations, LEXARA-RF operationalizes design principles for evaluation.Source: §2.1 · p. 1
2.2 Automated Visualization Evaluation
- Existing visualization benchmarks and learned evaluation pipelines depend on reference outputs, while NLP reference-free evaluators do not address CVA’s multimodal, design-sensitive needs.LEXARA-RF bridges this gap by grounding evaluation in data, visualization design principles, and conversational alignment.Source: §2.2 · p. 1
2.3 Conversational Quality for Analytical Dialogue
- Gricean principles support evaluating analytical dialogue for truthfulness, informativeness, relevance, and clarity, but had not been systematically operationalized as computable CVA metrics.LEXARA-RF checks consistency across text, visualization, data, and conversation history without reference data at training or evaluation time.Source: §2.3 · p. 2
3 LEXARA-RF: REFERENCE-FREE EVALUATION METRICS
LEXARA-RF scores CVA outputs as intrinsic artifacts using the prompt, visualization, explanation, data, schema, and conversation history. Its verification metrics cover expressiveness, effectiveness, and conversational alignment, with fixed inherited weights and a common scoring scale.
3 LEXARA-RF: REFERENCE-FREE EVALUATION METRICS
- LEXARA-RF verifies consistency, intent alignment, and design validity from the user utterance, visualization, explanation, data, schema, and conversation history.The metrics produce graded scores on a shared 0–100 scale; the 13 dimensions and fixed weights are inherited from Lexara [23].Source: §3 · p. 2
3.1 Expressiveness of the Visualization
- Expressiveness checks whether the visualization specification is consistent with the data and schema and aligned with the query’s analytical intent.Data Fidelity re-executes the specification’s transformations against the data, while Field Similarity, Filter Accuracy, and Sort Accuracy check query-derived fields and constraints.Source: §3.1 · p. 2
3.2 Effectiveness of the Visualization
- Effectiveness metrics assess whether chart types, axes, and visual encodings suit the task and data for accurate interpretation.Axis checks apply multiplicative penalties to conventions such as positional time encoding and zero baselines; encoding checks weigh channel-to-data-type fit and query relevance.Source: §3.2 · pp. 2–3
3.3 Conversational Alignment
- Conversational alignment evaluates natural-language claims against the visualization and data, disclosure of implicit choices, insightfulness, coherence, and follow-up relevance.Overall visualization and natural-language scores are means of their constituent metrics, with per-metric scores reported for interpretability.Source: §3.3 · p. 3
4 VALIDATION OF REFERENCE-FREE CVA METRICS
Validation on human-rated CVA responses found reference-free metrics aligned with human judgments and showed descriptive parity with reference-based formulations at the current sample size. Failure analyses indicate that structural errors tend to produce localized metric signals, while semantic ambiguity produces distributed signals.
4.1 Concurrent Validity: Human Alignment
- Human alignment was moderate overall, with deterministic execution-grounded metrics strongest and rubric-based interpretive metrics lower but consistently aligned.Spearman’s ρmed was 0.60 (range 0.42–0.74); Data Fidelity, Filter Accuracy, Sort Accuracy, and Factual Grounding reached ρ = 0.68–0.74, versus ρ = 0.42–0.52 for Insightfulness and Coherence.Source: §4.1 · pp. 3–4
4.2 Convergent Validity: Reference-Based Alignment
- Reference-free metrics failed to reject parity with reference-based formulations on 11 of 13 dimensions, but N = 120 does not establish strict measurement equivalence.The mean absolute correlation difference on the other two dimensions was 0.04 ± 0.01; detecting ∆ρ = 0.05 with 80% power would require approximately N ≈400.Source: §4.2 · p. 4
- On 36 divergent cases, 44% were valid alternatives, 32% phrasing-invariant disclosures, and 24% true errors; most divergences in this subset reflected legitimate task under-specification.The validation reused 120 responses stratified by ambiguity and task types and was presented descriptively rather than as an equivalence test.Source: §4 · p. 3, §4.2 · p. 4
4.3 Diagnostic Utility: Failure Localization
- Structurally grounded failures produced localized deviations in corresponding metrics, while semantically ambiguous failures produced correlated declines across conversational alignment metrics.The diagnostic analysis used a 90-response stratified subset and treated the structurally grounded subset of 54 responses across six modes as the primary basis for localization claims; examples include isolated Filter Accuracy declines and coordinated declines in Assumptions Disclosure, Insightfulness, and Coherence.Source: §4.3 · p. 4
5 DISCUSSION: LIMITATIONS & FUTURE WORK
LEXARA-RF aligns with expert judgment and localizes structural failures, but the evidence is bounded by sample size and evaluation scope. The authors position it as a complement to reference-based evaluation, with human review still needed for subjective judgments.
5 DISCUSSION: LIMITATIONS & FUTURE WORK
- At N = 120, LEXARA-RF reaches median ρ = 0.60, matches reference-based methods, exceeds surface-similarity NLG baselines, and localizes structural failures at ≥90% accuracy; parity is not strict equivalence.The authors attribute the parity qualification to limited statistical power.Source: §5 · p. 4
- Only 24% of reference-free versus reference-based disagreements are true metric errors, while 44% are valid alternatives, suggesting many flagged outputs reflect legitimate under-specification.Per-metric decomposition also localized failure dimensions in case studies, suggesting promise for regression triage and prompt debugging.Source: §5 · p. 4
- Rubric-based metrics show lower agreement and weaker localization, while interpretive subjectivity, judge variance, and overlapping rubrics plausibly contribute to distributed signals.Proposed improvement paths include ensemble judging, calibration against held-out human ratings, and narrower verification primitives; human review remains essential for interpretive aspects.Source: §5 · p. 4
- Validation is limited to Vega-Lite-style specifications, English utterances and judge prompts, and the Lexara corpus with inherited weights; broader data and grammars are needed for calibration.The authors also identify multi-table joins and calculated fields as richer CVA tasks for future extension.Source: §5 · p. 4
- The 120-response validation corpus is available online.Supplementary materials provide reproducibility details.Source: §5 · p. 4