How Reliable are AMR Parser Outputs? Uncertainty Estimation for AMR Parsing

1 Introduction

• AMR is a rooted semantic graph representation of sentence meaning, supporting applications including information extraction, question answering, summarisation, multilingual parsing, and trustworthy AI.Manually constructing AMR graphs is expensive and difficult to scale, motivating automatic parsers.Source: §1 · p. 1
• Despite improved AMR parsers, uncertainty estimation for AMR remains largely unexplored, while graph similarity metrics mainly measure aggregate performance against gold annotations.Those metrics offer limited insight into individual prediction reliability or whether confidence aligns with actual performance.Source: §1 · p. 1
• The study asks whether parsing performance and uncertainty correlate, how reliable beam candidates are, whether one metric suffices, and which parser balances performance and reliability.It presents a systematic black-box study of uncertainty estimation for AMR parsing.Source: §1 · p. 1

2 Problem Definition

• For each input sentence, the parser produces a set of k beam candidates, denoted C_i, with the top-1 candidate serving as its prediction.The dataset comprises sentence–gold-graph pairs; s_i denotes the sentence and g*_i its gold AMR graph.Source: §2 · p. 2
• In the black-box setting, parsing maps an input sentence to its candidate set without exposing model probabilities or internal states.The uncertainty estimator maps the candidate set to one scalar score per sentence.Source: §2 · p. 2
• Prediction quality is computed between the top-1 prediction and its gold graph, yielding a quality vector across the dataset.A desirable estimator assigns greater uncertainty to lower-quality predictions and maximizes PRR.Source: §2 · p. 2

3.1 Pairwise Similarity Matrix

• Pairwise similarities among beam candidates form a per-sentence matrix that supplies the input to uncertainty estimation.The matrix entry for a candidate pair is its AMR similarity score in [0, 1].Source: §3 · p. 2, §3.1 · p. 2

3 Methodology

• Across the pipeline, candidate agreement supplies uncertainty scores and PRR tests how effectively they identify low-quality outputs.The approach is conducted in a black-box setting without access to model probabilities or internal states.Source: §3 · p. 2, §3.3 · p. 3

3.2 Uncertainty Estimation

• Uncertainty estimators map candidate sets through similarity matrices to a scalar score, with greater candidate disagreement indicating greater uncertainty.The methods include lexical ROUGE-L similarity (Kuhn et al., 2023), average agreement, and graph-based spectral measures; the normalized Laplacian represents candidate connectivity, while Ecc measures spread in its eigenspace.Source: §3.2 · pp. 2–3

3.3 Prediction Rejection Evaluation

• PRR assesses whether rejecting the most uncertain predictions improves the mean quality of those remaining.It normalizes area under the prediction–rejection curve against oracle and random rejection; PRR = 1 denotes oracle performance, 0 random rejection, and values below 0 harmful uncertainty estimates.Source: §3.3 · p. 3

4.1 Experimental Setup

• On AMR 3.0’s test set, the study evaluates six parsers over 1,898 sentence–graph pairs using beam sizes k ∈ {3, 5, 8, 10} and five graph-similarity measures: Smatch, Smatch++, S2Match, SEMA, and WLK.Parsers were selected based on reported performance and code availability; LLM baselines were excluded due to substantially lower parsing performance.Source: §4.1 · p. 3

4.2 Results and Analysis

• ρ = 0.388 for Smatch++, ρ = 0.317 for S2Match, and ρ = 0.360 for SemBLEU; Smatch++ has the strongest performance–PRR correlation.MBSE and HMG-AMR-BART have mean PRRs of 0.183 and 0.154 under Smatch++, despite near-identical parsing accuracy.Source: §4.2 · p. 4
• PRR declines for most parsers as beams grow; HMG-AMR-T5 falls below zero at k = 10, while LEXSIM alone consistently improves and ECC degrades steepest.Moderate beam sizes provide the strongest trade-off between candidate diversity and uncertainty quality.Source: §4.2 · p. 4, Figure 2 · p. 4
• PRR varies by evaluation metric: transition-based parsers are stronger under S2Match than Smatch++, while estimator agreement also shifts between Smatch++ and SemBLEU.Source: §4.2 · pp. 4–5, Figure 3 · p. 5
• SPRING has the strongest overall balance, with the highest mean PRR (0.265) and positive uncertainty estimates across beam sizes and evaluation metrics.IBM-Parser is the most stable across metrics, while HMG-AMR-T5 degrades most steeply as beam size grows.Source: Figure 1 · p. 4, Figure 2 · p. 4, §4.2 · p. 5

4.3 Additional Analysis

• Short sentences receive higher mean uncertainty, but PRR is highest for medium-length sentences, followed by long and short sentences; graph-based estimators strongly agree.Reliability varies substantially with beam size across parsers, and no estimator is strongest for every parser at larger beams.Source: §4.3 · p. 5

• More accurate parsers generally yield more reliable uncertainty estimates, although reliability also varies with beam size and parser architecture.Source: §5 · p. 5
• Moderate beam sizes tend to yield more reliable uncertainty estimates than larger beams, and reliability depends on the evaluation metric.Source: §5 · p. 5
• Considering parsing accuracy and PRR together provides a more comprehensive assessment because parsers with similar accuracy can have substantially different PRR.Source: §5 · p. 5

Limitations

• The analysis uses beam-search candidates as its sole uncertainty source; alternative decoding strategies remain untested and may yield different candidate distributions and reliability characteristics.Beam decoding provides a consistent setting across models because most publicly available AMR parsers use it.Source: Limitations · p. 6
• AMR quality metrics capture distinct graph properties and can lead to different conclusions about uncertainty, leaving uncertainty-aware evaluation an open problem.The study evaluates uncertainty quality with PRR alongside existing AMR quality metrics.Source: Limitations · p. 6
• The study estimates uncertainty post hoc without changing parser architectures or training objectives; integrating it into training or decoding may further improve reliability.Source: Limitations · p. 6

Ethical Considerations

• The authors report using ChatGPT for writing and language refinement, Claude Code to accelerate research workflows, and Grammarly for proofreading; they say authors verified the final content and research.Source: Ethical Considerations · p. 6

A.1 Algorithm

• Algorithm 1 summarizes the complete procedure.Source: §A.1 · p. 8

A.2 Computational Complexity

• Pairwise similarity computation is the framework’s dominant cost, requiring O(k^2) graph comparisons per sentence, while estimator and PRR calculations add negligible overhead.The overall runtime across parsers, beam configurations, and sentences is O(P · B · N · k^2 · Csim).Source: §A.2 · p. 8

B Experiment Setup

• Appendix B provides implementation details intended to support reproduction of the experiments.Source: B Experiment Setup · p. 8

B.1 Dataset

• The dataset is the AMR 3.0 test set, containing 1,898 sentence–graph pairs, and all uncertainty experiments use parser outputs from this split.Source: §B.1 · p. 8

B.2 AMR Parsers

• Six publicly available AMR parsers are evaluated using official implementations and released pretrained checkpoints without architecture changes or retraining.Two instruction-tuned LLM baselines are included for reference but excluded from uncertainty analyses because of substantially lower AMR 3.0 performance.Source: §B.2 · pp. 8–9

B.3 Similarity Construction and Evaluation

• The framework constructs candidate similarities with four AMR-specific metrics and evaluates prediction quality for PRR with Smatch++, S2Match, and SemBLEU.Source: §B.3 · p. 9, §B.4 · p. 9

B.4 Uncertainty Estimation and Evaluation

• Uncertainty estimators include LEXSIM’s pairwise lexical-overlap approach and three graph-based approaches: EIGV, DEGMAT, and ECC.The estimators derive uncertainty from candidate diversity, graph connectivity, pairwise similarity, or embedding dispersion.Source: §B.4 · p. 9

B.5 Hardware and Runtime

• The complete evaluation pipeline across six parsers took approximately 150 hours of wall-clock time.Generating candidates and computing uncertainty metrics took approximately 25 hours per parser across beam sizes k ∈ {3, 5, 8, 10}.Source: Table 2 · p. 10, §B.5 · p. 10

C.1 AMR Parsing Performance

• SPRING leads the reported pre-BLINKification Smatch comparison at 0.8449, followed by MBSE at 0.8374 and HMG-AMR-BART at 0.8358.The instruction-tuned Qwen3-32B and Ministral-14B baselines perform substantially worse and are excluded from later uncertainty analyses.Source: Table 3 · p. 10, §C.1 · p. 10

C Additional Results

• The appendix’s additional-results analyses cover parser-level performance, the relationship between parsing quality and uncertainty, and estimator stability across beam sizes.Source: C Additional Results · p. 10

C.2 Relationship Between Parsing Performance and Reliability

• 0.388 for Smatch++, 0.317 for S2Match, and 0.360 for SemBLEU are the reported correlations between parser accuracy and mean PRR, but accuracy explains only part of uncertainty quality.PRR also varies across beam sizes for parsers with the same fixed top-1 accuracy, and its relationship with accuracy differs by evaluation metric.Source: §C.2 · p. 10, Figure 4 · p. 11

C.3 Uncertainty Estimator Stability Across Beam Sizes

• Mean PRR is highest at k = 3 and generally declines as beam size increases, with the sharpest drops between k = 5 and k = 8.Source: §C.3 · p. 11
• ECC-S2MATCH falls from 0.345 at k = 3 to 0.141 at k = 10, while EIGV variants also show substantial declines across beam sizes.EIGV variants drop from approximately 0.30–0.34 at k = 3 to 0.14–0.20 at k = 10, depending on the metric.Source: §C.3 · p. 11
• LEXSIM is the only estimator whose mean PRR rises with beam size, from 0.149 to 0.171 under Smatch++ and from 0.187 to 0.215 under S2Match.At k = 10, DEGMAT-SEMA retains PRR values of 0.182 under Smatch++ and 0.220 under S2Match.Source: §C.3 · p. 11
• S2Match yields higher mean PRR than Smatch++ across estimators and beam sizes; EIGV-S2MATCH scores 0.335 versus 0.301 at k = 3.The analysis identifies k = 3 to k = 5 as the strongest trade-off between candidate diversity and uncertainty-estimation reliability.Source: §C.3 · p. 12

C.4 Agreement Between UE

• Graph-based estimators EIGV, DEGMAT, and ECC agree strongly: most Smatch++ pairwise correlations exceed ρ = 0.87, with some pairs reaching ρ = 1.00.Agreement is also high under S2Match, where several pairs reach ρ = 1.00, and under SemBLEU, where most graph-based pairs have ρ ≥ 0.85.Source: §C.4 · p. 12, Figure 8 · p. 14
• LEXSIM correlates more weakly with graph-based estimators, ranging from ρ = −0.16 to ρ = 0.31 under Smatch++ and reaching ρ = −0.14 under SemBLEU.The results indicate that lexical overlap captures an uncertainty signal distinct from structural graph agreement, with the difference varying by metric.Source: §C.4 · pp. 12–13
• 0.335 for EIGV-S2MATCH versus 0.301 under Smatch++ at k = 3; changing the evaluation metric alters both PRR and estimator agreement.Under SemBLEU, WLK-based estimators show stronger agreement with other graph-based methods (ρ ≥ 0.85), reducing the separation seen under Smatch++ and S2Match.Source: §C.4 · p. 13

C.5 Parser Reliability Across Beam Sizes

• SPRING retains positive PRR across estimators and beam sizes under S2Match and SemBLEU, and PRR above 0.18 at k = 10 under Smatch++ except ECC.These results indicate robustness to candidate expansion across the evaluated metrics.Source: Figure 7 · p. 13, §C.5 · p. 14, Figure 9 · p. 15
• IBM-Parser is the most stable across beam sizes, with nearly flat PRR from k = 3 to k = 10 under all metrics; LEXSIM remains its strongest estimator.IBM-Neural shows a similar stability pattern, with LEXSIM highest at larger beams.Source: §C.5 · p. 14
• HMG-AMR-T5 degrades steeply: graph-based estimators fall below zero by k = 8 under Smatch++, and all estimators have negative PRR at k = 8 and k = 10 under SemBLEU.HMG-AMR-BART shows similar sensitivity, with ECC below zero at k = 10 under SemBLEU.Source: §C.5 · p. 14
• MBSE has competitive PRR at k = 3 across metrics but degrades at larger beams; under S2Match, ECC falls to ≈0.12 at k = 10, below LEXSIM and DEGMAT.The results distinguish maintaining reliable estimates as candidate diversity increases from having the highest parsing accuracy.Source: §C.5 · p. 14

C.6 Detailed Uncertainty Estimation Results

• At k = 10, DEGMAT-SEMA leads for SPRING and both HMG-AMR parsers, while LEXSIM leads for IBM-Parser, IBM-Neural, and MBSE; no estimator is optimal across all parsers.For SPRING, DEGMAT-SEMA scores 0.271 under Smatch++ and 0.247 under SemBLEU; the reported LEXSIM values are 0.266 and 0.162 for IBM-Parser, 0.266 and 0.169 for IBM-Neural, and 0.229 and 0.133 for MBSE.Source: §C.6 · pp. 15–16
• 0.137 under Smatch++ and 0.109 under SemBLEU for HMG-AMR-BART, and 0.069 and 0.023, respectively, for HMG-AMR-T5, are the strongest reported PRRs for those parsers, achieved by DEGMAT-SEMA.The estimator with the strongest reliability signal at k = 10 therefore varies across parsers.Source: §C.6 · p. 16
• ECC-based methods perform weakest overall at k = 10, including SemBLEU PRRs of −0.025 for HMG-AMR-BART and −0.148 for HMG-AMR-T5.The passage suggests eigenspace-based candidate spread becomes less informative as beam size increases for these parsers.Source: §C.6 · p. 16