Source-linked AI summary
POS-Signature Demonstration Retrieval for In-Context Dependency Parsing
Muhammad Azam Afridi, Rajab Ali, Marco Passarotti
TL;DR
Semantic similarity may be a poor proxy for demonstration relevance in structured prediction, motivating a POS-signature approach to dependency-parsing demonstrations. The policy retrieves length-compatible gold-parsed examples using weighted POS statistics, and it achieves the highest LAS among tested systems on English-EWT for both models.
Problem
Semantic similarity may be a poor proxy for demonstration relevance when structured outputs depend on syntax rather than topic.
Source: §1 · p. 1Method
The policy represents sentences with weighted POS statistics and retrieves three length-compatible gold-parsed demonstrations from the labeled training pool.
Source: §1 · pp. 1–2Results
POS-Signature retrieval achieves the highest LAS for both models on English-EWT, exceeding full-pool semantic retrieval by 1.9 and 2.7 points.
Source: §7 · p. 7Takeaways & Limitations
These exploratory results favor the tested POS-Signature retrieval approach for frozen LLM dependency parsing on English-EWT.
Source: §1 · p. 2, §7 · p. 7Findings cover only two models and English-EWT with gold sentence boundaries and tokens; generalization to other languages, domains, and end-to-end parsing remains untested.
Source: Limitations · p. 7
Abstract
from arXiv · showhide
Demonstrations for in-context learning are often retrieved by semantic similarity, which may be a poor proxy for relevance in structured prediction. We study demonstration selection for Universal Dependencies parsing with a POS-signature policy that represents sentences with weighted part-of-speech statistics and retrieves three length-compatible, gold-parsed examples from the labeled training split. We evaluate two open-weight language models without fine-tuning on all 2,077 sentences of the UD English Web Treebank test set, comparing the policy with shot- and format-matched fixed and semantic three-shot baselines, as well as zero-shot, chain-of-thought, and critique-refinement systems. POS-signature retrieval yields the highest labeled attachment score among the tested prompting systems for both models (72.4 and 73.2), outperforming semantic retrieval by 1.9 and 2.7 points; paired-bootstrap 95% confidence intervals exclude zero for both contrasts. Chain-of-thought and refinement comparisons vary across models. These exploratory results favor the tested POS-signature retrieval approach on English-EWT. We release the prompts, archived outputs, and evaluation code.
1 Introduction
Semantic similarity may be a poor guide to demonstration relevance for structured prediction, motivating a POS-signature retrieval policy for dependency parsing. Across two models, it achieves the highest LAS among tested prompting systems, while the CoT comparison varies by model.
1 Introduction
- Semantic proximity may poorly predict relevance for dependency parsing because topical similarity and shared syntactic structure need not coincide.The study asks whether POS-signature retrieval improves accuracy over semantic retrieval under the same prompt scaffold and three-example budget.Source: §1 · p. 1
- POS-signature retrieval achieves the highest LAS among tested prompting systems for both models, and all four primary retrieval contrasts remain significant after Holm correction.CoT improves over zero-shot for gpt-oss but reduces accuracy for Qwen.Source: §1 · p. 2
- The POS-signature policy retrieves three length-compatible gold-parsed examples using explicit POS statistics; predicted query tags guide retrieval but are hidden from the LLM.The central comparison tests complete policies: POS retrieval filters by length, while semantic retrieval searches the full training pool.Source: §1 · pp. 1–2
- The evaluation compares six prompting conditions across the full English-EWT test split, including three shot- and format-matched demonstration policies.It also includes zero-shot, chain-of-thought, and critique-refinement systems.Source: §1 · pp. 1–2
- The study releases prompts, archived outputs, evaluation code, and reproduction instructions.Reported results can be reproduced from archived outputs without additional model API calls; findings are scoped to two hosted models, one English treebank, gold tokenization, and labeled in-domain demonstrations.Source: §1 · p. 2
2 Related Work
Prior work applies syntactic or coverage-based selection to other structured tasks, while dependency-parsing retrieval comparisons remain limited. This study uses sentence-level POS statistics and compares the resulting policy with fixed and semantic retrieval; reasoning and refinement are broader system comparisons.
2 Related Work
- The closest cited retrieval study selects Arabic morphosyntactic-tagging and dependency-parsing examples by chrF++ overlap or sentence embeddings, tuning demonstration count on development data.This study instead fixes k = 3 and compares POS-signature retrieval with a matched sentence-embedding policy for English UD.Source: §2 · p. 2
- Prior selectors use dependency-subtree coverage for semantic parsing and syntactic plus lexical coverage for translation, unlike this study’s sentence-level POS statistics for dependency-tree prediction.The study holds demonstration count and serialization constant against fixed selection and Sentence-BERT retrieval.Source: §2 · p. 2
- CoT and iterative refinement are evaluated as broader prompting systems because they are not factor-matched retrieval baselines.Prior work reports that self-refinement can repair or introduce errors without external feedback.Source: §2 · p. 2
3 Method
The method evaluates supervised in-context dependency parsing with shared prompt conditions, then compares POS-signature retrieval against semantic retrieval. The signature combines weighted POS statistics and length compatibility; both policies supply three gold-parsed examples.
3.1 Task and Resource Setting
- The task predicts each token’s head and dependency relation from tokenized input; demonstrations contain complete gold parses, so the setting is supervised in-context parsing.The query’s gold POS tags, lemmas, morphological features, heads, and relations are withheld.Source: §3.1 · p. 2
3.2 Prompting Conditions
- Three few-shot policies share the task instruction, three-example budget, and demonstration format, whereas CoT and Critique-Refine alter example count, response format, or calls.Zero-Shot uses only the shared instruction; Fixed uses the same three manually selected examples, and POS-Signature and Semantic retrieve query-specific examples.Source: §3.2 · p. 3
3.3 POS-Signature Demonstration Retrieval
- POS-Signature represents sentences with weighted, globally normalized feature blocks for length, POS composition, and adjacent POS transitions, then applies length-compatible candidate filtering and selects three gold-parsed examples.Query signatures use Stanza-predicted UPOS, which is not shown to the LLM; the hand-specified weights are not learned or optimized.Source: §3.3 · p. 3
3.4 Semantic Retrieval Baseline
- Semantic Few-Shot ranks the full training pool by sentence-embedding cosine similarity and selects three examples without a length filter, so the comparison varies both representation and candidate pool.The three retrieval policies use a shared demonstration format.Source: §3.4 · p. 4, Table 1 · p. 4
4 Experimental Setup
The experiments use the complete English-EWT test split with gold-tokenized input and two hosted open-weight models. LAS is the primary metric under a fixed-denominator protocol, with paired sentence-bootstrap comparisons and a supervised parser as context.
4.1 Data
- The evaluation uses all 2,077 English-EWT test sentences and scores 21,998 non-punctuation tokens after excluding gold PUNCT tokens.Models receive gold-tokenized input, isolating dependency parsing from segmentation.Source: §4.1 · p. 4
4.2 Models and Inference Configuration
- The study evaluates gpt-oss-120b and Qwen2.5-72B-Instruct without fine-tuning, using temperature 0, top-p 1, seed 42, and a 6,000-token completion limit.Hosted inference is not guaranteed deterministic despite settings intended to reduce sampling variation; the two systems do not represent the broader LLM landscape.Source: §4.2 · p. 4
4.3 Metrics and Statistical Analysis
- LAS is primary and UAS secondary over a fixed denominator of 21,998 non-punctuation gold tokens; paired sentence-bootstrap inference uses 10,000 replicates for each contrast.The scoring protocol uses gold tokenization, retains relation subtypes, and differs from official CoNLL 2018 scoring.Source: §4.3 · pp. 4–5
4.4 Supervised Reference
- The supervised Stanza 1.11.1 biaffine parser scores 90.7 UAS and 88.0 LAS under the same evaluator and denominator, serving as a contextual rather than resource-matched baseline.The four focal retrieval comparisons are corrected jointly using Holm correction at α = .05.Source: §4.4 · p. 5, §4.3 · p. 5
5 Results
Across both models, POS-Signature Few-Shot leads the prompting systems, while CoT and refinement comparisons vary by model. Output compliance, relation-level patterns, sentence-length results, and other post-hoc diagnostics qualify these aggregate comparisons.
5.1 Overall Performance and Primary Contrasts
- 72.4 LAS on gpt-oss and 73.2 LAS on Qwen make POS-Signature Few-Shot the strongest prompted condition; Semantic Few-Shot reaches 70.5 LAS on both.Against Semantic Few-Shot, the LAS gains are 1.87 points on gpt-oss and 2.68 on Qwen; paired-bootstrap intervals exclude zero.Source: §5.1 · p. 5, Table 2 · p. 6
5.2 CoT and Refinement System Comparisons
- CoT gains 5.30 LAS points over Zero-Shot on gpt-oss but loses 1.18 on Qwen; refinement gains over CoT are 0.44 and 1.90 points, respectively.For Qwen, the Critique-Refine versus Zero-Shot difference is +0.72 points, with a confidence interval including zero; three missing second-pass records followed unscorable CoT outputs.Source: §5.2 · p. 5
5.3 Output Compliance
- Schema-valid rates are 99.7–100%; root warnings remain token-level scorable, and output compliance is distinct from attachment accuracy.The cited passages distinguish compliance from accuracy and note that the appendix reports a stricter, nonscoring tree audit.Source: §5.3 · pp. 5–6
5.4 Post-Hoc Diagnostic Analyses
- The post-hoc diagnostics localize aggregate differences by relation, sentence length, and refinement behavior, but do not establish causal mechanisms.These analyses are exploratory rather than tests of why the observed differences occur.Source: §5.4 · p. 6
5.4.1 Relation-Level Localization
- For nmod:poss, POS-Signature exceeds Semantic retrieval by 13.9 LAS points on gpt-oss and 23.0 on Qwen; nmod:unmarked has only 111 tokens of support.Only the retrieval policies obtain nonzero LAS on nmod:unmarked, so that pattern warrants caution.Source: §5.4.1 · p. 6
5.4.2 Sentence Length and Refinement Behavior
- Critique-Refine is numerically above POS-Signature in two gpt-oss length bins, while POS-Signature leads every Qwen bin; refinement yields net gains of 97 and 418 LAS-correct tokens.The gains are for gpt-oss and Qwen, respectively; refinement changes similar token shares of 10.3% and 9.6%.Source: §5.4.2 · p. 6
6 Discussion
The discussion treats POS-signature advantages as suggestive rather than mechanistically established, and CoT/refinement comparisons as model- and system-dependent. It also distinguishes schema compliance, structural validity, and parsing accuracy.
6.1 Interpreting the Retrieval-Policy Advantage
- POS composition, adjacent POS transitions, and sentence length may help identify constructions, but signatures omit head attachments and dependency labels, and similar signatures can map to different structures.The larger LAS than UAS advantage is consistent with demonstrations helping apply UD head and relation-label conventions; lexical-semantic information may complement POS patterns for some relations.Source: §6.1 · p. 6
6.2 Interpreting CoT and Refinement Comparisons
- These interpretations remain hypotheses: the comparison does not isolate POS features from length filtering, predicted-POS errors, or incidental lexical overlap, and the advantage is specific to tested models and data.CoT is a bundled system comparison across one model per family, and refinement can repair errors or overwrite correct arcs; some refinement contrasts remain uncertain.Source: §6.1 · p. 6, §6.2 · p. 7
6.3 Implications for Evaluating LLM Parsers
- Schema validity does not guarantee parsing success: tree-conforming rates are 96.5–98.4% for gpt-oss and 96.9–99.1% for Qwen, while best LAS is only 72–73%.The discussion recommends reporting serialization reliability, structural validity, and parsing accuracy separately; gpt-oss POS-Signature Few-Shot has the highest LAS but the lowest tree-conforming count.Source: §6.3 · p. 7
7 Conclusion
On English-EWT, POS-Signature retrieval has the highest LAS among the six prompting systems for both tested models, exceeding semantic retrieval by 1.9 and 2.7 points. These exploratory results favor the tested approach, but do not isolate POS features from length filtering or establish an alternative to supervised parsing.
7 Conclusion
- POS-Signature retrieval achieves the highest LAS for both models on English-EWT, exceeding full-pool semantic retrieval by 1.9 and 2.7 points.The reported outcome is explicitly exploratory.Source: §7 · p. 7
- The approach requires a supervised tagger and labeled demonstrations, and the study does not isolate POS features from length filtering or establish an alternative to supervised parsing.The authors identify matched ablations as still needed.Source: §7 · p. 7
Limitations
The retrieval comparison does not isolate individual policy components, and the evidence is limited to two models on English-EWT with gold sentence boundaries and tokens. Diagnostics and attachment scores also leave important questions about generalization, structural validity, and downstream utility open.
Limitations
- POS-Signature versus Semantic retrieval compares complete policies, so representation and candidate filtering effects are not isolated.The retrieval components and hyperparameters were not ablated; POS-Signature also relies on a supervised tagger and gold-annotated training demonstrations.Source: Limitations · p. 7
- Results cover two models and English-EWT with gold boundaries and tokens, leaving transfer to other languages, domains, and end-to-end parsing untested.The authors note that English word order and limited inflection may make local POS patterns informative in ways that do not transfer.Source: Limitations · p. 7
- Post-hoc diagnostics are exploratory, pretraining exposure to public EWT cannot be excluded, and token-level attachment scores do not establish tree validity or downstream utility.Structural validity is assessed separately.Source: Limitations · p. 8
Ethical Considerations
The appendix documents the study’s ethical handling, prompts, retrieval policies, scoring, and pairwise-comparison procedures. The released materials include prompts and archived outputs, while the reported evaluation uses a fixed test set and denominator.
Ethical Considerations
- The public UD English-EWT source text may contain identifying information; inputs and demonstrations were processed through DeepInfra and archived without additional de-identification.The artifact retains the dataset’s license and source notices, and generated parses are predictions rather than verified annotations.Source: Ethical Considerations · p. 8
A Prompt Templates / A.1 Shared Task Instruction
- The prompt appendix preserves shared task instructions and condition-specific templates; retrieval conditions differ in selected examples while sharing their base instruction and demonstration format.The shared instructions specify UD parsing and JSON output; CoT permits reasoning before the final parse.Source: A Prompt Templates · pp. 9–10, §A.1 · p. 10
A.2 Query and Demonstration Templates
- Queries use sentence text and token IDs/forms, while retrieval prompts append three demonstrations containing training sentences and gold parses.Both retrieval conditions use the same demonstration format.Source: §A.2 · p. 10
A.3 Critique-Refine Messages
- Critique-Refine reviews the extracted CoT parse in a separate message, with the extracted token array included in its template.Source: §A.3 · pp. 10–11
B Retrieval-Policy Comparison
- POS-Signature uses weighted POS statistics and length filtering, whereas Semantic retrieval uses sentence embeddings and searches the full training pool; both provide three gold-parsed examples.Their comparison evaluates complete policies rather than POS representation alone.Source: B Retrieval-Policy Comparison · p. 11
C Additional Results
- Corpus-level comparisons use 2,077 sentences and 21,998 non-punctuation gold tokens, assigning zero credit to missing or unscorable predictions without reducing the denominator.The appendix also reports diagnostic relation results and comparison tables.Source: C Additional Results · p. 11
C.1 Scoring and Statistical Tests
- Scoring defines UAS by correct heads and LAS by correct heads plus full relation labels; paired sentence-bootstrap samples estimate differences and pointwise confidence intervals.The bootstrap uses 10,000 paired samples, and the approximate p-value uses add-one smoothing and counts zero differences in both tails.Source: §C.1 · p. 11
C.2 Pairwise Comparisons
- Table 4 reports all 15 pairwise LAS comparisons per model, while distinguishing within-model Holm correction from the joint correction used for focal comparisons.Three absent Qwen Critique-Refine outputs remain in the full test-set denominator with zero credit.Source: §C.2 · p. 11
C.3 Results by Dependency Relation
The relation-level diagnostics show that POS-Signature’s advantage over Semantic retrieval is not uniform. These descriptive comparisons do not include relation-level significance tests.
C.3 Results by Dependency Relation
- The diagnostics report six relations with shared gold-token counts, but provide no relation-level significance tests.Source: §C.3 · p. 11
- Qwen performs better with Semantic retrieval on obl:unmarked, despite POS-Signature improving on Semantic retrieval for several other relations.Source: §C.3 · p. 11
- The two unmarked relations have small counts, so their relation-level comparisons warrant caution.Source: §C.3 · p. 11
C.4 Results by Sentence Length
Sentence-length results differ by model; refinement changes are more often beneficial for Qwen, while output validity is distinct from tree conformity.
C.4 Results by Sentence Length
- For gpt-oss, POS-Signature leads in the two shortest bins, while Critique-Refine is higher in the 21–30 and 31–50 bins; POS-Signature leads every Qwen bin.Length is measured in non-punctuation gold tokens; the longest bin contains only 11 sentences, and the results are descriptive without an interaction test or simultaneous confidence intervals.Source: §C.4 · pp. 11–12
C.5 Refinement Changes
- Both models change similar proportions of predictions, but Qwen introduces fewer new errors; transition counts use 21,801 tokens, versus 21,998 for full-corpus gains.An edit changes a predicted head, relation label, or both; a prediction is correct only when both fields match gold.Source: §C.5 · p. 12, Table 7 · p. 13
C.6 Output and Tree Validity
- Table 8 reports schema and tree validity; tree conformity additionally requires legal heads, one root, no self-loops, and paths from every token to the root.Qwen had six malformed Zero-Shot outputs, one malformed Fixed output, one malformed POS-Signature output, and three truncated CoT outputs; the latter prevented three Critique-Refine calls.Source: §C.6 · p. 13, Table 8 · p. 14