PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Verifiable Text Generation with Symbolic References

Lucas Torroba Hennigen, Shannon Shen, Aniruddha Nrusimha, Bernhard Gapp, David Sontag, Yoon Kim

arXiv:2311.09188v2cs.CLcs.AIcs.LG

TL;DR

LLM-generated text from structured data can remain imperfect, making manual verification important in high-stakes settings. SymGen interleaves prose with references to input fields; a human study found that annotations reduced average verification time by 20%.

  • Problem

    LLMs can generate fluent and faithful-enough text from structured data but remain imperfect, while high-stakes applications require manual verification against the conditioning information.

    Source: §1 · p. 1

  • Method

    SymGen prompts an LLM to interleave output text with symbolic references to fields in structured input, then renders those references as values with provenance cues.

    Source: §1 · p. 1

  • Results

    Experiments indicate that LLMs generate fluent text interleaved with field references, and SymGen annotations reduced average human verification time by 20%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluation suggests symbolic generation increases verifiability without degrading quality.

    Source: §8 · p. 10

    SymGen cannot completely prevent hallucinations: surrounding context may be incorrect even when symbolic references are accurate, yielding verifiable but incorrect text.

    Source: §6 · p. 9

Abstract

from arXiv · show

LLMs are vulnerable to hallucinations, and thus their outputs generally require laborious human verification for high-stakes applications. To this end, we propose symbolically grounded generation (SymGen) as a simple approach for enabling easier manual validation of an LLM's output. SymGen prompts an LLM to interleave its regular output text with explicit symbolic references to fields present in some conditioning data (e.g., a table in JSON format). The references can be used to display the provenance of different spans of text in the generation, reducing the effort required for manual verification. Across a range of data-to-text and question-answering experiments, we find that LLMs are able to directly output text that makes use of accurate symbolic references while maintaining fluency and factuality. In a human study we further find that such annotations can streamline human verification of machine-generated text. Our code will be available at http://symgen.github.io.

1 Introduction

LLMs can generate fluent and reasonably faithful text from structured data, but remain imperfect, leaving manual verification necessary for high-stakes applications. SymGen adds explicit links to source fields, and a human study found that its annotations reduced average verification time by 20%.

1 Introduction

  • High-stakes applications need fluent, accurate, and verifiable text from structured data, yet LLMs remain imperfect and their outputs may require manual checking.Examples include generating sports summaries from official statistics and grounding search output in search results.Source: §1 · p. 1
  • SymGen prompts an LLM to interleave output text with symbolic references to structured-data fields, which a parser renders as values and can expose as provenance cues.Unlike classic templated generation, SymGen offloads template specification to the LLM.Source: §1 · p. 1
  • A human study found that SymGen annotations reduced average verification time by 20%, while users also perceived verification as easier.Experiments also indicate that LLMs can produce fluent text interleaved with references to data fields.Source: §1 · p. 2

2 Text generation with symbolic references

SymGen prompts an LLM to produce prose interleaved with references to fields in structured input, then renders those references as their corresponding values. The paper describes direct and indirect prompting variants and evaluates text quality, reference accuracy, human verifiability, and extensions to question answering and mathematical reasoning.

2 Text generation with symbolic references

  • Figure 1 contrasts a standard LLM description with SymGen text whose highlighted spans link to source data, including a hoverable value tooltip.The example shows the number 30 linked to the value it references.Source: Figure 1 · p. 2
  • SymGen interleaves regular text with field references in a symbolic response, then substitutes each reference with its corresponding data value to render the final text.The references provide fine-grained attribution from text spans to source fields.Source: §2 · p. 2
  • Direct SymGen prompts the LLM to use Jinja-like field syntax in its output, whereas Indirect SymGen first generates regular text and then converts it into referenced text.The indirect variant conditions the symbolically grounded response on the initial text as a plan.Source: §2 · p. 2
  • Figure 2 illustrates the pipeline: a prompted LLM generates symbolic references, and a parser renders them while allowing prose without references and computed values from existing variables.The example includes a reference to visitor.city rendered as Portland and a computation combining two values.Source: Figure 2 · p. 3
  • Indirect SymGen can require roughly twice as many generated tokens and was unreliable with GPT-3.5, so the authors use it only with GPT-4.Its expected text-quality advantage depends on conversion being no harder than generating regular prose.Source: §2 · p. 3
  • The evaluation compares rendered-text quality with direct LLM generation, tests reference accuracy and human verifiability, and explores question answering and mathematical reasoning.Across the studies, the paper evaluates five datasets against a baseline without symbolic references.Source: §2 · p. 3

3 Textual quality study

The textual-quality study tests whether symbolic generation preserves baseline text quality across data-to-text and counterfactual obituary tasks. Results vary by dataset and setting, with rendering errors and unanswered questions qualifying performance.

3 Textual quality study

  • The study compares zero- and few-shot generation with prompted LLM baselines, assessing fluency on data-to-text tasks and factuality on counterfactual obituaries.SynthBio uses fictional entities; Rotowire summarizes basketball games from player box and team line scores, with 100 test examples sampled to manage API costs.Source: §3 · p. 3, §3.1 · p. 4, §3.2 · p. 5

3.1 Data-to-text

  • On SynthBio, both few-shot SymGen strategies match or exceed the baseline, while Rotowire SymGen generally trails it, with a smaller gap for GPT-4-32K indirect generation.For SynthBio, direct GPT-3.5-4K underperforms zero-shot, but few-shot examples bridge the gap.Source: §3.1 · pp. 4–5, Table 1 · p. 4
  • Rendering errors occur in many poor symbolic generations; GPT-3.5 and indirect SymGen produce more errors, while few-shot examples tend to reduce them.The authors attribute indirect errors partly to the difficulty of inserting references when JSON is incomplete or prose is not amenable to them.Source: Table 1 · p. 4, §3.1 · p. 5

3.2 Counterfactual text generation

  • The counterfactual dataset contains 167 famous scientists and altered variants designed to test whether models ignore data contradicting learned priors.GPT-3.5-4K obituaries are evaluated by asking schema-property questions using only the rendered text.Source: §3.2 · p. 5
  • In the counterfactual obituary task, SymGen slightly outperforms the baseline zero-shot but slightly underperforms few-shot; its higher unknown rate suggests a precision–recall trade-off.When conditioning on whether the model answered or returned “Unknown,” performance improves zero-shot and is comparable few-shot.Source: Table 2 · p. 5, §3.2 · p. 6

4 Verifiability Study

The study tests whether SymGen references are contextually accurate and whether their provenance annotations help people verify generated summaries. References were highly accurate, and annotations shortened verification time without changing error-detection likelihood.

4.1 Assessing symbolic reference accuracy

  • Because accurate rendered text can still cite the wrong field, references were judged for contextual correctness and compared with a regex baseline that matches values in text to fields.The evaluation sampled 20 Rotowire test examples using GPT-3.5-16K or GPT-4-32K.Source: §4.1 · p. 6
  • 99.77% accuracy with GPT-3.5-16K and 99.52% with GPT-4-32K exceeded regex-baseline accuracy of 35.40% and 46.10%, respectively.Three authors annotated 9,068 references, with 97.8% inter-annotator agreement.Source: §4.1 · p. 6

4.2 Human evaluation of improved verifiability

  • The human study used manually checked Rotowire summaries and compared annotated with unannotated versions while participants judged summary correctness and rated confidence and ease.Participants inspected provenance by hovering over numbers; 60 annotators were recruited.Source: §4.2 · pp. 6–7
  • 20% less time was needed to decide whether an error was present with annotations, while annotators were equally likely to identify errors.Annotators also felt 5% more confident and perceived verification as 14% easier.Source: §4.2 · p. 7

5 Extensions to other use-cases

The paper extends SymGen to structured-data question answering and mathematical reasoning. In math tasks, symbolic generation performs similarly to CoT on GSM8K and better on GSM-hard, though it trails PAL there.

5.1 Question answering over structured data

  • With GPT-4, financial QA answer quality is reasonably comparable across generation strategies, while SymGen underperforms when using GPT-3.5.The dataset contains 32 finance-related questions paired with structured company information.Source: §5.1 · p. 7, Table 3 · p. 8

5.2 Mathematical reasoning

  • SymGen interleaves natural-language reasoning with Jinja assignments that encode each computation step, unlike PAL and PoT, which recast problems as programs and execute them.This approach embeds symbolic computation within regular text as a scaffold.Source: §5.2 · p. 8, Figure 3 · p. 8
  • The math evaluation compares direct SymGen with CoT and PAL on GSM8K and GSM-hard, using 200 randomly selected GSM8K test problems and a corresponding GSM-hard testbed.GSM-hard replaces a randomly selected number in each GSM8K problem with a large random number of up to seven digits.Source: §5.2 · pp. 8–9
  • On GSM-hard, SymGen outperforms CoT with GPT-4-8K and GPT-3.5-4K but trails PAL, especially with GPT-3.5-4K; on GSM8K, SymGen and PAL perform similarly to CoT.The authors report that some SymGen errors were sensible but incorrect attempts at Jinja-based symbolic reasoning.Source: §5.2 · p. 9

6 Discussion & limitations

SymGen can produce accurate symbolic references without ensuring that the surrounding text is free of hallucinations. The studies primarily use closed-source models, with open-source models and fine-tuning left for future work.

6 Discussion & limitations

  • Accurate symbolic references do not ensure that surrounding context text is correct, so generations can be verifiable yet incorrect.The authors identify this as a limitation of SymGen.Source: §6 · p. 9
  • The current studies primarily focus on closed-source models; extending SymGen to open-source models and fine-tuning for templated generation remain future work.The authors note that, to their knowledge, training data for this task are not currently available.Source: §6 · p. 9

7 Related work

Related work spans template-based data-to-text generation, tool-augmented LLMs, and automatic verification. SymGen instead uses structured data to improve the verifiability of generated text.

7 Related work

  • Handcrafted data-to-text templates interleave prose with data-field placeholders and can strictly prevent hallucinations, but require manual, domain-specific work and may sound robotic.Kukich (1983), McKeown (1992), and McRoy et al. (2000) are cited for classic template approaches.Source: §7 · p. 9
  • Neural data-to-text methods use large datasets, while template-like neural methods still generally require hundreds or thousands of training examples, unlike this work’s zero- and few-shot settings.The passage traces approaches from induced templates to end-to-end neural networks and cites Wiseman et al. (2018), Ye et al. (2020), Li & Rush (2020), Wang et al. (2021), Puduppully et al. (2022), and Zhang et al. (2022) for template-like neural structures.Source: §7 · p. 9
  • Tool-augmented LLM work typically uses external modules to enhance capabilities, whereas SymGen treats structured data as a tool for improving verifiability.The passage also connects arithmetic computations in generations to work combining symbolic computations with language-based reasoning (Gao et al., 2022; Zhang et al., 2023).Source: §7 · p. 9
  • Automatic verification methods check whether generated responses are entailed by cited documents, use self-consistency and retrieval to correct mistakes, or call logical backends for deductive reasoning.The cited examples include Yue et al. (2023), Gao et al. (2023), Zhao et al. (2023), and Poesia et al. (2023).Source: §7 · pp. 9–10

8 Conclusion

The conclusion presents SymGen as a simple way to link generated text to input data and make outputs more verifiable. The evaluation suggests this can be done without degrading quality.

8 Conclusion

  • SymGen links portions of generated output to input data as a simple approach to more verifiable text generation.Source: §8 · p. 10
  • The evaluation suggests symbolic generation increases verifiability without degrading quality.Source: §8 · p. 10

A Obituary data evaluation

The obituary evaluation appendix identifies the Wikidata properties and corresponding questions used in question answering, and points to the evaluation prompt in App. H.3.

A Obituary data evaluation

B Financial question answering over structured data

This section evaluates whether LLMs can answer finance questions using structured company data. GPT-3.5-16K performs substantially worse than the baseline with symbolic generation, while GPT-4 strategies perform comparably.

Motivation

  • SymGen is also motivated by tool-augmented question answering, where LLMs condition generation on structured results returned by API calls.The passage frames this as a potential application of symbolic grounding.Source: B Financial question answering over… · p. 14

Dataset

  • The evaluation asks 32 finance-related questions about companies, drawing company information from the AlphaVantage company overview API.The questions include comparisons such as comparing two companies’ book values.Source: B Financial question answering over… · pp. 14–15

Experimental setup

  • One author judged responses acceptable or not based generally on whether the final answer was correct, using GPT-3.5-16K and GPT-4-32K.The in-context learning setting used three-shot prompting.Source: B Financial question answering over… · p. 15

Results

  • GPT-3.5-16K falls substantially short of the baseline with symbolic generation, while GPT-4 approaches perform comparably across generation strategies.Acceptability is judged for answers to finance-related questions using company information.Source: B Financial question answering over… · p. 15

C Financial question answering human evaluation

The financial QA evaluation uses company data to assess answers to pairwise and multiway comparison questions. Acceptability depends on question type, and the supplied table caption describes a separate conditioned obituary QA evaluation.

Financial QA evaluation

Evaluation setup

  • The structured input comes from AlphaVantage’s company overview endpoint, with numerical fields converted to JSON numbers; all questions are answerable from the data.One author annotated responses as acceptable or not.Source: C Financial question answering human… · p. 15

Acceptability criteria

  • For objective questions, an acceptable response needs a correct final answer even if its reasoning is incorrect; subjective answers require sensible, correct reasoning.A failure to generate an answer is unacceptable.Source: C Financial question answering human… · p. 15

Conditioned obituary results

  • Table 6’s caption describes conditioned obituary QA results for GPT-3.5-4K, reporting exact match and ROUGE against answers from source data.The results are conditioned on the model having answered rather than returning “Unknown.”Source: Table 6 · p. 17

D Conditioned obituary results

The conditioned obituary results are reported for cases where the QA model answered. The appendix details the human study’s annotation sequence, questions, and interface.

D Conditioned obituary results

  • Table 6 reports obituary results conditioned on the QA model not abstaining from an answer.The supplied passage gives the table reference and condition but not its result values.Source: D Conditioned obituary results · p. 17

E.1 Detailed breakdown of summary steps

  • Annotators saw an instruction page, three practice examples, and then four documents to assess; submissions were excluded from analysis after one failed practice check.Each annotator received four of five documents, split evenly between annotated and unannotated versions; each had a 50% chance of consistency.Source: §E.1 · pp. 17–18

E.2 Questions asked to annotators

  • For each document, annotators judged consistency and confidence; the exit survey asked whether annotations increased confidence or ease.The consistency judgment stopped the timer used to measure annotation time.Source: §E.2 · p. 18

E.3 Interface

  • The interface included instructions, a completed annotation page, an exit survey, and hover tooltips that described provenance and highlighted the corresponding data field.The tooltips and table were human-readable, with values written manually based on the known data schema.Source: §E.3 · p. 19, Figure 4 · p. 19

F Qualitative error analysis

The error analysis describes both reasoning mistakes and syntactically invalid symbolic expressions. These errors can make otherwise interpretable generations incorrect or unrenderable.

Qualitative examples

  • Figure 8 illustrates arithmetic errors, an incorrect increment, a likely misranking, and a syntactically incorrect Jinja expression that makes the generation unrenderable.The figure notes that symbolic output can expose reasoning behind errors, such as the model’s attempt to add one.Source: Figure 8 · p. 22

GSM error analysis

  • GSM failures included algebra-like variable assignments that were invalid Jinja, including an assignment expressing equal amounts of red, white, and purple paint.The expression was conceptually sensible but syntactically incorrect.Source: F Qualitative error analysis · p. 22

Financial error analysis

  • In financial QA, GPT-3.5-16K sometimes referenced comparison values without identifying the largest correctly, and most symbolic syntax errors involved 50DayMovingAverage.The authors suggest constrained decoding as a possible way to curtail syntax errors.Source: F Qualitative error analysis · p. 22

G Human study detailed results

The human-study appendix directs readers to Table 7 for detailed results and describes the comparison measures. It also records experimental settings and dataset-comparison dimensions.

G Human study detailed results

  • Table 7 compares a system with SymGen annotations to one without, reporting accuracy, verification time, perceived confidence, and perceived effort.Confidence and effort use 1–5 Likert scales; higher scores indicate more confidence and less effort.Source: G Human study detailed results · p. 22, Table 7 · p. 23
  • Table 8 compares datasets by domain, evaluation-set size, data-field types and average count, and whether each datapoint has a reference generation.Source: Table 8 · p. 23

H Experimental details

  • The experiments use the Azure API with model version 0613 unless otherwise specified, and Huggingface evaluate implementations of SacreBLEU, ROUGE, and BERTScore.Source: H Experimental details · p. 23

H.1 Datasets

This appendix part documents Rotowire preprocessing and counterfactual obituary data construction, alongside prompts for biography generation, basketball summaries, obituary and financial question answering, and GSM8K math solutions. The prompts illustrate symbolic references, answer rewriting, and calculations expressed through Jinja variables and operations.

H.1 Datasets

  • Rotowire data is converted to nested key–value structures, while counterfactual obituary records use scientist attributes with progressively corrupted values.The dataset includes entities with at least 75 sitelinks and dictionaries built from 27 manually selected Wikidata properties; absent properties are omitted and the first API value is kept when multiple values exist. Copies corrupt 50% or 100% of selected properties by sampling from empirical value distributions, excluding name, age, and gender.Source: §H.1 · pp. 23–24

H.2.1 SynthBio prompts

  • SynthBio prompts contrast baseline biographies with direct symbolic-reference generation and an indirect variant that rewrites a generated biography using JSON fields.The prompts instruct the model to reference only fields present in the JSON and to prefer references over copying values when possible.Source: §H.2.1 · pp. 25–29

H.2.2 Rotowire prompts

  • The Rotowire baseline prompt asks for a basketball-game summary that remains faithful to the statistics provided in JSON.Optional few-shot examples are included in the prompt format.Source: §H.2.2 · p. 30
  • The direct Rotowire prompt tells the model to summarize game statistics using Jinja-style references to existing JSON fields whenever possible.An example output uses references for team and player statistics while retaining ordinary prose between them.Source: §H.2.2 · pp. 33, 34, 37
  • The indirect Rotowire prompt first requests an ordinary game summary, then asks the model to rewrite it with references only to fields present in the JSON.The passages show a plain-text example and a rewritten example using symbolic references.Source: §H.2.2 · pp. 38, 39, 42

H.2.3 Obituary prompts

  • The obituary baseline prompt asks the model to write about a person described in JSON and faithfully mention every property.The supplied example begins with Niels Henrik Abel’s biographical fields.Source: §H.2.3 · pp. 42–43
  • For obituary generation, the direct prompt asks the model to reference existing JSON fields in its text, while the baseline example generates an unannotated obituary.The direct example uses Jinja-style references such as {{ data.name }} and {{ data.place_of_death }}.Source: §H.2.3 · p. 44

H.3 Obituary QA evaluation

  • The obituary QA prompt instructs the model to answer using only the supplied obituary, with a short answer or “Unknown” when it lacks the answer.The example asks where Herbert Spencer was born and gives “Derby” as an illustrative answer.Source: §H.3 · pp. 45–46

H.3.1 Financial prompts

  • Financial QA examples contrast plain answers with direct symbolic references and a two-step approach that first answers, then rewrites using JSON-field references.The direct prompt requires references to existing fields; the indirect prompt asks for a later rewrite in Jinja-style expressions.Source: §H.3.1 · pp. 49, 51, 52, 53
  • Financial prompts have the model rewrite a plain answer with Jinja references to existing JSON fields rather than copying values or inventing fields.The example references GOOG’s name, symbol, and market capitalization in its answer.Source: §H.3.1 · pp. 54–55

H.3.2 GSM8K prompts

  • GSM8K prompts ask the model to show math steps with Jinja expressions, allowing quoted variables and variable assignments but forbidding assignments to unknown variables.Examples encode arithmetic in named variables and use those variables in the answer.Source: §H.3.2 · pp. 55, 57
Loading 2311.09188v2…