PaperlayerView exact revision on arXiv

Source-linked AI summary

Controlled Attribute-Specific Summarization of Interrogative Dialogues

A Aditya Bhardwaj, Arjit Singh Arora, Md Shad Akhtar

arXiv:2609.28004v1cs.CLcs.AI

TL;DR

Interrogation transcripts can be verbose and disorganized, and errors in forensic summaries may distort investigative reasoning or omit legally salient facts. The paper presents CASPER, an attribute-specific prompting and iterative evaluation framework, and reports that it outperforms baselines in automatic and human evaluations.

  • Problem

    Interrogation transcripts can be verbose, fragmented, and disorganized, while existing summarization systems emphasize fluency or lexical overlap rather than structured investigative alignment.

    Source: §1 · pp. 1–2

  • Method

    CASPER uses attribute-guided prompting and iterative refinement, while RoleEval applies hierarchical role-specific assessment to summaries.

    Source: §1 · p. 3

  • Results

    CASPER outperforms baselines in automatic and human evaluations, with reported improvements in factual completeness, coherence, and relevance.

    Source: §6 · p. 20

  • Takeaways & Limitations

    The paper concludes that CASPER and MINDSum support evaluation of attribute-aligned forensic summaries.

    Source: §6 · p. 20

    The Officer, Inspector, and Senior Inspector roles were selected based on references intended to simulate the responsibilities of those positions.

    Source: Step 3: RoleEval – Performing… · p. 8

Abstract

from arXiv · show

Effective summarization of interrogative dialogues is a critical task in forensic and investigative settings, requiring high factual accuracy, coherence, and attribute-specific relevance. In this work, we introduce CASPER, a novel Chain-of-Thought Attribute-Specific Prompting for Evaluative Summarization framework that leverages structured prompting and iterative refinement to generate high-quality summaries of interrogator-witness interactions. We construct MINDSum, a dataset extending the MIND corpus, comprising 6,000 utterance pairs annotated with event details, factual statements, character descriptions, and fillers. CASPER employs RoleEval, a hierarchical evaluation mechanism where multiple roles (officer, inspector, senior inspector) iteratively assess summaries based on predefined criteria. By integrating entity extraction and structured feedback loops, CASPER significantly improves factual consistency and contextual completeness compared to existing baselines. Experimental results demonstrate that our framework outperforms standard summarization models on both lexical (ROUGE) and semantic (BERTScore) metrics, while human evaluation confirms its alignment with expert reasoning. Our findings underscore the potential of controlled summarization in high-stakes domains, paving the way for AI-driven forensic intelligence.

1. Introduction

Witness interrogations can be verbose, inconsistent, and cognitively disorganized, while errors in their summaries may distort investigative reasoning or omit legally salient facts. The paper frames forensic summarization around event details, factual details, and character descriptions, and introduces CASPER, RoleEval, and MINDSum to support this task.

1. Introduction

  • MINDSum extends the MIND corpus with 5,785 annotated witness-interrogator dialogue pairs and expert summaries aligned with three investigative attributes.The summaries are intended to preserve the testimony while avoiding overlap between categories.Source: §1 · p. 2
  • Forensic summaries must preserve chronological event details, objective factual claims, and structured descriptions of the people involved.The paper presents these as the core attributes needed to represent testimony for legal assessment.Source: §1 · pp. 2–3
  • CASPER combines attribute-guided prompting and iterative self-refinement to structure summaries around event details, factual details, and character descriptions.The framework uses modular prompt templates conditioned on forensic guidelines to manage conflicting constraints.Source: §1 · p. 3
  • RoleEval uses role-specific prompts and hierarchical review to assess summary quality against investigative rubrics.The paper reports combining RoleEval with ROUGE, BERTScore, and human assessment.Source: §1 · p. 3

2. Related Work

Prior controlled summarization research developed content planning, prompt-based control, and self-refinement, but the paper identifies gaps in enforcing multiple forensic attributes and evaluating investigative relevance. It positions CASPER as a forensic-grounded framework combining attribute-specific prompting, iterative role-based refinement, and hierarchical assessment.

2. Related Work

  • Earlier controlled summarization approaches use entity chains, control tokens, length or entity constraints, and hybrid extractive-abstractive methods, but often require fine-tuning and do not enforce forensic multi-attribute constraints.The paper also notes that ROUGE can miss temporal coherence, cross-witness consistency, and factual grounding.Source: §2 · p. 3
  • Prompt-based control includes entity-aware PromptSum, chain-of-thought prompting, and instance-level attribute control for content faithfulness in dialogue summarization.The passage cites PromptSum (Ravaut et al. 2023), CoT (Wei et al. 2022), and instance-level control (Liu et al. 2023).Source: §2 · p. 4
  • SumCoT decomposes summarization into identifying key elements and synthesizing a coherent summary, with reported ROUGE-L gains of +4.33/+4.77 on CN-N/DailyMail and BBC XSum.The passage attributes the method to Wang, Zhang, and Wang 2023.Source: §2 · p. 4
  • The paper distinguishes CASPER by decomposing testimony into event details, factual claims, and character descriptions while integrating entity-guided prompting, iterative role-based refinement, and hierarchical evaluation.It argues that lexical or semantic overlap metrics inadequately capture investigative relevance and forensic completeness.Source: §2 · pp. 4–5

3. Data Preparation and Annotation Framework

MINDSum extends MIND with interrogator-witness testimonies and attribute-aligned summaries. Its annotation process labels utterance pairs and produces summaries for each testimony, while dataset statistics describe the splits and dialogue-act distribution.

3. Data Preparation and Annotation Framework

  • MINDSum extends MIND with 376 testimonies covering 149 crime-related events and more than 5,785 interrogator-witness utterance pairs.The events include robbery, theft, and murder.Source: §3 · p. 5

3.1 Annotation

  • Annotators assign utterance pairs multilabel attribute tags, including an interrogation-filler label, then create event, factual, and character summaries for each testimony.The filler label filters non-essential, irrelevant, or redundant information from event narration.Source: §3.1 · p. 5
  • Two annotators and a senior linguist developed the annotations through three pilot rounds; annotators achieved Cohen’s κ = 0.83 on utterance labels.The senior linguist reviewed and edited summaries for relevance when necessary.Source: §3.1 · p. 5

3.2 Dataset Statistics

  • MINDSum has a 65:20:15 training:test:validation split, comprising 237 training testimonies (3,751 utterance pairs), 76 test testimonies (1,106 pairs), and 63 validation testimonies (928 pairs).Source: §3.2 · p. 6

4. Method

CASPER generates summaries conditioned on event, factual, or character attributes. It combines structured prompting and entity extraction with role-specific assessment and iterative refinement.

4. Method

  • CASPER generates an attribute-specific summary for event details, factual details, or character descriptions from a dialogue transcript.The framework integrates structured prompts, entity extraction, and hierarchical evaluation.Source: §4 · p. 7
  • CASPER iteratively refines summaries through a multiphase assessment involving officer, inspector, and senior inspector roles.The roles assess accuracy, coherence, and completeness.Source: Figure 3 · p. 7

Step 1: Dialogue Augmentation with Guiding Prompt.

CASPER augments a dialogue with a guiding prompt so an LLM can extract entities relevant to the testimony.

Step 1: Dialogue Augmentation with Guiding Prompt.

Step 2: Entity Extraction and Attribute-Specific Summarization.

CASPER uses extracted entities to generate attribute-specific questions and answers, then summarizes the answers for the selected attribute.

Step 2: Entity Extraction and Attribute-Specific Summarization.

  • Llama 3.1 8B extracts entities from the augmented dialogue, then answers targeted questions generated for the designated attribute.The resulting answer set is denoted Aa.Source: Step 2: Entity Extraction and… · p. 8
  • Llama 3.1 8B synthesizes the attribute-specific answers into a coherent, fluent summary in a zero-shot setting.The prompt limits the summary to 150 words and requires it to begin “According to the witness.”Source: Step 2: Entity Extraction and… · p. 8
  • The prompt directs the model to generate five questions related to the attribute and answer them.Source: Step 2: Entity Extraction and… · p. 8

Step 3: RoleEval – Performing Hierarchical Evaluation via Role-Specific Prompts.

RoleEval assesses generated summaries through successive role-specific reviews, with a senior inspector synthesizing earlier verdicts and the final verdict informing refinement.

Step 3: RoleEval – Performing Hierarchical Evaluation via Role-Specific Prompts.

  • The officer focuses on factual accuracy, while the inspector evaluates investigative relevance and identifies evidence important for further investigation.Source: Step 3: RoleEval – Performing… · p. 8
  • Officer, inspector, and senior inspector roles assess summaries in a hierarchy, with the senior inspector synthesizing earlier reviews into a final verdict.The roles were selected to simulate responsibilities associated with those positions.Source: Step 3: RoleEval – Performing… · pp. 8–9
  • Evaluators receive role-specific prompts, evaluation criteria, the summary, and the entity chain; GPT-4o mini generates verdicts that guide subsequent refinement.Source: Step 3: RoleEval – Performing… · p. 9
  • The senior inspector reviews legal and strategic implications, including objectivity, biases, inconsistencies, ambiguities, and alignment with legal standards.Source: Step 3: RoleEval – Performing… · p. 9
  • The evaluation criteria cover coverage, consistency, clarity, attribute relevance, and factual completeness, with verdicts recorded as role evaluations.Source: Step 3: RoleEval – Performing… · p. 9

5. Experiments, Results, and Discussion

Experiments compare CASPER with fine-tuned PLMs and prompted LLM baselines using lexical, semantic, and role-based evaluation. CASPER leads on RoleEval for event and factual details, while results also show differences across evaluation setups, case severity, and refinement iterations.

5.1 Baselines

  • Baselines include fine-tuned BART, T5, and PEGASUS, plus LLMs evaluated with CASPER’s attribute-specific zero-shot prompts.The comparison also includes SumCoT, PromptSum, PLASMA, DialoGPT, Tree of Thoughts, and Reflexion.Source: §5.1 · p. 10

5.2 Evaluation

  • Evaluation combines ROUGE-1, ROUGE-2, ROUGE-L, and BERTScore with five LLM-judged dimensions: coverage, consistency, clarity, attribute relevance, and factual completeness.RoleEval uses officer, inspector, and senior inspector scores on a 1-to-5 Likert scale and reports their mean.Source: §5.2 · pp. 10–11

5.3 Result Analysis

  • CASPER scores 86.06 on ED RoleEval and 80.67 on FD RoleEval, but its 66.68 on CD trails SumCoT (80.12) and GPT-4o (69.8).The comparison covers ROUGE, BERTScore, and RoleEval on MINDsum.Source: §5.3 · p. 11, Table 3 · p. 11

5.4 Ablation Study

  • Removing attribute labels and iterative refinement lowered RoleEval scores by 8.86 on ED, 11.5 on FD, and 5.48 on CD.Removing only iterative refinement lowered scores by 11.81 on ED, 11.5 on FD, and 5.53 on CD; removing only labels slightly boosted performance.Source: §5.3 · p. 11, §5.4 · p. 12

5.3 Result Analysis

  • A paired two-tailed t-test on 120 samples found CASPER’s performance improvements statistically significant.The reported analysis is presented in Table 4.Source: §5.3 · p. 11, Table 4 · p. 12

5.4.1 Hierarchical vs Non-Hierarchical Evaluation

  • Multi-step RoleEval produced summaries described as more informative and nuanced than single-step evaluation summaries.The analysis characterizes the multi-step approach as capturing broader context and enriching the summaries.Source: §5.4.1 · p. 12, Table 5 · p. 12

5.4.2 Hierarchical Evaluation vs Case Severity

  • In high-severity cases, senior inspectors consistently extracted more critical forensic elements than officers and inspectors.The authors recommend all three roles for high-severity cases, while lower-severity cases may use fewer roles.Source: §5.4.2 · pp. 12–13

5.4.3 Iterative Refinement

  • Refining summaries for K >= 2 reduced performance, which the authors associate with lower verbosity and inclusion of non-essential details.They suggest one iteration can suffice with comprehensive models such as GPT-4o and LLaMA 3.1.Source: §5.4.3 · p. 13

5.5 Verdict Assessment for Different Roles

RoleEval uses progressively senior reviewers to assess completeness, neutrality, and narrative coherence. Error analysis reports fewer summary-level and category-specific errors for CASPER than for baselines.

5.5 Verdict Assessment for Different Roles

  • RoleEval’s officer, inspector, and senior inspector verdicts progressively scrutinize completeness, interpretive depth, readability, and coherence.The framework is presented as a structured feedback loop, with examples spanning low-, mild-, and high-severity cases.Source: §5.5 · p. 13, Table 8 · p. 14, Table 9 · p. 15
  • In the low-severity prank-call case, higher-level reviewers flag missing identities and roles, then emphasize readability and clearer narrative structure.The officer captures the call sequence and realization that it was a prank.Source: §5.5 · p. 14, Table 8 · p. 14
  • In the mild-severity accident case, the inspector adds the passenger’s wedding urgency, while the senior inspector cautions against assumptions about intent.The review also questions the officer’s subjective reference to the passenger’s kindness.Source: §5.5 · pp. 14–15, Table 9 · p. 15
  • In the high-severity case, reviewers identify omitted participants and robbery details, then further clarify Lakshmi’s actions and Baban’s identity and role.The authors say these refinements add causal structure and address details missed by earlier assessments.Source: §5.5 · pp. 15–16, Table 10 · p. 16

5.6 Error Analysis

  • The error analysis examines 120 samples across three attributes and five failure types, distinguishing how many summaries are affected from the prevalence of each error type.The categories include attribute overlap, omission of nuance, hallucinated causality, entity drift, and lexical ambiguity.Source: §5.6 · pp. 17–18
  • CASPER has the lowest summary error density (Pe = 0.18), with ω3 = 0.02 for hallucinated causality and ω1 = 0.03 for attribute overlap.The authors report that more than 80% of CASPER summaries are free of the five defined failure types; PromptSum and SumCoT each exceed Pe = 0.50.Source: §5.6 · p. 18, Table 12 · p. 18
  • CASPER’s low error prevalence is associated with fewer hallucinated causal links, clearer attribute boundaries, and more stable references to people.The authors contrast this with baselines that omit details, introduce inaccuracies, or drift in entity references.Source: §5.6 · pp. 17–19

5.7 Human Evaluation Protocol

Human raters assessed summaries across five dimensions, while additional analyses examined RoleEval’s model-matching bias and agreement with human judgments. Profiling also reports inference time and the share attributed to iterative refinement.

5.7 Human Evaluation Protocol

  • Thirty graduate-level volunteers rated 45 summaries on five dimensions using a 1–5 Likert scale, with Cronbach’s α = 0.91.The dimensions were coverage, consistency, clarity, attribute relevance, and factual completeness.Source: §5.7 · p. 19, Table 13 · p. 19

5.8 Mitigating bias b/w RoleEval as a validator and RoleEval as a metric.

  • RoleEval assigns higher scores when the same LLM performs refinement and evaluation, while human scores are closer across different-model pairings.The analysis uses GPT-4o-mini and Phi-4; the human comparison uses 15 testimonies.Source: §5.8 · pp. 19–20, Table 14 · p. 20

5.9 Reliability of RoleEval as an evaluation metric

  • RoleEval correlates more strongly with human judgments than ROUGE or BERTScore.Table 15 reports the comparison of automatic metrics with human judgment.Source: §5.9 · p. 20, Table 15 · p. 20

5.10 Computation Cost and Inference Latency

  • A 50-utterance dialogue takes 10–15 seconds on an NVIDIA A100 GPU, with 60% of inference time attributed to three role-based refinement passes.Token generation during summarization and refinement is identified as the main bottleneck.Source: §5.10 · p. 20

6. Conclusion

The paper presents CASPER as attribute-specific prompting with iterative evaluation and introduces MINDSum as a forensic summarization benchmark. It reports better automatic and human evaluation results than baselines, while noting that CASPER is currently limited to monolingual English.

6. Conclusion

  • CASPER combines chain-of-thought attribute-specific prompting and iterative evaluation, conditioning summaries on event details, factual details, and character descriptions.RoleEval provides structured assessment; MINDSum is introduced as a dataset for evaluating attribute-aligned forensic summaries.Source: §6 · p. 20
  • CASPER outperforms baselines in automatic and human evaluations, with reported improvements in factual completeness, coherence, and relevance.The conclusion does not specify metric values.Source: §6 · p. 20
  • CASPER’s current functionality is limited to monolingual English, and the authors identify multilingual support as future work.Source: §6 · p. 20

7. Ethical Considerations

The authors describe privacy, consent, fairness, and responsible AI use as ethical considerations in developing MINDSum.

7. Ethical Considerations

  • MINDSum uses multiple annotators with Cohen’s κ = 0.83 and aims to avoid demographic stereotyping through neutral, fact-focused scenarios.The authors discourage dataset misuse in criminal investigations and emphasize transparency, accountability, and participant rights.Source: §7 · p. 21
Loading 2609.28004v1…