Summary Usefulness in Policy Briefing Documents Using Discourse Coherence Signals: Benchmark Study

Abstract

The rapid proliferation of government and institutional literature necessitates automated text summarization systems that can accurately distill complex policy briefing documents. However, traditional evaluation metrics for summarization primarily rely on lexical overlap and often fail to capture the actual usefulness of a summary for decision-makers. This paper investigates the predictive capacity of discourse coherence signals to determine the practical utility of automated summaries. By constructing a novel benchmark dataset comprising policy briefing documents and their corresponding summaries, we systematically extract and analyze both local and global discourse coherence features. These features include entity grid transitions, rhetorical structure parsing, and coreference resolution chains. We evaluate the performance of these discourse-based signals in predicting human-annotated usefulness scores through various machine learning classification models. The results indicate that models incorporating discourse coherence signals significantly outperform traditional readability and lexical overlap baselines in predicting how useful a summary will be to a policy analyst. Furthermore, we identify specific coherence transitions that act as strong indicators of high utility, suggesting that the logical flow of information is as critical as the factual content itself. This study provides a robust framework for evaluating automated summaries in specialized domains where accurate and coherent information synthesis is critical for high-stakes decision-making.