2003 Reports

Evaluating Content Selection in Human- or Machine-Generated Summaries: The Pyramid Scoring Method

Passonneau, Rebecca J.; Nenkova, Ani

From the outset of automated generation of summaries, the difficulty of evaluation has been widely discussed. Despite many promising attempts, we believe it remains an unsolved problem. Here we present a method for scoring the content of summaries of any length against a weighted inventory of content units, which we refer to as a pyramid. Our method is derived from empirical analysis of human-generated summaries, and provides an informative metric for human or machine-generated summaries.