Academic Commons

Presentations (Communicative Events)

Applying the Pyramid Method in DUC 2005

McKeown, Kathleen; Passonneau, Rebecca; Nenkova, Ani; Sigelman, Sergey

In DUC 2005, the pyramid method for content evaluation was used for the first time in a crosssite evaluation. We discuss the method used in creating pyramid models and performing peer annotation. Analysis of score averages for the peers indicates that the best systems score half as well as humans, and that systems can be grouped into better and worse performers. There were few significant differences among systems. High score correlations between sets from different annotators, and good interannotator agreement, indicate that participants can perform annotation reliably. We found that a modified pyramid score gave good results and would simplify peer annotation in the future.

Files

More About This Work

Academic Units
Computer Science
Publisher
Proceedings of the 2005 Workshop of the Document Understanding Conference (DUC)
Published Here
June 12, 2013
Academic Commons provides global access to research and scholarship produced at Columbia University, Barnard College, Teachers College, Union Theological Seminary and Jewish Theological Seminary. Academic Commons is managed by the Columbia University Libraries.