Presentations (Communicative Events)

Applying the Pyramid Method in DUC 2005

McKeown, Kathleen; Passonneau, Rebecca; Nenkova, Ani; Sigelman, Sergey

In DUC 2005, the pyramid method for content evaluation was used for the first time in a crosssite evaluation. We discuss the method used in creating pyramid models and performing peer annotation. Analysis of score averages for the peers indicates that the best systems score half as well as humans, and that systems can be grouped into better and worse performers. There were few significant differences among systems. High score correlations between sets from different annotators, and good interannotator agreement, indicate that participants can perform annotation reliably. We found that a modified pyramid score gave good results and would simplify peer annotation in the future.


More About This Work

Academic Units
Computer Science
Proceedings of the 2005 Workshop of the Document Understanding Conference (DUC)
Published Here
June 12, 2013