2006 Presentations (Communicative Events)

Applying the Pyramid Method in the 2006 Document Understanding Conference

Passonneau, Rebecca; McKeown, Kathleen; Sigelman, Sergey; Goodkind, Adam

The pyramid evaluation effort for the 2006 Document Understanding Conference involved twenty-two sites on twenty document sets. Each pyramid content model (one per document set) was constructed from four human summaries. Peer systems were scored using the modified pyramid score introduced in DUC 2005. ANOVAs with score as the independent variable and nine factors yielded three significant factors: document set, peer, and content responsiveness. There were many more significant differences among peer systems in 2006 than for DUC 2005. We speculate this is due to a combination of improved systems and improvements in our evaluation procedures.