Home

Using Density Estimation to Improve Text Categorization

Carl Sable; Kathleen McKeown; Vasileios Hatzivassiloglou

Title:
Using Density Estimation to Improve Text Categorization
Author(s):
Sable, Carl
McKeown, Kathleen
Hatzivassiloglou, Vasileios
Date:
Type:
Technical reports
Department:
Computer Science
Permanent URL:
Series:
Columbia University Computer Science Technical Reports
Part Number:
CUCS-012-02
Publisher:
Department of Computer Science, Columbia University
Publisher Location:
New York
Abstract:
This paper explores the use of a statistical technique known as density estimation to potentially improve the results of text categorization systems which label documents by computing similarities between documents and categories. In addition to potentially improving a system's overall accuracy, density estimation converts similarity scores to probabilities. These probabilities provide confidence measures for a system's predictions which are easily interpretable and could potentially help to combine results of various systems. We discuss the results of three complete experiments on three separate data sets applying density estimation to the results of a TF*IDF/Rocchio system, and we compare these results to those of many competing approaches.
Subject(s):
Computer science
Item views:
110
Metadata:
View

In Partnership with the Center for Digital Research and Scholarship at Columbia University Libraries/Information Services.