Microcoding the Lexicon with Co-occurrence Knowledge

Frank A. Smadja

Microcoding the Lexicon with Co-occurrence Knowledge
Smadja, Frank A.
Technical reports
Computer Science
Permanent URL:
Columbia University Computer Science Technical Reports
Part Number:
Neither syntax nor semantics can justify the use of a certain class of English word combinations. This class contains word pairs that often appear together in a given context of meaning. Such pairs are called co-occurrence relations or idiosyncratic collocations [3]. To correctly understand or produce natural language, such lexical relations need to be specifically encoded in lexicons [6]. [10]. [1]. In this paper, we show how word-based lexicons can be enriched with automatically acquired lexical relations. We call this process microcoding the lexicon, since it corresponds to the addition of lexical associations in a regular lexicon. We are using our enriched lexicon for language generation. Co-occurrence knowledge is particularly important for language generation, without it, awkward or incorrect sentences could be produced. In previous natural language work, co-occurrence knowledge was ignored or hand encoded. In contrast, we acquire it automatically from the analysis of large textual corpora. We describe the acquisition method based on EXTRACT [12], a co-occurrence compiler that retrieves lexical relations from the statistical analysis of a large corpus. We indicate how these lexical associations are entered in a word-based lexicon in a useful and coherent way for language generators. We then show how this information is used in COOK, a functional unification based language based generator that correctly handles collocation ally restricted sentences. Whenever possible, we use examples taken from the bank and stock market domains.
Computer science
Item views:
text | xml
Suggested Citation:
Frank A. Smadja, 1989, Microcoding the Lexicon with Co-occurrence Knowledge, Columbia University Academic Commons, http://hdl.handle.net/10022/AC:P:12114.

In Partnership with the Center for Digital Research and Scholarship at Columbia University Libraries/Information Services | Terms of Use