Theses Doctoral

Establishing Psychometrically-Sound Measures of Linguistic Skills in People With and Without Aphasia During Unstructured Conversation and Structured Narrative Monologue

Leaman, Marion C.

The body of work contained in this dissertation consists of seven studies investigating conversational skills in people with aphasia (PWA). The predominant focus is on establishing reliable measures of language skills in unstructured conversation in PWA. Overall, ten measures are investigated, and much of the work is concerned with establishing interrater reliability and test-retest stability. These measures are needed to determine generalization of aphasia intervention to conversation, to inform treatment decision-making, and to develop future interventions that have the capacity to improve language abilities at a conversation-level.

The initial work focused on microlinguistic skills (i.e., word and sentence-level language; Leaman & Edmonds, 2019a; 2019c), and then evolved to include macrolinguistic skills (discourse-level language) with a focus on global coherence (Leaman & Edmonds, in press) and topic initiation (Leaman & Edmonds, 2020). In addition, questions emerged regarding: a) the relationship of language production in monologue and in conversation (due to the predominance of monologue testing, as opposed to conversation, in clinical environments; b) normative data for the measures in monologue and in conversation; c) the sensitivity of the measures as treatment outcome measures. Research questions regarding items a and b are addressed in the novel research conducted for the dissertation (reported in the last two manuscripts in this document, i.e., Dissertation Studies 1 and 2 (DS1 and DS2)), and item c is addressed in Obermeyer et al., (in press).

In addition, a related outcome of this research is a methodology, The Conversation Collection Protocol (CCP). The CCP was developed to consistently collect unstructured conversations that would have similar interactional features that could be used as language samples. The protocol is based on conversational interactions in typical speakers, and is primarily informed by the Conversation Analysis literature (for an overview see Schegloff, 2007). The CPP was piloted to train SLP conversation partners to use typical, familiar, social, adult-style interactions during the conversations (rather than traditional therapy or instructional behaviors) in the first study (Leaman & Edmonds, 2019a). The protocol was further developed prior to data collection for the dissertation studies. In this development phase, the systemized training protocol was expanded to include excerpted readings from literature regarding conversational interaction, and a post-training quiz for the partners. In addition, a session fidelity protocol was developed and implemented.

Use of the CCP in all of the studies contributed to achieving similarity in the SLP partners’ interactional styles across conversational dyads, allowed fostering of social conversations which were desired (i.e., as opposed to interview-style conversations often used in the literature), and promoted the PWA to direct their own communication decisions and topics of discussion which in typical therapy interactions may be drastically limited by the clinician (Simmons-Mackie & Damico, 1999). The CCP resulted in high session fidelity (98-99%) across the 27 SLPs who participated in the two dissertation studies. The CCP training also resulted in a corpus of conversations that are similar in content and complexity (measured by mean length of utterance and type-token ratio), with similar topics and an equitable distribution of topic-initiating utterances between the PWA and the partners (Leaman & Edmonds, 2019a; DS1).

This research agenda is motivated by a clinical need and vision for a dramatic shift in aphasia intervention, which moves away from structured, decontextualized therapy tasks and towards use of everyday conversation as the primary vehicle of intervention. Prerequisite to development of such an intervention is development of outcome measures capable of capturing real-world changes in conversation. Without such measures, it is not possible to determine whether treatment has the intended effects on conversation.

Because conversation is a complex, multi-modal, and contextually-bound phenomenon, treatment that improves everyday conversation could potentially affect many aspects of communication. Change in conversation can be realized by treatment focused on verbal skills, nonverbal skills, compensatory strategies, participation, and/or partner training, and ideally should combine all of these communication parameters. Currently, measures and scales exist for each of these areas, except for in the area of language ability in conversation. It is this clinical and research gap, the lack of reliable measures to evaluate language in its most commonly used context, conversation, that fuels this line of research.

The publications, in press manuscripts, and two manuscripts resulting from the dissertation research are presented in their order of publication. Conclusions, clinical implications, and future directions are presented in each. However, in brief summary, the primary findings of this body of work are:

1. Reliability and Stability of Language in Conversation
Measurement of language production skills in PWA can be accomplished with a high-level of reliability and stability for all measures investigated except for: a) the measure of behavioral manifestations of lexical retrieval (LEXoth; Leaman & Edmonds, 2019a); and b) referential cohesion (REF), which demonstrated variability that precluded test-retest stability in two studies (Leaman & Edmonds, 2019a; DS1). The clinical implication is that language in unstructured conversation, for certain measures, is reliable and stable.

2. Language Production Relationships in Monologue and Conversation
Language production in monologue does not tend to parallel language production in unstructured conversation, thus performance during monologue therapy tasks cannot reliably predict conversational language production for most measures investigated (DS2). Consequently, language findings based on a monologue task (this research used a story narrative monologue) cannot be extrapolated for understanding of conversational language skills for most of the measures investigated. Thus, if the desired outcome of treatment is impact at a conversation level, evaluation of the effectiveness of the intervention to achieve this aim cannot be estimated with use of a story narrative monologue language sample. Further, development of intervention relevant to language needs in conversation cannot be adequately developed based on a story narrative monologue. Further investigation is needed regarding the relationship between single picture description tasks and conversation in terms of language production skills.

3. Topic Initiation Mechanisms in PWA and Their Partners
PWA often use similar mechanisms to alert listeners that a new topic is being initiated as their communication partners without aphasia, such as waiting for an old topic to end, or using a marker like “oh, and by the way…”. In interactions between individuals without communication disorders, these mechanisms are often layered and used simultaneously. However, the findings of this research demonstrate that as aphasia severity increases, individuals use fewer simultaneous mechanisms to introduce topics. In addition, for people with moderate to severe aphasia, the fewer topic initiation mechanisms they use, the less successful they are during their topic initiating utterances (Leaman & Edmonds, 2020). This work provides a broader analysis of topic initiating behaviors in PWA with a larger sample size (n=10) than had been previously available. In addition, it established a needed methodology for locating the beginning and end of topic locations in unstructured conversation. This ability to reliably locate topics within conversation is also key to the subsequent research regarding global coherence in conversation, which depends on analysis of each utterances coherence to the overall topic being discussed (Leaman & Edmonds, in press; DS1). An important clinical implication suggests that explicit teaching PWA to use simultaneous methods of topic initiation may have therapeutic benefit to support a greater level of successfulness when they initiate new topics during conversation. Further, teaching both PWA and their regular partners about mechanisms of topic initiation may facilitate improved awareness of these mechanisms with positive therapeutic effect in conversation.

4. Sensitivity of Linguistic Measures as Post-Treatment Outcomes
Evidence of stability and sensitivity of linguistic measures in conversation is provided in an intervention case study (Obermeyer et al., in press). As a case study, this research suggests preliminary evidence that a discourse-level intervention (Attentive Reading and Constrained Summarization -Written) can affect change in conversation, and that measures investigated in the research presented here may be sensitive to such change.

5. Development and Use of the CCP to Train SLPs as Conversation Partners
Although not addressed as a research question, the CCP appears to be effective as a systematic method to collect unstructured conversations suitable for language analysis. Further, the CCP training is brief (less than an hour), and a large group of SLPs (27) demonstrated learning and adherence to the protocol, as evidenced by high session fidelity and resulting conversations that are similar in terms of vocabulary use frequency, mean length of utterance, type-token ratio, and even distribution of topic-initiating turns between the PWA and their partners, and similar topical content across the conversations (Leaman & Edmonds, 2019a; DS1). Further, in the dissertation research over 90 conversations were collected using the CCP training (some were not analyzed due to subsequent ineligibility of the participant), and no conversation resembled an interview or traditional didactic therapy interaction. The clinical implication is that SLPs can be efficiently and effectively trained as conversation partners to collect unstructured (social) conversational samples for the purpose of assessment.

Next steps in this line of research are detailed in the conclusion of each of the seven articles and manuscripts. In addition, a summary of the findings and future directions based on the entire body of work are included in the Epilogue of this dissertation.


  • thumnail for Leaman_columbia_0054D_15934.pdf Leaman_columbia_0054D_15934.pdf application/pdf 2.81 MB Download File

More About This Work

Academic Units
Speech and Language Pathology
Thesis Advisors
Edmonds, Lisa A.
Ph.D., Columbia University
Published Here
July 14, 2020