Academic Commons Search Results
http://academiccommons.columbia.edu/catalog.rss?f%5Bdepartment_facet%5D%5B%5D=Human+Development&f%5Bsubject_facet%5D%5B%5D=Quantitative+psychology+and+psychometrics&q=&rows=500&sort=record_creation_date+desc
Academic Commons Search Resultsen-usEstimating the Q-matrix for Cognitive Diagnosis Models in a Bayesian Framework
http://academiccommons.columbia.edu/catalog/ac:176107
Chung, Meng-tahttp://dx.doi.org/10.7916/D857195BMon, 07 Jul 2014 11:49:03 +0000This research aims to develop an MCMC algorithm for estimating the Q-matrix in a Bayesian framework. A saturated multinomial model was used to estimate correlated attributes in the DINA model and rRUM. Closed-forms of posteriors for guess and slip parameters were derived for the DINA model. The random walk Metropolis-Hastings algorithm was applied to parameter estimation in the rRUM. An algorithm for reducing potential label switching was incorporated into the estimation procedure. A method for simulating data with correlated attributes for the DINA model and rRUM was offered.
Three simulation studies were conducted to evaluate the algorithm for Bayesian estimation. Twenty simulated data sets for simulation study 1 were generated from independent attributes for the DINA model and rRUM. A hundred data sets from correlated attributes were generated for the DINA and rRUM with guess and slip parameters set to 0.2 in simulation study 2. Simulation study 3 analyzed data sets simulated from the DINA model with guess and slip parameters generated from Uniform (0.1, 0.4). Results from simulation studies showed that the Q-matrix recovery rate was satisfactory. Using the fraction-subtraction data, an empirical study was conducted for the DINA model and rRUM. The estimated Q-matrices from the two models were compared with the expert-designed Q-matrix.Quantitative psychology and psychometrics, Statistics, Educational tests and measurementsMeasurement and Evaluation, Human DevelopmentDissertationsFactors Affecting Probability Matching Behavior
http://academiccommons.columbia.edu/catalog/ac:164326
Gao, Jiehttp://hdl.handle.net/10022/AC:P:21357Fri, 16 Aug 2013 16:58:44 +0000In life, people commonly face repeated decisions under risk or uncertainty. While normative economic models assume that people tend to make choices that maximize their expected utility, suboptimal behavior - in particular, probability matching - is frequently observed in research on repeated decisions. Probability matching is the tendency to match prediction probabilities of each outcome with the observed outcome probabilities in a random binary prediction task. For example, when people are faced with making with a sequence of predictions, such as repeatedly predicting the outcome of rolling a die with four sides colored green and two sides colored red, most people allocate about two-thirds of their predictions to green, and one-third to red. The optimal strategy, referred to as maximizing, would be to choose the outcome with the higher probability in every trial in the prediction task. Various causes for probability matching have been proposed during the past several decades. Here it is proposed that implicit adoption of a perfect prediction goal by decision makers might tend to elicit probability matching behavior. Thus, one factor that might affect the prevalence of probability matching behavior (investigated in Studies 1 and 2) is the type of performance goal. The manipulation in Study 1 contrasted single-trial prediction with prediction of four-trial sequences, which it is hypothesized might create an implicit perfect prediction goal for the sequence. In Study 2, three levels of goal were explicitly manipulated for each sequence: a perfect prediction goal, an 80% correct goal, and a 60% correct goal. In both studies it was predicted that more matching behavior would be observed for those who have a goal of perfect prediction than those who have a more reasonable (lower) goal. The results of both studies, conducted in an online worker marketplace, supported the goal-level hypothesis. The second factor proposed to affect the prevalence of probability matching is the type of conceptual schema describing the events to be predicted: independent events or complementary events. Study 3 investigated the effects of schema type and abstraction level of context on matching or maximizing behavior. Three abstraction levels of stories were included: abstract, concrete random devices, and real-world stories. The main hypothesis was that when the two options to be predicted are independent events, less matching and more maximizing behavior should be observed. Data from Study 3 supported the hypothesis that independent events tend to elicit more maximizing behavior. No effects of abstraction level were observed.Cognitive psychology, Quantitative psychology and psychometricsMeasurement and Evaluation, Human DevelopmentDissertationsSchematic Effects on Probability Problem Solving
http://academiccommons.columbia.edu/catalog/ac:174540
Gugga, Saranda Soniahttp://hdl.handle.net/10022/AC:P:20863Fri, 28 Jun 2013 09:50:44 +0000Three studies examined context effects on solving probability problems. Variants of word problems were written with cover stories which differed with respect to social or temporal schemas, while maintaining formal problem structure and solution procedure. In the first of these studies it was shown that problems depicting schemas in which randomness was inappropriate or unexpected for the social situation were solved less often than problems depicting schemas in which randomness was appropriate. Another set of two studies examined temporal and causal schemas, in which the convention is that events are considered in forward direction. Pairs of conditional probability (CP) problems were written depicting events E1 and E2, such that E1 either occurs before E2 or causes E2. Problems were defined with respect to the order of events expressed in CPs, so that P(E2|E1) represents the CP in schema-consistent, intact order by considering the occurrence of E1 before E2, while P(E1|E2) represents CP in schema-inconsistent, inverted order. Introductory statistics students had greater difficulty encoding CP for events in schema-inconsistent order than CP of events in conventional deterministic order. The differential effects of schematic context on solving probability problems identify specific conditions and sources of bias in human reasoning under uncertainty. In addition, these biases may be influential when evaluating empirical findings in a manner similar to that demonstrated in this paper experimentally, and may have implications for how social scientists are trained in research methodology.Cognitive psychology, Quantitative psychology and psychometrics, Educational psychologyssg34Measurement and Evaluation, Human DevelopmentDissertationsExamining Uncertainty and Misspecification of Attributes in Cognitive Diagnostic Models
http://academiccommons.columbia.edu/catalog/ac:174822
Chen, Chen-Miao Carolhttp://hdl.handle.net/10022/AC:P:20451Fri, 24 May 2013 09:17:55 +0000In recent years, cognitive diagnostic models (CDMs) have been widely used in educational assessment to provide a diagnostic profile (mastery/non-mastery) analysis for examinees, which gives insights into learning and teaching. However, there is often uncertainty about the specification of the Q-matrix that is required for CDMs, given that it is based on expert judgment. The current study uses a Bayesian approach to examine recovery of Q-matrix elements in the presence of uncertainty about some elements. The first simulation examined the situation where there is complete uncertainty about whether or not an attribute is required, when in fact it is required. The simulation results showed that recovery was generally excellent. However, recovery broke down when other elements of the Q-matrix were misspecified. Further simulations showed that, if one has some information about the attributes for a few items, then recovery improves considerably, but this also depends on how many other elements are misspecified. A second set of simulations examined the situation where uncertain Q-matrix elements were scattered throughout the Q-matrix. Recovery was generally excellent, even when some other elements were misspecified. A third set of simulations showed that using more informative priors did not uniformly improve recovery. An application of the approach to data from TIMSS (2007) suggested some alternative Q-matrices.Quantitative psychology and psychometricscc2410Measurement and Evaluation, Human DevelopmentDissertationsA Bayesian Multidimensional Scaling Model for Partial Rank Preference Data
http://academiccommons.columbia.edu/catalog/ac:160395
Tanaka, Kyokohttp://hdl.handle.net/10022/AC:P:20044Tue, 30 Apr 2013 12:44:35 +0000There has been great advancement on research for preferential choice in field of marketing. When we look at preferential choice data, there are two components to consider: the individuals and the items. Coombs (1950; 1964) introduced the unfolding technique on preferential choice data. In 1960, Bennett and Hays went on to create a multidimensional unfolding model. Hojo (1997;1998) showed rank data could be used in multidimensional scaling, however he did not implement a Bayesian technique. In 2010, Fong, DeSarbo, Park, and Scott proposed a new Bayesian vector Multidimensional Scaling (MDS) model which was applied to data from a five-point Likert scale survey. This paper focused on Bayesian approach choice behavior multidimensional space model for the analysis of partially ranked data (rank top 3 from J data) to provide a joint space of individuals and products, using MCMC procedure. The procedure is similar to what Fong, DeSarbo, Park, and Scott (2010) did but this study used partial rank data instead of Likert scale data. The goal of this study was to create a probability-based model that calculates the average product utility which indicates how popular the product is. Lambdas or the item loadings are the direction of the products and thetas are the direction for the individuals. In addition, this study dealt with rotational invariance by calculating the optimal lambda values for each iteration and each dimension by flipping the sign so it approaches the average value. To determine the number of dimensions of the datasets, the sum of squared loadings were calculated. We applied the MCMC procedure to simulated data in which we sampled the loadings from the normal distribution as well as loadings from the real datasets. In addition, we applied the MCMC procedure to the real dataset and created a multidimensional space for the products.Quantitative psychology and psychometricskjt2007Measurement and Evaluation, Human DevelopmentDissertationsThe Relation between Uncertainty in Latent Class Membership and Outcomes in a Latent Class Signal Detection Model
http://academiccommons.columbia.edu/catalog/ac:146637
Cheng, Zhifenhttp://hdl.handle.net/10022/AC:P:13139Fri, 04 May 2012 14:43:26 +0000Latent class variables are often used to predict outcomes. The conventional practice is to first assign observations to one of the latent classes based on the maximum posterior probabilities. The assigned class membership is then treated as an observed variable and used in predicting the outcomes. This widely used classify-analyze strategy ignores the uncertainty of being in a certain latent class for the observations. Once an observation is classified to the latent class with the highest posterior probability, its probability of being in the assigned class is treated as being one. In addition, once observations are classified to the latent class with the highest posterior probability, their representativeness of the class becomes the same because they will all have a probability of one of being in the assigned class. Finally, standard errors are underestimated because the residual uncertainty about the latent class membership is ignored. This dissertation used simulation studies and an analysis of a real-world data set to compare five commonly adopted approaches (most likely class regression, probability regression, probability-weighted regression, pseudo-class regression, and the simultaneous approach) for measuring the association between a latent class variable and outcome variables to see which one can better account for the uncertainty in latent class membership in such a situation. The model considered in the study was a latent class extension of the signal detection model (LC-SDT) by DeCarlo, which has proved to be able to address certain measurement issues in the educational field, more specifically, rater issues involved in essay grading such as rater effects and rater reliability. An LC-SDT model has the potential for wide applications in education as well as other areas. Therefore it is important to explore the issue of accounting for uncertainty in latent class membership within this framework. Three ordinal outcome variables having a negative, weak, and strong association with the latent class variable were considered in the simulations. Results of the simulations showed that the simultaneous approach performed best in obtaining unbiased parameter estimates. It also yielded larger standard errors than the other approaches which have been found by previous research to underestimate standard errors. Even though the simultaneous approach has its advantages, including outcome variables in a latent class model can affect parameters of the response variables. Therefore, cautions need to be taken when using this approach. The analysis results of the real-world data set confirmed the trends observed in the simulation studies.Quantitative psychology and psychometrics, Educational psychology, Statisticszc2133Measurement and Evaluation, Human DevelopmentDissertationsRater Drift in Constructed Response Scoring via Latent Class Signal Detection Theory and Item Response Theory
http://academiccommons.columbia.edu/catalog/ac:132272
Park, Yoon Soohttp://hdl.handle.net/10022/AC:P:10394Tue, 17 May 2011 15:29:48 +0000The use of constructed response (CR) items or performance tasks to assess test takers' ability has grown tremendously over the past decade. Examples of CR items in psychological and educational measurement range from essays, works of art, and admissions interviews. However, unlike multiple-choice (MC) items that have predetermined options, CR items require test takers to construct their own answer. As such, they require the judgment of multiple raters that are subject to differences in perception and prior knowledge of the material being evaluated. As with any scoring procedure, the scores assigned by raters must be comparable over time and over different test administrations and forms; in other words, scores must be reliable and valid for all test takers, regardless of when an individual takes the test. This study examines how longitudinal patterns or changes in rater behavior affect model-based classification accuracy. Rater drift refers to changes in rater behavior across different test administrations. Prior research has found evidence of drift. Rater behavior in CR scoring is examined using two measurement models - latent class signal detection theory (SDT) and item response theory (IRT) models. Rater effects (e.g., leniency and strictness) are partly examined with simulations, where the ability of different models to capture changes in rater behavior is studied. Drift is also examined in two real-world large scale tests: teacher certification test and high school writing test. These tests use the same set of raters for long periods of time, where each rater's scoring is examined on a monthly basis. Results from the empirical analysis showed that rater models were effective to detect changes in rater behavior over testing administrations in real-world data. However, there were differences in rater discrimination between the latent class SDT and IRT models. Simulations were used to examine the effect of rater drift on classification accuracy and on differences between the latent class SDT and IRT models. Changes in rater severity had only a minimal effect on classification. Rater discrimination had a greater effect on classification accuracy. This study also found that IRT models detected changes in rater severity and in rater discrimination even when data were generated from the latent class SDT model. However, when data were non-normal, IRT models underestimated rater discrimination, which may lead to incorrect inferences on the precision of raters. These findings provide new and important insights into CR scoring and issues that emerge in practice, including methods to improve rater training.Quantitative psychology and psychometrics, Educational tests and measurements, Statisticsysp2102Measurement and Evaluation, National Center for Disaster Preparedness, Human DevelopmentDissertations