Academic Commons Search Results
https://academiccommons.columbia.edu/catalog?action=index&controller=catalog&f%5Blanguage%5D%5B%5D=English&f%5Bsubject_facet%5D%5B%5D=Statistics&format=rss&fq%5B%5D=has_model_ssim%3A%22info%3Afedora%2Fldpd%3AContentAggregator%22&q=&rows=500&sort=record_creation_date+desc
Academic Commons Search Resultsen-usAdvantages of Synthetic Noise and Machine Learning for Analyzing Radioecological Data Sets
https://academiccommons.columbia.edu/catalog/ac:206927
Shuryak, Igor10.7916/D80V8JF4Wed, 05 Jul 2017 13:44:23 +0000The ecological effects of accidental or malicious radioactive contamination are insufficiently understood because of the hazards and difficulties associated with conducting studies in radioactively-polluted areas. Data sets from severely contaminated locations can therefore be small. Moreover, many potentially important factors, such as soil concentrations of toxic chemicals, pH, and temperature, can be correlated with radiation levels and with each other. In such situations, commonly-used statistical techniques like generalized linear models (GLMs) may not be able to provide useful information about how radiation and/or these other variables affect the outcome (e.g. abundance of the studied organisms). Ensemble machine learning methods such as random forests offer powerful alternatives. We propose that analysis of small radioecological data sets by GLMs and/or machine learning can be made more informative by using the following techniques: (1) adding synthetic noise variables to provide benchmarks for distinguishing the performances of valuable predictors from irrelevant ones; (2) adding noise directly to the predictors and/or to the outcome to test the robustness of analysis results against random data fluctuations; (3) adding artificial effects to selected predictors to test the sensitivity of the analysis methods in detecting predictor effects; (4) running a selected machine learning method multiple times (with different random-number seeds) to test the robustness of the detected “signal”; (5) using several machine learning methods to test the “signal’s” sensitivity to differences in analysis techniques. Here, we applied these approaches to simulated data, and to two published examples of small radioecological data sets: (I) counts of fungal taxa in samples of soil contaminated by the Chernobyl nuclear power plan accident (Ukraine), and (II) bacterial abundance in soil samples under a ruptured nuclear waste storage tank (USA). We show that the proposed techniques were advantageous compared with the methodology used in the original publications where the data sets were presented. Specifically, our approach identified a negative effect of radioactive contamination in data set I, and suggested that in data set II stable chromium could have been a stronger limiting factor for bacterial abundance than the radionuclides 137Cs and 99Tc. This new information, which was extracted from these data sets using the proposed techniques, can potentially enhance the design of radioactive waste bioremediation.Machine learning, Statistics, Radioactive pollution, Medical radiologyis144RadiologyArticlesBias Characterization in Probabilistic Genotype Data and Improved Signal Detection with Multiple Imputation
https://academiccommons.columbia.edu/catalog/ac:201409
Palmer, Cameron Douglas; Pe’er, Itsik10.7916/D8JS9QKNFri, 30 Jun 2017 18:33:07 +0000Missing data are an unavoidable component of modern statistical genetics. Different array or sequencing technologies cover different single nucleotide polymorphisms (SNPs), leading to a complicated mosaic pattern of missingness where both individual genotypes and entire SNPs are sporadically absent. Such missing data patterns cannot be ignored without introducing bias, yet cannot be inferred exclusively from nonmissing data. In genome-wide association studies, the accepted solution to missingness is to impute missing data using external reference haplotypes. The resulting probabilistic genotypes may be analyzed in the place of genotype calls. A general-purpose paradigm, called Multiple Imputation (MI), is known to model uncertainty in many contexts, yet it is not widely used in association studies. Here, we undertake a systematic evaluation of existing imputed data analysis methods and MI. We characterize biases related to uncertainty in association studies, and find that bias is introduced both at the imputation level, when imputation algorithms generate inconsistent genotype probabilities, and at the association level, when analysis methods inadequately model genotype uncertainty. We find that MI performs at least as well as existing methods or in some cases much better, and provides a straightforward paradigm for adapting existing genotype association methods to uncertain data.Multiple imputation (Statistics), Missing observations (Statistics), Genetics--Statistical methods, Genetics, Statisticscdp2130Computer ScienceArticlesFinding Alternatives to the Dogma of Power Based Sample Size Calculation: Is a Fixed Sample Size Prospective Meta-Experiment a Potential Alternative?
https://academiccommons.columbia.edu/catalog/ac:201635
Tavernier, Elsa; Trinquart, Ludovic; Giraudeau, Bruno10.7916/D89P31T1Fri, 30 Jun 2017 18:31:22 +0000Sample sizes for randomized controlled trials are typically based on power calculations. They require us to specify values for parameters such as the treatment effect, which is often difficult because we lack sufficient prior information. The objective of this paper is to provide an alternative design which circumvents the need for sample size calculation. In a simulation study, we compared a meta-experiment approach to the classical approach to assess treatment efficacy. The meta-experiment approach involves use of meta-analyzed results from 3 randomized trials of fixed sample size, 100 subjects. The classical approach involves a single randomized trial with the sample size calculated on the basis of an a priori-formulated hypothesis. For the sample size calculation in the classical approach, we used observed articles to characterize errors made on the formulated hypothesis. A prospective meta-analysis of data from trials of fixed sample size provided the same precision, power and type I error rate, on average, as the classical approach. The meta-experiment approach may provide an alternative design which does not require a sample size calculation and addresses the essential need for study replication; results may have greater external validity.Clinical trials, Meta-analysis, Epidemiology--Methodology, Public health, Epidemiology, StatisticsEpidemiologyArticlesReply
https://academiccommons.columbia.edu/catalog/ac:196838
Mason, Simon J.; Tippett, Michael K.; Weigel, Andreas P.; Goddard, Lisa M.; Rajaratnam, Balakanapathy10.7916/D8Z31ZKBFri, 30 Jun 2017 16:52:38 +0000Reply to a comment on the article: Conditional Exceedance Probabilities. Monthly Weather Review 135 (2010), 363–372 (available in Academic Commons at http://dx.doi.org/10.7916/D8PK0G2S).Climatic changes--Mathematical models, Statistics, Climatic changes--Forecasting, Atmospheresjm2103, mkt14, lmg107International Research Institute for Climate and Society, Applied Physics and Applied MathematicsArticlesComparative Validity of 3 Diabetes Mellitus Risk Prediction Scoring Models in a Multiethnic US Cohort: The Multi-Ethnic Study of Atherosclerosis
https://academiccommons.columbia.edu/catalog/ac:200753
Mann, Devin M.; Bertoni, Alain G.; Shimbo, Daichi; Carnethon, Mercedes R.; Chen, Haiying; Jenny, Nancy Swords; Muntner, Paul10.7916/D8SN093SFri, 30 Jun 2017 16:52:05 +0000Several models for estimating risk of incident diabetes in US adults are available. The authors aimed to determine the discriminative ability and calibration of published diabetes risk prediction models in a contemporary multiethnic cohort. Participants in the Multi-Ethnic Study of Atherosclerosis without diabetes at baseline (2000–2002; n = 5,329) were followed for a median of 4.75 years. The predicted risk of diabetes was calculated using published models from the Framingham Offspring Study, the Atherosclerosis Risk in Communities (ARIC) Study, and the San Antonio Heart Study. The mean age of participants was 61.6 years (standard deviation, 10.2); 29.3% were obese, 53.1% had hypertension, 34.9% had a family history of diabetes, 27.5% had high triglyceride levels, 33.8% had low high density lipoprotein cholesterol levels, and 15.3% had impaired fasting glucose. There were 446 incident cases of diabetes (fasting glucose level ≥126 mg/dL or initiation of antidiabetes medication use) diagnosed during follow-up. C statistics were 0.78, 0.84, and 0.83 for the Framingham, ARIC, and San Antonio risk prediction models, respectively. There were significant differences between observed and predicted diabetes risks (Hosmer-Lemeshow goodness-of-fit chi-squared test for each model: P < 0.001). The recalibrated and best-fit models achieved sufficient goodness of fit (each P > 0.10). The Framingham, ARIC, and San Antonio models maintained high discriminative ability but required recalibration in a modern, multiethnic US cohort.Diabetes--Epidemiology, Cohort analysis, Diabetes--Risk factors, Epidemiology, Statisticsds2231ArticlesConditional Exceedance Probabilities
https://academiccommons.columbia.edu/catalog/ac:196847
Mason, Simon J.; Galpin, Jacqueline S.; Goddard, Lisa M.; Graham, Nicholas E.; Rajaratnam, Balakanapathy10.7916/D8PK0G2SFri, 30 Jun 2017 16:50:29 +0000Probabilistic forecasts of variables measured on a categorical or ordinal scale, such as precipitation occurrence or temperatures exceeding a threshold, are typically verified by comparing the relative frequency with which the target event occurs given different levels of forecast confidence. The degree to which this conditional (on the forecast probability) relative frequency of an event corresponds with the actual forecast probabilities is known as reliability, or calibration. Forecast reliability for binary variables can be measured using the Murphy decomposition of the (half) Brier score, and can be presented graphically using reliability and attributes diagrams. For forecasts of variables on continuous scales, however, an alternative measure of reliability is required. The binned probability histogram and the reliability component of the continuous ranked probability score have been proposed as appropriate verification procedures in this context, but are subject to some limitations. A procedure is proposed that is applicable in the context of forecast ensembles and is an extension of the binned probability histogram. Individual ensemble members are treated as estimates of quantiles of the forecast distribution, and the conditional probability that the observed precipitation, for example, exceeds the amount forecast [the conditional exceedance probability (CEP)] is calculated. Generalized linear regression is used to estimate these conditional probabilities. A diagram showing the CEPs for ranked ensemble members is suggested as a useful method for indicating reliability when forecasts are on a continuous scale, and various statistical tests are suggested for quantifying the reliability.Climatic changes--Mathematical models, Statistics, Climatic changes--Forecasting, Atmospheresjm2103, lmg107International Research Institute for Climate and SocietyArticlesNine Justices Ten Years Statistical Retrospective
https://academiccommons.columbia.edu/catalog/ac:199318
Jackson, Robert J.; Vjgnarajah, Thiruvendran10.7916/D800024NFri, 30 Jun 2017 16:49:40 +0000The 2003 Term marked an unprecedented milestone for the Supreme Court for the first time in history nine Justices celebrated full decade presiding together over the nations highest court The continuity of the current Court is especially striking given that on average one new Justice has been appointed approximately every two years since the Courts expansion to nine members in 1837.2 Although the Harvard Law Review has prepared statistical retrospectives in the past3 the last decade presents rare opportunity to study the Court free from the disruptions of intervening appointments Presented here is review of the 823 cases decided by the Court over the past decade Of course bare statistics cannot capture the nuanced interactions among the Justices nor substantiate any particular theory about the complex dynamics of the Court Rather this statistical compilation and the preliminary observations articulated here are intended only as starting point modest effort to showcase trends that deserve closer attention and to jumpstart more robust analyses of how the Court despite its apparent stability has evolved over the past decade.Law reports, digests, etc., Law, Statistics, United States. Supreme Courtrj2317LawArticlesPredicting southern African summer rainfall using a combination of MOS and perfect prognosis
https://academiccommons.columbia.edu/catalog/ac:196920
Landman, Willem A.; Goddard, Lisa M.10.7916/D8959HHWFri, 30 Jun 2017 16:49:40 +0000A statistical-dynamical approach to probabilistic precipitation forecasts of southern African summer rainfall is described and validated. An ensemble of seasonal precipitation and circulation fields is obtained from the ECHAM4.5 atmospheric general circulation model (AGCM). Model output statistics (MOS) then spatially recalibrate the AGCM fields relative to observations. Although the MOS equations are built using the simulation data, in which observed SSTs force the AGCM, the same set of equations can be applied to the predicted data, in which predicted SSTs force the AGCM. The use of prediction data in a set of equations developed for simulations, assumes that the AGCM forecast skill approximates its simulation skill and that the systematic biases of the AGCM do not change in a prediction setting; this assumption is analogous to a perfect prognosis (PP) approach. Probabilistic forecast skill is assessed using this MOS-PP-recalibration scheme for 3 equi-probable categories using a 3-year-out cross-validation approach. High skill scores are found over the north-eastern interior of the region, with marginal skill over the remainder of the austral summer rainfall regions. When skill is assessed for only the wettest and driest of the years, high skill appears over most of the region.Atmospheric circulation, Statistics, Precipitation forecasting, Atmospherewal2113, lmg107International Research Institute for Climate and SocietyArticlesAssessing the predictability of extreme rainfall seasons over southern Africa
https://academiccommons.columbia.edu/catalog/ac:196917
Landman, Willem A.; Botes, Stephanie; Goddard, Lisa M.; Shongwe, Mxolisi10.7916/D8B56JPQFri, 30 Jun 2017 16:49:40 +0000A model output statistics (MOS) technique is developed to investigate the potential rainfall forecast skill for extreme seasons over southern Africa. Rainfall patterns produced by the ECHAM4.5 atmospheric GCM are statistically recalibrated to regional rainfall for the seasons of September–November, December–February, March–May and June–August. Archived records of the GCM simulated fields are related to observed rainfall through a set of canonical correlation analysis (CCA) equations. Probabilistic forecast skill (RPSS and ROC) of MOS-recalibrated simulations for 5 equi-probable categories is assessed using a 3-year-out cross-validation approach. High skill RPSS values are found for the DJF and MAM seasons. Although ROC scores for DJF and MAM are larger than 0.5 for all categories (scores less than 0.5 suggest negative skill), scores for DJF show that the extreme categories are more predictable than the inner categories and scores for MAM show that skill is mostly associated with the extremely wet category. The GCM's ability to reproduce tropical-temperate trough variability constitutes the main source of predictability for DJF and MAM.Atmospheric circulation, Statistics, Precipitation forecasting, Atmospherewal2113, lmg107International Research Institute for Climate and SocietyArticlesStatistical–Dynamical Seasonal Forecasts of Central-Southwest Asian Winter Precipitation
https://academiccommons.columbia.edu/catalog/ac:196890
Tippett, Michael K.; Goddard, Lisa M.; Barnston, Anthony G.10.7916/D8NK3DZFFri, 30 Jun 2017 16:49:40 +0000Interannual precipitation variability in central-southwest (CSW) Asia has been associated with East Asian jet stream variability and western Pacific tropical convection. However, atmospheric general circulation models (AGCMs) forced by observed sea surface temperature (SST) poorly simulate the region’s interannual precipitation variability. The statistical–dynamical approach uses statistical methods to correct systematic deficiencies in the response of AGCMs to SST forcing. Statistical correction methods linking model-simulated Indo–west Pacific precipitation and observed CSW Asia precipitation result in modest, but statistically significant, cross-validated simulation skill in the northeast part of the domain for the period from 1951 to 1998. The statistical–dynamical method is also applied to recent (winter 1998/99 to 2002/03) multimodel, two-tier December–March precipitation forecasts initiated in October. This period includes 4 yr (winter of 1998/99 to 2001/02) of severe drought. Tercile probability forecasts are produced using ensemble-mean forecasts and forecast error estimates. The statistical–dynamical forecasts show enhanced probability of below-normal precipitation for the four drought years and capture the return to normal conditions in part of the region during the winter of 2002/03.Precipitation forecasting, Statistics, Climatic changes--Forecasting, Atmospheremkt14, lmg107, agb52International Research Institute for Climate and Society, Applied Physics and Applied MathematicsArticlesPrior Design for Dependent Dirichlet Processes: An Application to Marathon Modeling
https://academiccommons.columbia.edu/catalog/ac:195557
Pradier, Melanie F.; Ruiz, Francisco Jesus Rodriguez; Perez-Cruz, Fernando10.7916/D8SN08V7Fri, 30 Jun 2017 00:47:30 +0000This paper presents a novel application of Bayesian nonparametrics (BNP) for marathon data modeling. We make use of two well-known BNP priors, the single-p dependent Dirichlet process and the hierarchical Dirichlet process, in order to address two different problems. First, we study the impact of age, gender and environment on the runners’ performance. We derive a fair grading method that allows direct comparison of runners regardless of their age and gender. Unlike current grading systems, our approach is based not only on top world records, but on the performances of all runners. The presented methodology for comparison of densities can be adopted in many other applications straightforwardly, providing an interesting perspective to build dependent Dirichlet processes. Second, we analyze the running patterns of the marathoners in time, obtaining information that can be valuable for training purposes. We also show that these running patterns can be used to predict finishing time given intermediate interval measurements. We apply our models to New York City, Boston and London marathons.Marathon running, Running races--Data processing, Nonparametric statistics, Stochastic processes, Statistics, Information sciencefr2392Data Science InstituteArticlesAre We Ready for Mass Fatality Incidents? Preparedness of the US Mass Fatality Infrastructure
https://academiccommons.columbia.edu/catalog/ac:192811
Merrill, Jacqueline A.; Orr, Mark; Chen, Daniel; Zhi, Qi; Gershon, Robyn R.10.7916/D8125SF8Fri, 30 Jun 2017 00:45:26 +0000Objective To assess the preparedness of the US mass fatality infrastructure, we developed and tested metrics for 3 components of preparedness: organizational, operational, and resource sharing networks.
Methods In 2014, data were collected from 5 response sectors: medical examiners and coroners, the death care industry, health departments, faith-based organizations, and offices of emergency management. Scores were calculated within and across sectors and a weighted score was developed for the infrastructure.
Results A total of 879 respondents reported highly variable organizational capabilities: 15% had responded to a mass fatality incident (MFI); 42% reported staff trained for an MFI, but only 27% for an MFI involving hazardous contaminants. Respondents estimated that 75% of their staff would be willing and able to respond, but only 53% if contaminants were involved. Most perceived their organization as somewhat prepared, but 13% indicated “not at all.” Operational capability scores ranged from 33% (death care industry) to 77% (offices of emergency management). Network capability analysis found that only 42% of possible reciprocal relationships between resource-sharing partners were present. The cross-sector composite score was 51%; that is, half the key capabilities for preparedness were in place.
Conclusions The sectors in the US mass fatality infrastructure report suboptimal capability to respond. National leadership is needed to ensure sector-specific and infrastructure-wide preparedness for a large-scale MFI.Mass casualties, Medical care, Emergency management, Disaster medicine, Medical sciences, Health services administration, Statisticsjam119, rg405NursingArticlesMicronutrients in HIV: A Bayesian MetaAnalysis
https://academiccommons.columbia.edu/catalog/ac:210017
Carter, George M.; Indyk, Debbie; Johnson, Matthew S.; Andreae, Michael; Suslov, Kathryn; Busani, Sudharani; Esmaeili, Aryan; Sacks, Henry S.10.7916/D8WM1D7SFri, 30 Jun 2017 00:43:41 +0000Background: Approximately 28.5 million people living with HIV are eligible for treatment (CD4&500), but currently have no access to antiretroviral therapy. Reduced serum level of micronutrients is common in HIV disease. Micronutrient supplementation (MNS) may mitigate disease progression and mortality. Objectives: We synthesized evidence on the effect of micronutrient supplementation on mortality and rate of disease progression in HIV disease.
Methods: We searched MEDLINE, EMBASE, the Cochrane Central, AMED and CINAHL databases through December 2014, without language restriction, for studies of greater than 3 micronutrients versus any or no comparator. We built a hierarchical Bayesian random effects model to synthesize results. Inferences are based on the posterior distribution of the population effects; posterior distributions were approximated by Markov chain Monte Carlo in OpenBugs.
Principal Findings: From 2166 initial references, we selected 49 studies for full review and identified eight reporting on disease progression and/or mortality. Bayesian synthesis of data from 2,249 adults in three studies estimated the relative risk of disease progression in subjects on MNS vs. control as 0.62 (95% credible interval, 0.37, 0.96). Median number needed to treat is 8.4 (4.8, 29.9) and the Bayes Factor 53.4. Based on data reporting on 4,095 adults reporting mortality in 7 randomized controlled studies, the RR was 0.84 (0.38, 1.85), NNT is 25 (4.3, ∞).
Conclusions: MNS significantly and substantially slows disease progression in HIV+ adults not on ARV, and possibly reduces mortality. Micronutrient supplements are effective in reducing progression with a posterior probability of 97.9%. Considering MNS low cost and lack of adverse effects, MNS should be standard of care for HIV+ adults not yet on ARV.Trace elements in nutrition, HIV infections--Treatment, Public health, Health services administration, Medical care, Statisticsmsj2119Human DevelopmentArticlesStatistics of surface divergence and their relation to air-water gas transfer velocity
https://academiccommons.columbia.edu/catalog/ac:194442
Asher, William E.; Liang, Hanzhuang; Zappa, Christopher J.; Loewen, Mark R.; Mukto, Moniz A.; Litchendorf, Trina M.; Jessup, Andrew T.10.7916/D8571BVQFri, 30 Jun 2017 00:40:37 +0000Air-sea gas fluxes are generally defined in terms of the air/water concentration difference of the gas and the gas transfer velocity,kL. Because it is difficult to measure kLin the ocean, it is often parameterized using more easily measured physical properties. Surface divergence theory suggests that infrared (IR) images of the water surface, which contain information concerning the movement of water very near the air-water interface, might be used to estimatekL. Therefore, a series of experiments testing whether IR imagery could provide a convenient means for estimating the surface divergence applicable to air-sea exchange were conducted in a synthetic jet array tank embedded in a wind tunnel. Gas transfer velocities were measured as a function of wind stress and mechanically generated turbulence; laser-induced fluorescence was used to measure the concentration of carbon dioxide in the top 300 μm of the water surface; IR imagery was used to measure the spatial and temporal distribution of the aqueous skin temperature; and particle image velocimetry was used to measure turbulence at a depth of 1 cm below the air-water interface. It is shown that an estimate of the surface divergence for both wind-shear driven turbulence and mechanically generated turbulence can be derived from the surface skin temperature. The estimates derived from the IR images are compared to velocity field divergences measured by the PIV and to independent estimates of the divergence made using the laser-induced fluorescence data. Divergence is shown to scale withkLvalues measured using gaseous tracers as predicted by conceptual models for both wind-driven and mechanically generated turbulence.Ocean-atmosphere interaction, Divergence theorem, Gas flow--Mathematical models, Surface waves (Oceanography), Oceanography, Mathematics, Statisticscjz9Lamont-Doherty Earth ObservatoryArticlesDistributed Bayesian Computation and Self-Organized Learning in Sheets of Spiking Neurons with Local Lateral Inhibition
https://academiccommons.columbia.edu/catalog/ac:192253
Buesing, Lars; Habenschuss, Stefan; Bill, Johannes; Nessler, Bernhard; Maass, Wolfgang; Legenstein, Robert10.7916/D8862G4XThu, 29 Jun 2017 23:25:22 +0000During the last decade, Bayesian probability theory has emerged as a framework in cognitive science and neuroscience for describing perception, reasoning and learning of mammals. However, our understanding of how probabilistic computations could be organized in the brain, and how the observed connectivity structure of cortical microcircuits supports these calculations, is rudimentary at best. In this study, we investigate statistical inference and self-organized learning in a spatially extended spiking network model, that accommodates both local competitive and large-scale associative aspects of neural information processing, under a unified Bayesian account. Specifically, we show how the spiking dynamics of a recurrent network with lateral excitation and local inhibition in response to distributed spiking input, can be understood as sampling from a variational posterior distribution of a well-defined implicit probabilistic model. This interpretation further permits a rigorous analytical treatment of experience-dependent plasticity on the network level. Using machine learning theory, we derive update rules for neuron and synapse parameters which equate with Hebbian synaptic and homeostatic intrinsic plasticity rules in a neural implementation. In computer simulations, we demonstrate that the interplay of these plasticity rules leads to the emergence of probabilistic local experts that form distributed assemblies of similarly tuned cells communicating through lateral excitatory connections. The resulting sparse distributed spike code of a well-adapted network carries compressed information on salient input features combined with prior experience on correlations among them. Our theory predicts that the emergence of such efficient representations benefits from network architectures in which the range of local inhibition matches the spatial extent of pyramidal cells that share common afferent input.Neuroplasticity, Neurons, Inhibition, Bayesian statistical decision theory, Neurosciences, Molecular biology, StatisticsStatisticsArticlesThe WTO Dispute Settlement System: 1995-2010 Some Descriptive Statistics
https://academiccommons.columbia.edu/catalog/ac:192343
Mavroidis, Petros C.; Horn, Henrik; Johannesson, Louise10.7916/D8B27TZZThu, 29 Jun 2017 23:15:50 +0000This paper reports descriptive statistics based on the WTO Dispute Settlement Data Set (Ver. 3.0). The data set contains approximately 67 000 observations on a wide range of aspects of the Dispute Settlement (DS) system, and is exclusively based on official WTO documents. It covers all 426 WTO disputes initiated through the official filing of a Request for Consultations from January 1, 1995, until August 11, 2011, and for these disputes it includes events occurring until July 28, 2011.1 In this paper however, we will omit data pertaining to 2011 and only consider the full years 1995—2010. In order to shed some light on differences across WTO Members in participation in the DS system, we will divide Members into five groups, as specified in detail in Table 1. Broadly speaking, these groups are: G2 - The European Union (EU), and the United States (US); IND - Other industrialized countries; DEV - Developing countries other than LDC; LDC - Least developed countries; BIC - Brazil, India and China. The EU is taken to be EU-15, since the enlargements came relatively late during the period we cover. For the most part, the choice in this regard makes little difference quantitatively, since most of the 12 countries acceding to the EU in 2004 and 2007 have been relatively inactive in the WTO. The LDC group corresponds to the list of LDCs prepared by the United Nations. A more discretionary line is drawn between IND and DEV. We have classified under IND, OECD Members, the non-OECD Members among the 12 countries that most recently became members of the EU, those that are currently at an advanced stage of their accession negotiations, as well as countries that are not OECD Members but have a very high per capita income, such as Singapore. The DEV group consists of all countries which do not fit into either of the above mentioned categories, and are not BIC countries either. BIC refers to Brazil, India, and China: the sheer number of cases in which Brazil, India and China have participated, as well as their overall participation in WTO, led us to these three countries as a separate group. The paper is structured as follows: Section 2 highlights the evolution of the total use of the DS system; Section 3 discusses some aspects of participation of the groups defined above when acting as complainants or respondents; Section 4 deals with the subject-matter of disputes; Section 5 highlights a few aspects of countries’ success with regard to the legal claims they made before panels; Section 6 provides information as to the nationality and the appointment process of WTO panelists; Section 7 focuses on the duration of dispute settlement procedures at different stages of the adjudication process; Section 8 concludes.Dispute resolution (Law), Statistics, Developed countries, Developing countries, Law, International law, World Trade Organization, European Unionpm2030LawArticlesIdentification and Validation of Structures in Neural Population Responses
https://academiccommons.columbia.edu/catalog/ac:b2rbnzs7j6
Elsayed, Gamaleldin Fathy10.7916/D8G73S1WThu, 29 Jun 2017 19:21:36 +0000A fundamental challenge of neuroscience is to understand how interconnected populations of neurons give rise to the remarkable computational abilities of our brains. Large neural datasets offer promise, but they are perilous: they are too complex to be studied with traditional single-neuron analyses, and thus require new analyses that can uncover structure at the level of the population. However, since these analyses operate on large datasets, our intuition whether structure is significant breaks down. Hence, we run the risk of over-interpreting structure from the population data that may have a simple explanation. Thus, with population analysis methods, there is also a need for methods that can validate the significance of structure identified. In this dissertation, I discuss topics covering both the identification and the validation of structure in population data. In the first part, I discuss novel methods for uncovering the computational strategy employed by the motor cortex to flexibly switch between different neural computations. I demonstrate that collective activity patterns of motor cortex neurons related to different computations are orthogonal yet can still be linked, indicating a degree of flexibility that was not displayed or predicted by existing cortical models. In the second part, I discuss a novel analytical framework to rigorously test the novelty of population-level findings, given a specified set of primary features such as correlations across time, neurons and experimental conditions. This framework provides a general tool for validating population findings across the brain and across population-level hypotheses.Neurosciences, Electrical engineering, Statistics, NeuronsNeurobiology and BehaviorThesesGLMLE: graph-limit enabled fast computation for fitting exponential random graph models to large social networks
https://academiccommons.columbia.edu/catalog/ac:185410
He, Ran; Zheng, Tian10.7916/D8S46QVQThu, 29 Jun 2017 03:44:14 +0000Large network, as a form of big data, has received increasing amount of attention in data science, especially for large social network, which is reaching the size of hundreds of millions, with daily interactions on the scale of billions. Thus analyzing and modeling these data to understand the connectivities and dynamics of large networks is important in a wide range of scientific fields. Among popular models, exponential random graph models (ERGMs) have been developed to study these complex networks by directly modeling network structures and features. ERGMs, however, are hard to scale to large networks because maximum likelihood estimation of parameters in these models can be very difficult, due to the unknown normalizing constant. Alternative strategies based on Markov chain Monte Carlo (MCMC) draw samples to approximate the likelihood, which is then maximized to obtain the maximum likelihood estimators (MLE). These strategies have poor convergence due to model degeneracy issues and cannot be used on large networks. Chatterjee et al. (Ann Stat 41:2428–2461, 2013) propose a new theoretical framework for estimating the parameters of ERGMs by approximating the normalizing constant using the emerging tools in graph theory—graph limits. In this paper, we construct a complete computational procedure built upon their results with practical innovations which is fast and is able to scale to large networks. More specifically, we evaluate the likelihood via simple function approximation of the corresponding ERGM’s graph limit and iteratively maximize the likelihood to obtain the MLE. We also discuss the methods of conducting likelihood ratio test for ERGMs as well as related issues. Through simulation studies and real data analysis of two large social networks, we show that our new method outperforms the MCMC-based method, especially when the network size is large (more than 100 nodes). One limitation of our approach, inherited from the limitation of the result of Chatterjee et al. (Ann Stat 41:2428–2461, 2013), is that it works only for sequences of graphs with a positive limiting density, i.e., dense graphs.Statisticsrh2528, tz33StatisticsArticlesEffect of Childhood Victimization on Occupational Prestige and Income Trajectories
https://academiccommons.columbia.edu/catalog/ac:184166
Fernandez, Cristina A.; Christ, Sharon L.; LeBlanc, William G.; McCollister, Kathyrn E.; Arheart, Kristopher L.; Dietz, Noella A.; Fleming, Lora E.; Muntaner, Carles; Muennig, Peter A.; Lee, David J.10.7916/D88C9V3DThu, 29 Jun 2017 03:43:09 +0000Background
Violence toward children (childhood victimization) is a major public health problem, with long-term consequences on economic well-being. The purpose of this study was to determine whether childhood victimization affects occupational prestige and income in young adulthood. We hypothesized that young adults who experienced more childhood victimizations would have less prestigious jobs and lower incomes relative to those with no victimization history. We also explored the pathways in which childhood victimization mediates the relationships between background variables, such as parent’s educational impact on the socioeconomic transition into adulthood.
Methods
A nationally representative sample of 8,901 young adults aged 18–28 surveyed between 1999–2009 from the National Longitudinal Survey of Youth 1997 (NLSY) were analyzed. Covariate-adjusted multivariate linear regression and path models were used to estimate the effects of victimization and covariates on income and prestige levels and on income and prestige trajectories. After each participant turned 18, their annual 2002 Census job code was assigned a yearly prestige score based on the 1989 General Social Survey, and their annual income was calculated via self-reports. Occupational prestige and annual income are time-varying variables measured from 1999–2009. Victimization effects were tested for moderation by sex, race, and ethnicity in the multivariate models.
Results
Approximately half of our sample reported at least one instance of childhood victimization before the age of 18. Major findings include 1) childhood victimization resulted in slower income and prestige growth over time, and 2) mediation analyses suggested that this slower prestige and earnings arose because victims did not get the same amount of education as non-victims.
Conclusions
Results indicated that the consequences of victimization negatively affected economic success throughout young adulthood, primarily by slowing the growth in prosperity due to lower education levels.Public health, Sociology, Statisticspm124Health Policy and ManagementArticlesDrinking Patterns and Alcohol Use Disorders in São Paulo, Brazil: The Role of Neighborhood Social Deprivation and Socioeconomic Status
https://academiccommons.columbia.edu/catalog/ac:184775
Silveira, Camila Magalhaes; Siu, Erica Rosanna; Anthony, James C.; Saito, Luis Paulo; Guerra de Andrade, Arthur; Kutschenko, Andressa; Viana, Maria Carmen; Wang, Yuan-Pang; Martins, Silvia S.; Andrade, Laura Helena10.7916/D89C6W9PThu, 29 Jun 2017 03:42:00 +0000Background
Research conducted in high-income countries has investigated influences of socioeconomic inequalities on drinking outcomes such as alcohol use disorders (AUD), however, associations between area-level neighborhood social deprivation (NSD) and individual socioeconomic status with these outcomes have not been explored in Brazil. Thus, we investigated the role of these factors on drink-related outcomes in a Brazilian population, attending to male-female variations.
Methods
A multi-stage area probability sample of adult household residents in the São Paulo Metropolitan Area was assessed using the WHO Composite International Diagnostic Interview (WMH-CIDI) (n = 5,037). Estimation focused on prevalence and correlates of past-year alcohol disturbances [heavy drinking of lower frequency (HDLF), heavy drinking of higher frequency (HDHF), abuse, dependence, and DMS-5 AUD] among regular users (RU); odds ratio (OR) were obtained.
Results
Higher NSD, measured as an area-level variable with individual level variables held constant, showed an excess odds for most alcohol disturbances analyzed. Prevalence estimates for HDLF and HDHF among RU were 9% and 20%, respectively, with excess odds in higher NSD areas; schooling (inverse association) and low income were associated with male HDLF. The only individual-level association with female HDLF involved employment status. Prevalence estimates for abuse, dependence, and DSM-5 AUD among RU were 8%, 4%, and 8%, respectively, with excess odds of: dependence in higher NSD areas for males; abuse and AUD for females. Among RU, AUD was associated with unemployment, and low education with dependence and AUD.Social sciences--Research, Statistics, Public healthssm2183EpidemiologyArticlesDynamical Phenotyping: Using Temporal Analysis of Clinically Collected Physiologic Data to Stratify Populations
https://academiccommons.columbia.edu/catalog/ac:184147
Albers, David J.; Elhadad, Noemie; Tabak, E.; Perotte, Adler; Hripcsak, George M.10.7916/D8W9581VThu, 29 Jun 2017 03:41:56 +0000Using glucose time series data from a well measured population drawn from an electronic health record (EHR) repository, the variation in predictability of glucose values quantified by the time-delayed mutual information (TDMI) was explained using a mechanistic endocrine model and manual and automated review of written patient records. The results suggest that predictability of glucose varies with health state where the relationship (e.g., linear or inverse) depends on the source of the acuity. It was found that on a fine scale in parameter variation, the less insulin required to process glucose, a condition that correlates with good health, the more predictable glucose values were. Nevertheless, the most powerful effect on predictability in the EHR subpopulation was the presence or absence of variation in health state, specifically, in- and out-of-control glucose versus in-control glucose. Both of these results are clinically and scientifically relevant because the magnitude of glucose is the most commonly used indicator of health as opposed to glucose dynamics, thus providing for a connection between a mechanistic endocrine model and direct insight to human health via clinically collected data.Medicine, Endocrinology, Statisticsdja2119, ne60, ajp2120, gh13Biomedical InformaticsArticlesStatistical Searches for Microlensing Events in Large, Non-Uniformly Sampled Time-Domain Surveys: A Test Using Palomar Transient Factory Data
https://academiccommons.columbia.edu/catalog/ac:185413
Price-Whelan, Adrian Michael; Agüeros, Marcel Andre; Fournier, Amanda P.; Street, Rachel; Ofek, Eran O.; Covey, Kevin R.; Levitan, David; Laher, Russ R.; Sesar, Branimir; Surace, Jason10.7916/D8HM57CMThu, 29 Jun 2017 03:41:47 +0000Many photometric time-domain surveys are driven by specific goals, such as searches for supernovae or transiting exoplanets, which set the cadence with which fields are re-imaged. In the case of the Palomar Transient Factory (PTF), several sub-surveys are conducted in parallel, leading to non-uniform sampling over its ~20,000 deg2 footprint. While the median 7.26 deg2 PTF field has been imaged ~40 times in the R band, ~2300 deg2 have been observed >100 times. We use PTF data to study the trade off between searching for microlensing events in a survey whose footprint is much larger than that of typical microlensing searches, but with far-from-optimal time sampling. To examine the probability that microlensing events can be recovered in these data, we test statistics used on uniformly sampled data to identify variables and transients. We find that the von Neumann ratio performs best for identifying simulated microlensing events in our data. We develop a selection method using this statistic and apply it to data from fields with >10 R-band observations, 1.1 × 109 light curves, uncovering three candidate microlensing events. We lack simultaneous, multi-color photometry to confirm these as microlensing events. However, their number is consistent with predictions for the event rate in the PTF footprint over the survey's three years of operations, as estimated from near-field microlensing models. This work can help constrain all-sky event rate predictions and tests microlensing signal recovery in large data sets, which will be useful to future time-domain surveys, such as that planned with the Large Synoptic Survey Telescope.Astronomy, Statisticsamp2217, maa17AstronomyArticlesSurveying Hard-to-Reach Groups Through Sampled Respondents in a Social Network
https://academiccommons.columbia.edu/catalog/ac:185373
McCormick, Tyler H.; Zheng, Tian; He, Ran; Kolaczyk, Eric10.7916/D8Z0372NThu, 29 Jun 2017 03:41:08 +0000The sampling frame in most social science surveys misses members of certain groups, such as the homeless or individuals living with HIV. These groups are known as hard-to-reach groups. One strategy for learning about these groups, or subpopulations, involves reaching hard-to-reach group members through their social network. In this paper we compare the efficiency of two common methods for subpopulation size estimation using data from standard surveys. These designs are examples of mental link tracing designs. These designs begin with a randomly sampled set of network members (nodes) and then reach other nodes indirectly through questions asked to the sampled nodes. Mental link tracing designs cost significantly less than traditional link tracing designs, yet introduce additional sources of potential bias. We examine the influence of one such source of bias using simulation studies. We then demonstrate our findings using data from the General Social Survey collected in 2004 and 2006. Additionally, we provide survey design suggestions for future surveys incorporating such designs.Statistics, Social sciences--Researchthm2105, tz33, rh2528StatisticsArticlesLatent demographic profile estimation in hard-to-reach groups
https://academiccommons.columbia.edu/catalog/ac:184956
McCormick, Tyler H.; Zheng, Tian10.7916/D8F76BFQThu, 29 Jun 2017 03:41:07 +0000The sampling frame in most social science surveys excludes members of certain groups, known as hard-to-reach groups. These groups, or subpopulations, may be difficult to access (the homeless, e.g.), camouflaged by stigma (individuals with HIV/AIDS), or both (commercial sex workers). Even basic demographic information about these groups is typically unknown, especially in many developing nations. We present statistical models which leverage social network structure to estimate demographic characteristics of these subpopulations using Aggregated relational data (ARD), or questions of the form “How many X’s do you know?” Unlike other network-based techniques for reaching these groups, ARD require no special sampling strategy and are easily incorporated into standard surveys. ARD also do not require respondents to reveal their own group membership. We propose a Bayesian hierarchical model for estimating the demographic characteristics of hard-to-reach groups, or latent demographic profiles, using ARD. We propose two estimation techniques. First, we propose a Markov-chain Monte Carlo algorithm for existing data or cases where the full posterior distribution is of interest. For cases when new data can be collected, we propose guidelines and, based on these guidelines, propose a simple estimate motivated by a missing data approach. Using data from McCarty et al. [Human Organization 60 (2001) 28–39], we estimate the age and gender profiles of six hard-to-reach groups, such as individuals who have HIV, women who were raped, and homeless persons. We also evaluate our simple estimates using simulation studies.Statisticsthm2105, tz33StatisticsArticlesA Practical Guide to Measuring Social Structure Using Indirectly Observed Network Data
https://academiccommons.columbia.edu/catalog/ac:185370
McCormick, Tyler H.; Moussa, Amal; DiPrete, Thomas A.; Ruf, Johannes; Gelman, Andrew E.; Teitler, Julien O.; Zheng, Tian10.7916/D86H4G9DThu, 29 Jun 2017 03:41:05 +0000Aggregated relational data (ARD) are an increasingly common tool for learning about social networks through standard surveys. Recent statistical advances present social scientists with new options for analyzing such data. In this article, we propose guidelines for learning about various network processes using ARD and a template to aid practitioners. We first propose that ARD can be used to measure “social distance” between a respondent and a subpopulation (individuals named Kevin, those in prison, or those serving in the military). We then present common methods for analyzing these data and associate each of these methods with a specific way of measuring social distance, thus associating statistical tools with their underlying social science phenomena. We examine the implications of using each of these social distance measures using an Internet survey about contemporary political issues.Statistics, Social sciences--Researchthm2105, am2810, tad61, ag389, jot8, tz33Sociology, Statistics, Social WorkArticlesBiodiversity and Ecosystem Multi-Functionality: Observed Relationships in Smallholder Fallows in Western Kenya
https://academiccommons.columbia.edu/catalog/ac:184813
Sircely, Jason; Naeem, Shahid10.7916/D8V986XHThu, 29 Jun 2017 03:40:59 +0000Recent studies indicate that species richness can enhance the ability of plant assemblages to support multiple ecosystem functions. To understand how and when ecosystem services depend on biodiversity, it is valuable to expand beyond experimental grasslands. We examined whether plant diversity improves the capacity of agroecosystems to sustain multiple ecosystem services—production of wood and forage, and two elements of soil formation—in two types of smallholder fallows in western Kenya. In 18 grazed and 21 improved fallows, we estimated biomass and quantified soil organic carbon, soil base cations, sand content, and soil infiltration capacity. For four ecosystem functions (wood biomass, forage biomass, soil base cations, steady infiltration rates) linked to the focal ecosystem services, we quantified ecosystem service multi-functionality as (1) the proportion of functions above half-maximum, and (2) mean percentage excess above mean function values, and assessed whether plant diversity or environmental favorability better predicted multi-functionality. In grazed fallows, positive effects of plant diversity best explained the proportion above half-maximum and mean percentage excess, the former also declining with grazing intensity. In improved fallows, the proportion above half-maximum was not associated with soil carbon or plant diversity, while soil carbon predicted mean percentage excess better than diversity. Grazed fallows yielded stronger evidence for diversity effects on multi-functionality, while environmental conditions appeared more influential in improved fallows. The contrast in diversity-multi-functionality relationships among fallow types appears related to differences in management and associated factors including disturbance and species composition. Complementary effects of species with contrasting functional traits on different functions and multi-functional species may have contributed to diversity effects in grazed fallows. Biodiversity and environmental favorability may enhance the capacity of smallholder fallows to simultaneously provide multiple ecosystem services, yet their effects are likely to vary with fallow management.Ecology, Statisticsjas2162, sn2121Ecology, Evolution, and Environmental BiologyArticlesPopulation Physiology: Leveraging Electronic Health Record Data to Understand Human Endocrine Dynamics
https://academiccommons.columbia.edu/catalog/ac:184150
Albers, David J.; Hripcsak, George M.; Schmidt, J. Michael10.7916/D8KW5DWSThu, 29 Jun 2017 03:40:36 +0000Studying physiology and pathophysiology over a broad population for long periods of time is difficult primarily because collecting human physiologic data can be intrusive, dangerous, and expensive. One solution is to use data that have been collected for a different purpose. Electronic health record (EHR) data promise to support the development and testing of mechanistic physiologic models on diverse populations and allow correlation with clinical outcomes, but limitations in the data have thus far thwarted such use. For example, using uncontrolled population-scale EHR data to verify the outcome of time dependent behavior of mechanistic, constructive models can be difficult because: (i) aggregation of the population can obscure or generate a signal, (ii) there is often no control population with a well understood health state, and (iii) diversity in how the population is measured can make the data difficult to fit into conventional analysis techniques. This paper shows that it is possible to use EHR data to test a physiological model for a population and over long time scales. Specifically, a methodology is developed and demonstrated for testing a mechanistic, time-dependent, physiological model of serum glucose dynamics with uncontrolled, population-scale, physiological patient data extracted from an EHR repository. It is shown that there is no observable daily variation the normalized mean glucose for any EHR subpopulations. In contrast, a derived value, daily variation in nonlinear correlation quantified by the time-delayed mutual information (TDMI), did reveal the intuitively expected diurnal variation in glucose levels amongst a random population of humans. Moreover, in a population of continuously (tube) fed patients, there was no observable TDMI-based diurnal signal. These TDMI-based signals, via a glucose insulin model, were then connected with human feeding patterns. In particular, a constructive physiological model was shown to correctly predict the difference between the general uncontrolled population and a subpopulation whose feeding was controlled.Statistics, Medicinedja2119, gh13, mjs2134Biomedical Informatics, NeurologyArticlesIncrease in Diarrheal Disease Associated with Arsenic Mitigation in Bangladesh
https://academiccommons.columbia.edu/catalog/ac:184172
Wu, Jianyong; Jahangir Alam, Yasuyuki Akita; van Geen, Alexander; Ahmed, Kazi Matin; Culligan, Patricia J.; Escamilla, Veronica; Feighery, John; Ferguson, Andrew S.; Knappett, Peter; Mailloux, Brian Justin; McKay, Larry D.; Serre, Marc L.; Streatfield, P. Kim; Yunus, Mohammad; Emch, Michael10.7916/D87D2T01Thu, 29 Jun 2017 03:40:33 +0000Background
Millions of households throughout Bangladesh have been exposed to high levels of arsenic (As) causing various deadly diseases by drinking groundwater from shallow tubewells for the past 30 years. Well testing has been the most effective form of mitigation because it has induced massive switching from tubewells that are high (>50 µg/L) in As to neighboring wells that are low in As. A recent study has shown, however, that shallow low-As wells are more likely to be contaminated with the fecal indicator E. coli than shallow high-As wells, suggesting that well switching might lead to an increase in diarrheal disease.
Methods
Approximately 60,000 episodes of childhood diarrhea were collected monthly by community health workers between 2000 and 2006 in 142 villages of Matlab, Bangladesh. In this cross-sectional study, associations between childhood diarrhea and As levels in tubewell water were evaluated using logistic regression models.
Results
Adjusting for wealth, population density, and flood control by multivariate logistic regression, the model indicates an 11% (95% confidence intervals (CIs) of 4–19%) increase in the likelihood of diarrhea in children drinking from shallow wells with 10–50 µg/L As compared to shallow wells with >50 µg/L As. The same model indicates a 26% (95%CI: 9–42%) increase in diarrhea for children drinking from shallow wells with ≤10 µg/L As compared to shallow wells with >50 µg/L As.
Conclusion
Children drinking water from shallow low As wells had a higher prevalence of diarrhea than children drinking water from high As wells. This suggests that the health benefits of reducing As exposure may to some extent be countered by an increase in childhood diarrhea.Public health, Statisticsafv2, pjc2104, af2460, bjm2103, me2159Civil Engineering and Engineering Mechanics, Environmental Science (Barnard College)ArticlesPolymorphisms in the Mitochondrial DNA Control Region and Frailty in Older Adults
https://academiccommons.columbia.edu/catalog/ac:184807
Moore, Anne Z.; Biggs, Mary L.; O'Connor, Ashley; Matteini, Amy; McGuire, Sarah; Beamer, Brock A.; Fallin, M. Danielle; Waltson, Jeremy; Fried, Linda P.; Chakravarti, Aravinda; Arking, Dan E.10.7916/D83R0RRHThu, 29 Jun 2017 03:40:06 +0000Background:
Mitochondria contribute to the dynamics of cellular metabolism, the production of reactive oxygen species, and apoptotic pathways. Consequently, mitochondrial function has been hypothesized to influence functional decline and vulnerability to disease in later life. Mitochondrial genetic variation may contribute to altered susceptibility to the frailty syndrome in older adults.
Methodology/Principal Findings:
To assess potential mitochondrial genetic contributions to the likelihood of frailty, mitochondrial DNA (mtDNA) variation was compared in frail and non-frail older adults. Associations of selected SNPs with a muscle strength phenotype were also explored. Participants were selected from the Cardiovascular Health Study (CHS), a population-based observational study (1989–1990, 1992–1993). At baseline, frailty was identified as the presence of three or more of five indicators (weakness, slowness, shrinking, low physical activity, and exhaustion). mtDNA variation was assessed in a pilot study, including 315 individuals selected as extremes of the frailty phenotype, using an oligonucleotide sequencing microarray based on the Revised Cambridge Reference Sequence. Three mtDNA SNPs were statistically significantly associated with frailty across all pilot participants or in sex-stratified comparisons: mt146, mt204, and mt228. In addition to pilot participants, 4,459 additional men and women with frailty classifications, and an overlapping subset of 4,453 individuals with grip strength measurements, were included in the study population genotyped at mt204 and mt228. In the study population, the mt204 C allele was associated with greater likelihood of frailty (adjusted odds ratio = 2.04, 95% CI = 1.07–3.60, p = 0.020) and lower grip strength (adjusted coefficient = −2.04, 95% CI = −3.33– −0.74, p = 0.002).
Conclusions:
This study supports a role for mitochondrial genetic variation in the frailty syndrome and later life muscle strength, demonstrating the importance of the mitochondrial genome in complex geriatric phenotypes.Genetics, Medicine, Statisticslf2296EpidemiologyArticlesOn Bootstrap Tests of Symmetry About an Unknown Median
https://academiccommons.columbia.edu/catalog/ac:184965
Zheng, Tian; Gastwirth, Joseph L.10.7916/D8X9296PThu, 29 Jun 2017 03:40:03 +0000It is important to examine the symmetry of an underlying distribution before applying some statistical procedures to a data set. For example, in the Zuni School District case, a formula originally developed by the Department of Education trimmed 5% of the data symmetrically from each end. The validity of this procedure was questioned at the hearing by Chief Justice Roberts. Most tests of symmetry (even nonparametric ones) are not distribution free in finite sample sizes. Hence, using asymptotic distribution may not yield an accurate type I error rate or/and loss of power in small samples. Bootstrap resampling from a symmetric empirical distribution function fitted to the data is proposed to improve the accuracy of the calculated p-value of several tests of symmetry. The results show that the bootstrap method is superior to previously used approaches relying on the asymptotic distribution of the tests that assumed the data come from a normal distribution. Incorporating the bootstrap estimate in a recently proposed test due to Miao, Gel and Gastwirth (2006) preserved its level and shows it has reasonable power properties on the family of distribution evaluated.Statisticstz33StatisticsArticlesHow many people do you know?: Efficiently estimating personal network size
https://academiccommons.columbia.edu/catalog/ac:185367
Zheng, Tian; Salganik, Matthew J.; McCormick, Tyler H.10.7916/D8FX78BTThu, 29 Jun 2017 03:40:03 +0000In this paper we develop a method to estimate both individual social network size (i.e., degree) and the distribution of network sizes in a population by asking respondents how many people they know in specific subpopulations (e.g., people named Michael). Building on the scale-up method of Killworth et al. and other previous attempts to estimate individual network size, we propose a latent non-random mixing model which resolves three known problems with previous approaches. As a byproduct, our method also provides estimates of the rate of social mixing between population groups. We demonstrate the model using a sample of 1,370 adults originally collected by McCarty et al. (2001). Based on insights developed during the statistical modeling, we conclude by offering practical guidelines for the design of future surveys to estimate social network size. Most importantly, we show that if the first names to be asked about are chosen properly, the simple scale-up degree estimates can enjoy the same bias-reduction as that from the our more complex latent non-random mixing model.Statistics, Social sciences--Researchtz33, thm2105StatisticsArticlesSigns of the 2009 Influenza Pandemic in the New York-Presbyterian Hospital Electronic Health Records
https://academiccommons.columbia.edu/catalog/ac:184153
Khiabanian, Hossein; Holmes, Antony B.; Kelly, Brendan J.; Gururaj, Mrinalini; Hripcsak, George M.; Rabadan, Raul10.7916/D82V2F0DThu, 29 Jun 2017 03:39:38 +0000Background
In June of 2009, the World Health Organization declared the first influenza pandemic of the 21st century, and by July, New York City's New York-Presbyterian Hospital (NYPH) experienced a heavy burden of cases, attributable to a novel strain of the virus (H1N1pdm).
Methods and Results
We present the signs in the NYPH electronic health records (EHR) that distinguished the 2009 pandemic from previous seasonal influenza outbreaks via various statistical analyses. These signs include (1) an increase in the number of patients diagnosed with influenza, (2) a preponderance of influenza diagnoses outside of the normal flu season, and (3) marked vaccine failure. The NYPH EHR also reveals distinct age distributions of patients affected by seasonal influenza and the pandemic strain, and via available longitudinal data, suggests that the two may be associated with distinct sets of comorbid conditions as well. In particular, we find significantly more pandemic flu patients with diagnoses associated with asthma and underlying lung disease. We further observe that the NYPH EHR is capable of tracking diseases at a resolution as high as particular zip codes in New York City.
Conclusion
The NYPH EHR permits early detection of pandemic influenza and hypothesis generation via identification of those significantly associated illnesses. As data standards develop and databases expand, EHRs will contribute more and more to disease detection and the discovery of novel disease associations.Medicine, Statistics, Public healthhk2524, abh2138, gh13, rr2579Biomedical InformaticsArticlesDiscovering influential variables: A method of partitions
https://academiccommons.columbia.edu/catalog/ac:184953
Chernoff, Herman; Lo, Shaw-Hwa; Zheng, Tian10.7916/D8PR7TVMThu, 29 Jun 2017 03:39:38 +0000A trend in all scientific disciplines, based on advances in technology, is the increasing availability of high dimensional data in which are buried important information. A current urgent challenge to statisticians is to develop effective methods of finding the useful information from the vast amounts of messy and noisy data available, most of which are noninformative. This paper presents a general computer intensive approach, based on a method pioneered by Lo and Zheng for detecting which, of many potential explanatory variables, have an influence on a dependent variable Y. This approach is suited to detect influential variables, where causal effects depend on the confluence of values of several variables. It has the advantage of avoiding a difficult direct analysis, involving possibly thousands of variables, by dealing with many randomly selected small subsets from which smaller subsets are selected, guided by a measure of influence I. The main objective is to discover the influential variables, rather than to measure their effects. Once they are detected, the problem of dealing with a much smaller group of influential variables should be vulnerable to appropriate analysis. In a sense, we are confining our attention to locating a few needles in a haystack.Computer science, Statisticsshl5, tz33StatisticsArticlesComment: Quantifying the Fraction of Missing Information for Hypothesis Testing in Statistical and Genetic Studies
https://academiccommons.columbia.edu/catalog/ac:184983
Zheng, Tian; Lo, Shaw-Hwa10.7916/D84T6H8MThu, 29 Jun 2017 03:39:16 +0000The authors suggest an interesting way to measure
the fraction of missing information in the context of
hypothesis testing. The measure seeks to quantify the
impact of missing observations on the test between two
hypotheses. The amount of impact can be useful information
for applied research. An example is, in genetics,
where multiple tests of the same sort are performed
on different variables with different missing rates, and
follow-up studies may be designed to resolve missing
values in selected variables.
In this discussion, we offer our prospective views on
the use of relative information in a follow-up study.
For studies where the impact of missing observations
varies greatly across different variables and where the
investigators have the flexibility of designing studies
that can have different efforts on variables, an optimal
design may be derived using relative information measures
to improve the cost-effectiveness of the followup.Statisticstz33, shl5StatisticsArticlesHow Many People Do You Know in Prison? Using Overdispersion in Count Data to Estimate Social Structure in Networks
https://academiccommons.columbia.edu/catalog/ac:185364
Zheng, Tian; Salganik, Matthew J.; Gelman, Andrew E.10.7916/D800011WThu, 29 Jun 2017 03:38:21 +0000Networks—sets of objects connected by relationships—are important in a number of fields. The study of networks has long been central to sociology, where researchers have attempted to understand the causes and consequences of the structure of relationships in large groups of people. Using insight from previous network research, Killworth et al. and McCarty et al. have developed and evaluated a method for estimating the sizes of hard-to-count populations using network data collected from a simple random sample of Americans. In this article we show how, using a multilevel overdispersed Poisson regression model, these data also can be used to estimate aspects of social structure in the population. Our work goes beyond most previous research on networks by using variation, as well as average responses, as a source of information. We apply our method to the data of McCarty et al. and find that Americans vary greatly in their number of acquaintances. Further, Americans show great variation in propensity to form ties to people in some groups (e.g., males in prison, the homeless, and American Indians), but little variation for other groups (e.g., twins, people named Michael or Nicole). We also explore other features of these data and consider ways in which survey data can be used to estimate network structure.Statistics, Social sciences--Researchtz33, ag389Statistics, Political ScienceArticlesBackward Genotype-Trait Association (BGTA)-Based Dissection of Complex Traits in Case-Control Designs
https://academiccommons.columbia.edu/catalog/ac:185325
Zheng, Tian; Wang, Hui; Lo, Shaw-Hwa10.7916/D8SF2V33Thu, 29 Jun 2017 03:38:20 +0000Background: The studies of complex traits project new challenges to current methods that evaluate association between genotypes and a specific trait. Consideration of possible interactions among loci leads to overwhelming dimensions that cannot be handled using current statistical methods. Methods: In this article, we evaluate a multi-marker screening algorithm--the backward genotype-trait association (BGTA) algorithm for case-control designs, which uses unphased multi-locus genotypes. BGTA carries out a global investigation on a candidate marker set and automatically screens out markers carrying diminutive amounts of information regarding the trait in question. To address the "too many possible genotypes, too few informative chromosomes" dilemma of a genomic-scale study that consists of hundreds to thousands of markers, we further investigate a BGTA-based marker selection procedure, in which the screening algorithm is repeated on a large number of random marker subsets. Results of these screenings are then aggregated into counts that the markers are retained by the BGTA algorithm. Markers with exceptional high counts of returns are selected for further analysis. Results and Conclusion: Evaluated using simulations under several disease models, the proposed methods prove to be more powerful in dealing with epistatic traits.We also demonstrate the proposed methods through an application to a study on the inflammatory bowel disease.Statistics, Genetics, Biometrytz33, hw2334, shl5Statistics, Microbiology and Immunology, BiostatisticsArticlesEstimating Preferences under Risk: The Case of Racetrack Bettors
https://academiccommons.columbia.edu/catalog/ac:184178
Jullien, Bruno; Salanie, Bernard10.7916/D8S75F6JThu, 29 Jun 2017 03:36:35 +0000In this paper we investigate the attitudes toward risk of bettors in British horse races. The model we use allows us to go beyond the expected utility framework and to explore various alternative proposals by estimating a multinomial model on a 34,443‐race data set. We find that rank‐dependent utility models do not fit the data noticeably better than expected utility models. On the other hand, cumulative prospect theory has higher explanatory power. Our preferred estimates suggest a pattern of local risk aversion similar to that proposed by Friedman and Savage.Economics, Statisticsbs2237EconomicsArticlesPreaching to the Unconverted
https://academiccommons.columbia.edu/catalog/ac:179470
Uriarte, Maria; Yackulic, Charles B.10.7916/D8SB44FMThu, 29 Jun 2017 02:57:01 +0000Rapid advances in computing in the past 20 years
have lead to an explosion in the development and
adoption of new statistical modeling tools (Gelman and
Hill 2006, Clark 2007, Bolker 2008, Cressie et al. 2009).
These innovations have occurred in parallel with a
tremendous increase in the availability of ecological
data. The latter has been fueled both by new tools that
have facilitated data collection and management efforts
(e.g., remote sensing, database management software,
and so on) and by increased ease of data sharing
through computers and the World Wide Web. The
impending implementation of the National Ecological
Observatory Network (NEON) will further boost data
availability. These rapid advances in the ability of
ecologists to collect data have not been matched by
application of modern statistical tools. Given the critical
questions ecology is facing (e.g., climate change, species
extinctions, spread of invasives, irreversible losses of
ecosystem services) and the benefits that can be gained
from connecting existing data to models in a sophisticated
inferential framework (Clark et al. 2001, Pielke
and Connant 2003), it is important to understand why
this mismatch exists. Such an understanding would
point to the issues that must be addressed if ecologists
are to make useful inferences from these new data and
tools and contribute in substantial ways to management
and decision making.Ecology, Statisticsmu2126Ecology, Evolution, and Environmental BiologyArticlesNew insights into old methods for identifying causal rare variants
https://academiccommons.columbia.edu/catalog/ac:195277
Hu, Inchi; Zheng, Tian; Huang, Chien-Hsun; Lo, Shaw-Hwa; Wang, Haitian10.7916/D8J38R1MWed, 28 Jun 2017 21:04:19 +0000The advance of high-throughput next-generation sequencing technology makes possible the analysis of rare variants. However, the investigation of rare variants in unrelated-individuals data sets faces the challenge of low power, and most methods circumvent the difficulty by using various collapsing procedures based on genes, pathways, or gene clusters. We suggest a new way to identify causal rare variants using the F-statistic and sliced inverse regression. The procedure is tested on the data set provided by the Genetic Analysis Workshop 17 (GAW17). After preliminary data reduction, we ranked markers according to their F-statistic values. Top-ranked markers were then subjected to sliced inverse regression, and those with higher absolute coefficients in the most significant sliced inverse regression direction were selected. The procedure yields good false discovery rates for the GAW17 data and thus is a promising method for future study on rare variants.Human genetics--Variation, Biometry--Statistical methods, Statistics--Methodology, Biometry, Statisticstz33, ch2526, shl5StatisticsArticlesUsing individual growth model to analyze the change in quality of life from adolescence to adulthood
https://academiccommons.columbia.edu/catalog/ac:192019
Chen, Henian; Cohen, Patricia R.10.7916/D8805135Wed, 28 Jun 2017 21:01:16 +0000Background: The individual growth model is a relatively new statistical technique now widely used to examine the unique trajectories of individuals and groups in repeated measures data. This technique is increasingly used to analyze the changes over time in quality of life (QOL) data. This study examines the change from adolescence to adulthood in physical health as an aspect of QOL as an illustration of the use of this analytic method.
Methods: Employing data from the Children in the Community (CIC) study, a prospective longitudinal investigation, physical health was assessed at mean ages 16, 22, and 33 in 752 persons born between 1965 and 1975.
Results: The analyses using individual growth models show a linear decline in average physical health from age 10 to age 40. Males reported better physical health and declined less per year on average. Time-varying psychiatric disorders accounted for 8.6% of the explained variation in mean physical health, and 6.7% of the explained variation in linear change in physical health. Those with such a disorder reported lower mean physical health and a more rapid decline with age than those without a current psychiatric disorder. The use of SAS PROC MIXED, including syntax and interpretation of output are provided. Applications of these models including statistical assumptions, centering issues and cohort effects are discussed.
Conclusion: This paper highlights the usefulness of the individual growth model in modeling longitudinal change in QOL variables.Human growth--Mathematical models, Quality of life--Statistical methods, Young adults--Health and hygiene, Medical sciences, Aging, Statisticsprc2Psychiatry, EpidemiologyArticlesBAMarray™: Java software for Bayesian analysis of variance for microarray data
https://academiccommons.columbia.edu/catalog/ac:192099
Ishwaran, Hemant; Rao, J. Sunil; Kogalur, Udaya B.10.7916/D8BR8QNZWed, 28 Jun 2017 21:00:37 +0000Background: DNA microarrays open up a new horizon for studying the genetic determinants of disease. The high throughput nature of these arrays creates an enormous wealth of information, but also poses a challenge to data analysis. Inferential problems become even more pronounced as experimental designs used to collect data become more complex. An important example is multigroup data collected over different experimental groups, such as data collected from distinct stages of a disease process. We have developed a method specifically addressing these issues termed Bayesian ANOVA for microarrays (BAM). The BAM approach uses a special inferential regularization known as spike-and-slab shrinkage that provides an optimal balance between total false detections and total false non-detections. This translates into more reproducible differential calls. Spike and slab shrinkage is a form of regularization achieved by using information across all genes and groups simultaneously.
Results: BAMarray™ is a graphically oriented Java-based software package that implements the BAM method for detecting differentially expressing genes in multigroup microarray experiments (up to 256 experimental groups can be analyzed). Drop-down menus allow the user to easily select between different models and to choose various run options. BAMarray™ can also be operated in a fully automated mode with preselected run options. Tuning parameters have been preset at theoretically optimal values freeing the user from such specifications. BAMarray™ provides estimates for gene differential effects and automatically estimates data adaptive, optimal cutoff values for classifying genes into biological patterns of differential activity across experimental groups. A graphical suite is a core feature of the product and includes diagnostic plots for assessing model assumptions and interactive plots that enable tracking of prespecified gene lists to study such things as biological pathway perturbations. The user can zoom in and lasso genes of interest that can then be saved for downstream analyses.
Conclusion: BAMarray™ is user friendly platform independent software that effectively and efficiently implements the BAM methodology. Classifying patterns of differential activity is greatly facilitated by a data adaptive cutoff rule and a graphical suite. BAMarray™ is licensed software freely available to academic institutions. More information can be found at http://www.bamarray.com.DNA microarrays--Data processing, Java (Computer program language), Bioinformatics, Bayesian statistical decision theory, Statistics, Information technologyubk2101StatisticsArticlesPAGE: Parametric Analysis of Gene Set Enrichment
https://academiccommons.columbia.edu/catalog/ac:194039
Kim, Seon-Young; Volsky, David Julian10.7916/D84X568JWed, 28 Jun 2017 20:59:43 +0000Background: Gene set enrichment analysis (GSEA) is a microarray data analysis method that uses predefined gene sets and ranks of genes to identify significant biological changes in microarray data sets. GSEA is especially useful when gene expression changes in a given microarray data set is minimal or moderate.
Results: We developed a modified gene set enrichment analysis method based on a parametric statistical analysis model. Compared with GSEA, the parametric analysis of gene set enrichment (PAGE) detected a larger number of significantly altered gene sets and their p-values were lower than the corresponding p-values calculated by GSEA. Because PAGE uses normal distribution for statistical inference, it requires less computation than GSEA, which needs repeated computation of the permutated data set. PAGE was able to detect significantly changed gene sets from microarray data irrespective of different Affymetrix probe level analysis methods or different microarray platforms. Comparison of two aged muscle microarray data sets at gene set level using PAGE revealed common biological themes better than comparison at individual gene level.
Conclusion: PAGE was statistically more sensitive and required much less computational effort than GSEA, it could identify significantly changed biological themes from microarray data irrespective of analysis methods or microarray platforms, and it was useful in comparison of multiple microarray data sets. We offer PAGE as a useful microarray analysis method.Bioinformatics--Methodology, Statistics, DNA microarrays--Data processing, Bioinformatics, Genetics, Biometrydjv4Pathology and Cell BiologyArticlesMedication-Wide Association Studies
https://academiccommons.columbia.edu/catalog/ac:173912
Ryan, P. B.; Madigan, David B.; Stang, P. E.; Schuemie, M. J.; Hripcsak, George M.10.7916/D8PG1PVXWed, 28 Jun 2017 20:28:18 +0000Undiscovered side effects of drugs can have a profound effect on the health of the nation, and electronic health-care databases offer opportunities to speed up the discovery of these side effects. We applied a “medication-wide association study” approach that combined multivariate analysis with exploratory visualization to study four health outcomes of interest in an administrative claims database of 46 million patients and a clinical database of 11 million patients. The technique had good predictive value, but there was no threshold high enough to eliminate false-positive findings. The visualization not only highlighted the class effects that strengthened the review of specific products but also underscored the challenges in confounding. These findings suggest that observational databases are useful for identifying potential associations that warrant further consideration but are unlikely to provide definitive evidence of causal effects.Pharmacology, Statistics, Bioinformaticsdm2418, gh13Statistics, Biomedical InformaticsArticlesLearning Theory Analysis for Association Rules and Sequential Event Prediction
https://academiccommons.columbia.edu/catalog/ac:173905
Rudin, Cynthia; Letham, Benjamin; Madigan, David B.10.7916/D82N50C1Wed, 28 Jun 2017 20:28:13 +0000We present a theoretical analysis for prediction algorithms based on association rules. As part of this analysis, we introduce a problem for which rules are particularly natural, called “sequential event prediction." In sequential event prediction, events in a sequence are revealed one by one, and the goal is to determine which event will next be revealed. The training set is a collection of past sequences of events. An example application is to predict which item will next be placed into a customer's online shopping cart, given his/her past purchases. In the context of this problem, algorithms based on association rules have distinct advantages over classical statistical and machine learning methods: they look at correlations based on subsets of co-occurring past events (items a and b imply item c), they can be applied to the sequential event prediction problem in a natural way, they can potentially handle the “cold start" problem where the training set is small, and they yield interpretable predictions. In this work, we present two algorithms that incorporate association rules. These algorithms can be used both for sequential event prediction and for supervised classification, and they are simple enough that they can possibly be understood by users, customers, patients, managers, etc. We provide generalization guarantees on these algorithms based on algorithmic stability analysis from statistical learning theory. We include a discussion of the strict minimum support threshold often used in association rule mining, and introduce an “adjusted confidence" measure that provides a weaker minimum support condition that has advantages over the strict minimum support. The paper brings together ideas from statistical learning theory, association rule mining and Bayesian analysis.Statistics, Artificial intelligencedm2418StatisticsArticlesGenerating Productive Dialogue between Consulting Statisticians and their Clients in the Pharmaceutical and Medical Research Settings
https://academiccommons.columbia.edu/catalog/ac:173832
Emir, Birol; Amaratunga, Dhammika; Beltangady, Mohan; Cabrera, Javier; Freeman, Roy; Madigan, David B.; Nguyen, Ha H.; Whalen, Edward Patrick10.7916/D8PK0D8NWed, 28 Jun 2017 20:27:47 +0000Due to the ever-increasing complexity of scientific technologies and resulting data, consulting statisticians are becoming more involved in the design, conduct, and analysis of biomedical research. This requires extensive collaboration between the consulting statistician and nonstatisticians, such as researchers, clinicians, and corporate executives. Consequently, a successful consulting career is becoming ever more dependent on the statistician's ability to effectively communicate with nonstatisticians. This is especially true when more complex, nontraditional analytical methods are required. In this paper, we examine the collaboration between statisticians and nonstatisticians from three different professional perspectives. Integrating these perspectives, we discuss ways to help the consulting statistician generate productive dialogue with clients. Finally, we examine how universities can better prepare students for careers in statistical consulting by incorporating more communication-based elements into their curriculum and by offering students ample opportunities to collaborate with nonstatisticians. Overall, we designed this exercise to help the consulting statistician generate dialogue with clients that results in more productive collaborations and a more satisfying work experience.Statistics, Bioinformatics, Medicinebe2166, dm2418, hhn2108, ew2320StatisticsArticlesBayesian Hierarchical Rule Modeling for Predicting Medical Conditions
https://academiccommons.columbia.edu/catalog/ac:173882
McCormick, Tyler H.; Rudin, Cynthia; Madigan, David B.10.7916/D8V69GP1Wed, 28 Jun 2017 20:26:51 +0000We propose a statistical modeling technique, called the Hierarchical Association Rule Model (HARM), that predicts a patient’s possible future medical conditions given the patient’s current and past history of reported conditions. The core of our technique is a Bayesian hierarchical model for selecting predictive association rules (such as “condition 1 and condition 2 → condition 3”) from a large set of candidate rules. Because this method “borrows strength” using the conditions of many similar patients, it is able to provide predictions specialized to any given patient, even when little information about the patient’s history of conditions is available.Mathematics, Statistics, Medicinethm2105, dm2418StatisticsArticlesA Hierarchical Model for Association Rule Mining of Sequential Events: An Approach to Automated Medical Symptom Prediction
https://academiccommons.columbia.edu/catalog/ac:173838
McCormick, Tyler H.; Rudin, Cynthia; Madigan, David B.10.7916/D89C6VJDWed, 28 Jun 2017 20:25:55 +0000In many healthcare settings, patients visit healthcare professionals periodically and report multiple medical conditions, or symptoms, at each encounter. We propose a statistical modeling technique, called the Hierarchical Association Rule Model (HARM), that predicts a patient’s possible future symptoms given the patient’s current and past history of reported symptoms. The core of our technique is a Bayesian hierarchical model for selecting predictive association rules (such as “symptom 1 and symptom 2 → symptom 3 ”) from a large set of candidate rules. Because this method “borrows strength” using the symptoms of many similar patients, it is able to provide predictions specialized to any given patient, even when little information about the patient’s history of symptoms is available.Mathematics, Statistics, Medicinethm2105, dm2418StatisticsArticlesAlgorithms for Sparse Linear Classifiers in the Massive Data Setting
https://academiccommons.columbia.edu/catalog/ac:173908
Balakrishnan, Suhrid; Madigan, David B.; Bartlett, Peter10.7916/D8Z0368XWed, 28 Jun 2017 20:24:23 +0000Classifiers favoring sparse solutions, such as support vector machines, relevance vector machines, LASSO-regression based classifiers, etc., provide competitive methods for classification problems in high dimensions. However, current algorithms for training sparse classifiers typically scale quite unfavorably with respect to the number of training examples. This paper proposes online and multi-pass algorithms for training sparse linear classifiers for high dimensional data. These algorithms have computational complexity and memory requirements that make learning on massive data sets feasible. The central idea that makes this possible is a straightforward quadratic approximation to the likelihood function.Statistics, Artificial intelligencedm2418StatisticsArticlesLocation Estimation in Wireless Networks: A Bayesian Approach
https://academiccommons.columbia.edu/catalog/ac:173820
Madigan, David B.; Ju, Wen-Hua; Krishnan, P.; Krishnakumar, A. S.; Zorych, Ivan10.7916/D82V2D74Wed, 28 Jun 2017 20:23:52 +0000We present a Bayesian hierarchical model for indoor location estimation in wireless networks. We demonstrate that out model achieves accuracy that is similar to other published models and algorithms. By harnessing prior knowledge, our model drastically reduces the requirement for training data as compared with existing approaches.Mathematics, Statisticsdm2418StatisticsArticlesA One-Pass Sequential Monte Carlo Method for Bayesian Analysis of Massive Datasets
https://academiccommons.columbia.edu/catalog/ac:173899
Balakrishnan, Suhrid; Madigan, David B.10.7916/D8B56GTPWed, 28 Jun 2017 20:23:50 +0000For Bayesian analysis of massive data, Markov chain Monte Carlo (MCMC) techniques often prove infeasible due to computational resource constraints. Standard MCMC methods generally require a complete scan of the dataset for each iteration. Ridgeway and Madigan (2002) and Chopin (2002b) recently presented importance sampling algorithms that combined simulations from a posterior distribution conditioned on a small portion of the dataset with a reweighting of those simulations to condition on the remainder of the dataset. While these algorithms drastically reduce the number of data accesses as compared to traditional MCMC, they still require substantially more than a single pass over the dataset. In this paper, we present "1PFS," an efficient, one-pass algorithm. The algorithm employs a simple modification of the Ridgeway and Madigan (2002) particle filtering algorithm that replaces the MCMC based "rejuvenation" step with a more efficient "shrinkage" kernel smoothing based step. To show proof-of-concept and to enable a direct comparison, we demonstrate 1PFS on the same examples presented in Ridgeway and Madigan (2002), namely a mixture model for Markov chains and Bayesian logistic regression. Our results indicate the proposed scheme delivers accurate parameter estimates while employing only a single pass through the data.Mathematics, Statisticsdm2418StatisticsArticlesAnalysis of Variance of Cross-Validation Estimators of the Generalization Error
https://academiccommons.columbia.edu/catalog/ac:173902
Markatou, Marianthi; Tian, Hong; Biswas, Shameek; Hripcsak, George M.10.7916/D86D5R2XWed, 28 Jun 2017 20:23:49 +0000This paper brings together methods from two different disciplines: statistics and machine learning. We address the problem of estimating the variance of cross-validation (CV) estimators of the generalization error. In particular, we approach the problem of variance estimation of the CV estimators of generalization error as a problem in approximating the moments of a statistic. The approximation illustrates the role of training and test sets in the performance of the algorithm. It provides a unifying approach to evaluation of various methods used in obtaining training and test sets and it takes into account the variability due to different training and test sets. For the simple problem of predicting the sample mean and in the case of smooth loss functions, we show that the variance of the CV estimator of the generalization error is a function of the moments of the random variables Y=Card(Sj ∩ Sj') and Y*=Card(Sjc ∩ Sj'c), where Sj, Sj' are two training sets, and Sjc, Sj'c are the corresponding test sets. We prove that the distribution of Y and Y* is hypergeometric and we compare our estimator with the one proposed by Nadeau and Bengio (2003). We extend these results in the regression case and the case of absolute error loss, and indicate how the methods can be extended to the classification case. We illustrate the results through simulation.Statistics, Artificial intelligencemm168, ht2031, spb2003, gh13Biostatistics, Biomedical Informatics, StatisticsArticles[Least Angle Regression]: Discussion
https://academiccommons.columbia.edu/catalog/ac:173841
Madigan, David B.; Ridgeway, Greg10.7916/D81V5C29Wed, 28 Jun 2017 20:23:33 +0000Algorithms for simultaneous shrinkage and selection in regression and classification provide attractive solutions to knotty old statistical challenges. Nevertheless, as far as we can tell, Tibshirani's Lasso algorithm has had little impact on statistical practice. Two particular reasons for this may be the relative inefficiency of the original Lasso algorithm and the relative complexity of more recent Lasso algorithms [e.g., Osborne, Presnell and Turlach (2000)]. Efron, Hastie, Johnstone and Tibshirani have provided an efficient, simple algorithm for the Lasso as well as algorithms for stagewise regression and the new least angle regression. As such this paper is an important contribution to statistical computing.Mathematics, Statisticsdm2418StatisticsArticlesCorrection: Separation and completeness properties for AMP chain graph Markov models
https://academiccommons.columbia.edu/catalog/ac:173887
Madigan, David B.; Levitz, Michael; Perlman, Michael D.10.7916/D8QF8R05Wed, 28 Jun 2017 20:23:16 +0000Correction of table 2 on page 1757 of 'Separation and completeness properties for AMP chain graph Markov models', Annals of Statistics, volume 29 (2001).Mathematics, Statisticsdm2418StatisticsArticlesBook Reviews: Principles of Data Mining. By David Hand, Heikki Mannila, and Padhraic Smyth.
https://academiccommons.columbia.edu/catalog/ac:173915
Madigan, David B.10.7916/D8DZ06D8Wed, 28 Jun 2017 20:23:08 +0000"Principles of Data Mining. By David Hand, Heikki Mannila, and Padhraic Smyth. MIT Press, Cambridge, MA, 2001. $50.00. xxxii+546 pp., hardcover. ISBN 0-262-08290-X. Is data mining the same as statistics? The distinguished authors of Principles of Data Mining struggle to make a distinction between the two subjects. In the end, what they have written is a fine applied statistics text." -- page 501Statisticsdm2418StatisticsReviewsBayesian Model Averaging: a Tutorial (with Comments by M. Clyde, David Draper and E. I. George, and a Rejoinder by the Authors)
https://academiccommons.columbia.edu/catalog/ac:173853
Hoeting, Jennifer A.; Madigan, David B.; Raftery, Adrian E.; Volinsky, Chris T.; Clyde, M.; Draper, David; George, E. I.10.7916/D84M92N7Wed, 28 Jun 2017 20:23:00 +0000Standard statistical practice ignores model uncertainty. Data analysts typically select a model from some class of models and then proceed as if the selected model had generated the data. This approach ignores the uncertainty in model selection, leading to over-confident inferences and decisions that are more risky than one thinks they are. Bayesian model averaging (BMA)provides a coherent mechanism for accounting for this model uncertainty. Several methods for implementing BMA have recently emerged. We discuss these methods and present a number of examples.In these examples, BMA provides improved out-of-sample predictive performance. We also provide a catalogue of currently available BMA software.Statisticsdm2418StatisticsArticlesSeparation and Completeness Properties for Amp Chain Graph Markov Models
https://academiccommons.columbia.edu/catalog/ac:173847
Levitz, Michael; Perlman, Michael D.; Madigan, David B.10.7916/D8X34VJGWed, 28 Jun 2017 20:23:00 +0000Pearl’s well-known d-separation criterion for an acyclic directed graph (ADG) is a pathwise separation criterion that can be used to efficiently identify all valid conditional independence relations in the Markov model determined by the graph. This paper introduces p-separation, a pathwise separation criterion that efficiently identifies all valid conditional independences under the Andersson–Madigan–Perlman (AMP) alternative Markov property for chain graphs (= adicyclic graphs), which include both ADGs and undirected graphs as special cases. The equivalence of p-separation to the augmentation criterion occurring in the AMP global Markov property is established, and p-separation is applied to prove completeness of the global Markov property for AMP chain graph models. Strong completeness of the AMP Markov property is established, that is, the existence of Markov perfect distributions that satisfy those and only those conditional independences implied by the AMP property(equivalently, by p-separation). A linear-time algorithm for determining p-separation is presented.Mathematics, Statisticsdm2418StatisticsArticlesA Characterization of Markov Equivalence Classes for Acyclic Digraphs
https://academiccommons.columbia.edu/catalog/ac:173896
Andersson, Steen A.; Madigan, David B.; Perlman, Michael D.10.7916/D8FX77J3Wed, 28 Jun 2017 20:22:39 +0000Undirected graphs and acyclic digraphs (ADG's), as well as their mutual extension to chain graphs, are widely used to describe dependencies among variables in multiviarate distributions. In particular, the likelihood functions of ADG models admit convenient recursive factorizations that often allow explicit maximum likelihood estimates and that are well suited to building Bayesian networks for expert systems. Whereas the undirected graph associated with a dependence model is uniquely determined, there may be many ADG's that determine the same dependence (i.e., Markov) model. Thus, the family of all ADG's with a given set of vertices is naturally partitioned into Markov-equivalence classes, each class being associated with a unique statistical model. Statistical procedures, such as model selection of model averaging, that fail to take into account these equivalence classes may incur substantial computational or other inefficiences. Here it is show that each Markov-equivalence class is uniquely determined by a single chain graph, the essential graph, that is itself simultaneously Markov equivalent to all ADG's in the equivalence class. Essential graphs are characterized, a polynomial-time algorithm for their construction is given, and their applications to model selection and other statistical questions are described.Mathematics, Statisticsdm2418StatisticsArticlesA Note on Equivalence Classes of Directed Acyclic Independence Graphs
https://academiccommons.columbia.edu/catalog/ac:173826
Madigan, David B.10.7916/D8TB150CWed, 28 Jun 2017 20:22:29 +0000Directed acyclic independence graphs (DAIGs) play an important role in recent developments in probabilistic expert systems and influence diagrams (Chyu [1]). The purpose of this note is to show that DAIGs can usefully be grouped into equivalence classes where the members of a single class share identical Markov properties. These equivalence classes can be identified via a simple graphical criterion. This result is particularly relevant to model selection procedures for DAIGs (see, e.g., Cooper and Herskovits [2] and Madigan and Raftery [4]) because it reduces the problem of searching among possible orientations of a given graph to that of searching among the equivalence classes.Mathematics, Statisticsdm2418StatisticsArticles[Bayesian Analysis in Expert Systems]: Comment: What's Next?
https://academiccommons.columbia.edu/catalog/ac:173856
Madigan, David B.10.7916/D8W37TFJWed, 28 Jun 2017 20:22:27 +0000"These papers represent two of the many different graphical modeling camps that have emerged from a flurry of activity in the past decade. The paper by Cox and Wermuth falls within the statistical graphical modeling camp and provides a useful generalization of that body of work. There is, of course, a price to be paid for this generality, namely that the interpretation of the graphs is more complex...The paper by Spiegelhalter, Dawid, Lauritzen and Cowell falls within the probabilistic expert system camp. This is a tour de force by researchers responsible for much of the astonishing progress in this area. Ten years ago, probabilistic models were shunned by the artificial intelligence community. That they are now widely accepted and used is due in large measure to the insights and efforts of these authors, along with other pioneers such as Judea Pearl and Peter Cheeseman..." -- page 261Mathematics, Statisticsdm2418StatisticsArticlesReporting of analyses from randomized controlled trials with multiple arms: a systematic review
https://academiccommons.columbia.edu/catalog/ac:180137
Ravaud, Philippe; Perrodeau, Elodie; Boutron, Isabelle; Baron, Gabriel10.7916/D837772TWed, 28 Jun 2017 20:20:55 +0000Background: Multiple-arm randomized trials can be more complex in their design, data analysis, and result reporting than two-arm trials. We conducted a systematic review to assess the reporting of analyses in reports of randomized controlled trials (RCTs) with multiple arms. Methods: The literature in the MEDLINE database was searched for reports of RCTs with multiple arms published in 2009 in the core clinical journals. Two reviewers extracted data using a standardized extraction form. Results: In total, 298 reports were identified. Descriptions of the baseline characteristics and outcomes per group were missing in 45 reports (15.1%) and 48 reports (16.1%), respectively. More than half of the articles (n = 171, 57.4%) reported that a planned global test comparison was used (that is, assessment of the global differences between all groups), but 67 (39.2%) of these 171 articles did not report details of the planned analysis. Of the 116 articles reporting a global comparison test, 12 (10.3%) did not report the analysis as planned. In all, 60% of publications (n = 180) described planned pairwise test comparisons (that is, assessment of the difference between two groups), but 20 of these 180 articles (11.1%) did not report the pairwise test comparisons. Of the 204 articles reporting pairwise test comparisons, the comparisons were not planned for 44 (21.6%) of them. Less than half the reports (n = 137; 46%) provided baseline and outcome data per arm and reported the analysis as planned. Conclusions: Our findings highlight discrepancies between the planning and reporting of analyses in reports of multiple-arm trials.Statistics, Medical sciencespr2341EpidemiologyArticlesCopy number variation genotyping using family information
https://academiccommons.columbia.edu/catalog/ac:180080
Darvishi, Katayoon; Mills, Ryan E.; Lee, Charles; Raby, Benjamin A.; Chu, Jen-hwa; Rogers, Angela; Ionita-Laza, Iuliana10.7916/D8HD7T0DWed, 28 Jun 2017 20:20:16 +0000Background: In recent years there has been a growing interest in the role of copy number variations (CNV) in genetic diseases. Though there has been rapid development of technologies and statistical methods devoted to detection in CNVs from array data, the inherent challenges in data quality associated with most hybridization techniques remains a challenging problem in CNV association studies. Results: To help address these data quality issues in the context of family-based association studies, we introduce a statistical framework for the intensity-based array data that takes into account the family information for copy-number assignment. The method is an adaptation of traditional methods for modeling SNP genotype data that assume Gaussian mixture model, whereby CNV calling is performed for all family members simultaneously and leveraging within family-data to reduce CNV calls that are incompatible with Mendelian inheritance while still allowing de-novo CNVs. Applying this method to simulation studies and a genome-wide association study in asthma, we find that our approach significantly improves CNV calls accuracy, and reduces the Mendelian inconsistency rates and false positive genotype calls. The results were validated using qPCR experiments. Conclusions: In conclusion, we have demonstrated that the use of family information can improve the quality of CNV calling and hopefully give more powerful association test of CNVs.Genetics, Statisticsii2135Mailman School of Public HealthArticlesHelping the decision maker effectively promote various experts’ views into various optimal solutions to China’s institutional problem of health care provider selection.
https://academiccommons.columbia.edu/catalog/ac:180132
Tang, Liyang10.7916/D8BP0147Wed, 28 Jun 2017 20:20:05 +0000Background: The main aim of China’s Health Care System Reform was to help the decision maker find the optimal solution to China’s institutional problem of health care provider selection. A pilot health care provider research system was recently organized in China’s health care system, and it could efficiently collect the data for determining the optimal solution to China’s institutional problem of health care provider selection from various experts, then the purpose of this study was to apply the optimal implementation methodology to help the decision maker effectively promote various experts’ views into various optimal solutions to this problem under the support of this pilot system. Methods: After the general framework of China’s institutional problem of health care provider selection was established, this study collaborated with the National Bureau of Statistics of China to commission a large-scale 2009 to 2010 national expert survey (n = 3,914) through the organization of a pilot health care provider research system for the first time in China, and the analytic network process (ANP) implementation methodology was adopted to analyze the dataset from this survey. Results: The market-oriented health care provider approach was the optimal solution to China’s institutional problem of health care provider selection from the doctors’ point of view; the traditional government’s regulation-oriented health care provider approach was the optimal solution to China’s institutional problem of health care provider selection from the pharmacists’ point of view, the hospital administrators’ point of view, and the point of view of health officials in health administration departments; the public private partnership (PPP) approach was the optimal solution to China’s institutional problem of health care provider selection from the nurses’ point of view, the point of view of officials in medical insurance agencies, and the health care researchers’ point of view. Conclusions: The data collected through a pilot health care provider research system in the 2009 to 2010 national expert survey could help the decision maker effectively promote various experts’ views into various optimal solutions to China’s institutional problem of health care provider selection.Business, StatisticsBusinessArticlesExpressionPlot: a web-based framework for analysis of RNA-Seq and microarray gene expression data
https://academiccommons.columbia.edu/catalog/ac:183139
Friedman, Brad; Maniatis, Tom10.7916/D82J6979Wed, 28 Jun 2017 20:18:15 +0000RNA-Seq and microarray platforms have emerged as important tools for detecting changes in gene expression and RNA processing in biological samples. We present ExpressionPlot, a software package consisting of a default back end, which prepares raw sequencing or Affymetrix microarray data, and a web-based front end, which offers a biologically centered interface to browse, visualize, and compare different data sets. Download and installation instructions, a user's manual, discussion group, and a prototype are available at http://expressionplot.comStatistics, Bioinformaticstm2472Biochemistry and Molecular BiophysicsArticlesProspect Theory as Efficient Perceptual Distortion
https://academiccommons.columbia.edu/catalog/ac:167407
Woodford, Michael10.7916/D8T43R03Wed, 28 Jun 2017 17:07:08 +0000The paper proposes a theory of efficient perceptual distortions, in which the statistical relation between subjective perceptions and the objective state minimizes the error of the state estimate, subject to a constraint on information processing capacity. The theory is shown to account for observed limits to the accuracy of visual perception, and then postulated to apply to perception of options in economic choice situations as well. When applied to choice between lotteries, it implies reference-dependent valuations, and predicts both risk-aversion with respect to gains and risk-seeking with respect to losses, as in the prospect theory of Kahneman and Tversky (1979).Statistics, Economics, Sociologymw2230EconomicsArticlesLearning to Believe in Sunspots
https://academiccommons.columbia.edu/catalog/ac:167710
Woodford, Michael10.7916/D85X26VBWed, 28 Jun 2017 16:57:54 +0000An adaptive learning rule is exhibited for the Azariadis (1981) overlapping generations model of a monetary economy with multiple equilibria, under which the economy may converge to a stationary sunspot equilibrium, even if agents do not initially believe that outcomes are significantly different in different "sunspot" states. The type of learning rule studied is of the "stochastic approximation" form studied by Robbins and Monro (1951);
methods for analyzing the convergence of this form of algorithm are presented that may be of use in many other contexts as well. Conditions are given under which convergence to a sunspot equilibrium occurs with probability one.Economics, Statisticsmw2230EconomicsArticlesThe Representation of Social Processes by Markov Models
https://academiccommons.columbia.edu/catalog/ac:165054
Singer, Burton; Spilerman, Seymour10.7916/D80G3W8BWed, 28 Jun 2017 16:56:20 +0000In this paper we consider a class of issues which are central to modeling social phenomena by continuous-time Markov structures. In particular, we discuss (a) embeddability, or how to determine whether observations on an empirical process could have arisen via the evolution of a continuous-time Markov structure; and (b) identification, or what to do if the observations are consistent with more than one continuous-time Markov structure. With respect to the latter topic, we discuss how to select the specific structure from the list of alternatives which should be associated with the empirical process. We point out that the issues of embeddability and identification are especially pertinent to modeling empirical processes when one has available only fragmentary data and when the observations contain "noise" or other sources of error. These characteristics, of course, describe the typical work situation of sociologists. Finally, we note the type of situation in which a continuous-time model is the proper structure to employ and indicate that issues analogous to the ones we describe here apply to modeling social processes with discrete-time structures.Sociology, Statisticsss50SociologyArticlesOn the relationship between total ozone and atmospheric dynamics and chemistry at mid-latitudes – Part 1: Statistical models and spatial fingerprints of atmospheric dynamics and chemistry
https://academiccommons.columbia.edu/catalog/ac:161210
Frossard, L.; Rieder, Harald; Ribatet, M.; Staehelin, J.; Maeder, J. A.; Di Rocco, S.; Davison, A. C.; Peter, T.10.7916/D86M3HVNTue, 27 Jun 2017 20:08:50 +0000We use statistical models for mean and extreme values of total column ozone to analyze "fingerprints" of atmospheric dynamics and chemistry on long-term ozone changes at northern and southern mid-latitudes on grid cell basis. At each grid cell, the r-largest order statistics method is used for the analysis of extreme events in low and high total ozone (termed ELOs and EHOs, respectively), and an autoregressive moving average (ARMA) model is used for the corresponding mean value analysis. In order to describe the dynamical and chemical state of the atmosphere, the statistical models include important atmospheric covariates: the solar cycle, the Quasi-Biennial Oscillation (QBO), ozone depleting substances (ODS) in terms of equivalent effective stratospheric chlorine (EESC), the North Atlantic Oscillation (NAO), the Antarctic Oscillation (AAO), the El Niño/Southern Oscillation (ENSO), and aerosol load after the volcanic eruptions of El Chichón and Mt. Pinatubo. The influence of the individual covariates on mean and extreme levels in total column ozone is derived on a grid cell basis. The results show that "fingerprints", i.e., significant influence, of dynamical and chemical features are captured in both the "bulk" and the tails of the statistical distribution of ozone, respectively described by mean values and EHOs/ELOs. While results for the solar cycle, QBO, and EESC are in good agreement with findings of earlier studies, unprecedented spatial fingerprints are retrieved for the dynamical covariates. Column ozone is enhanced over Labrador/Greenland, the North Atlantic sector and over the Norwegian Sea, but is reduced over Europe, Russia and the Eastern United States during the positive NAO phase, and vice-versa during the negative phase. The NAO's southern counterpart, the AAO, strongly influences column ozone at lower southern mid-latitudes, including the southern parts of South America and the Antarctic Peninsula, and the central southern mid-latitudes. Results for both NAO and AAO confirm the importance of atmospheric dynamics for ozone variability and changes from local/regional to global scales.Statistics, Atmospheric chemistry, Atmospherehr2302Lamont-Doherty Earth ObservatoryArticlesAnalyzing Postdisaster Surveillance Data: The Effect of the Statistical Method
https://academiccommons.columbia.edu/catalog/ac:157474
DiMaggio, Charles J.; Galea, Sandro; Abramson, David M.10.7916/D8G4513DTue, 27 Jun 2017 17:52:15 +0000Data from existing administrative databases and ongoing surveys or surveillance methods may prove indispensable after mass traumas as a way of providing information that may be useful to emergency planners and practitioners. The analytic approach, however, may affect exposure prevalence estimates and measures of association. We compare Bayesian hierarchical modeling methods to standard survey analytic techniques for survey data collected in the aftermath of a terrorist attack. Estimates for the prevalence of exposure to the terrorist attacks of September 11, 2001, varied by the method chosen. Bayesian hierarchical modeling returned the lowest estimate for exposure prevalence with a credible interval spanning nearly 3 times the range of the confidence intervals (CIs) associated with both unadjusted and survey procedures. Bayesian hierarchical modeling also returned a smaller point estimate for measures of association, although in this instance the credible interval was tighter than that obtained through survey procedures. Bayesian approaches allow a consideration of preexisting assumptions about survey data, and may offer potential advantages, particularly in the uncertain environment of postterrorism and disaster settings. Additional comparative analyses of existing data are necessary to guide our ability to use these techniques in future incidents.Public health, Statisticscjd11, sg822, dma3National Center for Disaster PreparednessArticlesR2WinBUGS: A Package for Running WinBUGS from R
https://academiccommons.columbia.edu/catalog/ac:154734
Sturtz, Sibylle; Ligges, Uwe; Gelman, Andrew E.10.7916/D80C55HHTue, 27 Jun 2017 15:43:29 +0000The R2WinBUGS package provides convenient functions to call WinBUGS from R. It automatically writes the data and scripts in a format readable by WinBUGS for processing in batch mode, which is possible since version 1.4. After the WinBUGS process has finished, it is possible either to read the resulting data into R by the package itself—which gives a compact graphical summary of inference and convergence diagnostics—or to use the facilities of the coda package for further analyses of the output. Examples are given to demonstrate the usage of this package.Statisticsag389StatisticsArticlesMultiple Imputation with Diagnostics (mi) in R: Opening Windows into the Black Box
https://academiccommons.columbia.edu/catalog/ac:154731
Su, Yu-Sung; Gelman, Andrew E.; Hill, Jennifer; Yajima, Masanao10.7916/D8VQ3CD3Tue, 27 Jun 2017 15:43:28 +0000Our mi package in R has several features that allow the user to get inside the imputation process and evaluate the reasonableness of the resulting models and imputations. These features include: choice of predictors, models, and transformations for chained imputation models; standard and binned residual plots for checking the fit of the conditional distributions used for imputation; and plots for comparing the distributions of observed and imputed data. In addition, we use Bayesian models and weakly informative prior distributions to construct more stable estimates of imputation models. Our goal is to have a demonstration package that (a) avoids many of the practical problems that arise with existing multivariate imputation programs, and (b) demonstrates state-of-the-art diagnostics that can be applied more generally and can be incorporated into the software of others.Statisticsag389StatisticsArticlesBayesian Statistical Pragmatism
https://academiccommons.columbia.edu/catalog/ac:154737
Gelman, Andrew E.10.7916/D8MC98QJTue, 27 Jun 2017 15:39:20 +0000I agree with Rob Kass’ point that we can and should make use of statistical methods developed under different philosophies, and I am happy to take the opportunity to elaborate on some of his arguments.Statisticsag389StatisticsArticlesSegregation in Social Networks Based on Acquaintanceship and Trust
https://academiccommons.columbia.edu/catalog/ac:154740
DiPrete, Thomas A.; Gelman, Andrew E.; McCormick, Tyler; Teitler, Julien O.; Zheng, Tian10.7916/D8F198DHTue, 27 Jun 2017 15:38:27 +0000Using 2006 General Social Survey data, the authors compare levels of segregation by race and along other dimensions of potential social cleavage in the contemporary United States. Americans are not as isolated as the most extreme recent estimates suggest. However, hopes that “bridging” social capital is more common in broader acquaintanceship networks than in core networks are not supported. Instead, the entire acquaintanceship network is perceived by Americans to be about as segregated as the much smaller network of close ties. People do not always know the religiosity, political ideology, family behaviors, or socioeconomic status of their acquaintances, but perceived social divisions on these dimensions are high, sometimes rivaling racial segregation in acquaintanceship networks. The major challenge to social integration today comes from the tendency of many Americans to isolate themselves from others who differ on race, political ideology, level of religiosity, and other salient aspects of social identity.Statisticstad61, ag389, thm2105, jot8, tz33StatisticsArticlesEditorial: Special Section on Statistical and Perceptual Audio Processing
https://academiccommons.columbia.edu/catalog/ac:144493
Ellis, Daniel P. W.; Raj, Bhiksha; Brown, Judith C.; Slaney, Malcolm; Smaragdis, Paris10.7916/D83T9SVCTue, 27 Jun 2017 14:12:37 +0000Human perception has always been an inspiration for automatic processing systems, not least because tasks such as speech recognition only exist because people do them—and, indeed, without that example we might wonder if they were possible at all. As computational power grows, we have increasing opportunities to model and duplicate perceptual abilities with greater fidelity, and, most importantly, based on larger and larger amounts of raw data describing both what signals exist in the real world, and how people respond to them. The power to deal with large data sets has meant that approaches that were once mere theoretical possibilities, such as exhaustive search of exponentially-sized codebooks, or real-time direct convolution of long sequences, have become increasingly practical and even unremarkable. A major consequence of this is the growth of statistical or corpus-based approaches, where complex relations, discriminations, or structures are inferred directly from example data (for instance by optimizing the parameters of a very general algorithm). An increasing number of complex tasks can be given empirically optimal solutions based on large, representative datasets. The traditional idea of perceptually-inspired processing is to develop a machine algorithm for a complex task such as melody recognition or source separation through inspiration and introspection about how individuals perform the task, and on the basis of direct psychological or neurophysiological data. The results can appear to be at odds with the statistical perspective, since perceptually-motivated work is often ad-hoc, comprising many stages whose individual contributions are difficult to separate. We believe that it is important to unify these two approaches: to employ rigorous, exhaustive techniques taking advantage of the statistics of large data sets to develop and solve perceptually-based and subjectively-defined problems. With this in mind, we organized a one-day workshop on Statistical and Perceptual Audio Processing as a satellite to the International Conference on Spoken Language Processing (ICSLP-INTERSPEECH), held in Jeju, Korea, in September 2004.Statistics, Psychophysiologyde171Electrical EngineeringArticlesMultiscale Representations for Manifold-Valued Data
https://academiccommons.columbia.edu/catalog/ac:140178
Rahman, Inam Ur; Drori, Iddo; Stodden, Victoria C.; Donoho, David L.; Schroeder, Peter10.7916/D87371F4Mon, 26 Jun 2017 21:43:59 +0000We describe multiscale representations for data observed on equispaced grids and taking values in manifolds such as: the sphere S2, the special orthogonal group SO(3), the positive definite matrices SPD(n), and the Grassmann manifolds G(n, k). The representations are based on the deployment of Deslauriers-Dubuc and Average Interpolating pyramids "in the tangent plane" of such manifolds, using the Exp and Log maps of those manifolds. The representations provide "wavelet coefficients" which can be thresholded, quantized, and scaled much as traditional wavelet coefficients. Tasks such as compression, noise removal, contrast enhancement, and stochastic simulation are facilitated by this representation. The approach applies to general manifolds, but is particularly suited to the manifolds we consider, i.e. Riemanian symmetric spaces, such as Sn-1, SO(n), G(n, k), where the Exp and Log maps are effectively computable. Applications to manifold-valued data sources of a geometric nature (motion, orientation, diffusion) seem particularly immediate. A software toolbox, SymmLab, can reproduce the results discussed in this paper.Statisticsvcs2115StatisticsArticlesA Flexible Bayesian Generalized Linear Model for Dichotomous Response Data with an Application to Text Categorization
https://academiccommons.columbia.edu/catalog/ac:173817
Eyheramendy, Susana; Madigan, David B.10.7916/D86M34ZFMon, 26 Jun 2017 20:40:43 +0000We present a class of sparse generalized linear models that include probit and logistic regression as special cases and offer some extra flexibility. We provide an EM algorithm for learning the parameters of these models from data. We apply our method in text classification and in simulated data and show that our method outperforms the logistic and probit models and also the elastic net, in general by a substantial margin.Mathematics, Statisticsdm2418StatisticsChapters (layout features)Mathematical Representations of Development Theories
https://academiccommons.columbia.edu/catalog/ac:168029
Singer, Burton; Spilerman, Seymour10.7916/D8NP22DSMon, 26 Jun 2017 20:39:04 +0000In this chapter we explore the consequences of particular stage linkage structures for the evolution of a population. We first argue the importance of constructing dynamic models of development theories and show the implications of various stage connections for population movements. A second focus concerns inverse problems: How the stage linkage structure may be recovered from survey data of the kind collected by developmental psychologists.Developmental psychology, Statisticsss50SociologyChapters (layout features)When Does Non-Negative Matrix Factorization Give a Correct Decomposition into Parts?
https://academiccommons.columbia.edu/catalog/ac:140175
Donoho, David L.; Stodden, Victoria C.10.7916/D88D05N7Mon, 26 Jun 2017 20:25:26 +0000We interpret non-negative matrix factorization geometrically, as the problem of finding a simplicial cone which contains a cloud of data points and which is contained in the positive orthant. We show that under certain conditions, basically requiring that some of the data are spread across the faces of the positive orthant, there is a unique such simplicial cone. We give examples of synthetic image articulation databases which obey these conditions; these require separated support and factorial sampling. For such databases there is a generative model in terms of "parts" and NMF correctly identifies the "parts". We show that our theoretical results are predictive of the performance of published NMF code, by running the published algorithms on one of our synthetic image articulation databases.Statisticsvcs2115StatisticsArticlesFast <em>l</em>1 Minimization for Genomewide Analysis of mRNA Lengths
https://academiccommons.columbia.edu/catalog/ac:140172
Drori, Iddo; Stodden, Victoria C.; Hurowitz, Evan H.10.7916/D80V8P4RMon, 26 Jun 2017 20:25:25 +0000Application of the virtual northern method to human mRNA allows us to systematically measure transcript length on a genome-wide scale [1]. Characterization of RNA transcripts by length provides a measurement which complements cDNA sequencing. We have robustly extracted the lengths of the transcripts expressed by each gene for comparison with the Unigene, Refseq, and H-Invitational databases [2, 3]. Obtaining an accurate probability for each peak requires performing multiple bootstrap simulations, each involving a deconvolution operation which is equivalent to finding the sparsest non-negative solution of an underdetermined system of equations. This process is computationally intensive for a large number of simulations and genes. In this contribution we present an efficient approximation method which is faster than general purpose solvers by two orders of magnitude, and in practice reduces our processing time from a week to hours.Genetics, Statisticsvcs2115StatisticsArticlesBreakdown Point of Model Selection When the Number of Variables Exceeds the Number of Observations
https://academiccommons.columbia.edu/catalog/ac:140168
Donoho, David L.; Stodden, Victoria C.10.7916/D84M9DXZMon, 26 Jun 2017 20:25:24 +0000The classical multivariate linear regression problem assumes p variables X1, X2, ... , Xp and a response vector y, each with n observations, and a linear relationship between the two: y = X beta + z, where z ~ N(0, sigma2). We point out that when p > n, there is a breakdown point for standard model selection schemes, such that model selection only works well below a certain critical complexity level depending on n/p. We apply this notion to some standard model selection algorithms (Forward Stepwise, LASSO, LARS) in the case where pGtn. We find that 1) the breakdown point is well-de ned for random X-models and low noise, 2) increasing noise shifts the breakdown point to lower levels of sparsity, and reduces the model recovery ability of the algorithm in a systematic way, and 3) below breakdown, the size of coefficient errors follows the theoretical error distribution for the classical linear model.Statisticsvcs2115StatisticsArticlesHigher-order Properties of Approximate Estimators
https://academiccommons.columbia.edu/catalog/ac:199766
Kristensen, Dennis; Salanie, Bernard10.7916/D8KK9BVXWed, 21 Jun 2017 14:27:44 +0000Many modern estimation methods in econometrics approximate an objective function, for instance, through simulation or discretization. These approximations typically affect both bias and variance of the resulting estimator. We first provide a higher-order expansion of such “approximate” estimators that takes into account the errors due to the use of approximations. We show how a Newton-Raphson adjustment can reduce the impact of approximations. Then we use our expansions to develop inferential tools that take into account approximation errors: we propose adjustments of the approximate estimator that remove its first-order bias and adjust its standard errors. These corrections apply to a class of approximate estimators that includes all known simulation-based procedures. A Monte Carlo simulation on the mixed logic model shows that our proposed adjustments can yield significant improvements at a low computational cost.Econometrics, Estimation theory, Statistics, Economics, Mathematics, Computer sciencedk2313, bs2237EconomicsReportsSemi-convergence of an Iterative Algorithm
https://academiccommons.columbia.edu/catalog/ac:194857
Vasilaky, Kathryn N.10.7916/D8SJ1KFXWed, 21 Jun 2017 14:26:51 +0000An iterative method is introduced for solving noisy, ill-conditioned inverse problems. Analysis of the semi-convergence behavior identifies three error components - iteration error, noise error, and initial guess error. A derived expression explains how the three errors are related to each other relative to the number of iterations. The Standard Tikhonov regularization method is just the first iteration of the iterative method and the derived noise damping filter is a generalization of the Standard Tikhonov filter. The derived filter is a function two parameters, a regularization parameter and the iteration number parameter. The new method is tested on image reconstruction from projections simulated data set.Iterative methods (Mathematics), Filters (Mathematics), Inverse problems (Differential equations), Statistics, Mathematicsknv4Earth InstituteReportsHierarchical Bayes models for daily rainfall time series at multiple locations from heterogenous data sources
https://academiccommons.columbia.edu/catalog/ac:199595
Shirley, Kenneth; Vasilaky, Kathryn N.; Greatrex, Helen L.; Osgood, Daniel E.10.7916/D8QF8SZ4Wed, 21 Jun 2017 14:26:48 +0000We estimate a Hierarchical Bayesian models for daily rainfall that incorporates two novelties for estimating spatial and temporal correlations. We estimate the within site time series correlations for a particular rainfall site using multiple data sources at a given location, and we estimate the across site covariance in rainfall based on location distance. Previous rainfall models have captured cross site correlations as a functions of site specific distances, but not within site correlations across multiple data sources, and not both aspects simultaneously. Further, we incorporate information on the technology used (satellite versus rain gauge) in our estimations, which is also a novel addition. This methodology has far reaching applications in providing more accurate and complex weather insurance contracts based combining information from multiple data sources from a single site, a crucial improvement in the face of climate change. Secondly, the modeling extends to many other data contexts where multiple datasources exist for a given event or variable where both within and between series covariances can be estimated over time.Rain and rainfall--Forecasting, Computer simulation, Rain and rainfall--Mathematical models, Statistics, Mathematics, Meteorologyknv4, hlg2124, do2126Earth Institute, International Research Institute for Climate and SocietyReportsHigher-order Properties of Approximate Estimators
https://academiccommons.columbia.edu/catalog/ac:188409
Kristensen, Dennis; Salanie, Bernard10.7916/D89886BKWed, 21 Jun 2017 13:58:02 +0000Many modern estimation methods in econometrics approximate an objective function, for instance, through simulation or discretization. These approximations typically affect both bias and variance of the resulting estimator. We first provide a higher-order expansion of such "approximate" estimators that takes into account the errors due to the use of approximations. We show how a Newton-Raphson adjustment can reduce the impact of approximations. Then we use our expansions to develop inferential tools that take into account approximation errors: we propose adjustments of the approximate estimator that remove its first-order bias and adjust its standard errors. These corrections apply to a class of approximate estimators that includes all known simulation-based procedures. A Monte Carlo simulation on the mixed logit model shows that our proposed adjustments can yield spectacular improvements at a low computational cost.Statistics, Economics, Mathematics, Computer sciencedk2313, bs2237EconomicsReportsDevelopment of a Parsimonious Set of City-level Environmental Performance Metrics for Jiyuan, Henan, China
https://academiccommons.columbia.edu/catalog/ac:188777
Guo, Dong; Bose, Satyajit10.7916/D8VQ322WWed, 21 Jun 2017 13:56:37 +0000The potential tradeoff between the twin goals of reducing environmental impact while maintaining growth will require China’s cities to evaluate the economic impact of urban pollution at the local level. Using economic input-output analysis, city level indicators of economic activity and environmental impact and available estimates of the benchmark relationships between output and pollution by sector, we outline a method to quantify in monetary terms the marginal damages of air pollution by sector at the city level. By applying the framework of environmental accounting to the pilot case of Jiyuan, a small city in Henan province, we demonstrate a method for local public agencies to facilitate administrative tracking of monetized air pollution based on underlying economic activity, and outline a minimum set of metrics which a small city in China must track in order to estimate the monetized damage of air pollution by sector. Our methodology leverages economy-wide aggregate models (Ho and Nielsen 2007, The World Bank 2007) to significantly reduce the metrics required for a simple approximation of the relative value added per unit of emission by sector for medium-sized cities in China.Public policy (Law), Statisticsdg2350, sgb2Earth Institute, School of Continuing EducationReportsSource codes for GLMLE algorithm
https://academiccommons.columbia.edu/catalog/ac:178966
He, Ran; Zheng, Tian10.7916/D8HH6HQRWed, 21 Jun 2017 13:55:17 +0000These are the R source codes for the algorithm proposed for fitting exponential random graph models (ERGMs) on large social networks in our paper "Estimation of exponential random graph models for large social networks via graph limits". Specifically, the ERGM model we implement is the one that consider homomorphism densities of edges, two-stars and triangles, the one we examine in the above paper.Statistics, Computer sciencerh2528, tz33StatisticsSoftwareSPAr package for Fan and Lo (2013) "A robust model-free approach for rare variants association studies incorporating gene-gene and gene-environmental interactions."
https://academiccommons.columbia.edu/catalog/ac:179424
Fan, Ruixue; Lo, Shaw-Hwa10.7916/D84Q7SN6Wed, 21 Jun 2017 13:55:16 +0000Recently more and more evidence suggest that rare variants with much lower minor allele frequencies play significant roles in disease etiology. Advances in next-generation sequencing technologies will lead to many more rare variants association studies. Several statistical methods have been proposed to assess the effect of rare variants by aggregating information from multiple loci across a genetic region and testing the association between the phenotype and aggregated genotype. One limitation of existing methods is that they only look into the marginal effects of rare variants but do not systematically take into account effects due to interactions among rare variants and between rare variants and environmental factors. In this article, we propose the summation of partition approach (SPA), a robust model-free method that is designed specifically for detecting both marginal effects and effects due to gene-gene (G×G) and gene-environmental (G×E) interactions for rare variants association studies. SPA has three advantages. First, it accounts for the interaction information and gains considerable power in the presence of unknown and complicated G×G or G×E interactions. Secondly, it does not sacrifice the marginal detection power; in the situation when rare variants only have marginal effects it is comparable with the most competitive method in current literature. Thirdly, it is easy to extend and can incorporate more complex interactions; other practitioners and scientists can tailor the procedure to fit their own study friendly. Our simulation studies show that SPA is considerably more powerful than many existing methods in the presence of G×G and G×E interactions.
This package is also maintained on the Comprehensive R Archive Network (http://cran.r-project.org). It contains the R programs, user's manual and example codes.Genetics, Statisticsrf2283, shl5StatisticsSoftwareSample Palomar Transient Factory light curves
https://academiccommons.columbia.edu/catalog/ac:167874
Price-Whelan, Adrian Michael; Agüeros, Marcel Andre; Fournier, Amanda P.; Street, Rachel; Ofek, Eran O.; Covey, Kevin R.; Levitan, David; Laher, Russ R.; Sesar, Branimir; Surace, Jason10.7916/D8CF9N1NTue, 20 Jun 2017 16:45:15 +0000These light curves are made available to the public as part of the publication of our recent paper, "Statistical Searches for Microlensing Events in Large, Non-Uniformly Sampled Time-Domain Surveys: A Test Using Palomar Transient Factory Data." We have selected ~10,000 light curves from the Palomar Transient Factory database that can be used to test the various statistical tools described in the paper.Astronomy, Statisticsamp2217, maa17AstronomyData (information)Fit GFuseTLP penalized conditional logistic regression model for high-dimensional one-to- one matched case-control data
https://academiccommons.columbia.edu/catalog/ac:174087
Zhou, Hui; Wang, Shuang; Zheng, Tian10.7916/D8028PNJTue, 20 Jun 2017 16:44:54 +0000Fit GFuseTLP penalized conditional logistic regression model for high-dimensional one-to- one matched case-control dataStatisticshz2106, sw2206, tz33Biostatistics, StatisticsSoftwareMultiplicative Multiresolution Analysis for Lie-group Valued Data Indexed by a Euclidean Parameter
https://academiccommons.columbia.edu/catalog/ac:155756
Stodden, Victoria C.10.7916/D8GM8GPCMon, 19 Jun 2017 17:46:15 +0000Lie-valued euclidean indexed data. These data might be: phase angles as functions of time or space, for example compass directions; 3D orientations of a rigid frame of reference as a function of time or space; or, quaternions as a function of time or space. This can also be extended to quotients of lie groups which gives us the ability to model points on S2, the unit sphere, as functions of time or space.Computer science, Statisticsvcs2115StatisticsPresentations (Communicative Events)Flexible Sparse Learning of Feature Subspaces
https://academiccommons.columbia.edu/catalog/ac:207319
Ma, Yuting10.7916/D83X8CBBThu, 15 Jun 2017 16:10:42 +0000It is widely observed that the performances of many traditional statistical learning methods degenerate when confronted with high-dimensional data. One promising approach to prevent this downfall is to identify the intrinsic low-dimensional spaces where the true signals embed and to pursue the learning process on these informative feature subspaces. This thesis focuses on the development of flexible sparse learning methods of feature subspaces for classification. Motivated by the success of some existing methods, we aim at learning informative feature subspaces for high-dimensional data of complex nature with better flexibility, sparsity and scalability.
The first part of this thesis is inspired by the success of distance metric learning in casting flexible feature transformations by utilizing local information. We propose a nonlinear sparse metric learning algorithm using a boosting-based nonparametric solution to address metric learning problem for high-dimensional data, named as the sDist algorithm. Leveraged a rank-one decomposition of the symmetric positive semi-definite weight matrix of the Mahalanobis distance metric, we restructure a hard global optimization problem into a forward stage-wise learning of weak learners through a gradient boosting algorithm. In each step, the algorithm progressively learns a sparse rank-one update of the weight matrix by imposing an L-1 regularization. Nonlinear feature mappings are adaptively learned by a hierarchical expansion of interactions integrated within the boosting framework. Meanwhile, an early stopping rule is imposed to control the overall complexity of the learned metric. As a result, without relying on computationally intensive tools, our approach automatically guarantees three desirable properties of the final metric: positive semi-definiteness, low rank and element-wise sparsity. Numerical experiments show that our learning model compares favorably with the state-of-the-art methods in the current literature of metric learning.
The second problem arises from the observation of high instability and feature selection bias when applying online methods to highly sparse data of large dimensionality for sparse learning problem. Due to the heterogeneity in feature sparsity, existing truncation-based methods incur slow convergence and high variance. To mitigate this problem, we introduce a stabilized truncated stochastic gradient descent algorithm. We employ a soft-thresholding scheme on the weight vector where the imposed shrinkage is adaptive to the amount of information available in each feature. The variability in the resulted sparse weight vector is further controlled by stability selection integrated with the informative truncation. To facilitate better convergence, we adopt an annealing strategy on the truncation rate. We show that, when the true parameter space is of low dimension, the stabilization with annealing strategy helps to achieve lower regret bound in expectation.Mathematical statistics, Machine learning--Statistical methods, Machine learning, Statisticsym2396StatisticsThesesMeasuring Spatial Extremal Dependence
https://academiccommons.columbia.edu/catalog/ac:202722
Cho, Yong Bum10.7916/D8PR7W8TThu, 15 Jun 2017 16:10:34 +0000The focus of this thesis is extremal dependence among spatial observations. In particular, this research extends the notion of the extremogram to the spatial process setting. Proposed by Davis and Mikosch (2009), the extremogram measures extremal dependence for a stationary time series. The versatility and flexibility of the concept made it well suited for many time series applications including from finance and environmental science.
After defining the spatial extremogram, we investigate the asymptotic properties of the empirical estimator of the spatial extremogram. To this end, two sampling scenarios are considered: 1) observations are taken on the lattice and 2) observations are taken on a continuous region in a continuous space, in which the locations are points of a homogeneous Poisson point process. For both cases, we establish the central limit theorem for the empirical spatial extremogram under general mixing and dependence conditions. A high level overview is as follows. When observations are observed on a lattice, the asymptotic results generalize those obtained in Davis and Mikosch (2009). For non-lattice cases, we define a kernel estimator of the empirical spatial extremogram and establish the central limit theorem provided the bandwidth of the kernel gets smaller and the sampling region grows at proper speeds. We illustrate the performance of the empirical spatial extremogram using simulation examples, and then demonstrate the practical use of our results with a data set of rainfall in Florida and ground-level ozone data in the eastern United States.
The second part of the thesis is devoted to bootstrapping and variance estimation with a view towards constructing asymptotically correct confidence intervals. Even though the empirical spatial extremogram is asymptotically normal, the limiting variance is intractable. We consider three approaches: for lattice data, we use the circular bootstrap adapted to spatial observations, jackknife variance estimation, and subsampling variance estimation. For data sampled according to a Poisson process, we use subsampling methods to estimate the variance of the empirical spatial extremogram. We establish the (conditional) asymptotic normality for the circular block bootstrap estimator for the spatial extremogram and show L2 consistency of the variance estimated by jackknife and subsampling. Then, we propose a portmanteau style test to check the existence of extremal dependences at multiple lags. The validity of confidence intervals produced from these approaches and a portmanteau style test are demonstrated through simulation examples. Finally, we illustrate this methodology to two data sets. The first is the amount of rainfall over a grid of locations in northern Florida. The second is ground-level ozone in the eastern United States, which are recorded on an irregularly spaced set of stations.Spatial analysis (Statistics), Statistics, Bootstrap (Statistics), Extremal problems (Mathematics)yc2500StatisticsThesesRandom Walk Models, Preferential Attachment, and Sequential Monte Carlo Methods for Analysis of Network Data
https://academiccommons.columbia.edu/catalog/ac:209294
Bloem-Reddy, Benjamin Michael10.7916/D8348R5QThu, 15 Jun 2017 16:10:23 +0000Networks arise in nearly every branch of science, from biology and physics to sociology and economics. A signature of many network datasets is strong local dependence, which gives rise to phenomena such as sparsity, power law degree distributions, clustering, and structural heterogeneity. Statistical models of networks require a careful balance of flexibility to faithfully capture that dependence, and simplicity, to make analysis and inference tractable. In this dissertation, we introduce a class of models that insert one network edge at a time via a random walk, permitting the location of new edges to depend explicitly on the structure of the existing network, while remaining probabilistically and computationally tractable. Connections to graph kernels are made through the probability generating function of the random walk length distribution. The limiting degree distribution is shown to exhibit power law behavior, and the properties of the limiting degree sequence are studied analytically with martingale methods. In the second part of the dissertation, we develop a class of particle Markov chain Monte Carlo algorithms to perform inference for a large class of sequential random graph models, even when the observation consists only of a single graph. Using these methods, we derive a particle Gibbs sampler for random walk models. Fit to synthetic data, the sampler accurately recovers the model parameters; fit to real data, the model offers insight into the typical length scale of dependence in the network, and provides a new measure of vertex centrality.
The arrival times of new vertices are the key to obtaining results for both theory and inference. In the third part, we undertake a careful study of the relationship between the arrival times, sparsity, and heavy tailed degree distributions in preferential attachment-type models of partitions and graphs. A number of constructive representations of the limiting degrees are obtained, and connections are made to exchangeable Gibbs partitions as well as to recent results on the limiting degrees of preferential attachment graphs.Computer networks, Markov processes, Information networks--Statistical methods, Monte Carlo method, Statisticsbmr2136StatisticsThesesDeconstructing Spinal Interneurons, one cell type at a time
https://academiccommons.columbia.edu/catalog/ac:209127
Gabitto, Mariano Ignacio10.7916/D8PN95XCThu, 15 Jun 2017 16:09:23 +0000Documenting the extent of cellular diversity is a critical step in defining the functional organization of the nervous system. In this context, we sought to develop statistical methods capable of revealing underlying cellular diversity given incomplete data sampling - a common problem in biological systems, where complete descriptions of cellular characteristics are rarely available. We devised a sparse Bayesian framework that infers cell type diversity from partial or incomplete transcription factor expression data. This framework appropriately handles estimation uncertainty, can incorporate multiple cellular characteristics, and can be used to optimize experimental design. We applied this framework to characterize a cardinal inhibitory population in the spinal cord.
Animals generate movement by engaging spinal circuits that direct precise sequences of muscle contraction, but the identity and organizational logic of local interneurons that lie at the core of these circuits remain unresolved. By using our Sparse Bayesian approach, we showed that V1 interneurons, a major inhibitory population that controls motor output, fractionate into diverse subsets on the basis of the expression of nineteen transcription factors. Transcriptionally defined subsets exhibit highly structured spatial distributions with mediolateral and dorsoventral positional biases. These distinctions in settling position are largely predictive of patterns of input from sensory and motor neurons, arguing that settling position is a determinant of inhibitory microcircuit organization. Finally, we extensively validated inferred cell types by direct experimental measurement and then, extend our Bayesian framework to full transcriptome technologies. Together, these findings provide insight into the diversity and organizational logic through which inhibitory microcircuits shape motor output.Interneurons, Bayesian statistical decision theory, Molecular biology--Statistical methods, Spinal cord, Neurosciences, StatisticsNeurobiology and BehaviorThesesAdvances in Credit Risk Modeling
https://academiccommons.columbia.edu/catalog/ac:206336
Neuberg, Richard10.7916/D84T6JZ0Thu, 15 Jun 2017 16:08:01 +0000Following the recent financial crisis, financial regulators have placed a strong emphasis on reducing expectations of government support for banks, and on better managing and assessing risks in the banking system. This thesis considers three current topics in credit risk and the statistical problems that arise there.
The first of these topics is expectations of government support in distressed banks. We utilize unique features of the European credit default swap market to find that market expectations of European government support for distressed banks have decreased -- an important development in the credibility of financial reforms.
The second topic we treat is the estimation of covariance matrices from the perspective of market risk management. This problem arises, for example, in the central clearing of credit default swaps. We propose several specialized loss functions, and a simple but effective visualization tool to assess estimators. We find that proper regularization significantly improves the performance of dynamic covariance models in estimating portfolio variance.
The third topic we consider is estimation risk in the pricing of financial products. When parameters are not known with certainty, a better informed counterparty may strategically pick mispriced products. We discuss how total estimation risk can be minimized approximately. We show how a premium for remaining estimation risk may be determined when one counterparty is better informed than the other, but a market collapse is to be avoided, using a simple example from loan pricing. We illustrate the approach with credit bureau data.Financial risk, Financial risk management, Finance--Statistical methods, Finance--Statistics, Credit--Management--Statistical methods, Statistics, Financern2325StatisticsThesesRents, Patronage, and Defection: State-building and Insurgency in Afghanistan
https://academiccommons.columbia.edu/catalog/ac:206825
Gopal, Anand10.7916/D81G0RWXThu, 15 Jun 2017 16:06:51 +0000Afghanistan has been one of the most protracted conflicts modern era, but theories of civil war onset fail to explain the war’s causes or its patterns of violence. This thesis examines the origins of the post-2001 period of the conflict through the perspective of state formation; although many civil wars today unfold in newly-forming states, the processes of center-periphery relations and elite incorporation have been little studied in the context of political violence. The thesis first describes how Afghanistan’s embeddedness in the international state system and global markets undermined the nascent state’s efforts to centralize and bureaucratize, leading instead to warlordism and neopatrimonialism. Second, it demonstrates that the development of an insurgency after 2001 was due not to ethnic grievance or rebel opportunities for profit, but rather to the degree to which local elites were excluded from state patronage. Third, it examines the role of ideology and social position in the Afghan Taliban movement. The dissertation seeks to offer a theory of political violence in Afghanistan that can, mutatis mutandis, help explain key features of civil war in newly-forming states.Insurgency, Nation-building, Civil war, Political violence, Sociology, Statisticsag3291SociologyThesesDeveloping an approach to determine generalizability: A review of efficacy and effectiveness trials funded by the Institute of Education Sciences
https://academiccommons.columbia.edu/catalog/ac:206699
Fellers, Lauren Ashley10.7916/D86D5ZN1Thu, 15 Jun 2017 16:06:42 +0000Since its establishment the Institute of Education Sciences has been creating opportunities and driving standards to generate research in education that is high quality rigorous, and relevant. This dissertation is an analysis of current practices in Goal III and Goal IV studies, in order to (1) better understand of the types of schools that agree to take part in these studies, and (2) an assess how representative these schools are in comparison to important policy relevant populations. This dissertation focuses on a subset of studies that were funded from 2005-2014 by the Department of Education, IES, under the NCER grants-funding arm. Studies included were those whose interventions were aimed at elementary students across core curriculum and ELL program areas. Study schools were compared to two main populations, the U.S population of elementary schools and Title I elementary schools, as well as these populations on a state level. The B-index, proposed by Tipton (2014) was the main value of comparison used to assess the compositional similarity, or generalizability, of study schools to these identified inference populations. The findings show that across all studies included in this analysis, participating schools were representative of the U.S. population of schools, B-index = 0.9. Comparisons were also made between this collection of schools and the respective populations at the state level. Results showed that these schools were not representative of any individual states (no B-index values were greater than 0.90). Across all included studies, schools that agreed to participate were more often located in urban areas, had higher rates of FRL students, had more minority students enrolled, and had more total students, in both district and school, than those schools in the population of U.S. schools. It is clear that the movement of education research is to be relevant to a larger audience. Through this study it is clear that, across studies, we are achieving some representation in IES funded studies. However, the finer comparisons, study samples to individual state and individual studies to these populations, show limited similarity between study schools and populations of interest to policy makers using these study findings to make decisions about their schools.Education--Research, Educational evaluation, Educational statistics, Education--Statistics, Statisticslaf2156Measurement and EvaluationThesesEstimation of Q-matrix for DINA Model Using the Constrained Generalized DINA Framework
https://academiccommons.columbia.edu/catalog/ac:198654
Li, Huacheng10.7916/D88W3DB2Thu, 15 Jun 2017 15:07:07 +0000The research of cognitive diagnostic models (CDMs) is becoming an important field of psychometrics. Instead of assigning one score, CDMs provide attribute profiles to indicate the mastering status of concepts or skills for the examinees. This would make the test result more informative. The implementation of many CDMs relies on the existing item-to-attribute relationship, which means that we need to know the concepts or skills each item requires. The relationships between the items and attributes could be summarized into the Q-matrix. Misspecification of the Q-matrix will lead to incorrect attribute profile. The Q-matrix can be designed by expert judgement, but it is possible that such practice can be subjective. There are previous researches about the Q-matrix estimation. This study proposes an estimation method for one of the most parsimonious CDMs, the DINA model. The method estimates the Q-matrix for DINA model by setting constraints on the generalized DINA model. In the simulation study, the results showed that the estimated Q-matrix fit better the empirical fraction subtraction data than the expert-design Q-matrix. We also show that the proposed method may still be applicable when the constraints were relaxed.Psychometrics, Educational tests and measurements, Statisticshl2536Measurement and EvaluationThesesOn Model-Selection and Applications of Multilevel Models in Survey and Causal Inference
https://academiccommons.columbia.edu/catalog/ac:200369
Wang, Wei10.7916/D8571C4QThu, 15 Jun 2017 15:07:06 +0000This thesis includes three parts. The overarching theme is how to analyze multilevel structured datasets, particularly in the areas of survey and causal inference. The first part discusses model selection of hierarchical models, in the context of a national political survey. I found that the commonly used model selection criteria based on predictive accuracy, such as cross validation, don't perform very well in the case of political survey and explore the possible causes. The second part centers around a unique data set on the presidential election collected through an online platform. I show that with adequate modeling, meaningful and highly accurate information could be extracted from this highly-biased data set. The third part builds on a formal causal inference framework for group-structured data, such as meta-analysis and multi-site trials. In particular, I develop a Gaussian Process model under this framework and demonstrate additional insights that can be gained compared with traditional parametric models.Social sciences--Statistical methods--Data processing, Statisticsww2243StatisticsThesesMethods for Personalized and Evidence Based Medicine
https://academiccommons.columbia.edu/catalog/ac:195007
Shahn, Zach10.7916/D8M0458SThu, 15 Jun 2017 15:06:44 +0000There is broad agreement that medicine ought to be `evidence based' and `personalized' and that data should play a large role in achieving both these goals. But the path from data to improved medical decision making is not clear. This thesis presents three methods that hopefully help in small ways to clear the path.
Personalized medicine depends almost entirely on understanding variation in treatment effect. Chapter 1 describes latent class mixture models for treatment effect heterogeneity that distinguish between continuous and discrete heterogeneity, use hierarchical shrinkage priors to mitigate overfitting and multiple comparisons concerns, and employ flexible error distributions to improve robustness. We apply different versions of these models to reanalyze a clinical trial comparing HIV treatments and a natural experiment on the effect of Medicaid on emergency department utilization.
Medical decisions often depend on observational studies performed on large longitudinal health insurance claims databases. These studies usually claim to identify a causal effect, but empirical evaluations have demonstrated that standard methods for causal discovery perform poorly in this context, most likely in large part due to the presence of unobserved confounding. Chapter 2 proposes an algorithm called Ensembles of Granger Graphs (EGG) that does not rely on the assumption that unobserved confounding is absent. In a simulation and experiments on a real claims database, EGG is robust to confounding, has high positive predictive value, and has high power to detect strong causal effects.
While decision making inherently involves causal inference, purely predictive models aid many medical decisions in practice. Predictions from health histories are challenging because the space of possible predictors is so vast. Not only are there thousands of health events to consider, but also their temporal interactions. In Chapter 3, we adapt a method originally developed for speech recognition that greedily constructs informative labeled graphs representing temporal relations between multiple health events at the nodes of randomized decision trees. We use this method to predict strokes in patients with atrial fibrillation using data from a Medicaid claims database.
I hope the ideas illustrated in these three projects inspire work that someday genuinely improves healthcare. I also include a short `bonus' chapter on an improved estimate of effective sample size in importance sampling. This chapter is not directly related to medicine, but finds a home in this thesis nonetheless.Evidence-based medicine, Medical care--Statistics, Personalized medicine, Statisticszss2101StatisticsThesesSpectral Filtering for Spatio-temporal Dynamics and Multivariate Forecasts
https://academiccommons.columbia.edu/catalog/ac:198310
Meng, Lu10.7916/D80Z7385Thu, 15 Jun 2017 15:06:43 +0000Due to the increasing availability of massive spatio-temporal data sets, modeling high dimensional data becomes quite challenging. A large number of research questions are rooted in identifying underlying dynamics in such spatio-temporal data. For many applications, the science suggests that the intrinsic dynamics be smooth and of low dimension. To reduce the variance of estimates and increase the computational tractability, dimension reduction is also quite necessary in the modeling procedure. In this dissertation, we propose a spectral filtering approach for dimension reduction and forecast amelioration, and apply it to multiple applications. We show the effectiveness of dimension reduction via our method and also illustrate its power for prediction in both simulation and real data examples. The resultant lower dimensional principal component series has a diagonal spectral density at each frequency whose diagonal elements are in descending order, which is not well motivated can be hard to interpret. Therefore we propose a phase-based filtering method to create principal component series with interpretable dynamics in the time domain. Our method is based on an approach of structural decomposition and phase-aligned construction in the frequency domain, identifying lower-rank dynamics and its components embedded in a high dimensional spatio-temporal system. In both our simulated examples and real data applications, we illustrate that the proposed method is able to separate and identify meaningful lower-rank movements. Benefiting from the zero-coherence property of the principal component series, we subsequently develop a predictive model for high-dimensional forecasting via lower-rank dynamics. Our modeling approach reduces multivariate modeling task to multiple univariate modeling and is flexible in combining with regularization techniques to obtain more stable estimates and improve interpretability. The simulation results and real data analysis show that our model achieves superior forecast performance compared to the class of autoregressive models.Mathematical statistics--Data processing, Dynamics, Dimension reduction (Statistics), Statisticslm2844StatisticsTheses