2025 Theses Doctoral

Two Problems in Geometry: Structural Insights into Interpolating Estimators in Overparameterized Models and Advances in Gaussian Entropic Inequalities

Ardeshir, Navid

This thesis explores two distinct areas of research.

The first part explores modern statistical learning schemes, where models are overparameterized with more parameters than training samples. Despite the fact that these models are able to interpolate the training data, they can often exhibit good generalization to unseen data due to algorithmic biases and implicitly favoring desirable estimators. In this part, we focus on studying the geometrical properties of such interpolating estimators, and their implications on statistical properties of overparameterized models. The problems addressed in the first part are motivated by the implicit biases that emerge in practical learning algorithms, examining both linear and nonlinear models.

For linear models, we study the structure of the solutions of hard-margin support vector machines (SVM) in high-dimensional binary classification, which corresponds to finding maximum margin hyperplane while classifying the labels in training data. Building on prior works, we investigate regimes where hard-margin SVM solution coincide with the minimum norm interpolating estimator known as ordinary least squares (OLS), and identify a sharp characterization of this coincidence in terms of distribution of the data.

For non-linear models, we study interpolating estimators based on (potentially infinitely wide) 2-layer neural networks with weight-decay regularization, and investigate their statistical properties for learning tasks with certain underlying structures. Our findings reveal that for certain tasks which are intrinsically "low-dimensional", the solutions found by these interpolators do not exhibit the same structure. Ultimately, we show that these interpolators are suboptimal for the proposed task.

Altogether, this part of the thesis shed further light on implicit biases of overparameterized models, by employing tools from geometry, functional analysis, and high-dimensional probability.

The second part of this thesis focuses on entropic information inequalities. In particular, we build on a fundamental result known as entropic Brascamp-Lieb inequality corresponding to superadditivity of relative entropy under linear transformations.

We extend the scope of this inequality to a broader class of inequalities involving higher moments of entropy in certain geometric settings.