2026 Theses Doctoral

Beyond Supervised Learning: The Statistical Benefits of Indirect Estimation of Long-Term Outcomes

Cheikhi, David

Policy evaluation is a central primitive in sequential decision making: it aims at quantifying the long-term outcome of a fixed policy. Reinforcement learning (RL) models such problems as an agent interacting with an environment, collecting rewards as it transitions between states; the quality of a policy is then quantified by its value function, a mapping from states to the expected sum of future rewards under that policy. Policy evaluation reduces to estimating this function from observed trajectories. Supervised learning approaches do so directly, regressing empirical returns onto states. While their simplicity is appealing, they ignore the sequential nature of the data. In contrast, indirect approaches enforce that the learned value function be consistent with plausible evolutions of the environment under the agent’s actions. Such methods are popular in RL and a large literature exists, developing new methods and deriving guarantees on their performance. While these methods differ in their exact design, they can typically be categorized into classes based on how they enforce consistency. The goal of this thesis is to understand the statistical consequences of these design choices.

The first chapter contrasts temporal difference learning (TD) with direct value estimation in finite-state Markov reward processes. We show that an intuitive inverse trajectory pooling coefficient completely characterizes the asymptotic ratio of mean-squared errors between the two approaches — large when distinct trajectories share intermediate states, near one when they do not. For advantage estimation, TD’s benefits are sharper: its errors are bounded in terms of a novel trajectory crossing time that can be much smaller than the problem’s horizon.

The second chapter focuses on another classical tension in policy evaluation: model-free versus model-based. We show that methods relying only on a correct value-function class — including LSTD — can be grossly statistically inefficient. The driver is the price of information loss 𝜌, an explicit ratio of Cramér–Rao bounds that lower-bounds the inefficiency of any model-free estimator relative to the model-based plug-in on a reference class M. When the value function and the constrained part of the dynamics are informationally entangled, 𝜌 > 1 and every model-free estimator pays a price; when they decouple, our theory does not rule out efficient model-free estimation. Linear systems, factored Markov reward processes, and linear-quadratic control illustrate three distinct regimes.

The third chapter proposes PEvalSuite, an empirical framework for systematically studying the statistical properties of policy evaluation methods in deep on-policy reinforcement learning, bridging the gap between theoretical understanding and practical application. We treat a value-fitting method as a mapping from a random dataset to a value network. Sampling many i.i.d. datasets yields i.i.d. realisations of the resulting value function, and pairing them with oracle ground truth at selected states, obtained by large-budget Monte Carlo, gives access to a wide range of metrics. The methodology extends to the advantage and policy-gradient estimators built on top. We exercise the framework on TD(𝜆) for value fitting and GAE(𝜆) for advantage estimation, across MuJoCo continuous-control environments. The analysis recovers existing insights and sharpens them with a finer-grained view.