2026 Theses Doctoral

Scalable Deep Reinforcement Learning for Inventory Management

Alvo, Matias

This dissertation studies scalable policy optimization methods for high-dimensional sequential decision-making problems, with a focus on inventory control. Policy-based reinforcement learning (RL) algorithms based on REINFORCE treat the environment as a black box, making them applicable under broad conditions but often leading to high sample complexity and instability. Across the three chapters, the central goal is to design RL algorithms that exploit problem structure-such as known dynamics, exogenous uncertainty, known network topology, and smoothness-rather than treating the environment as a black box. Doing so enables more data-efficient and stable policy optimization in complex operations problems.

The first chapter studies the use of pathwise gradients and differentiable simulation for inventory control. Given a simulator that can replay historical or simulated demand trajectories, we train neural policies by backpropagating through inventory dynamics using Hindsight Differentiable Policy Optimization (HDPO). This approach provides lower-variance learning signals than standard score-function policy gradients and enables reliable optimization in several inventory settings, including problems with realistic demand data. The chapter focuses on simple fully connected neural architectures and establishes the value of differentiable simulation as a policy optimization tool for inventory control.

The second chapter develops graph neural network policies for large-scale inventory networks. While fully connected architectures ignore the structure of multi-location systems, graph neural networks encode the physical flow network directly, enabling local decision-making, parameter sharing across similar locations, and coordination through message passing. We show that combining these architectures with HDPO substantially improves sample efficiency and scalability in complex inventory networks, and we investigate which architectural features drive these gains.

The third chapter extends differentiable policy optimization to problems with hybrid discrete-continuous action spaces. In such settings, pathwise gradients can be used for continuous decisions but not for discrete choices, while standard model-free estimators often scale poorly. We propose Hybrid Policy Optimization (HPO), which combines pathwise and score-function terms into an unbiased mixed gradient estimator. Experiments on inventory control and switched linear-quadratic regulator problems show that HPO scales more effectively than PPO as the continuous action dimension grows. The chapter also analyzes the structure of the mixed gradient and identifies conditions under which the interaction between discrete and continuous components becomes negligible.

Together, the chapters develop a framework for reliable reinforcement learning in structured operations problems. The main theme is that exploiting available structure can substantially improve credit assignment, generalization, and sample efficiency in high-dimensional decision problems.