2026 Theses Doctoral

Structure-aware Reinforcement Learning Algorithms for Generative Models Post Training

Zhao, Hanyang

Recently, reinforcement Learning (RL) has achieved remarkable success in enhancing the capabilities of modern generative models. RL can teach generative model to generate contents which are better aligned with human intent through reinforcement learning from human feedback (RLHF); it can also incentivize LLMs to reason through long chain of thought, showcasing expert level performance in math and coding benchmarks from verifiable rewards (RLVR). Yet to fully unleash the power of RL in post training large scaled pretrained generative models given the enormous amount of compute and resources required, designing and employing sample-efficient RL algorithms for generative models to learn from interactions have become of more than critical importance.

We notice that most existing RL methods for generative models have brute forcedly treated the underlying generative models as black boxes, ignoring or minimally leveraging the useful structure and properties of the underlying generative models. The topic of this thesis is to design RL algorithms that are structural aware of the underlying generative models, and domonstrate how the obtained RL algorithms could yield better performance by capturing and leveraging the underlying structures of modern generative models.

The background of this thesis mainly lies in RL and Generative Models. In Chapter 1, we first review the necessary background and foundations of RL in the discrete time scenario, including its application to fine-tuning generative models. The class of generative models we are going to discuss is diffusion probabilistic models with applications to either continuous distributions (images) or discrete distributions (texts).

In Chapter 2, we first present some new theoretical results for RL in continuous time and space, focusing on an infinite horizon scenario which has not been studied before, and extends the existing policy optimization results of discrete time to the continuous time and space, upon which we propose continuous-time policy gradient (CPG) and continuous-time proximal policy optimization (CPPO).

In Chapter 3, we showcase how the task of post training continuous-time continuous-space diffusion based generative models can be readily transformed into a special instance of continuous-time reinforcement learning. Built upon this insight, we propose Score as Action, a framework by treating the score function as an action/control over the denoising process of diffusion models, resulting to a RL algorithm which overcomes the bottleneck of existing algorithms and is more sample efficient.

In Chapter 4, we shift geers to the discrete space domain, and focus on diffusion language models. We showcase how to unlock more capability out of these base models via a unified off-policy RL framework to train dLLMs to reason not only better, but also faster. Our method enables more accurate learning from sample trajectories, and provides worst case performance guarantees that the baseline method lacks. Beyond improving model quality, we also introduce joint training of the model and its sampler utilizing the multi-token prediction capability of dLLMs. This allows dLLMs to adaptively control inference dynamics and further substantially improving the Pareto frontier of inference-time compute.

In Conclusion section, we conclude this thesis with unified themes, and we discuss the broader implications and future directions for the aforementioned research projects.