title={Dancegrpo: Unleashing grpo on visual generation},
author={Xue, Zeyue and Wu, Jie and Gao, Yu and Kong, Fangyuan and Zhu, Lingting and Chen, Mengzhao and Liu, Zhiheng and Liu, Wei and Guo, Qiushan and Huang, Weilin and others},
journal={arXiv preprint arXiv:2505.07818},
year={2025}
}
@inproceedings{black-etal:training,
title={Training diffusion models with reinforcement learning},
author={Black, Kevin and Janner, Michael and Du, Yilun and Kostrikov, Ilya and Levine, Sergey},
booktitle={International Conference on Learning Representations},
volume={2024},
pages={4965--4987},
year={2024}
}
@inproceedings{sohl-etal:deep,
title={Deep unsupervised learning using nonequilibrium thermodynamics},
author={Sohl-Dickstein, Jascha and Weiss, Eric and Maheswaranathan, Niru and Ganguli, Surya},
@@ -48,28 +48,55 @@ Unlike multimodal understanding models that typically produce textual responses,
In this subsection, we describe the application of RL in two representative classes of multimodal generation models: diffusion-based models and flow matching-based models.
\subsubsection{Diffusion-based Models}
We first briefly introduce the basic generation process of diffusion-based models.
There are many aspects of diffusion models, such as image representations and mathematical formulations, that cannot be covered in this paper. Interested readers can refer to existing surveys on diffusion models for further details \citep{yang-etal:diffusion,croitoru-etal:diffusion}.
Diffusion models generate new samples by learning a gradual denoising process \citep{sohl-etal:deep}. Specifically, given a clean data sample $\mathbf{x}_0$, the forward process gradually adds Gaussian noise to the sample through a series of diffusion steps. At each timestep $t$, the noisy sample is generated as:
We first briefly introduce the basic generation process of diffusion-based models to provide the necessary notations. There are many aspects of diffusion models, such as image representations and mathematical formulations, that cannot be fully covered in this paper. Interested readers can refer to existing surveys on diffusion models for further details \citep{yang-etal:diffusion,croitoru-etal:diffusion}.
Specifically, diffusion models generate samples through a gradual denoising process \citep{sohl-etal:deep}. Given a clean sample $\mathbf{x}_0$, the forward diffusion process progressively adds Gaussian noise to the sample. At each timestep $t$, the transition distribution is defined as:
where $q(\cdot)$ denotes the predefined forward noise process, $\beta_t$ controls the noise magnitude at timestep $t$, and $\mathcal{N}(\mu,\sigma^2)$ denotes a Gaussian distribution with mean $\mu$ and variance $\sigma^2$. Through this forward process, the original data distribution is gradually transformed into a simple Gaussian noise distribution.
where $q(\cdot)$ denotes the predefined forward diffusion process, $\beta_t$ controls the noise magnitude at timestep $t$, and $\mathcal{N}(\mu,\sigma^2)$ denotes a Gaussian distribution with mean $\mu$ and variance $\sigma^2$. Through this forward process, the original data distribution is gradually transformed into a simple Gaussian noise distribution.
The generation process performs the reverse procedure, where a neural network is trained to gradually remove noise from a random noise sample. Specifically, given a noisy sample $\mathbf{x}_t$, the reverse process predicts the previous state:
The generation process performs the reverse denoising procedure, where a neural network learns to recover clean samples from noisy inputs:
where $p_\theta(\cdot)$ denotes the learned reverse denoising process used for generation, and $\mu_\theta(\cdot)$ and $\Sigma_\theta(\cdot)$ denote the learned mean and variance of the reverse transition. By iteratively applying the denoising process from timestep $T$ to $0$, diffusion models can generate high-quality samples from random noise.
During training, diffusion models mainly focus on recovering the original data distribution, which can be regarded as a supervised learning objective. Similar to SFT in LLMs, such training paradigms optimize models toward the provided training data but do not explicitly consider human preferences. As a result, diffusion models may generate high-quality samples while still failing to satisfy fine-grained user requirements. For example, as described in \citep{lee-etal:aligning}, although text-to-image diffusion models can generate visually realistic images, they may struggle with specific attributes, such as generating a desired number of objects. Therefore, how to incorporate additional learning objectives to better align diffusion models with human preferences has become an important research topic.
To address this challenge, researchers have explored RL as an effective approach for aligning diffusion models with human preferences. A straightforward approach is to treat generated samples as optimization targets and use external reward signals to guide the generation process. Specifically, given a text prompt $\mathbf{z}$, a reward function $R_\mathrm{dm}(\mathbf{x}_0,\mathbf{z})$ evaluates the generated image $\mathbf{x}_0$, and the diffusion model is optimized to maximize the expected reward:
where $p_\theta$ denotes the learned reverse denoising process used for generation, $\mu_\theta(\cdot)$ and $\Sigma_\theta(\cdot)$ denote the learned mean and variance of the reverse transition. By iteratively applying the denoising process from timestep $T$ to $0$, diffusion models can generate high-quality samples from random noise.
Here, the reward function can be instantiated by different types of evaluators depending on the optimization objective. For example, it can measure semantic alignment between the generated image and the text prompt using a vision-language model or measure human preferences through a learned reward model. For example, given a prompt requiring ``three red apples on a table'', a reward function can assess whether the generated image contains the correct object number and color:
% DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models
% Training Diffusion Models with Reinforcement Learning
% Diffusion models can be directly optimized with RL without human annotation.
However, directly optimizing this objective with the RL formulation introduced in Section~\ref{sec:policy-gradient} is not straightforward. This is because that diffusion models generate samples through an iterative denoising process rather than an autoregressive generation process. Therefore, we need to redefine the RL formulation according to the characteristics of diffusion generation. Taking DPOK \citep{fan-etal:dpok} as an example, recent studies observe that the reverse diffusion process naturally forms a multi-step trajectory, where each denoising step can be viewed as an action conditioned on the current noisy state. Based on this observation, the diffusion generation process can be formulated as a Markov decision process. Specifically, we can consider the denoising procedure as a multi-step MDP and applies a policy gradient-based RL algorithm to optimize the reward obtained from generated images. The state corresponds to the current noisy latent representation, while the action represents the next denoising step:
\begin{eqnarray}
s_t=(\mathbf{z},\mathbf{x}_{T-t}),
\quad
a_t=\mathbf{x}_{T-t-1}
\end{eqnarray}
The policy is defined as the reverse diffusion transition:
where the final generated image receives the reward signal from the reward model. Based on this formulation, many RL techniques developed for LLM alignment can be naturally extended to diffusion model optimization, such as introducing a reference diffusion model for importance sampling \citep{black-etal:training} and applying GRPO \citep{xue-etal:dancegrpo}.
