Here, the reward function can be instantiated by different types of evaluators depending on the optimization objective. For example, it can measure semantic alignment between the generated image and the text prompt using a vision-language model or measure human preferences through a learned reward model. For example, given a prompt requiring ``three red apples on a table'', a reward function can assess whether the generated image contains the correct object number and color:
However, directly optimizing this objective with the RL formulation introduced in Section~\ref{sec:policy-gradient} is not straightforward. This is because that diffusion models generate samples through an iterative denoising process rather than an autoregressive generation process. Therefore, we need to redefine the RL formulation according to the characteristics of diffusion generation. Taking DPOK \citep{fan-etal:dpok} as an example, recent studies observe that the reverse diffusion process naturally forms a multi-step trajectory, where each denoising step can be viewed as an action conditioned on the current noisy state. Based on this observation, the diffusion generation process can be formulated as a Markov decision process. Specifically, we can consider the denoising procedure as a multi-step MDP and applies a policy gradient-based RL algorithm to optimize the reward obtained from generated images. The state corresponds to the current noisy latent representation, while the action represents the next denoising step:
However, directly optimizing this objective with the RL formulation introduced in Section~\ref{sec:policy-gradient} is not straightforward. This is because that diffusion models generate samples through an iterative denoising process rather than an autoregressive generation process. Therefore, we need to redefine the RL formulation according to the characteristics of diffusion generation. Taking DPOK \citep{fan-etal:dpok} as an example, recent studies observe that the reverse diffusion process naturally forms a multi-step trajectory, where each denoising step can be viewed as an action conditioned on the current noisy state. Based on this observation, the diffusion generation process can be formulated as a MDP. Specifically, we can consider the denoising procedure as a multi-step MDP and applies a policy gradient-based RL algorithm to optimize the reward obtained from generated images. The state corresponds to the current noisy latent representation, while the action represents the next denoising step:
\begin{eqnarray}
s_t=(\mathbf{z},\mathbf{x}_{T-t}),
\quad
...
...
@@ -96,7 +97,17 @@ The policy is defined as the reverse diffusion transition:
where the final generated image receives the reward signal from the reward model. Based on this formulation, many RL techniques developed for LLM alignment can be naturally extended to diffusion model optimization, such as introducing a reference diffusion model for importance sampling \citep{black-etal:training} and applying GRPO \citep{xue-etal:dancegrpo}.
where the final generated image receives the reward signal from the reward model. Based on this formulation, the optimization objective in Eq.~(\ref{eq:optimization_objective}) can be optimized using a simple policy gradient loss function:
where $\tau$ denotes the denoising trajectory $\{\mathbf{x}_{T},\mathbf{x}_{T-1},\cdots,\mathbf{x}_{0}\}$. Here, many RL techniques developed for LLM alignment can be naturally extended to diffusion model optimization, such as incorporating KL regularization \citep{fan-etal:dpok}, introducing a reference diffusion model for importance sampling \citep{black-etal:training}, and applying GRPO-based optimization \citep{xue-etal:dancegrpo}.
