
\section{Multimodal Reinforcement Learning}

Multimodal learning aims to build models that can process and integrate information from multiple modalities, such as vision and text. By combining complementary signals from different modalities, multimodal models can achieve more comprehensive perception and reasoning abilities than single-modality systems. These capabilities have enabled a wide range of applications, including image captioning, visual reasoning, speech understanding, and image generation \citep{baltruvsaitis-etal:2018multimodal}.

With the rapid development of LLMs, a natural direction is to extend their powerful reasoning and generation capabilities to multimodal domains. For example, visual language models can be constructed by aligning LLMs with visual encoders and training them on large-scale image-text pairs through supervised fine-tuning. However, extending LLM capabilities to new modalities introduces additional alignment challenges. Although an LLM may achieve strong alignment with human preferences in the language domain, such alignment does not necessarily transfer to other modalities. For example, a model may generate helpful textual responses while still producing visually inconsistent or misaligned outputs. Therefore, additional alignment strategies are required to optimize multimodal models according to modality-specific objectives.

RL provides a natural framework for addressing this challenge by optimizing multimodal models based on task-specific feedback signals. Compared with supervised fine-tuning, RL can directly optimize high-level objectives, such as human preferences, visual quality, and semantic consistency. In this section, we discuss the application of RL in different types of multimodal models. We first introduce RL for multimodal understanding models, where the output remains primarily textual but the input involves multiple modalities. We then discuss RL for multimodal generation models, including diffusion-based and flow matching-based models, where RL is used to optimize generated content quality.

\subsection{Multimodal Understanding Models}

Multimodal models can be roughly divided according to whether they generate non-textual content or use non-textual content as input. In this subsection, we focus on the latter case, namely multimodal understanding models. Although these models may process images, videos, audio, or other non-textual signals, their output is still usually a textual response. Therefore, from the perspective of RL training, they can still be viewed as conditional text generation models. The main difference from standard LLMs is that the condition now contains multimodal information in addition to a textual instruction. Once these inputs are encoded and passed into the LLM, the subsequent policy modeling, reward modeling, and RL training are largely consistent with the methods discussed in Section~\ref{sec:example-using-rl-training-llms}.

A representative example is the multimodal language model\footnote{A multimodal language model is defined as a model that integrates an LLM with a multimodal encoder, such as CLIP \citep{radford-etal:2021learning}, allowing the LLM to process inputs beyond text, such as images. Recent literature has also introduced the use of LLMs for generating outputs in non-text modalities, such as images and speech \citep{xu-etal:2025qwen2,zhang-etal:2024mm}. However, in this section, we focus on the former definition of multimodal language models.}, especially the Visual Language Model (VLM). A VLM typically connects a visual encoder to an LLM through a linear projector or related alignment module, enabling the LLM to perform visual and language understanding. VLMs are currently the most explored extension in multimodal language research, and they form the foundation for many open-source multimodal language models, such as Qwen2.5-VL \citep{qwenTeam:2025qwen2.5-VL} and LLaMA-3.2-11B-Vision \citep{grattafiori-etal:2024llama}. For a VLM, the input usually consists of one or multiple images and a textual instruction, denoted by $(\mathrm{\mathbf{I}}, \mathrm{\mathbf{x}})$, where $\mathrm{\mathbf{I}}$ represents the input images. These images are encoded into visual representations that are either concatenated with instruction embeddings or integrated into the LLM through cross-attention mechanisms. Since the model still generates a textual output $\mathbf{y}$, the RL training process for LLMs can be adapted to train VLMs with only minor changes \citep{yu-etal:2024rlhf,wang-etal:2024rovrm,zang2025internlm,ji2025safe}.

\begin{figure*}[!t]
    \centering
    %% \includegraphics[width=0.98\textwidth]{section7/Figures/rovrm.pdf}
    \input{section7/Figures/rovrm.tex}
    \caption{An overview of the multi-stage training approach for visual reward models.}
    \label{fig:preference-transfer}
\end{figure*}

Although the overall RL formulation can be reused, training VLMs with RL is still challenging in practice. The central difficulty is not the policy optimization algorithm itself, but the scarcity of high-quality multimodal preference data. Compared with textual preference data, visual preference data is more expensive to collect, harder to verify, and often more task-dependent. This makes it difficult to train a reliable visual reward model, which is a key component for applying RLHF-style methods to VLMs.

One straightforward way to alleviate this problem is to generate visual preference data through the automatic preference data generation method described in Section \ref{sec:automatic-preference-data-generation} \citep{yu-etal:2024rlaif}. Another alternative is a multi-stage training approach for the visual reward model, motivated by a simple idea: human preferences are already well captured in text, and these preferences can be transferred across modalities \citep{wang-etal:2024rovrm}. By leveraging textual preference data, this approach reduces the dependence on large-scale visual preference data when training a visual reward model. More specifically, as illustrated in Figure \ref{fig:preference-transfer}, we can train a visual reward model in the following three stages:
\begin{itemize}
    \item Stage 1: pre-training with large-scale textual preference data. Given the transferability of human preferences across different modalities, we can begin by using large-scale textual preference data to pre-train the visual reward model. Note that the projector parameters are frozen without images. This stage can be considered as providing a stronger starting point for training the visual reward model, as it enables the model to pre-learn general human preferences.
    \item Stage 2: fine-tuning with image caption-based preference data. Pre-learned human preferences cannot be directly applied to vision tasks due to both \textit{task gap} and \textit{modality gap}. To bridge the task gap, we fine-tune the reward model using image caption-based preference data in this stage. The rationale behind this approach is that general textual preference data does not cover vision-specific tasks, such as ``Please describe the content in this image''. We use such data to fine-tune the model, adapting it to vision-specific tasks. The projector parameters are frozen at this stage.
    \item Stage 3: fine-tuning with small-scale visual preference data. To further bridge the modality gap, we use visual preference data in the final stage to train the model. Note that in this phase, we also train the projector parameters.
\end{itemize}

In this process, not all preference data may align with the preferences used in subsequent stages, potentially leading to preference conflicts. To address this issue, we can enhance the visual reward model through data selection techniques, such as LESS \citep{xia-etal:2024less}. In fact, preference transfer is effective across modalities and has also been shown to work across different tasks and languages \citep{cheng-etal:2023everyone,wu-etal:2024reuse}. Interested readers can refer to these papers for more detailed discussions of these topics.

As discussed in Section~\ref{sec:generative-reward-models}, reward reasoning models have demonstrated superior performance in reward prediction. A natural question then arises: \textit{can reward reasoning capabilities also be transferred from text to multimodal settings?} Recent work by \cite{wang2026msrl} provides empirical evidence supporting this hypothesis. By following a similar multi-stage training paradigm, they show that reward reasoning capabilities can indeed be effectively transferred across modalities, further enhancing the performance of visual reward models. 



Apart from constructing better preference data, another approach to improving the visual reward model is to integrate additional image content, such as image captions, into reward prediction \citep{sun-etal:2023aligning}. This approach aims to achieve factually augmented reward prediction and reduce reward hacking. Specifically, in the original setup, the reward model predicts a reward based only on the image, input, and output; that is, the reward model's input is $[\mathbf{I}, \mathbf{x}, \mathbf{y}]$. In the factually augmented setup, the reward model also receives an additional textual image caption $\mathbf{C}$, resulting in an input of $[\mathbf{I}, \mathbf{C}, \mathbf{x}, \mathbf{y}]$. The basic idea is that the backbone of the visual reward model remains a well-trained LLM with in-context learning ability. By providing additional factual context about the image, we can help the reward model make more accurate and grounded reward predictions.

While our discussion primarily focuses on models for visual inputs in the context of visual language models, the RL techniques are adaptable across inputs of various modalities, such as video and audio.


\subsection{Multimodal Generation Models}

Unlike multimodal understanding models that typically produce textual responses, multimodal generation models aim to synthesize new content, such as images and videos. Recent advances in generative models, including diffusion models and flow matching models, have enabled high-quality content generation by learning complex data distributions. However, researchers have found that optimizing these models remains challenging because conventional maximum likelihood or reconstruction objectives cannot fully capture high-level human preferences, such as satisfying specific requirements (e.g., object quantities, colors, and visual layouts). To address this challenge, RL has been introduced to directly optimize multimodal generation models based on flexible reward signals \citep{lee-etal:aligning,fan-etal:dpok}. In this way, instead of relying solely on token-level or pixel-level reconstruction objectives, RL allows models to optimize task-specific criteria provided by humans or automatic evaluators. This property makes RL particularly suitable for multimodal generation, where generation quality is often difficult to describe through explicit supervision.

In this subsection, we describe the application of RL in two representative classes of multimodal generation models: diffusion models and flow matching models. 

\subsubsection{Diffusion Models}

We first briefly introduce the basic generation process of diffusion-based models to provide the necessary notations. There are many aspects of diffusion models, such as image representations and mathematical formulations, that cannot be fully covered in this paper. Interested readers can refer to existing surveys on diffusion models for further details \citep{yang-etal:diffusion,croitoru-etal:diffusion}.

Specifically, diffusion models generate samples through a gradual denoising process \citep{sohl-etal:deep}. Given a clean sample $\mathbf{x}_0$, the forward diffusion process progressively adds Gaussian noise to the sample. At each timestep $t$, the transition distribution is defined as:
\begin{eqnarray}
Q(\mathbf{x}_t|\mathbf{x}_{t-1}) = \mathcal{N} (\mathbf{x}_t;\sqrt{1-\beta_t}\mathbf{x}_{t-1},\beta_t\mathbf{I})
\end{eqnarray}

where $Q(\cdot)$ denotes the predefined forward diffusion process, $\beta_t$ controls the noise magnitude at timestep $t$, and $\mathcal{N}(\mu,\sigma^2)$ denotes a Gaussian distribution with mean $\mu$ and variance $\sigma^2$. Through this forward process, the original data distribution is gradually transformed into a simple Gaussian noise distribution.

The generation process performs the reverse denoising procedure, where a neural network learns to recover clean samples from noisy inputs:
\begin{eqnarray}
U_\theta(\mathbf{x}_{t-1}|\mathbf{x}_{t}) = \mathcal{N} (\mathbf{x}_{t-1}; \mu_\theta(\mathbf{x}_t,t), 
\Sigma_\theta(\mathbf{x}_t,t))
\end{eqnarray}

where $U_\theta(\cdot)$ denotes the learned reverse denoising process used for generation, and $\mu_\theta(\cdot)$ and $\Sigma_\theta(\cdot)$ denote the learned mean and variance of the reverse transition. By iteratively applying the denoising process from timestep $T$ to $0$, diffusion models can generate high-quality samples from random noise.

During training, diffusion models mainly focus on recovering the original data distribution, which can be regarded as a supervised learning objective. Similar to SFT in LLMs, such training paradigms optimize models toward the provided training data but do not explicitly consider human preferences. As a result, diffusion models may generate high-quality samples while still failing to satisfy fine-grained user requirements. For example, as described in \citet{lee-etal:aligning}, although text-to-image diffusion models can generate visually realistic images, they may struggle with specific attributes, such as generating a desired number of objects. Therefore, how to incorporate additional learning objectives to better align diffusion models with human preferences has become an important research topic.

To address this challenge, researchers have explored RL as an effective approach for aligning diffusion models with human preferences. A straightforward approach is to treat generated samples as optimization targets and use external reward signals to guide the generation process. Specifically, given a text prompt $\mathbf{z}$, a reward function $R_\mathrm{dm}(\mathbf{x}_0,\mathbf{z})$ evaluates the generated image $\mathbf{x}_0$, and the diffusion model is optimized to maximize the expected reward:
\begin{eqnarray}
\max_{\theta}
\mathbb{E}_{\mathbf{z}\sim p(\mathbf{z}),\mathbf{x}_0\sim p_\theta(\mathbf{x}_0|\mathbf{z})}
[R_\mathrm{dm}(\mathbf{x}_0,\mathbf{z})]
\label{eq:optimization_objective}
\end{eqnarray}

Here, the reward function can be instantiated by different types of evaluators depending on the optimization objective. For example, it can measure semantic alignment between the generated image and the text prompt using a vision-language model or measure human preferences through a learned reward model. For example, given a prompt requiring ``three red apples on a table'', a reward function can assess whether the generated image contains the correct object number and color:

However, directly optimizing this objective with the RL formulation introduced in Section~\ref{sec:policy-gradient} is not straightforward. This is because that diffusion models generate samples through an iterative denoising process rather than an autoregressive generation process. Therefore, we need to redefine the RL formulation according to the characteristics of diffusion generation. Taking DPOK \citep{fan-etal:dpok} as an example, recent studies observe that the reverse diffusion process naturally forms a multi-step trajectory, where each denoising step can be viewed as an action conditioned on the current noisy state. Based on this observation, the diffusion generation process can be formulated as a MDP. Specifically, we can consider the denoising procedure as a multi-step MDP and applies a policy gradient-based RL algorithm to optimize the reward obtained from generated images. The state corresponds to the current noisy latent representation, while the action represents the next denoising step:
\begin{eqnarray}
s_t=(\mathbf{z},\mathbf{x}_{T-t}), \quad a_t=\mathbf{x}_{T-t-1}
\end{eqnarray}

The policy is defined as the reverse diffusion transition:
\begin{eqnarray}
\mathrm{Pr}_\theta(a_t|s_t) = U_\theta(\mathbf{x}_{T-t-1}|\mathbf{x}_{T-t},\mathbf{z})
\end{eqnarray}

where the final generated image receives the reward signal from the reward model. Based on this formulation, the optimization objective in Eq.~(\ref{eq:optimization_objective}) can be optimized using a simple policy gradient loss function:
\begin{eqnarray}
\mathcal{L}_{\mathrm{dmrl}}(\theta) =
-\mathbb{E}_{\mathbf{z}\sim S_{z},\tau_\mathrm{dm}\sim \mathrm{Pr}_\theta(\cdot)}
\left[
\sum_{t=0}^{T-1}
\log U_\theta(\mathbf{x}_{T-t-1}|\mathbf{x}_{T-t},\mathbf{z}) R_{\mathrm{dm}}(\mathbf{x}_0,\mathbf{z})
\right]
\end{eqnarray}

where $S_{z}$ denotes the training set, $\tau_\mathrm{dm}$ denotes the denoising trajectory $\{\mathbf{x}_{T},\mathbf{x}_{T-1},\cdots,\mathbf{x}_{0}\}$. Here, many RL techniques developed for LLM alignment can be naturally extended to diffusion model optimization, such as incorporating KL regularization \citep{fan-etal:dpok}, introducing a reference diffusion model for importance sampling \citep{black-etal:training}, and applying GRPO-based optimization \citep{xue-etal:dancegrpo}.

\subsubsection{Flow Matching Models}

Different from diffusion models, flow matching models achieve the multimodal generation process based on ordinary differential equations (ODEs), where data transformation is modeled as a continuous-time flow. Instead of gradually adding and removing noise through a stochastic diffusion process, flow matching learns a continuous vector field that transports samples from a simple prior distribution to the target data distribution. In this subsection, we briefly introduce the training and generation processes of flow matching models, and then discuss how RL can be applied to optimize them. Note that we do not provide a detailed introduction to the underlying principles of flow matching models in this subsection. Interested readers can refer to existing tutorials for further details \citep{xiao-etal:ordinary}.

Let $\mathbf{x}_0 \sim X_0$ denote a data sample from the target distribution and $\mathbf{x}_1 \sim X_1$ denote a noise sample from the prior distribution. Flow matching constructs an intermediate state by interpolating between the data and noise distributions:
\begin{eqnarray}
\mathbf{x}_t=(1-t)\mathbf{x}_0+t\mathbf{x}_1
\end{eqnarray}

where $t\in[0,1]$ denotes the continuous time variable. Based on this interpolation process, flow matching trains a neural network to predict the velocity field that describes how samples move along the trajectory:
\begin{eqnarray}
\mathcal{L}_{\mathrm{fm}}(\theta)
= \mathbb{E}_{t,\mathbf{x}_0 \sim X_0 ,\mathbf{x}_1\sim X_1}
\left[
\left\|
\mathbf{v}_\theta(\mathbf{x}_t,t)-\mathbf{v}
\right\|^2
\right]
\end{eqnarray}

where $\mathbf{v}=\mathbf{x}_1-\mathbf{x}_0$ denotes the target velocity field and $\mathbf{v}_\theta(\mathbf{x}_t,t)$ denotes the learned velocity function. After training, the generation process starts from a noise sample $\mathbf{x}_1$ and follows the learned continuous flow by solving the ordinary differential equation:
\begin{eqnarray}
\frac{d\mathbf{x}_t}{dt} = \mathbf{v}_\theta(\mathbf{x}_t,t)
\end{eqnarray}

Similar to diffusion models, the continuous generation process of flow matching models can also be formulated as an MDP to enable RL optimization. Specifically, by discretizing the continuous flow trajectory, we can consider the generation process  as a sequence of decision steps, where the model determines how to transform the current state toward the target data distribution \citep{liu-etal:flow}. Under this formulation, the state, action, transition, initial state distribution, and reward function are defined as follows. The state at timestep $t$ is defined as:
\begin{eqnarray}
s_t=(\mathbf{z},t,\mathbf{x}_t)
\end{eqnarray}

The action corresponds to the next state predicted by the flow model:
\begin{eqnarray}
a_t=\mathbf{x}_{t-\Delta t}
\end{eqnarray}

Since the flow model deterministically predicts the velocity field, the policy can be represented as:
\begin{eqnarray}
\mathrm{Pr}_{\theta}(a_t|s_t) = \delta(a_t-\mathbf{v}_{\theta}(\mathbf{x}_t,t,\mathbf{z}))
\end{eqnarray}

where $\delta(\cdot)$ denotes the Dirac delta distribution, indicating that the next state is deterministically determined by the current state and the learned velocity field.
However, this deterministic formulation introduces challenges for RL optimization because policy gradient methods typically require stochastic sampling to explore different trajectories. To address this issue, Flow-GRPO converts the original ODE-based flow matching model into an equivalent stochastic differential equation (SDE), which preserves the marginal distribution while introducing sampling diversity \citep{liu-etal:flow}. Under this formulation, the transition probability becomes:
\begin{eqnarray}
\mathrm{Pr}_\theta(a_t|s_t) = \mathcal{N} (a_t;
\mu_\theta(\mathbf{x}_t,t,\mathbf{z}),
\sigma_t^2\mathbf{I})
\end{eqnarray}

where $\mu_\theta(\cdot)$ is determined by the learned velocity field and $\sigma_t$ controls the stochasticity during sampling. Based on this stochastic MDP formulation, RL algorithms can be applied to optimize flow matching models. Taking GRPO as an example, given a text condition $\mathbf{z}$, the flow model samples a group of generation trajectories $\{\tau^i_{\mathrm{fm}}\}_{i=1}^{G}$, where each trajectory represents a complete sampling process from the initial noise to the final generated sample. We evaluate the final generated sample of each trajectory using the reward function and obtain a group of rewards: $\mathbf{R}_{\mathrm{fm}} = \{R_{\mathrm{fm}}(\mathbf{x}_0^1,\mathbf{z}), R_{\mathrm{fm}}(\mathbf{x}_0^2,\mathbf{z}), \cdots, R_{\mathrm{fm}}(\mathbf{x}_0^G,\mathbf{z})\}$, where $R_{\mathrm{fm}}$ denotes the reward function for evaluating the generated sample. Similar to diffusion models, the reward function can be instantiated with different types of evaluators depending on the optimization objective, such as vision-language models for measuring text-image alignment. The advantage of each trajectory is then estimated based on its relative reward within the sampled group:
\begin{eqnarray}
\hat{A}_i= \frac{R_{\mathrm{fm}}(\mathbf{x}_0^i,\mathbf{z}) - 
\mathrm{mean}(\mathbf{R}_{\mathrm{fm}})}
{\mathrm{std}(\mathbf{R}_{\mathrm{fm}})}
\end{eqnarray}

Based on these estimated advantages, we formulate the GRPO objective as follows:
\begin{eqnarray}
\mathcal{L}_{\mathrm{fmgrpo}}(\theta) = -\mathbb{E}_{\mathbf{z}\sim S_{z}, \tau_{\mathrm{fm}}\sim \mathrm{Pr}(\cdot)}
\frac{1}{G} \sum_{i=1}^{G} \left[ \frac{1}{T} \sum_{t} \min
(\frac{\mathrm{Pr}_\theta(a_t^i|s_t^i)}{\mathrm{Pr}_{\theta_{\mathrm{old}}}(a_t^i|s_t^i)}\hat{A}_i,
\mathrm{clip}(\frac{\mathrm{Pr}_\theta(a_t^i|s_t^i)}{\mathrm{Pr}_{\theta_{\mathrm{old}}}(a_t^i|s_t^i)},1-\epsilon,1+\epsilon)\hat{A}_i)\right]
\end{eqnarray}

During training, trajectory sampling introduces a unique challenge for flow matching models. Diffusion models naturally support stochastic sampling because Gaussian noise is progressively involved in the generation process. In contrast, flow matching models typically generate samples through deterministic ODEs, where the intermediate states are uniquely determined once the initial noise is given. This deterministic process creates two difficulties for online RL: it makes the transition probability required by policy-gradient optimization difficult to compute, and, more importantly, provides limited randomness for policy exploration.
To address these issues, we can convert the deterministic ODE into an equivalent stochastic differential equation (SDE) that preserves the marginal distributions while introducing stochasticity into the generation process \citep{liu-etal:flow}. The resulting sampling process can be generally written as:
\begin{eqnarray}
d\mathbf{x}_t = \mathbf{v}^d_\theta(\mathbf{x}_t,t,\mathbf{z})dt +
\sigma_t d\mathbf{w}
\end{eqnarray}

where $\mathbf{v}^d_\theta(\cdot)$ is determined by the learned velocity field and $d\mathbf{w}$ denotes the Wiener process increment. Here, the drift function of the converted SDE can be constructed from the original flow velocity field as:
\begin{eqnarray}
\mathbf{v}^{d}_{\theta}(\mathbf{x}_t,t,\mathbf{z}) =
\mathbf{v}_{\theta}(\mathbf{x}_t,t,\mathbf{z}) +
\frac{\sigma_t^2}{2t}
\left[
\mathbf{x}_t + (1-t)\mathbf{v}_{\theta}(\mathbf{x}_t,t,\mathbf{z})
\right]
\end{eqnarray}

where $\mathbf{v}_{\theta}(\cdot)$ denotes the original velocity field and $\mathbf{v}^{d}_{\theta}(\cdot)$ denotes the drift function after the ODE-to-SDE conversion. Therefore, the stochastic generation process can be written as:
\begin{eqnarray}
d\mathbf{x}_t = \mathbf{v}^{d}_{\theta}(\mathbf{x}_t,t,\mathbf{z})dt +
\sigma_t d\mathbf{w}
\end{eqnarray}

In practice, we discretize the above SDE for trajectory sampling. Using Euler--Maruyama discretization, each sampling step can be written as:
\begin{eqnarray}
\mathbf{x}_{t+\Delta t} = \mathbf{x}_t + \mathbf{v}^{d}_{\theta}(\mathbf{x}_t,t,\mathbf{z})\Delta t +
\sigma_t\sqrt{\Delta t}\boldsymbol{\epsilon}
\quad
\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})
\end{eqnarray}

The last term injects stochasticity into each generation step, allowing the model to sample diverse trajectories from the same condition and thereby enabling effective exploration for online RL. After discretization, each generation step therefore corresponds to a stochastic Gaussian transition, allowing the model to sample diverse trajectories and compute the transition probabilities required by GRPO. 
