\newblock From word to world: Can large language models be implicit text-based world models?
\newblock In \emph{Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 8084--8111, 2026.
\bibitem[Li et~al.(2025{\natexlab{b}})Li, Hu, and Wang]{li-etal:encouraging}
\bibitem[Li et~al.(2025{\natexlab{b}})Li, Hu, and Wang]{li-etal:encouraging}
Zhiwei Li, Yong Hu, and Wenqing Wang.
Zhiwei Li, Yong Hu, and Wenqing Wang.
\newblock Encouraging good processes without the need for good answers: Reinforcement learning for llm agent planning.
\newblock Encouraging good processes without the need for good answers: Reinforcement learning for llm agent planning.
title={From word to world: Can large language models be implicit text-based world models?},
author={Li, Yixia and Wang, Hongru and Qiu, Jiahao and Yin, Zhenfei and Zhang, Dongdong and Qian, Cheng and Li, Zeping and Ma, Xiaoteng and Chen, Guanhua and Ji, Heng},
booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
where $p(\mathcal{E})$ denotes the environment distribution, and $p_e(\mathcal{Q})$ denotes the task distribution within environment $e$. The interaction trajectory $\tau$ follows the definition introduced in the previous subsections. This objective shows that agent performance depends not only on policy optimization, but also on the coverage and quality of the environments used for training.
where $p(\mathcal{E})$ denotes the environment distribution, and $p_e(\mathcal{Q})$ denotes the task distribution within environment $e$. The interaction trajectory $\tau$ follows the definition introduced in the previous subsections. This objective shows that agent performance depends not only on policy optimization, but also on the coverage and quality of the environments used for training.
However, constructing high-quality environments manually is expensive and difficult to scale. Real-world systems may be inaccessible, costly, or unsafe for large-scale exploration. Purely language-based simulations are easier to build, but they may generate inconsistent state transitions, invalid tool outputs, or unreliable evaluations. Thus, recent studies explore procedural and programmatic environment synthesis, where executable programs and structured states are used to provide a scalable and reliable interaction experience.
However, constructing high-quality environments manually is expensive and difficult to scale. Real-world systems may be inaccessible and costly. Purely language-based simulations are easier to build, but they may generate inconsistent state transitions \citep{li-etal:word2world}. Thus, recent studies explore procedural and programmatic environment synthesis, where executable programs and structured states are used to provide a scalable and reliable interaction experience.
\begin{figure}[t!]
\begin{figure}[t!]
\centering
\centering
...
@@ -302,9 +302,9 @@ Scaling environments also introduces substantial heterogeneity. Different enviro
...
@@ -302,9 +302,9 @@ Scaling environments also introduces substantial heterogeneity. Different enviro
\subsection{Learning from Agentic Experience}
\subsection{Learning from Agentic Experience}
\label{sec:learning_from_agentic_experience}
\label{sec:learning_from_agentic_experience}
AI systems are gradually moving from an era dominated by human-generated data toward an era in which agents increasingly learn from their own experience \citep{sutton-etal:welcome}. As discussed in Section~\ref{sec:building-agent-capabilities}, supervised data remains useful for teaching agents basic behaviors, such as planning, tool invocation, and response formatting. However, human demonstrations alone are unlikely to cover the full range of situations that an agent may encounter in complex environments.
AI systems are gradually moving from an era dominated by human-generated data toward an era in which agents increasingly learn from their own experience \citep{sutton-etal:welcome}. As discussed in Section~\ref{sec:building-agent-capabilities}, supervised data remains useful for teaching agents basic behaviors, such as planning, tool invocation, and response formatting. However, human demonstrations alone are unlikely to cover the full range of situations that an agent may encounter in complex and dynamic environments. One promising direction is to enable agents to learn from their own interaction experience. Through continuous interaction with environments, agents can collect successful and failed trajectories, and improve their future behaviors via these trajectories. In recent literature, this process is often referred to as \textit{agent self-evolution}, where agents continuously enhance their capabilities by leveraging accumulated experience. Here we discuss three commonly used approaches for learning from agentic experience.
The first approach is memory management. An agent can store useful information from previous interactions, such as successful solutions and failure patterns. When facing a new but related task, the agent can retrieve relevant memories and use them to guide its decisions. In this way, the agent does not need to solve every problem from scratch and can gradually accumulate knowledge across interactions.
The first approach is memory management though agentic experience. In practice, an agent can store useful information from previous interactions. When facing a new but related task, the agent can retrieve relevant memories and use them as additional context to guide its decisions. From the perspective of in-context learning, this process can also be viewed as a form of \textit{learning}. In this way, a straightforward approach to learn from experience is to update the memory based on interaction outcomes. For example, given a task, the agent can store both successful and failed trajectories in its memory system. When solving similar tasks in the future, the agent can retrieve these experiences, reuse effective strategies, and avoid previous mistakes, thereby improving task performance. This approach is usually simple and does not require additional training of the agent.
The second approach is skill optimization. Individual trajectories often contain reusable procedures, such as how to search for information, recover from an invalid tool call, or complete a common sequence of operations. The agent can abstract these procedures into higher-level skills and reuse them in future tasks. Skill optimization therefore converts low-level interaction experience into more general and efficient behaviors.
The second approach is skill optimization. Individual trajectories often contain reusable procedures, such as how to search for information, recover from an invalid tool call, or complete a common sequence of operations. The agent can abstract these procedures into higher-level skills and reuse them in future tasks. Skill optimization therefore converts low-level interaction experience into more general and efficient behaviors.
...
@@ -313,7 +313,7 @@ The third approach is trajectory refinement. Not all collected trajectories are
...
@@ -313,7 +313,7 @@ The third approach is trajectory refinement. Not all collected trajectories are
\subsubsection{Memory Management}
\subsubsection{Memory Management}
However, this approach introduces a key challenge in memory management: deciding when and what information should be updated in memory.
...
@@ -331,7 +331,7 @@ The third approach is trajectory refinement. Not all collected trajectories are
...
@@ -331,7 +331,7 @@ The third approach is trajectory refinement. Not all collected trajectories are
