Skip to main content
Agentic RL adds a trajectory adapter around an RL loop. The adapter runs the agent and environment, records each policy generation, assigns the environment reward, and returns trainer-ready data. The RL loop still owns scheduling, grouping, advantages, optimization, and weight publication. The cookbook’s async_rl_loop works with agentic rollouts, but it does not require a particular agent framework, environment, session service, or history-reconciliation policy.

​
Correctness boundary

For each logical trajectory, the adapter should:
  • preserve the exact generated token IDs and their aligned log probabilities;
  • mask prompts, tool results, and environment observations out of the loss;
  • keep every segment under one reward, GRPO group member, and advantage;
  • define what happens when a later prompt is not an exact token append;
  • distinguish an environment outcome from a broken or incomplete trace.
Never silently truncate or positionally zip mismatched token, log-probability, loss-mask, or routing data. Either repair the trace using an explicit policy or drop it before training.

​
Choose a trajectory architecture

There is no required architecture. Common choices include: Choose and test the policy for your harness. A history rewrite caused by a subagent, retry, dynamic system field, or context management is not inherently an error, and should not be labeled as compaction without evidence.

​
Logical trajectories and physical segments

One rollout_fn call is one logical trajectory and one completion in its prompt group. It may return multiple physical RolloutSample segments when histories branch or token ancestry diverges. Those segments retain the same reward and advantage; they do not become extra GRPO completions. If branches share a generated prefix, train that prefix once and keep it as masked context on later branches. Treat retry and failure behavior as an algorithm decision. A malformed trace should normally be retried within a bounded budget and then dropped. Assigning reward zero is appropriate only when the environment defines the failure as a real task outcome, not as a substitute for missing or misaligned training data.

​
Start from an example

The Harbor + OpenCode example runs local Docker environments, records OpenCode model calls through a policy adapter, builds a per-attempt token tree, and splits non-append token histories. It is one integration pattern, not a required Harbor or OpenCode dependency. For the loop itself, read Cookbook: Reinforcement Learning. For detailed implementation choices, calibration checks, failure policy, and session/cache guidance, read the agentic RL skill reference.