Skip to main content
Agentic RL uses the same rollout and optimization contract described in Cookbook: Reinforcement Learning. The async RL recipe still owns scheduling, GRPO grouping, advantages, training, and policy publication. The agentic adapter runs the multi-turn harness and returns exact, loss-aligned rollout data.

​
Why multi-turn RL needs TITO

An agent harness operates on messages: it reads an assistant tool call, executes the tool, appends the result, and submits the next request. RL trains on the exact token IDs and rollout-policy log probabilities used by inference. For each policy turn, TITO (token-in/token-out) records:
The boundary matters because a sampled action must become a message before the harness can continue. Rendering that message on the next turn is not guaranteed to reproduce the same token IDs. Tool-argument JSON may be reserialized, empty content may change representation, model-specific stop/message boundaries may overlap, or the harness may compact or rewrite its history. Messages alone cannot prove what the rollout policy sampled. If an exact prompt diverges from the previous exact checkpoint, TITO keeps both sides valid by closing the old training segment and starting a new one from the prompt that actually produced the next action. The new prompt is masked context; the exact sampled action remains trainable. All segments from one rollout keep the same reward, GRPO group membership, and advantage.

​
Choose whether online behavior may change

Fireworks supports two TITO behaviors. The meaningful distinction is whether TITO changes the prompt used to sample the next trainable action. full_history continuity handling may run after the rollout if every call already recorded its actual prompt IDs, sampled IDs, and logprobs. Re-rendering messages later is not sufficient. incremental must run before inference because constructing a different prompt after an action was sampled cannot change what the model saw. Neither mode makes a genuine history rewrite—such as compaction, pruning, or a subagent handoff—continuous without changing its meaning.

​
Fireworks support

The public Fireworks Cookbook separates the environment, harness, and task recipe. They are independent parts of one rollout rather than one Harbor integration.

​
Overall RL environment: Harbor

Harbor is the trial and environment layer. It loads a task, starts its Docker container or E2B sandbox, runs the verifier, produces the reward and artifacts, and tears the environment down. In the reference integration, the TITO sidecar runs inside that sandbox beside the agent harness.

​
Supported harness adapters

The reference adapters support OpenCode, Pi, and Mini-SWE-Agent over the same Harbor/TITO contract.

​
Supported task datasets

The Cookbook includes support for DABstep, Terminal-Bench 2.0, and DeepSWE.

​
Example: Pi with DABstep

Use the Pi+DABstep training script as the complete reference entrypoint. The Cookbook also provides task preparation scripts for Terminal-Bench and DeepSWE. These task entrypoints plug into the async RL lifecycles described in Cookbook: Reinforcement Learning; this page does not repeat the dedicated and serverless execution choices. Both prompt-construction modes use the same TITO engine, exact-token sampler boundary, artifact format, and RolloutRun materializer. Prompt construction defaults to full_history; select incremental explicitly with --tito-prompt-mode incremental only for a renderer/tokenizer contract that qualifies the additional suffix and junction behavior. An ordinary SFT/DPO renderer does not automatically qualify a model for agentic TITO.

​
The sidecar is a reference placement

The Cookbook starts one lightweight TITO sidecar inside every agent sandbox:
This placement requires no user-operated middleware fleet, public callback, or central stateful gateway. It also keeps each trajectory’s state and failure domain with its sandbox. The TITO invariants do not require this placement: The Cookbook sidecar is the maintained reference implementation, not the only valid TITO architecture. A custom integration may use offline materialization or a centralized gateway as long as it preserves the exact prompt, action, logprob, loss-mask, and rollout-identity contracts.