Algorithms

The algorithm examples show complete PPO and A2C configurations through r2l. They are useful when you want a runnable training loop before customizing lower-level hooks or samplers.

PPO

The PPO example trains on Pendulum-v1, writes training artifacts, reloads the best policy through InferenceRunner, and runs rendered inference episodes.

Run it from the workspace root:

cargo run -p r2l-examples --example ppo

use r2l::{Error, GymEnv, InferenceRunner, PPOBuilder, TrainingArtifactsConfig, TrainingLimit};

fn main() -> Result<(), Error> {
    const ENV_NAME: &str = "Pendulum-v1";

    // Path where the training artifacts are going to be stored. Training artifacts could include:
    // - Parameters of the model that was trained + the weights as a safetensor and the optional obs normalizer serialized
    // - Measurements on how the trained model perfoms after each training round
    // - Measurements on how long parts of the trainig run took
    const ARTIFACT_DIR: &str = "runs/pendulum";
    let artifacts_config = TrainingArtifactsConfig::new(ARTIFACT_DIR);

    // An environmnet builder how environments are to be constructed. Environment construction can
    // be elaborate (especially when working with external dependencies), so r2l opts to not pass
    // the environment directly (the environment would have to be Send for multi sampling), but
    // instead accepts anything that implements the `EnvBuilder` trait. Simplest example is just a
    // function/closure that returns the Env.
    let env_builder = || GymEnv::new(ENV_NAME, None);

    // The algorightm is constructed through a PPOBuilder. For A2C, the A2Cbuilder would be
    // equivalent. Builders expose a lot of common parameter setters. To check all the options, you
    // check https://docs.rs/r2l/latest/r2l/type.PPOBuilder.html.
    let mut ppo = PPOBuilder::new(env_builder, 10)?
        .with_training_artifacts(artifacts_config)
        .with_policy_hidden_layers(vec![64, 64])
        .with_lambda(0.95)
        .with_gamma(0.9)
        .with_learning_rate(0.001)
        .with_training_limit(TrainingLimit::rollouts(30))
        .build()?;

    // This kicks off and finishes training.
    ppo.train()?;

    // Once training in done, the training artifacts as serialized. You can reuse the trained model
    // by constructing an InferenceRunner. InferenceRunner can single step or run episodes on the
    // environment it recieves.
    let env = GymEnv::new(ENV_NAME, Some("human".to_owned()))?;
    let mut inference = InferenceRunner::load_from_env(ARTIFACT_DIR, env)?;
    for _ in 0..10 {
        inference.run_episode()?;
    }

    Ok(())
}

The important pieces are with_training_artifacts, which writes actor.safetensors, inference.yaml, and metrics files, and InferenceRunner::load_from_env, which rebuilds the inference runner from those saved files.

A2C

The A2C example selects the Candle backend, configures rollout collection, and uses a reporter channel to observe training statistics.

Run it from the workspace root:

cargo run -p r2l-examples --example a2c

use std::{
    sync::mpsc::{self, Receiver, Sender},
    thread,
};

use candle_core::Device;
use r2l::{A2CBuilder, A2CRolloutStats, SamplerExecutionMode, TrainingLimit};

fn main() -> anyhow::Result<()> {
    let (update_tx, update_rx): (Sender<A2CRolloutStats>, Receiver<A2CRolloutStats>) =
        mpsc::channel();

    let a2c_builder = A2CBuilder::gym("Pendulum-v1", 10)?
        .with_candle(Device::Cpu)
        .with_seed(0)
        .with_entropy_coefficient(0.2)
        .with_gradient_clipping(Some(0.5))
        .with_rollout_steps(2048)
        .with_execution_mode(SamplerExecutionMode::SingleThreaded)
        .with_training_limit(TrainingLimit::rollouts(300))
        .with_rollout_reporter(Some(update_tx));
    let mut a2c = a2c_builder.build()?;
    let t = thread::spawn(move || {
        while let Ok(stats) = update_rx.recv() {
            println!("avg reward: {}", stats.average_reward);
        }
    });
    a2c.train()?;
    drop(a2c);
    t.join()
        .map_err(|_| anyhow::anyhow!("A2C reporter thread panicked"))?;
    Ok(())
}

Both PPO and A2C builders expose the same broad setup concepts: choose an environment builder, select a backend, set rollout bounds, configure a learning schedule, then call build() and train().