Environment building

Algorithms are built from an EnvBuilder, not from a single environment instance. This lets samplers create one environment per worker and keeps environment state local to the worker that steps it.

The env_building example shows the supported construction styles:

• implementing Env for a custom environment;
• implementing EnvBuilder for a custom builder type;
• passing a closure or function that returns r2l_core::error::Result<E>;
• using GymEnvBuilder directly;
• using PPOBuilder::gym for Gymnasium environment ids.

Run it from the workspace root:

cargo run -p r2l-examples --example env_building

The full example is:

use r2l::{Env, EnvBuilder, EnvDescription, PPOBuilder, Snapshot, Space, VecTensor};
use r2l_core::error::Error;
use r2l_gym::GymEnvBuilder;

// Not a working implementation an actual env
pub struct MyEnv;

impl Env for MyEnv {
    type Tensor = VecTensor;

    fn reset(&mut self, _seed: u64) -> Result<Self::Tensor, Error> {
        Ok(VecTensor::new(vec![0., 0.], vec![2])?)
    }

    fn step(&mut self, _action: Self::Tensor) -> Result<Snapshot<Self::Tensor>, Error> {
        let state = VecTensor::new(vec![0., 0.], vec![2])?;
        let reward = 0.;
        let terminated = false;
        let truncated = false;
        let snapshot = Snapshot::new(state, reward, terminated, truncated);
        Ok(snapshot)
    }

    fn env_description(&self) -> EnvDescription<Self::Tensor> {
        let observation_space = Space::Box {
            min: None,
            max: None,
            shape: vec![2],
        };
        let action_space = Space::Discrete(2);
        EnvDescription::new(observation_space, action_space)
    }
}

struct MyEnvBuilder;

impl EnvBuilder for MyEnvBuilder {
    type Env = MyEnv;

    fn build_env(&self) -> Result<Self::Env, Error> {
        Ok(MyEnv)
    }
}

#[allow(clippy::unnecessary_wraps)]
fn build_env() -> Result<MyEnv, Error> {
    Ok(MyEnv)
}

fn main() -> anyhow::Result<()> {
    // Anything that implements Into<GymEnvBuilder> can be used with PPOBuilder::gym.
    // method. This includes &str, String and GymEnvBuilder itself (or your own implementation)
    let ppo_builder0 = PPOBuilder::gym("Pendulum-v1", 10)?;
    let _ppo0 = ppo_builder0.build()?;

    // Since GymEnvBuilder is an EnvBuilder, it can be used with PPOBuilder::new.
    let gym_env_builder = GymEnvBuilder::new("Pendulum-v1");
    let ppo_builder1 = PPOBuilder::new(gym_env_builder, 10)?;
    let _ppo1 = ppo_builder1.build()?;

    // This closure that returns an environment can be used as an environment builder
    let env_builder = || Ok(MyEnv);
    let ppo_builder = PPOBuilder::new(env_builder, 10)?;
    let _ppo = ppo_builder.build()?;

    // This function that returns an environment can also be used as an environment builder
    let ppo_builder3 = PPOBuilder::new(build_env, 10)?;
    let _ppo3 = ppo_builder3.build()?;

    // We can implement our own environment builder to be used with PPOBuilder::new.
    let ppo_builder4 = PPOBuilder::new(MyEnvBuilder, 10)?;
    let _ppo4 = ppo_builder4.build()?;
    Ok(())
}

For real environments, make sure env_description accurately describes the flattened observation and action spaces. The policy builder uses those spaces to choose the policy distribution and network dimensions.