Quickstart

Environment overview

Every benchmark task is registered as a Gymnasium environment. The base environment TwoRobot{Task}-v1 hosts levels L0, L1 and L2; each L3 variant is a standalone environment registered as TwoRobot{Task}L3-v1. Each environment contains two panda_wristcam agents — the action space is a dictionary keyed by panda_wristcam-0 (left) and panda_wristcam-1 (right).

L0 | TwoRobotStirSpoon-v1      L1 | TwoRobotStirSpoon-v1
L2 | TwoRobotStirSpoon-v1      L3 | TwoRobotStirSpoonL3-v1

Create & step an environment

Call configure_dual_task_level before gym.make. This resets all process-global level and mirror switches, then enables the requested level. Call it again whenever a process changes levels.

import gymnasium as gym
import mani_skill.envs  # Registers all benchmark environments.

from mani_skill.envs.tasks.tabletop.utils.dual_task_camera_utils import (
    configure_dual_task_level,
)

level = "L2"
base_env_id = "TwoRobotStirSpoon-v1"
env_id = base_env_id.replace("-v1", "L3-v1") if level == "L3" else base_env_id

configure_dual_task_level(level)
env = gym.make(
    env_id,
    obs_mode="state",
    control_mode="pd_joint_pos",
    reward_mode="dense",
    sim_backend="physx_cpu",
)

obs, info = env.reset(seed=0)
action = env.action_space.sample()
obs, reward, terminated, truncated, info = env.step(action)

evaluation = env.unwrapped.evaluate()
print(evaluation["success"])
env.close()

Sampling the environment action space returns the required two-arm structure; you can also write actions directly into the per-agent sub-spaces if you need asymmetric control.

Camera configuration

All 50 base environments and all 50 L3 environments expose two independent constructor arguments: hi_res (external scene cameras, default True) and wrist_sensor (two robot-mounted wrist cameras, default True).

Pass the options directly to gym.make. For example, a lightweight single-view configuration:

env = gym.make(
    "TwoRobotStirSpoon-v1",
    obs_mode="rgbd",
    render_mode="sensors",
    control_mode="pd_joint_pos",
    hi_res=False,
    wrist_sensor=False,
)

Evaluation & reward

Every task implements evaluate(). Its success field is the final rule-based simulator metric used for demonstration filtering and automated evaluation. Task-specific fields expose intermediate predicates such as reached, grasped, transported, placed or returned.

Dense rewards are built from ordered phases. RewardTracker keeps the peak completion of each phase and produces bounded progress signals. Use reward_mode="dense" for the phase reward, "sparse" for success-based reward, or "none" when only states/actions are being recorded.

Run a baseline

Every baseline has its own README with training and evaluation commands. A representative end-to-end run with ACT (frozen task-video configuration) looks like this:

export PYTHONPATH=$PWD:$PYTHONPATH
python -m examples.baselines.act.train_act_imitator \
  --human-root ./demos/demo_data \
  --sim-root ./demos/imitator_data \
  --human-dataset-file examples/baselines/lerobot_dataset/config/exp_configs/human_train_config_45.json \
  --sim-dataset-file examples/baselines/lerobot_dataset/config/exp_configs/sim_train_config_45.json \
  --task-mapping-file examples/baselines/lerobot_dataset/task_mapping.json \
  --input-mode video_only \
  --task-encoder-type frozen_backbone \
  --control-mode pd_joint_pos \
  --env-id TwoRobotPourCup-v1

Checkpoints go to runs/<run_name>/checkpoints/. Training requires LeRobot-format data — follow the data collection guide first if you don't have converted datasets yet. See the baselines page for all nine methods and their entry points.