Tutorial

Domain randomization

Two knobs control the visual and spatial diversity of every benchmark task: background texture randomization via TableSceneBuilder(..., random_background=True), and per-episode object-placement jitter via the xyz[:, :2] += rand * scale pattern in _initialize_episode.

Overview

Simulation makes it cheap to chop and change aspects of the scene that would be expensive in the real world. The Imitator Game task template exposes two ready-made domain-randomization hooks, both implemented in the shared mani_skill/utils/scene_builder/table/scene_builder.py and the task template at mani_skill/envs/tasks/_template/template_task.py:

Background textures

Randomized wall, table and ground textures sampled from the RoboTwin background-texture dataset, toggled with random_background=True.

Object placement

A uniform XY offset added to every object's spawn pose in _initialize_episode; the magnitude is the scale in xyz[:, :2] += torch.rand(...) * SCALE.

Robot init pose

Gaussian noise on the robot's initial joint configuration, controlled by robot_init_qpos_noise on TableSceneBuilder.

Background randomization

The background randomization is implemented by TableSceneBuilder.create_table_and_wall(...). When enabled, it samples random texture files for the wall, the table and the ground from the RoboTwin background-texture directory:

python

def create_table_and_wall(
        self,
        table_xy_bias=[0, 0],
        table_height=0.9,
        random_background=False,
        eval_mode=False
):
    ...
    if random_background:
        texture_type = "seen" if not eval_mode else "unseen"
        directory_path = str(Path.home() / ".maniskill" / "data" / "robotwin" / f"background_texture/{texture_type}")
        file_count = len([name for name in os.listdir(directory_path) ...])

        wall_texture   = np.random.randint(0, file_count)
        table_texture  = np.random.randint(0, file_count)
        ground_texture = np.random.randint(0, file_count)
        ...
    else:
        wall_texture, table_texture, ground_texture = None, None, None

Each of the three surfaces gets an independently sampled texture, so a single episode can mix different wall / table / ground appearances. The eval_mode flag switches the texture pool from the seen split to the unseen split, which is how you keep evaluation appearances out of the training distribution.

How to enable it

The TableSceneBuilder is constructed in your task's _load_scene. Set random_background=True in the constructor to turn the randomization on:

TableSceneBuilder(env, random_background=False, robot_init_qpos_noise=0.02)

random_background — enable RoboTwin texture randomization for the wall, table and ground. robot_init_qpos_noise — standard deviation of the Gaussian noise added to the robot's initial joint configuration at reset.

python

from mani_skill.utils.scene_builder.table import TableSceneBuilder

class TwoRobotMyTaskEnv(BaseEnv):
    def _load_scene(self, options: dict):
        self.table_scene = TableSceneBuilder(
            env=self,
            random_background=True,                # ← enables texture randomization
            robot_init_qpos_noise=self.robot_init_qpos_noise,
        )
        self.table_scene.build()   # uses random_background from the constructor
        # ... load your actors / articulations

Textures are sampled at scene build time

Textures are chosen inside build() / create_table_and_wall(), which runs during scene construction. If your environment is built once and reused, the background stays fixed for the lifetime of the scene unless the environment is reconfigured. With reconfiguration_freq set (e.g. 1 for single-env workflows), each reset rebuilds the scene and re-samples the textures.

The RoboTwin task examples (mani_skill/envs/tasks/tabletop/rm_tasks/) build the same builder — for example rm_microwave.py constructs TableSceneBuilder(self, robot_init_qpos_noise=...) and calls self.table_scene.build(random_background=False) directly. Both call styles are valid; passing the flag to the constructor keeps the default consistent.

Object placement randomization

In addition to the background, every object's spawn pose is randomized per episode. The template does this in _initialize_episode by adding a uniform XY jitter to the base pose before calling set_pose:

python

def _initialize_episode(self, env_idx: torch.Tensor, options: dict):
    with torch.device(self.device):
        b = len(env_idx)
        self.table_scene.initialize(env_idx)

        # Randomize apple pose (with optional L1 xy offset).
        xyz = torch.zeros((b, 3), device=self.device)
        xyz[:, 0] = -0.1
        xyz[:, 1] = -0.15
        xyz[:, 2] = self.apple_z
        xyz[:, :2] += torch.rand((b, 2), device=self.device) * 0.02   # ← placement jitter
        xyz = apply_l1_offset_xy(xyz, offset=(-0.1, 0.1))
        qs = torch.tensor([euler2quat(0, 0, np.pi / 6)] * b,
                          device=self.device, dtype=torch.float32)
        self.apple.set_pose(Pose.create_from_pq(p=xyz, q=qs))

        # Randomize basket pose (shared offset so items stay inside).
        basket_xyz = torch.zeros((b, 3), device=self.device)
        basket_xyz[:, 0] = 0.0
        basket_xyz[:, 1] = 0.1
        basket_xyz[:, 2] = self.basket_z
        basket_xyz[:, :2] += torch.rand((b, 2), device=self.device) * 0.02
        basket_xyz = apply_l1_offset_xy(basket_xyz, offset=(-0.1, 0.1))
        self.breadbasket.set_pose(...)

torch.rand((b, 2)) draws b uniform samples in [0, 1) for the X and Y axes. Multiplied by the scale, each object receives a one-sided offset in [0, scale) on each axis — so the default 0.02 meters gives a 2 cm × 2 cm offset range from the anchor point. For a symmetric range around the anchor, subtract 0.5 before scaling as shown below.

Tuning the jitter

The scale IS the randomization strength. Increase the multiplier to spread objects further apart across episodes, or decrease it to keep placements tight:

python

# Default: [0, 0.02) m per axis (2 cm × 2 cm one-sided range)
xyz[:, :2] += torch.rand((b, 2), device=self.device) * 0.02

# Stronger: [0, 0.1) m per axis
xyz[:, :2] += torch.rand((b, 2), device=self.device) * 0.1

# Even stronger: [0, 0.25) m per axis
xyz[:, :2] += torch.rand((b, 2), device=self.device) * 0.25

Two useful patterns

Symmetric noise — subtract half the scale so the nominal position stays at the center: xyz[:, :2] += (torch.rand((b, 2), ...) - 0.5) * 0.1. Shared offset for coupled objects — draw one offset and reuse it for every object that must stay together (the template does this for the apple and basket), so randomized placements don't break relative geometry.

Don't exceed the workspace

Keep the scale small enough that objects stay within the dual-Panda reachable workspace and don't collide with the wall or fall off the table (the table is 2.0 m × 1.2 m). Start from 0.02 and increase gradually while watching env.reset(seed=...) behavior.

Robot init qpos noise

The robot's initial joint configuration is also perturbed at every reset. In TableSceneBuilder.initialize() a Gaussian offset with standard deviation robot_init_qpos_noise is added to the rest qpos for the supported robot types:

python

qpos = np.array([
    0.0, np.pi / 8, 0, -np.pi * 5 / 8,
    0, np.pi * 3 / 4, np.pi / 4, 0.04, 0.04,
])
qpos = (self.env._episode_rng.normal(
    0, self.robot_init_qpos_noise, (b, len(qpos))) + qpos)
qpos[:, -2:] = 0.04  # gripper fingers always start open at 0.04
self.env.agent.reset(qpos)

The fingers (qpos[:, -2:]) are always reset to a fixed open position; the noise applies to the arm joints. This prevents the policy from memorizing a single initial arm configuration.

Combined example

A task that turns everything up: randomized background, stronger placement jitter, and a slightly noisier robot start:

python

class TwoRobotMyTaskEnv(BaseEnv):
    def __init__(self, *args, robot_uids=("panda_wristcam", "panda_wristcam"),
                 robot_init_qpos_noise=0.05, placement_noise=0.1, **kwargs):
        self.robot_init_qpos_noise = robot_init_qpos_noise
        self.placement_noise = placement_noise
        super().__init__(*args, robot_uids=robot_uids, **kwargs)

    def _load_scene(self, options: dict):
        self.table_scene = TableSceneBuilder(
            env=self,
            random_background=True,                       # randomized textures
            robot_init_qpos_noise=self.robot_init_qpos_noise,
        )
        self.table_scene.build()
        # ... load actors

    def _initialize_episode(self, env_idx: torch.Tensor, options: dict):
        with torch.device(self.device):
            b = len(env_idx)
            self.table_scene.initialize(env_idx)

            xyz = torch.zeros((b, 3), device=self.device)
            xyz[:, 0] = -0.1
            xyz[:, 1] = -0.15
            xyz[:, 2] = self.apple_z
            xyz[:, :2] += (torch.rand((b, 2), device=self.device) - 0.5) * 2 * self.placement_noise
            xyz = apply_l1_offset_xy(xyz, offset=(-0.1, 0.1))
            self.apple.set_pose(Pose.create_from_pq(
                p=xyz, q=torch.tensor([euler2quat(0, 0, np.pi / 6)] * b,
                                      device=self.device, dtype=torch.float32)))

Required assets

Background randomization reads textures from ~/.maniskill/data/robotwin/background_texture/{seen,unseen}/. Install them with the RoboTwin downloader (see the installation page):

bash

mkdir -p ~/.maniskill/data/robotwin
python -m mani_skill.utils.download_robotwin
cd ~/.maniskill/data/robotwin
unzip objects.zip && rm -rf objects.zip
unzip background_texture.zip && rm -rf background_texture.zip

random_background=True requires the texture dataset

If random_background=True but the background_texture/ folder is missing, the builder calls os.listdir on a non-existent directory and raises. Download the RoboTwin assets first.

Next steps

Build your own randomized task from the creating a task guide, or collect demonstrations for it with the data collection pipeline.