Tutorial

Domain randomization

Two knobs control the visual and spatial diversity of every benchmark task: background texture randomization via TableSceneBuilder(..., random_background=True), and per-episode object-placement jitter via the xyz[:, :2] += rand * scale pattern in _initialize_episode.

Overview

Simulation makes it cheap to chop and change aspects of the scene that would be expensive in the real world. The Imitator Game task template exposes two ready-made domain-randomization hooks, both implemented in the shared mani_skill/utils/scene_builder/table/scene_builder.py and the task template at mani_skill/envs/tasks/_template/template_task.py:

Background textures

Randomized wall, table and ground textures sampled from the RoboTwin background-texture dataset, toggled with random_background=True.

Object placement

A uniform XY offset added to every object's spawn pose in _initialize_episode; the magnitude is the scale in xyz[:, :2] += torch.rand(...) * SCALE.

Robot init pose

Gaussian noise on the robot's initial joint configuration, controlled by robot_init_qpos_noise on TableSceneBuilder.

Background randomization

The background randomization is implemented by TableSceneBuilder.create_table_and_wall(...). When enabled, it samples random texture files for the wall, the table and the ground from the RoboTwin background-texture directory:

python

def create_table_and_wall(
        self,
        table_xy_bias=[0, 0],
        table_height=0.9,
        random_background=False,
        eval_mode=False
):
    ...
    if random_background:
        texture_type = "seen" if not eval_mode else "unseen"
        directory_path = str(Path.home() / ".maniskill" / "data" / "robotwin" / f"background_texture/{texture_type}")
        file_count = len([name for name in os.listdir(directory_path) ...])

        wall_texture   = np.random.randint(0, file_count)
        table_texture  = np.random.randint(0, file_count)
        ground_texture = np.random.randint(0, file_count)
        ...
    else:
        wall_texture, table_texture, ground_texture = None, None, None

Robot init qpos noise

The robot's initial joint configuration is also perturbed at every reset. In TableSceneBuilder.initialize() a Gaussian offset with standard deviation robot_init_qpos_noise is added to the rest qpos for the supported robot types:

python

qpos = np.array([
    0.0, np.pi / 8, 0, -np.pi * 5 / 8,
    0, np.pi * 3 / 4, np.pi / 4, 0.04, 0.04,
])
qpos = (self.env._episode_rng.normal(
    0, self.robot_init_qpos_noise, (b, len(qpos))) + qpos)
qpos[:, -2:] = 0.04  # gripper fingers always start open at 0.04
self.env.agent.reset(qpos)

The fingers (qpos[:, -2:]) are always reset to a fixed open position; the noise applies to the arm joints. This prevents the policy from memorizing a single initial arm configuration.

How to enable it

The TableSceneBuilder is constructed in your task's _load_scene. Set random_background=True in the constructor to turn the randomization on:

python

from mani_skill.utils.scene_builder.table import TableSceneBuilder

class TwoRobotMyTaskEnv(BaseEnv):
    def _load_scene(self, options: dict):
        self.table_scene = TableSceneBuilder(
            env=self,
            random_background=True,                # ← enables texture randomization
            robot_init_qpos_noise=0.02,            # ← enables robot initial joint pose randomization
        )
        self.table_scene.build()   # uses random_background from the constructor
        # ... load your actors / articulations

Object placement randomization

In addition to the background, every object's spawn pose is randomized per episode. The template does this in _initialize_episode by adding a uniform XY jitter to the base pose before calling set_pose:

python

def _initialize_episode(self, env_idx: torch.Tensor, options: dict):
    with torch.device(self.device):
        b = len(env_idx)
        self.table_scene.initialize(env_idx)

        # Randomize apple pose (with optional L1 xy offset).
        xyz = torch.zeros((b, 3), device=self.device)
        xyz[:, 0] = -0.1
        xyz[:, 1] = -0.15
        xyz[:, 2] = self.apple_z
        xyz[:, :2] += torch.rand((b, 2), device=self.device) * 0.02   # ← placement jitter
        xyz = apply_l1_offset_xy(xyz, offset=(-0.1, 0.1))    
        qs = torch.tensor([euler2quat(0, 0, np.pi / 6)] * b,          # ← rotation jitter
                          device=self.device, dtype=torch.float32)
        self.apple.set_pose(Pose.create_from_pq(p=xyz, q=qs))

        # Randomize basket pose (shared offset so items stay inside).
        basket_xyz = torch.zeros((b, 3), device=self.device)
        basket_xyz[:, 0] = 0.0
        basket_xyz[:, 1] = 0.1
        basket_xyz[:, 2] = self.basket_z
        basket_xyz[:, :2] += torch.rand((b, 2), device=self.device) * 0.02
        basket_xyz = apply_l1_offset_xy(basket_xyz, offset=(-0.1, 0.1))
        self.breadbasket.set_pose(...)

torch.rand((b, 2)) draws b uniform samples in [0, 1) for the X and Y axes. Multiplied by the scale, each object receives a one-sided offset in [0, scale) on each axis — so the default 0.02 meters gives a 2 cm × 2 cm offset range from the anchor point. For a symmetric range around the anchor, subtract 0.5 before scaling.

Two useful patterns

Symmetric noise — subtract half the scale so the nominal position stays at the center: xyz[:, :2] += (torch.rand((b, 2), ...) - 0.5) * 0.1.
Shared offset for coupled objects — draw one offset and reuse it for every object that must stay together (the template does this for the apple and basket), so randomized placements don't break relative geometry.

Don't exceed the workspace

Keep the scale small enough that objects stay within the dual-Panda reachable workspace and don't collide with the wall or fall off the table (the table is 2.0 m × 1.2 m). Start from 0.02 and increase gradually while watching env.reset(seed=...) behavior.

Next steps

Build your own randomized task from the creating a task guide, or collect demonstrations for it with the data collection pipeline.