Creating a new task

Design templates

The repository ships four design templates that gate every contribution:

Task implementation contract

Each released task is a matched set of four components:

1. one base environment registered as TwoRobot{Task}-v1;
2. one standalone L3 environment registered as TwoRobot{Task}L3-v1;
3. one base and one L3 motion-planning solution registered in MP_SOLUTIONS; and
4. four aliases in the dataset task mapping.

The environment owns scene construction, episode initialization, success predicates, and reward phases. Level utilities own only the controlled L1/L2 changes shared across tasks. Motion-planning solutions access task geometry through the environment and produce the same two-agent pd_joint_pos action interface used by recorded demonstrations.

Create the environment

Start from the template at mani_skill/envs/tasks/_template/template_task.py:

1. Copy it to mani_skill/envs/tasks/tabletop/dual_tasks/_NNN_your_task.py.
2. Change the @register_env id, class name, and docstring.
3. Import them in mani_skill/envs/tasks/tabletop/__init__.py.
4. Replace the assets in _load_scene() — use object_loader.py.
5. Update _initialize_episode(), evaluate(), and compute_dense_reward().

The L0/L1/L2 class is registered as TwoRobotTemplateTask-v1; the L3 class as TwoRobotTemplateTaskL3-v1. Rename both before use.

Loading objects

Assets live under ~/.maniskill/data (see the installation page). The template provides loaders for all four namespaces:

from mani_skill.envs.tasks._template.object_loader import (
    load_ycb_object, load_robotwin_object, load_partnet_object,
    load_sketchfab_object, z_offset_to_table,
)

# YCB object (dynamic)
self.apple, _ = load_ycb_object(self.scene, "013_apple", position=[0, 0.5, 0],
                                scale=0.7, mass=0.5)

# RoboTwin container (static goal)
self.basket, _ = load_robotwin_object(self.scene, "076_breadbasket",
                                      position=[0, -0.5, 0],
                                      rotation=(np.pi / 2, 0, 0),
                                      is_static=True)

# PartNet articulated object (microwave, cabinet, ...)
self.microwave, _ = load_partnet_object(self.scene, "microwave", "7119",
                                        position=[0.3, 0.1, 0],
                                        robot_base_position=[0, -1, 0])

# External GLB asset
self.scale, _ = load_sketchfab_object(self.scene, "balance_scale",
                                      position=[0.2, 0.0, 0.0])

How levels work

The level is chosen before gym.make via configure_dual_task_level(level), or via the --l0/--l1/--l2 flags of two_robot_run. Do not add level logic to the policy. Full semantics are on the levels page.

Required method contract

Every task environment must implement:

• _load_scene(options) — build table + load all actors/articulations
• _initialize_episode(env_idx, options) — randomize per-episode state
• _after_reconfigure(options) — compute z-offsets for table contact
• evaluate() — return dict with a success tensor (rule-based)
• _get_obs_extra(info) — extra observation fields
• compute_dense_reward(obs, action, info) — reward
• compute_normalized_dense_reward(obs, action, info)

Camera configuration (_default_sensor_configs, hi_res, wrist_sensor) and the dual-agent setup are identical to the released tasks; keep them as-is so the data pipeline (two_robot_run, collect_data, h5_to_lerobot) works unchanged. Keep the same __init__ signature (hi_res, wrist_sensor, robot_init_qpos_noise), the same camera sets, RewardTracker phases, and rule-based evaluate() with a success field.

Add a motion-planning solution

Every released task is four components — base env, L3 env, base solution, and L3 solution. Copy the reference solution and implement solve():

cp mani_skill/examples/motionplanning/dual/solutions/two_robot_pick_cube_ycb.py \
   mani_skill/examples/motionplanning/dual/solutions/_NNN_your_task.py

The runner calls solve(env, seed=None, debug=False, vis=False) and expects the tuple (left_res, right_res):

from mani_skill.envs.tasks import YourTaskEnv
from mani_skill.examples.motionplanning.panda.motionplanner import (
    PandaArmMotionPlanningSolver,
)


def solve(env: YourTaskEnv, seed=None, debug=False, vis=False):
    env.reset(seed=seed)

    left_planner = PandaArmMotionPlanningSolver(
        env, debug=debug, vis=vis,
        base_pose=env.unwrapped.agent.agents[0].robot.pose,
        visualize_target_grasp_pose=vis, print_env_info=False,
        multi_robot_id=0,                      # left robot
    )
    right_planner = PandaArmMotionPlanningSolver(
        env, debug=debug, vis=vis,
        base_pose=env.unwrapped.agent.agents[1].robot.pose,
        visualize_target_grasp_pose=vis, print_env_info=False,
        multi_robot_id=1,                      # right robot
    )
    env = env.unwrapped

    # Always pass the other robot's gripper state to avoid collisions.
    left_planner.move_to_pose_with_screw(
        my_pose, other_gripper_state=right_planner.gripper_state)
    left_planner.close_gripper(other_gripper_state=right_planner.gripper_state)
    ...
    left_res = left_planner.open_gripper(other_gripper_state=right_planner.gripper_state)
    return left_res, left_res

Key planner API (PandaArmMotionPlanningSolver):

Useful helpers: get_actor_obb(actor), compute_grasp_info_by_obb(...) from ../panda/utils.py to build grasp poses from an object's oriented bounding box.

Register the solution

In solutions/__init__.py:

from ._NNN_your_task import solve as solveYourTask

In two_robot_run.py, add to MP_SOLUTIONS:

MP_SOLUTIONS = {
    ...,
    "TwoRobotYourTask-v1": solveYourTask,
}

If the task is L0/L1/L2, its L3 variant goes into solutions_l3/ and is registered with the ...L3-v1 key.

Add a robot

New robots (URDF agents) are contributed via the robot template under mani_skill/agents/robots/_template/, which documents how to register a URDF agent and its supported controllers. Robots such as the single-arm Panda, dual-arm Panda pairs, xArm6+Allegro, Fetch, so100, and widowxai are already supported by TableSceneBuilder.initialize().

Verify

python -m mani_skill.examples.motionplanning.dual.two_robot_run \
  -e TwoRobotTemplateTask-v1 -n 1 --only-count-success --record-dir demos