Baselines

Baseline overview

Each baseline is a self-contained directory under examples/baselines/ with its own README describing the method, training and rollout entry points, and the key hyperparameters used to reproduce the paper's experiments.

Shared infrastructure

ACT

The policy receives a robot observation stream together with a task representation extracted from the human demonstration video, and predicts a chunk of dual-arm actions with a DETR-style transformer decoder. The reference training command uses the frozen task-video (video-only) configuration:

TAG=<15|30|45>
BACKBONE=<dinov2_vitl14|siglip2_so400m|videomae_large>

python -m examples.baselines.act.train_act_imitator \
  --human-root demos/demo_data \
  --sim-root demos/imitator_data \
  --human-dataset-file examples/baselines/lerobot_dataset/config/exp_configs/human_train_config_${TAG}.json \
  --sim-dataset-file examples/baselines/lerobot_dataset/config/exp_configs/sim_train_config_${TAG}.json \
  --task-mapping-file examples/baselines/lerobot_dataset/task_mapping.json \
  --input-mode video_only \
  --task-encoder-type frozen_backbone \
  --frozen-backbone-type ${BACKBONE} \
  --frozen-backbone-num-frames 10 \
  --frozen-backbone-adapter-layers 1 \
  --frozen-backbone-seq-patches 32 \
  --pred-horizon 24 \
  --batch-size 128 \
  --total-epochs 10 \
  --warmup-epochs 1 \
  --lr 1e-4 \
  --control-mode pd_joint_pos \
  --env-id TwoRobotPourCup-v1 \
  --max-episode-steps 500 \
  --no-include-depth

Additional model hyperparameters used in the reported ablation runs: --hidden-dim 1024 --dim-feedforward 4096 --nheads 16 --enc-layers 12 --dec-layers 12. Checkpoints are written to runs/<run_name>/checkpoints/, including final_model.pt.

Diffusion Policy

The model conditions a 1D action diffusion model (UNet) on robot observations together with a task representation extracted from the human demonstration video:

python -m examples.baselines.diffusion_policy.train_dp_imitator \
  --human-root demos/demo_data \
  --sim-root demos/imitator_data \
  --human-dataset-file examples/baselines/lerobot_dataset/config/exp_configs/human_train_config_${TAG}.json \
  --sim-dataset-file examples/baselines/lerobot_dataset/config/exp_configs/sim_train_config_${TAG}.json \
  --task-mapping-file examples/baselines/lerobot_dataset/task_mapping.json \
  --input-mode video_only \
  --task-encoder-type frozen_backbone \
  --frozen-backbone-type ${BACKBONE} \
  --frozen-backbone-num-frames 10 \
  --control-mode pd_joint_pos \
  --batch-size 256 \
  --total-epochs 10 \
  --lr 1e-4 \
  --num-dataload-workers 24 \
  --env-id TwoRobotPourCup-v1 \
  --max-episode-steps 500 \
  --unet-dims 352 704 1408 2816

Other baselines

Every baseline has its own README with training and evaluation commands. Highlights:

• VQ-BeT — two-stage: pretrain the VQ-VAE (pretrain_vqvae_imitator.py), then train the behavior Transformer (train_vqbet_imitator.py).
• UniSkill — cross-embodiment skill learning: train the inverse-dynamics module (diffusion/train_uniskill.py), then the conditional diffusion policy (diffusion/train_cond_dp.py).
• XSkill — skill discovery + transfer: stage 1 pretrains the encoder (scripts/stage1_pretrain_encoder.py), stage 2 transfers the skill (scripts/stage2_skill_transfer.py).
• GR00T N1.6 — flow-matching VLA, LoRA fine-tuned on the shared bimanual dataset with its dedicated uv environment:

cd examples/baselines/gr00t
uv sync
source .venv/bin/activate
# then, from the repository root:
CUDA_VISIBLE_DEVICES=0 python -m gr00t.experiment.launch_finetune \
  --embodiment-tag NEW_EMBODIMENT \
  --human-config-path examples/baselines/lerobot_dataset/config/exp_configs/human_train_config_${TAG}.json \
  --sim-config-path examples/baselines/lerobot_dataset/config/exp_configs/sim_train_config_${TAG}.json \
  --lerobot-version v3 \
  --language-source human_desc \
  --batch-size 256 \
  --num-epochs 10 \
  --save-epochs 1 \
  --use-backbone-lora --backbone-lora-rank 64 \
  --use-llm-lora --llm-lora-rank 64 \
  --tune-projector \
  --base-model-path <GR00T-N1.6-3B cache path> \
  --dataset-path demos/imitator_data

• RDT-1B — diffusion transformer fine-tuned with LoRA (train_rdt_lora.py) or from scratch (train_rdt_scratch.py).
• π0 / π0.5 — flow-matching VLA in JAX (train_pi_lerobot_jax.py), depends on the pinned openpi package.
• OpenVLA — LoRA fine-tuning with a discrete / regression / diffusion action head (train_openvla.py).

The paired dataset contract

The released baselines consume the shared contract in examples/baselines/lerobot_dataset:

• ACT, Diffusion Policy, and VQ-BeT use HumanSimPairedDataset directly.
• OpenVLA-OFT, π0.5, and RDT adapt the paired sample to their model-specific token, tensor, or JAX layouts.
• XSkill and UniSkill enable the skill-frame fields through wrappers around the paired interface.
• GR00T uses its own episode loader but reads the same LeRobot directories, experiment configs, task mapping, and descriptions.

The common training arguments are therefore the human and simulation data roots, their experiment config files, task_mapping.json, the task-description files, camera selection, observation/action horizons, and input_mode. Model-specific commands and hyperparameters are documented in each policy's README. See the data collection page for conversion and split setup.