We aim to use only Aloha the Demonstrator to collect demonstrations, train a policy, and control the demonstrator itself. I will walk through the whole process using the official SDK, cobot_magic, provided by Agilex. This note largely follows the official tutorial of Agilex. Here I will focus on things customized for our goal.
The SDK provides the following modules.
aloha-devel/ # policy learning and inference
camera_ws/ # camera configuration
collect_data/ # demonstration collection
Piper_ros_private-ros-noetic/ # Piper the arm
tools/ # quick scripts
Let's first prepare a runtime environment
source ~/cobot_magic/camera_ws/devel/setup.bash
source ~/cobot_magic/Piper_ros_private-ros-noetic/devel/setup.bash
Run bash tools/build if the camera module has not been built; otherwise, check camera serial numbers using tools/camera_serial.sh and fill them in 'camera_ws/src/ros_astra_camera/launch/multi_camera.launch' accordingly. In my case, I made a copy of the launch file and edited the copy '.../multi_camera_yann.launch', so I can launch the cameras by
roslaunch astra_camera multi_camera_yann.launch
The launch file defines camera-related topics (e.g., depth and pixel image) that will be published on a ROS node. Check camera status by running rqt_image_view and select ROS topics of your interest.
First, let's activate can's:
bash Piper_ros_private-ros-noetic/can_muti_activate.sh
where I made the following modifications to rename can's
EXPECTED_CAN_COUNT=2
if [ "$EXPECTED_CAN_COUNT" -ne 1 ]; then
declare -A USB_PORTS
USB_PORTS["1-1.1:1.0"]="can_piper_left:1000000"
USB_PORTS["1-1.2:1.0"]="can_piper_right:1000000"
fi
If you are unclear which piper corresponds to which USB port, simply unplug all of them, re-plugin one by one, and check ports every time you plug in the USB of a new piper by running
bash Piper_ros_private-ros-noetic/find_all_can_port.sh
Before launch pipers, we will need to modify can names specified in Piper_ros_private-ros-noetic/src/piper/launch/start_ms_piper.launch accordingly. In my case, I made a copy of the launch file and edited the copy '.../start_ms_piper.launch', so my launch command is
roslaunch piper start_ms_piper_yann.launch mode:=0 auto_enable:=false
So far, the demonstrator should have been set up for demonstration collection.
Check the script 'collect_data/yann_collect_data_agilex.py'. I added shortcut keys to facilatate collection. An example running can be:
python yann_collect_data_agilex.py --dataset_dir ~/data --task_name passover --max_timesteps 500 --episode_idx 0 --episode_max 75
In this example, I demonstrated a passover task for 75 times, where I repeatedly pick up a banana from a plate on the left using the left gripper and pass it over to the right gripper and release it on a plate on the right. You can always stop for a break and continue the collection process by specifying the start episode in --episode_idx START_EPISODE_ID.
Apart from replaying the collected episodes:
python yann_replay_data.py --dataset_dir ~/data --task_name passover --only_pub_master --episode_idx 0
I played with them in this Jupyter notebook 'collect_data/visualize_episodes.ipynb'.
Check run.sh in /media/raid/workspace/zhaoyanpeng/code/cobot_magic/aloha-devel. The running environment is configured as:
mamba create -n tongvla python=3.10 -y
mamba install ffmpeg==7.1.1 -c conda-forge
mamba install git-lfs -c conda-forge
pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu126
cd /media/raid/workspace/zhaoyanpeng/code/cobot_magic/aloha-devel
pip install -r requirements-new.txt
CMAKE_POLICY_VERSION_MINIMUM=3.5 proxychains4 pip install robomimic
Check infer.sh in /home/oops/code/volcano-aloha-devel. The running environment is the same as in learning.
Before inference, we need to reset pipers to the control mode. You might succeed following the official tutorial: power off and on pipers, and launch them by
roslaunch piper start_ms_piper_yann.launch mode:=1 auto_enable:=true
but it does not always work out in my case. I have to make sure the pipers are in the control mode first, using this script '/home/agilex/project/control.py'. First uncomment piper.MasterSlaveConfig(0xFC, 0, 0, 0) and run it to see if the specified piper works. That is it if it works; otherwise, comment out the line and rerun to check out. Usually, it will work the second time. If not, repeat the process until it does.
By the way, here is a wrapper I found promising, making controling easier. Agilex also provides a README and minimal demos of setting up pipers in the correct mode. Take a look at piper_sdk/demo/V2/piper_ctrl_joint.py and piper_sdk/demo/V2/piper_ctrl_enable.py.
So far, I have assumed policy inference and controling (via rospy) on the same machine mounted on the robot, but, in reality, policy inference is computation extensive, so ideally, it should be deployed on a server, and the machine mounted on the robot solely hosts a ROS service to send current observations and receive next actions. To see how to implement this paradigm, checkout my practice in this post.
Even better, we can wrap up the policy model within a small client, solely receiving observations and sending out predicted actions; it is agnostic of what policy model it is in the background and communicates between the policy model and the ROS service. We are working on this right now. Stay tuned!