We aim to use only Aloha the Demonstrator to collect demonstrations, train a policy, and control the demonstrator itself. I will walk through the whole process using the official SDK, cobot_magic, provided by Agilex. This note largely follows the official tutorial of Agilex. Here I will focus on things customized for our goal.
The SDK provides the following modules.
aloha-devel/ # policy learning and inference
camera_ws/ # camera configuration
collect_data/ # demonstration collection
ATTENTION: Please use the default system-wide compiler tools like cmake, gcc, g++, and python, deactivate all conda/mamba environments, and remove all customized $PATH; otherwise, the compiler cmake may complain that necessary headers and libraries are missing, for example, "Could NOT find boost," "Compatibility with CMake < 3.5 has been removed from CMake," and "/usr/include/pybind11/detail/common.h:215:10: fatal error: Python.h: No such file or directory."
There should not be any issues if you have been following the tutorial on a fresh OS, but there might be some pitfalls casued by your OS setups.
If it complains "Unable to locate package python3-rosdep" when installing development tools and ROS tools, simply change python3-rosdep to python3-rosdep2.
| droot = "/home/xxxx/data" | |
| kroot = f"{droot}/kelm-corpus/updated-2021" | |
| kname = "kelm_generated_corpus.jsonl" | |
| ename = "entities.jsonl" | |
| tekgen_train = "quadruples-train.tsv" | |
| tekgen_test = "quadruples-test.tsv" | |
| tekgen_val = "quadruples-validation.tsv" |
| #!/usr/bin/sh | |
| root=./ | |
| export OMP_NUM_THREADS=32 | |
| export CUDA_VISIBLE_DEVICES=$2 | |
| run_type=$1 | |
| gpu_list=(${CUDA_VISIBLE_DEVICES//,/ }) |
| ('three_hop.json', 0, '/mnt/yann/code/clevr-caption/question_generation_2.0/CAPTION_templates/three_hop.json') | |
| How many <Z4> <C4> <M4> <S4>s are <R3> the <Z3> <C3> <M3> <S3> [that is] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>? | |
| What number of <Z4> <C4> <M4> <S4>s are <R3> the <Z3> <C3> <M3> <S3> [that is] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>? | |
| There is a <Z3> <C3> <M3> <S3> [that is] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>; how many <Z4> <C4> <M4> <S4>s are <R3> it? | |
| There is a <Z3> <C3> <M3> <S3> [that is] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>; what number of <Z4> <C4> <M4> <S4>s are <R3> it? | |
| There are <ANS> <Z4> <C4> <M4> <S4>s [that are] <R3> the <Z3> <C3> <M3> <S3> [that is] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>. | |
| The number of <Z4> <C4> <M4> <S4>s [that are] <R3> the <Z3> <C3> <M3> <S3> [that is] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S> is <ANS>. | |
| There is a <Z3> <C3> |
| ('one_hop.json', 0, '/mnt/yann/code/clevr-caption/question_generation_2.0/CAPTION_templates/one_hop.json') | |
| What number of <Z2> <C2> <M2> <S2>s are <R> the <Z> <C> <M> <S>? | |
| How many <Z2> <C2> <M2> <S2>s are <R> the <Z> <C> <M> <S>? | |
| There is a <Z> <C> <M> <S>; how many <Z2> <C2> <M2> <S2>s are <R> it? | |
| There is a <Z> <C> <M> <S>; what number of <Z2> <C2> <M2> <S2>s are <R> it? | |
| The number of <Z2> <C2> <M2> <S2>s [that are] <R> the <Z> <C> <M> <S> is <ANS>. | |
| There are <ANS> <Z2> <C2> <M2> <S2>s [that are] <R> the <Z> <C> <M> <S>. | |
| There is a <Z> <C> <M> <S>; there are <ANS> <Z2> <C2> <M2> <S2>s [that are] <R> it. |
| ('two_hop.json', 0, '/mnt/yann/code/clevr-caption/question_generation_2.0/CAPTION_templates/two_hop.json') | |
| How many <Z3> <C3> <M3> <S3>s are <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>? | |
| What number of <Z3> <C3> <M3> <S3>s are <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>? | |
| There is a <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>; how many <Z3> <C3> <M3> <S3>s are <R2> it? | |
| There is a <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>; what number of <Z3> <C3> <M3> <S3>s are <R2> it? | |
| There are <ANS> <Z3> <C3> <M3> <S3>s [that are] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>. | |
| The number of <Z3> <C3> <M3> <S3>s [that are] <R2> the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S> is <ANS>. | |
| There is a <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>; there are <ANS> <Z3> <C3> <M3> <S3>s [that are] <R2> it. |
| ('same_relate.json', 0, '/mnt/yann/code/clevr-caption/question_generation_2.0/CAPTION_templates/same_relate.json') | |
| Are there any other things that have the same size as the <Z> <C> <M> <S>? | |
| Is there anything else that has the same size as the <Z> <C> <M> <S>? | |
| Is there any other thing that has the same size as the <Z> <C> <M> <S>? | |
| Are there any other things that are the same size as the <Z> <C> <M> <S>? | |
| Is there anything else that is the same size as the <Z> <C> <M> <S>? | |
| Is there any other thing that is the same size as the <Z> <C> <M> <S>? | |
| There are some other things that have the same size as the <Z> <C> <M> <S>. |
| ('zero_hop.json', 0, '/mnt/yann/code/clevr-caption/question_generation_2.0/CAPTION_templates/zero_hop.json') | |
| How many <Z> <C> <M> <S>s are there? | |
| What number of <Z> <C> <M> <S>s are there? | |
| There are <ANS> <Z> <C> <M> <S>s. | |
| The number of <Z> <C> <M> <S>s is <ANS>. | |
| ('zero_hop.json', 1, '/mnt/yann/code/clevr-caption/question_generation_2.0/CAPTION_templates/zero_hop.json') |