This challenge has ended!
This documentation is only for the Real Robot Challenge 2020 which has ended. Following challenges have their own documentation, see the challenge website for more information.
Before you dive into the code, please check the rules of the simulation phase here: https://real-robot-challenge.com/simulation_phase.
Here we explain the quantitative-evaluation procedure of the simulation phase. We provide instructions how to install the simulation package, how to integrate your own code, and how to submit your code and results. We also explain how we will execute your code to verify the reported scores.
Download and install the rrc_simulation repository, as described here: Install Software for Simulation Phase.
Note that the simulation phase will be done completely in Python. In later phases, using the real robot, you will also have the option to use C++.
The task is to manipulate a cube in a given way using the TriFinger robot (see Details of the Tasks). You may use any approach to create a control policy which solves this task. To avoid confusion, no changes are allowed to existing files, please create new files for your code. The only existing files you may and must change are:
rrc_simulation/scripts/evaluate_policy.py: This file will be executed during evaluation, so you must replace the placeholder policy with your policy. You may also replace our gym environment with your own if you wish to e.g. use a different action space, observation space or reward function. The most important thing in this script is that it stores the action log at the end, which will be used for evaluation according to our gym environment.
rrc_simulation/environment.yml: You will replace this file with your own conda environment, necessary to execute your code. You can do this e.g. with
conda env export > environment.yml
or by creating the file manually.
Once you are done developing, please run the evaluation command:
rrc_simulation/scripts directory. This will execute the
evaluate_policy.py file discussed above for multiple goals. Then it will
simulate the action sequence stored by
evaluate_policy.py and print the
resulting score in the terminal.
Note that rrc_evaluate will execute
evaluate_policy.py in the folder from
which it is called. You may wish to develop your
evaluate_policy.py file in different folder, but at the end you must place
the file to be used for evaluation at
since this is the path we will assume.
Finally, you will create a submission containing
your score, which you copied from the terminal and you will enter into a form
your rrc_simulation folder (as a .zip) containing
the unchanged original files
and the output_dir (as a .zip) generated by
Below, we describe how we will evaluate the submission. We highly recommend that participants go through exactly these steps with their tentative submission to ensure that everything will run smoothly on our side.
On our side, we will then
download your folder to a computer with the specs described below,
install the conda environment according to the same instructions as you did (see Install Software for Simulation Phase),
rrc_simulation/scriptsfolder. If it does not terminate within 1 hour, we will abort it and assign the minimum score.
We then compare the score obtained to the reported score to ensure that they are identical (up to variations due to the random seed).
We will also verify all the action logs to ensure that simulation parameters have not been modified by the user code.
Evaluation Computer Specs¶
CPU: 8 cores, 3GHz
GPU: will not be available during the evaluation (you may of course use the GPU for training on your side)
OS: Ubuntu 18.04 or MacOS (you will be able to choose)