Attention

This challenge has ended!

This documentation is only for the Real Robot Challenge 2020 which has ended. Following challenges have their own documentation, see the challenge website for more information.

Evaluation

For Phase 2 of the challenge, the same tasks and evaluation metrics are used as for Phase 1 (see Details of the Tasks) – just now on the real robots instead of using simulation. There is only one difference: To give you more time on the actual robot, the episode length is increased to 120000 steps, corresponding to 2 minutes at 1 kHz.

Compute Score for a Single Run

The evaluation is done using the recorded log files. To compute the accumulated reward which you achieved in one run, use evaluate_log.py which you can execute using the Singularity image:

./realrobotchallenge.sif rosrun robot_fingers evaluate_log.py \
    --robot-log robot_data.dat --camera-log camera_data.dat --goal-file goal.json

The files specified in the arguments correspond to the generated files with the same name, see Complete List of Generated Files.

Weekly Evaluation

To give you an idea were you are standing, we will do an evaluation round once every week. For this, we will take the latest version of your software from the configured git repository and run it multiple times on different goals of all difficulty levels. We then compute the weighted reward over all levels like we did in Phase 1.

The result of this weekly evaluation will be published using your submission system usernames on the challenge website. Note that the usernames are randomly assigned, so as long as you don’t make it public the ranking is anonymous.

The final evaluation after the end of the phase will be done in the same way.