Evaluation

Compute Score for a Single Run

The evaluation is done using the recorded log files. To compute the accumulated reward which you achieved in one run, use evaluate_log.py which you can execute using the Singularity image:

./rrc2021.sif ros2 run robot_fingers evaluate_log --task move_cube_on_trajectory \
    --log-dir path/to/log_files

The given “log-dir” is expected to contain the files generated during the run, see Complete List of Generated Files.

The example above is for Stage 1. For Stage 2, simply replace the specified task with “rearrange_dice”.

Weekly Evaluation

To give you an idea were you are standing, we will do an evaluation round once every week. For this, we will take the latest version of your software from the configured git repository and run it times on different goals. We then take the median reward of all runs.

The result of this weekly evaluation will be published using your submission system usernames on the challenge website. Note that the usernames are randomly assigned, so as long as you don’t make it public the ranking is anonymous.

The final evaluation after the end of the stage will be done in the same way.