**********
Evaluation
**********

Compute Score for a Single Run
==============================

The evaluation is done using the recorded log files.  To compute the accumulated
reward which you achieved in one run, use ``evaluate_log.py`` which you can
execute using the Singularity image:

.. code-block:: bash

   ./rrc2021.sif ros2 run robot_fingers evaluate_log --task move_cube_on_trajectory \
       --log-dir path/to/log_files

The given "log-dir" is expected to contain the files generated during the run,
see :ref:`list_of_generated_files`.

The example above is for Stage 1.  For Stage 2, simply replace the specified
task with "rearrange_dice".


.. _weekly_evaluation:

Weekly Evaluation
=================

To give you an idea were you are standing, we will do an evaluation round once
every week.  For this, we will take the latest version of your software from the
configured git repository and run it times on different goals.  We then take
the median reward of all runs.

The result of this weekly evaluation will be published using your
submission system usernames on the `challenge website`_.  Note that the
usernames are randomly assigned, so as long as you don't make it public the
ranking is anonymous.

The final evaluation after the end of the stage will be done in the same way.


.. _challenge website: https://real-robot-challenge.com