**********
Evaluation
**********

For Phase 2 of the challenge, the same tasks and evaluation metrics are used as
for Phase 1 (see :doc:`../simulation_phase/tasks`) -- just now on the real
robots instead of using simulation.
There is only one difference:  To give you more time on the actual robot, the
episode length is increased to 120000 steps, corresponding to 2 minutes at 1
kHz.


Compute Score for a Single Run
==============================

The evaluation is done using the recorded log files.  To compute the accumulated
reward which you achieved in one run, use ``evaluate_log.py`` which you can
execute using the Singularity image:

.. code-block:: bash

   ./realrobotchallenge.sif rosrun robot_fingers evaluate_log.py \
       --robot-log robot_data.dat --camera-log camera_data.dat --goal-file goal.json

The files specified in the arguments correspond to the generated files with the
same name, see :ref:`list_of_generated_files`.


.. _weekly_evaluation:

Weekly Evaluation
=================

To give you an idea were you are standing, we will do an evaluation round once
every week.  For this, we will take the latest version of your software from the
configured git repository and run it multiple times on different goals of all
difficulty levels.  We then compute the weighted reward over all levels like we
did in Phase 1.

The result of this weekly evaluation will be published using your
submission system usernames on the `challenge website`_.  Note that the
usernames are randomly assigned, so as long as you don't make it public the
ranking is anonymous.

The final evaluation after the end of the phase will be done in the same way.


.. _challenge website: https://real-robot-challenge.com