Documentation
README
Plan as if the experimentation stage will not happen, because usually it does not
The pipeline has a stage for running experiments and a stage for analysing them. On a wall-clocked run they are frequently theoretical.
Measured over a nineteen-task scored arm, with a four-hour cap per task:
| final stage the run reached | runs |
|---|---|
| 01 literature survey | 6 |
| 02 hypothesis generation | 3 |
| 03 study design | 5 |
| 04 implementation | 4 |
| 06 analysis | 1 |
Nineteen of nineteen hit the cap. None finished the walk. Six never left the first stage; four of those six spent between 13 and 22 attempts on it. So the predictions file that got scored was, on most tasks, whatever an early stage happened to produce on the way past.
This is the opening of the README. Read the full README on GitHub.