# Testsuite

Please make sure you have read the section about testruns in the scip documentation https://scipopt.org/doc/html/TEST.php before you continue reading.

In general the execution of the testruns is the following:

- To start a testrun call `make test` for a local one, `make testcluster` for one on the cluster.
  (These are for running scip tests but variants exist for other solvers, see below)
- The `Makefile` will call the corresponding `check_<solver>.sh` or `check_cluster_<solver>.sh` script
  with the correct variables. These scripts will
  + configure the environment variables for local and cluster runs `. configuration_set.sh`, `. configuration_cluster.sh`
  + configure the test output files such as the .eval, the .tmp and the .set files `. configuration_logfiles.sh`
  + cluster run scripts have a `waitcluster.sh` script in the loop that makes the scripts wait
    instead of overloading the cluster queue with jobs.
  + run `evalcheck_cluster.sh` to evaluate and save any old testruns that lay around.
  + `. configuration_tmpfile_setup_{cbc,cplex,gurobi,scip,xpress}.sh`
    reset and fill a tmpfile to run the solver with. Tmpfile will set correct limits, read in settings, and control
    display of the solving process.
  + with `run.sh` (or `run.*.sh`) run or submit the jobs for the current testrun
  + some local `check_*.sh` scripts run `evalcheck.sh` or `evalcheck_cluster.sh` automatically
    after the testrun, for the others and for clusterruns the user has to do it themselves.
    The `evalcheck*.sh` scripts evaluate the testrun and concatenate the individual
    logfiles. The naming here are a bit misleading, the main difference between the two files is that
    the `_cluster` script is more involved and cares about cleaning up after itself. Both of them can
    be used to evaluate a `check.*.eval` file. Recommended is to use the command
    `evalcheck_cluster.sh results/check.*.eval` from the `check` folder.

## local make targets

### run a local testrun with scip

make test
  - `check.sh`
    + `. configuration_set.sh`
    + `. configuration_logfiles.sh` END
    + `evalcheck_cluster.sh` END
    + `. configuration_tmpfile_setup_{cbc,cplex,gurobi,scip,xpress}.sh` END
    + `run.sh` END

### run a local testrun with fscip

make testfscip
  - `check_fscip.sh`
    + `. configuration_set.sh`
    + `. configuration_logfiles.sh` END
    + `evalcheck_cluster.sh` END
    + `run_fscip.sh` END

### run a local testrun with glpk, mosek, symphony

make test{glpk,mosek,symphony}
  - `check_{glpk,mosek,symphony}.sh`
    +`getlastprob.awk` END
    +`evalcheck.sh`

### generate a coverage report

make coverage
  - `check_coverage.sh`
    + `evalcheck.sh`

### count feasible solutions

make testcount
  - `check_count.sh`
    + `getlastprob.awk` END
    + `evalcheck_count.sh`
       - `. configuration_solufile.sh` END
       - `check_count.awk` END

## cluster make targets

### run a clusterrun with scip, cbc, cpx, gurobi, xpress

make testcluster{,cbc,cpx,gurobi,xpress}
  - `check_cluster.sh`
    + `. configuration_cluster.sh`
    + `. configuration_set.sh`
    + `waitcluster.sh` END
    + `. configuration_logfiles.sh` END
    + `. configuration_tmpfile_setup_{cbc,cplex,gurobi,scip,xpress}.sh` END
    + `run.sh` END

### run a clusterrun with fscip

make testclusterfscip
  - `check_cluster_fscip.sh`
    + `. configuration_cluster.sh`
    + `. configuration_set.sh`
    + `waitcluster.sh` END
    + `. configuration_logfiles.sh` END
    + `run_fscip.sh` END

### run a clusterrun with mosek

make testclustermosek
  - `check_cluster_mosek.sh`
    + `. configuration_cluster.sh`
    + `waitcluster.sh` END
    + `run.sh` END

### start a gams testrun on the cluster

make testgamscluster
  - `check_gamscluster.sh`
    + `schulz.sh` END
    + `. configuration_solufile.sh` END
    + `waitcluster.sh` END
    + `rungamscluster.sh` END
    + `finishgamscluster.sh`
      - evalcheck_gamscluster.sh
        + `. configuration_solufile.sh` END
        + `check_count.awk` END

## other Scripts

### Generate comparison of two testruns.

- `allcmpres.sh`
  + `cmpres.awk` END

### compares averages of several SCIP result files

- `average.sh`
  + `average.awk` END

### compute averages over instances for different permuations

- `permaverage.sh`
  + `permaverage.awk` END

### compare different versions of runs with permuations

- `permcmpresall.sh`
  + `permcmpresall.awk` END

# Files

## AWK files

- `check_*.awk` END
  Parse and check the check files of a testrun and output a .res file table.

- `check_count.awk`
  Count feasible solutions

- `getlastprob.awk` END
  Get the last problem of a check.*-outfile

- `cmpres.awk` END
  Check Comparison Report Generator - Compare two res files.

- `average.awk` END
  compute averages of several SCIP result files

- `permaverage.awk` END
  compute averages over instances for different permuations

- `permcmpresall.awk` END
  compare different versions of runs with permuations

## Bash Scripts

### Configuration

- `. configuration_tmpfile_setup_{cbc,cplex,gurobi,scip,xpress}.sh` END
  Resets and fills a batch file TMPFILE to run the solver with.
  Tmpfile will set correct limits, read in settings, and control display of the solving process.

- `. configuration_logfiles.sh` END
  Configures the right test output files such as the .eval, the .tmp and the .set files to run a test on.

- `. configuration_solufile.sh` END
  Configures SOLUFILE env variable from name of testset.

- `. configuration_cluster.sh`
  Configures environment variables for cluster runs.
  It is to be invoked inside a `check_cluster*.sh` script.

- `. configuration_set.sh`
  Configures environment variables that are needed for test runs both on the cluster and locally.
  It is to be invoked inside a `check(_cluster)*.sh` script.
  + `. configuration_solufile.sh` END

### Running

- `run.sh` END
  The script executing EXECNAME on one instance and producing the logfiles.
  Can be executed either locally or on a cluster node.
  Is to be invoked inside a `check(_cluster)*.sh` script.

- `run_fscip.sh` END
  The script executing fscip on one instance and producing the logfiles.
  Can be executed either locally or on a cluster node.
  Is to be invoked inside a `check(_cluster)*.sh` script.

- `rungamscluster.sh` END
  The script executing gams on one instance and producing the logfiles.
  Can be executed either locally or on a cluster node.
  Is to be invoked inside a `check(_cluster)*.sh` script.

### Evaluation

- `evalcheck.sh`
  Evaluates one or more testrun by each checking the check*-outfile.
  Is to be invoked inside a `check*.sh` script.
  + `. evaluate.sh`
    + `. configuration_solufile.sh` END
    + `check_*.awk` END

- `evalcheck_cluster.sh` END
  Evaluates a testrun and concatenates the individual logfiles, possibly uploads to rubberband.
  + `. evaluate.sh`
    + `. configuration_solufile.sh` END
    + `check_*.awk` END

- `. evaluate.sh`
  Depending on the solver used for the testrun calls corresponding check_*.awk on the testrun files and writes the output in a .res file.
  + `. configuration_solufile.sh` END
  + `check_*.awk` END

- `finishgamscluster.sh`
  Cleans up after gams testrun.
  + evalcheck_gamscluster.sh
    - `. configuration_solufile.sh` END
    - `check_count.awk` END

### helper

- `waitcluster.sh` END
  In order to not overload the cluster, no jobs are submitted if the queue is too full
  instead, this script waits until the queue load falls under a threshold and returns
  for the calling script to continue submitting jobs.

### uncategorized

- `schulz.sh` END
  Supervises processes and sends given signals via kill when elapsed time exceeds given thresholds.

## folders and other files

- `CMakeLists.txt` is part of the cmake system, this file's main purpose is to add tests.

- The directory `mipstarts/` contains some files used by `CMakeLists.txt` to generate tests.

- In the directory `coverage/` are all necessary files located that are used in producing the coverate report for SCIP.

- The directory `interactiveshell/` contains files for `CMakeLists.txt` that get configured to batchfiles to be executed by SCIP.

- In the `testset/` directory reside the *.test and *.solu files to be specified via `TEST=short`.
  A `.test` file lists problem files, one file per line, absolute and relative paths to its location.
  A `.solu` file with the same basename as the `.test` file contains information about feasibility and best known objective value.
  It is optional for a testrun.

- In the `instances` direcotry are some example instances that can be solved with SCIP.
