# os-autoinst [![CI](https://github.com/os-autoinst/os-autoinst/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/os-autoinst/os-autoinst/actions) [![codecov](https://codecov.io/gh/os-autoinst/os-autoinst/branch/master/graph/badge.svg)](https://codecov.io/gh/os-autoinst/os-autoinst)

The OS-autoinst project aims at providing a means to run fully automated
tests. Especially to run tests of basic and low-level operating system
components such as bootloader, kernel, installer and upgrade, which can
not easily and safely be tested with other automated testing frameworks.
However, it can just as well be used to test firefox and openoffice
operation on top of a newly installed OS.

os-autoinst can be executed alone, but is currently designed to be
executed together with openQA, the web user interface that allows to run
more than one os-autoinst instance at the same time.

More information on os-autoinst and openQA can be found on
<http://os-autoinst.github.io/openQA/>

# Getting started

Under openSUSE the `os-autoinst` package can be installed from the
official repository or from our [devel
repository](https://build.opensuse.org/project/show/devel:openQA). For
further details, have a look at the [openQA
documentation](http://open.qa/docs/#_development_version_repository).

For building os-autoinst manually check out the build instructions
below.

The main executable `isotovideo` can read test parameters from the
command line or read test parameters from a file named `vars.json`. This
file stores the values of the different variables that will configure
the behavior of the test execution.

A container is provided and can be pulled and the main execution can be
called in one step, for example using the `podman` container engine for
tests defined in the current directory on x86_64 if your environment
supports KVM virtualization acceleration:

``` sh
podman run --rm -it -v .:/tests registry.opensuse.org/devel/openqa/containers/isotovideo:qemu-kvm casedir=/tests
```

Use the image variant ending with `qemu-x86` on x86_64 if no KVM support
is available.

Take a look on
<https://registry.opensuse.org/cgi-bin/cooverview?srch_term=project%3Ddevel%3AopenQA>
for all available container images.

Additional test variables can be supplied on the command line. There are
some variables used by os-autoinst itself and other that are used by the
tests. A minimal command line can look like this:

``` sh
isotovideo distri=opensuse casedir=/full/path/for/tests iso=/full/path/for/iso
```

As alternative or completementary a corresponding `vars.json` with
additional parameters could be:

``` javascript
{
   "DISTRI" :      "opensuse",
   "CASEDIR" :     "/full/path/for/tests",
   "NAME" :        "test-name",
   "ISO" :         "/full/path/for/iso",
   "VNC" :         "91",
   "BACKEND" :     "qemu",
   "DESKTOP" :     "kde"
}
```

Be advised that the file `vars.json` is also modified by `os-autoinst`
so make sure to backup handcrafted versions of this file.

For more concrete instructions read on in the "How to run test cases"
section below. Find sections about "How to contribute" or "Build
instructions" further below.

# How to run test cases

This following instructions shows how to run test cases. First one needs
to clone the test distribution. Check out
[os-autoinst-distri-example](https://github.com/os-autoinst/os-autoinst-distri-example)
for an example of a minimal test distribution.

Example for openSUSE’s tests:

```sh
mkdir distri && cd distri
git clone git@github.com:os-autoinst/os-autoinst-distri-opensuse.git opensuse
cd opensuse/products/opensuse
git clone git@github.com:os-autoinst/os-autoinst-needles-opensuse.git needles
```

Example for openQA’s self-tests ("openQA-in-openQA" test):

```sh
mkdir distri && cd distri
git clone git@github.com:os-autoinst/os-autoinst-distri-openQA.git openqa
cd openqa
git clone git@github.com:os-autoinst/os-autoinst-needles-openQA.git needles
```

Then create a working directory for the test execution, e.g.:

```sh
mkdir /tmp/os-autoinst-run && cd /tmp/os-autoinst-run
```

Create a minimal `vars.json` config file within that directory, e.g.:

<div>

<div class="title">

vars.json

</div>

``` json
{
   "ARCH" : "x86_64",
   "BACKEND" : "qemu",
   "CASEDIR" : "/path/to/os-autoinst-distri-opensuse",
   "DESKTOP" : "gnome",
   "DISTRI" : "opensuse",
   "ISO" : "/path/to/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20160715-Media.iso",
   "PRODUCTDIR" : "/path/to/os-autoinst-distri-opensuse/products/opensuse",
   "VNC" : 90,
}
```

</div>

You will need to correct the file paths to point to real locations. Some
of the variables you can use are listed
[here](doc/backend_vars.md). Test case specific variables are
listed in the distri directories e.g.
[os-autoinst-distri-opensuse/variables](https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/variables.md).

Then you can run the `isotovideo` script within the created working
directory. When doing a manual build, that script can be found at the
top-level of the os-autoinst Git checkout.

All of these examples were using the QEMU backend which is usually the
easiest backend to handle and therefore recommended. If you need to
develop and test other backends, have a look at the [backend-specific
documentation](doc/backends.md).

When using the QEMU backend it is possible to access the system under
test via VNC:

```sh
vncviewer localhost:91 -ViewOnly -Shared
```

Run `isotovideo` with the environment variable `RUN_VNCVIEWER` set to
autostart a VNC viewer on the right port.

Run `isotovideo` with the environment variable `RUN_DEBUGVIEWER` to
start the internal debug screenshot viewer updated with an always recent
screenshot of the test run.

# Develop test modules

Individual test modules are written with one test module per file using
the test API in Perl code. Experimental support for test modules in the
Python programming language is provided.

Find more details about how to write tests on
<http://open.qa/docs/#_how_to_write_tests>

# Verifying a runtime environment

To check if your hardware is able to successfully execute os-autoinst
based tests one can execute openQA tests, all the development tests or
simply call something like

```sh
podman run --pull=always --rm -it --entrypoint '' registry.opensuse.org/devel/openqa/containers/os-autoinst_dev:latest /bin/sh -c 'git -C /opt clone --depth 1 https://github.com/os-autoinst/os-autoinst && make -C /opt/os-autoinst/ test-perl-testsuite TESTS=t/99-full-stack.t'
```

which only requires the container runtime environment "podman" and will
run a container based os-autoinst full-stack test, here without KVM
hardware accelerated virtualization support.

# How to contribute

If you want to contribute to this project, please clone and send pull
requests via <https://github.com/os-autoinst/os-autoinst>.

More information on the contribution can be found on
<http://os-autoinst.github.io/openQA/contact/>, too.

Issues are tracked on
<https://progress.opensuse.org/projects/openqav3/>.

For an overview of the architecture, see
[doc/architecture.md](doc/architecture.md).

Additionally the AI generated os-autoinst deepwiki can provide a good
overview about the complete project but without a guarantee for
correctness: <https://deepwiki.com/os-autoinst/os-autoinst>

## Rules for commits

- Every commit is checked by our CI system as soon as you create a pull
  request but you **should** run the os-autoinst tests locally. Check
  out the build instructions for further details.

<!-- -->

- For git commit messages use the rules stated on [How to Write a Git
  Commit Message](http://chris.beams.io/posts/git-commit/) as a
  reference

<!-- -->

- Every pull request is reviewed in a peer review to give feedback on
  possible implications and how we can help each other to improve

If this is too much hassle for you feel free to provide incomplete pull
requests for consideration or create an issue with a code change
proposal.

## Deprecation approach

In case you want to deprecate functionality consider the use of the
function `backend::baseclass::handle_deprecate_backend`.

# Build instructions

## Installing dependencies

On openSUSE one can install the package `os-autoinst-devel` which
provides all the dependencies to build and run os-autoinst for the
corresponding version of the sources. To build a current version of
`os-autoinst` it is recommended to install `os-autoinst-devel` from
[devel:openQA](https://build.opensuse.org/project/show/devel:openQA) as
the distribution-provided packages might be too old or miss
dependencies. This is particularly true for openSUSE Leap. Also see [the
openQA docs](https://open.qa/docs/#_development_version_repository).

The required dependencies are also declared in `dependencies.yaml`. (The
names listed within that file are specific to openSUSE but can be easily
transferred to other distributions.)

## Conducting the build

Simply call

```sh
make
```

in the top folder which automatically creates a build directory and
builds the complete project.

Call

```sh
make help
```

to list all available targets. One especially useful target for
developers is `make` `setup-hooks` which installs git pre-commit hooks
to check commit styles.

The above commands use a convenience Makefile calling `cmake`. For
packaging, when using an IDE or to conduct the steps manually it is
suggested to use CMake directly and do the following: Create a build
directory outside of the source directory. The following commands need
to be invoked within that directory.

Configure build: `cmake $path_to_os_autoinst_checkout`

You can specify any of the standard CMake variables, e.g.
`-DCMAKE_BUILD_TYPE=Debug` and
`-DCMAKE_INSTALL_PREFIX=/custom/install/prefix`.

The following examples assume that GNU Make is used. It is possible to
generate for a different build tool by adding e.g. `-G` `Ninja` to the
CMake arguments.

Build executables and libraries: `make symlinks`

This target also creates symlinks of the built executables and libraries
within the source directory so `isotovideo` can find them.

Run all tests: `make check`

By default CTest is invoked in verbose mode because prove already
provides condensed output. Add `-DVERBOSE_CTEST=OFF` to the CMake
arguments to avoid that.

Run all Perl tests (`*.t` files found within the `t` and `xt`
directories): `make test-perl-testsuite`

Run individual tests by specifying them explicitly:
`make test-perl-testsuite TESTS="t/15-logging.t t/28-signalblocker.t"`

Run perl author tests: `make test-local-author-perl`

Run all author tests: `make test-local`

Notice that the user needs to include the test directory for each test
(either t for normal or xt for developer-centric tests) when specifying
individual tests.

Add additional arguments to the `prove` invocation, e.g. enable verbose
output: `make test-perl-testsuite PROVE_ARGS=-v`

Gather coverage data while running tests:
`make test-perl-testsuite WITH_COVER_OPTIONS=1`

Generate a coverage report from the gathered coverage data: `make coverage`

If no coverage data has been gathered so far the `coverage` target will
invoke the testsuite automatically.

Reset gathered coverage data: `make coverage-reset`

Install files for packaging: `make install DESTDIR=…`

Automatically tidy all perl files: `tools/tidyall`

Tidy all changed perl files: `tools/tidyall --git`

Further notes:

- When using the `test-perl-testsuite` target, `ctest` is not used (and
  therefore `ctest` specific tweaks have no effect).

- One can always run Perl tests manually via `prove` after the build has
  been conducted with `make` `symlinks`. Note that some tests need to be
  invoked within the `t` directory. An invocation like `prove` `-vI..`
  `-I../external/os-autoinst-common/lib` `28-signalblocker.t` is
  supposed to work.

- It is also possible to run `ctest` within the build directory directly
  instead of using the mentioned targets.

- All mentioned variables to influence the test execution (`TESTS`,
  `WITH_COVER_OPTIONS`, …) can be combined and can also be used with the
  `coverage` target.

# Running isotovideo as CI check

We provide a container to run `isotovideo` which can be used to run
QEMU-based tests directly in a CI runner. Checkout this [example
workflow](https://github.com/os-autoinst/os-autoinst-distri-example/blob/main/.github/workflows/isotovideo-action.yml)
for how it can be used. The [README of the example test
distribution](https://github.com/os-autoinst/os-autoinst-distri-example/blob/main/README.md#local-testing-and-ci-environment)
also contains further details.

# Standalone tool for image search

The script `imgsearch` in the repository’s script folder allows to use
the fuzzy image comparison independently of the normal test execution.
Invoke the script with no parameters to show its usage. There is also an
[example
file](https://github.com/os-autoinst/os-autoinst/blob/master/t/imgsearch/expected-output.json)
showing what output you can expect. There is one key for each file to be
searched. The best matching image to be found will show up as `match`
and the other images under `candidates`. If no image matches well
enough, `match` will be `null`.

To use the script the previously shown build instructions need to be
executed (including the invocation of the `symlinks` target).

# History of os-autoinst

At a time Bernhard M. Wiedemann who later joined was on the openSUSE
testing team and was assigned the task of testing the installer. Which
meant tedious and dull work of waiting for 4GB ISO files to download
when it’s not even clear if those things even boot. And as the Perl
founder Larry Wall states, important traits of programmers are laziness,
impatience and hybris. Which quickly led to developing os-autoinst to
automate installations ;) See
<https://lizards.opensuse.org/2010/04/29/making-of-the-opensuse-install-video/>
and
<https://lizards.opensuse.org/2010/05/25/automated-opensuse-testing/>
for Bernhard’s blog posts.

# Further notes

When using the QEMU backend, also ensure your user running os-autoinst
has access to `/dev/kvm`.

```sh
modprobe kvm-intel || modprobe kvm-amd
chgrp kvm /dev/kvm ; chmod g+rw /dev/kvm # maybe redundant
# optionally use a new user; just to keep things separate
useradd -m USERNAME -G kvm
passwd USERNAME # and/or add ~USERNAME/.ssh/authorized_keys
```
