KNEM: Fast Intra-Node MPI Communication

# KNEM

# Documentation

See also the interface documentation for details on how to use KNEM within your applications or MPI implementations.

### Installing KNEM

KNEM configures, builds and installs as usual.

     $ ./configure --prefix=/opt/knem
     $ make
     $ make install

If building from a GIT glone, you may need to install automake >=1.10 and autoconf >=2.61, and then prepare the script before configure:

     $ ./autogen.sh

Everything is installed in the prefix directory (`/opt/knem` by default), including the kernel module and udev rules file. If you want modprobe and udev to properly handle KNEM, the relevant files should be installed in `/lib/modules` and `/etc/udev/rules.d` instead. To do so:

     $ /path/to/knem/install/sbin/knem_local_install

Usually, `make install` will run only once per cluster and install in a shared directory, while `knem_local_install` will run once per machine since `/lib/modules` and `/etc/udev/rules.d` are not shared between machines.

If you ever want to uninstall KNEM, you may run:

     $ rmmod knem
     $ /path/to/knem/install/sbin/knem_local_install --uninstall
     $ make uninstall

It is also possible to build KNEM as a RPM package which bundles the result of both `make install` and `knem_local_install`:

     $ rpmbuild -tb knem-<version>.tar.gz

KNEM also supports DKMS (Dynamic Kernel Module Support) which lets the system automatically rebuild and install modules for new kernels. See `dkms.conf` for details.

### Starting KNEM

To start KNEM, assuming you ran `knem_local_install`, just run:

     $ modprobe knem

By default, udev will set the `/dev/knem` device group to rdma. Thus any KNEM user should be added to this group first.

If you did not run `knem_local_install`, you may still load the kernel module and setup the device group manually:

     $ insmod /path/to/knem/install/lib/modules/$(uname -r)/knem.ko
     $ chgrp rdma /dev/knem

If the rdma group does not exist on your machine, you may modify the udev rule to set the mode to `0666` and not to change the file group.

### Configuring your MPI implementation to use KNEM

MPICH2 supports KNEM since version 1.1.1 (it is therefore also supported by MVAPICH2 1.5). To use it, you should add the following options to the configure line:

     --with-nemesis-local-lmt=dma --with-knem=/path/to/knem/install/

It will use KNEM starting at 64kB messages, and offload copies to DMA engine after 2MB. To change the latter threshold, use the environment variable `MPICH_NEM_LMT_DMA_THRESHOLD`.

Open MPI supports KNEM since version 1.5.0. To use it, you should add the following option to the configure line:

     --with-knem=/path/to/knem/install/

KNEM will be used at runtime after 4kB. You may change this threshold with `--mca btl_sm_eager_limit 32768`. You may also disable KNEM entirely at runtime with `--mca btl_sm_use_knem 0`. KNEM will not offload copies to DMA engine by default. You should use `--mca btl_sm_knem_dma_min 1048576` to enable copy offload starting at 1MB.

To check that KNEM is properly used, see Getting Statistics. Make sure your users are members of the rdma group so that they are granted access to the `/dev/knem` device file.

If you want to port your MPI implementation to KNEM, see the interface documentation.

### Getting Statistics

To check that KNEM is properly used, how it processed requests, why some of them failed, you may read statistics:

      $ cat /dev/knem
      [...]
       Requests submitted                           : 10243
       Requests processed (total)                   : 10236
                processed (using DMA)               : 0
                processed (offloaded to thread)     : 0
      [...]
       Requests rejected (invalid flags)            : 5
                rejected (not enough memory)        : 2
      [...]
       Requests failed during memcpy from/to user   : 1
                failed during DMA copy              : 0

These statistics may also be cleared by a privileged user:

      # echo 1 > /dev/knem

If the knem module was loaded with parameter `statsverbose=1` (appended to the `insmod` or `modprobe` command-line), reading from `/dev/knem` will also list the currently open memory regions for the current process (and even from all processes if the user is privileged).

      # cat /dev/knem
      [...]
       Context 5d343d48
        Region 5d344ad1 SingleUse
        Listed 1 regions
       Context 5d343d49
        Listed 0 regions
       Listed 8 contexts

Last updated on 2011/07/12.