# Nemesis Runtime Netmod Selection

This page explains the mechanism used by Nemesis to select network
modules at runtime. With this mechanism the list of network modules is
specified in configure and are then statically compiled in to the MPICH
library. At runtime the user sets an environment variable to choose
which network module to use.

## Building and Execution

The `--with-device=` configure option can be used to specify multiple
network modules to be compiled into the library. The list of network
modules is specified following `ch3:nemesis:` as a comma-separated list
(with no spaces). The first network module listed will be the default.
E.g.:

```
./configure --with-device=ch3:nemesis:tcp,mx
```

At runtime, the network module is chosen using the
`MPICH_NEMESIS_NETMOD` environment variable. E.g.:


```
MPICH_NEMESIS_NETMOD=mx mpiexec -n 4 ./cpi
```

If `MPICH_NEMESIS_NETMOD` is not set, the first network module listed in
configure is used.

## Implementation

Each network module defines a function pointer table,
`MPID_nem_netmod_funcs_t`. This function pointer table is named
`MPIDI_nem_<module name>_<module funcs>`. E.g. for the tcp network
module, the function pointer table is defined in `tcp_init.c` as:

```
MPID_nem_netmod_funcs_t MPIDI_nem_tcp_funcs = {
    MPID_nem_tcp_init,
    MPID_nem_tcp_finalize,
    MPID_nem_tcp_connpoll,
    MPID_nem_tcp_send,
    MPID_nem_tcp_get_business_card,
    MPID_nem_tcp_connect_to_root,
    MPID_nem_tcp_vc_init,
    MPID_nem_tcp_vc_destroy,
    MPID_nem_tcp_vc_terminate
   };
```

`MPID_nem_netmod_funcs` is an array of the function pointer tables for
each configured netmod. It's defined in mpid_nem_net_array.c which is
generated by configure from mpid_nem_net_array.c.in. In this file,
`MPID_nem_netmod_strings` is also defined which holds the names of the
netmods.

Which netmod to use is determined in `MPID_nem_choose_netmod()` by
reading the `MPICH_NEMESIS_NETMOD` environment variable and searching
for a match in `MPID_nem_netmod_strings`. Then `MPID_nem_netmod_func` is
set to point to the corresponding entry in the `MPID_nem_netmod_funcs`
array.

Any call to a network module function is done by dereferencing
`MPID_nem_netmod_func`.

The variable `MPID_nem_netmod_id` is the index of the function pointer
table for the current netmod in the `MPID_nem_netmod_funcs` array:

```
MPID_nem_netmod_func == MPID_nem_netmod_funcs[MPID_nem_netmod_id]`
```

### Variables

- `MPID_nem_netmod_funcs_t` - Struct for function pointers for a single
  netmod (`mpid_nem_nets.h`)
- `MPID_nem_netmod_funcs_t MPIDI_nem_<MODULE NAME>_module_funcs` -
  Function pointer table for netmod `<MODULE NAME>` (defined in each
  netmod)
- `MPID_nem_netmod_funcs_t *MPID_nem_netmod_funcs[]` - Array of
  pointers to function pointer tables from each configured netmod
  (generated by configure from `mpid_nem_net_array.c.in`)
- `char MPID_nem_netmod_strings[][]` - Array of string names for each
  netmod (generated by configure from `mpid_nem_net_array.c.in`)
- `MPID_nem_netmod_funcs_t *MPID_nem_netmod_func` - Global pointer to
  the function pointer table for the selected netmod. Dereference this
  to call the netmod function. (`mpid_nem_network.c`)
- `MPICH_NEMESIS_NETMOD` - Environment variable specifying which netmod
  to use
- `MPID_nem_netmod_id` - Index of the function pointer table for the
  current netmod. (`mpid_nem_network.c`)

```
MPID_nem_netmod_func == MPID_nem_netmod_funcs[MPID_nem_netmod_id]
```

- `MPIDI_NEM_<MODULE NAME>_MODULE` - Index of the function pointer
  table for the netmod *MODULENAME* (generated by configure from
  `mpid_nem_net_module_defs.h.in`)

## Future Work

- Multiple netmods - This can be done having each vc contain a pointer
  to the appropriate netmod function pointer (that part's easy). How
  do we decide/specify which netmod to use for which vc?
- Dynamically loaded netmods
