.. _DeveloperDocumentation:

Developer Documentation
=======================

Developers need to know more than users.  For example, they need to know about the test suite, and writing test cases.  They need to know how to perform a developer installation.  They need to know how to edit and compile this documentation.  They need a step-by-step release process.



.. index:: Producing generating reproducible C++ code

.. _ReproducibleCCode:

Generating reproducible C++ code
--------------------------------

Since moving to Python 3, the C++ code that XMDS2 generates can vary from invocation to invocation. That means you can run "xmds2 testscript.xmds", look at the generated C++ code, run "xmds2 testscript.xmds" again and obtain slightly different C++ code. The reason for this is that XMDS2 uses sets internally to store a number of object types such as vectors and transforms. In Python 2, when you iterate over a set, the ordering was in principle not guaranteed, but in practice is almost always was. Python 3, however, randomizes its hash seeds on every run, so that the order you get things out of a set really won't be same run to run.

This means that the C++ code that XMDS2 generates will differ run-to-run, but when compiled and executed will still produce the same result. When working on XMDS it is often easier to debug things if the C++ code doesn't change. To make this happen you have to tell Python not to randomize its hash table seeds, which you can do by setting an environment variable via

.. parsed-literal::

  export PYTHONHASHSEED=0

This ensures any Python programs launched in that terminal will produce reproducible code.



.. index:: Test scripts

.. _TestScripts:

Test scripts
------------

Every time you add a new feature and/or fix a new and exciting bug, it is a great idea to make sure that the new feature works and/or the bug stays fixed.  Fortunately, it is pleasantly easy to add a test case to the testing suite.

1. Write normal XMDS script that behaves as you expect.
2. Add a ``<testing>`` element to your script.  You can read the description of this element and its contents below, and have a look at other testcases for examples, but the basic structure is simple:.

.. parsed-literal::

      <:ref:`testing <TestingElement>`> 
        <:ref:`command_line <CommandLineElement>`> <:ref:`/command_line <CommandLineElement>`>
        <:ref:`arguments <ArgumentsElement>`>
          <:ref:`argument <ArgumentElement>` />
          <:ref:`argument <ArgumentElement>` />
          ...
        <:ref:`/arguments <ArgumentsElement>`>
        <:ref:`input_xsil_file <InputXSILFileElement>` />
        <:ref:`xsil_file <XSILFileElement>`>
          <:ref:`moment_group <MomentGroupElement>` />
          <:ref:`moment_group <MomentGroupElement>` />
          ...
        <:ref:`/xsil_file <XSILFileElement>`>
      <:ref:`/testing <TestingElement>`>
      
3. Put into the appropriate ``testsuite/`` directory.
4. run ``./run_tests.py`` This will automatically generate your ``_expected`` files.
5. Commit the ``.xmds``, ``*_expected.xsil`` file and any ``*_expected*`` data files.

There are examples of test cases in the testsuite directory. Feel free to add to them! Ideally a testcase should run quickly (a second or less, unless it's specifically testing a huge number of dimensions or something similar), and not produce very much data (since we have to bundle test results with the XMDS2 code, resulting in larger downloads). This is because there are a *lot* of test cases and running the full testsuite is taking an increasing amount of time.

  
.. _TestingElement:

<testing> element
~~~~~~~~~~~~~~~~~

The element that wraps everything, e.g.

.. parsed-literal::

    <testing>
       ...
    </testing>

This element is required for a test case script.



.. _CommandLineElement:

<command_line> element
~~~~~~~~~~~~~~~~~~~~~~

The test runner assumes it can fire off your test script by just using the filename (without the .xmds extension). So if you have a test case script called MyTest.xmds, it assumes that it can be run via

.. parsed-literal::
    xmds2 MyTest.xmds
    ./MyTest

If the command to run your executable isn't "./MyTest" you can override it with the <command_line> element. For example, if you need your script to be run as an MPI job with 8 processors, you could specify

.. parsed-literal::
    <testing>
       <command_line> mpirun -n 8 ./MyTest </command_line>
       ...
    </testing>

This element is optional.


<arguments> element
~~~~~~~~~~~~~~~~~~~

This can be used to add additional arguments to the command used to run the executable. So, for example, if you wanted to pass lattice sizes to the script, you could use

.. parsed-literal::
    <testing>
      <arguments> --latticesize_x 10 --latticesize_y 10 </arguments>
      <command_line> mpirun -n 8 ./MyTest </command_line>
      ...
    </testing>

which would get the test runner to execute your test case with

.. parsed-literal::
    mpirun -n 8 ./MyTest --latticesize_x 10 --latticesize_y 10 

This element is optional.



.. _InputXSILFileElement:

<input_xsil_file> element
~~~~~~~~~~~~~~~~~~~~~~~~~

This is a list of all .xsil files used by your testcase as input, e.g. if you are using a file to initialise your vectors. So, for example, if you had previously output the files "input_file_breakpoint.xsil" and the associated "input_file_breakpoint.h" and your test case script needs to load them for vector intialisation, you'd need to inform the test runner of this fact via

.. parsed-literal::
  <testing>
    <input_xsil_file name="input_file_breakpoint.xsil" />
    ... 
  </testing>

This element is optional.



.. _XSILFileElement:

<xsil_file> element
~~~~~~~~~~~~~~~~~~~

This element is used to compare the results of the test with some previously generated expected results. An example would be

.. parsed-literal::
  <testing>
    <xsil_file name="MyTest.xsil" expected="MyTest_expected.xsil" absolute_tolerance="1e-6" relative_tolerance="1e-5" />
    ...

The "name" attribute specifies the filename of the source data. Generally speaking it will be name of the output file that your test case creates when it runs. The "expected" attribute specifies the filename of the canonical correct data against which you wish to compare.

You can specify the "absolute_tolerance" attribute or the "relative_tolerance" attribute or both. If the comparison between the generated data and expected data exceeds these specifications the test will be counted as failed.

This element is optional but recommended. If you don't specify it, the only way the test will fail is if it fails to compile and link, or crashes for some other reason (division by zero etc). 



.. _MomentGroupElement:

<moment_group> element
~~~~~~~~~~~~~~~~~~~~~~

This element is used if you want to specify the tolerances per moment group, rather than having the same tolerance for all moment groups. Tolerances specified per moment group override the tolerances specified in the <xsil_file_name> element. The syntax is

.. parsed-literal::
  <testing>
      <xsil_file name="MyTest.xsil" expected="MyTest_expected.xsil" absolute_tolerance="1e-6" relative_tolerance="1e-5" />
      <moment_group number="1" absolute_tolerance="1e-7" relative_tolerance="1e-6" />
      <moment_group number="2" absolute_tolerance="1e-3" relative_tolerance="1e-6" />
      ...

This element is optional.


.. index:: Documentation format

.. _UserDocumentation:

XMDS Documentation
------------------

.. Author: Justin Lewer, 2013.

Documentation in XMDS is written as reStructuredText files (.rst), which are then parsed into HTML files to be displayed on their website. 

You can find the user documentation folder located ``admin/userdoc-source``. This is where all of the .rst files are kept. If you’re wanting to add documentation to the site, you’ll need to create your own .rst file, with the name of the webpage as the filename.

RST is a relatively simple language, which is basically simplified HTML markup. For documentation on how to make Lists, Href Links, Embed images etc, you should check here;

http://docutils.sourceforge.net/docs/user/rst/quickref.html
http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html

However, you should easily be able to use some of the pre-existing .rst files in the project as a template to create yours.

Once your documentation is in this folder, it should be deployed along with the project to their website when you run create_release_version.sh, which can be found in the /Trunk/xpdeint/admin folder. If you would like to test to see what your rst file generates without running this shell script, you can use the Makefile in the userdoc-source folder, by running “make html”.

NOTE: Before you can run the create_release_version.sh file, there are a few packages you will need. This command uses latex to generate the XMDS2 pdf, so you’ll be needing the following packages; ``texlive-fonts-recommended``, ``texlive-lang-cjk``, ``texlive-latex-base``.


.. index:: XML schema updates

.. _HowToAddElementToValidator:

How to update ``XMDS2`` script validator (XML schema)
-------------------------------------------------------

.. Author: Damien Beard. 2013.

.. image:: images/IntroduceNewElement.png
           :align: right

This is a short guide to adding an element to XMDS2, so that it can be validated by the XMDS2 script validator. In this guide, the example being used will be the addition of a matrix element to the validator.  The matrix will have a  ‘name’ and a ‘type’ (so it can be called later, and the type is known for future reference). Each matrix will also need a ‘row’ component, and possibly an initialisation value.

Navigate to ``xpdeint/support/xpdeint.rnc``. This is a RelaxNG compact file, which specifies the XML schema which is only used for issuing warnings to users about missing or extraneous XML tags / attributes. Add the following lines to the end of the file (so that it is outside all other brackets in the file):

.. code-block:: none
    
    Matrix = element matrix {
        attribute name { text }
        , attribute type { text }?
        , element components { text }
        , element initialisation {
            attribute kind { text }?
        }?
    }

Save this file, and then in the terminal navigate to the folder ``xpdeint/support/`` and run ``make``. This updates the XML based file ``xpdeint/support/xpdeint.rng``, which is the file the parser uses to validate elements in XMDS2. This file which is  used is in RelaxNG format, but RelaxNG compact is easier to read and edit.

Commit both ``xpdeint/support/xpdeint.rnc`` and ``xpdeint/support/xpdeint.rng`` to the code repository.


.. index:: Adding an integrator

.. _HowToAddIntegrator:

How to introduce a new integrator Stepper into the XMDS2 environment
--------------------------------------------------------------------

.. Author: Damien Beard. 2013.

.. image:: images/IntroduceNewIntegrationTechnique.png
           :align: right

This is a short guide to adding a new stepper containing a new mathematical technique to XMDS2, which can then be used by to integrate equations. This guide describes the logistics of introducing a new stepper and as such, the code inside the stepper template is outside the scope of this document. The new stepper which will be used in this guide will be called ‘IntegrateMethodStepper’.

Navigate to the ``xpdeint/Segments/Integrators`` directory. Create a file called ``IntegrateMethodStepper.tmpl`` in this directory. In this file, implement the new integration algorithm (follow the convention of existing steppers in that folder). In this same folder, open the file named ``__init__.py`` and add the following line to the bottom of the file and save it:

.. code-block:: none

    import IntegrateMethodStepper

Navigate up until you are in the ``xpdeint`` directory. Open the file ``XMDS2Parser.py``, and 'find' the algorithm map (Ctrl+F > algorithmMap works for most text editors). The mnemonic ‘IM’ will be used for our Stepper. If the stepper uses fixed step sizes, then add the following line to the algorithm map:

.. code-block:: none

    'IM':   (Integrators.FixedStep.FixedStep, Integrators.IntegrateMethodStepper.IntegrateMethodStepper),

Otherwise, if your stepper is an adaptive Stepper, add the following line:

.. code-block:: none

    'IM':   (Integrators.AdaptiveStep.AdaptiveStep, Integrators.IntegrateMethodStepper.IntegrateMethodStepper),

In the terminal, navigate to the ``xpdeint`` directory, and run make over the entire directory. 'IM' can now be used to specify the new Stepper as your integration algorithm inside your .xmds files, e.g.

.. code-block:: xpdeint

    <integrate algorithm="IM" interval="5.0" steps="2000">
        ...
    </integrate>


.. _LogicalBreakDownParsingProcess:


.. index:: XMDS2 parsing process

Logical breakdown of XMDS2 Parsing Process
------------------------------------------

.. Author: Damien Beard. 2013.

The following information is intended to assist developers in understanding the logical process undertaken by the XMDS2 system when parsing an .xmds file. The documentation was not designed to be exhaustive, but rather to help paint a picture of part of the way XMDS2 works. 

The flowcharts have been created in open source diagram drawing program Dia, and compiled into .png files which are displayed below. This page contains links to the original .dia files, so if you find any error in the information below (or you'd like to extend it, by adding in more information), please update the .dia files and commit them (and their compiled versions) to svn. 

Overall process for parsing XML file in XMDS2
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. image:: images/Overall_Flowchart.png
   :align: center

The original .dia file can be downloaded `here <_images/Overall_Flowchart.dia>`__.

parser2.py parses XML file (Sub process 3)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. image:: images/Subprocess3_ParsingXMLFile.png
   :align: center

You can download the original dia file `here <_images/Subprocess3_ParsingXMLFile.dia>`__.

Pass file to XMDS2Parser to parse xmlDocument with parseXMLDocument() (Sub process 3.4)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. image:: images/Subprocess3_4_parseXMLDocument.png
   :align: center

You can download the original dia file `here <_images/Subprocess3_4_parseXMLDocument.dia>`__.

Parse Top Level Sequence elements (Sub process 3.4.11)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. image:: images/Subprocess3_4_11_ParseTopLvlSeqElements.png
   :align: center

You can download the original dia file `here <_images/Subprocess3_4_11_ParseTopLvlSeqElements.dia>`__.

Parse Integrate Element (Sub process 3.4.11.2)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. image:: images/Subprocess3_4_11_2_ParseIntegrateElement.png
   :align: center

You can download the original dia file `here <_images/Subprocess3_4_11_2_ParseIntegrateElement.dia>`__.



.. index:: XMDS2 project directory layout

Directory layout
----------------

XMDS2's code and templates
~~~~~~~~~~~~~~~~~~~~~~~~~~

All ``.tmpl`` files are Cheetah template files.  These are used to generate C++ code.  These templates are compiled as part of the XMDS2 build process to ``.py`` files of the same name.  Do not edit the generated ``.py`` files, always edit the ``.tmpl`` files and regenerate the corresponding ``.py`` files with ``make``.

* ``xpdeint/``: 
	* ``Features/``: Code for all ``<feature>`` elements, such as ``<globals>`` and ``<auto_vectorise>``
		* ``Transforms/``: Code for the Fourier and matrix-based transforms (including MPI variants).
	* ``Geometry/``: Code for describing the geometry of simulation dimensions and domains.  Includes code for ``Geometry``, ``Field`` and all ``DimensionRepresentations``.
	* ``Operators/``: Code for all ``<operator>`` elements, including ``IP``, ``EX`` and the temporal derivative operator ``DeltaA``.
	* ``Segments/``: Code for all elements that can appear in a ``<segments>`` tag.  This includes ``<integrate>``, ``<filter>``, and ``<breakpoint>``.
		* ``Integrators``: Code for fixed and adaptive integration schemes, and all steppers (e.g. ``RK4``, ``RK45``, ``RK9``, etc.)
	* ``Stochastic/``: Code for all random number generators and the random variables derived from them.
		* ``Generators/``: Code for random number generators, includes ``dSFMT``, ``POSIX``, ``Solirte``.
		* ``RandomVariables/``: Code for the random variables derived from the random number generators.  These are the gaussian, poissonian and uniform random variables.
	* ``SimulationDrivers/``: Code for all ``<driver>`` elements.  In particular, this is where the location of MPI and multi-path code.
	* ``Vectors/``: Code for all ``<vector>`` elements, and their initialisation.  This includes normal ``<vector>`` elements as well as ``<computed_vector>`` and ``<noise_vector>`` elements.
	* ``includes/``: C++ header and sources files used by the generated simulations.
	* ``support/``: Support files
		* ``wscript``: ``waf`` build script for configuring and compiling generated simulations
		* ``xpdeint.rnc``: Compact RelaxNG XML validation for XMDS scripts.  This is the source file for the XML RelaxNG file ``xpdeint.rng``
		* ``xpdeint.rng``: RelaxNG XML validation for XMDS scripts.  To regenerate this file from ``xpdeint.rnc``, just run ``make`` in this directory.
	* ``waf/``: Our included version of the Python configuration and build tool ``waf``.
	* ``waf_extensions/``: ``waf`` tool for compiling Cheetah templates.
	* ``xsil2graphics2/``: Templates for the output formats supported by ``xsil2graphics2``.
	* ``wscript``: ``waf`` build script for XMDS2 itself.
	* ``CodeParser.py``: Minimally parses included C++ code for handling nonlocal dimension access, IP/EX operators and IP operator validation.
	* ``Configuration.py``: Manages configuration and building of generated simulations.
	* ``FriendlyPlusStyle.py``: Sphinx plug-in to improve formatting of XMDS scripts in user documentation.
	* This directory also contains code for the input script parser, code blocks, code indentation, and the root ``_ScriptElement`` class.


Support files
~~~~~~~~~~~~~

* ``admin/``: Documentation source, Linux installer and release scripts.
	* ``developer-doc-source/``: source for epydoc python class documentation (generated from python code).
	* ``userdoc-source/``: source for the user documentation (results visible at www.xmds.org and xmds2.readthedocs.org).
	* ``xpdeint.tmbundle/``: TextMate support bundle for Cheetah templates and XMDS scripts
* ``bin/``: Executable scripts to be installed as part of XMDS2 (includes ``xmds2`` and ``xsil2graphics2``).
* ``examples/``: Example XMDS2 input scripts demonstrating most of XMDS2's features.
* ``testsuite/``: Testsuite of XMDS2 scripts.  Run the testsuite by executing ``./run_tests.py``
