@section Binary IO Design Notes

The aim of the binary IO is to provide a consistent mechanism for saving and
restoring all VXL objects to streams/files, in a cross platform manner.

@subsection Structure

The original intention was that ALL classes with data should implement
@code{b_write(os)} and @code{b_read(is)} as member functions, and that
the external @code{vsl_b_write(os,object);} would be implemented using
the member function.  The advantages of this scheme are that

@itemize @bullet
@item Member functions can save all the state of a class without making everything public
@item Keeping the IO in the class makes it more likely that it will be kept up to date
@end itemize

Unfortunately a problem arises with templated containers, such as @code{vbl_array_2d<T>}.
Creating a container of type T requires that @code{vsl_b_write(os,T)} is implemented,
otherwise the @code{b_write} for the array will not compile.  This means that
adding such code to the library would break existing code, and force un-necessary
implementation of reading/writing functions (even when creating arrays of
objects which are unlikely ever to be saved, such as GUI Widgets).

A related problem is that to keep the core libraries independent, one would have
to duplicate the basic binary IO of built in types in every library.  When templating
over a class outside the library, one might have to write a specialisation of the IO
functions to get them to compile.

The simplest solution is to write the code as `clip-on' libraries, as has been done.

However, for level-2 libraries, in general, neither of these problems will arise.
In this case it is strongly recommended by the authors that IO be built into the
class.

It is interesting to note that after vsl was written, and entirely independently
as far as these authors are aware, a section on IO was added to the comp.lang.c++
FAQ.  The recommendations in the FAQ match the design of vsl very closely, in the
design of binary formats, the serialisation scheme, and the base-class loader
scheme.

@subsection Format of Fundamental types

All values are stored in little endian format (as used by Intel and DEC
alpha processors.) We had to choose one, and we mostly use Intel platforms.

@subsubsection Format of Integer types

The original code from which @code{vsl} was derived was cross platform only in the sense
that it worked on either big-endian 32-bit platforms or little-endian 32-bit platforms.
This was not an unreasonable assumption at the time it was written.  However during the
requirements capture for @code{vsl}, it became clear that being able to handle 64-bit
fundamental types would be necessary.  For example the long on an alpha-64 is 64 bits.

It is fundamentally impracticable to deal quietly and correctly with the problems this
causes.  In particular, a number greater than 4G can be represent by a 64-bit long.  It
can be successfully saved, but a platform with a 32-bit long simply cannot represent
the value.  Nevertheless it is necessary to deal with the problems in a predictable manner.
There were several options.

@enumerate
@item Fix the size of the various types.
@item Record the size of the various types in a header at the start of the file, and then
save values in native type.
@item Use an encoding scheme which works for any size.
@end enumerate

The first option has the advantage of being the simplest (on the platform which matches
the sizes chosen.) It has the disadvantage of either being twice as large as required on
some platforms, and not able to represent the full range of values on others.

The second option, has the advantages of doing the correct thing on all platforms,
and being very efficient when files are loaded and
saved on the same platform.  This is especially true if the endian-ness was also flagged
in the header and values saved in native format.  The disadvantage is that the code for
reading files on each platform must know about every other possibility.  So if there are
@math{n} platforms supported, the numbers of options that need to be programmed is
@math{n^2}.

The third option, suggested by Peter Vanroose, has the advantage that it does the correct
thing on all platforms.  Also, there is only one format,
and so the amount of code to write only grows linearly with the number of platforms rather
than quadratically as in option 2.  The final
advantage is that the store file will likely be smaller, since numbers are represented
in as few bytes as possible.  It has
the disadvantage of some extra computation and memory overhead when both reading and writing.

The performance of the encoding and decoding required for option 3 was measured.
On an Intel Pentium 3, using MSVC 6.0 in full optimisation mode,
25 million integers were encoded or decoded at once.
The results do not include memory allocation overhead but does include some OS overhead.

@example
Type     Range of values  Range of encoded size  Encoding time   Decoding time
                                 / bytes         /clock cycles   /clock cycles
unsigned    0  -> 127             1                  18              24
int        -64 ->  63             1                  19              28
unsigned    0  ->  4G     1,2,3,4,but mostly 5       37              95
int        -2G ->  2G     1,2,3,4,but mostly 5       47             105
@end example

This means that a 850 MHz PC can encode about 50 million small valued integers per
second, and decode about 35 million.  The decoding takes longer due to the need for testing
for integer overflow.

We consider these speeds to be fast enough. and so option 3 was chosen.

The arbitrary length format stores the number in a series of bytes.  Of each byte, 7 bits
are used to represent the number and the most significant bit is used to flag the end of the number.
The flag bit is 0 if there
are more bytes left for this integer, and 1 if this is the last byte.  The bytes are stored
in little endian order, and the signed numbers are stored using 2s complement notation.

A consequence of this choice is that for example a number can be written from a short, and read
into a long.  We strongly advise against doing this, as it relies on the implementation details.

@subsubsection Format of Floating Point Types

Since almost all platforms use IEEE format floating point types to represent floats
and doubles, we have used this format (in little endian order) to save them to disk.

A downside of this decision is that it is unclear how to store long doubles.  The 80bit format
that is native to Intel platforms would not appear to be sufficiently general.
The best alternative would be to switch to variable length encoded floating point values.
It might be easier just to design a 128 bit format.  Until then I/O of long doubles is not
supported.  Please contact the designers if you need to add it.

@subsection Changing the IO format

The @code{vsl} system stores a IO schema version number at the start of the stream.  This will enable
a clean backwards compatible upgrade of the IO schema.  It will not however make the rewrite of
@code{vsl} trivial.  Contact the designers if you wish to change the schema.

@subsection File Magic Numbers

If you use Unix, the following can be inserted in your /etc/magic file so that the @command{file} command
will recognise @code{vsl} files.

@example
#------------------------------------------------------------------------------
# VXL: file(1) magic for VXL binary IO data files
#
# from Ian Scott <scottim@@sf.net>
#
# VXL is a collection of C++ libraries for Computer Vision.
# See the vsl chapter in the VXL Book for more info
# http://www.isbe.man.ac.uk/public_vxl_doc/books/vxl/book.html
# http:/vxl.sf.net

2       lelong  0x472b2c4e      VXL binary file,
>0      leshort >0      schema version no %d

@end example

@subsection Serialisation

A common approach to performing serialisation, is to store the serial number in the shared object
itself.  This is the approach used by the Microsoft Foundation Classes, and by DEX (the IUE's IO
and serialisation scheme.)

There are two disadvantages to this.  The first is that you have to modify
the object being serialised.

The second is that this scheme can get confused if a set of objects
is being written to two streams.  When do you clear the already saved flag for each object?
What happens if the output to the two streams is being interleaved? Will a shared object get
saved to one stream and not the other?

A different approach is used by @code{vsl}.  Here the serialisation record is kept by the stream object.
Pointers are used to uniquely identify each shared object within each computer, and a serial number
is generated to be saved on the stream.  Since there is now a record for each pair
(stream, shared object) there will not be any clashes when saving to multiple streams.  It also
avoids having to modify the shared object.

@subsection Loading by base class pointer: Design Overview

When using polymorphism, there are frequently times when one needs to
save and restore an object just using a base class pointer to it.
@code{vsl} provides facilities to do this.

There are two cases to consider
@itemize @bullet
@item IO is provided by the class itself
@item IO is provided by external `clip-on' classes.
@end itemize

The former is the preferred method.

The latter is provided to allow binary IO for third-party libraries which cannot be
modified.  However, for technical reasons (see design notes below) it is also
used to provide binary IO for the polymorphic hierarchies in the VXL core
libraries.

@subsubsection IO provided within the class

@example
#include <vsl/vsl_binary_loader.h>

class vxl_my_class : public vxl_baseclass
@{
// ...
  virtual void b_write(vsl_b_ostream&) const;
  virtual void b_read(vsl_b_ostream&);
  virtual vcl_string is_a() const;
  virtual bool is_class(vcl_string const&) const;
  virtual vxl_baseclass* clone() const;
@};

//: Provide examples of each type of polymorphic object that might appear in the stream.
void vxl_configure_loaders()
@{
  vsl_add_to_binary_loader(vxl_my_class());
@}

void demonstrate_save(vsl_b_ostream& os)
@{
  vxl_baseclass *base_ptr = new vxl_my_class;

  // Write the object to the stream, together with
  // an identifier indicating what type it is.
  vsl_b_write(os,base_ptr);

  // Tidy up
  delete base_ptr;
@}

void demonstrate_load(vsl_b_istream& is)
@{
  vxl_baseclass *base_ptr;

  vsl_b_read(is,base_ptr);

  // Show the object
  vcl_cout<<"Loaded object: "<<base_ptr<<vcl_endl;

  // Tidy up
  delete base_ptr;
@}
@end example

@subsubsection How loading by baseclass pointer works (in-class case)

When an object is saved by baseclass pointer, using @code{vsl_b_write(os,baseclass const*)},
the name of the class is written first, then the object itself.

When one comes to load the object, using @code{vsl_b_read(is,baseclass* &)}, the following
occurs:

@enumerate
@item The singleton loader object (@code{vsl_binary_loader<baseclass>}) is invoked.
@item The loader reads the name of the class from the stream.
@item The loader compares this name with a list of possible objects,
      which have been supplied by earlier calls to @code{vsl_add_to_loader(derived_class())}.
@item If the loader finds a match, it creates a clone of the named class object.
@item This clone loads the data from the stream.
@item The loader then sets the baseclass pointer to point to this new object.
@end enumerate

So you now have a shiny pointer to the object.  Note: The caller is then responsible
for the object that the loader created.

The above methods also work if the
pointer that was saved was NULL.  The loader will detect this and set the baseclass
pointer to zero.

@subsubsection IO provided by external `clip-on' classes

@example
class vxl_my_class : public vxl_baseclass
@{
// ...
  // No IO here
@};

//: Provide IO for vxl_my_class
class vxl_my_class : public vxl_baseclassIO
@{
 public:
  //: Constructor
  vxl_my_classIO();

  //: Destructor
  virtual ~vxl_my_classIO();

  //: Create new object of type vxl_my_class on heap
  virtual vxl_baseclass* new_object() const;

  //: Write derived class to os using baseclass reference
  virtual void b_write_by_base(vsl_b_ostream& os, vxl_baseclass const& base) const;

  //: Write derived class to os using baseclass reference
  virtual void b_read_by_base(vsl_b_istream& is, vxl_baseclass& base) const;

  //: Copy this object onto the heap and return a pointer
  virtual vxl_baseclassIO* clone() const;

  //: Return name of class for which this object provides IO
  virtual vcl_string target_classname() const;

  //: Return true if \a b is of class target_classname()
  //  Typically this will just be "return b.is_class(target_classname())"
  //  However, third party libraries may use a different system
  virtual bool is_io_for(vxl_baseclass const& b) const;
@};

//: Provide IO class for each type of polymorphic object that might appear in the stream.
void vxl_configure_loaders()
@{
  vsl_add_to_binary_loader(vxl_my_class_io());
@}

// The actual IO calls are then identical to those described above for the
// case of the IO provided within the class

@end example

See below for details of the implementation of the clip-on IO libraries.

@subsubsection How loading by baseclass pointer works (external IO case)

Essentially this is similar to the within-class-IO case, in that a string
is written to the stream to indicate which class is saved, and when reading
the stream this is used to indicate what to generate on the heap.
However, the details of the implementation are more complicated.

When an object is saved by baseclass pointer, using @code{vsl_b_write(os,baseclass const*)},
a function in @code{vsl_clipon_binary_loader<baseclass,baseclass_io>} is invoked.
It contains a list of IO objects, each of which is queried
(using the @code{io.is_io_for(baseclass const&)} function to see if it can deal with the
given class.  If so a name, @code{io.target_classname()}, is saved to the stream, then the
IO object is used to save the target object.

When one comes to load the object, using @code{vsl_b_read(is,baseclass* &)}, the following
occurs:

@enumerate
@item The singleton loader object (@code{vsl_clipon_binary_loader<baseclass,baseclass_io>}) is invoked.
@item The loader reads the name of the class from the stream.
@item The loader compares this name with a list of possible objects,
      which have been supplied by earlier calls to @code{vsl_add_to_loader(derived_class())}.
@item If the loader finds a match, it uses the IO object to create a new object on the heap
@item The io object then loads the data from the stream into the object on the heap
@item The loader then sets the baseclass pointer to point to this new object.
@end enumerate

So you now have a shiny pointer to the object.  Note: The caller is then responsible
for the object that the loader created.

The above methods also work if the
pointer that was saved was NULL.  The loader will detect this and set the baseclass
pointer to zero.

@subsection Future Work

@enumerate
@item Currently, a few error checks in the IO still result in an error message and @code{std::abort()}.  Since
IO code is error prone for reasons often beyond the control of the programmer, this behaviour should be
replaced with a standard error message a setting of the stream's fail flag, and return.
@item Design a format for @code{long double} and add it @code{to vsl_binary_io.h}.
@item Many of the function and file names do not conform to VXL standards - that each
function and class should be named after the file-stem that declares and defines it.
@end enumerate

@subsection Credits

Your principle designers for this chapter have been Tim Cootes and Ian Scott.
We based the first draft on the design of the Binary IO system in RADIAL, which was
originally designed in Manchester by Dave Bailes and Dave Cooper.

Finally, the hard work of programming @code{vsl} and the vxl/io libraries was performed by the
C++ user's group at the Dept. of Imaging Science, University of Manchester;
Danny Allen, Tim Cootes, Nick Costen,
Ian Scott, Christine Beeston, David Cristinacce, Franck Bettinger, Louise Butcher, and
John Kang.

@section Adding Binary I/O to a New Class

The golden rule for trouble free binary IO is
@quotation
Always write the input and output code in tandem - the output should
precisely mirror the input.
@end quotation

First you need to evaluate whether the class needs binary I/O.
For example, the @code{vnl_matrix_ref} can just be left to inherit from
@code{vnl_matrix} because it does not have any additional member
variables.
Often iterators tend to be transient things which should not be
saved.

Where one does wish to save state, there are four cases to consider:
@enumerate
@item The class is a non-polymorphic (plain) class with no virtual functions
@item The class is a base class with virtual functions
@item The class is a derived class with virtual functions
@item The class is derived from a plain class but is not strictly polymorphic,
      and has no virtual functions
@end enumerate

The last must be considered because various classes in vxl are derived
from plain classes.  The base classes in these cases are kept as
plain classes to improve memory efficiency.

@subsection Non-polymorphic (Plain) Classes

For plain classes (those with no virtual functions), and those derived from plain classes,
you need to add the following functions to the class definition in the .h file.
@example
  //: Binary save self to stream.
  void b_write(vsl_b_ostream& os) const;

  //: Binary load self from stream.
  void b_read(vsl_b_istream& is);

  //: Return IO version number;
  short version() const;

  //: Print an ascii summary to the stream
  void print_summary(vcl_ostream& os) const;

  //: Return a platform independent string identifying the class
  vcl_string is_a() const;

  //: Return true if the argument matches the string identifying the class or any parent class
  bool is_class(vcl_string const&) const;

@end example
You will also need to add the following global scope helper function
declarations to the .h file

@example
//: Binary save vnl_my_class to stream.
void vsl_b_write(vsl_b_ostream& os, vnl_my_class const& b);

//: Binary load vnl_my_class from stream.
void vsl_b_read(vsl_b_istream& is, vnl_my_class& b);
@end example

NOTE: YOU SHOULD ALSO ADD APPROPRIATE TEST PROGRAMS : See below

@subsection Base Classes

For base classes (those classes at the base of a polymorphic hierarchy, with virtual functions),
you need to add the following functions to the class definition in the .h file.

@example
  //: Binary save self to stream.
  virtual void b_write(vsl_b_ostream& os) const;

  //: Binary load self from stream.
  virtual void b_read(vsl_b_istream& is);

  //: Return IO version number;
  short version() const;

  //: Print an ascii summary to the stream
  virtual void print_summary(vcl_ostream& os) const;

  //: Return a platform independent string identifying the class
  virtual vcl_string is_a() const=0;

  //: Return true if the argument matches the string identifying the class
  virtual bool is_class(vcl_string const&) const;

  //: Create a copy of the object on the heap.
  // The caller is responsible for deletion
  virtual vanyl_my_base_class* clone() const=0;

@end example
You will also need to add the following global scope helper function declarations to the .h file

@example
  //: Binary save vnl_my_class to stream.
  void vsl_b_write(vsl_b_ostream& os, vnl_my_class const& b);

  //: Binary load vnl_my_class from stream.
  void vsl_b_read(vsl_b_istream& is, vnl_my_class& b);

  //: Allows derived class to be loaded by base-class pointer
  //  A loader object exists which is invoked by calls
  //  of the form "vsl_b_read(os,base_ptr)".  This loads derived class
  //  objects from the disk, places them on the heap and
  //  returns a base class pointer.
  //  In order to work the loader object requires
  //  an instance of each derived class that might be
  //  found.  This function gives the model class to
  //  the appropriate loader.
  void vsl_add_to_binary_loader(vanyl_my_base_class const& b);

  //: Stream output operator for class pointer
  void vsl_print_summary(vcl_ostream& os, vanyl_my_base_class const* b);
@end example

NOTE: YOU SHOULD ALSO ADD APPROPRIATE TEST PROGRAMS : See below

@subsection Derived Classes

For derived classes in a polymorphic hierarchy, which include virtual functions,
you need to add the following functions to the class definition in the .h file.

@example
  //: Binary save self to stream.
  virtual void b_write(vsl_b_ostream& os) const;

  //: Binary load self from stream.
  virtual void b_read(vsl_b_istream& is);

  //: Return IO version number;
  short version() const;

  //: Print an ascii summary to the stream
  virtual void print_summary(vcl_ostream& os) const;

  //: Return a platform independent string identifying the class
  virtual vcl_string is_a() const;

  //: Return true if the argument matches the string identifying the class or any parent class
  bool is_class(vcl_string const&) const;

  //: Create a copy of the object on the heap.
  // The caller is responsible for deletion
  virtual vanyl_my_base_class* clone() const;

@end example

In this case you should not need any global helper functions, as they
are included in the base class.

NOTE: YOU SHOULD ALSO ADD APPROPRIATE TEST PROGRAMS : See below

@subsection The Implementation Code to Add

The following is a template for the standard code for the implementation of each of the
methods and functions declared above.

@subsubsection Version Numbering

This identifies the I/O version numbering.  It needs to be incremented when the
class's binary I/O changes.

There is no reason why this version number can't be
used for other purposes.  So long as the @code{b_read()} method is updated to deal with it.

It is common and perfectly acceptable to modify a classes I/O during very early development without
changing the version number.

@example
//: Return IO version number
short vanyl_my_class::version() const
@{
  return 1;
@}
@end example

@subsubsection Saving Binary State to a Stream

Classes should first save their version number.  Derived classes should then
call the base class @code{b_write()}, if the base class has any data members.
Finally write the classes data members.

If data is duplicated within the class,
(e.g. some workspace data) you will have to decide whether to save and reload the data,
or else not save it and regenerate it on loading.

The standard form for the method is
@example
//: Binary save self to stream.
void vanyl_my_class::b_write(vsl_b_ostream& os) const
@{
  vsl_b_write(os, version());
  vanyl_my_base_class::b_write(os); // vanyl_my_base_class is parent of vanyl_my_class
  vsl_b_write(os, this->my_value);
  ...
@}
@end example

@subsubsection Restoring Binary State from a Stream

This function is less complicated than it looks.  The complexity of the switch statement is there
solely to give as much backwards compatibility as you are willing to program.

@example
//=======================================
//: Binary load self from stream.
void vanyl_my_class::b_read(vsl_b_istream& is)
@{
  if (!is) return;

  short ver;
  vsl_b_read(is, ver);
  switch (ver)
  @{
   case 1:
    vanyl_my_base_class::b_read(is); // vanyl_my_base_class is parent of vanyl_my_class
    vsl_b_read(is, this->my_value);
    ...
    break;

   default:
    vcl_cerr << "I/O ERROR: vanyl_my_class::b_read(vsl_b_istream&)\n"
             << "           Unknown version number "<< ver << '\n';
    is.is().clear(vcl_ios::badbit); // Set an unrecoverable IO error on stream
    return;
  @}
@}
@end example

@subsubsection Print a Human Readable Summary to a Stream

Classes should print a short summary of their contents.  Devolve printing
of complex members and parent classes to that class, only if it is sensible.

Do not give too much information.  For instance if your class is a
container print the number of elements , and then no more than
perhaps the first 5 elements, with ellipsis to indicate if there
are more.

To aid clear printing the following rule has been found to be helpful.
@quotation
If a summary can be printed on a single line, then do not output a linefeed.
If a summary contains linefeeds in the middle, then finish the summary with a linefeed.
@end quotation
This approach gives flexibility in the output format whilst preserving readability
and predictability for programmers.

The standard form for the method is
@example
//: Output a human readable summary to the stream
void vanyl_my_class::print_summary(vcl_ostream& os) const
@{
  os<<"Important Value: "<<this->my_value<<" .. ";
  ...
  // optionally os << vcl_endl;
@}
@end example

@subsubsection Platform Independent Class Identification

Unfortunately neither the @code{type_info} class nor the @code{type_info::name} field given
by RTTI are platform independent.  To allow loading of a class by base class
pointer, the @code{is_a()}
method is used to identify exactly which
object to create.  It can also be used for other purposes.

If the class is not part of an inheritance hierarchy or will not ever be loaded by base class
pointer, the @code{is_a()} method is not necessary and can be left out.

@example
//: Return a platform independent string identifying the class
vcl_string vanyl_my_class::is_a() const
@{
  return vcl_string("vanyl_my_class");
@}
@end example

The @code{is_class()} methods are not used by the vsl system, but are useful when
you don't have RTTI.
@example
//: Return true if the argument matches the string identifying the class or any parent class
bool vanyl_my_class::is_class(vcl_string const& s) const
@{
  return s==vanyl_my_class::is_class || vanyl_parent_class::is_class(s);
@}
@end example

@subsubsection Polymorphic Copy Creation onto the Heap

The base class loader scheme needs to be able to create a new object on the heap
from a base_class pointer to an old one.  If the class is not part of an inheritance
hierarchy or will not ever be loaded by base class
pointer, the @code{clone()} method is not necessary and can be left out.

The code below assumes that either a working copy constructor has been defined, or else
that compiler generated default copy constructor is suitable (e.g. the class has no pointer
or reference member variables.)

@example
//: Create a copy of the object on the heap.
// The caller is responsible for deletion
vanyl_my_base_class* vanyl_my_class::clone() const
@{
  return new vanyl_my_class(*this);
@}
@end example

@subsubsection Helper Functions.

@example
//: Binary save vnl_vector to stream.
void vsl_b_write(vsl_b_ostream& os, vanyl_my_class const& v)
@{
  v.b_write(os);
@}

//: Binary load vnl_vector from stream.
void vsl_b_read(vsl_b_istream& is, vanyl_my_class& v)
@{
  v.b_read(is);
@}
@end example
It is acceptable to inline the previous two functions, in which case a half-decent compiler
will optimise them completely away.

@example
//: Output a human readable summary to the stream
void vsl_print_summary(vcl_ostream& os, vanyl_my_class const& v)
@{
  os << v.is_a() << ": ";
  vsl_indent_inc(os);
  v.print_summary(os);
  vsl_indent_dec(os);
@}

@end example

@subsubsection Base Classes

The implementation code for functions to be added to base classes is as follows

@example
//==============================================
//: Allows derived class to be loaded by base-class pointer.
//  A loader object exists which is invoked by calls
//  of the form "vsl_b_read(os,base_ptr);".  This loads derived class
//  objects from the stream, places them on the heap and
//  returns a base class pointer.
//  In order to work the loader object requires
//  an instance of each derived class that might be
//  found.  This function gives the model class to
//  the appropriate loader.
void vsl_add_to_binary_loader(vanyl_my_base_class const& b)
@{
  vsl_binary_loader<vanyl_my_base_class>::instance().append(b);
@}

//==============================================
//: Stream summary output for base class pointer
void vsl_print_summary(vcl_ostream& os, vanyl_my_base_class const* b)
@{
  if (b)
    return vsl_print_summary(*b);
  else
    return os << "No vanyl_my_base_class defined.\n";
@}
@end example

If the base class is abstract, then you should not to provide
implementations for the following methods.  Instead declare them
as pure virtual methods in the .h file.
@example
  //: Create a copy on the heap and return base class pointer
  virtual my_base_class* clone() const = 0;

  //: Print class to os
  virtual void print_summary(vcl_ostream& os) const = 0;

  //: Save class to binary file stream
  virtual void b_write(vsl_b_ostream& bfs) const = 0;

  //: Load class from binary file stream
  virtual void b_read(vsl_b_istream& bfs) = 0;

@end example

@subsection Serialisation - Saving Objects with Shared Ownership

There is problem when you want to save an object whose ownership is shared by
several other objects.  The most common way this happens is if several objects
(A1, A2, ..., An) contain pointers to a single other object B.  If all the objects save B,
when the stream is reloaded, there wile be n copies of B in memory, instead of the
single shared one there was before saving.

If one object can be designated the @emph{owner} (for example if one object, say A1,
is responsible for deleting B,) then A1 is responsible for saving and loading
object B.  This prevents multiple copies of B being created during loading, but all the
other A objects do not have their pointer set up correctly.

The standard way of dealing with these problems involves giving each shared object a unique
@dfn{serial number}, and the process of performing IO on shared objects is called
@dfn{serialisation}.

If you are using shared objects though existing classes such as @code{vbl_smart_ptr}, or
@code{vil_image}, then you do not need to do anything, since these classes handle the
serialisation.

If you are managing shared ownership in any of your classes, you will need to write
the serialising code yourself.  There is no language support for serialisation in C++
(unlike Java), however the @code{vsl} library does provide some support.

The basic algorithm performed by all of the owner objects during saving is
@itemize @bullet
@item If it is a null pointer, save a unique serial number.
@item else if object B has not been saved before
  @itemize @bullet
  @item generate a unique serial number for object B
  @item save the serial number
  @item save object B
  @end itemize
@item else if we have already saved B
  @itemize @bullet
  @item find unique serial number for object B
  @item save the serial number
  @end itemize
@item end
@end itemize

The basic algorithm performed by all of the owner objects during loading is
@itemize @bullet
@item load unique serial number
@item if it is the null-pointer id, set my pointer to 0
@item else if this is the first time we have seen this serial number
  @itemize @bullet
  @item load object B
  @item set my pointer to point to B
  @end itemize
@item else if we have seen the serial number before
  @itemize @bullet
  @item find location in memory of object B with given serial number
  @item set my pointer to point to B
  @end itemize
@item end
@end itemize

You can use the serialisation record in @code{vsl_b_istream} and
@code{vsl_b_ostream} to record the serial numbers, and whether
you have already saved object B.

The best way to explain the detail is in this simplified and
cut-down version of @file{vxl/vbl/io/vbl_io_smart_ptr.txx}
In this case all the smart_ptr<T> objects are the As, and the
objects they point to are the Bs.

@example
#include <vsl/vsl_binary_io.h>
#include <vbl/vbl_smart_ptr.h>

//: Binary save smart_ptr and serialised *smart_ptr to stream.
template<class T>
void vsl_b_write(vsl_b_ostream& os, vbl_smart_ptr<T> const& p)
@{
  // write version number
  const short io_version_no = 1;
  vsl_b_write(os, io_version_no);

  if (p.ptr() == 0)  // Deal with Null pointers first.
  @{
    vsl_b_write(os, 0ul); // Use 0 to indicate a null pointer.
                          // True serialisation IDs are always 1 or more.
    return;
  @}

  // Get a serial_number for object being pointed to
  unsigned long id = os.get_serial_number(p.ptr());

  // Find out if this is the first time the object being
  // pointed to is being saved
  if (id == 0)
  @{
    // Store a record of the address of B and get a serial number
    id = os.add_serialisation_record(p.ptr());

    vsl_b_write(os, id);     // Save the serial number

    // If you get a compiler error in the next line, it could be because your type T
    // has no vsl_b_write(vsl_b_ostream&, T const*)  defined on it.
    // See the documentation in the .h file to see how to add it.
    vsl_b_write(os, p.ptr());  // Only save the actual object if
                               // it hasn't been saved before to this stream
  @}
  else
  @{
    vsl_b_write(os, id);         // Save the serial number
  @}
@}

//=============================================================================
//: Binary load self from stream.
template<class T>
void vsl_b_read(vsl_b_istream& is, vbl_smart_ptr<T>& p)
@{
  short ver;
  vsl_b_read(is, ver);
  switch (ver)
  @{
   case 1: @{
    unsigned long id; // Unique serial number identifying object
    vsl_b_read(is, id);

    if (id == 0) // Deal with Null pointers first.
    @{
      p = 0;
      return;
    @}

    T* pointer = (T*) is.get_serialisation_pointer(id);
    if (pointer == 0) // Not loaded before
    @{
      // If you get a compiler error in the next line, it could be because your type T
      // has no vsl_b_read(vsl_b_ostream&, T* &)  defined on it.
      // See the documentation in the .h file to see how to add it.
      vsl_b_read(is, pointer);                 // load object B
      is.add_serialisation_record(id, pointer); // remember location of B
    @}

    p.set_ptr(pointer); // This operator method will set the internal
                        // pointer in vbl_smart_ptr.
    break; @}
   default:
    vcl_cerr << "vbl_smart_ptr::b_read() Unknown version number "
             << ver << vcl_endl;
    vcl_abort();
  @}
@}
@end example

You do not need to write any special code for the owned object, except
to provide member functions for loading and saving the object by
pointer.  These will already exist if you have written polymorphic IO
for the class of object B.  If not, the following examples might help.

@example
// Save with base class pointers
void vsl_b_read(vsl_b_istream& is, class_B* & p)
@{
  delete p;
  bool not_null_ptr;
  vsl_b_read(is, not_null_ptr);
  if (not_null_ptr)
  @{
    p = new class_B;
    vsl_b_read(is, *p);
  @}
  else
    p = 0;
@}

template<class T>
void vsl_b_write(vsl_b_ostream& os, class_B const* p)
@{
  if (p==0)
  @{
    vsl_b_write(os, false); // Indicate null pointer stored
  @}
  else
  @{
    vsl_b_write(os,true); // Indicate non-null pointer stored
    vsl_b_write(os, *p);
  @}
@}
@end example

The upshot of this is that serialisation can be tricky, but not
necessarily difficult.  You are advised to manage shored ownership
though smart pointers which will do all the serialisation (and
memory management) for you.

@section Adding Binary I/O to Level-1 Libraries

There are several reasons for doing slightly different things for
classes that are members of a level-1 library.
@enumerate
@item No level-one library can depend on any library except vcl.
@item We do not wish to `break' any existing code
@item There is a desire to keep the level 1 libraries small.  IO is available if required.
@end enumerate

Binary IO for the level-1 libraries (@code{vbl},@code{vil},@code{vgl},@code{vnl})
is implemented using
`clip-on' libraries, the code for which lives in `io' subdirectories of
each library.  Thus the code for the IO for @code{vnl_vector<T>} lives in
@file{vnl/io/vnl_io_vector.h/.txx}.  Binary IO for @code{vcl} containers is provided in
the library @code{vsl}.

The `clip-on' libraries provide a set of external functions and classes which
allow binary save and restore of the level 1 classes through their public
member functions.

Essentially this means writing the following functions for each class:

@example
  //: Binary save vnl_my_class to stream.
  void vsl_b_write(vsl_b_ostream& os, vnl_my_class const& c);

  //: Binary load vnl_my_class from stream.
  void vsl_b_read(vsl_b_istream& is, vnl_my_class& c);

  //: Stream output operator for class pointer
  void vsl_print_summary(vcl_ostream& os, vnl_my_class const& c);
@end example

Each one is written using only the public access functions of the class.

For instance, @code{vgl_point_2d} has binary IO as follows:

@example
// This is core/vgl/io/vgl_io_point_2d.h
#ifndef vgl_io_point_2d_h_
#define vgl_io_point_2d_h_
//:
// \file
// \author Tim Cootes

#include <vgl/vgl_point_2d.h>
#include <vsl/vsl_binary_io.h>

//: Binary save vgl_point_2d to stream.
template <class T>
void vsl_b_write(vsl_b_ostream& os, vgl_point_2d<T> const& p);

//: Binary load vgl_point_2d from stream.
template <class T>
void vsl_b_read(vsl_b_istream& is, vgl_point_2d<T>& p);

//: Print human readable summary of a vgl_point_2d object to a stream
template <class T>
void vsl_print_summary(vcl_ostream& os, vgl_point_2d<T> const& p);

#endif // #ifndef vgl_io_point_2d_h_
@end example

The implementation is

@example
// This is core/vgl/io/vgl_io_point_2d.txx

#include "vgl_io_point_2d.h"
#include <vsl/vsl_binary_io.h>

//==============================================================================
//: Binary save vgl_point_2d to stream.
template<class T>
void vsl_b_write(vsl_b_ostream& os, vgl_point_2d<T> const& p)
@{
  vsl_b_write(os, p.version());
  vsl_b_write(os, v.x());
  vsl_b_write(os, v.y());
@}

//==============================================================================
//: Binary load vgl_point_2d from stream.
template<class T>
void vsl_b_read(vsl_b_istream& is, vgl_point_2d<T>& p)
@{
  if (!is) return;

  short w;
  vsl_b_read(is, w);
  switch (w)
  @{
   case 1:
    vsl_b_read(is, p.x());
    vsl_b_read(is, p.y());
    break;

   default:
    vcl_cerr << "I/O ERROR: vsl_b_read(vsl_b_istream&, vgl_point_2d<T>&)\n"
             << "           Unknown version number "<< w << '\n';
    is.is().clear(vcl_ios::badbit); // Set an unrecoverable IO error on stream
    return;
  @}
@}

//==============================================================================
//: Output a human readable summary of a vgl_point_2d object to the stream
template<class T>
void vsl_print_summary(vcl_ostream& os, vgl_point_2d<T> const& p)
@{
  os<<'(';
  vsl_print_summary(p.x());
  os<<','
  vsl_print_summary(p.y());
  os<<')';
@}

#define VGL_IO_POINT_2D_INSTANTIATE(T) \
template void vsl_print_summary(vcl_ostream&, vgl_point_2d<T > const&); \
template void vsl_b_read(vsl_b_istream&, vgl_point_2d<T >&); \
template void vsl_b_write(vsl_b_ostream&, vgl_point_2d<T > const&)

@end example

@subsection `Clip-On' IO for Polymorphic Hierarchies

Here are some examples:

@subsubsection Clip-on IO BaseClass Header

Here is an example of a @file{BaseClassIO.h}

@example
// This is BaseClassIO.h
#ifndef BaseClassIO_h_
#define BaseClassIO_h_
//:
// \file
// \author Tim Cootes

#include <vsl/vsl_binary_io.h>

// Predeclare classes
class BaseClass;

//: Base for objects which provide IO for classes derived from BaseClass
class BaseClassIO
@{
 public:
  //: Constructor
  BaseClassIO();

  //: Destructor
  virtual ~BaseClassIO();

  //: Create new object of type BaseClass on heap
  virtual BaseClass* new_object() const;

  //: Write derived class to os using baseclass reference
  virtual void b_write_by_base(vsl_b_ostream& os, BaseClass const& base) const;

  //: Write derived class to os using baseclass reference
  virtual void b_read_by_base(vsl_b_istream& is, BaseClass& base) const;

  //: Copy this object onto the heap and return a pointer
  virtual BaseClassIO* clone() const;

  //: Return name of class for which this object provides IO
  virtual vcl_string target_classname() const;

  //: Return true if \a b is of class target_classname()
  //  Typically this will just be "return b.is_class(target_classname())"
  //  However, third party libraries may use a different system
  virtual bool is_io_for(BaseClass const& b) const;
@};

//: Add example object to list of those that can be loaded.
//  The vsl_binary_loader must see an example of each derived class
//  before it knows how to deal with them.
//  A clone is taken of \a b
void vsl_add_to_binary_loader(BaseClassIO const& b);

//: Binary save to stream by baseclass pointer
void vsl_b_write(vsl_b_ostream& os, BaseClass const* b);

//: Binary read from stream by baseclass pointer
void vsl_b_read(vsl_b_istream& is, BaseClass* & b);

//: Binary save vgl_my_class to stream.
void vsl_b_write(vsl_b_ostream& os, BaseClass const& b);

//: Binary load vgl_my_class from stream.
void vsl_b_read(vsl_b_istream& is, BaseClass& b);

//: Print human readable summary of object to a stream
void vsl_print_summary(vcl_ostream& os, BaseClass const& b);

@end example

@subsubsection Clip-on IO BaseClass Implementation

For a BaseClass, one creates @file{BaseClassIO.cxx} as follows

@example
// This is BaseClassIO.cxx
#include "BaseClassIO.h"
//:
// \file
// \author Tim Cootes

#include <BaseClass.h>
#include <vsl/vsl_clipon_binary_loader.txx>

//: Constructor
BaseClassIO()::BaseClassIO()
@{
@}

//: Destructor
BaseClassIO()::~BaseClassIO()
@{
@}

//: Create new object of type BaseClass on heap
BaseClass* BaseClassIO()::new_object() const
@{
  return new BaseClass;
@}

//: Write derived class to os using baseclass reference
void BaseClassIO()::b_write_by_base(vsl_b_ostream& os, BaseClass const& base) const
@{
  vsl_b_write(os,base);
@}

//: Write derived class to os using baseclass reference
void BaseClassIO()::b_read_by_base(vsl_b_istream& is, BaseClass& base) const
@{
  vsl_b_read(is,base);
@}

//: Copy this object onto the heap and return a pointer
BaseClassIO* BaseClassIO()::clone() const
@{
  return new BaseClassIO(*this);
@}

//: Return name of class for which this object provides IO
vcl_string BaseClassIO()::target_classname() const
@{
  return string("BaseClass");
@}

//: Return true if \a b is of class target_classname()
bool BaseClassIO()::is_io_for(BaseClass const& b) const
@{
  return b.is_a()==target_classname();
@}

//==============================================================================
//: Binary write to stream.
void vsl_b_write(vsl_b_ostream& os, BaseClass const& p)
@{
  const short io_version_no = 1;
  vsl_b_write(os, io_version_no);
  vsl_b_write(os, p.x());
  vsl_b_write(os, p.y());
  // ... Insert rest of xxxxxx code here
@}

//==============================================================================
//: Binary load from stream.
void vsl_b_read(vsl_b_istream& is, BaseClass& p)
@{
  if (!is) return;

  short v;
  vsl_b_read(is, v);
  switch (v)
  @{
   case 1:
    vsl_b_read(is, p.x());
    vsl_b_read(is, p.y());
    // ... Insert rest of xxxxxx code here
    break;

   default:
    vcl_cerr << "I/O ERROR: vsl_b_read(vsl_b_istream&, BaseClass&)\n"
             << "           Unknown version number "<< v << '\n';
    is.is().clear(vcl_ios::badbit); // Set an unrecoverable IO error on stream
    return;
  @}
@}

//==============================================================================
//: Output a human readable summary to the stream
void vsl_print_summary(vcl_ostream& os, BaseClass const& p)
@{
  os<<'('<<p.x()<<','<<p.y()<<')';
  // ... Insert rest of xxxxxx code here
@}

//: Add example object to list of those that can be loaded.
//  The vsl_binary_loader must see an example of each derived class
//  before it knows how to deal with them.
//  A clone is taken of \a b
void vsl_add_to_binary_loader(BaseClassIO const& b)
@{
  vsl_clipon_binary_loader<BaseClass,BaseClassIO>::instance().add(b);
@}

//: Binary save to stream by baseclass pointer
void vsl_b_write(vsl_b_ostream& os, BaseClass const* b)
@{
  vsl_clipon_binary_loader<BaseClass,BaseClassIO>::instance().write_object(os,b);
@}

//: Binary read from stream by baseclass pointer
void vsl_b_read(vsl_b_istream& is, BaseClass* & b)
@{
  vsl_clipon_binary_loader<BaseClass,BaseClassIO>::instance().read_object(is,b);
@}

// Explicitly instantiate loader
VSL_CLIPON_BINARY_LOADER_INSTANTIATE(BaseClass, BaseClassIO);

@end example

@subsubsection Clip-on IO Derived Class Header

For a DerivedClass, one creates @file{DerivedClassIO.h} as follows

@example
// This is DerivedClassIO.h
#ifndef DerivedClassIO_h_
#define DerivedClassIO_h_
//:
// \file
// \author Tim Cootes

#include <BaseClassIO.h>

//: Provide IO for DerivedClass
class DerivedClass : public BaseClassIO
@{
 public:
  //: Constructor
  DerivedClassIO();

  //: Destructor
  virtual ~DerivedClassIO();

  //: Create new object of type DerivedClass on heap
  virtual BaseClass* new_object() const;

  //: Write derived class to os using baseclass reference
  virtual void b_write_by_base(vsl_b_ostream& os, BaseClass const& base) const;

  //: Write derived class to os using baseclass reference
  virtual void b_read_by_base(vsl_b_istream& is, BaseClass& base) const;

  //: Copy this object onto the heap and return a pointer
  virtual BaseClassIO* clone() const;

  //: Return name of class for which this object provides IO
  virtual vcl_string target_classname() const;

  //: Return true if \a b is of class target_classname()
  //  Typically this will just be "return b.is_class(target_classname())"
  //  However, third party libraries may use a different system
  virtual bool is_io_for(BaseClass const& b) const;
@};

//: Binary save vgl_my_class to stream.
void vsl_b_write(vsl_b_ostream& os, DerivedClass const& b);

//: Binary load vgl_my_class from stream.
void vsl_b_read(vsl_b_istream& is, DerivedClass& b);

//: Print human readable summary of object to a stream
void vsl_print_summary(vcl_ostream& os, DerivedClass const& b);

@end example

@subsubsection Clip-on IO Derived Class Implementation

For a DerivedClass, one creates @file{DerivedClassIO.cxx} as follows

@example
// This is DerivedClassIO.cxx
#include "DerivedClassIO.h"
//:
// \file
// \author Tim Cootes

#include <DerivedClass.h>

//: Constructor
DerivedClassIO()::DerivedClassIO()
@{
@}

//: Destructor
DerivedClassIO()::~DerivedClassIO()
@{
@}

//: Create new object of type DerivedClass on heap
BaseClass* DerivedClassIO()::new_object() const
@{
  return new DerivedClass;
@}

//: Write derived class to os using baseclass reference
void DerivedClassIO()::b_write_by_base(vsl_b_ostream& os, BaseClass const& base) const
@{
  vsl_b_write(os,base);
@}

//: Write derived class to os using baseclass reference
void DerivedClassIO()::b_read_by_base(vsl_b_istream& is, BaseClass& base) const
@{
  vsl_b_read(is,base);
@}

//: Copy this object onto the heap and return a pointer
BaseClassIO* DerivedClassIO()::clone() const
@{
  return new DerivedClassIO(*this);
@}

//: Return name of class for which this object provides IO
vcl_string DerivedClassIO()::target_classname() const
@{
  return vcl_string("DerivedClass");
@}

//: Return true if \a b is of class target_classname()
bool DerivedClassIO()::is_io_for(BaseClass const& b) const
@{
  return b.is_a()==target_classname();
@}

//==============================================================================
//: Binary write to stream.
void vsl_b_write(vsl_b_ostream& os, DerivedClass const& p)
@{
  const short io_version_no = 1;
  vsl_b_write(os, io_version_no);
  vsl_b_write(os, p.x());
  vsl_b_write(os, p.y());
  //... Insert rest of xxxxxx code here
@}

//==============================================================================
//: Binary load from stream.
void vsl_b_read(vsl_b_istream& is, DerivedClass& p)
@{
  if (!is) return;

  short v;
  vsl_b_read(is, v);
  switch (v)
  @{
   case 1:
    vsl_b_read(is, p.x());
    vsl_b_read(is, p.y());
    //... Insert rest of xxxxxx code here
    break;

   default:
    vcl_cerr << "I/O ERROR: vsl_b_read(vsl_b_istream&, DerivedClass&)\n"
             << "           Unknown version number "<< v << '\n';
    is.is().clear(vcl_ios::badbit); // Set an unrecoverable IO error on stream
    return;
  @}
@}

@end example

There.  Nothing to it really.

@section Test Programs

It is important to write test programs for all the IO.  Below is a
summary of how it is done for classes in vgl.  Extrapolate as
appropriate.

The test programs for vgl live in the subdirectory core/vgl/tests.

To add a test for a class you

@enumerate
@item Write a test function in a file : @file{test_classname_io.cxx}
@item Add a call to that function in the file @file{test_driver.cxx}
@end enumerate

For instance, to test the IO in class @code{vgl_point_2d<T>}, we
write

@file{test_point_2d_io.cxx}:
@example
#include <vcl_iostream.h>

#include <vgl/vgl_point_2d.h>
#include <vgl/io/vgl_io_point_2d.h>

#include <testlib/testlib_test.h>

void test_point_2d_io()
@{
  vcl_cout << "*******************************\n"
           << "Testing vgl_point_2d<double> io\n"
           << "*******************************\n";

  vgl_point_2d<double> p_out(1.2,3.4), p_in;

  vsl_b_ofstream bfs_out("vgl_point_2d_test_double_io.bvl.tmp");
  TEST("Created vgl_point_2d_test_double_io.bvl.tmp for writing", (!bfs_out), false);
  vsl_b_write(bfs_out, p_out);
  bfs_out.close();

  vsl_b_ifstream bfs_in("vgl_point_2d_test_double_io.bvl.tmp");
  TEST("Opened vgl_point_2d_test_double_io.bvl.tmp for reading", (!bfs_in), false);
  vsl_b_read(bfs_in, p_in);
  TEST("Finished reading file successfully", (!bfs_in), false);
  bfs_in.close();

  TEST("p_out == p_in", p_out, p_in);

  vsl_print_summary(vcl_cout, p_out);
  vcl_cout << vcl_endl;
@}

TESTMAIN(test_point_2d_io);
@end example

The macros @code{TEST} and @code{TESTMAIN} are defined in the file @file{testlib/testlib_test.h}

We must then add a call to this test to @file{test_driver.cxx}:

@example
#include <testlib/testlib_register.h>

DECLARE( test_point_2d_io );
DECLARE( test_point_3d_io );
// More tests ....

void
register_tests()
@{
  REGISTER( test_point_2d_io );
  REGISTER( test_point_3d_io );
  // More tests ....
@}

DEFINE_MAIN;
@end example

When one runs make (under Unix) all the programs are compiled and the
tests automatically run.  A message will be output giving a summary of
how many were successful/unsuccessful.

@subsection Summary

To summarise, when adding a test program to v?l you need to

@enumerate
@item Create a test function in @file{tests/test_classname_io.cxx}
@item Add a call to the function in @file{tests/test_driver.cxx}
@end enumerate
