\documentclass[]{article}
\usepackage{lmodern}
\usepackage{amssymb,amsmath}
\usepackage{color}
\usepackage{fixltx2e} % provides \textsubscript
% use upquote if available, for straight quotes in verbatim environments
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
% use microtype if available
\IfFileExists{microtype.sty}{%
\usepackage{microtype}
\UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
}{}
\usepackage[unicode=true]{hyperref}
\hypersetup{colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue,
            pdfborder={0 0 0},
            breaklinks=true}
%\urlstyle{same}  % don't use monospace font for urls
\IfFileExists{parskip.sty}{%
\usepackage{parskip}
}{% else
\setlength{\parindent}{0pt}
\setlength{\parskip}{6pt plus 2pt minus 1pt}
}
\setlength{\emergencystretch}{3em}  % prevent overfull lines
\providecommand{\tightlist}{%
  \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
\setcounter{secnumdepth}{0}
% Redefines (sub)paragraphs to behave more like sections
\ifx\paragraph\undefined\else
\let\oldparagraph\paragraph
\renewcommand{\paragraph}[1]{\oldparagraph{#1}\mbox{}}
\fi
\ifx\subparagraph\undefined\else
\let\oldsubparagraph\subparagraph
\renewcommand{\subparagraph}[1]{\oldsubparagraph{#1}\mbox{}}
\fi

% set default figure placement to htbp
\makeatletter
\def\fps@figure{htbp}
\makeatother

\newcommand{\zdimension}{$z$-dimension}

\newcommand{\latin}[1]{\emph{#1}}
\newcommand{\ie}{\latin{i.\,e.}}
\newcommand{\eg}{\latin{e.\,g.}}
\newcommand{\etc}{\latin{etc}}

\begin{document}

\section{Notes on cave surveying and GIS}

The idea that reduced cave survey data should be easily readable into a
Geographic Information System (GIS) platform such as
\href{http://www.qgis.org/}{QGIS} is practically a no-brainer, as it can
then be integrated with other geographical data such as maps, satellite
imagery, digital elevation models, and the like. This is much closer
to being achievable than one might think. Here's the contents of a
typical \href{https://survex.com/}{survex} \verb}.3d} file as exposed
by running \verb}dump3d}:

\begin{itemize}
\tightlist
\item
  survey metadata: title, date, and co-ordinate reference system;
\item
  strings of survey legs with metadata: names, flags (normal, duplicate,
  splay, surface);
\item
  survey stations with metadata: names, flags (exported, entrance,
  fixed, surface) and passage cross-sections (LRUD data).
\end{itemize}

Now compare this to a typical
\href{https://en.wikipedia.org/wiki/Shapefile}{ESRI shapefile}, or the
\href{https://en.wikipedia.org/wiki/GeoPackage}{GeoPackage} data format
from the
\href{https://en.wikipedia.org/wiki/Open_Geospatial_Consortium}{Open
Geospatial Consortium}, which are well known containers for GIS vector
data. These formats specify:

\begin{itemize}
\tightlist
\item
  a co-ordinate reference system, and possible other metadata;
\item
  geometries comprising points, lines, polylines (line strings), and
  polygons, with or without \zdimension\ (elevation) data;
\item
  geometry attributes consisting of records of various kinds that are
  user-configurable.
\end{itemize}

At this point you are supposed to slap yourself on the head and ask
why on earth we haven't been using a GIS-compatible format for storing
reduced survey data all along!  It's certainly flexible enough
to contain all the information normally included in a \verb}.3d} file.

\subsection{Spatial Reference Systems}\label{spatial-reference-systems}

In order for this to work smoothly, we first have to be on top of our
\href{https://en.wikipedia.org/wiki/Spatial_reference_system}{spatial
reference system (SRS)} in general GIS parlance, or co-ordinate
reference system (CRS) in QGIS language. The following notes hopefully
contain enough of the truth to be useful. Something closer to the truth
can be found in
\href{https://www.ordnancesurvey.co.uk/docs/support/guide-coordinate-systems-great-britain.pdf}{{\it A guide to coordinate systems in 
Great Britain}} by the Ordnance Survey (OS).

An SRS usually comprises:

\begin{itemize}
\item
  a \href{https://en.wikipedia.org/wiki/Geodetic_datum}{geodetic
    datum} or Terrestrial Reference System, which is usually an
  ellipsoid which specifies the overall shape of the earth's surface:
  \href{https://en.wikipedia.org/wiki/World_Geodetic_System}{WGS84}
  used in GPS, and 
  \href{https://en.wikipedia.org/wiki/Ordnance_Survey_National_Grid}{OSGB36}
  used by the Ordnance Survey in the UK are two examples;
\item
  a map projection which is nearly always a
  \href{https://en.wikipedia.org/wiki/Transverse_Mercator_projection}{Transverse
    Mercator} projection, such as the
  \href{https://en.wikipedia.org/wiki/Universal_Transverse_Mercator_coordinate_system}{Universal
    Transverse Mercator (UTM)} system used in GPS: the map projection
  tries to flatten \emph{locally} the curved surface of the earth
  (there is always some compromise involved here) and a given geodetic
  datum may have more than one map projection depending which part of
  the Earth's surface one is interested in (UTM has 60 zones for
  instance, and all karst in mainland Britain lies in UTM zone 30N);
\item
  a co-ordinate system (typically metric) defined on top of the map
  projection, usually specifying a `false origin' so that
  co-ordinates are always positive.
\end{itemize}

Given the geodetic datum one can always work with latitudes and
longitudes, but these aren't terribly convenient for cave survey data
crunching. Also beware that the same point on the earth's surface may
have a different latitude and longitude according to the choice of
datum: this is known as a datum shift, and a well-known example is the
\href{https://en.wikipedia.org/wiki/Ordnance_Survey_National_Grid\#Datum_shift_between_OSGB_36_and_WGS_84}{datum
  shift between WGS84 and OSGB36} that nowadays only shows up in
\href{http://www.natureonthemap.naturalengland.org.uk/MagicMap.aspx}{Magic
  Map}.

WGS84 is pretty much universally used nowadays on the internet, for
example Google's \href{https://developers.google.com/kml/}{Keyhole
  Markup Language (KML)} only supports WGS84 latitude and longitude,
to upload to \href{https://en.wikipedia.org/wiki/Google_Earth}{Google
  Earth}.  Also most GPS devices report latitude and longitude for
WGS84, though more often than not you won't see this directly but
rather get metric UTM co-ordinates, or metric British National Grid
co-ordinates in the UK.

To further add to the confusion, latitude and longitude can be
reported in decimal degrees; or degrees, minutes, and seconds (or even
degrees and decimal minutes). For example the entrance to Dow Cave is
at NGR SD\,983{\small78}\,743{\small00}, which translates to (WGS84)
$54^\circ\,9'\,52.2''$\,N $2^\circ\,1'\,34.8''$\,W where one decimal
place in the seconds corresponds to approximately 3\,m on the ground,
or (WGS84) $54.16450^\circ$\,N $2.02634^\circ$\,W where five decimal
places corresponds approximately to 1\,m on the ground, or (WGS84) 30U
6\,002\,262\enskip563\,570 for UTM. Online converters between British
National Grid references and WGS84 latitudes and logitudes can be
found on the internet by searching for `OSGB36 to WGS84 converter'. To
check things, the WGS84 latitude and longitude in decimal degrees can
be copied and pasted into Google maps for example, or for that matter
directly into the Google search engine.

Note there is a confusing difference between the
\href{https://en.wikipedia.org/wiki/Web_Mercator}{Web Mercator} often
used for online maps, and the standard conformal Mercator projection.
For this reason the US DoD has declared the Web Mercator projection
`to be unacceptable for any official use', which is a good enough
reason for me to avoid where possible.

In the UK, Ordnance Survey (OS)
\href{https://en.wikipedia.org/wiki/Ordnance_Survey_National_Grid}{British
National Grid} co-ordinates provide a metric SRS which is convenient for
cave survey data. Typically one fixes cave entrances using the numeric
part of the national grid reference (NGR). The NGR can be specified in two
ways. The most convenient way is to use the OS grid letter system in
which a pair of letters specifies a
$100\,\mathrm{km}\times100\,\mathrm{km}$ square. Then, within
that, a 10-figure national grid reference (NGR) specifies a location to
within a square metre. This system (two letters plus 10 figures) is what
is usually encountered when using a GPS device set to the British
National Grid. Many datasets in the Cave Registry have entrance fixes
specified as 10-fig NGRs, without the grid letters which are assumed
known.

Alternatively, and more commonly in GIS, one can use an all-numeric
12-figure NGR in which the leading figures signal the
$100\,\mathrm{km}\times100\,\mathrm{km}$ 
square. For example in the all-numeric scheme the
entrance to Dow Cave is at NGR\,{\small3}983{\small78}\,{\small4}743{\small00}.

In the letter-based system the co-ordinates are often truncated to 8-fig
or 6-fig NGRs, to reflect the accuracy of the GPS device for instance
(thus 8-fig NGRs are used in the new Northern Caves). In case you forgot
your school geography lessons, recall that the correct way to truncate
an NGR is to \emph{drop} the least significant figures, not to round to
the nearest 10 or 100. This is because an 8-fig (or 6-fig) NGR actually
specifies a $10\,\mathrm{m}\times10\,\mathrm{m}$
(or $100\,\mathrm{m}\times100\,\mathrm{m}$) \emph{square} and not an
approximate position as such. Thus the 6-fig NGR for the Dow Cave
entrance is NGR SD\,983\,743.

To check NGRs in the UK, one can use the `Where am I?' tool in the
\href{http://www.natureonthemap.naturalengland.org.uk/MagicMap.aspx}{Magic
Map} application. Note that unless explicitly set to use WGS84, Magic Map reports OSGB36 latitude and longitude,
which as mentioned is offset from WGS84 by a datum shift of up to
50--100\,m. Beware copying and pasting these OSGB36 latitudes and
longitudes into Google Maps!

Elsewhere in the world, or for that matter in the UK as well, the UTM
system offers a convenient metric SRS for embedding cave survey data.
Typically one fixes the entrance co-ordinates as the numeric part of
the UTM position, making a note of the UTM grid zone. Online
converters from WGS84 latitude and longitude to UTM or back are easily
found (thus, for example, the Dow Cave entrance above). Perhaps it's
restating the obvious but if you accidentally paste OSGB36 latitudes
and longitudes into a UTM converter, you will likely be out by
50--100\,m.

\subsection{Georeferencing cave survey
data}\label{georeferencing-cave-survey-data}

Back to cave surveying: for most surveys the earth's surface can be
regarded as essentially flat, so one is working in a 3d world with
eastings, northings, and altitudes, with the origin of the co-ordinate
system chosen at one's convenience. Perhaps for synoptic maps of very
large karst areas, one might be worried about the curvature of the
earth's surface, but for the most part assuming the world is flat should
introduce negligible errors, at least in comparison to the errors that
typically creep into cave survey projects.

As long as this local cave co-ordinate system can be tied into one of
the known geodetic SRS schemes (\ie\
\href{https://en.wikipedia.org/wiki/Georeferencing}{\emph{georeferenced}}),
then any feature in the cave will have a known position in GIS terms,
and can thus be tied into any other georeferenced data such as maps,
satellite imagery, digital elevation models, \etc. Given that most cave
surveying is done in metres, it is obviously convenient to tie into an
SRS which uses metric co-ordinates, such as UTM or British National
Grid. Note that once you've tied the dataset into a recognised SRS, any
GIS platform worth its salt will be able to re-project into a different
SRS, and will be able to display and combine information from different
sources irrespective of the SRS.

The easiest way to georeference cave survey data, with a modern survex
distribution, is to \verb}*fix} cave entrances with appropriate
co-ordinates and make judicious use of the \verb}*cs} commands (for
co-ordinate system): use a plain \verb}*cs} command to specify the
input SRS that the entrance co-ordinates are given in, and a
\verb}*cs out} command to specify what the output SRS should be. In
the UK for instance one can use this to convert between the OS grid
letter system and the all-numeric scheme.

The cave survey data used in the examples below is included in the
repository under the \verb}DowProv} directory. It is for the
\href{http://www.mudinmyhair.co.uk/}{Dow Cave - Providence Pot system}
(Great Whernside, Wharfedale, UK), and is essentially a snapshot of the
data held in the \href{http://cave-registry.org.uk/}{Cave Registry Data
Archive}. Note that the \verb}.svx} files have
\href{https://en.wikipedia.org/wiki/Newline}{unix-style line endings} so
on Windows you might have to use something like
\href{https://notepad-plus-plus.org/}{Notepad{\small++}} to look at them. The
processed data is \verb}DowProv.3d}, generated using survex 1.2.32.

Back to georeferencing, the cave-specific file \verb}DowCave.svx} (for
example) contains

\begin{verbatim}
*begin DowCave
*export entrance
...
*entrance entrance
*fix entrance 98378 74300 334
*equate entrance dow1.1
...
\end{verbatim}

and the master file \verb}DowProv.svx} contains

\begin{verbatim}
*cs OSGB:SD
*cs out EPSG:7405
...
*begin DowProv
*include DowCave
...
\end{verbatim}

(obviously this is only one of many possible ways to add the metadata
into the survex files).

Thus the file \verb}DowCave.svx} contains a \verb}*fix} which
specifies the entrance location as a 10-fig NGR
SD\,98378\,74300, without the SD part. The easting and
northing here (and elevation
\href{https://en.wikipedia.org/wiki/Ordnance_datum}{ODN}) were obtained
by field work. Then the file \verb}DowProv.svx} specifies input SRS is
the SD square, and asks that the reduced data should be exported
using the all-numeric British National Grid scheme, here codified with 
\href{http://spatialreference.org/}{European Petroleum Survey Group
  (EPSG)} code \verb+EPSG:7405+.
Using EPSG numbers avoids potential misunderstanding when
importing into a GIS platform, for example in QGIS one can find the
exact exported SRS easily enough by searching on the EPSG number.

If you check the processed survey in \verb}aven}, or run
\verb}3dtopos} on the \verb}.3d} file, the processed entrance
co-ordinates are now indeed

\begin{verbatim}
(398378.00, 474300.00,   334.00 ) dowprov.dowcave.entrance
\end{verbatim}

Whilst this may seem like a crazily over-the-top way to add a `3' and
`4' to the entrance co-ordinates, it is actually very simple to
implement: one only needs to add two lines (the \verb}*cs} and
\verb}*cs out} commands) to the survex file. The benefit is that it
is robust, clean, and unambiguous. Moreover, the output SRS is included
as metadata in the \verb}.3d} file; thus with \verb}dump3d} one sees

\begin{verbatim}
CS +init=epsg:7405 +no_defs
\end{verbatim}

(this is in fact a \href{http://proj4.org/}{PROJ.4} string which species
the map projection, and can be directly pushed to a GIS application).

As a slightly less trivial example, one can ask for the reduced survey
data to be re-projected as UTM co-ordinates. This can be done almost
totally trivially by replacing the previous \verb}*cs out} command
with \verb}*cs out EPSG:3042} which specifies the output SRS is
(WGS84) UTM zone 30N (this includes zone 30U). If we now reduce the data
with \verb}cavern} and check with \verb}3dtopos} we find the Dow
Cave entrance has magically moved to

\begin{verbatim}
(563570.22, 6002262.20,   384.57 ) dowprov.dowcave.entrance
\end{verbatim}

and the exported SRS from \verb}dump3d} is

\begin{verbatim}
CS +init=epsg:3042 +no_defs
\end{verbatim}

As expected, the entrance location in UTM is the same as obtained by
converting the original NGR first to WGS84 latitude and longitude, then
to UTM, using the online converters. Note that in re-projecting to UTM,
we also get a vertical datum shift.

For another example, the CUCC Austria data set which comes as sample
data with the survex distribution can be georeferenced by adding the
following to the top of the \verb}all.svx} file:

\begin{verbatim}
*cs custom "+proj=tmerc +lat_0=0 +lon_0=13d20 +k=1 
    +x_0=0 +y_0=-5200000 +ellps=bessel 
    +towgs84=577.326,90.129,463.919,5.137,1.474,5.297,2.4232 
    +units=m +no_defs"
*cs out EPSG:31255
\end{verbatim}

The first 4 lines are all one line in the real file, and this 
specifies the custom SRS in which the co-ordinates of the
surface fixed points in the Austria data set are specified. The second
line determines the output SRS. This doesn't really matter too much as
long as the SRS can be recognised by the GIS platform: this example uses
the MGI / Austria Gauss-Kr\"uger (GK) Central SRS (\verb+EPSG:31255+), where the
\emph{only} difference compared to custom SRS is in the \verb:+y_0: false
origin. Another output SRS could be \verb}EPSG:3045} which
is (WGS84) UTM zone 33N.

I've gone into these examples in some detail as the survex documentation
on the \verb}*cs} command is rather spartan.

As a further benefit, providing the survex data files include correctly
formatted \verb}*date} commands (as the Dow-Providence dataset does),
the \verb}*cs} commands make survex aware of the geodetic SRS and
magnetic declination corrections can be automatically added. This is
another reason one might want to `do things properly' with \verb}*cs}
commands. The \verb}DowProv.svx} master file thus also contains the
lines (the first two are just comments)

\begin{verbatim}
; Mag dec calculated for SD 97480 72608 alt 225m
; Dowbergill Bridge, just above Kettlewell
*declination auto 97480 72608 225
\end{verbatim}

This correctly applies the magnetic declination using the
\href{https://en.wikipedia.org/wiki/International_Geomagnetic_Reference_Field}{International
Geomagnetic Reference Field (IGRF)} model, calculated at the specified
location in the input SRS, and applied to \emph{all} the included survey
files, in this case taking into account the range of dates which spans
some 30 years.

\subsection{GIS import methods}\label{gis-import-methods}

\subsubsection{Using DXF}\label{quick-and-dirty-two-dimensional-flat-import}

Once the dataset has been georeferenced as just described, a DXF file
can be generated using the survex \verb}cad3d} tool, or exporting as
such from \verb}aven}.  This DXF file can then be directly imported
into QGIS.  At present this route does not neatly capture
\zdimension\ (elevation) data.

\subsubsection{DXF conversion using GDAL}\label{three-dimensional-import}

This route requires command-line access to the
\href{http://www.gdal.org/ogr_utilities.html}{GDAL utilities}.

From the DXF file, the centreline can be extracted by running

\begin{verbatim}
ogr2ogr -f "ESRI Shapefile" DowProv_centreline.shp DowProv.dxf \
  -where "Layer='CentreLine'" -a_srs EPSG:7405
\end{verbatim}

We take the opportunity here to add an SRS to match that used in the
georeferenced survey data. The resulting shapefile can then be imported
in QGIS, and this route \emph{does} preserve \zdimension\ (elevation)
data. 

Similarly the stations with labels (and elevations) can be extracted by
running

\begin{verbatim}
ogr2ogr -f "ESRI Shapefile" DowProv_stations.shp DowProv.dxf \
  -where "Layer='Labels'" -a_srs EPSG:7405
\end{verbatim}

\subsubsection{QGIS import plugins (preferred route)}\label{import-using-qgis-plugin}

Features of these plugins include:
\begin{itemize}
\tightlist
\item no dependencies, natively reads binary .3d files ;
\item import stations and legs with full metadata ;
\item features carry \zdimension\ (elevation) data ;
\item create passage walls, cross-sections, and polygons from LRUD data ;
\item CRS can be set from a PROJ.4 string embedded in \verb}.3d} file ;
\item save results into a GeoPackage (\verb+.gpkg+) shapefile.
\end{itemize}

More details and installation instructions can be
found in the
\href{https://github.com/patrickbwarren/qgis-survex-import}{QGIS2} and
\href{https://github.com/patrickbwarren/qgis3-survex-import}{QGIS3}
plugin repositories.
Additional scripts can be found in the \verb+extra+  directories.

\subsection{Georeferencing images, maps, and old
surveys}\label{georeferencing-images-maps-and-old-surveys}

Georeferencing here refers to assigning a co-ordinate system to an image
or map, or a scanned hard copy of a survey. The actual steps require
identifying so-called Ground Control Points (GCPs), which are
identifiable features on the map for which actual co-ordinates are
known. One way to do this is to use the
\href{https://docs.qgis.org/2.8/en/docs/user_manual/plugins/plugins_georeferencer.html}{GDAL
Georeferencer} plugin in QGIS.
Georeferencing surveys may be easier if there is more than one entrance
and the positions are known, or there is already a surface grid. If
there is only one entrance then tracing a centerline in Inkscape and
using the survex output tool as described
\href{https://github.com/patrickbwarren/inkscape-survex-export}{here}
may help.

\subsubsection{Copying}\label{copying}

These notes are licensed under a
\href{https://creativecommons.org/licenses/by-sa/4.0/}{Creative
  Commons Attribution-ShareAlike 4.0 International License (CC BY-SA
  4.0)}.

These notes are distributed `as is', in the hope that they will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
FITNESS FOR A PARTICULAR PURPOSE.

\subsubsection{Copyright}\label{copyright}

Copyright \copyright\ (2017--22) Patrick B Warren.

\end{document}
