Farrago Developer Jumpstart

# Getting Started With Farrago

This document provides instructions on how to build and run Farrago and its dependencies (including Fennel).

## Supported Platforms

So far, the Fennel C++ library only builds and runs on Linux and Windows. This means that if you want to get a fully functional Farrago build, you have to be running one of these operating systems. However, the Java portion is likely to run on other platforms where JDK 1.5 is available. You can use it to test parsing, validation, optimization, and the metadata aspects of DDL, as well as queries against foreign data accessible via Java (a sample HSQLDB source is provided for testing JDBC, and the MDR HOWTO explains MDR/JMI usage).

So, if you're working on a task that doesn't require Fennel, you can do the development anywhere. If you need Fennel, then you have a few options:

- use your own Linux or Windows box if you have one (Linux is strongly recommended; the Windows C++ build/debug environment is slow and touchy)
- use one of the Sourceforge compile farm machines (Fennel has been successfully tested on some of them)
- work on porting Fennel to your platform!

### Important Windows Note

If you are using Windows, please read and follow the Cygwin setup instructions before going any further.

## Get the Source Code

There are two ways to get the Farrago source code; either download a full Eigenbase source release or pull the latest code from the Eigenbase Perforce server. (If you just want to take a look-see through the codebase, you can use p4web.)

If you have downloaded a full Eigenbase source release, you can skip the rest of this section about Perforce and continue on with Initial Build; note that in this case we'll refer to the top-level directory created when you unpacked the release archive as `dev` in this and other developer docs.

If you're new to Perforce, you might want to read the official Perforce Quick Start. If you don't already have a p4 client program, download the appropriate binary from Perforce (you only need the client, not the server).

There are a number of ways to configure Perforce depending on the platform (on Windows through the GUI, on Unix through environment variables or a .p4env file). The essential information you need to provide is:

- P4PORT=perforce.eigenbase.org:1666
- P4USER=your-sourceforge-user-name
- P4CLIENT=your-user-name.your-hostname

In order to make changes, you'll need to send your username to the Farrago project admin and request write access. If you don't have a Sourceforge account yet, go get one. For read-only clients (like if you just want a snapshot of the latest code), use the predefined username "guest" (the server license has a user limit, but there's no limit on the number of clients, so feel free to create multiple workspaces on as many machines as needed).

Once you've got Perforce set up, run `p4 client` (or edit your default clientspec from P4WIN). Map your view to match this example:

    View:
            //open/dev/thirdparty/... //YOURCLIENT/dev/thirdparty/...
            //open/dev/fennel/... //YOURCLIENT/dev/fennel/...
            //open/dev/farrago/... //YOURCLIENT/dev/farrago/...

We'll refer to the `YOURCLIENT/dev` local filesystem directory as `dev` in this and other developer docs.

While editing your client, you should also set the `LineEnd` attribute to `share`, especially if you are developing on Windows or using Samba. This prevents DOS CRLF's from getting checked in.

Finally, run `p4 sync` to get all the source files and thirdparty distributions into your client workspace. This may take a while because some of the thirdparty distributions are large.

## Initial Build

Before proceeding, make sure you have JDK 1.5 or higher installed, with JAVA_HOME set to the JDK installation root.

For the first build in a new sandbox, there's a lot of one-time setup that needs to be done, including building some of the third-party components from source. This process is automated by the script `dev/farrago/initBuild.sh`. Change to the `dev/farrago` directory and run the initial build as follows:

    initBuild.sh --with-fennel

      or

    initBuild.sh --without-fennel

If you're building without Fennel, this should be fairly quick (under ten minutes on most machines); with Fennel it will take quite a bit longer. If all goes well, you'll be left with a complete Farrago build for your platform. Otherwise, you'll need to dig into the script and its output to see what's going wrong. If you can't figure it out, ask for help on the farrago-developers mailing list. You should probably include the last 100 lines of build output including the error.

### Fennel and Autotools

If you are building with Fennel, the initBuild script will check to see if you have a autotools versions (autoconf, automake, and libtool) which are known to work; if you don't, the script will fail before starting the C++ build. If this occurs, you can either follow the directions in the script error message to build and install the autotools from thirdparty, or update your machine using the mechanism provided by your OS distribution (Linux or Cygwin).

If you need help with problems in the C++ portion of the build, use the fennel-developers mailing list.

## Environment Variables

The initial build script takes care of setting some necessary environment variables internally, but once it completes they are left unset. For subsequent runs and builds of Farrago, you will need to make sure these are set. Scripts are provided for this purpose. You can run them automatically from a login script such as `.bash_profile`, or run them manually before working with Farrago:

    source dev/fennel/fennelenv.sh dev/fennel
    source dev/farrago/farragoenv.sh dev/thirdparty

As usual, you only need the Fennel script if Fennel support is enabled. Don't execute these scripts directly; you must use `.` or `source` in order for the environment variable settings to take effect for subsequent commands.

## Incremental Builds

Once you already have a working build, updating to the latest code (assuming you're using the Perforce server) requires following these steps:

1. `p4 sync`. You'll probably want to examine the output to see what has changed (or you can ignore it and blindly carry out the rest of the steps unconditionally).
2. If anything under `dev/thirdparty/...` has changed, cd to that directory and run `make all`. New thirdparty components and upgrades to new versions are fairly rare. Upgrades to C++ components usually require you to repeat the original initBuild.sh process.
3. If anything under `dev/fennel/...` has changed (and you're building with Fennel support), cd to that directory and run `make`. If it has been a long time since your last sync, you may get errors, in which case you should run `make clean` and then try again. If you still get errors, see below on forcing a full rebuild.
4. cd to `dev/farrago` and run `ant test` to make sure you'll be working from a good build (WARNING: this will rebuild your catalog automatically, destroying any data in your Farrago database). To avoid running the tests, run `ant createCatalog` instead.

If none of the C++ components under thirdparty have been upgraded, and you'd like to force a rebuild of everything else, skip the steps after the sync and instead run

    cd dev/farrago
    initBuild.sh --with-fennel --skip-fennel-thirdparty-build

This will recapitulate the original full build, but skipping the time-consuming thirdparty C++ components.

## Catalog Dump

At any time, you can get a dump of the catalog by running `ant exportCatalog`; the output will be an XMI file in `dev/farrago/catalog/xmi/FarragoCatalogExport.xmi`. This can be useful for debugging the effects of DDL execution.

## Catalog Backup/Restore

The `ant createCatalog` task (invoked automatically as part of `ant test`) creates a backup of the initial catalog (and database if Fennel is enabled). If your catalog gets corrupted, you can recover with `ant restoreCatalog`. This is a lot faster than `ant createCatalog` (which is another way to get back to a good state).

## Interactive SQL

You can execute interactive SQL from a command-line via the open-source JDBC sqlline project. Just run the `sqllineEngine` script from dev/farrago. After connecting, try `select * from sales.emps;` (but if you disabled Fennel, don't be disappointed when the result set comes back empty).

If you prefer using a GUI, try the `dbmanEngine` script instead.

## Tracing

See the wiki docs for instructions on tracing Farrago.

## Unit Testing

See the wiki docs for information on Farrago unit tests.

## Debugging

The developer FAQ contains some useful information on debugging, including how to set up Eclipse.

A standalone debugger (JSwat) is also provided in the thirdparty distribution. Run `ant jswat.sqlline` or `ant jswat.isql` to use this debugger for execution of interactive SQL commands. For sqlline, input/output is performed through the JSwat Output pane at the bottom left of the screen. The `junitDebug` script can also be used to debug one unit test at a time. If you want to use another debugger, the `jswatDebug` ant macro definition in `dev/farrago/buildMacros.xml` might be helpful as a template for passing the right startup information. Also, as part of a successful build, the full classpath is always dumped to the file `dev/farrago/classpath.gen`.

## Client/Server Operation

So far, all of the execution modes discussed have been embedded engine mode, where the Farrago engine is loaded directly into a single process such as sqlline. This is the most useful for debugging, but scripts are also available to test client/server mode. Run the `farragoServer` script to start the server in a shell (after startup, it waits listening for connections; enter the `!quit` command in that shell to shut it down). From another shell, run `sqllineClient` or `dbmanClient` to connect and execute commands.

## Coding

Please read the coding conventions documentation. Before starting development projects, consider using private branching.

## Feedback

If you have trouble with any of this, or have other questions, use the farrago-developers mailing list.