Metro Test Harness User Guide

# JAX-WS/Tango Test Harness User Guide

$Id: index.html,v 1.47 2010-07-21 18:31:28 ramapulavarthi Exp $

## Table of contents

1. Running Tests

1. Configuring the Harness Classpath for Different Delivery Vehicles

1. JAX-WS RI Workspace

2. JAX-WS RI Binary Image

3. WSIT Workspace

4. WSIT Binary Image

5. Overriding jars

2. Running Tests Manually

3. Debugging Test Failures

1. Looking Inside Working Directory

4. Testing Different Transports/Containers

1. In-VM (No Container)

2. Tomcat

1. Embedded Tomcat

2. Remote Tomcat

3. Local Tomcat

3. Jetty

1. Embedded Jetty

4. Glassfish

1. Remote Glassfish

2. Local Glassfish

5. Running Batch Tests

6. Summarizing Test Results

7. Generating Coverage Report

2. Writing Tests

1. Conventions

2. Test Descriptor

1. Linking to Relevant Bugs

2. Specifying test requirements from the Container

3. Specifying verbatim wsimport options

4. Specifying test specific options

3. Client Test Scripts

1. Invoking Services

2. More on Script Execution Environment

3. Common utility

4. Side-effect free tests

4. Java Client

5. Service

1. WSDL

6. Requiring certain modes

7. Resources

8. Common source files

3. Technical Assistance

## 1. Running Tests

### 1.1. Configuring the Harness Classpath for Different Delivery Vehicles

The harness can be used to test different projects that use the JAX-WS RI. Normally, the harness will be able to automatically guess which project you are working on, and set up classpath automatically. However, you can always use one of the following options to manually specify it to override this default behavior.

#### 1.1.1. JAX-WS RI Workspace

The following option tells the harness to pick up the JAX-WS RI classes from the JAX-WS workspace. Specifically, with this option, the harness will look into various `build/classes` and library jars.

    $ java -jar path/to/harness.jar -cp:jaxws path/to/jaxws-ri-workspace test-dir1 test-dir2 ...

#### 1.1.2. JAX-WS RI Binary Image

The following option tells the harness to pick up the RI classes from a JAX-WS binary dist image. Using this option, the harness will only look at the RI library jars found under the `lib` directory of the top-level dist image directory that you specify.

    $ java -jar path/to/harness.jar -cp:jaxws-image path/to/jaxws-ri-image test-dir1 test-dir2 ...

#### 1.1.3. WSIT Workspace

The following option tells the harness to pick up the WSIT classes from a WSIT workspace. Specifically, with this option, the harness will look into various `build/classes` and library jars.

    $ java -jar path/to/harness.jar -cp:wsit path/to/wsit-workspace test-dir1 test-dir2 ...

#### 1.1.4. WSIT Binary Image

The following option tells the harness to pick up the WSIT classes from a binary dist image. Using this option, the harness will only look at the WSIT library jars (webservices.jar & webservices-tools.jar) found under the `lib` directory of the dist image directory that you specify.

    $ java -jar path/to/harness.jar -cp:wsit-image path/to/wsit-image test-dir1 test-dir2 ...

#### 1.1.5. Overriding jars

Sometimes it's convenient to be able to override some of the test subject jars with an override (like applying a patch.) You can use `-cp:override` option to do this.

### 1.2. Running Tests Manually

Each test is packaged in a directory. The following command runs a single test stored in `testdata/abc/def`:

    $ java -jar path/to/harness.jar testdata/abc/def

If you specify multiple directories, all the tests in the specified directories are run.

    $ java -jar path/to/harness.jar testdata/abc/*

The `-r` option tells the harness to recursively search for all tests inside the given directory:

    $ java -jar path/to/harness.jar -r testdata/addressing

| Consider defining an alias to `java -jar path/to/harness.jar` or a wrapper shell script so that you don't have to type the path name manually.

### 1.3. Debugging Test Failures

The following command executes a test with debugger support enabled. Run this command from a console, then set up a remote debugging with your IDE to port 8000. This allows you to set breakpoints anywhere both on the client and the server.

    $ java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=8000 -jar path/to/harness.jar testdata/abc/def

| You'll use this command very often, so consider defining an alias to `java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=8000 -jar path/to/harness.jar`

If your test data resides in the Tango workspace or the JAX-WS workspace, the harness recognizes their directory layouts and pick up classes from respective `build/class` directories and libraries. So if you make a change to the source code, all you need to do is to compile them to class files.

| Do not run any targets that generate jar files, as those unnecessary work will slow you down.

If your change to the code is to the runtime and does not affect the code generation, then use the `-skip` option to skip the code generation. This allows you to quickly rerun the runtime portion of the tests without wasting time running all the tools just to produce the same results.

    $ java -jar path/to/harness.jar -skip testdata/abc/def

#### 1.3.1. Looking Inside Working Directory

The test harness leaves all the artifacts generated during a test execution inside a working directory called `work` in each test directory. Sometimes inspecting what's generated becomes important, so this is the directory structure of the work directory:

    work/                    (root test directory)
     +- client-source/       (client Java artifacts generated by wsimport from service WSDL)
     +- client-classes/      (compiled client Java artifacts. Generated from client-source by Javac)
     +- services
         +- <serviceName>    (this intermediate directory is only if you have multiple services)
             +- war          (root of the exploded war file image that's deployed to the server)
             +- gen-src      (generated server Java artifacts by wsimport)

Unless the `-skip` option is used, the work directory is `"rm -rf"`ed before a test execution. There's no need to manually remove them.

### 1.4. Testing Different Transports/Containers

The harness can run tests by using a variety of transports and containers.

#### 1.4.1. In-VM (No Container)

By default, the test harness uses the in-vm transport. There's no additional option required for this mode. This is the preferred mode of testing because of its quick turn-around time. This mode also allows you to use a single debug session to debug both the client and the server side.

Some delivery vehicles (notably the distribution images) do not have the code needed to use the in-vm transport. To test those, find the matching `jaxws-local-transport.jar` and use the `-transport` option to specify it.

    $ java -jar path/to/harness.jar -cp:wsit-image path/to/wsit -transport path/to/jaxws-local-transport.jar ...

#### 1.4.2. Tomcat

This test harness supports Tomcat 5.0.x or 5.5.x. There are a few different ways of using Tomcat.

##### 1.4.2.1. Embedded Tomcat

The following command causes the harness to run Tomcat within the same VM. This mode is bit like the local transport, in the sense that both the server and the client runs inside the same VM, but this mode run tests by using HTTP and full servlet container.

    $ java -jar path/to/harness.jar -tomcat-embedded path/to/tomcat test-dir1 test-dir2 ...

`path/to/tomcat` should point to the directory where you have Tomcat installed. The harness will load Tomcat jar files from there.

This mode is preferrable mode of testing for Tomcat, as this provides the quickest turn-around time, and worry-free. This mode is the only mode in Tomcat where you can use a single debug session to debug both the client and the server.

##### 1.4.2.2. Remote Tomcat

The following command causes tests to run by using Tomcat launched externally. The parameter to the -tomcat-remote option is `adminUserName: adminPassword@ hostname: port` .

    $ java -jar path/to/harness.jar -tomcat-remote admin:admin@localhost:8080 test-dir1 test-dir2 ...

It is your responsibility to launch Tomcat. In this mode, the harness will only deploy and undeploy services from this running instance. The tomcat used for this purpose should NOT have JAX-WS/Tango binaries installed in `common/lib` or anywhere else. The harness will include everything necessary into a war file, and this is the only way to make sure that you are testing the bits that you are supposed to be testing.

| To find out the admin username and password, look at `$TOMCAT_HOME/conf/tomcat-users.xml` and find the entry that has "manager" as one of the roles. If none is found, simply add the following two lines to the file:
    <role rolename="manager"/>
    <user username="admin" password="admin" roles="manager"/>

| To debug the server side in this set up, you need to launch Tomcat with the debugger support enabled. Run `catalina.sh jpda run` to do this, and you can attach your debugger to Tomcat remotely.

| (Tomcat 5.0.x only) By default, Tomcat is configured to output logs to multiple places, including files in the `logs` directory and stdout/stderr. This is often inconvenient for debugging, as you are not sure which file you should monitor. You can modify `$TOMCAT_HOME/conf/server.xml` to avoid this problem, by sending all the outputs to stdout. To do so, find all "<Logger>" elements in server.xml and replace them with `SystemOutLogger`, like this:
    <Logger className="org.apache.catalina.logger.SystemOutLogger" />

| When tests are aborted via Ctrl+C, web applications may be left running on this remote machine. When the test is run next time, they will be removed (or you can remove them manually.)

##### 1.4.2.3. Local Tomcat

Run the following command to use Tomcat in the local mode. In this mode, the haress launches Tomcat as a child process, run tests, and terminates Tomcat when everything is done.

    $ java -jar path/to/harness.jar -tomcat-local path/to/tomcat test-dir1 test-dir2 ...

See the tips for the remote tomcat support for how to configure Tomcat to report logs to one place.

| On Windows systems, aborting the harness via Ctrl+C will leave the child Tomcat process running (does anyone know how to fix that?) Those orphan processes can be killed from task manager (be careful not to kill other Java processes), or better yet, process explorer.

#### 1.4.3. Jetty

This test harness supports Jetty 6.x.

##### 1.4.3.1. Embedded Jetty

The following command causes the harness to run Jetty within the same VM. See the embedded tomcat description for more about what it means to embed a container.

    $ java -jar path/to/harness.jar -jetty-embedded path/to/jetty-6.1.0 test-dir1 test-dir2 ...

#### 1.4.4. Glassfish

This test harness supports Glassfish 1.0.

##### 1.4.4.1. Remote Glassfish

To test with existing instance of Glassfish, use the following option. This mode assumes that you have already started a domain, and the harness will simply deploy/undeploy applications to this running service:

    $ java -jar path/to/harness.jar -glassfish-remote localhost test-dir1 test-dir2 ...

The "-glassfish-remote" takes the connection string as the parameter. The full syntax of the paramter is `[USER:PASS@]HOST[:PORT][-HTTPURL]`. The first four parts up to '-' specifies the Glassfish admin port. USER and PASS is used for the administrator login, and PORT is the admin TCP port. The last HTTPURL specifies the URL with which client test programs talk to the service. All components but HOST are defaulted, so if your glassfish installation is out of the box, then all you need to specify is the host name where Glassfish runs.

| Glassfish has its copy of JAX-WS, so testing with Glassfish requires us to replace the runtime of Glassfish. At this point the harness is not capable of doing that.

##### 1.4.4.2. Local Glassfish

The test harness is capable of starting and stopping Glassfish on its own before/after running tests. This mode assumes that you have Glassfish installed locally. Use the following command line option to run in this mode:

    $ java -jar path/to/harness.jar -glassfish-local path/to/glassfish test-dir1 test-dir2 ...

The harness will create a domain configuration in a temporary directory and run tests from there, so this running mode will not touch the files in `path/to/glassfish` --- it is simply used to load jar files. All the TCP ports are chosen randomly from 20000-50000 ranges, to avoid collisions with other tests that happen to be running on the same machine. See the console output for the HTTP port and admin port, in case you need to manually access them for debugging. The admin user name and password will be set to "admin" and "adminadmin" respectively.

This mode is convenient for fully automated Glassfish testing, because you do not need a dedicated Glassfish installation per each test set up, and the harness chooses the right TCP ports to avoid conflicts. However, starting and stopping Glassfish takes significant amount of time, so it's almost impossible to use for interactive debugging.

### 1.5. Running Batch Tests

There are a few command-line options that often go with automated batch test for a large number of tests. First, use `-p N` to run tests by using N number of threads. This should be used on a multi-processor system to reduce the turn-around time of the test, and therefore ideal for continuous testing.

### 1.6. Summarizing Test Results

The `-report DIR` option causes the harness to generate JUnit test report files into the specified directory. This allows the reports to be combined with other JUnit or TestNG-based tests (such as per-class unit test.) These XML reports can be then formatted into HTML (via <junitreport> ant task) or picked up by Hudson for further processing.

### 1.7. Generating Coverage Report

The test harness can generate EMMA coverage report of the tests by specifying the `-emma OUTPUTFILE` option. The coverage will include both the runtime as well as tools. See Emma User's Guide for how to generate various reports from the coverage data file.

## 2. Writing Tests

### 2.1. Conventions

A single end-to-end test consists of a client talking to services. Each test is organized into one directory. A test directory could look like the following:

    testdir/
     +- test-descriptor.xml     (metadata file that describes a test)
     +- custom-client.xml       (optional client customization file)
     +- *.bsh                   (client test scripts)
     +- common/                 (optional common source files)
     |   +- *.java
     +- client/                 (optional client source files)
     |   +- *.java
     +- resources/              (optional resource files if needed by test)
     |   +- mascot.jpeg
     +- AddNumbers.wsdl         (optional WSDL that represents service)
     +- **/*.java               (Java code that implements service)
     +- web.xml                 (optional web.xml file, if specified used for packaging the war)

### 2.2. Test Descriptor

The file `test-descriptor.xml` is the heart of the test data, and it contains the required configuration information for each test. A typical "fromwsdl" test descriptor could look like the following:

    <descriptor>
       <description>this is a very useful test that tests stuff</description>
       <client href="*.bsh"/>
       <service>
         <wsdl href="AddNumbers.wsdl"/>
       </service>
    </despcriptor>

The structure of the test descriptor is defined in a schema, and the harness performs an XML validation to make sure the data is in the expected format.

#### 2.2.1. Linking to Relevant Bugs

It is recommended that you link test cases to relevant bugs, so that in the future we can take advantage of information programatically. To do this, use the bugs attribute on the descriptor element. It takes whitespace-separated list of bugs,where each of them are either 7-digit bugster ID or a java.net proeject name followed by an issue number.

    <descriptor bugs="6420352 wsit-123 jax-ws-234">
      ...
    </descriptor>

#### 2.2.2. Specifying test requirements from the Container

If your test relies on a certain capability from the container, Specify such requirements in "uses" attribute. Harness checks the capabilities of the container to continue with running the test or skipping it. Supported values are "servlet30", "servlet","ri","multi-endpoint","skip-lwhs".

Please use this option with caution, so that tests can retain its generality.

    <descriptor uses="servlet30">
        <description>this test runs only on servlet 3.0 based container</description>
        ....
    </despcriptor>

#### 2.2.3. Specifying verbatim wsimport options

If your test requires a certain wsimport command line options, you can specify verbatim by using the nested `wsimport-client` and `wsimport-server` elements. These are passed as-is to when the harness invokes wsimport for generating client artifacts and generating server artifacts, respectively. Options can be separated by tab, space, CR, and NL.

Please use this option with caution, so that tests can retain its generality.

    <descriptor>
       <description>this is a very useful test that tests stuff</description>
       <wsimport-client>-abc -def -ghi</wsimport-client>
    </despcriptor>

#### 2.2.4. Specifying test specific options

If your test special configuration from the default harness behavior, `test-options` is the place to specify those.

    Supported Options are:
        -noPackage : to not use the default harness behavior of specifying -p option to wsimport.

    <descriptor>
       <description>this is a very useful test that tests stuff</description>
       <test-options>-noPackage</test-options>
    </despcriptor>

### 2.3. Client Test Scripts

Client test scripts are the programs that are executed after all the services are set up. These scripts are expected to make some service invocations and perform assretions. Test scripts can be written as:

- files. They can be anywhere and have any name, since the harness locates them through `<client href="...">` inside the descriptor. However, we recommend keeping them at the top directory and use ".bsh" suffix.
- Inline inside the test descriptor, as illustrated below. This is particularly convenient for small test clients. The name attribute indicates human readable short summary of a test:

      <descriptor>
        <description>this is a very useful test that tests stuff</description>
        <client name="echo test">
          assertEquals("hello",fooPort.echo("hello"));
        </client>
        ...
      </descriptor>

Scripts are written in the Beanshell scripting language. BeanShell is mostly compatible with Java, so you can write any valid Java programs, such as:

    import java.math.*;
    BigInteger bi = new BigInteger("35.12");

As a scripting language, BeanShell allows you to save typing. Specifically,

1. You need not perform any explicit casting, since variables are not statically typed.
2. No explicit variable declaration is required (although allowed)
3. Auto-boxing/unboxing ala Tiger
4. Type parameters are not needed (in fact they are not supported)

So you can write a program like this:

    import java.math.*;
    bi = new Holder(new BigInteger("35.12")); // create JAX-WS holder object with BigInteger

See beanshell syntax documentation for more about BeanShell syntax.

#### 2.3.1. Invoking Services

To isolate the tests from the environment, and to make tests as reusable as possible, the harness creates services and port objects on behalf of the test programs. This allows the harness to configure ports correctly, so that they can talk to the right service.

For this reason, it's important for the client program not to create port objects on its own. See the following illustration:

    // :-< BAD EXAMPLE
    port = new HelloService().getHelloPort();
    port.sayHelloTo("duke");

    // :-) GOOD EXAMPLE
    helloPort.sayHelloTo("duke");

To find out what port variables are imported under what names, check the output from the test harness. Right before the client script executes, you should see a line like this:

    injected services: helloService

    injected ports: addNumbersPort

The only exception to this "don't create your ports" rule is when you need `Dispatch` objects, where you'd need to use the pre-defined variable "Address", for ex: helloPortAddress (not fully cooked. Feedback welcome) :

Junit based tests can use System property to access the endpoint address, for ex: System.getProperty("helloPortAddress")

    // create dispatch
    helloService.addPort(new QName("foo","bar"), javax.xml.ws.soap.SOAPBinding.SOAP11HTTP_BINDING, endpointAddress);

#### 2.3.2. More on Script Execution Environment

For productivity, scripts are run in a configured environment. Client scripts are expected to take advantage of those.

1. Script can invoke any of the `junit.framework.Assert` methods without any qualifier. So you can just write `assertEquals(5,3+2)`. This is the prefered way of testing assertions.
2. The following Java packages are implicitly imported.
   - java.lang
   - java.util
   - java.util.cocurrent
   - javax.xml.soap
   - javax.xml.ws
   - javax.xml.ws.soap
   - javax.xml.bind
   - javax.xml.namespace
   - javax.xml.transform
   - javax.xml.transform.sax
   - javax.xml.transform.dom
   - javax.xml.transform.stream
   - (client artifact packages)
   So you could just say `JAXBContext.newInstance(...)` instead of `javax.xml.bind.JAXBContext.newInstance(...)`. Client artifact packages are all imported, too, so for example if you need to create a bean for service invocation, you can do as follows:

       bean = new FooBarBean(); // instead of new test.fromwsdl.helloLit.client.FooBarBean();
       bean.setName("Duke");
       bean.setId(53);

3. Helper methods. The harness defines a few helper methods that are avaialble to the script as if they are global functions. See this source code and this source code for the complete list and their semantics.

#### 2.3.3. Common utility

Often it's useful to factor out the common variable/function/etc for multiple <client> scripts in a single test suite. This can be done by defining <pre-client> element in the test descriptor. If specified, the pre-script will be executed before each client test script is executed.

#### 2.3.4. Side-effect free tests

Many tests are written in such a way that they do not have any side-effect on both server or client (when I say side-effect, I mean things like updating global counters or touching database or anything of that sort.) Test descriptor can mark those tests as "side-effect free" by adding the attribute as follows:

    <descriptor>
    ...
    <client href="abc.bsh" sideEffectFree="true" />
    </descriptor>

With the '-concurrent-side-effect-free' option, tests marked in this way will be run in concurrent environment. That is to say, the same test will be run repeatedly in multiple threads. This allows the runtime to be exercised in a new way.

### 2.4. Java Client

If your client test scripts need some Java helper code (such as handlers), or if you have some JUnit test classes written as Java classes, you can put them under the `testdir/client` directory. All classes that ends with "Test" will be considered JUnit tests and executed like client test scripts.

Also, all the Java source files are compiled and made available to client test scripts (for example so that you can access those classes.) The "client" package itself will be also imported into the client test scripts, so classes in the "client" package can be accessed by their short name.

### 2.5. Service

Code that constitutes a service needs to be written by using Java. By default, those files should be placed into the same test directory. If you are converting existing test services to this harness, most likely the only thing that needs to change is the declared package name.

In a rare circumstance, where you need multiple independently deployed services, you can use `<service basedir="..."/>` value to move service source files into subdirectories to isolate them.

| To set a binding in fromjava scenarios, use the `javax.xml.ws.BindingType` annotation on your service class. For example,
    @WebService
    @BindingType(SOAPBinding.SOAP11HTTP_BINDING)
    class FooBarSgervice {
      ...
    }

#### 2.5.1. WSDL

For scenarios which requiring building the service from a WSDL, there is a sub-element of service named `wsdl` with has an attribute named `href`. `href` should name the WSDL file, relative to the service's `basedir`.

When present, WSDL is first complied and then the artifacts are compiled together with the rest of the Java source files. The harness uses the `-p` option of wsimport to place the generated artifacts into the "right" package (infered from the directory name of the service.)

To further customize the compilation process, place a JAX-WS customization file named `custom-server.xml` to the service top directory (to refer to WSDL file and schema files from this customization file, just use the relative path from the customization file to WSDL/schemas like `"./foo.xsd"`.) You can also use `custom-schema-server.xml` to specify additional JAXB customization file.

### 2.6. Requiring certain modes

A test can declare that it requires a particular kind of a transport by declaring the transport attribute. The possible values are the values you specify in the scheme portion of the endpoint address URI (The major values include "http" and "local", but open-ended as new transports are added.)

    <descriptor transport="http">
      ...

A test can also declare that it's applicable only to a certain verion(s) of JAX-WS/WSIT, by using the version attributes `@since`, `@until`, and `@excludeFrom`.

1. The @since attribute indicates that a test will require the given version of the test subject or later. For example, `since="2.0"` indicates that it will run on 2.0 FCS, 2.0 EAs, etc, but not on 1.1.
2. The @until attribute indicates that a test can only be run against the given version of the test subject or earlier. For example `until="1.*"` indicates that the test will run on all versions of 1.x but not on 2.0.
3. The @excludeFrom takes a list of whitespace-separated version numbers, and it indicates that tests be excluded from these specific versions. This is used to mark known regressions and such things.

Those three attributes can be combined, and they can be specified either on <descript> element (in which case the entire test is subject to the constraints), or on individual <client> element (in which case only that client test is subject to the constraints.)

See this document for more formal definition of what can be a valid version number, and how they are ordered.

Similarly, Junit based tests can specify version requirements by using annotation @com.sun.xml.ws.test.VersionRequirement(since="2.1.*", until="2.2",excludeFrom="") on the TestClass.

### 2.7. Resources

If test programs need to use image files, binary files, or other data files, they should be put in `resources` directory at test's top-level. This is optional. Those resources can be loaded by using the helper methods as follows:

    File f = resource("foo.jpg");  // load resources/foo.jpg

For resources that are XML, you can define them by using <xml-resource> elements.

    <!-- define it inline -->
    <xml-resource name="foo">
      <echoRequest>123</echoRequest>
    </xml-resource>

    <!-- define it externally -->
    <xml-resource name="bar" href="my.xml" />

Such XML resources are injected into clients as variables of the type `XmlResource` , so you can use them like this:

    echoPort.echoString(foo.asString());

### 2.8. Common source files

If test programs need to share some Java source files between the server and the client, those files can be put into the 'common' directory. They will be compiled and made available to both client and server.

(not fully cooked. Feedback welcome) we are hoping to define more convenience methods and define how to access resources from server.

## 3. Technical Assistance

Talk to Ken, Bhakti, Bobby, or Kohsuke if you have technical problems with the harness and need in-person assistance.

We use the java.net ws-test-harness project for developing the harness. We use the `dev@ws-test-harness.dev.java.net` alias and the SWAN IRC `#ws-test-harness` for communication.

Our goal is to improve your developer productivity. Please use the issue tracker for keeping track of RFEs and problems.