Lobo - Continuous Tuning

Lobo, Continuous Tuning

GETTING STARTED

First of all, get the java archive from the OnCast site.
The jar file has everything you need to write, run and report performance tests over your application.
There are two versions of the jar file: one with all dependencies embedded and one without. You can find them at the following links:

Lobo Jar
http://www.oncast.com.br/dev/lobo/dist/oncast-lobo-1.0.alpha.jar

Lobo Jar with dependencies
http://www.oncast.com.br/dev/lobo/dist/oncast-lobo-with-deps-1.0.alpha.jar

An alternative would be downloading the full Lobo package which contains:

The libraries (both with and without dependencies);
The dependencies;
The getting started document;
The user manual;
The GPL license;
Usage examples;
and the source code.

The full package can be found at:

http://www.oncast.com.br/dev/lobo/dist/oncast-lobo-1.0.alpha.zip.

For information on version updates and other Lobo material, always refer to the Lobo web-site:

http://www.oncast.com.br/dev/lobo

2. Writing performance tests

You want to performance-profile some code, let's say, the method "search" in the following class:

public class SearchEngine {
   public static int search(List<Integer> list, Integer i) {
      return Collections.binarySearch(list, i);
   }
}

Let's write a profile-case to collect performance metrics on the code above. This is just another java class with an annotation:

import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import br.oncast.dev.lobo.Profile;

public class ProfileSearchEngine {
   @Profile
   public void profileSearch() {
      SearchEngine.search(generateAThousandNumbers(3), 3);
   }

   private List<Integer> generateAThousandNumbers(int unique) {

    (...)

   }
}

That's it! You are now monitoring the performance of the code in the annotated method body!

Lobo is able to profile portions of the profile scenario method body if you want. Refer to the user manual to find ways to do that.

3. Running performance tests on Ant

Now that we have an annotated profiling scenario we must run the performance tests.
The purpose of this step is basically to output an XML file with the result metrics from the annotated scenarios. The file name will be profile-results.xml and its content will be something like this:

<profile-report>
    <profile-case case-class="ProfileSearchEngine">
        <profile-scenario scenario-method="profileSearch">
            <profile-metric key="scenario-timespan" type="SINGLE" value="300.0" />
        </profile-scenario>
    </profile-case>
</profile-report>

To do that let's make use of the Ant build infra-structure. What we need is a build file like this:

<?xml version="1.0"?>
<project name="MyProject" default="build-and-profile">
    <path id="profiler-classpath">
        <fileset file="lib/oncast-lobo-with-deps-1.0.alpha.jar" />
    </path>
    <taskdef name="profile" classname="br.oncast.dev.lobo.task.ProfilerTask" classpathref="profiler-classpath" />

    <target name="build-and-profile">
        <mkdir dir="build" />

        <mkdir dir="build/classes" />
        <javac srcdir="src/java" destdir="build/classes" />
        <javac srcdir="src/performance" destdir="build/classes" classpathref="profiler-classpath" />

        <profile output="build/profile-results.xml">
            <classpath>
                <path location="build/classes" />
            </classpath>
            <batchProfile>
                <fileset dir="build/classes" includes="**/Profile*.class" />
            </batchProfile>
        </profile>
    </target>
</project>

The snippet marked in bold green defines the profile task, the one that actually runs the profile cases.

The snippet marked in red runs the defined profile task. It is composed by the tags classpath and batchProfile:

The classpath tag defines the classpath to be used. It doesn't need to include the profile classes;
The batchProfile tag specifies the files that define profile cases. Note that these classes must be actually classes (.class), in other words, must be already compiled.

Run this build and you'll have your code profiled.

4. Collecting results for comparison

In this point we already have an XML file, called profile-results.xml, with the collected performance metrics. But, what good makes an XML? Well, if you have enough of them, you'll be able to track how your performance evolved due to your refactorings and improvements.

So, let's collect them!

Lobo defines a standard way to collect and maintain the profile information for posterior referencing and reporting. It also makes use of Ant and is perfect for an Ant based continuous integration system like CruiseControl.

Here is what you have to add to your CruiseControl build file to collect and store the performance metrics generated in the previous step:

This snippet represents the lines that should be added to your build file to join several build metrics XML files into a single merge XML. A good place to place this task is in your master CruiseControl build file. With this, it will store the metrics collected on each build.

After this task is executed for the first time, a merge.xml file is created with the information of the profile-results.xml. Every subsequent calls to this task will append into the merge file the metrics present at the profile-results.xml.

So, once merged, you may consider the information found in the profile-results.xml safe. Remember, of course, to maintain the merge.xml in a safe place. Every time you have new profile information, run the merge task again and it will be stored in the merge.xml file.

Ok, we had a lot of XML files, now we have just one. How does it help me monitor performance? Take a look at the following topic...

5. Reporting performance evolution

If you came till here you probably have a way to collect performance metrics for all your builds and store them into a single, centralized, place. To take advantage of all this information we need better ways to interpret them.

The Lobo offers a reporting facility that help you compare the values of the metrics of different builds. For instance, if you have a metric that exposes the response time for a web-service, you'll be able to see how the web-service response time evolved during the development.

The reporting tool will generate a graphic for each of the metrics defined, like the following:

If these were the response times for your web-service we could say that between builds 1 and 2 you did a great improvement in performance, but it started growing back from builds 2 to 3.

How to generate this graphic? Another Ant task. This one now takes the merge.xml file as input and outputs the graphics organized in an HTML. Let's get to it:

The profile-report task does all the formatting. In this case, it will take the information found in the merge.xml file generating all the reports to the report directory.

A good place to place this task would be right after generating the merge file, so you would have always the most up-to-date metrics report at your disposal. Remember to put a link in your build artifacts page to the performance report, so guaranteeing the visibility.