PA Runner for Continuous Integration

by Paul Bianciardi » Fri Dec 19, 2014 14:40 pm


This is a guide to integrating Performance Analyser (PA) into a continuous integration (CI) build chain. It has been tested with Jenkins and Team City, but should also work with other solutions.

It involves using a simple, self-contained ‘runner’ that allows PA jobs to be run externally, as well as extracting the run-level data. This is then extended further to allow assertions (via performance budgets) to be defined. Tests must be within those budgets to be considered successful, so you can then fail a build chain if they do not comply.

The runner will execute based on the given configuration, and simply finish with an exit code of 0 if the tests have passed or 1 if any of the performance budgets have been breached.

Most CI tools can detect the failure state of an external application and so can interface with the runner easily.

Application configuration and installation


There is a library dependency from HTTP Components:  From this, you need:

  • httpclient.jar
  • httpcore.jar
  • commons-logging.jar

Note that the actual downloaded JARs will be versioned in the filename, for example httpclient-4.3.6.jar

Download and extract the binaries listed. You will need these three, plus the main pa-runner.jar:

Download pa-runner.jar

Jenkins portal configuration

This will walk you through setting up a new Jenkins build job.

1. Create a new Jenkins item. Give it a name (e.g. Check-main-sites) and select Build a free-style software project.

image2014-12-18 15-14-44

2. Click OK to get to the configuration page.

3. Scroll down and under Build, select Add build step, then the Execute shell option:

image2014-12-18 15-34-28

4. Add the following as the command to run:

java -cp pa-runner.jar:httpclient-4.3.6.jar:httpcore-4.3.3.jar:commons-logging-1.1.3.jar: app.PAJobRunner config.json

This command includes the parameter config.json at the end. This is the configuration that the PA Runner will use. It is documented below, but can be named anything.

If you want a different configuration file name or location, change the config.json parameter accordingly.  If no parameter is defined, config.json is assumed to be in the same location as the other files.

5. If you add an output file for the runner (see configuration options below), then you can have Jenkins save this as an artefact for retrieval later on. We will assume you have configured it to be called paRunnerOutput.log, in which case, add the following Jenkins configuration to the Post-build Actions section:

image2014-12-18 15-39-58

Ignore any messages about the file not matching anything yet – that’s because it’s not been run yet.

If you don’t set an output file in the configuration, leave this section blank.

6. Save the new Jenkins item

Jenkins server configuration

Now that you have created the new build configuration, you should have a matching workspace on the Jenkins server. This will have the same name as the item you created (Check-main-sites in our example).

Copy the four JAR files referenced in the dependencies above into this folder.

This is also the folder into which you put the configuration file (in our example, config.json).  The content of this file is documented below.

Team City build configuration

1. Create a new command line build configuration in Team City, specifying the working directory and the custom script that is relevant to your environment.


2. If Java is not in the path for the Team City build user, simply add the full path to the java executable, for example:

C:\ProgramData\Oracle\Java\javapath\java -cp pa-runner.jar;httpclient-4.3.6.jar;httpcore-4.3.3.jar;commons-logging-1.1.3.jar; app.PAJobRunner config.json

Note that the classpath separator when running under Windows is different than for Jenkins under Linux, as it uses semicolons instead of colons, as shown in the above example.

3. You can store the output log from the runner as an artefact stored with each Team City build.  In our example, we are outputting to a file called paRunnerOutput.log (as configured in the config.json file).

Go to the General Settings for the Team City build and add the artefact path, for example C:\PA\Runner\paRunnerOutput.log


Configure your PA job

In your PA account, create a new job for the URLs you want to test against.  For example, you might want a multi test to check a variety of your key URLs.

In our example, we’ve selected a number of BBC URLs:

image2014-12-18 15-58-3

Note that the PA Runner will check against URL, so ensure they are unique per job. If you need the same URL in different browsers, put these into different job templates and run more than one.

Save the job template (you can also run it to ensure it is testing the URLs you want).

We need the ID of the job template, so go to the Run/Edit, Templates page, hover over the runner icon and look at the URL of the link:

image2014-12-18 16-3-59

Make a note of this ID (12345 in this example).

PA Runner configuration

The PA Runner relies on a configuration to define how it should connect to and run PA jobs and also what performance budgets you want to check against.

By default this will be config.json, but can be changed as detailed above.

Here’s a sample config file, which is in JSON format:

    'config': {
        'portalHubURL': '',
        'paPortalURL': '',
        'username': '',
        'password': 'mypassword',
        'accountName': 'My PA Account',
        'jobTemplateId': '12345',
        'maxSecWaitForJob': '300',
        'pathForOutput': 'paRunnerOutput.log'

    'perfBudgets': [
            'url': '',
            'budgets': [
                    'download duration (sec)': '5.0',
                    'page size (bytes)': '400000',
                    'object count': '80'
            'url': '',
            'budgets': [
                    'download duration (sec)': '15.0',
                    'page size (bytes)': '1200000',
                    'object count': '200'

The config object is used to define the key configuration for the whole runner, with the following options available:

Property Optional Description
portalHubURL Optional The URL of the main portal
If not specified, the default value used by the application will be:
paPortalURL Optional The URL of the main portal. If not specified, the default value used by the application will be:
username Mandatory The username used to log in to the portal
password Mandatory The password used to log in to the portal
accountName Mandatory The PA account name to use for running all PA jobs (as you see when you log into the portal hub as a user)
jobTemplateId Mandatory The ID of the job template that new jobs will be created from, as detailed above – in our example, this is 12345
maxSecWaitForJob Optional The maximum number of seconds to wait for the job to finish. If not specified, the default of 300 seconds will be used. If the job does not finish in this amount of time, this will be considered a run failure in Jenkins
pathForOutput Optional If specified, then a log file containing all of the budget assertions and all of the performance data will be written to that location after each run

Note that the runner will log into the portal hub and PA account as the user you specify in the configuration file via the username and password parameters.

You may want a specific account that has access only to the PA account within which you want to run tests.  Account admins can set up their own user accounts – alternatively, contact Customer Support for help with this.

You will also see the pathForOutput parameter.  If set, this will be the file location where the runner will create an output with all the results.  This filename should match what was set in Jenkins, with respect to archiving the artefacts, as documented above.

Use the perfBudgets section to build up the performance budgets you want to check against. Each budget requires a URL, which is what the runner will check against within the job. For that URL, you can then define multiple sets of budgets, all of which must be met for the job to pass.

The metrics that can be set as budgets are any of those available at the test run level, for example:

  • render start (sec)
  • dom load (sec)
  • on load (sec)
  • http load (sec)
  • download duraton (sec)
  • visually complete (sec)
  • speed index
  • page size (bytes)
  • object count

The value you set is the value at which you consider the budget breached; i.e. the actual value must be less than the value in the configuration, for it to pass.

Seeing it in action – Jenkins

That’s it – the configuration is all done now.

Run a build in the normal way, via the Build Now link:

image2014-12-18 16-16-10

You can also access the Console Output as shown to see how the run is going, for example:

image2014-12-18 16-27-28

In the console, you will see the PASS or FAILURE of every URL being checked, against each budget.

This information is also included in the artefact if you set that up, which was paRunnerOutput.log in our example.  In addition, you will get all of the performance metrics in the artefact, which is useful for looking at previous runs, for example:

image2014-12-18 16-30-4

That’s it!  You can include this new Jenkins configuration in your existing build chain, such as running on commit, etc.

Seeing it in action – Team City

Click on the Run… button and a new build will start:


You can also see it running in real time by looking at the Build.log from the running drop down:


This will show you the output of the runner in progress (or after it’s finished):


The result of this will be a PASS or FAILURE and the Team City build will pass or fail accordingly.

Looking back historically

You can look back at the runner output or log output (if you configured the artefacts) by looking at the build history.  This allows you to review the build.log or paRunnerOutput.log for any of the builds.


Leave a Reply

Your email address will not be published. Required fields are marked *