We have been using Hudson to do CI on our grails application for a few months now. We use it to run unit, integration, jsunit, and functional (via selenium) tests, as well as code coverage metrics using Cobertura. Since this was the first project that we used Hudson for, the person who originally set it up just put all of these steps into a single job. It worked fairly well, but a single build was taking between 20-30 minutes, depending on how extensive the changes were. This can be a real pain, especially during very active development cycles when you most need rapid feedback. I decided to look into how I could run the tests in parallel, and hopefully get our build times to under 5 minutes. This is what I have come up with so far. It’s not perfect, but the tests do run more quickly (around 5 1/2 minutes total, primarily limited by the speed of the Selenium tests).
Primary Job – Poll and Build
The primary job is very simple. It polls Subversion and executes clean and test-compile targets. (We use maven with the grails-plugin to compile the app.) It normally only takes 30-40s to run. It doesn’t do a full package/install because that adds another minute to the build time.
Build steps for the primary job
Using the Groovy Plugin
One shortcoming of not running the package step is that the job doesn’t generate a unique file that can be used for fingerprinting like a jar or a war. You need to turn on fingerprinting in order to aggregate test results. (More on that subject, below.)
To resolve this, I used the Groovy plugin to execute a one-line script to generate a unique file for this purpose. I pass in the path and build information as script parameters rather than environment properties because the Groovy plugin doesn’t resolve Hudson environment variables when they are set in the Properties field. This seems like a big shortcoming, so perhaps I just misunderstand how to pass them correctly as properties.
A side-effect of using the Groovy command option rather than invoking an existing Groovy script is that you end up with a copy of the script in your workspace directory. In fact, you get a copy of it for each build. I am not certain why the plugin needs to generate the script file at all as opposed to just running it in memory. Hopefully, this will get corrected in a future release. For now, I may put the script in a regular file just so I don’t have to clear out all the temps.
Downstream Jobs – Test, Test, Test
The tests are divided into 3 downstream jobs that are kicked off when the primary job has a successful build. The jobs run the unit and integration tests, the Selenium tests, and the JsUnit tests, respectively. I didn’t bother splitting out the unit and integration tests into separate jobs because they take less time combined than do the Selenium tests. I could easily split them out later.
Cloning the Workspace
In order to speed up the downstream jobs, I didn’t want to pull the code for each and recompile. Instead, I used the Clone Workspace SCM Plugin to make copies of the primary job’s workspace for each downstream job. This adds only a few seconds to the primary job in order to create the zip archive.
Cloning a workspace - upstream project view
Cloning a workspace - downstream project view
I had two small issues with the plugin:
- The zip command doesn’t include empty directories by default. This can be an issue if any of your tests in downstream jobs expect some directories to be there already.
- In the downstream job, the working directory is set to the workspace directory itself rather than the (sub-)directory pulled from Subversion as it is in the primary job. This makes sense since the Clone Workspace plugin does clone the entire workspace and not just what was pulled from the SCM system. It just threw off some of the tests that expected files at specific paths. (Yes, I know – the tests should be more flexible. It’s “on the list…”)
I need to do more research to see if I can use this approach when I spin up some slave machines. I will post when I tackle that issue.
Downstream Build Steps
The build steps for the downstream jobs look like this:
Unit and Integration Tests Job
Funtional Tests Job
JsUnit Tests Job
You can see that the two jobs which run the server-side test jobs execute a shell command rather than a regular maven target. This was because of the working directory issue I mentioned above. I use the shell to cd into the proper directory first, and then execute the tests. A little hackish, but it was a good experiment with using the Execute shell build step.
You can also see how many different ports are used by the tests – the JsUnit acceptor servlet, the Selenium server servlet, the app itself when launched for Selenium, etc. I use the Hudson Port Allocator Plug-in to keep tests from stomping on each other.
Aggregating Test Results and Fingerprinting
I turned on test aggregation in the primary job so I could get a single view of test results. In order for this to work, the primary job needs to know how to associate specific downstream builds with specific builds in it. This is done through the fingerprinting feature. None of the docs mention this connection. I didn’t figure it out until I did a build, clicked on the Aggregate Test Results link, and saw this error message:
Generating a unique-per-build file using the above Groovy script (discussed above) let me link the jobs together for aggregating.
NOTE: You do not select the “Publish JUnit test result report” option in the primary job as part of this feature. That option is only used in the jobs that actually run the tests. If you turn it on in the primary job, you will get an error during the build because there are no JUnit files for it to process.
The Aggregate Test Results feature is nice since it provides links through to the test results pages for the downstream builds and a simple table listing the fail/total counts from the downstream builds.
Aggregate results report for a good build
Unfortunately, there appears to be a bug in it where it will not report results for any downstream job that has failures. A failing downstream job is shown in the build details page for the primary job
Aggregate results summary with failing job
but you can see that the Aggregated Test Result link lists “no failures.” Clicking that link shows the summary table, but it is missing a line for the downstream job with failures:
Aggregate results, but failing job is missing
In addition to having this bug, the feature does not show the test result trend chart or the tests by packages in the primary build. This makes it of very limited usefulness.
I was able to accomplish my primary goal of cutting our CI build times, but not without losing some required output. Most of the shortcomings for this approach are related to one core issue – Multiple jobs means multiple places to configure and view results. For example, I had to configure failure email notifications in all downstream jobs rather than just one. Also, there is no way to get an aggregated code coverage report that spans all the tests (unit, integration, and functional). I could live with not having a single view of all test failures since I can get that info from the downstream jobs, but not having accurate code coverage metrics is not an option. I have to figure out a way around that.
Since most of the problems with this configuration were related to aggregating the test results (both direct results and coverage stats), my next step will be to try a “diamond” build configuration using the Join Plugin. Hopefully, I can pull all of the test results, coberturra.ser, and other such files into the bottom of the diamond to get a single place to view the status of the build.
I also want to get CodeNarc output displaying for the job via the Violations plugin. I can generate the CodeNarc output, and the Violations tries to parse it, but it then crashes with an NPE. I need to narrow down what part of the file is causing the exception so I can report the issue.