My Test Setup

This post talks about the Functional testing Setup that I am using at work. Our product is a small Web based product with a PHP backend and a XForms / normal Web Forms front end running on Apache Web Server. We use a combination of tools that help us in testing the product.

Redmine : This is a issue tracking system written in RoR (Ruby on Rails). It allows multiple projects to be managed. We set up a separate project for Test Cases. Each test case is entered as an issue in Redmine. Redmine also allows the definition of categories on the fly . It allows defintion of custom fields and updation of custom fields using a Web API. Redmine allows a system wide definition of a query. It also allows this Query to be queried over RSS . The following is an example of how we use Redmine in our tests


@Test
@CustomTest(1470)
public void checkPermissions() {
}


This isvery helpful in the following ways :

• All search operations can be performed on the Web using Redmine's search facilities. Tests can be searched by category / exported to PDF etc
• It is easy to have discussions on the test suites in the Wiki / Forums
• Pseudo code can be explained in the Redmine server instead of the documents
• Test Description is shown in the Test Results (as part of TestNG) which helps in the person reviewing the results to understand the test results in context
  

What is missing :

• From the IDE (IntelliJ) I would like to click on the @CustomTest(1470) and open the browser
• We need to write an utility that would parse the groovy code and update Redmine so that it will indicate the tests implementing this

Alternate systems :
We tried using Test Case Management Systems but somehow we were not successful in maintaining them. The main problem we had was that the test case description in the tool would be very different from the actual tests that are implemented.

TestNG: This is the test driver and is extremely powerful with good support for annotations and a great community. It has a very good support for data driven testing. All our tests are written are written in groovy and this plays very well with TestNG. We use TestNG ant task to orchestrate our testing and this uses the testng.xml . Most of our tests use the @Factory methods to read the data from an xml file. We use Groovy's neat XML Parser methods which makes this a breeze . This is helpful in the following ways :

• Use the same method in different tests by passing different parameters in the xml files. This can be achieved by using the @Param in the testng.xml file
• It has a great listener support and a great reporting system . We were able to pick up the test description from Redmine and display them in the test results. All tests were also hyperlinked to Redmine
  

Hudson This is the Continous Integration server that we use. We use this in conjunction with a few ant taskts. We use Hudson to check out from Subversion and then use a ant task to set up databases and restart apache. Hudson then invokes another ant task which starts the TestNG tests . Hudson is useful to keep the results and lets you track changes and failures to the tests. One important feature of Hudson that we use is the chained build (This is a hudson plugin) which starts one job on the completion of a previous build. Hudson has very good support for over riding values in the build.properties . This allows us to check in the build.properties but over ride them in the Hudson job for a daily run

Tellurium This is the heart of the whole system and this implements the Test cases that are run by TestNG. This makes functional testing easy to maintain