My Test Setup

This post talks about the Functional testing Setup that I am using at work. Our product is a small Web based product with a PHP backend and a XForms / normal Web Forms front end running on Apache Web Server. We use a combination of tools that help us in testing the product.

Redmine : This is a issue tracking system written in RoR (Ruby on Rails). It allows multiple projects to be managed. We set up a separate project for Test Cases. Each test case is entered as an issue in Redmine. Redmine also allows the definition of categories on the fly . It allows defintion of custom fields and updation of custom fields using a Web API. Redmine allows a system wide definition of a query. It also allows this Query to be queried over RSS . The following is an example of how we use Redmine in our tests


@Test
@CustomTest(1470)
public void checkPermissions() {
}


This isvery helpful in the following ways :

• All search operations can be performed on the Web using Redmine's search facilities. Tests can be searched by category / exported to PDF etc
• It is easy to have discussions on the test suites in the Wiki / Forums
• Pseudo code can be explained in the Redmine server instead of the documents
• Test Description is shown in the Test Results (as part of TestNG) which helps in the person reviewing the results to understand the test results in context
  

What is missing :

• From the IDE (IntelliJ) I would like to click on the @CustomTest(1470) and open the browser
• We need to write an utility that would parse the groovy code and update Redmine so that it will indicate the tests implementing this

Alternate systems :
We tried using Test Case Management Systems but somehow we were not successful in maintaining them. The main problem we had was that the test case description in the tool would be very different from the actual tests that are implemented.

TestNG: This is the test driver and is extremely powerful with good support for annotations and a great community. It has a very good support for data driven testing. All our tests are written are written in groovy and this plays very well with TestNG. We use TestNG ant task to orchestrate our testing and this uses the testng.xml . Most of our tests use the @Factory methods to read the data from an xml file. We use Groovy's neat XML Parser methods which makes this a breeze . This is helpful in the following ways :

• Use the same method in different tests by passing different parameters in the xml files. This can be achieved by using the @Param in the testng.xml file
• It has a great listener support and a great reporting system . We were able to pick up the test description from Redmine and display them in the test results. All tests were also hyperlinked to Redmine
  

Hudson This is the Continous Integration server that we use. We use this in conjunction with a few ant taskts. We use Hudson to check out from Subversion and then use a ant task to set up databases and restart apache. Hudson then invokes another ant task which starts the TestNG tests . Hudson is useful to keep the results and lets you track changes and failures to the tests. One important feature of Hudson that we use is the chained build (This is a hudson plugin) which starts one job on the completion of a previous build. Hudson has very good support for over riding values in the build.properties . This allows us to check in the build.properties but over ride them in the Hudson job for a daily run

Tellurium This is the heart of the whole system and this implements the Test cases that are run by TestNG. This makes functional testing easy to maintain

Experience with Web Testing Frameworks

This series of posts details my experiences with Web Testing Frameworks. These are completely personal observations and your mileage may vary. I will write about using Selenium , Canoo Web Test and Tellurium.

Cutting to the end of the chase : We have chosen to stick with Tellurium since it matches our needs the most . In this post I will detail why we chose Tellurium and some of the important concepts of Tellurium. In the next post we will look at an example of using Tellurium for testing


I am not a QA person and most of my testing is from a Development / Unit Testing / Regression testing point of view. One of the main problems we had with a separate QA team trying test automation was that the QA team generally lagged behind the product cycle. Things change in the product and it was difficult for the QA Team to invest in an automation test . We decided to have the development team own the automation tests . We had a framework to run the automtaion tests daily and this detected changes more frequently.

We were looking for a framework that was

• Easy for the java developers to use , understand and extend
  
• Modular and will typically allow changes to be localised
• Something that can be used with the same java IDE to debug

Other essential attributes were

• Open source with an active community to support
• Integration with a Testrunner framework to enable data driven testing
  

Tellurium fits all of the above. The difficult thing about Tellurium was the difficulty in locating it . Searching in google for Tellurium never yielded for much :-)

The good thing about Tellurium was that it had very good documentation. There are examples with live website (Tellurium's own wiki) which has a sufficient variety and beyond the usual "Hello World" examples. There are complete tests that show integration with Test runners like

• TestNG
• JUnit

I am a major fan of TestNG and the examples helped to jumpstart with Tellurium and TestNG.

A major strength of Tellurium which distinguishes it from Selenium is the fact that it allows for separation of UI from the Test framework. This is very important and helps you write maintainable code over a long run. It allows to bundle a page / set of related pages into a UI module and allows to define widgets which can be reused.

Two of the most important concepts of Tellurium are the Table and List UI widgets which help in handling repetitive content. The Container widget also is handy for maintaining a structure.

That completes an introduction to why I chose Tellurium as a testing framework. In the next post I will write about selectors and look at tests using Tellurium

Unit Testing

There has been lot of writing on why Unit Testing is important as a software developer. Common Sense dictates that you do Unit Testing. But software engineers rarely do. Voltaire was right. The typical progression is something like this :

When you start a feature you diligently start testing . Some assumptions are made. As the feature develops the assumptions change , become invalid. On completing a feature some complicated features get tested but basic features get missed out. If the company that you work for does not make it mandatory for Unit Testing then this completely gets dropped by the wayside until a customer notices it . Or worse it shows up when the Big Boss demonstrates this to a high powered investment team

One of the reasons I feel that Unit Testing does not get done is because of the over confidence of the software developer . Engineers assume that using standard components work . The do not factor for Murphy's law The answer in all cases to the question "Have you tested the feature? " is almost the same
But the manual says that this should work . How can this not work.

Unit Testing is also expensive. It requires a lot of investment of time and effort. And it requires a steep learning curve. It is not a Fill It Shut It style of work. It needs constant tweaking . As a software developer time is premium and it makes complete sense to spend most of the time on a software development rather than unit testing .