My Test Setup

This post talks about the Functional testing Setup that I am using at work. Our product is a small Web based product with a PHP backend and a XForms / normal Web Forms front end running on Apache Web Server. We use a combination of tools that help us in testing the product.

Redmine : This is a issue tracking system written in RoR (Ruby on Rails). It allows multiple projects to be managed. We set up a separate project for Test Cases. Each test case is entered as an issue in Redmine. Redmine also allows the definition of categories on the fly . It allows defintion of custom fields and updation of custom fields using a Web API. Redmine allows a system wide definition of a query. It also allows this Query to be queried over RSS . The following is an example of how we use Redmine in our tests


@Test
@CustomTest(1470)
public void checkPermissions() {
}


This isvery helpful in the following ways :

• All search operations can be performed on the Web using Redmine's search facilities. Tests can be searched by category / exported to PDF etc
• It is easy to have discussions on the test suites in the Wiki / Forums
• Pseudo code can be explained in the Redmine server instead of the documents
• Test Description is shown in the Test Results (as part of TestNG) which helps in the person reviewing the results to understand the test results in context
  

What is missing :

• From the IDE (IntelliJ) I would like to click on the @CustomTest(1470) and open the browser
• We need to write an utility that would parse the groovy code and update Redmine so that it will indicate the tests implementing this

Alternate systems :
We tried using Test Case Management Systems but somehow we were not successful in maintaining them. The main problem we had was that the test case description in the tool would be very different from the actual tests that are implemented.

TestNG: This is the test driver and is extremely powerful with good support for annotations and a great community. It has a very good support for data driven testing. All our tests are written are written in groovy and this plays very well with TestNG. We use TestNG ant task to orchestrate our testing and this uses the testng.xml . Most of our tests use the @Factory methods to read the data from an xml file. We use Groovy's neat XML Parser methods which makes this a breeze . This is helpful in the following ways :

• Use the same method in different tests by passing different parameters in the xml files. This can be achieved by using the @Param in the testng.xml file
• It has a great listener support and a great reporting system . We were able to pick up the test description from Redmine and display them in the test results. All tests were also hyperlinked to Redmine
  

Hudson This is the Continous Integration server that we use. We use this in conjunction with a few ant taskts. We use Hudson to check out from Subversion and then use a ant task to set up databases and restart apache. Hudson then invokes another ant task which starts the TestNG tests . Hudson is useful to keep the results and lets you track changes and failures to the tests. One important feature of Hudson that we use is the chained build (This is a hudson plugin) which starts one job on the completion of a previous build. Hudson has very good support for over riding values in the build.properties . This allows us to check in the build.properties but over ride them in the Hudson job for a daily run

Tellurium This is the heart of the whole system and this implements the Test cases that are run by TestNG. This makes functional testing easy to maintain

Experience with Web Testing Frameworks

This series of posts details my experiences with Web Testing Frameworks. These are completely personal observations and your mileage may vary. I will write about using Selenium , Canoo Web Test and Tellurium.

Cutting to the end of the chase : We have chosen to stick with Tellurium since it matches our needs the most . In this post I will detail why we chose Tellurium and some of the important concepts of Tellurium. In the next post we will look at an example of using Tellurium for testing


I am not a QA person and most of my testing is from a Development / Unit Testing / Regression testing point of view. One of the main problems we had with a separate QA team trying test automation was that the QA team generally lagged behind the product cycle. Things change in the product and it was difficult for the QA Team to invest in an automation test . We decided to have the development team own the automation tests . We had a framework to run the automtaion tests daily and this detected changes more frequently.

We were looking for a framework that was

• Easy for the java developers to use , understand and extend
  
• Modular and will typically allow changes to be localised
• Something that can be used with the same java IDE to debug

Other essential attributes were

• Open source with an active community to support
• Integration with a Testrunner framework to enable data driven testing
  

Tellurium fits all of the above. The difficult thing about Tellurium was the difficulty in locating it . Searching in google for Tellurium never yielded for much :-)

The good thing about Tellurium was that it had very good documentation. There are examples with live website (Tellurium's own wiki) which has a sufficient variety and beyond the usual "Hello World" examples. There are complete tests that show integration with Test runners like

• TestNG
• JUnit

I am a major fan of TestNG and the examples helped to jumpstart with Tellurium and TestNG.

A major strength of Tellurium which distinguishes it from Selenium is the fact that it allows for separation of UI from the Test framework. This is very important and helps you write maintainable code over a long run. It allows to bundle a page / set of related pages into a UI module and allows to define widgets which can be reused.

Two of the most important concepts of Tellurium are the Table and List UI widgets which help in handling repetitive content. The Container widget also is handy for maintaining a structure.

That completes an introduction to why I chose Tellurium as a testing framework. In the next post I will write about selectors and look at tests using Tellurium

Unit Testing

There has been lot of writing on why Unit Testing is important as a software developer. Common Sense dictates that you do Unit Testing. But software engineers rarely do. Voltaire was right. The typical progression is something like this :

When you start a feature you diligently start testing . Some assumptions are made. As the feature develops the assumptions change , become invalid. On completing a feature some complicated features get tested but basic features get missed out. If the company that you work for does not make it mandatory for Unit Testing then this completely gets dropped by the wayside until a customer notices it . Or worse it shows up when the Big Boss demonstrates this to a high powered investment team

One of the reasons I feel that Unit Testing does not get done is because of the over confidence of the software developer . Engineers assume that using standard components work . The do not factor for Murphy's law The answer in all cases to the question "Have you tested the feature? " is almost the same
But the manual says that this should work . How can this not work.

Unit Testing is also expensive. It requires a lot of investment of time and effort. And it requires a steep learning curve. It is not a Fill It Shut It style of work. It needs constant tweaking . As a software developer time is premium and it makes complete sense to spend most of the time on a software development rather than unit testing .

JBoss Seam and the 32 bit Vs 64 bit saga

Choosing a Java Virtual Machine to deploy our application proved a tough task . Our development environment is primarily a mix of Linux desktops (Fedora 5, CentOS, Ubuntu ).

We started with JBoss Seam 1.2.GA and initially chose Sun's JDK 1.5 on a 32 bit Operating Systems . This turned out to be a disaster during development as our our already long "Compile Deploy Test" cycle took even longer. We hit the dreaded PermGen Space problem (JBOSS Seam Mailing List)very often. We switched to JRockit 1.5 32 bit version and found that it did not solve the problem completely . The time between PermGen Exceptions increased from every 2nd deploy to every 5th deploy.

On a hunch we moved the Operating System from 32 bit to 64 bit and also moved the JVM from 32 bit to 64 bit . That combination of OS and JVM solved the PermGen Space problem completely. We tried running the 64 bit Sun JVM and found that it still had the Class Loader problem.

Moving to a 64 bit Operating Systems had its own share of problems . The primary one being non availability of Flash plugin for the 64 bit browser on the CentOS. The developers were reluctant to move since that stopped them from wathcing YouTube movies :-) . Also circa 2007 not many libraries were available for 64 bit Linux compared to the 32 bit . I think that situation has improved a lot in the current years particularly with the Ubuntu Operating System. There is a flash plugin which can be installed with apt-get . Also the CentOS versions had a work around which allows the 32 bit flash plugin to be installed on the 64 bit machines

Another issue that we faced (from a System Administration point of view) was that we could not run a 64 bit OS on the laptops. The Dell laptop 6400 that I had ( circa 2006) did not support running 64 bit Operating System and I had to settle running for a 32 bit version on the laptop and a 64 bit version on the desktop. This meant I had to maintain 2 local copies of the repository since otherwise a disconnect from the Internet meant no possible updates

One of the advantages of moving to the 64 bit Operating System meant that we could take advantage of more memory. Take a look at the 3GB memory barrier for a description of why 32 bit OS cannot use the 4GB memory completely.

Payback time

The blogs and the open source community has been incredible treasure trove of knowledge . We have searched and found a lot of answers to our problems . There are some problems for which we found no answers . We groped around some of these problems and did find some solutions and workarounds. We did discuss about contributing back to the community . This blog post is a small way of contributing back to the open source community
Our project uses the JBoss Seam Application framework . I plan to write a series of articles focusing on the pains , joys, advantages and disadvantages of using these technologies .
The major technologies that we use are JSF, Facelets, Rich Faces . Our application is deployed on top of the JBoss AS . I am not an expert in these technologies. These views are my own and the methods described in these posts did work for me . Your mileage may vary. But I will try to help in case you face any problems

Working for Startups in India

Read an article by Kamla Bhat on working for startups in India. This is post that I could really emphatise with . I have lost team members since their future in-laws did not think much about their son inlaws working in a startup. I have also had cases where parents thought that their sons did not stand a chance in the marriage market if they worked in a startup .

Posted using ScribeFire from Windows.

Powered by ScribeFire.