X

Move your VMware and KVM applications to the cloud without making any changes

  • October 31, 2013

How To Build an Effective Web UI Testing Lab

Testing your web application manually can be painful - keeping track of changes, hours of going through all application pages, and coverage is never 100%. To reduce the drain on resources (man hours and computing), shorten the testing time cycles, and enlarge coverage — automation is a must.


We've actually implemented an automated testing lab on AWS using Ravello. Here's a short technical how-to guide for building an effective web UI testing lab.

http://www.youtube.com/watch?v=GgLJr7En6wM

Learn more about automated testing smart labs in the cloud and check it out by starting your free trial.



The most common open source package for automated web UI testing is Webdriver (Selenium2) project, with its “Grid” configuration.
Grid allows you to:

  • Scale by distributing tests on several machines (parallel execution)
  • Manage multiple environments from a central point, making it easy to run the tests against a vast combination of browsers / os.

So, actually, the only things left to do is:

  1. Write code for the tests and the infrastructure.
  2. Install Grid on your compute resources.

Grid is very simple to instruct, basically running a java process, one for your Grid-HUB that is distributing the testing requests to multiple Grid-NODE processes located on your different testing devices endpoints (e.g: Ubuntu-desktop / Win7 / MacOS).

Once your test infrastructure is configured correctly to run parallel tests on multiple environments/browsers, Grid will do the rest…

The problem, as always = resources

Since testing became so easy, hundreds of tests are coded, and without notice the testing suite runtime reaches hours, and we are back to square one. We increased coverage however yet again we face a long testing phase that causes longer testing cycles. So more parallelism is required, therefore more compute resources.

Possible solutions

Provision more VMs/CPUs in your local lab, or even buy more HW if limit reached. This can be a reasonable solution if we have unlimited cash flow, but this solution in neither scalable nor economical.
Terminate the local Grid, and contact one of many SAAS providers for a cloud based Grid. This is an out-of-the-box solution, you just run your existing tests on an external grid located in the provider’s premises. (e.g., SauceLabs)Pros:

 

  • Very easy to integrate, we don’t even have to know what Grid is.
  • Logs and reports are very clear.
  • Can be used for manual testing for environments that we don’t have in-house.
  • Scalable (but not limitless, they have their own limit in addition to your funds limit).

 

Cons:

  • There is no way to control the execution node yourself, in case we need some additional service/application on the running node it might be impossible.
  • Expensive enough - At 1st look, plans look attractive but after a short POC you realize that higher parallelism hits the roof and monthly charges can reach thousands of $.
  • Offers more capabilities than we actually need (such as a browser that we don’t wish to test on).
  • We might end up paying more than we actually use, or on the other hand, misestimate our needs and renegotiate after realizing it’s not enough.
Use cloud service to orchestrate your own Grid in the cloud Pros:

 

  • Unlimited resources. you can start small and grow as much as you need.
  • Pay per use.

 

Cons:

  • You need to build your test VMs in the cloud. If you use specific client software and OS versions that are not supported by your cloud provider, you have a problem.
  • If you are not already a user of a cloud service, moving to the cloud adds yet another learning curve.

 

Or - use Ravello Systems’ customized application for web ui testing, A.K.A: UI-testing-farm

After reviewing all our options we decided to be original - instead of compromising for a solution that is not 100% suitable for our needs we decided to implement the Grid ourselves, in the Cloud, over Ravello.

Pros:

  • Out-of-the-box (Grid implemented once)
  • Scalable (limitless), and super flexible – nodes can click-added to Grid.
  • Fully controlled, we can easily add installation/services to execution nodes
  • Suitable for our needs; we can easily modify it ourselves, and control the uptime, we pay only for our flexible needs.
  • API control allows us to start/stop the grid machine by demand for testing cycles.
  • Because our test targets are located in the cloud, we can (using Ravello) set the Grid application and the testing target on the same cloud GEO (any of them). This potentially increases network performance as well.
  • Easy to create personalized version of an application.
  • Supports running of desktop or client applications, not just web-based applications.

Cons:

  • MacOsx envs are not an option here as well (yet!).

By choosing the Ravello-UI-Farm solution, we have a one-click “on demand” application that contains our entire UI testing grid, using the common and familiar webdriver infrastructure.

Applications can be started/stopped on demand, so there is no need to pay for “sleeping time”. This can be integrated in pre/post testing cycles on the CI system, using Ravello-API or Ravello-Maven-Plugin.

If the Application is too large/small for our requirements - once created from the available Ravello public blueprint, the application can be easily edited - removing NODE vms if we can settle for lower parallelism, or adding NODE VMs from the VM library of we want to expand our testing parallelism. No coding/scripting necessary whatsoever. Just drag and drop.

All resources in the farm application can be accessed and manipulated to fit personal needs as if located in-house. Our tests require specific files and services configured on the executing node to be used in test runtime.

 

We also enjoy GEO locationing. Because our testing target is hosted on EC2, by hosting our testing grid in Amazon (using Ravello) we get much better testing performance than hosting it in our offices (Middle East). If we decide to migrate our testing targets (servers) to different clouds (using Ravello), our testing grid can migrate there as well, easy enough.

Ravello UI-Testing Farm architecture example

The testing application is constructed of:

  • 1 x Linux Ubuntu server vm running selenium webdriver (role=hub) as java process.
  • N x Llinux Ubuntu desktop vms running n x selenium webdriver (role=node) java processes.

Each node has Google-Chrome and Firefox installed, in addition to VNC server to ensure X-oriented connections to the VM.

HUB virtual machine startup configuration:

java -jar /opt/selenium/selenium-server-standalone-2.28.0.jar -role hub -Xmx2048m

*This starts the HUB process, listening for node registrations and tests sessions.

NODE vms startup configuration (triggered only after X was started):

for DISPLAY 0 : java -jar /opt/selenium/selenium-server-standalone-2.28.0.jar -role node -port 6666 -browser browserName=chrome,maxInstances=1 -Dwebdriver.chrome.driver=/opt/selenium/chromedriver -hub http://hub:4444/grid/register

for DISPLAY 1 : java -jar /opt/selenium/selenium-server-standalone-2.28.0.jar -role node -browser browserName=firefox,maxInstances=1 -hub http://hub:4444/grid/register

*This starts the webdriver processes for Chrome and Firefox on different DISPLAY to prevent interference.

Example of a working UI-testing farm we use

One HUB, 10 VMs each hosting 2 webdriver nodes: Firefox and Chrome, capable of running 10 parallel tests for each of the browser types. Meaning, we can run 10 tests on both browsers simultaneously. If our single test average run time is 2 minutes, let’s see the full test suite runtime breakdown:

# of unique tests Actual running tests(x2 browser types) Test suite runtime
10 20 2 minutes
100 200 20 minutes
300 600 60 minutes

 

Grid status( partial picture)

Example of a Grid status implemented in Ravello ui farm application, contains google-chrome and firefox browser instances. Browse to http://<hub-url>:4444/grid/console

Grid Hub Console for your selenium UI testing farm

 

UI testing project example

Here are some typical test project snippets using: java / maven / testng / selenium.

Starting a webdriver instance:

Test class:

Testing suite.xml:

*the parallel=”tests” will run all test cases in parallel : we have 10 nodes so we set it to 10.

*the parallel=”instances” in each test will cause all data provider in class factory to run in parallel, meaning all browsers will run in parallel. (this is just an example of a use case, parallel=”methods”, can be set as well for different data-provider configuration, causing all tests in all classes to run in parallel, regardless to order or browser type)

Using ravello-maven-plugin to start/stop the farm application:

Code snippet example to check hub status & wait for all nodes to start:

Using htmldriver (in this example)

This queries the hub status for number of live nodes until the required number of nodes are started. this should run once as a pre-condition for the test suite (@BeforeSuite).

Try it out by starting your free trial. You can also read more about our solution for testing labs in the cloud .

References

https://code.google.com/p/selenium/wiki/Grid2

http://testng.org/doc/index.html

http://www.amazon.com/Selenium-Testing-Cookbook-Gundecha-Unmesh/dp/1849515743

 

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.