Testing your web application manually can be painful - keeping track of changes, hours of going through all application pages, and coverage is never 100%. To reduce the drain on resources (man hours and computing), shorten the testing time cycles, and enlarge coverage — automation is a must.
We've actually implemented an automated testing lab on AWS using Ravello. Here's a short technical how-to guide for building an effective web UI testing lab.
The most common open source package for automated web UI testing is Webdriver (Selenium2) project, with its “Grid” configuration.
“Grid allows you to:
So, actually, the only things left to do is:
Grid is very simple to instruct, basically running a java process, one for your Grid-HUB that is distributing the testing requests to multiple Grid-NODE processes located on your different testing devices endpoints (e.g: Ubuntu-desktop / Win7 / MacOS).
Once your test infrastructure is configured correctly to run parallel tests on multiple environments/browsers, Grid will do the rest…
Since testing became so easy, hundreds of tests are coded, and without notice the testing suite runtime reaches hours, and we are back to square one. We increased coverage however yet again we face a long testing phase that causes longer testing cycles. So more parallelism is required, therefore more compute resources.
|Provision more VMs/CPUs in your local lab, or even buy more HW if limit reached.||This can be a reasonable solution if we have unlimited cash flow, but this solution in neither scalable nor economical.|
|Terminate the local Grid, and contact one of many SAAS providers for a cloud based Grid.||This is an out-of-the-box solution, you just run your existing tests on an external grid located in the provider’s premises. (e.g., SauceLabs)Pros:
|Use cloud service to orchestrate your own Grid in the cloud||Pros:
After reviewing all our options we decided to be original - instead of compromising for a solution that is not 100% suitable for our needs we decided to implement the Grid ourselves, in the Cloud, over Ravello.
By choosing the Ravello-UI-Farm solution, we have a one-click “on demand” application that contains our entire UI testing grid, using the common and familiar webdriver infrastructure.
Applications can be started/stopped on demand, so there is no need to pay for “sleeping time”. This can be integrated in pre/post testing cycles on the CI system, using Ravello-API or Ravello-Maven-Plugin.
If the Application is too large/small for our requirements - once created from the available Ravello public blueprint, the application can be easily edited - removing NODE vms if we can settle for lower parallelism, or adding NODE VMs from the VM library of we want to expand our testing parallelism. No coding/scripting necessary whatsoever. Just drag and drop.
All resources in the farm application can be accessed and manipulated to fit personal needs as if located in-house. Our tests require specific files and services configured on the executing node to be used in test runtime.
We also enjoy GEO locationing. Because our testing target is hosted on EC2, by hosting our testing grid in Amazon (using Ravello) we get much better testing performance than hosting it in our offices (Middle East). If we decide to migrate our testing targets (servers) to different clouds (using Ravello), our testing grid can migrate there as well, easy enough.
Ravello UI-Testing Farm architecture example
The testing application is constructed of:
Each node has Google-Chrome and Firefox installed, in addition to VNC server to ensure X-oriented connections to the VM.
HUB virtual machine startup configuration:
java -jar /opt/selenium/selenium-server-standalone-2.28.0.jar -role hub -Xmx2048m
*This starts the HUB process, listening for node registrations and tests sessions.
NODE vms startup configuration (triggered only after X was started):
for DISPLAY 0 : java -jar /opt/selenium/selenium-server-standalone-2.28.0.jar -role node -port 6666 -browser browserName=chrome,maxInstances=1 -Dwebdriver.chrome.driver=/opt/selenium/chromedriver -hub http://hub:4444/grid/register
for DISPLAY 1 : java -jar /opt/selenium/selenium-server-standalone-2.28.0.jar -role node -browser browserName=firefox,maxInstances=1 -hub http://hub:4444/grid/register
*This starts the webdriver processes for Chrome and Firefox on different DISPLAY to prevent interference.
One HUB, 10 VMs each hosting 2 webdriver nodes: Firefox and Chrome, capable of running 10 parallel tests for each of the browser types. Meaning, we can run 10 tests on both browsers simultaneously. If our single test average run time is 2 minutes, let’s see the full test suite runtime breakdown:
|# of unique tests||Actual running tests(x2 browser types)||Test suite runtime|
Example of a Grid status implemented in Ravello ui farm application, contains google-chrome and firefox browser instances. Browse to http://<hub-url>:4444/grid/console
Here are some typical test project snippets using: java / maven / testng / selenium.
Starting a webdriver instance:
*the parallel=”tests” will run all test cases in parallel : we have 10 nodes so we set it to 10.
*the parallel=”instances” in each test will cause all data provider in class factory to run in parallel, meaning all browsers will run in parallel. (this is just an example of a use case, parallel=”methods”, can be set as well for different data-provider configuration, causing all tests in all classes to run in parallel, regardless to order or browser type)
Using ravello-maven-plugin to start/stop the farm application:
Code snippet example to check hub status & wait for all nodes to start:
Using htmldriver (in this example)
This queries the hub status for number of live nodes until the required number of nodes are started. this should run once as a pre-condition for the test suite (@BeforeSuite).