Leveraging a disaster recovery site for development - Part 1
By Amir Javanshir on May 09, 2011
As part of the software development lifecycle, the application testing environments are often overlooked and mostly left to the appreciation of each developer, who routinely end up using their own PC or laptop for that. The main advantage here is that it gives developers full control on the testing environment without interfering with other developers or worse, with the production system.
There are however some serious drawbacks with this method:
- Installing and configuring the various layers of software is time consuming and unproductive. Not to mention that developers (rightfully) test a lot by messing around with their environment or data, so the burden of installing and configuring is a repeating one.
- Developers are unable to leverage and test the scalability of their code on the hundreds of threads that modern production servers offer.
- Copying hundreds of gigabytes from the production database to every laptop is not an option, leaving the developers test their code on small, often outdated, data sets.
With the advent of virtualization, using ready-to-boot virtual images of a fully pre-installed and configured application testing environment on an internal cloud has become a very attractive option, removing the drawbacks listed above while still offering full control on the environment to each developer. Few companies can however afford to run a dedicated grid of servers to host these virtual environments. On the other hand, many companies do have a lot of dormant processing and storage power: the Disaster Recovery (DR) site whose purpose is to ensure operational continuity in case of a disaster on the primary production site, and often equipped with equally performing servers than the primary site.
Could one think of leveraging the DR site as a testing environment? There would be 3 benefits to it:
- It offers a testing cloud at no cost, no extra hardware to purchase or deploy.
- It gives access to the latest production data for testing, thus capturing latest trends in using the application.
- It makes the DR site itself more secure.
On that last point, it is indeed not uncommon to find out too late (i.e. when you fail over) that a DR site --same for backup tapes, stand-by replacement systems, etc by the way-- is not as operational as one needs it; think of faulty memory e.g. Contrarian to the common idea that recovery / backup systems must be sanctuarized and left idle, the best way to be sure that these systems are fully operational is to have them working / worked on, on a regular basis.
Of course, whatever you do on the DR site should not interfere with the primary objective of a DR site: it must continuously run the backup for the production site and, at any point of time, it must be ready to take over operations. This vital requirement calls for a strong separation between the stand-by production and development test environments at the DR site.In a following post, we will describe in details how we used the Oracle Solaris Zones and ZFS technologies to setup such an architecture, at no extra cost --these features come bundled with Solaris 10. An architecture that we designed for one of our partners, a major ISV in the Healthcare sector, eager to set up a quick and flexible framework to create pre-configured development / test environments for developers.