MySQL Fabric Server Farm Testing Overview
By rudrap on Apr 29, 2014
MySQL SystemQA: MySQL Fabric server farm Setup and Testing overview
MySQL Fabric is an open-source solution released by the MySQL Engineering team at Oracle. It is an extensible and easy to use system for managing a MySQL deployment for Sharding and High-availability. High availability is generally used to describe the ability of a system to continue operating and provide resources to the application when a failure occurs in one or more of the following categories in a fault domain: hardware, software, or application. With high availability feature, MySQL fabric manages several servers in different groups using replication topology. At any point of failure it creates servers available to do the regular activities. To test similar functionalities, we have designed a setup in JET framework which will handle multiple servers in multiple machines. What we did? We have started with a fabric setup class in JET framework, which starts few servers, creates groups and shards. Initially we ran this on a single machine to verify the setup is fine. Later we moved to 2-3 physical machines to create up to 100 servers, 20 groups and 15 shards. We encountered with a failure as the servers can not be started in individual machines. This was because of the limitations of the physical machine i.e we can not start more that 15-20 servers in a 8gb ram machine. We create a server farm using 2 physical machines and 18 Oracle VMs. There we were able to create a MySQL Fabric setup using 230 servers, 40 groups and 35 shards. This setup was done in linux environment. We then used one window 7 machine and multiple MySQL versions to run the setup. What we have tested using this setup? We have used the setup to run :
- Feature testing (fabric CLI features)
- Failover testing ( failing a server/VM and ensure the recovery)
- Concurrent testing (running multiple activities using threads)
- Scalability testing
- Crash Testing includes crashing of fabric state-store, multiples Vms, Multiple servers.
- Randomize testing includes running randomly selected activities in multiple tests.
- Blogs about detail of Crash and randomize testing