Saturday Apr 05, 2014

MySQl Fabric Testing

MySQL Fabric Testing :

MySQL SysQA team is responsible to test the MySQL Fabric at a system level.We do

1) Feature/Functional/Regression Testing   
2) Complex/Integration/concurrent Testing
3) Scalability testing
4) Performance/Atomic Testing
5) Fabric Recovery/Security Testing

MySQL Fabric Testing Examples:

1) multiple activities performed at same time. e.g.

  • Shard split

  • promote a slave

  • insert records in a shard

  • demote another group

  • remove master from the group

  • add old master to the group

2) promote and demote for 100 times
3) scalability testing for 20 mn and 50 mn records and run activities to ensure fabric does not crash
4)1000 connections at the same time to check if there is crash.
5)comparing performance of conn/python,conn/j fabric support against conn/python,conn/j
6) fabric server using ndbCluster
7) DDLs on a sharded table
8) 100-1000 servers in multiple machines with 50 groups and 45 shards (will change based on topology) to ensure there is no crash
9) 10+ activities in parallel to check if there is any hangs in activites
10) 20+ activites one after another
11) Fabric tests integration with mysqlutilities (failover, mysqlreplicate,etc)
12) Backup of shard table (using multisource replication)
13) 10 tables sharded using a single mapping and do DDLs and DMLs in each shard
14) Fabric test with DDLs and DMLs in backend
15) Minor updown of servers for each master or slave

How We did it?

- Created/updated an Automated and Multi Threaded program to run the Fabric Activities and transactions in parallel.

- Running Hudson jobs to run the tests regularly

Future activities:

- Crash testing and Randomize testing for MySQL Fabric server farm.

Tuesday Dec 31, 2013

Profiling mysqld on windows

Profiling mysqld on windows using visual studio 2012 profiler tool (vsperfcmd)

Recently, when I was doing performance assessment of one  innodb feature 
using sysbench standard tool then I observed performance drop in the feature.

I had some  hands on experience with  codeanalyst (AMD's tool) so I did sampling profiling. 
I attached profiler vsperf to mysqld when sysbench client was doing transaction with some 
concurrent threads and captured the data for 1 minute.
Just for information, there are 2 technique of profiling -1 sampling and instrumentation. 
For instrumentation, I had limitation because a)our  test  framework starts mysqld in instrumentation 
profiler needs to start mysqld  
b) in-between framework does shutdown  for some clean-up of data and 
c) instrumentation profiling requires debug build.

The codeanalyst does not provide very detailed data like 'vsperf' provides. 

Using visual studio IDE, we can profile (sampling or instrumentation). 
In my sampling profiling I measured CPU clocks and it was quite easy to do through IDE. 
It saves session so that we can refer profiled  data and compare with another saved session profiled data. 
The list of events for your hardware, you can find by running:

vsperfcmd /querycounters (note: in my installation vsperfcmd was 
@C:\Program Files (x86)\Microsoft Visual Studio 10.0\Team Tools\Performance Tools\x64)

‘Event based profiling’ like profiling for event 'L2 cachemiss'  using IDE was giving me trouble and I got no data 
for L2CacheMiss event-profiling. 
I could not figure out like why event-profiling through IDE is not able to capture any data.
So, I thought of trying command-line option for this and command-line approach worked out and I got the data I needed.

Here, a sample batch file I wrote to do event (L2CacheMiss) based profiling:

@echo on
cd C:\Program Files (x86)\Microsoft Visual Studio 10.0\Team Tools\Performance Tools\x64
rem VSPerfCmd.exe /attach:%1   /Output:"C:\amrendra\wl6906_analysis\report\baseline_L2LinesOut.vsp" /Start:SAMPLE /COUNTER:L2Misses,1000
rem VSPerfCmd.exe /attach:%1   /Output:"C:\amrendra\wl6906_analysis\report\baseline_L2LinesOut_2.vsp" /Start:SAMPLE /COUNTER:L2LinesOut,100
VSPerfCmd.exe /attach:%1   /Output:"C:\amrendra\wl6658\baseline\baseline_cpu.vsp" /Start:SAMPLE 
VSPerfCmd.exe /GlobalOn
rem ping -n 1 -w 60000 >NUL
rem wscript c:\amrendra\wait.vbs 60
timeout /T 60 /NOBREAK
VSPerfCmd.exe /GlobalOff
VSPerfCmd.exe /detach
VSPerfCmd.exe  /Shutdown
rem do not use vsperfcmd /admin:service,start
rem vsperfcmd /querycounters

Some reference links can be useful, found on msdn:

  • For events detail I referred:

Monday Jun 03, 2013

Testing MEB impact over MySQL Server features

MySQL SystemQA: Testing MEB impact over MySQL Server features:

     MySQL SysQA team is responsible to test the MySQL Enterprise backup at a system level. Along with the System testing , SysQA team worked on MEB impact over MySQL Server features to ensure the new version of MySQl server is compatible with latest release of MEB.

What we do.

- Analyse each Server feature and findout if that impacts MEB.

- Run an Automated test that integrates the server functionalities along with MEB. The test is done in parallel with running server feature.

- Run the feature at each level of MEB run and verify if it works fine.

- Run the feature at different boundary conditions and verify the backup.

How we Did.

- Created a MultiThreaded program to run the MEB and Server feature in parallel.

- Run different transactions along with the server feature and run backup. Restore the backup and - Start a MySQL server with the restored data.

- Verify the data against the original data directory.


MySQL Server Feature: Online alter

- Created different online alter scenarios (with boundary conditions) and prepared a SQL file.

- Included the same in the Program.

- Modify the parameter of the program to trigger the Transactions at certain point of MEB run.(run the transactions when the backup command starts copying the table impacted)

- Check if backup is successful

- Restore the new data directory and start a server.

- Check if the transactions are updated in the table.

Future activities:

High load integration with the automation script to ensure the MEB works with new server features.

Friday May 03, 2013

Welcome to MySQL Testing blog

This blog will discuss the testing aspects of the MySQL products, testing we do for different mysql products,  how we qualify the releases, some of the approaches we use, test tools we use.
[Read More]

This blog will discuss the testing aspects of the MySQL products, testing we do for different mysql products, how we qualify the releases, some of the approaches we use, test tools we use.


« April 2014