Saturday Oct 30, 2010

Into the sunset

sunset

After four years working with the MySQL team, under three different companies, it's time for me to pursue a new career.

Tomorrow is my last working day at Oracle. (Working? But it's Sunday! So, ehm, kind of, anyway, you got the point.)

You may look at my personal blog in the coming days to know what I am going to do next.

Right now, I would like to just say thank you to all my colleagues in the MySQL Team, and to all the community people with whom I have shared the wonderful experience of these four years.

And also, thanks to all the ones who have offered me a job. Really. There were many offers. I am sorry I could not take them all and I had to choose one in the end, but thanks, my friends, for believing in me.

See you around!

P.S.: This blog will not be updated anymore.

Monday Jul 13, 2009

The MySQL Librarian is here!

I have had a wish for a few years. I wanted to find a way to put together the valuable information that the community produces about MySQL, a way that would let me easily find the interesting content that I may have missed when on vacation, or when busy with a conference, a company meeting, or a long stream of coding.

That wish started to take shape last year, when I was traveling with Dups during the East Coast tour. I drove, he took notes. He drove, I took more notes. During meals and walking breaks we discussed and refined the idea. When we went back home, a plan was ready. Dups started coding in January.

At first, his changes were completely invisible. He was refactoring the Planet MySQL code to integrate it with the advanced features that had been developed for the main site. After a series of secondary changes, there came the substantial one. The voting system allows people to show their appreciation or dislike of a post. The tagging system makes the blog posts easy to search.

However, that's only blogs. What about presentations, videos, pictures? Welcome to the MySQL Librarian, the coronation of my wish. My congratulations to Dups for coding it under the pressure that a community relations manager is subject to, while traveling across the continent several times, talking at conferences, dealing with communities, paperwork, blogging, his cat, and some more stray duties that fall on his head regularly.

The Librarian is the place where the community creates, collects, improves, and maintain high quality contents. What you find in the Librarian is not the result of a search engine. It's the refined outcome of cherry picking MySQL related articles, videos, presentations, and pictures, done by dedicated users. The MySQL Community. You. Me. Everyone interested.

There is a detailed description of the Librarian in the Dev Zone. No need to duplicate it here.

Please, share my dream. Come make the MySQL community Librarian the best place to find and post valuable information.

Thanks to the early testers for their patience and valuable advice. The MySQL Librarian is good also because of their comments!

Thursday Jun 11, 2009

MySQL has a new release model

In an earlier post, the pursuit of openness, I announced that MySQL is working at a new release model.

There are still a few details to sort out, but the general plan is ready. The new release model has been approved and starts to operate immediately.

The basic principles of this model are:

  • The trunk is always at least in beta quality.
  • A milestone starts in beta quality ( never in alpha) with a merge between the trunk and a stage tree;
  • Milestone releases, with RC quality, released every three to six months.
  • Integration windows between milestones allow the insertion of new features from stage trees
  • GA releases happen every 12 to 18 months;
  • There are not more than two releases in active support.

The practical consequences of this model adoption is that what was planned for the previous development is now canceled. MySQL 6.0 planned features (Falcon, Maria, Online Backup) are not a priority for the time being.

The next stage tree will be Azalea, which will include the 6.0 features that are stable enough to have a chance to be merged with 5.4 (mainly, subquery optimization batched key access, the fix for Bug#989, out parameters in stored procedures, information_schema.parameters, and some more).

The fundamental difference between this version and the previous one is that Azalea is not blocking. In the previous model, nothing could be released until all the features were ready. In this model, if the features in Azalea are not stable by the time of the intended GA for 5.4, we will rollback, and release only what is ready and stable.

This sort of train model, which has been quite successful with other open source projects, is more dynamic, easy to understand, and more open to participation.

The details of the model are explained in a MySQL University session, today, June 11, at 14:00 UTC (16:00 CET), where Tomas Ulin, director of engineering for the MySQL server, will explain the model and answer questions.

Wednesday Jun 10, 2009

webinar on Data Reduction and Smoothing in MySQL

If you have missed Michael McFadden's session at the last MySQL Conference, here's a chance to catch up.

On June 11, at 17:00 UTC Michael McFadden will present at a free webinar, on the subject of Faster Data Reduction and Smoothing for Analysis & Archival in MySQL.

Don't let the "For ISVs" distract you. This session is a collection of very practical and down to earth tips for tasks that can be in the TODO list of any DBA.

In addition to being practical, Michael's advice is justified by rigorous statistical analysis, and the tips he provides have both the benefits of practical testing and a scientific justification.

Did I make you curious enough? I may add that the tips provided in this lesson will privilege SQL instead of external languages coding, but some Python will be there as a bonus for the patient attendees.

Thursday Jun 04, 2009

MYSQL Planet now with tags and search

Planet MySQL

All this started during a long drive from Charlottesville to Washington, back in November 2008, when I and Dups discussed the status of MySQL Community web presence.

We agreed that we needed to enhance the usefulness of the tools for the community, and MySQL Planet was the first candidate for change. Externally, you have noticed very little until now. First, a login, then the voting system, the Buzz, the Italian, Japanese, and Russian aggregators, an improved treatment for group blogs, and finally the Tags and Searching.

Behind the scenes, there is much more. Dups has been refactoring most of the Planet MySQL code, with the goal of integrating it with the main site. The idea is to eventually allow users to search and use the information available in several formats, such as blogs, forums, presentations, articles, events, and to connect them to each other.

With today's release, we finally see a good reason for logging in to Planet MySQL. Now you can edit tags, thus making the blog posts more useful and easily searchable.

Kudos to Dups and Lenz for the results we got so far. However, there is much more in the making. Stay tuned for the next improvements.

Friday May 08, 2009

MySQL Event in Montreal : one more big shot

Dups is organizing an MySQL event in Montreal.

Also Kaj Arnö is coming along, and will show the results of an interesting research about social networking.

Event: Meet and talk to MySQL Gurus

Where: 1800 McGill College Avenue, Suite 800, Montreal

When: 5-7pm, Monday May 18th

Maximum: 25 people, please RSVP by email or blog comment, or, if you are a Facebook user (and who isn't?) you can do it from this event page.

Tuesday Apr 28, 2009

MySQL Community awards 2009

Attending the MySQL Users Conference in 2006, I had one of the best days of my career. At the morning keynote, my name was called, and I found myself on stage, together with Markus Popp, Roland Bouman, and Rasmus Lerdorf, being awarded a Community Member of the year crystal ball. That day is permanently in my mind as a very fond memory.

For this reason, it is a particular pleasure for me to be in a position to suggest the next ones who will hold the community awards. It is a collegial decision, not my own. Each member of the community team submits a few names, we discuss the pros and the cons, and then we settle for the first three names in the list.

This year, the agreement fell on three names, who were included for different reasons.

MYSQL UC 2009 - Marc Delisle Marc Delisle should be familiar to anyone working with MySQL.

As the author of phpMyAdmin, he deserves gratitude and respect from the community.

For the few who are unfamiliar with it, phpMyAdmin is a web based GUI for MySQL, which has helped millions of users to improve their experience with MySQL from beginner to expert.

Probably the only question that everyone in the community will ask is "what? He was never awarded before?". Indeed. This award was long overdue.

Good job, Marc! The MySQL community would not be as large and happy without your contributions.

MYSQL UC 2009 - Ronald Bradford Ronald Bradford has been a community member for long time, with with a genuine passion for MySQl and its ecosystem.

I remember when, in 2006, he fought Sheeri in an auction to get a T-shirt with the developers name. At the end of the day, he had earned himself a mention in MySQL source code, and a solid reputation as a lover of community.

After a brief period as MySQL employee, Ronald has gone back to the trenches, and his contributions to the community have always been constant and of high quality.

Whenever I can, I enjoy working with him. He is a fine analyst and a hard worker.

Well done, Ronald!

MySQL UC2009 - Shlomi Noach

Shlomi Noach is the new blood among the award winners.

His contributions are proof that you don't need to be a superhero to become a good community member. As Sheeri said in her keynote, you just need to do something well. If there is something that you enjoy and can help others, don't ask questions, and do it.

Shlomi's fresh approach in the community arena is a very welcome and necessary addition. The community needs both the extremely advanced articles of Matt Yonkovit and Shlomi's wake-up posts that make some experts scratch their heads and keep them on their toes.

Thanks, Shlomi. Keep being curious! We like it.

The pursuit of openness

When I joined MySQL in 2006, after several profitable years as a consultant, I had a dream. I wanted to improve the product that had contributed to my professional success.

The first thing that I learned when I started the uphill task is that it was far more difficult than expected. MySQL called itself open source, but the development practices were for all practical purposes closed source. At the same time, I found that MySQL, below the surface, is an organization with complex and well oiled engineering practices.

Indeed, opening up the cathedral, as Lenz put it, was a hard nut to crack. We had a closed source revision control system, and our developers loved it so much, that any proposal to change it was met with strong opposition. We discussed technical matters behind the firewall. Our code review process, although technically demanding and very thorough, was kept in our internal mailing lists.

Sakila closed 2

Contributions were scarce, with no good signs for the future. The binary split between community and enterprise servers made bug fixes contributions very unlikely. The impossible roadmap, with contributions going theoretically to version 6.1, in the distant features, practically discouraged any new proposal of code contribution, except for a few brave souls. On top of it, the Contributor License Agreement (CLA) had only recently been set up, yet formed a formidable obstacle for many people.

Sakila closed 1

Faced with such challenges, several people left. Either to new jobs inside the company or to new shores. The challenge is that you can't fix such a problem in one sweep. Changing practices in a company, especially from a position in which you can't take decisions, is really hard.

But not everyone left. The ones who stayed, in spite of the adverse circumstances, share a strong belief in the values of open source, and were determined to succeed, no matter what it takes.

With a little help of my team mates, Colin, Lenz, and Dups, I continued to support Kaj's internal action to spearhead the changes. While my younger colleagues were busy managing the community, Kaj and I were gently wrestling the rest of the company, to steer its direction towards a more open course.

The results arrived. Slowly, but surely.

  • In 2007, many developers opened the IRC gates, and started using freenode.net;
  • June 2008 - The open source Bazaar replaced the previous lock-in proprietary software Bitkeeper, which didn't allowed handling of the source from the outside;
  • In October 2008, we got rid of the CLA, in favor of the more open Sun Contributor Agreement.
Sakila open 1

The major obstacles were still the binary split and the long release cycle. There were much discussion internally, pleading and cajoling, to convince people that we were on the wrong path, and that our business model required a more open approach. Our team was under pressure. We were between the hammer of the community, shouting at us for not being open, and the anvil of the sales and marketing teams, unwilling to accept the assumption that a happy community generates future customers.

We kept going, and we kept pushing, until we got the announcement at the MySQL Conference 2009, stating the end of the binary split. Now the community server binaries will be published as often as the enterprise ones. The rationale of this request is that the strength of MySQl is in its wide community. We claim that the MySQL server is tested by millions of users, and yet we were giving the GA (mature) binaries only to a handful of customers.

This looked like a privilege, but it was in fact, from an engineering standpoint, a disadvantage. As a customer, I would gladly adopt a software that has been installed by a few million people, rather than being the privileged first one to try it in production.

With this announcement, we are back to the position that has made MySQL so popular and trusted. A database with a large user community.

The last barrier to fall will be the roadmap. So far, we have been tied to an impossible roadmap, with a wish list of features that had little chance of being released. The experience of MySQL 5.1, which took 37 months to release, has taught us a hard lesson. The next version, 6.0, has been in alpha for 25 months, with no near end in sight. Now we took the hard decision to replace the previous roadmap, where we piled features over features in an impossible attempt of pleasing future customers, with a milestone model, where we release features as they are mature, without waiting for all the features in the list to be ready. It means that we renounce the goal to release Falcon and Maria and the Online Backup as one single big bundle released at the same time in 6.0, and we put them aside. The MySQL engineers will schedule features for release as soon as they reach beta status, without waiting for the trailing ones.

Furthermore, we have now a process in place to accept contributions and to review them, with active participation from core developers. Yet, we have still work to do, to smooth the rough edges both internally and externally.

Sakila open 2

To our developers, who have not succumbed to the pressure, and have delivered top class software, while accepting our team's (not always) gentle suggestions to more openness, a hearty thank you!

To the many who have believed in MySQL and in our single and team efforts, thank you! To the many, who have suggested improvements, and contributed actively to our changes, thank you! To the ones who have not joined the flame war when unjustified rumors spread across the net, thank you!

To the ones who spread FUD, and twist every tiny piece of news in the shape of a conspiracy, I say wake up. Have you considered the possibility that we actually strive for the best for our community?

To the naysayers, who shout at us and abuse our goodwill from their self appointed righteousness, which often hides a secret business agenda, I say that we have always be working with the community in mind, and we will continue to improve. The path is still uphill, but the community team, the guardians of MySQL ubiquity, is happy with the progress so far, and will continue on this course!

Monday Apr 27, 2009

Presentation slides online at slideshare

I decided to give a try to slideshare. So I uploaded the slides from my most recent talks, and will eventually catch up with the old ones. My slides repository is http://www.slideshare.net/datacharmer.

If you are looking for the slides from MySQL Conference 2009, here are the shortcuts:

Wednesday Apr 01, 2009

Simulating Workload with MySQL Proxy web seminar - April 2, 2009 10:00PT

On April 2nd, at 10:00 PT (13:00 ET, 18:00 UTC, 19:00 CET), there is a free web seminar on Simulating Workload with MySQL Proxy.

I will introduce MySQL Proxy, and Diego Medina, MySQL QA Engineer, will talk about the juicy part.

Participation is free. To enlist, you need to register online and then you can attend the event from the comfort of your home or office.

Tuesday Mar 24, 2009

Follow us on Twitter

If you are a twitter user, you may like to know that there are many people from MySQL who are regular twitters.

For a start, you may want to follow these ones:

People

  1. kajarno Kaj Arnö
  2. datacharmer Giuseppe Maxia
  3. bytebot Colin Charles
  4. LenzGr Lenz Grimmer
  5. dups Dups Wijayawardhana

Non-people

  1. mysqlconf MySQL Users Conference, which is going to be used during the conference to collect feedback for the panel keynotes.
  2. planetmysql Planet MySQL feeds
  3. mysqluk MySQL UK news

To find more people in the ecosystem, you may check the ones followed by the people above, and you will find most of the important players. Happy twittering!

Monday Mar 23, 2009

A taste of MySQL certification on Facebook

Do you plan to take the MySQL Certification?

Or, are you just curious to know how good you are at MySQL ?

If you use Facebook, and chances are that you do, then you should give a try to the MySQL Quiz Application.

Well, it is not like the real certification, but it's fun, and some of the questions are hard enough to give you pause.

The MySQL Quiz Application was created by Dave Stokes, the MySQL certification manager. Thanks, Dave!

Sakila quiz

Wednesday Mar 18, 2009

MySQL tour of California prior to the Users Conference

Mark your calendars!

Just before the Users Conference, 4 MySQL enthusiasts will tour California Universities and user groups.

This is the conclusion of the Dups on rails journey. When the heroic Dups arrives to California, he will have company. Colin Charles will tour Northern California with Dups. Yours truly and Sheeri K. Cabral will take care of the South.

Everyone is invited!

Are you in California and want to attend? You are welcome! I will publish more details as we finalize them.

Do you want to contribute with a guest appearance? Contact us! (datacharmer [at] sun [dot] com, dups [at] sun [dot] com).

Schedule

North California
13-Apr-2009 Univ. of San Francisco
15-Apr-2009 Univ. of California, Berkley
16-Apr-2009 San Jose State University
16-Apr-2009 Stanford University, Palo Alto
17-Apr-2009 Univ. of California, Davis
South California
14-Apr-2009 Cal Poly, San Luis Obispo
15-Apr-2009 UCLA, Los Angeles
15-Apr-2009 Univ. of California, Irvine
16-Apr-2009 USC, Los Angeles
17-Apr-2009 MySQL Meetup Los Angeles

I would like to thank Dups, for starting this epic journey, and all the colleagues and Sun Campus Ambassadors who have contributed to the organization.

How MySQL tests server binaries before a release

What happens when the binary files of a fresh MySQL version is published on the web?

You may have noticed that the date on the release notes is not the same as the date the downloads are available. Sometimes there is a two weeks gap, sometimes more. Many people in the community have asked what is going on in the meantime.

The answer is a lot of hard work. The code is built for all the operating systems supported by MySQL, and tested in each platform (1). During this process, portability problems, test case glitches, and other things not caught in the normal daily build and test are fixed.

This task involves QA engineers, Build engineers, the Maintenance team, with help and cooperation from the Services, Development, and Community teams.

I asked our Build and QA Teams to tell what happens between the date a release is "branched off" from the Bazaar tree and the date it is available in the downloads page. This is the list of what goes on. It's very impressive that the regression test suite, which looks huge and intimidating to the casual user, is just a tiny bit compared to the rest of the torture tests that the server goes through.

BTW, this is just the tip of the iceberg. QA is a continuous process, not just a set of tests at the end. Sun/MySQL uses a continuous build and test process triggered on code check in, on different products and branches, in total 574 build and test runs a day (more about that in a separate article). The release process adds to that some more testing.

Thanks, MySQL engineers!


The following text was provided by Kent Boortz, Senior Production Engineer in the Build Team.

Tests run during the build process

During the build all packages was tested using the regression test suites. These tests are mainly testing SQL features, from basic queries to replication and partitioning, but also the “mysql” command line client, the “mysqldump” tool and other client tools.

The server can be run in different “modes”, SQL standards and protocols. Because of this, the same test suite is run several times with different combinations of these modes and protocols. As there are time limits not all combinations are run, but what is believed to be a fair sample of combinations to catch regressions. The test runs for each package are

  1. The main suite, except replication tests, was run against the bundled debug enabled server
  2. The main suite was run against the server in default protocol mode
  3. The main suite was run against the server using prepared statements protocol mode
  4. The main suite was run against the embedded server library
  5. The 'funcs_1' suite was run against the server using prepared statements protocol mode
  6. The 'stress' suite was run against the server in default protocol mode

There are some suites not run against all packages, or was disabled

  1. The 'jp' suite was run against the server in normal mode, but not against the Windows server (this is a bug the suite was not run, there is no good reason to skip this)
  2. The 'funcs_2' test suite was not run, disabled because of Bug#20447
  3. The 'nist' test was run using both normal and prepared statements protocol, in all builds but the RPM builds

Tests run after the release binaries are built

Some package tests are run after the packages are built, not in direct connection to the build process.

  1. The package names were verified
  2. RPM packages were checked for unwanted dependencies, like the IA-64 RPMs build using the Intel icc compiler, verified that no icc runtime libraries are needed to install
  3. A simple link test was done against the client and embedded libraries, to try catch missing symbols, missing headers or defects in the output of the “mysql_config” script, that defines the usage

Install Test

The binaries are installed and basic smoke test was done on all supported platforms to catch any install related issues. Testing on some of the platforms was done using automated scripts while on the others it was done manually.


System Test Suite

This is a concurrency/longevity test and can be configured to run with with different storage engines, concurrent users and time.

This test suite contains a number of of Insert/Update/Delete and Selects tests using stored procedures and triggers. The test also has an aspect of Integration testing, since it uses Events, Partitions, etc together and covers scenarios for various storage engines like testing MyISAM and Innodb together, e.g. a trigger on the Innodb table writes into a MyISAM table.

We tested the scenarios that contain inserts, updates, deletes and selects using stored procedures and triggers for innodb and myisam tables, separately. Each of these tests were run for a period of 24 hours with 50 concurrent users


Replication

This is also a concurrency/longevity test and executes Insert/Update/Delete scenarios using stored procedures and triggers but with replication. We tested 3 different scenarios one each for replication types RBR, SBR and MBR, Each of these tests were run with 100 concurrent users for 6 each hours with Innodb tables


High Concurrency

This is Concurrency/Longevity/Stress testing and executes an OLTP scenario on one table. The concurrency and the time period can be configured. We ran this test on Linux platform , with 2000 concurrent users over a period of 8 hours using InnoDB table


MySQL_Bench

This is a single user performance benchmark and tests various scenarios, providing one specific angle to performance benchmark. With this test we catch \*performance results / regressions per operation\* (Sometimes "operation" = one SQL query, but often this is block of statements/queries). We ran this test using Prepared statements with MyISAM and InnoDB tables separately, Each test was run 3 times and the average time computed for each operation. The resulting performance numbers were compared with previous versions of MySQL server for identifying regressions..


DBT-2 Benchmark (TPC-C)

This toolkit implements the TPC-C benchmark standard, measuring the performance for OLTP scenario by using the New order transactions per minute (NOTPM). The tests are completely configurable. We ran the tests for CPU bound analysis with 16, 40 and 80 concurrent users with InnoDB tables and then compared the performance with both 5.1.30 and 5.0 version to detect performance regressions on SUSE platform.


Upgrade/Downgrade testing

Executes upgrade/downgrade scenario of the MySQL server, checking that objects can be created/altered/dropped and data can be inserted/updated/deleted/selected between the previous and current versions of MySQL. Objects include Permissions, Tables,Views,Triggers, Stored Procedures, events and partition and a large number of datatypes. Both live and dump/dump scenarios are tested.

We tested minor version upgrades from pervious versions of 5.1 to 5.1.31 and also major version upgrades 5.0.x to 5.1.31


Sysbench

We use the Sysbench tests to measure database server performance (OLTP benchmark) and compare this against the performance of previous mysql server versions.

We ran the OLTP_RO, OLTP_RW and also the atomic queries using InnoDB and MyISAM tables for 4, 16, 64, 128, 256 threads


Large dataset testing

This test suite runs insert/update/delete/select scenarios using indexes against a very large database containing up to 1 billion rows in the tables. We try to uncover areas where Query Analyser or Optimizer perhaps is not using the right approach. Large datasets provide clearer results on if internal query analysis & optimization has been done correctly.


Tests Using Random Query Generator

Additionally, replication tests were run in the 5.1 replication team tree using Random Query Generator tool. These tests were run on myisam tables, with all replication modes, with a "simple" workloads (DML only) and a "complex" workload (DML, DDL, implicit commit, other statements interesting for replication). Few bugs were found and fixed during this process.


How we deal with bugs during the release phase

As we use the "release often" process for enterprise releases, a bug found during the release process might not cause the process to stop to fix that bug, and rebuild it all.

If considered serious enough it will, but in some cases a bug is filed and the correction is targeted for the next or a later "maintenance" release (or what they are named).

Over time we will get better and better at test automation, preventing these kind of bugs to sneak in. But at this time it might, and means a maintenance release could be a regression to an unlucky user hit by that specific bug.

Having said that, it is rare that we find a new bug during the release process that is ignored, but it could happen, as said a side effect of the "release often" release process.

(1) Not all tests are run in all platforms. Some performance tests are only run on the most popular ones, to compare performance across versions.

Tuesday Mar 03, 2009

BoFs welcome at the MySQL Users Conference and Expo 2009

As usual, the MySQL Users Conference and Expo includes Birds of a Feather (BoF) sessions.

For the uninitiated, a BoF is an informal discussion group about a common topic. There is no registration and no fee for a BoF. You can organize one even if you don't plan to attend the main event. If your BoF is accepted and scheduled, you can organize it as you like. You won't get a free pass for a BoF, though.

We provide the rooms. You bring in the ideas. Submit a BoF proposal!

Almost everything is suitable for a BoF. We only ask that it is related to MySQL.

  • Do you have a project that you are developing and it is still in vaporware stage? No chances for the main conference, but perfectly OK for a BoF.
  • A topic that you feel is missing from the main schedule? Make a BoF.
  • Your favorite connector is neglected? A BoF may revamp the community attention.
  • Have you created a killer scheme that solves MySQL replication one-master-many-slaves limitation? Show it in a BoF.
  • Your fork of the Archive engine is the best thing since sliced bread? Boast it in a BoF!

Imagination is the only limit. Get a good idea. Make it known!

About

Giuseppe Maxia

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today