Thursday Jan 08, 2009

[NB65] Backward Compatibility Testing Report

The goal when producing new version of NetBeans Platform is to be backward compatible, unless otherwise stated. This means that if you developed a module/plugin for version A of the platform then it should work even in platform version B that was developed a few years later. If there is a module for which the previous statement is not true, then the compatibility is compromised. Discovering such modules is generally bad sign. It can either mean that there is an unwanted incompatible change, and in such case we shall fix it prior to release, or this change is desirable and highly justified, and in such case it should be properly documented. In either case, backward compatibility is the strongest commitment of the NetBeans platform. Undesired incompatible changes have to be eliminated.

As the testing of the compatibility in an artificial testing environment might be helpful it wouldn't meet our goals. Therefore we(QE) started a community program that should give us more reliable results.

We received reports for 5 different products. You can look at full reports at http://wiki.netbeans.org/NB65BackwardCompatibilityTestingReport. I have to say that we expected more reports then "only" three. On the other hand, these reports are from people and companies that build real product on NetBeans Platform for years therefore we are happy that they haven't found any incompatibilities in NetBeans Platform - platform9 cluster of NetBeans IDE 6.5.

We will start similar compatibility testing for NetBeans IDE 7.0 when it will reach its Beta.

Thursday Nov 13, 2008

Report exceptions in an easy way

Reporting bugs can be quite frustrating and boring process. The process works generally in following way. Go to a bug tracking system that usually has a web interface. Then you have to log in (isn't it painful already?). Then you report your problem/bug/issue. You can also attach a stack traces, thread dumps or screenshots to the bug report. And then somebody evaluates it and closes it as duplicate of another bug that somebody reported earlier. Isn't there an easier way how to report bugs?

Of course, there is a better and easier way. At least in NetBeans IDE(NB). The NB shows you Exception dialog whenever a exception appears in the software. If you will look carefully at the dialog you will notice Review and Report Problem button.



When you click the Review and Report Problem button then can report the exception into our database. You can provide additional comments that can help to reproduce or identify the problem. If you wanna be contacted later or if you are interested what will happen with the report then you can provide your user/password to netbeans.org.



If you are wondering what data are sent from your computer you can look at them. Just press View Data button. You'll see log of uigestures collector.

Your report will be uploaded to our server, entered into database(see the message). It will be evaluated if it is a duplicate of existing report or new report. New reports that fulfill the criteria will be then entered to Issuezilla as new bug reports.



More details about the exception reporter tool:

Tuesday Nov 11, 2008

QE said GO for NetBeans 6.5

That is great news - NetBeans 6.5 has final GO from QE. What it means? NetBeans Quality Engineering team finished the certification testing of last Release Candidate of the NetBeans 6.5. We haven't found any stopper. Therefore we can proceed to releasing it. It will take a time before it will be uploaded, all the pages need to be changed etc. The official release date should be November 20th. Why I said "should be" ? You never know what happens... Crossing finger for NB 6.5 publishing.

Thursday Nov 06, 2008

Where are the P1s coming from?

It seems that it repeats every release. The worst bugs appear a second before the end of the release cycle. I always think "did we forget to test this situation?". the answer is "No". We did our best. Unfortunately, we are not able to cover all the different configurations and usages that the users can create. As written in various books and papers it's mission impossible

We have a P1 in NetBeans 6.5 FCS candidate that looks really serious. The IDE starts to delete users' files on the disc. Real stopper, right?
Fortunately, it isn't easy to get into this situation. When you could face it? You need a project in a version control system (most likely SVN) then you have to "too many open files". It means exceed the limit for open files in your system. The it can happen that the IDE goes mad and deletes some of the files.

Good news is that we have fix and we are testing it. It will take a time... The fix is in Filesystems which is one of the core IDE functionalities. Everybody builds on it.

UPDATE: thank you to all who helped with the final testing of the latest fixes in NetBeans 6.5! We are done. No regressions, no new stoppers.

Wednesday Nov 05, 2008

Gimme test results in html

When you are developing a code in NetBeans repository then you should provide a tests that prove that your code works, of course. When you run your tests then it will generate the test result for you. But these test results are in XML which is not as readable as you'd expect.
There is a way how to convert the XML results to HTML. You can use junitreport task in your ant script. And to make it even more convenient for developers and testers we added new target test-generate-html to the common.xml that is used in all netbeans projects.
You should run this target after your tests. It will provide you more readable results. The html results are also generated after commit validation tests (commit-validation target) in NB build.

Thursday Aug 21, 2008

Patch3 for Netbeans 6.1

We are close to finishing of testing of next patch for the NetBeans 6.1. It seems that the patch doesn't cause any regressions and fixes the bugs that it promised. Unfortunately, some of the bugs are almost unverifiable because they appear only in some unknown conditions. Anyway, we haven't found any regression during the testing that could be caused by the fixes.

The patch should be released on August 27 if there won't be any other issues reported by other teams. If you are not patient then you can try the patch now. On your own risk, of course. Add new update center and set up the URL field to this link. The updates will appear in the Plugin manager or in bubble.

Please, be aware of this bug when you are installing a patch - Bubble can install different set of update than via Plugins.

Friday Aug 08, 2008

Testing Beta of NetBeans 6.5


Beta of NetBeans 6.5 is coming next week. We are quite busy with testing of the builds, verifying the issues etc. As usually most of the users will try this beta build rather then a Release candidate (RC). However the quality of the Betas is a beta quality. All the NetBeans' team tries to keep the quality as high as possible but it is not possible to fix all the issues. However there will be enough time to fix them later.


If you are NetBeans fans then this is time to participate. We need to know about your problems. Then we can solve them. Yes, we are testing it like crazy however I'm senior QA engineer therefore I know that we can not cover/simulate/reproduce all the scenarios that the users can come up.
Please, click "Review and Report Problem" if an exception will appear to you. It is simple and easy. You can post it as a guest.
There will be also a Beta Satisfaction Survey that will appear after 5 days from the 1st start of IDE. Don't hesitate to fill it. In this way you can let us know about the problematic parts of the IDE. All the feedback help us to focus the power of the teams.

Do you wanna know more?

Tuesday Jul 15, 2008

A11Y testing of window that appears right after NetBeans start

We are testing accessibility this week. It means to check all the dialogs/menus/wizards/etc. in IDE that they implement Accessible interface, that the provide AccessibleName and Description. We also check the correct mnemonics, tab traversal. This kind of testing is required by Section 508 - more details.

We use a tool for our testing. It is UI Accessibility Tester. We install it as NB module and then run it with icon from NetBeans IDE. But sometimes you need to start the A11Y tester before the application is started. It's needed in cases when you are testing dialogs that appears right after the startup of NetBeans - e.g. Registration wizard.

Start UI Accessibility Tester before the application
  1. Copy a11y.jar from a11y.nbm/netbeans/modules/ to $JDKHOME/jre/lib/ext directory under your JDK installation
  2. Modify (create one if it doesn't exist) your $JDKHOME/jre/lib/accessibility.properties file to include the following line - assistive_technologies=org.netbeans.a11y.tester.UIAccessibilityTester
  3. Start IDE
More detailed instructions at a1y.netbeans.org.

Monday Jul 14, 2008

Patch 2 for NetBeans 6.1 (really bad story)

There is nothing like bugless software.


And NetBeans is not exception. We in QA do our best to find all the bugs that we can. However the bugs are hidden very well. Sometimes they appear in a special configuration. Sometimes we just miss them (yes, we are humans). The live cycle of the release can be long therefore some users don't want to wait for next release to get their bug fixed. They want them fixed immediately. That was the reason why NetBeans started to produce patches for the releases. That was just a background info for the patches in NetBeans.

We produced patch 2 for NetBeans 6.1. QE is responsible for the testing of the patch. If the fixes in patch doesn't break anything in the NetBeans that the users have been using for while. This time the problems appears at the end of the testing cycle. All it started with problem with rights of updater.jar on Mac OS. Unfortunately, the Mac users couldn't update the jar because of wrong setup of the rights. There is always a workaround. And they(.JARa and Jirka) found one. But then the real fun started. Another problem with user/admin rights. WTF! I don't know if you ever tried to test this scenario but it is enormously boring - install as admin, update, switch user, test it. This kind of scenarios cannot be automated :( My thanks to .JARa who took care about it.
The result was "we cannot provide update for everything". There are un-updatable parts of the IDE. This must be fixed for next release.

We are glad that the patch 2 is out (and we don't have to test it anymore, till the next patch). Enjoy the fixes!
About

Team blog of NetBeans Core and Platform Quality Assurance team

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today