Wednesday Dec 07, 2005


It wasn't a word 5 minutes ago, but it is now. I spent all night practicing Buildery, or the act of building vast numbers of Solaris packages. Just as soon as Subversion 1.3 is released, I will unleash a raft of new packages via Blastwave. This will include Apache 2.2.0, Subversion 1.3.0, neon 0.25.4, and SVK 1.05. Most of the long night was going through the Blastwave build system and ensuring that configurations and package admin files for all of the 32 Perl modules that SVK depends on are up to date. Actually, SVK has 49 dependencies, if you count VCP and it's 16 dependencies, but I stopped counting VCP because I can't get it to work properly. VCP is an interesting module. It is used by SVK to allow the same command set to operate on Subversion, CVS, or Perforce repositories. I'm not certain how many people would actually use such a feature, but looking at the code, it appears that a great deal of effort has gone into it.

This work is part of an effort to get the build system back into shape after a long hiatus. In the beginning, I downloaded a version of GARNOME, one of the (finest) build systems for cutting edge GNOME sources. At the time, I was just trying to install a GNOME environment on my Solaris 8 desktop, as I can't really stand using CDE for any length of time. Also, as I was building GNOME, I was building a set of GNU utilities which I used for development and production support in Sun's Newark manufacturing facility. Each distribution was compiled and packaged manually, and was taking a long time, as the number of useful utilities I wanted to provide was increasing.

Eventually, I decided that it would be a great idea to put these package builds into a GARNOME style build sytem. GARNOME itself is based on the GAR ports system, which was developed by the Linux Bootable Business Card (Linux-BBC) project. GAR makes building software extremely easy -- just set a fiew fields in a template Makefile, and away you go. Over time, each time I received a request to add more Perl modules to our servers, or identified a new tool which was required for some purpose, I'd stick it in my GAR system. I then developed a GAR extension (GAR supports additional extensions as included make files) which would create Solaris packages. This system grew to approximately 300 software distributions, bundled into four packages (most of the 300 were CPAN modules).

In August 2004, I joined the Blastwave project. I had decided at that point, that it made no sense for me to maintain 100% of the packages that I used, when someone else was already doing it for me. I would contribute the packages that I build that nobody else had picked up, and reduce my workload. From day one, I used the GAR build system which I adapted for internal use. However, the Solaris packaging system, at this time based on make rules only -- phear, was error prone, and not as flexible as I'd like. For one thing, it could really only build a single package out of each build directory, and I needed it to build any number of packages from a single distribution install. In response, I split out the packaging logic into a utility called mkpackage.

The mkpackage utility reads a spec file, and performs some actions as a result. The idea is roughly like RPM, only it is far simpler, because, for better or worse, SVR4 packages are much more limited than RPM. It would solve so many problems if SVR4 packages adopted a provides/requires system, or at least supported minimum/maximum package version numbers rather than exact match. Maybe this functionality will appear once the pkg tools are eventually released to OpenSolaris. Anyway, the spec file is very simple. Here is the spec for apache2c, the core Apache 2.2.0 package:

%var            bitname apache2c
%var            pkgname CSWapache2c
%include        url file://%{GARDIR}/pkglib/csw_dyndepend.gspec
%var            desc Apache 2.2 web server (core)
%copyright      url file://%{WORKSRC}/LICENSE

Simple enough. The %var directive sets a variable for use later in the same file, or any included files. In the above case the bitname variable is used to construct the package filename. The %include directive does exactly what you think it does -- include another spec file. This directive takes two arguments: the first is a method, which in this case is url, indicating that the final argument is a URI which mkpackage must fetch. Any URI which is supported by LWP::UserAgent can be used here, including http://, ftp://, https://, and so on. The other supported method is exec, which will run some command and include the result.

The content of any %include is evaluated immediately. In the above case, the include resolves to this spec:

%include        url file://%{GARDIR}/pkglib/csw_vars.gspec
%include        url file://%{GARDIR}/pkglib/csw_prototype.gspec
%pkginfo        url file://%{GARDIR}/pkglib/csw/pkginfo
%depend         url file://%{GARDIR}/pkglib/csw/depend
%include        url file://%{GARDIR}/pkglib/std_depend.gspec

This is mostly includes, but shows the pkginfo and standard depend file for CSW (Community SoftWare -- Blastwave). There are several directives like %pkginfo which correspond to the different package admin files that SVR4 packages support, including %prototype, %preinstall, %postinstall, %space, and so on. These are treated a bit differently than an include. They create a file in the mkpackage work directory (configurable) called %{pkgname}.<admfile>, e.g. CSWapache2c.pkginfo. As the source file is written into the work directory, any %{variables} in that file are replaced with variables from the environment, those fabricated by mkpackage, and any set in prior %var statements. For instance, the standard CSW pkginfo file is:


The MKP_ variables are supplied by mkpackage, and SPKG_ variables are exported to mkpackage by GAR. Everything is replaced when the pkginfo file is written, and any variables that cannot be replaced will cause mkpackage to complain.

As I mentioned earlier, there is a way to execute any arbitrary command and include the result. It is also possible to execute an arbitrary command to create one of the standard files. This is most often used to create depend and prototype files. Default outputs from pkgproto may be fine for most cases, but Blastwave has a set of standards for prototypes that need to be followed. That means that certain ownership needs to be assigned to files, and certain directories and filetypes need to be excluded. For this, I created a simple program called cswproto which takes arguments similar to pkgproto, and spits out a CSW-compliant prototype file. This utility is called as follows:

%prototype      exec cswproto -s %{TIMESTAMP} -v basedir=%{DESTDIR} %{DESTDIR}=/

This finds any files in the destination directory %{DESTDIR} which are newer than the timestamp file %{TIMESTAMP} (created and updated by an implicit make rule in GAR). As I don't build as root, I install software off to a temporary directory (usually /tmp/a). This means that the resultant prototype from cswproto always expects to pull the real file from /tmp/a/some/real/file. However, in the GAR system, it is possible (and useful in a number of circumstances) to change the DESTDIR at will. This makes life difficult for prototype files with a fixed DESTDIR path. The -v switch to cswproto replaces instances of %{DESTDIR} in each line of the prototype with the package build time variable $basedir. Before the transformation, a sample prototype line might be:

f none /opt/csw/lib/ 0755 root bin

After the transformation, the same line would be:

f none /opt/csw/lib/$basedir/opt/csw/lib/ 0755 root bin

The basedir variable is then supplied to pkgmk, so that it can find all files in the prototype, regardless of the current DESTDIR setting.

This system has proven to be very flexible -- I can now build any number of packages I require from a single software installation. For example, after building and installing PHP 5, I can now build 21 separate packages out of the installed files, separating out less commonly used modules that might have additional dependencies. This benefits users directly because less dependencies means less software to download for common tasks, and less disk space occupied by bits that will never be used.

Wednesday Dec 08, 2004

Variety is the Spice of Blastwave

With all of the talk recently regarding the Blastwave project, I polled some of the maintainers to get an idea of the reasons these people give of their free time (and in many cases, of their pocketbook), to support open source packages for Solaris. The response was great, and therefore I've kicked off an effort to document some of these reasons. Articles are in no particular order, beyond what I feel like writing about in a given day :). And now, onward.

Life without spice, like an OS without a color capable ls command, is bland. In the past, the variety of software available for Solaris earned it a reputation for being a bit bland. Solid as a rock, but at the end of the day, rocks alone don't get the job done. Since the very early days of Solaris, system administrators, engineers and other technically inclined Solaris users decided to take the entrepreneurial approach. They began to compile and package for Solaris the vast number of software projects in the wild which typically only target Linux. These projects made Solaris more enticing to users and admins that in the past were spoiled by the riches of open source projects readily available for Linux.

Blastwave continues in this proud tradition of from-the-people-for-the-people, providing more functionality for Solaris, improving it's usablility, and at the same time, it's reputation. There is a stunning array of software that Blastwave makes available in Solaris standard SysV packages. At the time of this writing, the most recent release of great note is Gnome 2.8. Installing or upgrading Gnome on a Blastwave system is a snap -- just run pkg-get -i gnome, and all of the required packages, including dependencies, are installed for you.

If you're not into Gnome, you can always grab the K Desktop Environment (KDE) instead. It is packaged with end users in mind, including dtlogin integration, so after installation, it's as easy as logging out of your current environment and logging back in in KDE! And the fun doesn't stop with the desktop (although it could, if you're happy at that point). If you are an administrator, the variety of software available, including Apache (both 1.3.x and 2.0.x), MySQL 4.0.x, Subversion 1.1.1, and many, many more. In fact, at the time of this writing, Blastwave has 932 high quality open source software packages available -- just install and go!

Check out Blastwave today, and see if the software you need is available. If it isn't available, check the request form to see if someone has already requested it, and if not, add it to the list (or better yet, build a package and sign on as a maintainer).

As always, if you or your company are powered by Blastwave, please consider donating to the project, purchasing a single DVD, or even a DVD subscription -- every little bit helps to ensure the future of high-quality open source software packages for the Solaris Operating Environment.

Monday Nov 29, 2004

S.O.S -- Save Blastwave!

For the last several months, I've been involved with the Blastwave project. This project produces high-performance, high-quality Solaris packaged open source software. As many packages as possible are compiled using the Sun One compiler suite, and many packages include both 32- and 64-bit libraries and binaries. The packages aim to provide the best out-of-box experience to the end user, with little or no post-installation configuration required. Some of you may have heard of this project; some may be using these packages right now. It's the ideal way to get open source software like Gnome 2.6.2 and KDE 3.3.1 up and running on everything from Solaris 8 through to 10, on both SPARC and x86 processors (with AMD64 support on the way). In fact, there are currently 878 packages available from Blastwave. Packages are available via a package management utility, pkg-get, which is much like the Debian Linux apt-get utility. This makes it easy to update existing packages, or find new packages to install. A much better alternative is to receive all of the packages on a single DVD! The single DVD is available for USD$20, with a monthly subscription service available for 20% of the single DVD cost.

So, now that I'm done with the sales pitch -- what's all this about an S.O.S?!? Well, this project has so much momentum at the moment on the community side, but next to no momentum on the financial side. The project's founder, Dennis Clarke, has footed the bill up to this point -- that includes server space for the build farm, bandwidth for synchronizing mirror sites, bandwidth for the web page, and up-front costs for production and shipping of the DVDs (not to mention the labor). At this point, the project accounts are dry, there are a scant 30 DVD subscriptions to date, and there have been no offers of corporate support or sponsorship! Some of the maintainers have managed to scrounge up servers to support update and maintenance of the build farm and supporting servers, but that's just part of the solution. For more background info, please read this thread on comp.unix.solaris started by Blastwave maintainer Mark Round.

So my plea is this -- if you use Blastwave, please buy a DVD, or better yet get a subscription. Save yourself the time spent downloading! If you are part of a company (or own a company) that is powered by Blastwave, please consider lending the project a hand through corporate sponsorship. The project has come so far, and achieved so much that it would be a disaster to lose it all. Please support Blastwave and Solaris Open Source software!

Thursday Sep 30, 2004

Learn mdb in 30 minutes

As part of my involvement with the Blastwave community software project, I maintain SPARC and x86 versions of the Subversion packages for Solaris. When I became the maintainer for this software, I inherited with it a strange bug, which now is causing me problems. I currently use CVS for my day to day revision control needs, but I'd like to switch to SVN if only for the cool factor, but also to evaluate how it could eventually be used to replace the ClearCase repository my organization uses.

The issue is with the Apache 2 module mod_dav_svn, which enables SVN repository access via the WebDAV (distributed authoring and versioning) protocol. This is a great feature, as then it is possible to take advantage of modules like mod_auth_ldap to manage access control, or mod_perl to provide even more sophisticated functionality. The bug I was tracing occurs any time a file is committed to a SVN repository via DAV. The commit will fail, and the Apache error logs will contain a line or two like this:

[Thu Sep 30 16:00:10 2004] [notice] child pid 18115 exit signal Illegal instruction (4)

The Apache worker thread dies, and is replaced by another thread, waiting for a new request. (It seems to me that this should probably have a higher severity than notice) On the client side, the svn client outputs this useful message:

svn: Commit failed (details follow):
svn: MKACTIVITY of /svn/!svn/act/7cba529c-56e5-0310-9002-f184ad8e84d2:
Could not read status line: connection was closed by server. (http://server:5957)

So how was I going to track down what was happening with mod_dav_svn if it was being launched by Apache in response to an incoming commit request from my subversion client? I was going to learn how to use mdb, that's how. The first thing I did was learn how to launch a program in the debugger. It's easy: mdb /path/to/program. Apache requires a set of arguments, specifically the -X argument, which forces it to stay in the foreground and such. Finding out how to set arguments was a bit more difficult.

I'm somewhat familiar with the GNU debugger gdb, but this was my first time into mdb or any sort of Solaris debugging of this type (I've done very very low level hardware, firmware and software debugging on the Sun Fire mid-frame server line, but never applications). Using the mdb dcmd ::dcmds lists out the available commands. ::help <dcmd> for any dcmd will give the usage for that dcmd. I picked out the ::run dcmd, which is how arguments are passed into the target executable, and how execution is restarted from the top.

Now that I know this much, I can see what is happening to this module. Once the server is started with the ::run dcmd, I tried to commit a small change to the sample svn repository on my server. As soon as I connect, mdb dumps out with a SIGILL (Illegal Instruction). Using the ::stack dcmd prints out the stack trace:

bash$ mdb -uM /opt/csw/apache2/sbin/httpd 
Loading modules: [ ]
> ::run -X -f /opt/csw/apache2/etc/svn.conf
mdb: stop on SIGILL
mdb: target stopped at:
0x169f08:       unimp     0
mdb: You've got symbols!
Loading modules: [ ]
> ::stack
0x169f08(fca7bbec, 178010, 1, fff, 176340, ff36d518)`dav_svn_get_txn+0x44(177f90, 177eba, 0, 243f0, fedccdd4, fe62949c)`dav_svn_prep_activity+0xc(177ee0, 177eba, fe62aaac, fe60a5e4, 177ee0, 0)`dav_svn_get_resource+0x5a0(176378, 2f, 0, 0, fca7bda4, 177ee0)
0xfe705564(176378, 800, 0, fca7bda4, 27500, 0)
0xfe70f08c(176378, 1, fe62ac04, 0, 800, fe72c9fc)
ap_invoke_handler+0x178(176378, fe710298, e313c, e2f94, 0, 0)
ap_process_request+0x30(176378, 4, 176378, 0, fe4614a0, 0)
0x2aa60(152ca0, 0, 152cf8, 8ac00, 8ac00, 1000)
ap_process_connection+0xc8(152ca0, 2aa04, e32a8, 0, 152c98, 8d9f0)
0x34790(f81f0, 152bb0, 18, 152ca0, 0, 18)`_lwp_start(0, 0, 0, 0, 0, 0)

From the stack trace, we can see that the issue is occuring in some unnamed function called by the mod_dav_svn function dav_svn_get_txn. Setting a breakpoint on this function can be done using the ::bp dcmd using the symbol name. Now we can try the commit again with a new breakpoint and see where it gets us:

> ::bp dav_svn_get_txn
> ::run -X -f /opt/csw/apache2/etc/svn.conf
mdb: stop at dav_svn_get_txn
mdb: target stopped at:`dav_svn_get_txn: save      %sp, -0x90, %sp
mdb: You've got symbols!

Excellent. Now we have control over mod_dav_svn before it hits the illegal instruction. Mdb shows in the output that we have just saved off the stack pointer after dav_svn_get_txn is called by dav_svn_prep_activity. Looking in the actual source code for dav_svn_get_txn shows that the first steps in this function are to initialize and open a Berkeley DB (BDB) database, which is used by this installation of subversion to manage transactions in the repository. Mdb will now allow us to single step through the code using the short dcmd :s which steps one instruction at a time. If single stepping gets into a C library function or other function that is not critical to the current debugging task, specify :s out to step out of the function:

> :s
mdb: target stopped at:`svn_path_join+0x28:      call      +0x26d9c    <`strlen>
> :s
mdb: target stopped at:`svn_path_join+0x2c:      nop
> :s out
mdb: stop at`dav_svn_get_txn+0x34
mdb: target stopped at:`dav_svn_get_txn+0x34:    mov       1, %o2

This shows mdb stepping out of the strlen() function called from dav_svn_get_txn. Quite a bit of single stepping is needed to get to the actual error, as we call into the Apache support libraries eventually. I knew from some previous research that others had found the issue to be inside of the aprutil library, so I knew not to step out of any apr_ functions. One of the biggest pains I found with mdb was single stepping. What I really wanted to do was to break on a function and single step through everything within libaprutil (but only libaprutil), printing each disassembled line. I couldn't find out how to do this in 30 minutes using the Solaris Modular Debugger Guide, and I also couldn't find how to repeat the command I just sent to mdb. What I really needed, and continue to need, as I still haven't found the answer, is a command like gdb '.', which repeats the last command verbatim. With this, I would be able to type :s, then use a single character to step through lines, rather than two (50% efficiency improvement :). Eventually, after carpal tunnel started to set in, I hit the SIGILL:

> :s
mdb: target stopped at:
0xff3515c0:     ld        [%o0 + 0xf0], %l3
> :s
mdb: target stopped at:
0xff3515c4:     jmpl      %l3, %o7
> :s
mdb: target stopped at:
0xff3515c8:     clr       %o1
> :s
mdb: target stopped at:
0x169fc8:       unimp     0
> ::dis -w
0x169fa0:                       unimp     0
0x169fa4:                       unimp     0
0x169fa8:                       unimp     0
0x169fac:                       unimp     0
0x169fb0:                       cb13      -0x440000   <0xffd29fb0>
0x169fb4:                       unimp     0
0x169fb8:                       unimp     0
0x169fbc:                       unimp     0
0x169fc0:                       unimp     0x69
0x169fc4:                       unimp     0
0x169fc8:                       unimp     0      <--- culprit
0x169fcc:                       unimp     0
0x169fd0:                       unimp     0
0x169fd4:                       unimp     2
0x169fd8:                       swapa     [%g4 + -0x18] %asi, %i7
0x169fdc:                       swapa     [%g5 + %l0] 03, %i7
0x169fe0:                       unimp     0x20
0x169fe4:                       unimp     0xa
0x169fe8:                       unimp     0
0x169fec:                       unimp     0
0x169ff0:                       unimp     0

Using the ::dis dcmd disassembles instructions near the current code pointer. The -w argument shows a window on either side of the current line. One more step and the process would throw a SIGILL and the apache request will die. A little more break point experimentation resulted in this call sequence:`dav_svn_get_txn`apr_dbm_open`apr_posix_perms2mode

It looks like Forte generated a bit of bad code in libaprutil, which could be due to the fact that it is compiled with a high optimization level. This supported the claims of some users on the Subversion user mailing list who indicated that the issues were within libaprutil. The user had recompiled APR, Apache 2 and Subversion with GCC 3.4, and the SIGILL problem went away. The next step for me is to work with the apr and aprutil maintainer to produce some unoptimized versions of the APR libraries.

Tuesday Sep 14, 2004

Sun Open Source Summit

I attended (via webcast + IRC) the Sun Open Source Summit which was organized by the Sun Software CTO, and held at the Santa Clara Auditorium. It's a shame I couldn't be there in person, but c'est la vie (translation: it's not my day job, but I wish it were). I managed to attend all of the afternoon Auditorium tracks, including the Lightning Talks (which were amusing in their brevity). Talks were given on projects such as Rome, an open Atom/RSS toolkit for Java, and of course Tonic, which is upcoming Open Source Solaris. Of particular interest was hearing Joerg Shilling, author of cdrtools and star among others. Joerg is also involved with the Blastwave Community Software project, which I recently joined. Blastwave provides high quality, low touch Solaris packages for a wide variety of Open Source software</plug>.

The Auditorium also hosted several main sessions this afternoon which I was able to attend:

  • Open Source Business Models

    “Charging money for access to source code is a tried and true way to make lots of money. But thanks to projects like Apache and Linux, more and more customers (including Sun customers) are demanding code access and a marked procurement preference for F/OSS code. Why is this happening and how will proprietary code producers make the switch to F/OSS while still maintaining profits?”

  • Community vs. Control

    “In many ways creating very high-quality software is all about controlling the engineering environment, certainly what gets checked in and how changes are vetted. Sun's engineering process was designed to deliver proven results to our customers. Yet increasingly there is a competing priority of attracting and engaging communities of outside developers who may not have patience and discipline for the established process. We know from some of our previous projects with the F/OSS community that Sun Control is seen as a bad thing. We are told that giving up control will reap rewards, but what will it do to projects like Solaris?”

  • Sun Lessons

    “This [...] session is a chance to share lessons learned during Sun's many experiments hosting or participating in F/OSS communities. Which assumptions were completely validated and which were so far off base that we had to regroup? How did we solve the challenges that came up? Which practices worked the best? What were the unintended outcomes (both positive and negative)?”

  • Sun and Tonic

    “Tonic is the codename for Sun's OpenSolaris project. Another name we considered for this talk was "The Truth about Tonic". Moderated by Claire Giordano from Solaris. Come listen to DE Andy Tucker, engineering manager Karyn Ritter, community manager Jim Grisanzio, and senior staff engineer Bart Smaalders give an overview of the program, talk about why we are open sourcing Solaris, provide a status update, and answer your questions.”

All of the sessions were very well attended, and discussions usually ended with the F/OSS community members encouraging Sun to embrace open source, with full support of the community. There were so many informed, well articulated discussions, that I could not begin to summarize everything that happened during the afternoon sessions. I'll leave this to the real journalists -- primarily James Turner from LinuxWorld Magazine. This is a great time to be involved with Open Source, and a fantastic opportunity for Sun to win back hearts and minds in the broader community. I'm personally exited about the possible interaction between the CSW project and Open Solaris, so watch this space.

Thursday Aug 19, 2004

March of the Browser

I can hardly believe how far browser technology has advanced over the time I've been connected to the net. From the early GUI days of Mosaic, through to today, there's been a tremendous amount of progress. Installing a new update for Firefox today made me think back to all of the different browsers that Sun has standardized on internally since I started work here four and a half years ago.

When I first started, the standard was Hot Java, believe it or not. It wouldn't render most of the sites I wanted to visit, was dog slow, and had next to no customization possibilities. Shortly after using Hot Java for the first time, I moved to Netscape 4.x, and internal software standards changed to use this version of Netscape as well. Once nearly everyone had transitioned to NS4, NS6 was released. The big march includes many points of interest, including NS6, Mozilla 1.2, 1.3, 1.4beta, Mozillas with Gnome integration, and those without, NS7.1, and finally Firefox. Firefox is the future. Love it.

The best feature of Firefox is the ability to write extensions for the browser, which can be downloaded, installed and managed directly by the browser. No need to muck about with plugin directories, etc. Also, extensions use the same language (XUL) that the browser uses for it's UI. Extensions written by 3rd parties integrate with the browser seamlessly, like they were part of the base package. The first thing I do when I install Firefox is add in a good set of extensions.

Top 10 Firefox Extensions

  1. AdBlock - Best... extension... ever. Allows you to wildcard sites which serve ads, blocking the graphics, but wait, there's more. It will even collapse and remove the HTML elements in which the ads were embedded -- no more ugly whitespace where that half page banner ad used to be!
  2. BugMeNot - This was in stiff competition with AdBlock for #1 on my list. Ever want to read a New York Times or Washington Post story, but didn't want the hassle of submitting your 'free registration'? BugMeNot generates random form data for you, and submits the form for you.
  3. All-in-One Gestures - A very comprehensive mouse gestures extension. Allows you to assign complex gestures to a variety of tasks, such as basic browser navigation, bookmarking, tab management and more.
  4. SwitchProxy Tool - Configure more than one web proxy, including anonymous redirectors, and switch between them using a toolbar interface. For Sun users, this is handy, as you can set up a Sun internal proxy on your home browser, and switch to it when connected via the VPN.
  5. Web Developer - If you do any work authoring web pages or dynamic content, install this extension. It adds a toolbar, context menu, and sidebar with a myriad of web features, such as viewing and editing page CSS, converting form GETs to POSTs (and vice versa), and showing content as if it were displayed on a different size screen (e.g. parent screen is 1280x1024, but you can browse at 800x600 to see how your site would appear). Too many features to mention here (read: I'm too lazy to type them all out).
  6. Java Script Debugger - This extension, known as Venkmann, is the JavaScript (or is it ECMAScript) debugger for Firefox/Mozilla. Very handy when used along with the Web Developer extension.
  7. Duplicate Tab - This is a great extension. It allows you to open a new tab which is the exact duplicat of another, including forward and back history.
  8. Diggler - Ever wish there was a button to clear the location bar contents? Diggler does that, and also gives you a menu of options for dealing with tabs, popup and image blocking. I'm sure Dirk would be proud.
  9. Sage - Sage is a RSS and ATOM feed aggregator. I've yet to really put it to task, but I've seen enough people raving about this one that I had to add it to the list (and my installation). It would probably work well for aggregating feeds from
  10. Basics - This very simple extension adds a 'new tab' button to the tab bar. Very handy

That's the unabbreviated list. It just so happens I only install 10 extensions, so it's perfect for a Top 10 list. Give Firefox 0.9.3 a try today!




« June 2016