Sunday Aug 04, 2013

Some times it’s a book sprint, other times it’s a marathon

One of the areas the X.Org Foundation has been working on in recent years is trying to bring new developers on board. Programs like Google Summer of Code and The X.Org Endless Vacation of Code (EVoC) help with this somewhat, but we’ve also been trying to find ways to lower the barrier to entry, both so the students in those programs can learn more and so that other developers have an easier time joining us.

Some of our efforts here have been technological - one of the driving efforts of the conversions from Imake to automake and from CVS to git was to make use of tools developers would already be familiar and productive with from other projects. The Modularization project, which broke up X.Org from one giant tree into over 200 small ones, had the goal of making it possible to fix a bug in a single library or driver without having to download and build many megabytes of software & fonts that were not being changed.

Another area of effort has been around the X.Org documentation, as I’ve written about before. Several years ago, Bart Massey suggested we try organizing a Book Sprint to write a guide for new X.Org developers, similar to the sprint he’d participated in to write a Google Summer of Code Mentoring Guide and a matching GSoC Student Guide. The Board agreed to let him try organizing one, but finding a time to bring everyone together was challenging.

We finally ended up holding a first session as a virtual gathering over the internet one weekend in March 2012. Bart, Matt Dew, Keith Packard, Stéphane Marchesin, and I used Gobby and Mumble to agree on an outline and write several chapters, along with creating some accompanying diagrams using graphviz dot. Unfortunately, the combination of the work from home setting, in which people kept dropping in and out as their home life intervened, the small number of people, and lack of an experienced organizer made this not as productive as other book sprints have been for other projects.

After the first weekend we had drafts of about 7 chapters and a preface, and outlines of several more. We also had a large chunk of prewritten material on graphics drivers that Stéphane had been working on already to link with. Over the next few months, Bart edited and formatted the chapters we had, while we got a little more written, including recruiting additional authors such as Peter Hutterer.

We then gathered again, this time in person, hosted by Matthias Hopf at the Georg-Simon-Ohm-University in Nürnberg, Germany, over the two days prior to the 2012 X.Org Developer’s Conference. We had some additional people join us this time, including Martin Peres, Lucas Stach, Ben Skeggs, and probably more I’ve forgotten. At this session, Bart, Matt & I worked on polsihing the rough edges on the guide we’d been creating, while the others worked more on the driver guide that Stéphane had started. Bart figured out the tools needed to generate ebook formats of the guide, such as epub, and made a cover and pulled together an overall structure for the guide, and by the end of that sprint, we had something we could read on the Android ebook reader he'd brought with him.

And then, we waited. Bart tried to figure out how to make the setup maintainable and reproducible, and how to make it available to our users, but his day job as a University professor kept taking time away from that. Bart also gave a lightning talk on our book sprint experience at the Write the Docs conference in Portland earlier this year covering what worked and what we learned didn’t work in our virtual sprint, and asking for ideas or help on how to go forward.

Meanwhile, the burden of fighting the spammers on the X.Org & FreeDesktop.Org wikis had gotten overwhelming on the current wiki software, so the freedesktop sitewranglers evaluated different solutions, and after looking at how well ikiwiki had been working for the past few years on the XCB wiki decided to move the rest of the FreeDesktop hosted wikis to ikiwiki as well. One major advantage is that it let us use the existing freedesktop authentication infrastructure for wiki accounts instead of having to find different ways to let our users in while keeping the spammers out. Another benefit is that it uses Markdown and git to update the wiki, so we can easily push files in Markdown format to the wiki to publish them now.

In a shocking coincidence, the geeks who wrote the X.Org Developer's Guide also used Markdown & git to author it, so with just a very little conversion to make links work between sections, it was possible to publish the guide to the wiki, where you can find it now:

It’s not perfect, it’s not even fully finished, but it’s better than nothing, and since it’s now on a wiki, it’s even easier for others to fill in the missing pieces or improve the existing bits.

Stephane is still working on the companion guide to graphics drivers, covering the stack from DRM and KMS in the kernel up to Xorg & Mesa. He posted the PDF of the draft as it was at the end of the March book sprint, but not yet a version with the contributions added from the September followup.

When working on applications higher up the stack, not hacking on the graphics stack itself, we refer developers to the documentation for their toolkits, or to general guides to OpenGL programming, as those are beyond what we can document ourselves in X.Org.

And for other open source projects, if you’d like a chance to have a professionally organized, in person doc sprint for your project, applications are being taken until August 7 for 2-4 projects to hold sprints as part of the Google Summer of Code Doc Camp in October, including travel expenses for up to 5 participants from the sponsors.

Thursday Oct 25, 2012

Documentation Changes in Solaris 11.1

One of the first places you can see Solaris 11.1 changes are in the docs, which have now been posted in the Solaris 11.1 Library on docs.oracle.com. I spent a good deal of time reviewing documentation for this release, and thought some would be interesting to blog about, but didn't review all the changes (not by a long shot), and am not going to cover all the changes here, so there's plenty left for you to discover on your own.

Just comparing the Solaris 11.1 Library list of docs against the Solaris 11 list will show a lot of reorganization and refactoring of the doc set, especially in the system administration guides. Hopefully the new break down will make it easier to get straight to the sections you need when a task is at hand.

Packaging System

Unfortunately, the excellent in-depth guide for how to build packages for the new Image Packaging System (IPS) in Solaris 11 wasn't done in time to make the initial Solaris 11 doc set. An interim version was published shortly after release, in PDF form on the OTN IPS page. For Solaris 11.1 it was included in the doc set, as Packaging and Delivering Software With the Image Packaging System in Oracle Solaris 11.1, so should be easier to find, and easier to share links to specific pages the HTML version.

Beyond just how to build a package, it includes details on how Solaris is packaged, and how package updates work, which may be useful to all system administrators who deal with Solaris 11 upgrades & installations. The Adding and Updating Oracle Solaris 11.1 Software Packages was also extended, including new sections on Relaxing Version Constraints Specified by Incorporations and Locking Packages to a Specified Version that may be of interest to those who want to keep the Solaris 11 versions of certain packages when they upgrade, such as the couple of packages that had functionality removed by an (unusual for an update release) End of Feature process in the 11.1 release.

Also added in this release is a document containing the lists of all the packages in each of the major package groups in Solaris 11.1 (solaris-desktop, solaris-large-server, and solaris-small-server). While you can simply get the contents of those groups from the package repository, either via the web interface or the pkg command line, the documentation puts them in handy tables for easier side-by-side comparison, or viewing the lists before you've installed the system to pick which one you want to initially install.

X Window System

We've not had good X11 coverage in the online Solaris docs in a while, mostly relying on the man pages, and upstream X.Org docs. In this release, we've integrated some X coverage into the Solaris 11.1 Desktop Adminstrator's Guide, including sections on installing fonts for fontconfig or legacy X11 clients, X server configuration, and setting up remote access via X11 or VNC. Of course we continue to work on improving the docs, including a lot of contributions to the upstream docs all OS'es share (more about that another time).

Security

One of the things Oracle likes to do for its products is to publish security guides for administrators & developers to know how to build systems that meet their security needs. For Solaris, we started this with Solaris 11, providing a guide for sysadmins to find where the security relevant configuration options were documented. The Solaris 11.1 Security Guidelines extend this to cover new security features, such as Address Space Layout Randomization (ASLR) and Read-Only Zones, as well as adding additional guidelines for existing features, such as how to limit the size of tmpfs filesystems, to avoid users driving the system into swap thrashing situations.

For developers, the corresponding document is the Developer's Guide to Oracle Solaris 11 Security, which has been the source for years for documentation of security-relevant Solaris API's such as PAM, GSS-API, and the Solaris Cryptographic Framework. For Solaris 11.1, a new appendix was added to start providing Secure Coding Guidelines for Developers, leveraging the CERT Secure Coding Standards and OWASP guidelines to provide the base recommendations for common programming languages and their standard API's. Solaris specific secure programming guidance was added via links to other documentation in the product doc set.

In parallel, we updated the Solaris C Libary Functions security considerations list with details of Solaris 11 enhancements such as FD_CLOEXEC flags, additional *at() functions, and new stdio functions such as asprintf() and getline(). A number of code examples throughout the Solaris 11.1 doc set were updated to follow these recommendations, changing unbounded strcpy() calls to strlcpy(), sprintf() to snprintf(), etc. so that developers following our examples start out with safer code. The Writing Device Drivers guide even had the appendix updated to list which of these utility functions, like snprintf() and strlcpy(), are now available via the Kernel DDI.

Little Things

Of course all the big new features got documented, and some major efforts were put into refactoring and renovation, but there were also a lot of smaller things that got fixed as well in the nearly a year between the Solaris 11 and 11.1 doc releases - again too many to list here, but a random sampling of the ones I know about & found interesting or useful:

Sunday Jan 23, 2011

X11R7.6 Documentation Improvements

Late last month (about three months late in fact, sorry about that), X.Org announced the release of X11R7.6. While the announcement, release notes, and changelog give details on all the changes that went into this release since the X11R7.5 release a year earlier, the one I worked on the most (aside from the work of producing and posting the modules to be released) was the modernization of the X documentation.

When the X.Org Foundation inherited stewardship of the X Window System, the documentation was in a wide variety of formats. The programs and most libraries included man pages in the traditional nroff format. The source tree also contained a large number of specifications of the protocols and libraries, and these were in a variety of formats, including troff, TeX, FrameMaker, and LinuxDoc. These were mostly optimized for print output and generated Postscript documents which were included in the releases since few people had all the tools necessary to process all of those, especially the commercial FrameMaker software. This also was a problem for developers who needed to update the docs, and either didn't know all the different formats or who didn't have tools like FrameMaker.

One of the early decisions of X.Org was that we wanted all our docs to be in open formats with open source toolchains. The format we decided to standardize on for the long form documents such as the specifications was DocBook, a open format that was already adopted by projects such as GNOME. Later, during the split of the monolithic X Window System, we decided that once the documents were converted, they would be moved to the modules that they documented, so that the documents would be updated and released in sync with the code. Over the following years, we made a little progress, converting a few documents here and there, but not making a serious dent.

In 2010 though, two volunteers really turned this around - Matt Dew converted the bulk of our specifications to DocBook/XML, and Gaetan Nadon integrated those into the modules and set up the autoconf macros and Makefile build rules to convert them to the various output formats in a standardized fashion. The xmlto tool is used as a front-end processor to drive backend tools including xsltproc and Apache FOP to generate output documents in text, html, PostScript, and/or PDF formats.

You can see the effects of this if you compare documents from the X11R7.5 release with some from the X11R7.6 release. For instance, in the html versions of the Xlib spec, not only has the formatting greatly improved from the 7.5 version to the 7.6 version, the new one also now has a hyperlinked table of contents and index, and many cross-reference links between individual sections. In the pdf version, the 7.5 version is a simple encapsulation of the postscript output that is at least nicely formatted since the original troff was designed for print output. In the 7.6 version, fop generated the output directly for PDF, and was able to produce output that took advantage of the additional features of PDF, such as including the same extended set of hyperlinks as the HTML version, and a set of "bookmarks" for quick navigation to sections listed in the table of contents.

If you're building the sources, there's a standardized set of configure flags to control the documentation tools used during the build, such as --with-fop to enable the use of fop to generate Postscript and PDF output or --without-fop to disable it. If you're installing prebuilt packages, you may find the documents in various formats provided in the module specific subdirectory of the documentation directory for your system, such as /usr/share/doc/libX11 for the documents provided with libX11. Even if the packager didn't preformat the html, text, or pdf for you, the default Makefiles install the xml versions there so you can format them when needed, or read online with a Docbook-viewing tool such as GNOME's Yelp.

While this is a huge step forward, we're not done yet. Many of the API docs still need to be converted from old-style K&R C prototypes to the ANSI C89 style used in the code base, and other formatting cleanups are needed. Matt is working on ways to not just have hyperlinked cross-references inside a single document, but across the documentation set. There's plenty of work to go around for additional people to help, such as improving the style-sheets, proofreading the documents to find more areas to fix, adding cross-references where they could be useful, so more help is always welcome. And of course, the huge pile of docs we have are almost exclusively developer focused - end user documentation is mainly limited to man pages, which aren't always as helpful as they could be, but we need user feedback to let us know the areas that need more help, since the developers don't have to rely on the man pages to figure out how to use the software, so don't notice the gaps.

But now at least we've got good momentum going, and I'm hopeful each future release will continue to show improvement.

About

Engineer working on Oracle Solaris and with the X.Org open source community.

Disclaimer

The views expressed on this blog are my own and do not necessarily reflect the views of Oracle, the X.Org Foundation, or anyone else.

See Also
Follow me on twitter

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today