Wikis, Docs, and the Reuse Proposition
By Eric Armstrong on Nov 04, 2007
I was recently asked two very interesting questions:
Is there an open source CMS that takes a Wiki approach to (DITA) editing?and
When you were looking at Doc wikis, did you see any that you especially liked?
These questions explore the two basic approaches to a DITA/Wiki merge: Find a CMS that works like a Wiki, or find a way to make a Wiki work
like DITA. Those two options are the subject of this post.
CMS that works like a Wiki
The Darwin Informaton Typing Architecture (DITA) is an XML-based
document format that was designed from the ground up for reuse. It
rocks. Content Managment Systms (CMSes) are designed to hold XML data.
So in theory, a CMS system that lets you edit like a Wiki would be
everything you need. But getting a system like that to work is a pretty
There are 4 open source Java-based Wiki's I know of that are worth a thorough look: eXist, Daisy (InfoWorld review), Lenya, and Mirage. There could be more. I was planning to try them and do a comparison, but I'm going to be hard-pressed to find the time at the moment. (I hate when I don't have time for the really important work. This the kind of evaluation and analysis I excel at, and nothing makes me happier.)
Drupal-afficiando Josh Huckabee recently mentioned some Drupal-like open source CMS systems implemented in Ruby. In my reply to his blog, he mentioned: RailFrog, Mephisto and Radiant CMS. They are no doubt worth a look, as well
But the real question in this scenario is the editing. Given that the
CMS provides for version management, conflict prevention or resolution,
and link management, the question is: How will engineers and the open
source community be able to collaborate on the docs?
A browser-based editing system is obviously a requirement. For DITA, the solution costs $18k -- for an online editor from Just Systems called XMAX. It's the only one that works really well. However, it's a single-vendor, proprietary solution that doesn't let an organization like Sun leverage its technical expertise, should a desire to extend it emerge at some point in the future.
So the online options are limited, and the situation is unlikely to change anytime soon. We probably won't see many open source editors for DITA, precisely because the features that make DITA so powerful are also difficult to implement. It's three most powerful features are the ones that require a specialized DITA editor rather than a generic XML editor. Those features are modular composition (topics and "maps"), conditional text (metadata), and transclusions (content references).
Another alternative, as Kay Ramme reminds me, is to use a WebDAV-enabled CMS and interact with it using a WebDAV-enabled editor. There are several commercial editors that fit that bill, but a most intriguing option is the idea of using OpenOffice and ODF<-->DITA transforms, as described in Building a Bridge: DITA, DocBook, and ODF. That will be an interesting option to explore, when the transforms are available.
But even if terrific editors were available today, the even bigger hurdle is the fact that DITA is a difficult format to work with. It imposes many restrictions on what you can do, and where. The restrictions are benevolent, in that they help you produce topics that are more readily reusable, but they are restrictions, nonetheless. Initial experiences with DITA can be very frustrating until you internalize the knowledge of how it works.
Once you internalize that knowledge, you can be fluent. But the difficulty of doing so suggests that DITA is best suited for people who spend their day writing, rather than people who spend the majority of their time doing other things, like developing.
In short: DITA provides tremendous capacity for reuse, but it is a difficult format to author in. But what if we could achieve some degree of reuse in an easier data format? That's the subject of the next section.
Wiki that Works Like DITA
This is where the big win is to be found, in my opinion. I'm pretty sure
that win will come in the form of a Ruby-based MediaWiki clone that runs
on the Java platform using JRuby. The Wiki is called MediaCloth. The ease of coding in Ruby will help to make the necessary extensions
possible, while the scalability and ubiquity of the Java platform add up
to ease of installation. (And Java's class libraries don't hurt, either.)
Here's the deal:
- There are 5 kinds of reuse that technical docs generally need, across products, versions, platforms, audiences, and document types (tutorials, courses, reference docs, etc). Reuse generally isn't important across all dimensions, but it is almost always important across some of them.
- Wikis are the killer app for collaborative authoring, especially with online WYSIWYG editors, but they weren't originally designed for reuse. On the other hand, we're beginning to see some very interesting movement in that direction.
- There are four features in particular that enable reuse in DITA:
- Creation of small, reusable topics/modules.
- A composition mechanism that joins them into larger units, like DITA "maps" (and the attendant processing mechanisms).
- Conditional text and variables (for topics with small differences in different settings)
- Transclusions (for small bits that need to be included in multiple topics)
- At least one group at Sun, led by Steve Wilson, has declared that they're not interested in producing books. With their "no books" policy, they're doing all their docs on a Wiki.
- That effort has set a very important precedent for an organization that had historically been wedded to books.
- The downside to this approach is that it doesn't allow for conditional text. There is no composition mechanism, either, but it does allow for a topic-oriented approach, and the right Wiki allows for transclusions. That's 2 of the 4 critical features. And there are ways to address the missing features.
- Clayton Cornell gave us a demo of the TOC functionality the Open Office doc'n group has implemented in MediaWiki. They've been able to use the template mechanism, page hierarchies, and show/hide CSS to create page-level TOCs that point to other Wiki pages. The result is a book. They're working on the code to process the nested TOCs into a PDF. Quite impressive stuff.
- MediaWiki is the well-known system that is used for Wikipedia. So it's obviously well-tested and robust. It's written in PHP, which makes it "easy to extend" according to Clayton. But there is a downside:
- It doesn't let us leverage our expertise in a JVM-based language.
- While it's possible to find a pre-compiled binary of PHP for Solaris, that site that offers it gives a list of 16 libraries it depends on, with no inkling of how you find out if they exist on your system, where you get them if they don't, how to install them, and what if anything you need to configure to enable the PHP interpreter to find them. So like most things Solaris, installation requires a significant amount of background expertise.
(Clayton notes that CoolStack provides all of the needed extras like a recent php binary. This significantly reduces the background expertise. But it still looks complex to me, and it looks like you need to be root to do it.)
- However, there are Java-based and Ruby-based versions of MediaWiki. The Java-based version is JAMWiki (http://jamwiki.org). It has 12 developers, and it's most recent release was Oct 2007. It will be a strong provider of MediaWiki functionality, but it will be more difficult to extend in new directions, such as those discussed below.
- The Ruby-based version is MediaCloth. Running that
version in JRuby gives the best of both worlds--Java's ease of install, native threads, performance scalability, extensive class libraries, and equally-extensive documentation, plus Ruby's extensibility. (The extensibility is key, whether it runs in JRuby or Ruby.)
- MediaCloth was last released in late 2005, but one of its 4 developers is Sun's Gregory Murphy, who assures me that development is going on in the trunk, and that they are very close to full MediaWiki compatibility.
Note: MediaCloth is the internal engine that converts Wiki text to HTML. It needs to be built into a Wiki server for online interactions. But that separation makes it easy for this purpose. And it maintains a parser generator to generate the conversion engine. You just define the language and the parser generator does the rest. So the parser generator could be used for other things like, say, a DITA map language or a dialogue-mapping language to enable online design-and-decision discussions.
So MediaCloth looks like the eventual winner, to me.
I favor a Ruby-based solution because it's so much
easier to extend. And extensions will be needed in a
variety of areas, mostly to create the potential for reuse:
- WYSIWYG editing
- Conditional Text
- Variables & Steps
I haven't used the last two. But I use the
commercial Xinha editor to write these posts, and I love it.
Transclusions need to be present, as well. Several Wikis already have that capability, which lets you inline material from another page. MediaWiki Templates provide transclusion capability for a
single block of text, and there is an extension that allows you to pick and choose marked segments from other pages: http://www.mediawiki.org/wiki/Extension:Labeled_Section_Transclusion
But that's MediaWiki. The next step is to make sure that MediaCloth has similar functionality. (It might even be able to take advantage of DITA concepts, when doing so.)
An important requirement for reuse is conditional text. But that capability can be implemented in Ruby. There are two possible levels of implementation:
- Basic categories (fairly straightforward)
- Hierarchical categories (pretty much a thought experiment, at this point)
Ruby excels at creating custom languages. Most of the time, in Tim Bray's words, they're just "methods on steroids". But that simple capacity produces highly readable syntax. So it should be possible to construct a mechanism that looks something like this:
solaris? % cd /java/pubs
windows? > cd c:\\java\\pubs
Method names can include punctuation in Ruby, and the standard idiom is to have boolean methods end with a question mark. The Wiki interpreter could pass the remainder of the line as a string or, at worst, the string might need to be encoded in quotation marks. The method would then either return the string or swallow it, depending on the settings of global variables.
Of course, it would be necessary to ensure that ordinary text isn't mistaken for a method name, so it might be necessary to encode the method calls using an rhtml tag, like this:
<% solaris? ... %>
When edited in a WYSIWYG editor, the methods would just appear to be text, so the editor wouldn't need any special features. (If the code is in an rhtml tag like <%...%>, then presumably the editor will just treat it as text. If not, some other encoding may be needed, like [%...%].)
For multiple lines of text, the syntax might have to wind look something like this (but this is virtually guaranteed to work):
<% solaris? << END_TEXT
This is the text that will be
included when "solaris" has been specified
in the metadata
That example uses a Ruby "here" document--a way of constructing a multi-line string using the notation << HERE ... HERE, where you can substitute any label you choose for HERE. (The convention is all caps.)
- A small inset in the page would let the user choose their platform (solaris or windows, in this example).
- If the solaris platform is chosen, that information is shown, while the windows information is hidden:
% cd /java/pubs
- If multiple selections are chosen (selection count > 1), then the platform labels are shown, as well:
<b>solaris:</b> % cd /java/pubs
<b>windows:</b> > cd c:\\java\\pubs
- Selections made in the interface would be stored in a cookie on the user's system, so every page they visit would reflect their preferred settings
For all of it's power, DITA only provides for single-level categories. Hierarchies aren't supported.
But they are clearly needed.
Take the Solaris platform, for example. There is information that is generic to all Solaris systems, and information that is unique to 64-bit systems. Although I haven't seen an example yet, there is also the potential for some information that is unique to 32-bit Solaris systems.
The most desirable form of metadata would be a hierarchy. There would be a "solaris" category, with two subcategories: "32" and "64". It would then be possible to mark something as "solaris", "solaris/32", or "solaris/64".
With a system like that, text marked as solaris/64 would automatically be included when "solaris" is selected. Similarly, text marked as "solaris" would be automatically included when "solaris/64" is selected.
But DITA doesn't work that way. To simulate that behavior, you need to define 3 metadata categories: solaris, solaris-32, and solaris-64. Then, when generating a document for 32-bit solaris, you need to remember to include both solaris and solaris-32. Similarly for solaris-64.
Although I don't currently see exactly how to do so, I feel sure that it would be feasible to implement more robust hierarchical semantics in a (J)Ruby-based implementation.
Variables and Steps
Variables could be implemented using a mechanism similar to conditionals. It's even easier, though, because Ruby is an expression language, in which the value of a method is the last expression evaluated. So substitutions could be implemented either with variables or methods.
There are multiple ways to get the job done. It's just a matter of choosing a good syntax.
Here is an example of variable substitution:
<% platform! %>
Adding the exclamation point to the name is a Ruby convention that identifies a destructive operation. Here, it's being used to identify a substitution. If only one platform has been selected, a value like "Solaris" would be returned. If multiple, a value like "Solaris/Linux/Windows" or "Solaris, Linux, Windows" would be returned, depending on how the underlying method was coded.
On the other hand, it might be easier to use a syntax that looks like this:
<% value_of :platform %>
where the colon in front of "platform" (:platform) indicates that platform is a symbol.
Steps are similar. They're simply variables whose value changes as they are referenced. Many Wikis find it difficult to accommodate numbered steps that contain paragraphs within a step. The indentation gets messed up and/or the numbering is destroyed. But variable substitution could be used to invoke a method that substitutes something like "Step #2" for syntax like this:
<% start_step! %> Do something....
The page-level TOCs that Clayton's group put together constitute a composition mechanism. But at the moment, it's a single-level hierarchy. In other words, page X always points to sub-pages a, b, and c. That may make it harder to reuse page X in a different context
On the other hand, it's possible that conditional text and transclusions will provide a sufficient level of reuse. But if it turns out that topic reuse does matter, then some sort of extension will be needed so that the TOC and next/previous pointers depend on context, rather than being buried in the page.
Problem: The Containment Hierarchy
The problem stems from the containment hierarchy:
- Each page contains a link to the previous and next page, as well as the TOC.
- The TOC, in turn, contains a link to the previous and next section.
The containment hierarchy defeats reuse, because a page is
irrevocably tied to its original context. To solve the
problem, the containment hierarchy needs to be inverted, using a system that will need to look remarkably like DITA
With that inversion, the map arranges pages into a hierarchy, and previous/next links for sections and pages are generated when the page is constructed.
There are several interesting implications for a system of that kind:
- A page will have one source file, but multiple renditions (one per context)
- The previous/next links in the rendition pages may be difficult to generate renditions dynamically. If so, the pages will need to be pre-generated and cached, in which case editing a single source page will need to trigger the re-generation of each rendition page that is dependent on it.
- It will be important to find every rendition-context in which a page appears, so that you don't make a change that renders the text invalid in a different context.
This is an area that is fraught with difficulty, so it makes
sense to hold off implementing a solution until it turns out
to be necessary. At that point, a DITA-style mapping model may
be a good one to follow.
Solution? Page Containers
In response to this idea, Frank Peters wrote:
> One way to change that would be to set up page "containers" that
> transclude the actual content. In this way, the content itself would
> not know where it belongs and consequently could be reused in
> multiple contexts.
That is certainly a good thought. And if the containers are wiki pages themselves, they would easy easier to edit. They could probably use a more restricted, YAML-style syntax. (The standard Ruby syntax for properties files.)
On the other hand, that solution doesn't take advantage of DITA maps and the Open Toolkit that processes them into so many different deliverables (HTML, PDF, etc.) The section that follows shows how we might be able to do that.
Solution? DITA Maps
The DITA Open Toolkit (DITA-OT, or OT) includes a HTML to DITA conversion script (h2d, in the demo/ directory). I've used it to convert to the specialized DITA subtypes (concept, task, and reference). It does a terrific job, but with those restricted topic types, there are always several bits that need to be cleaned up because they're out of place.
The one thing I haven't done is to convert to the generic "topic" type. There are few restrictions in that format, so it's likely that the clean-up problem would pretty much disappear. If so, then Wiki pages could be fairly easily treated as generic DITA topics.
So let's assume that we have the conversion script in place,
and that pages converts cleanly to generic DITA topics. In
- Books can be defined using DITA maps
- The DITA-OT can be used to create PDFs (solving one of the current problems faced by Wiki-based documentation).
- DITA-OT can also be used to create single-page HTML files, Java help, HTML help, DocBook, and other formats.
- Rake can be used to manage the dependencies, so that:
- Editing a page regenerates the DITA topic
- All outputs that depend on that topic are regenerated.
(That processing can be done in very smart ways, by extending Rake for DITA processing, as described in Incremental DITA Builds.)
A mechanism of that kind could provide a possible solution,
assuming that HTML files convert fairly cleanly to generic
topics. That leaves a few problems left to solve:
- Conditional-text methods need to produce conditional metadata tagging in DITA. That conversion needs to happen after h2d runs. The method name can then be converted to one or more metadata attributes on the element the text is in. (JAMOT. No technical issues here. Just A Matter Of Time.)
- Wiki transclusions need to be converted to DITA content references. (Before h2d? Or afterward, using some sort of method invocation, as with conditional text?)
- Page-level TOCs need to be generated from the map, and need to be dynamically included when the page is rendered, so the TOC someone sees on a page depends on which "book" they're in at the time. (This is the big one. Could require extensive hacking of the wiki, or a wicked-cool plugin at the very least.)
MediaCloth running on JRuby looks like one heck of a good option for a doc-centric Wiki. With some clever code, it may well provide for modular reuse, as well as easy online editing.
- The Convergence of Structure and Chaos
(Paul Prescod's paper showing similarities between DITA and Wikis.)
- Online Document Collaboration
(Blog post that explores differences between DITA and Wikis.)
- DITA: One Cool Document Format
(Blog post the explains why DITA is so powerful.)
- MediaWiki Transclusions
- Incremental DITA Builds
- Open source WYSIWYG editors
- PHP downloads
- JRuby Project
- Open Source CMS Systems (Ruby-based)
- eXist: http://exist.sourceforge.net/
- Daisy: http://www.cocoondev.org/daisy/index.html
- Lenya: http://lenya.apache.org/
- Mirage: https://mirage.dev.java.net/public/CMSFramework-DesignDiagram.html
- Online Editors