Wikis, Docs, and the Reuse Proposition

Wiki systems make it easy to edit documents online. That makes them terrific for document collaboration. But current Wiki formats don't allow the kind of reuse that the DITA document format was designed for. But it may be possible to implement some of DITA's best features using a clever combination of JavaScript, CSS, and an extensible Wiki. I suspect it can be done most easily using a Ruby-based Wiki like MediaCloth.


I was recently asked two very interesting questions:

Is there an open source CMS that takes a Wiki approach to (DITA) editing?
and
When you were looking at Doc wikis, did you see any that you especially liked?

These questions explore the two basic approaches to a DITA/Wiki merge: Find a CMS that works like a Wiki, or find a way to make a Wiki work like DITA. Those two options are the subject of this post.

Note:
Paul Prescod first advanced the idea of a DITA-based Wiki,  but those are two different worlds that may never come together completely, as described in this blog post. There is a possible solution, however. It may be possible to implement the DITA features that support reuse using a clever combination of JavaScript, CSS and an extensible Wiki.

CMS that works like a Wiki

The Darwin Informaton Typing Architecture (DITA) is an XML-based document format that was designed from the ground up for reuse. It rocks. Content Managment Systms (CMSes) are designed to hold XML data. So in theory, a CMS system that lets you edit like a Wiki would be everything you need. But getting a system like that to work is a pretty tricky proposition.

There are 4 open source Java-based Wiki's I know of that are worth a thorough look: eXist, Daisy (InfoWorld review), Lenya, and Mirage. There could be more. I was planning to try them and do a comparison, but I'm going to be hard-pressed to find the time at the moment. (I hate when I don't have time for the really important work. This the kind of evaluation and analysis I excel at, and nothing makes me happier.)

Update:
Drupal-afficiando Josh Huckabee recently mentioned some Drupal-like open source CMS systems implemented in Ruby. In my reply to his blog, he mentioned: RailFrog, Mephisto and Radiant CMS.  They are no doubt worth a look, as well

But the real question in this scenario is the editing. Given that the CMS provides for version management, conflict prevention or resolution, and link management, the question is: How will engineers and the open source community be able to collaborate on the docs?

A browser-based editing system is obviously a requirement. For DITA, the solution costs $18k -- for an online editor from Just Systems called XMAX. It's the only one that works really well. However, it's a single-vendor, proprietary solution that doesn't let an organization like Sun leverage its technical expertise, should a desire to extend it emerge at some point in the future.

So the online options are limited, and the situation is unlikely to change anytime soon. We probably won't see many open source editors for DITA, precisely because the features that make DITA so powerful are also difficult to implement. It's three most powerful features are the ones that require a specialized DITA editor rather than a generic XML editor. Those features are modular composition (topics and "maps"), conditional text (metadata), and transclusions (content references).

Another alternative, as Kay Ramme reminds me, is to use a WebDAV-enabled CMS and interact with it using a WebDAV-enabled editor. There are several commercial editors that fit that bill, but a most intriguing option is the idea of using OpenOffice and ODF<-->DITA transforms, as described in Building a Bridge: DITA, DocBook, and ODF. That will be an interesting option to explore, when the transforms are available.

But even if terrific editors were available today, the even bigger hurdle is the fact that DITA is a difficult format to work with. It imposes many restrictions on what you can do, and where. The restrictions are benevolent, in that they help you produce topics that are more readily reusable, but they are restrictions, nonetheless. Initial experiences with DITA can be very frustrating until you internalize the knowledge of how it works.

Once you internalize that knowledge, you can be fluent. But the difficulty of doing so suggests that DITA is best suited for people who spend their day writing, rather than people who spend the majority of their time doing other things, like developing.

In short: DITA provides tremendous capacity for reuse, but it is a difficult format to author in. But what if we could achieve some degree of reuse in an easier data format? That's the subject of the next section.

Wiki that Works Like DITA

This is where the big win is to be found, in my opinion. I'm pretty sure that win will come in the form of a Ruby-based MediaWiki clone that runs on the Java platform using JRuby. The Wiki is called MediaCloth. The ease of coding in Ruby will help to make the necessary extensions possible, while the scalability and ubiquity of the Java platform add up to ease of installation. (And Java's class libraries don't hurt, either.)

Here's the deal:

  • There are 5 kinds of reuse that technical docs generally need, across products, versions, platforms, audiences, and document types (tutorials, courses, reference docs, etc). Reuse generally isn't important across all dimensions, but it is almost always important across some of them.

  • Wikis are the killer app for collaborative authoring, especially with online WYSIWYG editors, but they weren't originally designed for reuse. On the other hand, we're beginning to see some very interesting movement in that direction.

  • There are four features in particular that enable reuse in DITA:
  • Creation of small, reusable topics/modules.
  • A composition mechanism that joins them into larger units, like DITA "maps" (and the attendant  processing mechanisms).
  • Conditional text and variables (for topics with small differences in different settings)
  • Transclusions (for small bits that need to be included in multiple topics)
  • At least one group at Sun, led by Steve Wilson, has declared that they're not interested in producing books. With their "no books" policy, they're doing all their docs on a Wiki.
  • That effort has set a very important precedent for an organization that had  historically been wedded to books.

  • The downside to this approach is that it doesn't allow for conditional text. There is no composition mechanism, either, but it does allow for a topic-oriented approach, and the right Wiki allows for transclusions. That's 2 of the 4 critical features. And there are ways to address the missing features.

  • Clayton Cornell gave us a demo of the TOC functionality the Open Office doc'n group has implemented in MediaWiki. They've been able to use the template mechanism, page hierarchies, and show/hide CSS to create page-level TOCs that point to other Wiki pages. The result is a book. They're working on the code to process the nested TOCs into a PDF. Quite impressive stuff.

  • MediaWiki is the well-known system that is used for Wikipedia. So it's obviously well-tested and robust. It's written in PHP, which makes it "easy to extend" according to Clayton. But there is a downside:
  • It doesn't let us leverage our expertise in a JVM-based language.

  • While it's possible to find a pre-compiled binary of PHP for Solaris, that site that offers it gives a list of 16 libraries it depends on, with no inkling of how you find out if they exist on your system, where you get them if they don't, how to install them, and what if anything you need to configure to enable the PHP interpreter to find them. So like most things Solaris, installation requires a significant amount of background expertise.
    http://sunfreeware.com/programlistsparc10.html#php
(Clayton notes that CoolStack provides all of the needed extras like a recent php binary. This significantly reduces the background expertise. But it still looks complex to me, and it looks like you need to be root to do it.)
http://cooltools.sunsource.net/coolstack/index.html

  • However, there are Java-based and Ruby-based versions of MediaWiki. The Java-based version is JAMWiki (http://jamwiki.org). It has 12 developers, and it's most recent release was Oct 2007. It will be a strong provider of MediaWiki functionality, but it will be more difficult to extend in new directions, such as those discussed below.

  • The Ruby-based version is MediaCloth. Running that version in JRuby gives the best of both worlds--Java's ease of install, native threads, performance scalability, extensive class libraries, and equally-extensive documentation, plus Ruby's extensibility. (The extensibility is key, whether it runs in JRuby or Ruby.)

  • MediaCloth was last released in late 2005, but one of its 4 developers is Sun's Gregory Murphy, who assures me that development is going on in the trunk, and that they are very close to full MediaWiki compatibility.
Note: MediaCloth is the internal engine that converts Wiki text to HTML. It needs to be built into a Wiki server for online interactions. But that separation makes it easy for this purpose. And it maintains a parser generator to generate the conversion engine. You just define the language and the parser generator does the rest. So the parser generator could be used for other things like, say, a DITA map language or a dialogue-mapping language to enable online design-and-decision discussions.

So MediaCloth looks like the eventual winner, to me. I favor a Ruby-based solution because it's so much easier to extend. And extensions will be needed in a variety of areas, mostly to create the potential for reuse:

  • WYSIWYG editing
  • Transclusions
  • Conditional Text
  • Variables & Steps
  • Composition

WYSIWYG editing

A good visual editor needs to be added, if it's not already present. I know of one really good WYSIWYG editor (Xinha), and Clayton mentioned a couple of other popular  editors that are JavaScript-based, and that are reported to work well:

I haven't used the last two. But I use the commercial Xinha editor to write these posts, and I love it.

Transclusions

Transclusions need to be present, as well. Several Wikis already have that capability, which lets you inline material from another page. MediaWiki Templates provide transclusion capability for a single block of text, and there is an extension that allows you to pick and choose marked segments from other pages: http://www.mediawiki.org/wiki/Extension:Labeled_Section_Transclusion

But that's MediaWiki. The next step is to make sure that MediaCloth has similar functionality. (It might even be able to take advantage of DITA concepts, when doing so.)

Conditional Text

An important requirement for reuse is conditional text. But that capability can be implemented in Ruby. There are two possible levels of implementation:

  • Basic categories (fairly straightforward)
  • Hierarchical categories (pretty much a thought experiment, at this point)

Basic Categories

Ruby excels at creating custom languages. Most of the time, in Tim Bray's words, they're just "methods on steroids". But that simple capacity produces highly readable syntax. So it should be possible to construct a mechanism that looks something like this:

  solaris?  % cd /java/pubs
windows?  > cd c:\\java\\pubs

Method names can include punctuation in Ruby, and the standard idiom is to have boolean methods end with a question mark. The Wiki interpreter could pass the remainder of the line as a string or, at worst, the string might need to be encoded in quotation marks. The method would then either return the string or swallow it, depending on the settings of global variables.

Of course, it would be necessary to ensure that ordinary text isn't mistaken for a method name, so it might be necessary to encode the method calls using an rhtml tag, like this:

  <% solaris?  ... %>

When edited in a WYSIWYG editor, the methods would just appear to be text, so the editor wouldn't need any special features. (If the code is in an rhtml tag like <%...%>, then presumably the editor will just treat it as text. If not, some other encoding may be needed, like [%...%].)

For multiple lines of text, the syntax might have to wind  look something like this (but this is virtually guaranteed to work):

<% solaris? << END_TEXT
This is the text that will be
included when "solaris" has been specified
in the metadata
END_TEXT
%>

 That example uses a Ruby "here" document--a way of constructing a multi-line string using the notation << HERE ... HERE, where you can substitute any label you choose for HERE. (The convention is all caps.)

When translated into HTML, show/hide CSS and a tiny bit of JavaScript would implement the conditionalization semantics:

  • A small inset in the page would let the user choose their platform (solaris or windows, in this example).

  • If the solaris platform is chosen, that information is shown, while the windows information is hidden:

       % cd /java/pubs
  • If multiple selections are chosen (selection count > 1), then the platform labels are shown, as well:
       <b>solaris:</b>  % cd /java/pubs
<b>windows:</b>  > cd c:\\java\\pubs

  • Selections made in the interface would be stored in a cookie on the user's system, so every page they visit would reflect their preferred settings

  • The default CSS setting would be to "show all", in case the javascript were unintentionally left off  the page

Hierarchical categories

For all of it's power, DITA only provides for single-level categories. Hierarchies aren't supported. But they are clearly needed.

Take the Solaris platform, for example. There is information that is generic to all Solaris systems, and information that is unique to 64-bit systems. Although I haven't seen an example yet, there is also the potential for some information that is unique to 32-bit Solaris systems.

The most desirable form of metadata would be a hierarchy. There would be a "solaris" category, with two subcategories: "32" and "64". It would then be possible to mark something as "solaris", "solaris/32", or "solaris/64".

With a system like that, text marked as solaris/64 would automatically be included when "solaris" is selected. Similarly, text marked as "solaris" would be automatically included when "solaris/64" is selected.

But DITA doesn't work that way. To simulate that behavior, you need to define 3 metadata categories: solaris, solaris-32, and solaris-64. Then, when generating a document for 32-bit solaris, you need to remember to include both solaris and solaris-32. Similarly for solaris-64.

Although I don't currently see exactly how to do so, I feel sure that it would be feasible to implement more robust hierarchical semantics in a (J)Ruby-based implementation.

Variables and Steps

Variables could be implemented using a mechanism similar to conditionals. It's even easier, though, because Ruby is an expression language, in which the value of a method is the last expression evaluated. So substitutions could be implemented either with variables or methods.

There are multiple ways to get the job done. It's just a matter of choosing a good syntax.

Here is an example of variable substitution:

<% platform! %>

Adding the exclamation point to the name is a Ruby convention that identifies a destructive operation. Here, it's being used to identify a substitution. If only one platform has been selected, a value like "Solaris" would be returned. If multiple, a value like "Solaris/Linux/Windows" or "Solaris, Linux, Windows" would be returned, depending on how the underlying method was coded.

On the other hand, it might be easier to use a syntax that looks like this:

<% value_of :platform %> 

where the colon in front of "platform" (:platform) indicates that platform is a symbol.

Steps are similar. They're simply variables whose value changes as they are referenced. Many Wikis find it difficult to accommodate numbered steps that contain paragraphs within a step. The indentation gets messed up and/or the numbering is destroyed. But variable substitution could be used to invoke a method that substitutes something like "Step #2" for syntax like this:

<% start_step! %> Do something....

Composition

The page-level TOCs that Clayton's group put together constitute a composition mechanism. But at the moment, it's a single-level hierarchy. In other words, page X always points to sub-pages a, b, and c. That may make it harder to reuse page X in a different context

On the other hand, it's possible that conditional text and transclusions will provide a sufficient level of reuse. But if it turns out that topic reuse does matter, then some sort of extension will be needed so that the TOC and next/previous pointers depend on context, rather than being buried in the page.

Problem: The Containment Hierarchy

The problem stems from the containment hierarchy:

  • Each page contains a link to the previous and next page, as well as the TOC.
  • The TOC, in turn, contains a link to the previous and next section.

The containment hierarchy defeats reuse, because a page is irrevocably tied to its original context. To solve the problem, the containment hierarchy needs to be inverted, using a system that will need to look remarkably like DITA "maps".

With that inversion, the map arranges pages into a hierarchy, and previous/next links for sections and pages are generated when the page is constructed.

There are several interesting implications for a system of that kind:

  • A page will have one source file, but multiple renditions (one per context)

  • The previous/next links in the rendition pages may be difficult to generate renditions dynamically. If so, the pages will need to be pre-generated and cached, in which case editing a single source page will need to trigger the re-generation of each rendition page that is dependent on it.

  • It will be important to find every rendition-context in which a page appears, so that you don't make a change that renders the text invalid in a different context.

This is an area that is fraught with difficulty, so it makes sense to hold off implementing a solution until it turns out to be necessary. At that point, a DITA-style mapping model may be a good one to follow.

Solution? Page Containers

In response to this idea, Frank Peters wrote:

> One way to change that would be to set up page "containers" that
> transclude the actual content. In this way, the content itself would
> not know where it belongs and consequently could be reused in
> multiple contexts.

That is certainly a good thought. And if the containers are wiki pages themselves, they would easy easier to edit. They could probably use a more restricted, YAML-style syntax. (The standard Ruby syntax for properties files.)

On the other hand, that solution doesn't take advantage of DITA maps and the Open Toolkit that processes them into so many different deliverables (HTML, PDF, etc.) The section that follows shows how we might be able to do that.

Solution? DITA Maps

The DITA Open Toolkit (DITA-OT, or OT) includes a HTML to DITA conversion script (h2d, in the demo/ directory). I've used it to convert to the specialized DITA subtypes (concept, task, and reference). It does a terrific job, but with those restricted topic types, there are always several bits that need to be cleaned up because they're out of place.

The one thing I haven't done is to convert to the generic "topic" type. There are few restrictions in that format, so it's likely that the clean-up problem would pretty much disappear. If so, then Wiki pages could be fairly easily treated as generic DITA topics.

So let's assume that we have the conversion script in place, and that pages converts cleanly to generic DITA topics. In that case:

  • Books can be defined using DITA maps .

  • The DITA-OT can be used to create PDFs (solving one of the current problems faced by Wiki-based documentation).

  • DITA-OT can also be used to create single-page HTML files, Java help, HTML help, DocBook, and other formats.

  • Rake can be used to manage the dependencies, so that:

  1. Editing a page regenerates the DITA topic
  2. All outputs that depend on that topic are regenerated.
    (That processing can be done in very smart ways, by extending Rake for DITA processing, as described in Incremental DITA Builds.)

A mechanism of that kind could provide a possible solution, assuming that HTML files convert fairly cleanly to generic topics. That leaves a few problems left to solve:

  • Conditional-text methods need to produce conditional metadata tagging in DITA. That conversion needs to happen after h2d runs. The method name can then be converted to one or more metadata attributes on the element the text is in. (JAMOT. No technical issues here. Just A Matter Of Time.)

  • Wiki transclusions need to be converted to DITA content references. (Before h2d? Or afterward, using some sort of method invocation, as with conditional text?)

  • Page-level TOCs need to be generated from the map, and need to be dynamically included when the page is rendered, so the TOC someone sees on a page depends on which "book" they're in at the time. (This is the big one. Could require extensive hacking of the wiki, or a wicked-cool plugin at the very least.)

Bottom Line

MediaCloth running on JRuby looks like one heck of a good option for a doc-centric Wiki. With some clever code, it may well provide for modular reuse, as well as easy online editing.

Resources

Comments:

Eric, wow. I wonder how much of a problem it is that the typical wiki treats each page as one blob of text w/ markup? For my personal websites I use Drupal and out-of-the-box it also treats each page as one blob of text w/ markup, but you can add a module (Content Creation Kit, or CCK) and it's featureset supports turning each page into a series of fields. Such as ... I've defined a content type for "Digital Camera Information" and it has a series of fields describing camera features like resolution, image formats, memory cards, URL's for reviews, summaries of the reviews, etc. Rather than each page being a single blob of text w/ markup, each page is constructed out of a series of fields. It supports fields with several data types and the result is vaguely similar to a database table. But since you seem focused on Documents (with a capital-D) what I'm discussing doesn't translate properly. But maybe capital-D-Documents is a writing style which doesn't fit well for online information dispersal? Maybe the online form of documentation should be a cloud of datums.. that with appropriate tagging can be pulled together into a capital-D-Document? The most powerful feature about Drupal is its vocabulary system which has immensely flexible categorization capabilities. There may be a way with vocabulary categorization to do something along the lines you mentioned under "Page Containers". The cloud of datums which is the snippets of writing and other data that goes into a capital-D-Document could be organized by appropriate tagging into an appropriate sequence. I think.

Posted by David Herron on November 07, 2007 at 08:13 AM PST #

Hey, David. Thanks for sharing those insights into Drupal. I've heard about for some time, but haven't had a chance to investigate.

It sounds like Drupal has excellent transclusion capabilities, allowing you to build up a page from components, as well as the ability easily break a page down into its sub-components. That's cool.

But you're right the capital-D "Document" strategy requires aggregating pages into books or other larger collections of information. I believe I've heard that Drupal has some of those capabilities, as well.

The one thing I'm not sure of is how well Drupal supports reuse, as topics (pages) are aggregated in different ways, into different collections. But it sounds like any effort aimed at implementing the system I describe should start with an evaluation of Drupal, if only to identify ideas worth "borrowing".
:_)

Posted by Eric Armstrong on November 07, 2007 at 08:36 AM PST #

That was was an off the top of my head brainstorming of something Drupal might be capable of.

Drupal does have CCK which as I said allows for a sort of structured document. There is also what they call the 'Views module' which let's one construct a query into the content on the site. The views module is pretty complex and there are several closely related modules that add capabilities to views. One thing you can do with it is cause data from a View to be inserted into another page, so that's pretty close to the Inclusions concept you mentioned. If CCK makes content items into something akin to a database table, then Views is akin to database queries.

I had some more thinking on my way home last night. A more concrete to describing the brainstorm is .. one could have this "cloud" of snippets, a snippet is a few paragraphs, a section, a table of data, derivatives or aggregates of multiple data tables, etc. These snippets can be tied together into a 'Document' via some kind of tagging. The tagging ought to be seen as metadata and you described pretty well some useful kinds of tags. I'd also say the tags could describe the chapter/section/subsection ordering of snippets into a Document. And by keeping metadata separated from content snippets I think the snippets could be used in multiple contexts.

But I think taking an approach like this could also end up with Documents that don't have good writing style. If each snippet is written as a standalone thing, then how do you get good flow of wording between sections? This is especially true if you have snippets created by multiple authors, and snippets are reused in multiple documents. A good snippet would stand well on its own, but that interferes with flow as it's embedded in Documents.

Since Drupal is implemented in PHP it might be better as a source of ideas than a source of code.

Posted by David Herron on November 08, 2007 at 03:47 AM PST #

Oh, uh, "The one thing I'm not sure of is how well Drupal supports reuse, as topics (pages) are aggregated in different ways, into different collections." Um, out of the box Drupal autogenerates pages based on the vocabulary terms attached to a 'node'. The vocabulary terms is the categorization, you can have multiple vocabularies, each with their own terms. The autogenerated pages list the teasers for all nodes having a given vocabulary term. For example on www.7gen.com (my site) I have a vocabulary 'Topics' and a term 'Electric Vehicles', and every posting having this vocabulary+term has its' teaser listed on the autogenerated page for this vocabulary+term. I have a different vocabulary, 'Websites', in which is also a term 'Electric Vehicles'. Despite it being the same text it is a different term because it is in a different vocabulary.

There are some modules that let you insert contents of one node into another page. I haven't used that module.

Drupal's Book module is part of the core capabilities. It structures pages into a tree hierarchy. Each page in a book has a parent, or else it has a 'null parent' in which case it's the opening page of a 'book'. Each book page can have multiple children. The printable version of a Drupal 'book' is a traversal of the pages. And there's a module which can export a Drupal 'book' into other formats.

Posted by David Herron on November 08, 2007 at 03:55 AM PST #

In view of using an advanced content management system to build your dynamic portal engine, it might be useful to have a look at WEBRIQ.
http://www.webriq.com

Posted by Philippe Bodart on February 11, 2009 at 11:28 PM PST #

Is there a way to make backspace work? Looks like pressing backspace does not update the results.

Posted by Exchange on October 11, 2010 at 11:36 AM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today