My Apache WebDAV/Windows Nightmare
By Eric Armstrong on Jul 01, 2008
Subversion is WebDAV-ready. Cool! "Just enable WebDAV in Apache". Riiigght... Like it was really that simple! But even after you get it working, there are problems: DreamWeaver's synchronization mechanisms leave a lot to be desired, and XMetaL access depends on mapping a drive in Windows--a mechanism that simply doesn't work--unless you manage to get SSL working in your WebDAV server, so you can make an https connection (something I never managed to do in Apache).
- Initial Installation
- Access Path Issues
- Browsing Issues
- DreamWeaver's Synchronization Model
- XMetaL's Dependency on Windows
The goal was to use Subversion (SVN) as a poor man's CMS, and take advantage of great PC-based editors like DreamWeaver (for HTML) and XMetaL (for DITA). Eventually, we could add pre-commit checks and utilities to give us some of the advanced functionality we'd really like--like link management and metadata change management--but in the meantime we could do everything manually to get by.
All we had to do was install Subversion and enable the WebDAV interface in Apache. But many hurdles later, I'm exhausted from jumping over them. Every one requires me to look through 20 web pages in search of a solution, and each time I surmount one obstacle, it's only to find a new one standing in my way.
To be fair, Subversion was really easy to install. And it was easy to set up a Subversion repository. But then things started getting dicey.
The first step in enabling WebDAV was to find the Solaris packages we needed. We needed Apache2 (we were running Apache 1.0), and we needed Subversion's WebDAV plugin for Apache. I took a look at what was out there to figure out what we needed, and then slowly backed away in horror.
There were dozens of packages to choose from, all of which had dependencies on other packages, which had dependencies themselves, and so on, in a spasm of dependency recursions. I didn't know what we had, which ones to choose, or how to make them play together once I collected them. Fortunately, the sys admins came to our rescue. They installed Apache2. collected the stuff we needed, and told us where to find it. One problem down.
The next step was to figure out the Apache configuration. It turns out that you want to work with the configuration file for the HTTP "demon". (As Unix server-side processes are generally known.) That's httpd.conf. You add the WebDAV module and the SVN module that plugs into it. Then you enable an additional config file: extras/httpd-dav.conf.
Into that file, you put your directory specification and location directive. Both are important. The directory specification tells Apache where the files live, while the location directive tells Apache which URLs to route to it.
There was also the minor issue of when to specify the SVN root directory and when to specify the trunk/ subdirectory. In general, you point Apache to the root directory, but point all clients to the trunk/ subdirectory (or, conceivably, to one of the directories in the branches/ hierarchy.
So far, so good. It only took about a week to get the details ironed out. We were then up and running! Well, almost, anyway. It turned out that there were still a few access-path issues to sort out...
Access Path Issues
We were running Apache on a Solaris box, with the intention of accessing it using DreamWeaver and XMetaL running on PCs. The idea was to maintain Subversion access at the same time, for folks running emacs and vi on Solaris. There were some interesting issues, as shown in the following diagram (click for a larger version):
At this point, I hadn't gotten around to trying XMetaL, so that path was still "To Be Determined". (When I got to it, its status became "You can't get there from here".)
The WebDAV path in DreamWeaver was super simple to set up. Fill in a couple of entries in DreamWeaver's Remote Site dialog, and things just worked. There was even a Test button in that dialog to help you figure out when you got the configuration right. But after that, things got a lot less simple. As you'll see in the next section, WebDAV access doesn't work all that well in DreamWeaver 8.
That left Subversion client access. At the top of the diagram, over on the right, you can see that creating a shell on the Solaris box "just works". Of course, the Subversion repository had to be set up with the right permissions. It had to be owned by "nobody", so that the Apache process could make changes to it. And "nobody" had to be in the same group as the user running the shell process, so that the shell process had access rights to update the SVN repository.
When running a Subversion client on the PC, however, only one in three access paths worked. Tortoise SVN was the client of choice (TSVN). There is even a TSVN plugin for DreamWeaver, which is another point in its favor. And it worked beautifully--but only when it was pointed to the SVN repository with a URL. Interestingly enough, that option works via WebDAV. So a WebDAV-enabled server is a requirement in this scenario. But once set up, things just work.
The other two options were to use TSVN with a local child workspace or with a mapped drive on the Solaris server. With a local child, updates worked, but commits failed because the SMB process was running as user "root", with an unknown group, so it did not have permissions to update the SVN repository. (That option could be made to work by changing process permissions, but there could be unexpected side effects. In any case, process ownership was in the domain of the sys admins, so we didn't go any further down that path.)
The other interesting case was using TSVN with a child workspace that was on a mapped drive. The drive was on the Solaris server, with access mediated by SMB. But the path to the parent workspace differs, depending on whether you are running in a TSVN client or in a Solaris shell--creating a situation in which it was possible to create a workspace on the Solaris box that could be updated either from TSVN or from a Solaris shell, but not both. And TSVN commits would always fail, due to the permissions problems.
In the end, there were three access paths that worked:
- Direct WebDAV access from DreamWeaver
- Indirect WebDAV access using a URL with TSVN on the PC
- SVN client access from a shell on the Solaris system
Subversion has the interesting characteristic that the master repository consists of the database, and only the database. There isn't a copy of files for people to look at or copy from as part of a publication process. To see the content, you must use a child workspace or WebDAV access.
WebDAV access lets people get to the repository to update it over the web, but it requires a username and password. That was fine for our department, who would be updating files, but we also wanted browsing to be open to the rest of the organization.
For a limited access, inside-the-firewall repository like this one, it made sense to skip the procedure for enabling LDAP authentication in Apache. I'm sure it wouldn't have taken more than another week. Users would then have the ability to change their passwords. Lacking that capability, each username and password had to be manually added to a file for Apache to consult.
We wanted users to see the results of the latest changes to the repository, but it turned out that there wasn't an easy way to provide it. With direct browsing, the URL would be
something like http://<ourServer>/<SvnRepo>/trunk. That wasn't terrible, but it wasn't as nice as we'd like. And there were .svn directories hanging around that Apache had to be told to ignore.
What we wanted, of course, was a clean "content mirror" for users to browse--one that didn't have "trunk" in the path. There were three possibilities, but only one of them worked:
The first possibility was to create a <location> directive in Apache that pointed to the trunk directory. That would take care of the URL, and Apache could then be told to ignore .svn directories. But that directive didn't work. Once the directory is turned over to the WebDAV module, it's no longer available as a target for redirection.
The second possibility was to use the Subversion export function, whose sole goal is to make a clean copy of a repository, suitable for browsing. I figured I could put it in a post-commit hook. It would be somewhat costly as the repository grew larger, but it would be a quick and easy solution for now. But the gods of computing laughed at me. "Ha, ha", they said, "we didn't put this in the documentation, but 'export' only works from a child workspace, you can't do it from the master repository!" So much for that possibility.
- The only remaining possibility was the one that everyone winds up doing, for the simple reason that nothing else works as well: Create a separate child workspace and update it in a post-commit hook. Then use an Apache <location> directive to direct the browser's URL to the trunk/ subdirectory.
In our case, we wound up creating:
- web_svn/ -- the Subversion repository
- web_svn_content_mirror/ -- the updated child
- http://<ourServer>/web -- an Apache redirect to web_svn_content_mirror/trunk
DreamWeaver's Synchronization Model
At this point, we had a WebDAV-enabled Subversion repository and a way to browse it. While we were doing that, we were also experimenting with DreamWeaver 8. I'm a big fan of DreamWeaver. For HTML editing, there is nothing finer. But it's site-update model simply doesn't work well in a multi-user, WebDAV setting.
Before getting to that point, though, there were a couple of other problems to work through:
The first problem was in doing what came naturally -- creating one directory and using it as both a DreamWeaver site and as a Subversion child workspace. (It was all the more natural, given a TSVN plugin for DreamWeaver.) But that arrangement turns out to be problematic. DreamWeaver keeps "site synchronization" files somewhere. Those files get out of whack when you commit changes from the child workspace. After that, the synchronization files don't match either the server state or the child state. DreamWeaver doesn't seem to recover well from the situation, and you wind up with weird files in your repository (probably synchronization files).
- The other problem was most likely the result of my attempt to create a mirror directory under the SVN root (adjacent to the trunk/ directory), instead of outside it. At that point, things got really confused, with spurious files and copies of the hierarchy showing up all over the place. I was forced to delete the main repository and recreate it. So it was a bad idea in itself, made worse by using the same directory for a DreamWeaver site and an SVN child.
Once those problems were ironed out, the repository was showing the right files once again. It remained necessary to warn others to use separate directories for TSVN and WebDAV, but we could live with that. The real problem turned out to be the DreamWeaver synchronization model.
FTPVoyager remains the exemplar for how synchronization operations should work. Unfortunately, the implementation in DreamWeaver 8 falls far short of that standard.
That model isn't documented anywhere I could see, but it turns out to work like this:
A DreamWeaver "site" has two views: the local view (files on your local system) and a remote view (generally the web site you're updating). You can also think of them as the "local site" and the "remote site".
When you delete, rename, or move a file in the local view, that change occurs only in the local copy of the files.
When you delete, rename, or move a file the remote view, that change occurs only in the remote files. (Those operations were specifically added to the http-access protocols to support WebDAV operations like these.)
However, when you "synchronize" the views, only new and modified files participate in the synchronization.
At times, it's handy to do operations that only affect one site. But DreamWeaver need two versions of the move, rename, and delete operations: One that only affects the site you're viewing, and one that replicates the changes at the other site. But you only get one, and the documentation doesn't tell you which one it is.
It's one thing when structural changes made to your local site aren't automatically reflected at the remote site. You can live with that in a single-user system, where you are the only one making changes. But it's quite another situation when structural changes aren't reflected in other user's local sites. Because of that, a file you thought you had removed could magically reappear at any moment.
One seemingly plausible solution for that problem would be to eschew the local view altogether, and confine yourself to working in the remote view. But there are perils with that approach:
When you edit a file on the remote site, DreamWeaver makes a copy in your local site. All saves go there. If you don't enable "automatic upload to server on save", and forget to upload the file manually, you can wind up overwriting your changes the next time you edit that file. (A dialog gives you a warning, but it's easy to make the wrong choice by mistake.)
- If you do enable "automatic upload to server on save", then every time you save the file, you add another version to the repository history, and you have to wait for the operation to complete before you can get back to editing. It takes a noticeable amount of time, even with a fast connection. With a slow connection, it's not really practical.
What's really need is "auto upload on file close". But DreamWeaver 8 doesn't work that way. (The alternative discussed in the next solution could be a terrific solution--if only it worked!)
- When you open a dialog to browse for a file to link to, it's your local site that is browsed. So you have to copy files to your local site to link to them.
Worse, when you start DreamWeaver, you always start in the local view, even if you left the site in the remote view. That makes it easy to edit the wrong copy of a file--and that file could be one that is not supposed to exist. If anyone does that, and uploads their changes, an old file that should be gone will mysteriously reappear, and the edit will have been made to the wrong file.
Since DreamWeaver doesn't sync structural changes, the only way to be sure you avoid such problems solution is for every user to delete all the files in their local sites and copy over a new set, whenever any user makes a structural change.
But a system that depends on everyone doing the right thing is obviously pretty brittle--especially when it's so difficult to figure out what the right thing is. So it's only a matter of time until unfortunate consequences ensue.
After playing with it for a while, the folks on my team pretty much gave up on WebDAV and stuck to using TSVN--a strategy that turns out to be the only realistic solution for XMetaL, as well.
XMetaL's Dependency on Windows
When I began to explore XMetaL's WebDAV capabilities, things sounded really good. "Just use Windows file explorer to map a drive to the WebDAV repository, then access that drive like any other file on your system". Wow, I thought. Pretty cool. I never knew Windows could do that!
That solution would not only be super easy, it could also solve DreamWeaver's syncrhonization problems. You could forget about creating a remote site, and just edit files in the mapped drive as though they were local files. Awesome.
After some more investigating, I got even more excited. It turns out that Windows copies a file to a temp directory when you open it for editing. That sounded like it could be the ideal implementation: All of your saves go to the local disk, with an automatic upload to the remote repository when you close the file!
That was truly the solution! I was now several weeks into the process, but it would worth it to spend a few more days to make this access path work!
Ah, such optimism. No amount of infernal difficulty seems to quench my unshakable belief that the next technology I encounter will "just work". Call it faith in human nature. I just naturally expect that things will be implemented in the most useful way possible. And I continue to hold that belief, despite truckloads of evidence to the contrary. I guess it's a character flaw. Anyway, here's what I found:
When I tried to map a drive, I got an error that said my username and password were not recognized.
- After I entered my user name and the mapping attempt failed, the username field was filled in with <domain>/<username>.
That was a significant clue as to the problem. Many web searches and failed recommendations later, I came across the stunning revelation that Windows XP and Vista, in their infinite wisdom, add the system domain to the user name when mapping to a WebDAV target, for reasons known only to the authors of the implementation.
I found nearly a dozen workarounds at various sites around the web. All of them failed or were impractical. One interesting possibility was to make multiple entries in the password database--one for each system a user might be coming from. But they all had to be added manually, using a command line program (openssl) that had you typing the username and password, and then confirming the password by typing it in again. But the big failing is that it wouldn't let users log in from just anywhere--which is sort of the point of web access.
That left me with one remaining solution: To enable SSL in Apache, and tell users to map to the drive using https -- because Windows apparently doesn't summarily attach the system domain name in that case. So I tried that:
- I found a pretty good set of directions for that process here: http://www.hss.caltech.edu/~sysadmin/public-cookbook.html. But they didn't happen to mention the need to create a security certificate.
I found an answer for that problem in the first section of an Apache FAQ, which told how to create a self-signed certificate: http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html
With that, I got everything operational, only to find that the server was now requiring the client to present an identification certificate of its own!
Back to the web, yet again. I found one web page that said to add the OPTIONS keyword in the Apache <directory> specification, but the person who wrote it must have done something else to solve the problem, because that solution certainly didn't work here.
So at this point, I've given up on making an Apache WebDAV server that Windows XP can map to. After recounting the horror story to my co-workers, one of them pointed to me to this page, which does a pretty good job of describing how, despite an initially robust implementation of WebDAV, Windows got off-track starting with Windows XP:
That page has a couple of additional workarounds, too. But they're mostly Windows-side workarounds. Apparently, you need several of them, because the same workaround won't work for all Windows systems! But if one of them really does work, I doubt that I'll ever know. I've already spent more time than I can afford.
Bottom line: I am forced to admit defeat. I hate that. It's depressing. WebDAV seemed like a great way to access a remote repository. But the DreamWeaver synchronization model leaves a lot to be desired, and Windows drive mapping just won't work.
And XMetaL? Well, XMetaL is just plain hosed. XMetaL depends on Windows drive mapping to make a WebDAV connection, so there is no way I can see for XMetaL to access a WebDAV repository that's served up by Apache (unless you have somehow managed to penetrate the mysteries of SSL). If that repository is Subversion, the Tortoise SVN client can be pressed into service. But otherwise, I have no idea how XMetaL users would be able to access a WebDAV-enabled CMS served by Apache.