Wednesday Jan 03, 2007

Grenoble/Prague Trip Report

A long trip report.[Read More]

Thursday Dec 07, 2006

Have You Tried Mercurial?

I've been using Mercurial over the last few months trying to understand what it will take to transition from Teamware. The Open Solaris team did quite a bit of analysis on the available open source Source Code Management (SCM) choices.

Besides getting my silly comic posted, I just wanted to make sure that people understood the meaning of a Distributed Source Code Management (DSCM) as described in the Open Solaris DCM Requirements Document. The SCM needs of Open Solaris are very similar to the needs of OpenJDK. The critical concept with a DSCM is what the Open Solaris requirements document calls "Unbiased and disconnected distribution" or the ability to create completely separate repositories or source trees that can operate independently from one another with full SCM functionality. This is more than just a simple parent and child relationship. Any two repositories that have a common ancestor can exchange changes. With Teamware this was with revisions to individual SCCS s.\* files, one per managed file. With Mercurial it's changesets, which includes changes to multiple files.

In general many of the basic concepts between Teamware and Mercurial are the same, but it's different enough that we all will be on a learning curve for a while. Teamware has been used by the JDK developers for over 10 years. Speaking for myself, Mercurial is a nice system and as we get comfortable with it I think this will be a great DSCM solution for us.

So on to the comic... About a year ago we traveled along the northern coast of California and stopped at the "Trees of Mystery" on Highway 101. We didn't take the tour, but I couldn't help but take a picture. Using it now seemed appropriate as I play around with the 35,000 files in the primary source base of the JDK.



Wednesday Oct 11, 2006

Thunderbird, RSS Feeds, and Tracking Forums & Blogs

Thunderbird, RSS Feeds, and Tracking Forums & Blogs

Watch all the forums, read the blogs, deal with your email, what a pain. I'm always looking for a better way to track these things, and the best so far appears to be using the latest Thunderbird and RSS feeds. Maybe this is old news to you, in which case, ignore the cranky old man behind the mail settings.

If you have the latest Thunderbird, then you can do this:

  1. If you need to use proxies, go to Thunderbird Preferences, select 'General', then use the "Connection Settings". I had some problems with the default values put into "No proxy For:", so you may want to be careful what gets put in that field, I'd recommend you check your browser settings and duplicate those.
  2. Create a new account for RSS feeds.
  3. Select your new account and go to 'Manage Subscriptions'.
  4. Click on Add, and pull this window up in the corner so you can easily paste in an RSS feed URL when you get one.
  5. Start up your browser and find an RSS feed URL, copy it.
  6. Paste the RSS feed URL into the Thunderbird RSS Subscriptions "Feed URL" and hit OK.
  7. Repeat the 'Add' process for any RSS feeds you want.

Sometimes getting the RSS feed can be tricky, and it seems that every forum is slightly different. The RSS feed URLs I'm using are:

  • Planet JDK (blogs):
  • Some of the JDK forums I watch:
  • Some of the forums I watch (these were tricky to get right):
  • Some of the forums on my projects:

Once you get the pattern down for any site, all you need is the forumID which is seen in the URL for the main site. I've found that using Thunderbird this way has made keeping track of these forums, and blogs much easier.

Hope this helps someone. have fun!


Saturday Aug 20, 2005

Adventures with MKS and CYGWIN on Windows

Adventures with MKS and CYGWIN on Windows

Using unix toolkits on Windows machines can be a frustrating experience. I just thought I note a few items I've discovered trying to make things work with either kit.

The JDK uses makefiles and shell scripts in it's build process and has historically used MKS on Windows. Recently the use of CYGWIN has become important and we have been trying to make the choice of Windows unix utilities an option and not force the adoption of either. We may find it difficult to police that both always work but our intention is to allow for a variety of developers. Getting shell scripts and makefiles to work with both has been interesting.

In addition, on Windows we have been trying to make the builds a bit smarter by having it detect or find various components installed on the system with fewer interventions on the user's (builder actually) part, e.g. automatically finding the shorter windows paths to things instead of insisting on the user providing the shorter paths or changing the default install locations of components.

  • PATHNAMES: Try and stick with the windows style of directory names, but using a / instead of a \\ character as the file separator character. With few exceptions, most windows applications seem to accept the C:/ style and it makes life easiest if you can avoid the \\ characters.

  • NMAKE: One exception to the C:/ rule is NMAKE, anything passed into NMAKE needs to be transformed into the C:\\ style.

  • MKS 8.7: Try and use MKS 8.7 or newer. It has a utility dosname that seems to work well, without dosname, or a reliable one, things will be difficult.

  • 64BIT: Supposedly you need at least MKS 8.1 for 64bit Windows, but I've heard reports that even MKS 6.1 can work on 64bit Windows. I've used 8.7. I recently tried to install CYGWIN on a 64bit Windows machine and finally gave up. A co-worker infoirmed me that if the packages are downloaded on a 32bit machine, then copied over to the 64bit machine and installed, it should work. Something is wrong with the CYGWIN install process on Windows 64bit.

  • SPACES: With MKS try and use dosname, and in particular use 'dosname -s'. The dosname utility helps considerably in dealing with changing directory names with spaces to their non-space equivalents (so called short names) and also changing all the backslash characters (\\) to slash (/), which is essential inside makefiles and shell scripts. With CYGWIN use cygpath, and in particular 'cygpath -s -m' to get the short names in the 'mixed' mode, e.g. the "C:/" style of pathname. It's easy in GNU makefiles to check to see if a directory has spaces, just use:

    ifneq ($(DIR),$(word 1,$(DIR)))
      # DIR contains spaces
  • DOSNAME and CYGPATH: Want directory names and filenames that exist. If the entry doesn't exist you will get undefined results. Also, if the pathname contains spaces, you will need to quote it.

  • MIXED: You want the mixed mode of paths because many executables will not understand the cygwin style of directory names, e.g. /usr/bin, or "/cygdrive/c/Program Files". The MKS dosname returns mixed mode, and the 'cygpath -m' will too.

  • BACKSLASH: Inside a shell script or in a makefile, deal with the \\ -> / transformation first. If something processes your pathnames in a unix way before you deal with this, you will end up with no file separators at all, they will all assumed to be character escapes.

  • DOUBLE SLASH: In the process of transforming pathnames, you sometimes end up with double slashes, make sure you process these out, sometimes they can cause certain utilities problems or make them behave differently, like 'jar'.

  • DRIVES: Watch out for the difference between "C:" and "C:/", they are different. With windows it's usually wiser to always end directory names with /, but inside makefiles that can be a problem, causing the DOUBLE SLASH problem. In general you just need to be careful in Makefiles which variables contain a prefix path (ending with a /) vs. a pathname used like $(DIR)/foobar.c.

  • UNAME: CYGWIN's uname returns "CYGWIN_NT-5.0", or something similar, effectively the version of CYGWIN, not really the version of the system or machine OS. The MKS uname returns something like "Windows_NT". In general CYGWIN is it's own system, and MKS is just a set of Windows utilities.

  • UMASK: MKS seems to have problems with file/directory permissions, usually leaving permissions wide open or ill-defined. CYGWIN does better.

  • Administrator: MKS has had some problems being used by anyone other than the one who installed it, I'm not sure what this problem is. CYGWIN may or may not have this problem.

  • PATH: MKS PATH is the same as Windows PATH, using the ';' path separator character, and must be quoted if it has any embedded spaces. It can contain a mix of \\ and / file separator characters apparently. With CYGWIN PATH is unix style, using : path separator characters and NOT using any C: path names, all pathnames need to be in the cygwin style like /cygdrive/c.

  • ROOTDIR: If MKS is installed on the system, the environment variable ROOTDIR will refer to the installation location. I'm not sure if CYGWIN has any way to know where the base install location is.

  • SHELL: To detect if the shell you are in there are various ways. I usually assume that the MKS or CYGWIN basic utitities are in your search path, so I assume they have fired up a MKS or CYGWIN shell of some kind. This seems to make sense so far. One way is to assume the proper 'uname' is in the search path and use it:

        if [ "$(uname -a | fgrep Cygwin)" = "" -a -d "${ROOTDIR}" ] ; then
            mkshome=$(dosname -s "${ROOTDIR}")
            fullpathprocessor="${mkshome}/mksnt/dosname -s"
        elif [ "$(uname -a | fgrep Cygwin)" != "" -a -f cygpath ] ; then
            fullpathprocessor="cygpath -a -m -s"
            echo "WARNING: Cannot figure out if this is MKS or CYGWIN"
    Notice that I added the 'cygpath -a' option to get an absolute path. The MKS dosname always returns an absiolute path, so adding the -a option to cygpath makes it do the same thing. Inside a makefile (GNU make) you might:
    ifeq ($(shell uname), Windows_NT)
      PLATFORM = windows
      OS_VERSION := $(shell uname -r)
    ifneq (,$(findstring CYGWIN,$(shell uname)))
      PLATFORM = windows
      OS_VERSION := 5
      USING_CYGWIN = true
      export USING_CYGWIN
  • MAKEFILES: The trick to keeping your makefiles sane is to try and stick with the mixed mode paths as much as possible, and isolate the pathname transformation to a make macro:

    CYGPATH_CMD=cygpath -a -s -m
    define FullPath
    $(subst //,/,$(shell $(CYGPATH_CMD) "$1"))
    else # MKS
    DOSNAME_CMD:=dosname -s
    define FullPath
    $(subst //,/,$(shell $(DOSNAME_CMD) "$1"))
    Note that there is no leading spaces or indenting above. The define macros replicate all spaces, so be careful with these things. Also, note that we add quotes around what we pass dosname or cygpath. To use the above macro you would:
    ABS_DIR:=$(call FullPath,"$(subst \\,/,$(DIR))")
    Note that the first thing I do is a substitute of \\ with /, then I quote that result and send it to FullPath.
  • PREFIX PATHS: If you need a setting that is a prefix path (ends with /) that could also be empty, I created this macro:

    define PrefixPath
    $(if $1,$(subst //,/,$1/),)
    Which also takes care of the // characters, or the fact that what it was given already had a trailing /.
  • EXISTS: Getting dynamic in makefiles is a bit tricky but using the $(shell) you can just about do anything. Here is a macro that picks the first directory that exists or a default value:

    define DirExists
    $(shell \\
      if [ -d "$1" ]; then  \\
        echo "$1"; \\
      elif [ -d "$2" ]; then \\
        echo "$2"; \\
      else \\
        echo "$3"; \\
  • ENVIRONMENT VARIABLES: It appears that CYGWIN inherits Windows system variables with their names mapped to all uppercase, so things like windir, SystemDrive, SystemRoot, and MSDEVDIR, turn into WINDIR, SYSTEMDRIVE, SYSTEMROOT, and MSDEVDIR. I don't know why. The MKS shells don't seem to have this behavior. Makefiles and scripts may need to check for both names.

  • VCVARS32.BAT: For some reason, it appears that VC6 (Visual Studio 98?) somehow managed to setup a system variable MSDEVDir during installation, regardless of using vcvars32.bat. So after VC7 (Visual Studio .NET 2003) is installed, VC71COMNTOOLS is in the system variables now, but MSDEVDir still refers to the old MSDEV98, at least until the VC7 vcvars32.bat script is run, then MSDEVDir refers to the new .NET 2003 version. So be careful.

Hope this information was helpful, it was rather painful to learn, maybe by posting this I've saved someone some time. Many thanks to my Sun co-workers, especially Martin, Vijay, Tim, Graham, Jim, and others for the input and hints they have provided.

If there are mistakes or errors in the above, please post a comment. Or if you have more to add, please add a comment.

Thursday Jul 15, 2004

The Importance of Internal Productivity Tools

Critical to all engineering efforts is that someone makes sure the teams are productive in their day to day activities, this includes internal infrastructure tools. Anything that helps get the job done and/or improves the quality of life for engineers.

Various blogs on JDK development procedures, including building, build infrastructure, testing, and source maintenance.


« June 2016