Tuesday Jun 14, 2005

wx and You: your friend in managing workspace change

wx and You: your friend in managing workspace change

wx and You: your friend in managing workspace change

"Relieves OpenSolaris putback anxiety or your money back."

The following is a webpage I wrote to introduce Sun developers to a source code workspace management tool, wx, that I extensively modified after getting burned when using it to collapse a delta on a renamed file. Actually it was about a year after that incident that I realized, after talking to different folks at Sun, how wx could be modified to make my life easier (I had a hard time remembering all the development rules and I hate having to do things manually that can be automated). During my recovery from back surgery I started hacking on wx and discovered there was a lot that could be done to make the tool better. At the time I was not thinking that my version would be official so I called it wwx and let some of my Kerberos team members try it out. Within a month I started getting calls/e-mails from other Sun developers with thanks, suggestions and bug reports on wwx. Things snowballed from there and eventually my changes made it into the official version of wx.

Note that the contents below are oriented towards internal development in Sun using Teamware workspaces so some items may not be applicable to OpenSolaris development. It looks like we're moving to Mercurial controlled workspaces which are not compatible with wx. Instead one should use the Cadmium extensions to Mercurial. You can read more about this on the OpenSolaris site.

Contents

  1. About wx

  2. Examples

    1. How I normally use wx

    2. Getting a project gate/workspace ready for putback

  3. Advanced Usage

    1. How to skip particular pbchk/nits checks

    2. How to skip files in webrevs

    3. How to edit the active list (and deal with new files that haven't been "created")

    4. Environment Variables

    5. Tips for use with bugtraq

    6. How to return a file to its parent version

    7. How to initialize in a STC workspace (non-default src path)

    8. How to debug wx

  4. Credits

About wx

wx's reason for being is to help the developer keep track of file changes in a workspace and follow the Solaris source gate rules. Over the last several months I decided to enhance wx to be more functional and robust. As a result wx now does the following:

  • Keeps track of changed and new files in the active list.

  • Keeps track of renamed and deleted files in the renamed list.

  • Keeps track of comments associated with files in the active list.

  • Provides a wrapper around the underlying SCCS and workspace commands used to manipulate files in a workspace. This allows wx to update the active and renamed lists automatically and also do some checking to help the developer conform to the OpenSolaris gate rules.

  • The pbchk command does a number of new checks on active files and comments to make sure they conform to OpenSolaris rules. This includes looking for "sccs rmdel" which causes Teamware problems, too many delta's, files that are checked out that are not in the active list, poorly formed active list comments, active list comments that don't match those in the SCCS delta and whether the RTI is approved for the bugs listed in the active list comments.

  • The webrev command generates accurate webrevs that include renamed/delete info. A webrev is is a set of HTML code diffs that allow easy review with a browser.

  • The backup and restore commands allow easy backup and restore of files in the active and renamed list to a directory in your home directory which is usually backed up and thus safer than a workspace on a build system.

  • The putback/pb command allows the developer to do a putback with more safety and convenience.

  • The reedit command is enhanced to make it safer to use on files that have been renamed. Note, the reedit command is used to collapse deltas of files in the active list which is useful if the file was merged as a result of a resolve or anything else that could cause the file to be checked out and in more than once. Note, there is also a new “redelget” command that collapses file histories but leaves the file checked in. This is currently the only way to collapse new files.

  • Provides several informational commands so the developer can see what files are renamed, deleted, new, etc...

  • The mv and rm commands check for cyclic renames which cause problems for Teamware.

  • Provides a number of command aliases for ease of use.

  • See "wx help" for a list of all the commands, aliases and flags.

wx issues that the new wx fixes

  • Old "wx reedit" treats renamed files as new files, losing the file history in the process. New wx uses the Teamware nametable hash info to determine if a file has been renamed or is new. This is one reason new wx is slower than old wx but more accurate in this regard. And this is why both the parent workspace and local workspace nametables must be accurate. If wx thinks a file is new it will warn and ask the user if it is okay to proceed with the reedit (answer "no" if the file is not new).

  • Old "wx reedit" used the modification time of the SCCS delta files to determine if the parent file contains a delta that the child doesn't. This is risky because the child delta could be modified by a check in and the result would be that the reedit would lose the parent's delta code changes. New wx looks at the latest delta comment in the parent and determines if this exists in the child's delta history. If it doesn't then wx skips the reedit for that file and warns the user that a bringover is required.

  • Old "wx new" lists renamed files as new. New wx doesn't do that but it is slower as a result of more checking.

  • Not really a bug but old wx uses the paradigm of either editing the active list manually to add entries or using the update command to search all the directories to find files that are checked out (adding them to the active list). This means that the user must remember to update the active list after they check out files using "sccs edit". New wx can automatically update the active list when a file is being checked out with "wx co file" (edit/checkout/co are all aliases for the same command). This requires the developer to do fewer steps and the active list is more likely to stay in sync with the file changes in a workspace.

The best way to use new wx is to do all file manipulation in a workspace with the wx commands (do not use SCCS commands). Doing so consistently automatically updates the active, new and renamed lists and thus does not require that the update command be run in order to update these lists. Be aware that in order for wx to work properly the active and renamed file lists must be accurate. So if you do use SCCS commands or workspace filemv or filerm then you will need to use the 'wx update' command. Note, if you decide to remove a newly created file which is in your active list from your workspace, use 'wx rm file' to remove the file so the active list is updated accurately.

New wx now keeps track of file renames and deletes in a renamed list. This is separate from the active list since the active list only stores info on files that have been edited or are newly created. This separation allows wx commands like reedit to run only on active files and not files that have only been renamed/deleted. Note, if a file in the active list is renamed it will also appear in the renamed list.

New wx assumes that the parent workspace associated with the current child workspace contains the same set of files (except for new files) that the child workspace contains. Be careful if you change the parent of the child workspace since wx will assume that if it cannot locate a file in the parent then the local file is new. If this assumption is wrong, wx can output erroneous information or in the case of the reedit command, delete the SCCS history. Use "wx new" to see which files wx thinks are new.

Examples

Here's how I normally use wx:

  1. Create local workspace then do "ws workspace" and bringover files. Note, you do not have to do "ws workspace" in order to use wx in a workspace. Instead, just cd to a workspace and start using wx (note, wx will complain if you ws'ed to one workspace and cd to another and then try to use wx).

  2. "wx init". Initialize wx in a workspace (creates the wx subdir and some files for tracking changes). If I haven't checked out any files I use the "no update" option which creates a empty active list (fast). If you have modified files you should use one of the other updated options listed. Note, you can keep an active list sorted by default or sort it manually by using the "wx update -q -s" command.

  3. "wx co file_to_modify ...". (Note, co is an alias for checkout/edit. This does a "sccs edit file_to_modify ..." and adds an entry to the active list for each file on the command line. For example I would "cd usr/src" then "wx co Makefile" to check out the Makefile for editing which would also put an entry for usr/src/Makefile in my active list.) Use "wx list" or "wx active" to list files in the active list.

  4. "wx create new_file ...". New files need to be "created" to be recognized by the Teamware tools. This command starts a new file history for the new file and adds a entry in the active list. Use "wx new" to list new files in the active list.

  5. "wx rm file_to_delete ...". Use this if you need to remove a file in the OpenSolaris approved way. wx will actually move the file from usr/ to deleted_files/ and add an entry in the renamed list. Note, for new files the files are not renamed and wx will ask if you want to remove the files completely from the workspace. Typically you would answer yes but make sure you have a backup of the files just in case you need them. The renamed list will not be updated in this case but entries in the active list will be removed. Use "wx renamed" to list renamed files.

  6. "wx mv file new_file". Use this if you need to rename file to new_file. Will add an entry in the renamed list. The active list will be updated with the new name if file is in the active list. Use "wx renamed" to list renamed files. See "wx help" for more info.

  7. "wx bu -t" (bu is alias for backup, -t only backup if needed). Use this periodically to backup up files that wx are tracking. The files are backed up in the wx.backup subdir in your home directory which is useful since most build systems don't backup workspaces unlike your home directory. Note, "wx backup -i" provides info about the backup dir and files. There are other flags to control compression. See "wx help" for more info.

  8. When I have checked out and modified all the files required for the bug fix I build and test. If the testing looks good I then do:

  9. "wx bu -t". Just to be safe.

  10. "wx ci -c comment_file". Check in all active files. The check in comments will be taken from the active list comments which will be replaced by the comments in the comment_file. If you do this you can skip editing the active list comments manually.

  11. "wx webrev". Generates HTML based code diffs under the webrev/ directory at top of workspace. Have code reviewed using a web browser.

  12. Once the code review is done and all changes have been made I file an RTI using the web RTI page. XXX Note, for OpenSolaris I am not sure how an RTI is filed but I will add a pointer to that info when I find it.

  13. Login to the putback gate system, "ws /net/path_to_your_workspace", and reparent your workspace if necessary (See "man workspace").

  14. "wx pbchk". Check files and comments for OpenSolaris gate rules conformance. Note, nits runs a subset of pbchk checks and is more useful for checking your files while you've got them checked out.

  15. If "wx pbchk" complains about multiple SCCS deltas/file versions, use "wx redelget -m" to collapse the deltas so there is only one version difference between the local file and the parent. This will also reset the file history for new files to 1.1. Note, reci is an alias for redelget.

  16. "wx pb -n". Do a trial putback and check for conflicts. If there are conflicts, run "bringover files_in_conflict then "wx resolve" to resolve the conflicts and collapse the deltas created by the merge ("wx resolve" uses the reedit subcommand to collapse the SCCS deltas). Note, pb is an alias for putback. You can see the list of files that wx provides to the putback command by doing "wx pblist". "wx pbcom" will display the putback comments. If you resolved conflicts then repeat steps 10-15

  17. "wx pb". Do the real putback. Note, wx will do some checking and will prompt the user before actually doing the putback. It will pass the comments found in the active list to the putback command and will display the putback comments before the prompt to do the putback. You can also use "wx pbcom" to see what the putback comments will be.

  18. After the putback is done, if I intend to do more bug fix work in this workspace I'll save the current wx directory (either tar it or rename it), remove the current wx directory if I didn't rename it and then do "wx init" to initialize wx with fresh state. That way there is less chance of getting confused about the changes made for the new bug fix.

Getting a project gate/workspace ready for putback to the OpenSolaris gate:

    Note, the following describes the model that I use when working on a project that will require a significant amount of time and code change. In order to reduce the pain involved, my model uses a project source gate created by copying the source from the official OpenSolaris gate. Development occurs in workspaces that are children of this project gate with intermediate putbacks from this child workspaces going back to the project gate. This allows the project to use it's own rules regarding number of deltas in a putback and other issues without regard to the official OpenSolaris gate rules. When the project is done and ready to be putback to the Open Solaris gate these are the steps I use (featuring wx):

    First, a caveat about using wx in a project gate. Never use "wx reedit" or redelget (or any equivalent alias that collapses a SCCS delta) in a project gate that is the parent of developer workspaces. This will confuse the Teamware commands like putback and bringover when used between a child workspace and the project gate. Generally, it is best to avoid the wx reedit/redelget commands until it is time to putback to the OpenSolaris gate.

  1. Create a local copy of the official OpenSolaris gate (I'll call it opensolaris_copy in this example). The parent of opensolaris_copy should be the official OpenSolaris gate.

  2. "putback usr/src delete_files" from the project gate to opensolaris_copy. This leaves the project gate intact and any modification (collapsing) of file SCCS delta histories for the real OpenSolaris putback will be done in opensolaris_copy. Note, if the putback reports conflicts these will need to be resolved in the project gate. The resolve will change files in the project gate but child workspaces will be able to properly bringover those changes. Backup the project gate before doing a resolve. Again, do NOT use wx reedit/redelget in a project workspace if you want bringover to work in child workspaces.

  3. "ws opensolaris_copy". Set the local ON clone copy as the current workspace.

  4. "wx init -ft". Initialize the opensolaris_copy workspace using a thorough wx update. This will create the active and renamed lists with all the files that were changed in the project gate.

  5. "wx list". Make sure active list looks okay.

  6. "wx new". Make sure new list looks okay.

  7. "wx renamed". Make sure renamed list looks okay.

  8. "wx ea". Update the active list comments. The comments will be placed in the SCCS delta history for files in the active list when the "wx redelget" step is done below and will also be used as the putback comment.

  9. "wx redelget". Collapse all the active list files SCCS delta history so there will only be one SCCS delta for each file in the active list when they are checked in. Note, recheckin and reci are aliases for redelget.

  10. Do a nightly build in the opensolaris_copy gate. If any files need to be modified, do the modification in the project gate and putback the changes to the local copy. Do a "wx deltachk" to see if there are any delta problems.

  11. Follow steps 13-16 from the "How I normally use wx" list of wx steps. to putback the files from the opensolaris_copy gate to the official OpenSolaris gate.

Advanced Usage

How to skip particular pbchk/nits checks

Some of the checks that the pbchk and nits commands run will skip active files listed in a file called wx/command.NOT where command is the name of the wx command doing the particular check. For example if I want to skip cstyle checking of all my active list files when I run "wx pbchk" I would first do "wx list >wx/cstyle.NOT". Note, I only skip checks for which I know I have an exemption from the gatekeepers. For example for some open source code in Solaris there are exemptions from the cstyle rules. Similarly for files containing non-Sun copyrights one may want to list files that the user is sure there is not a copyright problem in "wx/copyright.NOT".

How to skip file in webrevs

List files (one per line) in wx/webrev.NOT that you don't want to include in webrevs generated by "wx webrev".

How to edit the active list (and deal with new files that haven't been "created")

The original wx required the user to edit the active list to associate comments with the active files (note, wx checkin now provides a -c comment_file option which will automatically update the comments in the active list as well as use them for the SCCS check-in delta). Also, some people like to manually add new file entries to the active list before they were 'sccs created' so they could use wx for backups and nits checking, etc... The format of an entry in the active list is:

[filepath]
#empty line
[one or more comment lines]
#empty line

where [filepath] is the file path of an existing or new file relative to the top of the workspace. Here's an example entry with three comment lines (note, each entry starts with the filepath, there is not a empty line before in the example below):


usr/src/cmd/krb5/klist/klist.c

4772119 Enabling Kerberos's Triple DES/SHA-1 Enctype Support
PSARC/2002/178 Enabling Kerberos's Triple DES/SHA-1 Enctype Support
4705662 GSS/Kerberos clients requesting dec-cbc-crc before des-cbc-md5


Note, if you are adding new files to the active list you should run "wx new -t" to populate the new list. New files that haven't been created will be created when "wx delget" is run (note checkin and ci are aliases for delget).

Environment Variables

  • PUTBACK: specifies the command to do the putback. This is useful if you want to use something like Casper Dik's turbo-dir.flp scripts as in this example: "export PUTBACK='cm_env -g -o putback'"

  • WXDIFFCMD: specifies the diff command and args for the wx diffs commands like diffs and pdiffs. This is similar to the CDIFFCMD and UDIFFCMD environment variables that webrev uses. A good setting is: "export WXDIFFCMD='diff -bw -U 5'"

  • WXWEBREV: specifies the webrev command and args used by the "wx webrev" command.

  • WXDIR: specifies the directory where wx will keep it's state files. The default location is in wx/ at the top of the workspace however via WXDIR you can change this to point to a different path like /tmp/my_wx. This is useful when you want to run wx in a workspace where you don't have write permission.

Tips for use with bugtraq

  • For the suggested fix field in bugtraq use "wx pdiffs" output. If that's too large, save it in a file, add it as an attachment and mention this in the suggested fix note.

How to return a file to its parent version

To undo all changes in a file that you've checked out and set it to the current version in the parent workspace use: wx reset file
Note, this will bringover the file from the parent and will undo local file renames.

To undo all changes in a file and return it to the version when it was originally brought over from the parent do:

  1. wx reedit file

  2. wx unedit file

This does not bringover the file from the parent so the file contents will be that of the last bringover of that file. It also does not undo a file rename.

How to initialize in a STC workspace (non-default src path)

Since the source in a STC workspace doesn't use the wx default path of usr/src you'll need to initialize using "wx init usr/ontest" in order for it to find the source. Note, if the source you're working on isn't found under usr/src from the top of your workspace then you'll need to initialize using the common path starting from the top of your workspace.

How to debug wx

To turn on debug output in wx do "wx command -D command args". This will send debug info to stderr so to page the output use 2>&1.

Credits

Thanks go to Jeff Bonwick who wrote wx, David Robinson who had a great idea about using the nametable hashes, Brent Callaghan who wrote webrev, John Beck (code review), Anup Sekhar (testing), Glenn Barry, Arun Perinkolam, Wyllys Ingersoll, Chin-Long Shu, Alastair McDermott, Matt Simmons, Valerie Anne Bubb, Bill Sommerfeld, Heiner Steven, Nikolay Molchanov, Alan Burlison, Nico Williams, Ann-Marie Westgate, James Carlson, Jeff Smith, and last but not least my mother for pointing out that my caching methodology could be more generalized.


Technorati Tag:
Technorati Tag:



About

user12615206

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today