Tuesday Mar 10, 2009

hg merge results in a need for hg commit, which bombs because of uncommitted changes

So I had a workspace I needed to get integrated back to the ON gate. I last worked on it about Feb 14th and I couldn't remember which machine, let alone which workspace it was located in. I found one that looked right and started the merge off:

[th199096@aus-build-x86 smurf]> hg pull -u
pulling from ssh://onnv.eng//export/onnv-clone
searching for changes
adding changesets
adding manifests
adding file changes
added 646 changesets with 5986 changes to 4868 files (+1 heads)
not updating, since new heads added
(run 'hg heads' to see heads, 'hg merge' to merge)
[th199096@aus-build-x86 smurf]> hg merge
abort: outstanding uncommitted changes

I've seen this before, it normally means my workspace has been corrupted. I can at least recover to get the original version of the changes:

[th199096@aus-build-x86 smurf]> hg rollback
rolling back last transaction
[th199096@aus-build-x86 smurf]> hg commit
nothing changed
[th199096@aus-build-x86 smurf]> hg status
! usr/src/pkgdefs/SUNWmptsas/copyright
! usr/src/pkgdefs/SUNWmptsas/depend

But I don't see what I never committed! Try again:

[th199096@aus-build-x86 smurf]> hg merge
abort: there is nothing to merge
[th199096@aus-build-x86 smurf]> hg pull -u
pulling from ssh://onnv.eng//export/onnv-clone
searching for changes
adding changesets
adding manifests
adding file changes
added 646 changesets with 5986 changes to 4868 files (+1 heads)
not updating, since new heads added
(run 'hg heads' to see heads, 'hg merge' to merge)
[th199096@aus-build-x86 smurf]> hg merge
abort: outstanding uncommitted changes
[th199096@aus-build-x86 smurf]> hg rollback
rolling back last transaction
[th199096@aus-build-x86 smurf]> hg list
modified:
        usr/src/uts/common/fs/nfs/nfs_server.c

Okay, that looks right for what I need to modify. I then proceeded to create a new workspace into which I was going to copy over the file. Once I did that, I did a diff and realized that the reason I was getting an error was that this set of changes had already been integrated into the ON gate!


Originally posted on Kool Aid Served Daily
Copyright (C) 2009, Kool Aid Served Daily

Sunday Feb 22, 2009

Setting up a Mercurial gate/clone at Cthon 09

So one of the minor things which went wrong for our booth was I forgot that we needed to integrate code here at Connectathon 2009. We had a gate (thanks to Rick Mesta) working, but the new Mercurial gate had been put in place since our last event (a Fall Bakeathon) and we just forgot about it. I.e., we did have a Mercurial gate in place, but not one that mimicked the ON gate. We did a couple of integrations the old way (mainly need to remember 'hg push -u'), but there is additional sanity checking which needed to go on.

So I dragged out my notes from Setting up a Development Project Gate and went to work. I cheated in that I already had most of the configuration changes I had made. But I still had to get the system running.

One thing I have to say about my colleague Rick is that he was very thorough in setting up our build machine. But he did things differently than I would have done them. I'm not saying he did them wrong, just differently. He also did more than I thought I asked him to do for the setup. For example, he set the build server up to be a NIS server - which actually saved us several hours when we started to use the machine.

Anyway, I get the gate up and running without any major modifications to the instructions. And it is easy for me to get a clone of the gate either via ssh or local file access. But for the life of me, I can't get the gate to respond to 'hd pbchk':

[th199096@build nfs]> vi bootparam_xdr.c
[th199096@build nfs]> hg reparent ssh://nfs4hg@build//ws/nfs41-gate
[th199096@build nfs]> hg commit
[th199096@build nfs]> hg pbchk
Warning: Parent workspace ssh://nfs4hg@build//ws/nfs41-gate is not accessible
active list will be incomplete

A little backstory - I said Rick configured the machine to be a NIS server and he also configured the gate to be accessible over NFS via the automounter as /ws/nfs41-gate. I had taken a clone of that and changed the automounter mount points to be:

[th199096@build ~]> ypcat -k auto.ws | grep nfs41
nfs41-clone build:/export/gate/nfs41-clone
nfs41-orig build:/export/gate/nfs41-gate
nfs41-gate build:/export/gate/nfs41-gate

But in the Austin lab, we don't export our gate, the only access is via ssh. And that means I use the fully qualified path to the zfs filesystem that has the gate. In this case, I configured everything to reference /export/gate/foo. And since I could get a clone of nfs41-clone, I thought that was sufficient.

This hosed me and it wasn't until I tried the following:

[th199096@build nfs41-clone]> hg reparent ssh://nfs4hg@build//export/gate/nfs41-gate
[th199096@build nfs41-clone]> hg pbchk
CDDL block check:
Error: Invalid line in CDDL block in file usr/src/uts/common/fs/nfs/bootparam_xdr.c

that things started to work. The issue here is that there are no restrictions on getting things out of the read-only clone. But to access the gate, there are restrictions in that you have to be the 'nfs4hg' user and the paths in the config files have to match those passed in by the script that runs when the ssh command is issued.

There are two ways to fix this problem - either change the paths in the config files or teach everyone to use the '/export/gate/nfs41-gate' path. Each has its own advantage, if it were just me, I'd use the full path because it removes a dependency on the automounter. But for this event, everyone is already trained to using the other path.

The other issue here is that as the build machine is also the NIS server, we can not easily have a local account for 'nfs4hg'. With a development gate and a level of personal responsibility (i.e., you know who it is who will be ragging you if you make a mistake), I'm willing to live with that. I'm not sure I could make that same call with the ON gate.


Originally posted on Kool Aid Served Daily
Copyright (C) 2009, Kool Aid Served Daily

Tuesday Dec 02, 2008

Setting up some backup clone gates

One of my colleagues had a gate wiped on a disk crash (using ZFS, but only 1 disk). That got people wondering about our backup strategy. We have distributed developers, so we've banked on them having private copies. Oh, and we also push directly to OpenSolaris.

So now I'm looking at adding clones in two remote sites - on machines I don't own. And by that, I mean that I can't create a local gatekeeper user account. Which means that I'll have ssh trust issues. I.e., this will not work:

changegroup.2 = /usr/bin/hg push -R /pool/ws/nfs41-gate ssh://cistern.central//export/ws/th199096/nfs41-clone

I could have a cron job pull changes, but besides being gross, there is a slight chance it will not grab something. No, I want the push to the gate to still copy to the clones.

The answer is to mimic Dave Marker's code for updaing OpenSolaris (it is updateoso.py in the onnv-gk-tools repository seen at ssh://anon@hg.opensolaris.org/hg/scm-migration/onnv-gk-tools). Before I do that though, there is the issue that I'll need local copies of that repository as well. I could use the one that is in use for onnv, but then again, if I mess things up, Dave might recall that Oklahoma has a no helmet law and come over to visit. Much safer to create my own copy.

The biggest issue was making sure to add the new key to the two machines. I've got this set up and running.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Thursday Nov 20, 2008

Locking down a gate/respository

So I'm not a mean gatekeeper - I try to not lock down our gate. I feel like I should just be able to ask people not to integrate anything. But the reality is that you can't count on everyone getting the message. So, you need to lock down the gate.

I knew how to do that in Teamware, but I've never had that need with Mercurial. Until today that is - a nasty branch merge with onnv_103.

So to lock down a Mercurial gate with Sun's extensions, you can use lock.py:

[nfs4hg@aus1500-home ~]> which lock.py
/pool/nfs4hg/bin/lock.py
[nfs4hg@aus1500-home nfs41-gate]> lock.py -n -R /pool/ws/nfs41-gate
[write]: None

Okay, no one has a write lock, so let's grab one:

[nfs4hg@aus1500-home nfs41-gate]> lock.py -R /pool/ws/nfs41-gate
[nfs4hg@aus1500-home nfs41-gate]> lock.py -n -R /pool/ws/nfs41-gate
[write]:
        nfs4hg
        th199096

By the way, where is this configured?

[nfs4hg@aus1500-home ~]> grep lock /pool/ws/nfs41-gate/.hg/hgrc 
#   5. lockdir must be readable by whomever will pull/push
lockdir = public/lock
wlock = nfs4hg, th199096
# These hooks check the lock before anything happens.
prechangegroup.0 = python:hook.lockchk.lockchk
# then comment it out, let them push, uncomment, and unlock the gate.

So the lock only works on the gate and not the clone. You can find more about this in the source of lock.py.

Ohh, and even though I haven't tested it, you release the gate easily enough with a unlock.py.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Thursday Oct 09, 2008

A better fix for that non-responsive tool repository

I ran into another machine which got stuck on that 'hg style' issue I reported in Getting around a tool repository which is not updating and I got mad enough to start looking at it. We've still got path dependencies in BFU to stuff inside SWAN and I was hoping we could get away from it with the new tools.

I started trying to figure out where the Python script was that was being run. I wanted to find the hard-coded reference to /ws/onnv-tools/onbld/etc/hgstyle. Knowing that I had just seen a Flag Day on this, I looked in Flag day: new default output style for Mercurial. And the solution was staring right at me!

Mark had coded it correctly, no path hardcoding! I could simply change:

	[ui]
	username=Mark J. Nelson 
	style=/ws/onnv-tools/onbld/etc/hgstyle

to be:

	[ui]
	username=Mark J. Nelson 
	style=/opt/onbld/etc/hgstyle

Sweet, great to have my faith restored and an effective solution.

But now I need to really fix the automount maps in the lab, we keep on tripping over this issue with a stale repository and we have a working one we should be using.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Wednesday Oct 08, 2008

Branch merge to a specific tagged revision

I need to do a branch merge between nfs41-gate and onnv-clone. And specifically, I want to not get the 'tip', but rather the tag for release 100. I found a good reference - Chapter 8 Managing releases and branchy development.

So I'll follow along with it. I need the tag:

[th199096@jhereg onnv-play]> hg tags
tip                             7782:716c23b2ce2e
onnv_100                        7757:bf4a45ecb669
onnv_99                         7613:e49de7ec7617
onnv_98                         7473:fad192e9bc57

It turns out I don't need much more:

[th199096@jhereg nfs41-100]> hg reparent ssh://onnv.eng//export/onnv-clone
[th199096@jhereg nfs41-100]> hg tags | more
tip                             7744:763bfa203d1a
closedv1                        7742:9fab48a31a4a
onnv_99                         7652:e49de7ec7617

So I haven't merged yet:

[th199096@jhereg nfs41-100]> hg pull -u -r onnv_100
pulling from ssh://onnv.eng//export/onnv-clone
searching for changes
adding changesets
adding manifests
adding file changes
added 64 changesets with 475 changes to 462 files (+1 heads)
not updating, since new heads added
(run 'hg heads' to see heads, 'hg merge' to merge)

The tag can be used as a revision!


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Monday Oct 06, 2008

Pushing your open gate to OpenSolaris

In Setting up a Development Project Gate, I showed you the steps I took to setup a shared development gate. The only thing I left out was the automatic push to a Mercurial repository on OpenSolaris. I'll take you through these steps now.

By the way, thanks to Dave Marker for sharing how ONNV does this and writing some Python tools to automate the majority of the push.

Get a repository

First you need to get a project leader for your project to create a workspace for you on hg.opensolaris.org. They can do this by logging to OpenSolaris and going to the project page. Then they select 'SCM Management'. Next 'Add Repository' and remember to click the Mercurial radio button on the next screen. The page is pretty explanatory and the only thing to remember is that it might spew up a message about the project already existing. It also seems to take 5-10 minutes for the project space to show up.

Which if you know about it is okay. You can take this time to configure who has commit rights. (You'll need one account for sure, more on that later.)

As evidenced in Setting up a NFS41 gate, I had a hard time figuring out where the gate was located. It can be reached as:

hg clone ssh://anon@hg.opensolaris.org/hg/<PROJECT>/<GATE-NAME>

Note you need to configure anonymous access on the repository page for this to work. I'm not sure, but I believe this also implies anyone can write back to the gate. Tie this in with a lack of tools to maintain the gate out there (including deletion) and you can see this might be a problem.

Configure a SSH public key

This part is the scary part - scary in that you have to decide if you want automatic pushes or manual pushes.

Manual pushes

If you want manual pushes, then you need to make sure you have a SSH public key setup as described in opensolaris.org SSH key help. Then whenever you want to push a change, you would do:

hg push -R <your gate> -e "ssh -q -F ssh://<YOUR OSOL USERNAME>@hg.opensolaris.org/hg/<PROJECT>/<GATE-NAME>"

You don't have to read much more of the rest of this entry, except perhaps to make sure your proxy host is correct.

Automatic pushes

If you want automatic pushes, then you need to configure a special key pair without a passphrase. Note that this is a risk you need to manage yourself. It needs to be blank because you don't want to even store the passphrase anywhere. I've done this type of thing in the past for production systems and the trick is that you want to lock down the box you have stored the private version of the key

Follow the directions as described in opensolaris.org SSH key help to publish this key. Note that no-one will know that this passphrase is empty for this key. They still need the private copy of the key. We have to make sure that is locked down tight!

  • Use a system with a non-standard root password.
  • Strictly control admin access to the system
    • Don't hand out the root password.
    • Don't make sudo too permissive.
  • Pick a non-global account to store the data in.
  • Make sure the permissions are 700 on the directory where the key is stored.
  • If you share that homedir, make sure to read The myth of security with AUTH_SYS, which means:
    • Do not accept the default access list ooptions.
    • Set root= or use anon=-1.
    • Set the rw= and/or ro= hosts appropriately.
    • Strongly consider kerberizing the share.

There is probably more you can do, so the first thing is to accept responsibility for this setup.

Okay, having scared you, it is now time to configure this special ssh configuration:

[nfs4hg@aus1500-home ~/tmp]> mkdir opensolaris
[nfs4hg@aus1500-home ~/tmp]> chmod 700 opensolaris/
[nfs4hg@aus1500-home ~/tmp]> cp ~/opensolaris/\* .
[nfs4hg@aus1500-home ~/tmp]> more config 
Host \*.org
        GlobalKnownHostsFile /pool/nfs4hg/opensolaris/known_hosts
        ProxyCommand /usr/lib/ssh/ssh-socks5-proxy-connect -h 192.18.43.19 %h %p
        IdentityFile /pool/nfs4hg/opensolaris/id_dsa
        User <YOUR OSOL USERNAME>

You'll have to figure out whether or not you need the proxy configuration or not. And you will need to fix-up the paths for the GlobalKnownHostsFile and IdentityFile.

Also note that I've got this account already setup with a ~/.ssh directory to allow integrations to the gate. It doen't have DS keys, but I like to keep these apart. And note that id_dsa and id_dsa.txt are the keypair that you generated with the empty passphrase.

Edit hook/updateoso.py

In the copy of hook/updateoso.py I got from ssh://anon@hg.opensolaris.org/hg/scm-migration/onnv-gk-tools, I had the following minor diffs:

[nfs4hg@aus1500-home ~]> diff /pool/onnv-gk-tools/hook/updateoso.py onnv-gk-tools/hook/updateoso.py
33c33
< OSOREPO = "ssh://hg.opensolaris.org/hg/nfsv41/nfs41-gate"
---
> OSOREPO = "ssh://hg.opensolaris.org/hg/onnv/onnv-gate"
43c43
< This script must be run as user "nfs4hg".
---
> This script must be run as user "onhg".
76c76
<     home = pwd.getpwnam("nfs4hg")[5]
---
>     home = pwd.getpwnam("onhg")[5]
95c95
<     if utility.check_user("nfs4hg", m) is False:
---
>     if utility.check_user("onhg", m) is False:
97c97
<         m.write('''Can't update OpenSolaris. User != "nfs4hg"\\n''')
---
>         m.write('''Can't update OpenSolaris. User != "onhg"\\n''')

Note that I grabbed this script right after Dave Marker put it back, so I got a cutting edge version of it. I know he is about to change it such that the user and osol path are configurable variables outside of this file. I.e., you would make changes in a project specific configuration file.

Anyway, once those changes are made, go a head and make sure this line is enabled in your official clone's hgrc:

# These hooks are run from bghook() in the background
bg-changegroup.0 = python:hook.updateoso.updateoso

I'm assuming you have modeled your development gate ala onnv (and as I described in Setting up a Development Project Gate). If you haven't, then I think all you need to do is add this as the last bg.changeroup entry in your gate's hgrc.

And now you are good to go!

You might want to do the manual push I described above to seed the gate.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Friday Oct 03, 2008

Hey, the source browser is up and running

Check out nfsv41/nfs41-gate.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

First successful push to nfs41-gate on opensolaris

We had the first developer make a real push to the nfs41-gate on the OpenSolaris NFSv41 Project Repository.

Jim Wahlig had this to push:

[thud@adept nfs41-gate]> hg incoming
comparing with ssh://anon@hg.opensolaris.org/hg/nfsv41/nfs41-gate
searching for changes
changeset:   7743:c672b1cb86be
user:        Thomas Haynes 
date:        Thu Oct 02 22:28:30 2008 -0500
summary:     Added tag closedv1 for changeset 9fab48a31a4a

changeset:   7744:763bfa203d1a
tag:         tip
user:        jwahlig@aus-build3
date:        Fri Oct 03 11:52:59 2008 -0500
summary:     fix stable storage on x86.

The only issue we encountered was that mail did not get accepted for the nfs41-discuss mailing list. I'll have to look at that.

You can also see above the tag I pushed for closedv1.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

How to get a Mercurial workspace after creating a ZFS clone

I wrote about how I didn't know how to mix Mercurial and ZFS data sets together to get a new clone on a new dataset. Dave Marker provided this insight:

zfs create pool/ws/th199096/spe-build
cd /pool/ws/th199096/spe-build
hg init
echo "[paths]" > .hg/hgrc
echo "default = ssh://anon@hg.opensolaris.org/hg/nfsv41/nfs41-gate" >> .hg/hgrc
hg pull -u

The trick is realizing that there is nothing magical about 'hg clone'.

And if at this point I want to do a closed gate, I can use my normal incantation because it wil be on the same dataset.

And I can then create a ZFS snapshot and clone that to my heart's desire.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Thursday Oct 02, 2008

Setting up a Development Project Gate

When we transitioned from TeamWare (tw) to Mercurial (hg), I made several attempts to craft a group workspace. The killer always seemed to be that I had to manually do an 'hg update' in the gate and that was too difficult to remember. In the end, I decided to mimic what the ON gatekeepers were doing. At first, I did it all by brute force, copying everything that they had in place. Eventually, since I didn't know Python, I started asking Dave Marker for help. And boy, did I get some. Anyway, here is what I went through to set up nfs41-gate and nfs41-clone.

Setup a restricted user account

You want to create a restricted user account for a couple of reasons. At first I thought this was to just keep people from sneaking a look at the hgrc files and such, but there is a broader need in that you want to force all writes to the gate to come through a single account. That way you can configure the push process to always go through some sanity checks. You don't want this to be your regular account, because it will restrict your ability to do things. I'll show that to you in a bit.

[nfs4hg@aus1500-home hook]> grep nfs4hg /etc/passwd 
nfs4hg:x:3530:1813:Mr. NFS4 HG:/pool/nfs4hg:/usr/bin/tcsh
[nfs4hg@aus1500-home hook]> grep 1813 /etc/group
mhg::1813:th199096

You want a uid and gid which is not in the NIS maps and you want the account to be local to your gate machine.

Setup a ZFS filesystem for your gate and clone

You want to leverage ZFS because snapshots and clones are your safety nets. A snapshot saves you from a bad command and a clone lets you try new things in a sandbox.

zfs create pool/ws/nfs41-gate
zfs create pool/ws/nfs41-clone
chown nfs4hg:mhg /pool/ws/nfs41-gate /pool/ws/nfs41-clone

Populate your gate and clone

Be sure to login as your restricted user:

warlock % ssh aus1500-home 
aus1500-home % su - nfs4hg

Get used to doing it this way - you won't be able to ssh directly as nfs4hg before too long.

I'm going to assume a new branch off of onnv-gate. If you have an existing tw or hg workspace, you can substitute in the relevant commands to create the gate. I never want to speak of migrating TeamWare to Mercurial.

I don't know why, but I have to do something like:

cd /pool/ws/nfs41-gate
hg clone ssh://anon@hg.opensolaris.org/hg/onnv/onnv-gate
mv onnv-gate/.hg\* .
mv onnv-gate/usr .
rm -rf onnv-gate

I find Mercurial doesn't like it if the target directory already exists. If you are inside SWAN, make sure to also get the closed bits:

cd usr
hg clone ssh://anon@onnv.eng/export/onnv-clone/usr/closed

And now make your clone off of your new gate. You want to make sure that the hg paths are correct:

[nfs4hg@aus1500-home nfs41-clone]> hg paths
default = /pool/ws/nfs41-gate

Retrieve the gatekeeping Python extensions

You can get these via:

[thud@adept ~/foo]> hg clone ssh://anon@hg.opensolaris.org/hg/scm-migration/onnv-gk-tools
destination directory: onnv-gk-tools
requesting all changes
adding changesets
adding manifests
adding file changes
added 40 changesets with 171 changes to 50 files
35 files updated, 0 files merged, 0 files removed, 0 files unresolved

The onnv-gk-tools are the heart of setting up your project gate. For the nfs41-gate, I put them outside of both the gate and the clone:

[nfs4hg@aus1500-home ws]> zfs list | grep onnv-gk
pool/onnv-gk-tools                                                630K  4.31T   630K  /pool/onnv-gk-tools

Again, leverage ZFS for things like this tool set.

Read onnv-gk-tools/README at this point. I may have done things from there and forgotten to mention them here.

Copy hgrc files to gate and clone

You'll want to copy the existing hgrc files over to your gate and clone:

cp gate-hgrc /pool/ws/nfs41-gate/.hg/hgrc
cp gate-closed-hgrc /pool/ws/nfs41-gate/usr/closed/.hg/hgrc
cp clone-hgrc /pool/ws/nfs41-clone/.hg/hgrc
cp clone-closed-hgrc /pool/ws/nfs41-clone/usr/closed/.hg/hgrc

Set permissions on the gate and clone

I can't say this enough, from the README:

gate enforcement:
        Mercurial only lets you pull if you can read {REPO}/.hg
        Mercurial only lets you push if you can write {REPO}/.hg

        So {GATE} and {GATE}/usr/closed are owned by onhg/gk
        Mode is set to 0770

        {CLONE} and {CLONE}/usr/closed are also owned by onhg/gk
        But mode is set to 0775

Comment out temp hack in hook/notify.py

So, this should be configurable, but for now you want to comment out the following to avoid spamming a mailing list:

[nfs4hg@aus1500-home onnv-gk-tools]> diff hook/notify.py ~/onnv-gk-tools/hook/notify.py
80c80
<     # m.msg["Bcc"] = "onnv-flagdays@onnv.eng"
---
>     m.msg["Bcc"] = "onnv-flagdays@onnv.eng"

Note that you want to make sure the '#' is added right where the 'm' was - I hear Python really cares about indentation.

Copy on-hg.py to the homedir

You'll want to copy this file to the restricted account's homedir and rename it as well. You don't want to inadvertently refer to the ON one at any point:

cp on-hg.py ~/nfs4-hg.py

Edit the new etc/config.py file

This is the step that configures Mercurial to understand your gate.

I'm not going to step through the changes, I feel they are explanatory. Note though that later we will see that some of the Python scripts do not make use of parts of this file. I.e., GATE_USER could be used in the hook/updateoso.py file.

Hmm, and so far, all of the user accounts and paths needed for these changes exist.

[nfs4hg@aus1500-home onnv-gk-tools]> diff etc/config.py ~/onnv-gk-tools/etc/config.py 
87c87
< GATE_NAME = "nfs41"
---
> GATE_NAME = "onnv"
89c89
< GATE_WS = "/pool/ws/%s-gate" % (GATE_NAME)
---
> GATE_WS = "/ws/%s-gate" % (GATE_NAME)
91c91
< CLONE_WS = "/pool/ws/%s-clone" % (GATE_NAME)
---
> CLONE_WS = "/ws/%s-clone" % (GATE_NAME)
94c94
< GATE_DIR = "/pool/ws/nfs41-gate"
---
> GATE_DIR = "/export/onnv-gate"
96c96
< CLONE_DIR = "/pool/ws/nfs41-clone"
---
> CLONE_DIR = "/export/onnv-clone"
99,104c99,104
< GATE_HOST = "aus1500-home"
< GATE_ALTHOST = "aus1500-home"
< GATE_HOST_X = "aus1500-home"
< GATE_HOST_S = "aus1500-home"
< GATE_DOMAIN = "central"
< GATE_MAIL = "aus1500-home.central"
---
> GATE_HOST = "elpaso"
> GATE_ALTHOST = "juarez"
> GATE_HOST_X = "elpaso"
> GATE_HOST_S = "juarez"
> GATE_DOMAIN = "sfbay"
> GATE_MAIL = "onnv.eng"
106,110c106,110
< GATEKEEPER = "th199096"
< ASSTGATEKEEPER = "rmesta"
< TECHLEAD = "th199096"
< ASSTTECHLEAD = "rmesta"
< CTEAMLEAD = "webaker"
---
> GATEKEEPER = "dm120769"
> ASSTGATEKEEPER = "suha"
> TECHLEAD = "jbeck"
> ASSTTECHLEAD = "nickto"
> CTEAMLEAD = "muolla"
112,113c112,113
< ALIAS_GK = "th199096@%s" % (GATE_MAIL)
< ALIAS_GATEKEEPER = "th199096@%s" % (GATE_MAIL)
---
> ALIAS_GK = "gk@%s" % (GATE_MAIL)
> ALIAS_GATEKEEPER = "gatekeeper@%s" % (GATE_MAIL)
115,116c115,116
< GATE_USER = "nfs4hg"
< GATE_GROUP = "mhg"
---
> GATE_USER = "onhg"
> GATE_GROUP = "gk"
118,119c118,119
< SNAPS_DIR = "/pool/ws/snapshot"
< BUILDS_DIR = "/pool/ws/builds"
---
> SNAPS_DIR = "/export/snapshot"
> BUILDS_DIR = "/export/builds"

Edit the hgrc files

Now we go back and edit the hgrc files for the various pieces. These modifications tell the gate how to interact with the clone, etc.

Gate's hgrc

I will annotate these changes:

[nfs4hg@aus1500-home .hg]> diff hgrc ~/onnv-gk-tools/gate-hgrc 
17c17
< hook = /pool/onnv-gk-tools/hook
---
> hook = /export/onnv-gate/public/python/hook

Okay, we need to tell the gate where our config.py file is and how to use the extensions. The above does that. Note that if we do not make this change, we could impact ON.

20c20
< gatename = nfs41-gate
---
> gatename = onnv-gate
23c23
< wlock = nfs4hg, th199096
---
> wlock = onhg, dm120769, suha
27,28c27,28
< recv = pnfs-core@sun.com
< #logmail = onnv-gate-putback-log@onnv.eng
---
> recv = onnv-gate-notify@onnv.eng
> logmail = onnv-gate-putback-log@onnv.eng
32,33c32,33
< recv = thomas.haynes@sun.com
< rti = False
---
> recv = onnv-putback-diffs@onnv.eng
> rti = True

With a development gate, you bypass the RTI process. So, we should bypass the checking for it.

36,38d35
< [web]
< baseurl = http://aus1500-home.central
< 
40c37
< url = http://aus1500-home.central/pool/ws/nfs41-gate
---
> url = http://onnv.sfbay/net/onnv.sfbay
43c40
< temp = /pool/nfs4hg/webrev
---
> temp = /space/webrev

Ah, we will want to create this directory. I understand that you want this directory to have parents that do not have ".hg/" or "Codemgr_wsdata/". If there is even one with a subdirectory of with these names, it will mess things up.

45,48c42,49
< #[rti]
< #webrticli = /net/webrti/export/home/bin/webrticli
< #url = http://webrti.sfbay/rti/xml/index.php
< #project = on
---
> # advocate is only set for restricted builds.
> # When set only those listed (separated by commas) are valid RTI advocates.
> # Any others used will cause a rollback from rti.py
> [rti]
> webrticli = /ws/onnv-gate/public/bin/webrticli
> url = http://webrti.sfbay/rti/xml/index.php
> project = on
> #advocate = John (dot) Beck (at) sun (DOT) com

We really, really want to bypass RTI checking and John really doesn't want to be spammed by this. (And this is the only place I changed the source.)

51c52
< comchk = False
---
> comchk = True

We know a comments in a development gate are not going to be valid, so do no checks.

74c75
< #pretxnchangegroup.2 = python:hook.rti.rti
---
> pretxnchangegroup.2 = python:hook.rti.rti

Again, no RTI at all!

82c83
< changegroup.0 = /usr/bin/hg push -R /pool/ws/nfs41-gate /pool/ws/nfs41-clone
---
> changegroup.0 = /usr/bin/hg push -R /export/onnv-gate /export/onnv-clone
91d91
< 

Ahh, we should get the above out of etc/config.py, no?

BTW: The two lines I care about most here are:

changegroup.0 = /usr/bin/hg push -R /pool/ws/nfs41-gate /pool/ws/nfs41-clone
changegroup.1 = /usr/bin/hg update

Basically, after a push occurs, first push that change to the clone and then run update. See, I don't want to be doing that manually!

Also, note that because of the following lines:

[gatehooks]
gatename = nfs41-gate
logdir = public/log
lockdir = public/lock

You will want to create:

cd /pool/ws/nfs41-gate
mkdir -p public/log
mkdir public/lock

Gate's closed hgrc

All of the above changes apply, but the only real diff is the following:

81,82c81
< # push to hg.os.o will be done out of cron.
< changegroup.0 = /usr/bin/hg push -R /pool/ws/nfs41-gate/usr/closed /pool/ws/nfs41-clone/usr/closed
---
> changegroup.0 = /usr/bin/hg push -R /export/onnv-gate/usr/closed /export/onnv-clone/usr/closed
91d89

And again, the changes will automatically occur to the clone.

Clone's hgrc

This one is much simpler, mainly because there is not much there:

[nfs4hg@aus1500-home .hg]> cd /pool/ws/nfs41-clone/.hg
[nfs4hg@aus1500-home .hg]> diff hgrc ~/onnv-gk-tools/clone-hgrc 
12c12
< default = /pool/ws/nfs41-gate
---
> default = /export/onnv-gate

We tell the clone where the parent is located.

15c15
< hook = /pool/onnv-gk-tools/hook
---
> hook = /export/onnv-gate/public/python/hook
19c19
< gate = file:/pool/ws/nfs41-gate
---
> gate = file:/export/onnv-gate
36d35

Where are the hooks and the gate? All of this should be in etc/config.py.

BTW an important line here is:

# These hooks are run from bghook() in the background
bg-changegroup.0 = python:hook.updateoso.updateoso

When the clone gets updated, then we will push a change out to OpenSolaris!

If you don't have a repository out there, shame on you! Well, just comment out this line.

Clone's closed hgrc

Exact same diffs as above.

Understanding some things in the hgrcs

A big difference between the gate and the clone is in the hgrcs. The gate is write only and the clone is read only.

So the clone has to prevent writes before they occur. This line does that:

# This prevents boneheaded gatekeepers and gives a more useful message
# to gatelings who trust our hooks.
prechangegroup.0 = python:hook.cloneincoming.cloneincoming

And I haven't figured out how the gate keeps people from reading. Note, yes I have, see onnv-gk-tools/README. So part of the above may be wrong....

Modify that file in the homedir

These appear pretty self-explanatory:

[nfs4hg@aus1500-home ~]> diff nfs4-hg.py onnv-gk-tools/on-hg.py 
45c45
< HGLOGIN = "nfs4hg"
---
> HGLOGIN = "onhg"
54,55c54,55
<     "/pool/ws/nfs41-gate",
<     "/pool/ws/nfs41-gate/usr/closed",
---
>     "/export/onnv-gate",
>     "/export/onnv-gate/usr/closed",

Configuring ssh access

Okay, we almost have everything done that I remember. At this point, you need to start sending emails to your developers for them to send you in their SSH public keys -- see opensolaris.org SSH key help. They need to do this for ON anyway.

Once you get them, then you will add them to the ~/.ssh/authorized_keys of your restricted account. The format of each entry will be:

command="~/nfs4-hg.py 'th199096 ' ",no-port-forwarding,no-X11-forwarding,no-agent-forwarding [the contents of their id_ds.pub file]

You will have one per user.

The format needed here is discussed in onnv-gk-tools/README.

Disclaimer

I did all of this a month or so ago. I am reconstructing what I did. I may have missed some steps.

All of the steps reported here are mine. All mention of possible bugs is my opinion.

Notes

Cron jobs

The only cron job I have running is:

[nfs4hg@aus1500-home ~/onnv-gk-tools]> crontab -l
7 3 \* \* \* /pool/ws/scripts/buildtags.sh /pool/ws/nfs41-clone/developer.sh

This will rebuild the cscope and tags databases in the clone. I could do this in another clone, but I like it occurring in a well known place. I do not want it in the gate.

Automatic push to OpenSolaris

I haven't provided the details on how to configure an automatic push to OpenSolaris...

A good link for jumping off: How to Use Mercurial (hg) Repositories


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Setting up a NFS41 gate

We just opened up a new Mercurial gate of NFSv41 on OpenSolaris.org. Eventually it will automatically push changes as they occur to our gate. I also need to figure out a way to automatically update the closed-bins.

The hardest part was figuring out the naming convention. Some links of interest are Some work on libMicro; Mercurial transition notes and finally How to Use Mercurial (hg) Repositories. Look for For Project Leads: How to set up a Mercurial repository.

Update: Also, SCMVolunteers, look for Setting up a new (Mercurial) Project repository on OpenSolaris.org.

In any event, you can grab a copy of the source at:

hg clone ssh://anon@hg.opensolaris.org/hg/nfsv41/nfs41-gate

Note the lack of a double '/' after the FQDN - normally I would take that as a sign of a bug with Mercurial.

Note that while this compiles, you can't run it without a corresponding closed-bins.

Eventually, you should be able to browse the source via Cross Reference: nfs41-gate.

And a big thanks to David Marker for providing the help necessary to getting this to go live!


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Tuesday Sep 30, 2008

branch gatekeeping 101

So I maintain nfs41-gate which is a development branch of onnv-gate. With the introduction of Mercurial as our version control system, my life has changed a bit, but the basic tasks for gatekeeping stay the same:

Build binaries
When developers change the source base, build new BFU bits for QA and/or other developers. I.e., reference bits without anyone else's changes present.
Branch merge with the ON gate
When something we want is introduced into ON or we don't want to drift too far, we sync up with onnv-gate.

I've noticed that I find merging to be easier with Mercurial, so it tends to happen more often.

Building Binaries

I typically already have an existing workspace and I'll do an incremental build in it. Also, I do this for both sparc and i386. I don't have to worry about conflicts or merging since nothing changes in the child. A typical session would be:

[th199096@aus-build-x86 ~]> ws /builds/th199096/nfs41-gk

Workspace                    : /builds/th199096/nfs41-gk
Workspace Parent             : ssh://aus1500-home//pool/ws/nfs41-clone
Proto area ($ROOT)           : /builds/th199096/nfs41-gk/proto/root_i386
Root of source ($SRC)        : /builds/th199096/nfs41-gk/usr/src
Root of test source ($TSRC)  : /builds/th199096/nfs41-gk/usr/ontest
Current directory ($PWD)     : /builds/th199096/nfs41-gk

[th199096@aus-build-x86 nfs41-gk]>  hg pull -u
pulling from ssh://aus1500-home//pool/ws/nfs41-clone
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 93 changes to 93 files
93 files updated, 0 files merged, 0 files removed, 0 files unresolved
[th199096@aus-build-x86 nfs41-gk]> ws usr/closed/

Workspace                    : /builds/th199096/nfs41-gk/usr/closed
Workspace Parent             : ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed
Proto area ($ROOT)           : /builds/th199096/nfs41-gk/usr/closed/proto/root_i386
Root of source ($SRC)        : /builds/th199096/nfs41-gk/usr/closed/usr/src
Root of test source ($TSRC)  : /builds/th199096/nfs41-gk/usr/closed/usr/ontest
Current directory ($PWD)     : /builds/th199096/nfs41-gk/usr/closed

[th199096@aus-build-x86 closed]> hg pull -u
pulling from ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 3 changes to 3 files
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
[th199096@aus-build-x86 closed]> exit
exit
[th199096@aus-build-x86 nfs41-gk]> `which nightly` -in nightly.env 

At which point I go do something else, like blog about what I am doing.

I said typical, except that this change set (which gets rid of auth records in the MDS as a byproduct) happens to have touched something in closed. We rarely seem to make changes there.

Okay, the build is done (remember to check the logs) and in this case it did not fail. So push the BFU bits out and then send out email telling people about the new reference bits.

[th199096@aus-build-x86 nfs41-gk]> ~/gk/hg-push.sh nfs41-gk i386 2008-09-30
ARCH=i386
BASE=/builds/th199096
AUS=/net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386
NIGHTLY=/builds/th199096/nfs41-gk/archives/i386/nightly
NIGHTLY_ND=/builds/th199096/nfs41-gk/archives/i386/nightly-nd
DATER=/builds/th199096/nfs41-gk/archives/i386/2008-09-30
DATER_ND=/builds/th199096/nfs41-gk/archives/i386/2008-09-30-nd
+ mv /builds/th199096/nfs41-gk/archives/i386/nightly /builds/th199096/nfs41-gk/archives/i386/2008-09-30 
+ mv /builds/th199096/nfs41-gk/archives/i386/nightly-nd /builds/th199096/nfs41-gk/archives/i386/2008-09-30-nd 
+ cp -r /builds/th199096/nfs41-gk/archives/i386/2008-09-30 /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386/2008-09-30 
+ cp -r /builds/th199096/nfs41-gk/archives/i386/2008-09-30-nd /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386/2008-09-30-nd 
+ rm /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386/latest /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386/latest-nd 
+ cd /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386 
+ ln -s 2008-09-30 latest 
+ ln -s 2008-09-30-nd latest-nd 

Branch merging

This case is a bit more complicated and can be summarized by:

  1. Take a clone of nfs41-clone - call it nfs41-sync
  2. Reparent it to onnv-clone (onnv-gate is write only)
  3. Pull across the changes and merge them.
    This is actually the only difficulty in the entire process
  4. On sparc and x86 build boxes, get clones of nfs41-sync and do full builds.
    Incrementals are not sufficient in this case.
  5. Populate a pNFS community (client, DS, and MDS) with these changes and make sure the Connectathon tests all pass.
    Depending on the scope of the changes and/or the difficulty of the merge, we may skip this item. -- pure judgment call. Also, note that these clones will become the basis for future incremental builds as described in the previous section.
  6. If there have been further integrations to nfs41-gate, reparent to nfs41-clone (which is automatically kept up to date with nfs41-gate), and do the pull/merge cycle until everything is up to date. You may have to rebuild and retest. Much easier with a small group of developers to ask them not to integrate.
  7. ZFS snapshot nfs41-gate to make rolling back the changes easier.
  8. Reparent nfs41-sync to nfs41-gate and integrate the changes.

By not changing the nfs41-gate until the final moment, I can throw everything away if needed. And believe me, as painful as that is, I've done it. Also, note that when I talk about a workspace above, I am also taling about working in parallel with the closed version of it.

But now onto a detailed example:

Get a backup snapshot of the gate:

[th199096@aus1500-home ~]> zfs snapshot pool/ws/nfs41-clone@sync99

Now grab your copy for merging

[th199096@aus1500-home ~]> cd /pool/ws/th199096/
[th199096@aus1500-home th199096]> ~/bin/hg-clone ssh://aus1500-home//pool/ws/nfs41-clone nfs41-syn
c
397b36b5473d
=== clone open tree: ssh://aus1500-home//pool/ws/nfs41-clone ===
requesting all changes
adding changesets
adding manifests
adding file changes
added 7515 changesets with 101024 changes to 51313 files
updating working directory
42507 files updated, 0 files merged, 0 files removed, 0 files unresolved
2a39f20bc20e
=== clone closed tree: ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed ===
requesting all changes
adding changesets
adding manifests
adding file changes
added 968 changesets with 8269 changes to 4389 files
updating working directory
2677 files updated, 0 files merged, 0 files removed, 0 files unresolved

~/bin/hg-clone is a simple script to get both the open and closed versions of the gate.< /p>

And reparent it to onnv-clone

[th199096@aus1500-home th199096]> ws nfs41-sync/

Workspace			: /pool/ws/th199096/nfs41-sync
Workspace Parent		: ssh://aus1500-home//pool/ws/nfs41-clone
Proto area ($ROOT)		: /pool/ws/th199096/nfs41-sync/proto/root_i386
Root of source ($SRC)		: /pool/ws/th199096/nfs41-sync/usr/src
Root of test source ($TSRC)  : /pool/ws/th199096/nfs41-sync/usr/ontest
Current directory ($PWD)	: /pool/ws/th199096/nfs41-sync

[th199096@aus1500-home nfs41-sync]> hg reparent ssh://onnv.eng//export/onnv-clone

Pull and merge

[th199096@aus1500-home nfs41-sync]> hg pull -u
pulling from ssh://onnv.eng//export/onnv-clone
searching for changes
adding changesets
adding manifests
adding file changes
added 210 changesets with 2122 changes to 1808 files (+1 heads)
not updating, since new heads added
(run 'hg heads' to see heads, 'hg merge' to merge)
[th199096@aus1500-home nfs41-sync]> hg merge
merging usr/src/cmd/Makefile
merging usr/src/cmd/zfs/zfs_iter.c
merging usr/src/cmd/zfs/zfs_main.c
merging usr/src/lib/Makefile
merging usr/src/lib/libzfs/common/libzfs.h
merging usr/src/lib/libzfs/common/libzfs_dataset.c
merging usr/src/lib/libzfs/common/mapfile-vers
merging usr/src/pkgdefs/Makefile
merging usr/src/pkgdefs/SUNWcsu/prototype_com
merging usr/src/pkgdefs/SUNWhea/prototype_com
merging usr/src/pkgdefs/etc/exception_list_sparc
merging usr/src/uts/common/Makefile.files
merging usr/src/uts/common/Makefile.rules
merging usr/src/uts/common/fs/zfs/dsl_dataset.c
merging usr/src/uts/common/fs/zfs/zfs_ioctl.c
merging usr/src/uts/common/sys/Makefile
merging usr/src/uts/common/sys/fs/zfs.h
merging usr/src/uts/intel/Makefile.intel.shared
merging usr/src/uts/intel/os/minor_perm
merging usr/src/uts/intel/os/name_to_major
merging usr/src/uts/sparc/Makefile.sparc.shared
merging usr/src/uts/sparc/os/minor_perm
merging usr/src/uts/sparc/os/name_to_major
1785 files updated, 23 files merged, 62 files removed, 0 files unresolved
(branch merge, don't forget to commit)

So, I used filemerge to do any manual editing in the above merge. It is invoked in my .hgrc:

# Merge tool
[merge-patterns]
\*\* = filemerge

[merge-tools]
filemerge.executable = /ws/onnv-tools/teamware/bin/filemerge
filemerge.args = -a $base $local $other $output
filemerge.checkchanged = true
filemerge.gui = true

Then I would commit and repeat the cycle for the closed branch:

[th199096@aus1500-home nfs41-sync]> hg commit
[th199096@aus1500-home nfs41-sync]> ws usr/closed

Workspace			: /pool/ws/th199096/nfs41-sync/usr/closed
Workspace Parent		: ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed
Proto area ($ROOT)		: /pool/ws/th199096/nfs41-sync/usr/closed/proto/root_i386
Root of source ($SRC)		: /pool/ws/th199096/nfs41-sync/usr/closed/usr/src
Root of test source ($TSRC)  : /pool/ws/th199096/nfs41-sync/usr/closed/usr/ontest
Current directory ($PWD)	: /pool/ws/th199096/nfs41-sync/usr/closed

[th199096@aus1500-home closed]> hg reparent ssh://onnv.eng//export/onnv-clone/usr/closed
[th199096@aus1500-home closed]> hg pull -u
pulling from ssh://onnv.eng//export/onnv-clone/usr/closed
searching for changes
adding changesets
adding manifests
adding file changes
added 15 changesets with 145 changes to 137 files (+1 heads)
not updating, since new heads added
(run 'hg heads' to see heads, 'hg merge' to merge)
[th199096@aus1500-home closed]> hg merge
135 files updated, 0 files merged, 10 files removed, 0 files unresolved
(branch merge, don't forget to commit)
[th199096@aus1500-home closed]> hg commit

The next step is a clone/build on one of the build machines. As this looks a lot like the cloning in this section and the build from the prior, I'm going to leave it out.

After the build and verification is done, we prepare the nfs41-gate for the integration. Because of the branch merge comments not being in the approved RTI format, we need to turn off sanity checking for this operation. Note that it is okay to keep on for developers pushing to the gate:

[th199096@aus1500-home ~]> su - nfs4hg
Password:
Sun Microsystems Inc.	SunOS 5.11	snv_92	January 2008
[nfs4hg@aus1500-home ~]> cd /pool/ws/nfs41-gate/usr/closed/.hg
[nfs4hg@aus1500-home .hg]> cp hgrc hgrc.good
[nfs4hg@aus1500-home .hg]> vi hgrc
[nfs4hg@aus1500-home .hg]> diff hgrc hgrc.good
73c73
< #pretxnchangegroup.1 = python:hook.sanity.sanity
---
> pretxnchangegroup.1 = python:hook.sanity.sanity

Reparent to the gate and push

[th199096@aus1500-home closed]> hg reparent ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate/usr/clos
ed
[th199096@aus1500-home closed]> hg push
pushing to ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate/usr/closed
searching for changes
Are you sure you wish to push? [y/N]: y
pushing to ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate/usr/closed
...
remote: Preparing gk email...
remote: ...gk email sent

Fix the .hgrc back to turn on sanity checking and repeat for the open bits.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Thursday Sep 25, 2008

Hit a Doh! whilst gatekeeping

I was trying to integrate a branch merge of the nfs41-gate and ON. And I'm keeping tight notes because I'm trying to get Piyush to be the gatekeeper. Anyway, here is what went wrong:

Reparent to the gate:

% hg reparent ssh://aus1500-home//pool/ws/nfs41-gate

And push:

% hg push
pushing to ssh://aus1500-home//pool/ws/nfs41-gate
searching for changes
Are you sure you wish to push? [y/N]: y
remote: abort: could not lock repository /pool/ws/nfs41-gate: Permission denied
abort: unexpected response: empty string

This worked last time I tried it. And Rich Brown was able to make a change this morning. Now, I could just blame him, but that is too easy. I use google and find [Volo-dev] I am unable to push my changes to the gate. I don't get past the comment:

You are trying to push back the changes as user "anon".

Before I realize what is going on. I forgot to reparent the gate correctly:

% hg reparent ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate
% hg push
pushing to ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate
searching for changes
Are you sure you wish to push? [y/N]: y
pushing to ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate
searching for changes
Are you sure you wish to push? [y/N]: y
remote: adding changesets
remote: adding manifests
...
remote: Preparing gk email...
remote: ...gk email sent

All is well and Rich is off the hook!


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily
About

tdh

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today