Thursday Nov 20, 2008

Locking down a gate/respository

So I'm not a mean gatekeeper - I try to not lock down our gate. I feel like I should just be able to ask people not to integrate anything. But the reality is that you can't count on everyone getting the message. So, you need to lock down the gate.

I knew how to do that in Teamware, but I've never had that need with Mercurial. Until today that is - a nasty branch merge with onnv_103.

So to lock down a Mercurial gate with Sun's extensions, you can use lock.py:

[nfs4hg@aus1500-home ~]> which lock.py
/pool/nfs4hg/bin/lock.py
[nfs4hg@aus1500-home nfs41-gate]> lock.py -n -R /pool/ws/nfs41-gate
[write]: None

Okay, no one has a write lock, so let's grab one:

[nfs4hg@aus1500-home nfs41-gate]> lock.py -R /pool/ws/nfs41-gate
[nfs4hg@aus1500-home nfs41-gate]> lock.py -n -R /pool/ws/nfs41-gate
[write]:
        nfs4hg
        th199096

By the way, where is this configured?

[nfs4hg@aus1500-home ~]> grep lock /pool/ws/nfs41-gate/.hg/hgrc 
#   5. lockdir must be readable by whomever will pull/push
lockdir = public/lock
wlock = nfs4hg, th199096
# These hooks check the lock before anything happens.
prechangegroup.0 = python:hook.lockchk.lockchk
# then comment it out, let them push, uncomment, and unlock the gate.

So the lock only works on the gate and not the clone. You can find more about this in the source of lock.py.

Ohh, and even though I haven't tested it, you release the gate easily enough with a unlock.py.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Thursday Oct 09, 2008

A better fix for that non-responsive tool repository

I ran into another machine which got stuck on that 'hg style' issue I reported in Getting around a tool repository which is not updating and I got mad enough to start looking at it. We've still got path dependencies in BFU to stuff inside SWAN and I was hoping we could get away from it with the new tools.

I started trying to figure out where the Python script was that was being run. I wanted to find the hard-coded reference to /ws/onnv-tools/onbld/etc/hgstyle. Knowing that I had just seen a Flag Day on this, I looked in Flag day: new default output style for Mercurial. And the solution was staring right at me!

Mark had coded it correctly, no path hardcoding! I could simply change:

	[ui]
	username=Mark J. Nelson 
	style=/ws/onnv-tools/onbld/etc/hgstyle

to be:

	[ui]
	username=Mark J. Nelson 
	style=/opt/onbld/etc/hgstyle

Sweet, great to have my faith restored and an effective solution.

But now I need to really fix the automount maps in the lab, we keep on tripping over this issue with a stale repository and we have a working one we should be using.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Wednesday Oct 08, 2008

Branch merge to a specific tagged revision

I need to do a branch merge between nfs41-gate and onnv-clone. And specifically, I want to not get the 'tip', but rather the tag for release 100. I found a good reference - Chapter 8 Managing releases and branchy development.

So I'll follow along with it. I need the tag:

[th199096@jhereg onnv-play]> hg tags
tip                             7782:716c23b2ce2e
onnv_100                        7757:bf4a45ecb669
onnv_99                         7613:e49de7ec7617
onnv_98                         7473:fad192e9bc57

It turns out I don't need much more:

[th199096@jhereg nfs41-100]> hg reparent ssh://onnv.eng//export/onnv-clone
[th199096@jhereg nfs41-100]> hg tags | more
tip                             7744:763bfa203d1a
closedv1                        7742:9fab48a31a4a
onnv_99                         7652:e49de7ec7617

So I haven't merged yet:

[th199096@jhereg nfs41-100]> hg pull -u -r onnv_100
pulling from ssh://onnv.eng//export/onnv-clone
searching for changes
adding changesets
adding manifests
adding file changes
added 64 changesets with 475 changes to 462 files (+1 heads)
not updating, since new heads added
(run 'hg heads' to see heads, 'hg merge' to merge)

The tag can be used as a revision!


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Monday Oct 06, 2008

Pushing your open gate to OpenSolaris

In Setting up a Development Project Gate, I showed you the steps I took to setup a shared development gate. The only thing I left out was the automatic push to a Mercurial repository on OpenSolaris. I'll take you through these steps now.

By the way, thanks to Dave Marker for sharing how ONNV does this and writing some Python tools to automate the majority of the push.

Get a repository

First you need to get a project leader for your project to create a workspace for you on hg.opensolaris.org. They can do this by logging to OpenSolaris and going to the project page. Then they select 'SCM Management'. Next 'Add Repository' and remember to click the Mercurial radio button on the next screen. The page is pretty explanatory and the only thing to remember is that it might spew up a message about the project already existing. It also seems to take 5-10 minutes for the project space to show up.

Which if you know about it is okay. You can take this time to configure who has commit rights. (You'll need one account for sure, more on that later.)

As evidenced in Setting up a NFS41 gate, I had a hard time figuring out where the gate was located. It can be reached as:

hg clone ssh://anon@hg.opensolaris.org/hg/<PROJECT>/<GATE-NAME>

Note you need to configure anonymous access on the repository page for this to work. I'm not sure, but I believe this also implies anyone can write back to the gate. Tie this in with a lack of tools to maintain the gate out there (including deletion) and you can see this might be a problem.

Configure a SSH public key

This part is the scary part - scary in that you have to decide if you want automatic pushes or manual pushes.

Manual pushes

If you want manual pushes, then you need to make sure you have a SSH public key setup as described in opensolaris.org SSH key help. Then whenever you want to push a change, you would do:

hg push -R <your gate> -e "ssh -q -F ssh://<YOUR OSOL USERNAME>@hg.opensolaris.org/hg/<PROJECT>/<GATE-NAME>"

You don't have to read much more of the rest of this entry, except perhaps to make sure your proxy host is correct.

Automatic pushes

If you want automatic pushes, then you need to configure a special key pair without a passphrase. Note that this is a risk you need to manage yourself. It needs to be blank because you don't want to even store the passphrase anywhere. I've done this type of thing in the past for production systems and the trick is that you want to lock down the box you have stored the private version of the key

Follow the directions as described in opensolaris.org SSH key help to publish this key. Note that no-one will know that this passphrase is empty for this key. They still need the private copy of the key. We have to make sure that is locked down tight!

  • Use a system with a non-standard root password.
  • Strictly control admin access to the system
    • Don't hand out the root password.
    • Don't make sudo too permissive.
  • Pick a non-global account to store the data in.
  • Make sure the permissions are 700 on the directory where the key is stored.
  • If you share that homedir, make sure to read The myth of security with AUTH_SYS, which means:
    • Do not accept the default access list ooptions.
    • Set root= or use anon=-1.
    • Set the rw= and/or ro= hosts appropriately.
    • Strongly consider kerberizing the share.

There is probably more you can do, so the first thing is to accept responsibility for this setup.

Okay, having scared you, it is now time to configure this special ssh configuration:

[nfs4hg@aus1500-home ~/tmp]> mkdir opensolaris
[nfs4hg@aus1500-home ~/tmp]> chmod 700 opensolaris/
[nfs4hg@aus1500-home ~/tmp]> cp ~/opensolaris/\* .
[nfs4hg@aus1500-home ~/tmp]> more config 
Host \*.org
        GlobalKnownHostsFile /pool/nfs4hg/opensolaris/known_hosts
        ProxyCommand /usr/lib/ssh/ssh-socks5-proxy-connect -h 192.18.43.19 %h %p
        IdentityFile /pool/nfs4hg/opensolaris/id_dsa
        User <YOUR OSOL USERNAME>

You'll have to figure out whether or not you need the proxy configuration or not. And you will need to fix-up the paths for the GlobalKnownHostsFile and IdentityFile.

Also note that I've got this account already setup with a ~/.ssh directory to allow integrations to the gate. It doen't have DS keys, but I like to keep these apart. And note that id_dsa and id_dsa.txt are the keypair that you generated with the empty passphrase.

Edit hook/updateoso.py

In the copy of hook/updateoso.py I got from ssh://anon@hg.opensolaris.org/hg/scm-migration/onnv-gk-tools, I had the following minor diffs:

[nfs4hg@aus1500-home ~]> diff /pool/onnv-gk-tools/hook/updateoso.py onnv-gk-tools/hook/updateoso.py
33c33
< OSOREPO = "ssh://hg.opensolaris.org/hg/nfsv41/nfs41-gate"
---
> OSOREPO = "ssh://hg.opensolaris.org/hg/onnv/onnv-gate"
43c43
< This script must be run as user "nfs4hg".
---
> This script must be run as user "onhg".
76c76
<     home = pwd.getpwnam("nfs4hg")[5]
---
>     home = pwd.getpwnam("onhg")[5]
95c95
<     if utility.check_user("nfs4hg", m) is False:
---
>     if utility.check_user("onhg", m) is False:
97c97
<         m.write('''Can't update OpenSolaris. User != "nfs4hg"\\n''')
---
>         m.write('''Can't update OpenSolaris. User != "onhg"\\n''')

Note that I grabbed this script right after Dave Marker put it back, so I got a cutting edge version of it. I know he is about to change it such that the user and osol path are configurable variables outside of this file. I.e., you would make changes in a project specific configuration file.

Anyway, once those changes are made, go a head and make sure this line is enabled in your official clone's hgrc:

# These hooks are run from bghook() in the background
bg-changegroup.0 = python:hook.updateoso.updateoso

I'm assuming you have modeled your development gate ala onnv (and as I described in Setting up a Development Project Gate). If you haven't, then I think all you need to do is add this as the last bg.changeroup entry in your gate's hgrc.

And now you are good to go!

You might want to do the manual push I described above to seed the gate.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Trying to figure out printing and variables in Python

I'm pretty used to referencing variables inside print blocks in Perl. I'm not at all comfortable with Python. I have a block of code that I want to change the 'onhg' to come out of a config file. So I set up a scratch directory and make a bare bones implementation:

[th199096@jhereg etc]> ls -laiR ~/scratch/
/home/th199096/scratch/:
total 42
       749 drwxr-xr-x   3 th199096 staff          4 Oct  6 16:36 .
         3 drwxr-xr-x  39 th199096 staff         55 Oct  6 16:35 ..
       752 drwxr-xr-x   2 th199096 staff          6 Oct  6 16:42 etc
       750 -rwxr-xr-x   1 th199096 staff       1297 Oct  6 16:42 updateoso.py

/home/th199096/scratch/etc:
total 25
       752 drwxr-xr-x   2 th199096 staff          6 Oct  6 16:42 .
       749 drwxr-xr-x   3 th199096 staff          4 Oct  6 16:36 ..
       753 -rw-r--r--   1 th199096 staff       1052 Oct  6 16:40 __init__.py
       754 -rw-r--r--   1 th199096 staff        243 Oct  6 16:41 __init__.pyc
       751 -rwxr-xr-x   1 th199096 staff         94 Oct  6 16:42 config.py
       756 -rw-r--r--   1 th199096 staff        257 Oct  6 16:42 config.pyc

Where the config file simply has:

GATE_USER = "onhg"
GATE_GROUP = "gk"

OSOREPO = "ssh://hg.opensolaris.org/hg/onnv/onnv-gate"

And the updateoso has:

import os, pwd, subprocess, sys

from mercurial import hg, repo
from mercurial.node import hex

sys.path.insert(1,
    os.path.realpath(os.path.join(os.path.dirname(__file__), "..")))
from etc import config

__USAGE = """
updateoso.py [-n] <-R repo root>
    -n: dry run, no email sent (displayed on stdout)
    -R: root dir of repo (where .hg is)

Attempt to send changes to 
%s

This script must be run as user "onhg".
You should set up RBAC and use pfexec(1).
""" % (config.OSOREPO)
__USAGE = __USAGE.strip()

print >> sys.stderr, __USAGE

Well, in isolation, I can already see what I am going to have to do. All I need to do is replace the 'ohng' with a %s and add a second argument:

[th199096@jhereg ~/scratch]> diff updateoso.py updateoso.py.first 
39c39
< This script must be run as user "%s".
---
> This script must be run as user "ohng".
41c41
< """ % (config.OSOREPO, config.GATE_USER)
---
> """ % (config.OSOREPO)

And we get:

[th199096@jhereg ~/scratch]> ./updateoso.py
updateoso.py [-n] <-R repo root>
    -n: dry run, no email sent (displayed on stdout)
    -R: root dir of repo (where .hg is)

Attempt to send changes to 
ssh://hg.opensolaris.org/hg/onnv/onnv-gate

This script must be run as user "onhg".
You should set up RBAC and use pfexec(1).

I ought to be able to test this inside the interactive shell:

[th199096@jhereg ~/scratch]> python
Python 2.4.4 (#1, Aug 25 2008, 03:30:42) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import updateoso
updateoso.py [-n] <-R repo root>
    -n: dry run, no email sent (displayed on stdout)
    -R: root dir of repo (where .hg is)

Attempt to send changes to 
ssh://hg.opensolaris.org/hg/onnv/onnv-gate

This script must be run as user "onhg".
You should set up RBAC and use pfexec(1).
>>> config.GATE_USER = "duke"
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
NameError: name 'config' is not defined

Okay, I should have known that wasn't going to work. It would probably work in the code (we'll see later), but for now this will work:

>>> updateoso.config.GATE_USER = "duke"
>>> reload(updateoso)
updateoso.py [-n] <-R repo root>
    -n: dry run, no email sent (displayed on stdout)
    -R: root dir of repo (where .hg is)

Attempt to send changes to 
ssh://hg.opensolaris.org/hg/onnv/onnv-gate

This script must be run as user "duke".
You should set up RBAC and use pfexec(1).
<module 'updateoso' from 'updateoso.pyc'>

To be honest, I knew the reference would work, but I expected it to be reset. In retrospect, I can see that I reloaded updateoso and etc/config. Just something to get used to. I could force it to 'reset' via:

>>> reload(etc/config)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
NameError: name 'etc' is not defined
>>> from etc reload(config)
  File "<stdin>", line 1
    from etc reload(config)
                  \^
SyntaxError: invalid syntax
>>> reload(updateoso.config)
<module 'etc.config' from 'etc/config.pyc'>
>>> reload(updateoso)
updateoso.py [-n] <-R repo root>
    -n: dry run, no email sent (displayed on stdout)
    -R: root dir of repo (where .hg is)

Attempt to send changes to 
ssh://hg.opensolaris.org/hg/onnv/onnv-gate

This script must be run as user "onhg".
You should set up RBAC and use pfexec(1).
<module 'updateoso' from 'updateoso.pyc'>

Took me a bit to figure out the syntax.

Okay, can I see the change from the script:

[th199096@jhereg ~/scratch]> diff updateoso.py updateoso.py.second 
45,48d44
< 
< config.GATE_USER = "gark"
< print >> sys.stderr, __USAGE
< 

I don't expect this to work. And it doesn't.

This script must be run as user "onhg".
...
This script must be run as user "onhg".

How about a test driver script?

[th199096@jhereg ~/scratch]> more test.py 
import updateoso

print "Now change the user"

updateoso.config.GATE_USER = "nark"

reload(updateoso)

And that works:

This script must be run as user "onhg".
...
Now change the user
...
This script must be run as user "nark".

Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Sunday Oct 05, 2008

Learning a new language - python

So I decided to learn python - why? because it is used by Mercurial. And there is at least one of the gatekeeping scripts which I needed to hack for the nfs41-gate.

I bought Learning Python, Third Edition by Mark Lutz because the local Borders did not have Programming Python, Third Edition. From some reviews I read, it would probably have been a better fit for me.

I know I can find most of what I want on the net, but I wanted a printed resource.

Anyway, I had a question right off the bat - about whether if file a imports modules b and c, what happens if c also imports b? Deeper in the book that I've read, it does state that an import is equivalent to load the file if it is not already loaded. But that doesn't help me learn the language. :->

The following example is quite simple, but effective in answering the question for me:

a.py

> cat a.py
#!/usr/bin/python

title = "This is the file a.py!"

print title

print "importing b from a"
import b

print "importing c from a"
import c

b.py

> cat b.py
#!/usr/bin/python

title = "This is the file b.py!"

print title

c.py

> cat c.py
#!/usr/bin/python

import b

title = "This is the file c.py!"

print title

Test 1

> ./a.py
This is the file a.py!
importing b from a
This is the file b.py!
importing c from a
This is the file c.py!

So we see that if a has loaded it, then c will not. How about the other way?

a2.py

> cat a2.py
#!/usr/bin/python

title = "This is the file a.py!"

print title

print "importing c from a"
import c

print "importing b from a"
import b

print "and b's title is"
print b.title

Test 2

> ./a2.py
This is the file a.py!
importing c from a
This is the file b.py!
This is the file c.py!
importing b from a
and b's title is
This is the file b.py!

We see that c loads b and that b's attributes are visible from a.

What would really help me here is if b could state the call stack of what is importing it.

Test 3

A simple change fails:

> cat b.py
#!/usr/bin/python

title = "This is the file b.py!"

print title

print "Called from", __file__

but it does show the effect of byte code compilation:

> ./a.py
This is the file a.py!
importing b from a
This is the file b.py!
Called from /home/tdh/python/b.py
importing c from a
This is the file c.py!
> ./a2.py
This is the file a.py!
importing c from a
This is the file b.py!
Called from /home/tdh/python/b.pyc
This is the file c.py!
importing b from a
and b's title is
This is the file b.py!

I can see the "nesting" if I pop into an interactive session:

> python
>>> import a2
This is the file a.py!
importing c from a
This is the file b.py!
Called from b.pyc
This is the file c.py!
importing b from a
and b's title is
This is the file b.py!
>>> a2.__dict__.keys()
['c', 'b', 'title', '__builtins__', '__file__', '__name__', '__doc__']
>>> a2.c.__dict__.keys()
['b', 'title', '__builtins__', '__file__', '__name__', '__doc__']
>>> a2.c.b.__dict__.keys()
['__builtins__', '__name__', '__file__', '__doc__', 'title']

But this doesn't answer my question of how to figure this out recursively. I.e., I guess I am looking for a parent "pointer" and I could walk it to get my answer.

But I've still learned more than just reading the book linearly.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Friday Oct 03, 2008

Hey, the source browser is up and running

Check out nfsv41/nfs41-gate.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

First successful push to nfs41-gate on opensolaris

We had the first developer make a real push to the nfs41-gate on the OpenSolaris NFSv41 Project Repository.

Jim Wahlig had this to push:

[thud@adept nfs41-gate]> hg incoming
comparing with ssh://anon@hg.opensolaris.org/hg/nfsv41/nfs41-gate
searching for changes
changeset:   7743:c672b1cb86be
user:        Thomas Haynes 
date:        Thu Oct 02 22:28:30 2008 -0500
summary:     Added tag closedv1 for changeset 9fab48a31a4a

changeset:   7744:763bfa203d1a
tag:         tip
user:        jwahlig@aus-build3
date:        Fri Oct 03 11:52:59 2008 -0500
summary:     fix stable storage on x86.

The only issue we encountered was that mail did not get accepted for the nfs41-discuss mailing list. I'll have to look at that.

You can also see above the tag I pushed for closedv1.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

How to get a Mercurial workspace after creating a ZFS clone

I wrote about how I didn't know how to mix Mercurial and ZFS data sets together to get a new clone on a new dataset. Dave Marker provided this insight:

zfs create pool/ws/th199096/spe-build
cd /pool/ws/th199096/spe-build
hg init
echo "[paths]" > .hg/hgrc
echo "default = ssh://anon@hg.opensolaris.org/hg/nfsv41/nfs41-gate" >> .hg/hgrc
hg pull -u

The trick is realizing that there is nothing magical about 'hg clone'.

And if at this point I want to do a closed gate, I can use my normal incantation because it wil be on the same dataset.

And I can then create a ZFS snapshot and clone that to my heart's desire.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Thursday Oct 02, 2008

How to tie our closed-bins to the new gate

I was trying to relax and I realized we would have an ongoing problem in keeping the new ssh://anon@hg.opensolaris.org/hg/nfsv41/nfs41-gate in sync with our copy of the closed binaries. But, I think we will be saved by a couple of things:

  1. We don't update the internal nfs41-gate automatically with every change in the onnv-gate. We actually normally sync up with the 2 week releases. This means that random changes to the closed source will not impact the osol gate. As a matter of fact, we control when a change causes a respin of the closed bits.
  2. Just like the ON gatekeepers tag their gate every 2 weeks, we could also use 'hg tag' to mark when the closed binaries changed. We could store the closed binaries on the project download page and when an external developer saw the tag change, they could then pick up a new copy

Plus with setting the mail to go out to the dev mailing list, people would be able to see a need to pickup a new set of closed binaries.

[thud@adept src]> hg incoming
comparing with ssh://anon@hg.opensolaris.org/hg/nfsv41/nfs41-gate
searching for changes
changeset:   7743:c672b1cb86be
tag:         tip
user:        Thomas Haynes 
date:        Thu Oct 02 22:28:30 2008 -0500
summary:     Added tag closedv1 for changeset 9fab48a31a4a

Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Tuesday Sep 30, 2008

branch gatekeeping 101

So I maintain nfs41-gate which is a development branch of onnv-gate. With the introduction of Mercurial as our version control system, my life has changed a bit, but the basic tasks for gatekeeping stay the same:

Build binaries
When developers change the source base, build new BFU bits for QA and/or other developers. I.e., reference bits without anyone else's changes present.
Branch merge with the ON gate
When something we want is introduced into ON or we don't want to drift too far, we sync up with onnv-gate.

I've noticed that I find merging to be easier with Mercurial, so it tends to happen more often.

Building Binaries

I typically already have an existing workspace and I'll do an incremental build in it. Also, I do this for both sparc and i386. I don't have to worry about conflicts or merging since nothing changes in the child. A typical session would be:

[th199096@aus-build-x86 ~]> ws /builds/th199096/nfs41-gk

Workspace                    : /builds/th199096/nfs41-gk
Workspace Parent             : ssh://aus1500-home//pool/ws/nfs41-clone
Proto area ($ROOT)           : /builds/th199096/nfs41-gk/proto/root_i386
Root of source ($SRC)        : /builds/th199096/nfs41-gk/usr/src
Root of test source ($TSRC)  : /builds/th199096/nfs41-gk/usr/ontest
Current directory ($PWD)     : /builds/th199096/nfs41-gk

[th199096@aus-build-x86 nfs41-gk]>  hg pull -u
pulling from ssh://aus1500-home//pool/ws/nfs41-clone
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 93 changes to 93 files
93 files updated, 0 files merged, 0 files removed, 0 files unresolved
[th199096@aus-build-x86 nfs41-gk]> ws usr/closed/

Workspace                    : /builds/th199096/nfs41-gk/usr/closed
Workspace Parent             : ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed
Proto area ($ROOT)           : /builds/th199096/nfs41-gk/usr/closed/proto/root_i386
Root of source ($SRC)        : /builds/th199096/nfs41-gk/usr/closed/usr/src
Root of test source ($TSRC)  : /builds/th199096/nfs41-gk/usr/closed/usr/ontest
Current directory ($PWD)     : /builds/th199096/nfs41-gk/usr/closed

[th199096@aus-build-x86 closed]> hg pull -u
pulling from ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 3 changes to 3 files
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
[th199096@aus-build-x86 closed]> exit
exit
[th199096@aus-build-x86 nfs41-gk]> `which nightly` -in nightly.env 

At which point I go do something else, like blog about what I am doing.

I said typical, except that this change set (which gets rid of auth records in the MDS as a byproduct) happens to have touched something in closed. We rarely seem to make changes there.

Okay, the build is done (remember to check the logs) and in this case it did not fail. So push the BFU bits out and then send out email telling people about the new reference bits.

[th199096@aus-build-x86 nfs41-gk]> ~/gk/hg-push.sh nfs41-gk i386 2008-09-30
ARCH=i386
BASE=/builds/th199096
AUS=/net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386
NIGHTLY=/builds/th199096/nfs41-gk/archives/i386/nightly
NIGHTLY_ND=/builds/th199096/nfs41-gk/archives/i386/nightly-nd
DATER=/builds/th199096/nfs41-gk/archives/i386/2008-09-30
DATER_ND=/builds/th199096/nfs41-gk/archives/i386/2008-09-30-nd
+ mv /builds/th199096/nfs41-gk/archives/i386/nightly /builds/th199096/nfs41-gk/archives/i386/2008-09-30 
+ mv /builds/th199096/nfs41-gk/archives/i386/nightly-nd /builds/th199096/nfs41-gk/archives/i386/2008-09-30-nd 
+ cp -r /builds/th199096/nfs41-gk/archives/i386/2008-09-30 /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386/2008-09-30 
+ cp -r /builds/th199096/nfs41-gk/archives/i386/2008-09-30-nd /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386/2008-09-30-nd 
+ rm /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386/latest /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386/latest-nd 
+ cd /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386 
+ ln -s 2008-09-30 latest 
+ ln -s 2008-09-30-nd latest-nd 

Branch merging

This case is a bit more complicated and can be summarized by:

  1. Take a clone of nfs41-clone - call it nfs41-sync
  2. Reparent it to onnv-clone (onnv-gate is write only)
  3. Pull across the changes and merge them.
    This is actually the only difficulty in the entire process
  4. On sparc and x86 build boxes, get clones of nfs41-sync and do full builds.
    Incrementals are not sufficient in this case.
  5. Populate a pNFS community (client, DS, and MDS) with these changes and make sure the Connectathon tests all pass.
    Depending on the scope of the changes and/or the difficulty of the merge, we may skip this item. -- pure judgment call. Also, note that these clones will become the basis for future incremental builds as described in the previous section.
  6. If there have been further integrations to nfs41-gate, reparent to nfs41-clone (which is automatically kept up to date with nfs41-gate), and do the pull/merge cycle until everything is up to date. You may have to rebuild and retest. Much easier with a small group of developers to ask them not to integrate.
  7. ZFS snapshot nfs41-gate to make rolling back the changes easier.
  8. Reparent nfs41-sync to nfs41-gate and integrate the changes.

By not changing the nfs41-gate until the final moment, I can throw everything away if needed. And believe me, as painful as that is, I've done it. Also, note that when I talk about a workspace above, I am also taling about working in parallel with the closed version of it.

But now onto a detailed example:

Get a backup snapshot of the gate:

[th199096@aus1500-home ~]> zfs snapshot pool/ws/nfs41-clone@sync99

Now grab your copy for merging

[th199096@aus1500-home ~]> cd /pool/ws/th199096/
[th199096@aus1500-home th199096]> ~/bin/hg-clone ssh://aus1500-home//pool/ws/nfs41-clone nfs41-syn
c
397b36b5473d
=== clone open tree: ssh://aus1500-home//pool/ws/nfs41-clone ===
requesting all changes
adding changesets
adding manifests
adding file changes
added 7515 changesets with 101024 changes to 51313 files
updating working directory
42507 files updated, 0 files merged, 0 files removed, 0 files unresolved
2a39f20bc20e
=== clone closed tree: ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed ===
requesting all changes
adding changesets
adding manifests
adding file changes
added 968 changesets with 8269 changes to 4389 files
updating working directory
2677 files updated, 0 files merged, 0 files removed, 0 files unresolved

~/bin/hg-clone is a simple script to get both the open and closed versions of the gate.< /p>

And reparent it to onnv-clone

[th199096@aus1500-home th199096]> ws nfs41-sync/

Workspace			: /pool/ws/th199096/nfs41-sync
Workspace Parent		: ssh://aus1500-home//pool/ws/nfs41-clone
Proto area ($ROOT)		: /pool/ws/th199096/nfs41-sync/proto/root_i386
Root of source ($SRC)		: /pool/ws/th199096/nfs41-sync/usr/src
Root of test source ($TSRC)  : /pool/ws/th199096/nfs41-sync/usr/ontest
Current directory ($PWD)	: /pool/ws/th199096/nfs41-sync

[th199096@aus1500-home nfs41-sync]> hg reparent ssh://onnv.eng//export/onnv-clone

Pull and merge

[th199096@aus1500-home nfs41-sync]> hg pull -u
pulling from ssh://onnv.eng//export/onnv-clone
searching for changes
adding changesets
adding manifests
adding file changes
added 210 changesets with 2122 changes to 1808 files (+1 heads)
not updating, since new heads added
(run 'hg heads' to see heads, 'hg merge' to merge)
[th199096@aus1500-home nfs41-sync]> hg merge
merging usr/src/cmd/Makefile
merging usr/src/cmd/zfs/zfs_iter.c
merging usr/src/cmd/zfs/zfs_main.c
merging usr/src/lib/Makefile
merging usr/src/lib/libzfs/common/libzfs.h
merging usr/src/lib/libzfs/common/libzfs_dataset.c
merging usr/src/lib/libzfs/common/mapfile-vers
merging usr/src/pkgdefs/Makefile
merging usr/src/pkgdefs/SUNWcsu/prototype_com
merging usr/src/pkgdefs/SUNWhea/prototype_com
merging usr/src/pkgdefs/etc/exception_list_sparc
merging usr/src/uts/common/Makefile.files
merging usr/src/uts/common/Makefile.rules
merging usr/src/uts/common/fs/zfs/dsl_dataset.c
merging usr/src/uts/common/fs/zfs/zfs_ioctl.c
merging usr/src/uts/common/sys/Makefile
merging usr/src/uts/common/sys/fs/zfs.h
merging usr/src/uts/intel/Makefile.intel.shared
merging usr/src/uts/intel/os/minor_perm
merging usr/src/uts/intel/os/name_to_major
merging usr/src/uts/sparc/Makefile.sparc.shared
merging usr/src/uts/sparc/os/minor_perm
merging usr/src/uts/sparc/os/name_to_major
1785 files updated, 23 files merged, 62 files removed, 0 files unresolved
(branch merge, don't forget to commit)

So, I used filemerge to do any manual editing in the above merge. It is invoked in my .hgrc:

# Merge tool
[merge-patterns]
\*\* = filemerge

[merge-tools]
filemerge.executable = /ws/onnv-tools/teamware/bin/filemerge
filemerge.args = -a $base $local $other $output
filemerge.checkchanged = true
filemerge.gui = true

Then I would commit and repeat the cycle for the closed branch:

[th199096@aus1500-home nfs41-sync]> hg commit
[th199096@aus1500-home nfs41-sync]> ws usr/closed

Workspace			: /pool/ws/th199096/nfs41-sync/usr/closed
Workspace Parent		: ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed
Proto area ($ROOT)		: /pool/ws/th199096/nfs41-sync/usr/closed/proto/root_i386
Root of source ($SRC)		: /pool/ws/th199096/nfs41-sync/usr/closed/usr/src
Root of test source ($TSRC)  : /pool/ws/th199096/nfs41-sync/usr/closed/usr/ontest
Current directory ($PWD)	: /pool/ws/th199096/nfs41-sync/usr/closed

[th199096@aus1500-home closed]> hg reparent ssh://onnv.eng//export/onnv-clone/usr/closed
[th199096@aus1500-home closed]> hg pull -u
pulling from ssh://onnv.eng//export/onnv-clone/usr/closed
searching for changes
adding changesets
adding manifests
adding file changes
added 15 changesets with 145 changes to 137 files (+1 heads)
not updating, since new heads added
(run 'hg heads' to see heads, 'hg merge' to merge)
[th199096@aus1500-home closed]> hg merge
135 files updated, 0 files merged, 10 files removed, 0 files unresolved
(branch merge, don't forget to commit)
[th199096@aus1500-home closed]> hg commit

The next step is a clone/build on one of the build machines. As this looks a lot like the cloning in this section and the build from the prior, I'm going to leave it out.

After the build and verification is done, we prepare the nfs41-gate for the integration. Because of the branch merge comments not being in the approved RTI format, we need to turn off sanity checking for this operation. Note that it is okay to keep on for developers pushing to the gate:

[th199096@aus1500-home ~]> su - nfs4hg
Password:
Sun Microsystems Inc.	SunOS 5.11	snv_92	January 2008
[nfs4hg@aus1500-home ~]> cd /pool/ws/nfs41-gate/usr/closed/.hg
[nfs4hg@aus1500-home .hg]> cp hgrc hgrc.good
[nfs4hg@aus1500-home .hg]> vi hgrc
[nfs4hg@aus1500-home .hg]> diff hgrc hgrc.good
73c73
< #pretxnchangegroup.1 = python:hook.sanity.sanity
---
> pretxnchangegroup.1 = python:hook.sanity.sanity

Reparent to the gate and push

[th199096@aus1500-home closed]> hg reparent ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate/usr/clos
ed
[th199096@aus1500-home closed]> hg push
pushing to ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate/usr/closed
searching for changes
Are you sure you wish to push? [y/N]: y
pushing to ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate/usr/closed
...
remote: Preparing gk email...
remote: ...gk email sent

Fix the .hgrc back to turn on sanity checking and repeat for the open bits.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Thursday Sep 25, 2008

Hit a Doh! whilst gatekeeping

I was trying to integrate a branch merge of the nfs41-gate and ON. And I'm keeping tight notes because I'm trying to get Piyush to be the gatekeeper. Anyway, here is what went wrong:

Reparent to the gate:

% hg reparent ssh://aus1500-home//pool/ws/nfs41-gate

And push:

% hg push
pushing to ssh://aus1500-home//pool/ws/nfs41-gate
searching for changes
Are you sure you wish to push? [y/N]: y
remote: abort: could not lock repository /pool/ws/nfs41-gate: Permission denied
abort: unexpected response: empty string

This worked last time I tried it. And Rich Brown was able to make a change this morning. Now, I could just blame him, but that is too easy. I use google and find [Volo-dev] I am unable to push my changes to the gate. I don't get past the comment:

You are trying to push back the changes as user "anon".

Before I realize what is going on. I forgot to reparent the gate correctly:

% hg reparent ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate
% hg push
pushing to ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate
searching for changes
Are you sure you wish to push? [y/N]: y
pushing to ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate
searching for changes
Are you sure you wish to push? [y/N]: y
remote: adding changesets
remote: adding manifests
...
remote: Preparing gk email...
remote: ...gk email sent

All is well and Rich is off the hook!


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily
About

tdh

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today