Thursday Nov 20, 2008

Locking down a gate/respository

So I'm not a mean gatekeeper - I try to not lock down our gate. I feel like I should just be able to ask people not to integrate anything. But the reality is that you can't count on everyone getting the message. So, you need to lock down the gate.

I knew how to do that in Teamware, but I've never had that need with Mercurial. Until today that is - a nasty branch merge with onnv_103.

So to lock down a Mercurial gate with Sun's extensions, you can use lock.py:

[nfs4hg@aus1500-home ~]> which lock.py
/pool/nfs4hg/bin/lock.py
[nfs4hg@aus1500-home nfs41-gate]> lock.py -n -R /pool/ws/nfs41-gate
[write]: None

Okay, no one has a write lock, so let's grab one:

[nfs4hg@aus1500-home nfs41-gate]> lock.py -R /pool/ws/nfs41-gate
[nfs4hg@aus1500-home nfs41-gate]> lock.py -n -R /pool/ws/nfs41-gate
[write]:
        nfs4hg
        th199096

By the way, where is this configured?

[nfs4hg@aus1500-home ~]> grep lock /pool/ws/nfs41-gate/.hg/hgrc 
#   5. lockdir must be readable by whomever will pull/push
lockdir = public/lock
wlock = nfs4hg, th199096
# These hooks check the lock before anything happens.
prechangegroup.0 = python:hook.lockchk.lockchk
# then comment it out, let them push, uncomment, and unlock the gate.

So the lock only works on the gate and not the clone. You can find more about this in the source of lock.py.

Ohh, and even though I haven't tested it, you release the gate easily enough with a unlock.py.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Wednesday Oct 08, 2008

Branch merge to a specific tagged revision

I need to do a branch merge between nfs41-gate and onnv-clone. And specifically, I want to not get the 'tip', but rather the tag for release 100. I found a good reference - Chapter 8 Managing releases and branchy development.

So I'll follow along with it. I need the tag:

[th199096@jhereg onnv-play]> hg tags
tip                             7782:716c23b2ce2e
onnv_100                        7757:bf4a45ecb669
onnv_99                         7613:e49de7ec7617
onnv_98                         7473:fad192e9bc57

It turns out I don't need much more:

[th199096@jhereg nfs41-100]> hg reparent ssh://onnv.eng//export/onnv-clone
[th199096@jhereg nfs41-100]> hg tags | more
tip                             7744:763bfa203d1a
closedv1                        7742:9fab48a31a4a
onnv_99                         7652:e49de7ec7617

So I haven't merged yet:

[th199096@jhereg nfs41-100]> hg pull -u -r onnv_100
pulling from ssh://onnv.eng//export/onnv-clone
searching for changes
adding changesets
adding manifests
adding file changes
added 64 changesets with 475 changes to 462 files (+1 heads)
not updating, since new heads added
(run 'hg heads' to see heads, 'hg merge' to merge)

The tag can be used as a revision!


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Sunday Oct 05, 2008

Learning a new language - python

So I decided to learn python - why? because it is used by Mercurial. And there is at least one of the gatekeeping scripts which I needed to hack for the nfs41-gate.

I bought Learning Python, Third Edition by Mark Lutz because the local Borders did not have Programming Python, Third Edition. From some reviews I read, it would probably have been a better fit for me.

I know I can find most of what I want on the net, but I wanted a printed resource.

Anyway, I had a question right off the bat - about whether if file a imports modules b and c, what happens if c also imports b? Deeper in the book that I've read, it does state that an import is equivalent to load the file if it is not already loaded. But that doesn't help me learn the language. :->

The following example is quite simple, but effective in answering the question for me:

a.py

> cat a.py
#!/usr/bin/python

title = "This is the file a.py!"

print title

print "importing b from a"
import b

print "importing c from a"
import c

b.py

> cat b.py
#!/usr/bin/python

title = "This is the file b.py!"

print title

c.py

> cat c.py
#!/usr/bin/python

import b

title = "This is the file c.py!"

print title

Test 1

> ./a.py
This is the file a.py!
importing b from a
This is the file b.py!
importing c from a
This is the file c.py!

So we see that if a has loaded it, then c will not. How about the other way?

a2.py

> cat a2.py
#!/usr/bin/python

title = "This is the file a.py!"

print title

print "importing c from a"
import c

print "importing b from a"
import b

print "and b's title is"
print b.title

Test 2

> ./a2.py
This is the file a.py!
importing c from a
This is the file b.py!
This is the file c.py!
importing b from a
and b's title is
This is the file b.py!

We see that c loads b and that b's attributes are visible from a.

What would really help me here is if b could state the call stack of what is importing it.

Test 3

A simple change fails:

> cat b.py
#!/usr/bin/python

title = "This is the file b.py!"

print title

print "Called from", __file__

but it does show the effect of byte code compilation:

> ./a.py
This is the file a.py!
importing b from a
This is the file b.py!
Called from /home/tdh/python/b.py
importing c from a
This is the file c.py!
> ./a2.py
This is the file a.py!
importing c from a
This is the file b.py!
Called from /home/tdh/python/b.pyc
This is the file c.py!
importing b from a
and b's title is
This is the file b.py!

I can see the "nesting" if I pop into an interactive session:

> python
>>> import a2
This is the file a.py!
importing c from a
This is the file b.py!
Called from b.pyc
This is the file c.py!
importing b from a
and b's title is
This is the file b.py!
>>> a2.__dict__.keys()
['c', 'b', 'title', '__builtins__', '__file__', '__name__', '__doc__']
>>> a2.c.__dict__.keys()
['b', 'title', '__builtins__', '__file__', '__name__', '__doc__']
>>> a2.c.b.__dict__.keys()
['__builtins__', '__name__', '__file__', '__doc__', 'title']

But this doesn't answer my question of how to figure this out recursively. I.e., I guess I am looking for a parent "pointer" and I could walk it to get my answer.

But I've still learned more than just reading the book linearly.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Friday Oct 03, 2008

How to get a Mercurial workspace after creating a ZFS clone

I wrote about how I didn't know how to mix Mercurial and ZFS data sets together to get a new clone on a new dataset. Dave Marker provided this insight:

zfs create pool/ws/th199096/spe-build
cd /pool/ws/th199096/spe-build
hg init
echo "[paths]" > .hg/hgrc
echo "default = ssh://anon@hg.opensolaris.org/hg/nfsv41/nfs41-gate" >> .hg/hgrc
hg pull -u

The trick is realizing that there is nothing magical about 'hg clone'.

And if at this point I want to do a closed gate, I can use my normal incantation because it wil be on the same dataset.

And I can then create a ZFS snapshot and clone that to my heart's desire.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Thursday Oct 02, 2008

How to tie our closed-bins to the new gate

I was trying to relax and I realized we would have an ongoing problem in keeping the new ssh://anon@hg.opensolaris.org/hg/nfsv41/nfs41-gate in sync with our copy of the closed binaries. But, I think we will be saved by a couple of things:

  1. We don't update the internal nfs41-gate automatically with every change in the onnv-gate. We actually normally sync up with the 2 week releases. This means that random changes to the closed source will not impact the osol gate. As a matter of fact, we control when a change causes a respin of the closed bits.
  2. Just like the ON gatekeepers tag their gate every 2 weeks, we could also use 'hg tag' to mark when the closed binaries changed. We could store the closed binaries on the project download page and when an external developer saw the tag change, they could then pick up a new copy

Plus with setting the mail to go out to the dev mailing list, people would be able to see a need to pickup a new set of closed binaries.

[thud@adept src]> hg incoming
comparing with ssh://anon@hg.opensolaris.org/hg/nfsv41/nfs41-gate
searching for changes
changeset:   7743:c672b1cb86be
tag:         tip
user:        Thomas Haynes 
date:        Thu Oct 02 22:28:30 2008 -0500
summary:     Added tag closedv1 for changeset 9fab48a31a4a

Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Setting up a NFS41 gate

We just opened up a new Mercurial gate of NFSv41 on OpenSolaris.org. Eventually it will automatically push changes as they occur to our gate. I also need to figure out a way to automatically update the closed-bins.

The hardest part was figuring out the naming convention. Some links of interest are Some work on libMicro; Mercurial transition notes and finally How to Use Mercurial (hg) Repositories. Look for For Project Leads: How to set up a Mercurial repository.

Update: Also, SCMVolunteers, look for Setting up a new (Mercurial) Project repository on OpenSolaris.org.

In any event, you can grab a copy of the source at:

hg clone ssh://anon@hg.opensolaris.org/hg/nfsv41/nfs41-gate

Note the lack of a double '/' after the FQDN - normally I would take that as a sign of a bug with Mercurial.

Note that while this compiles, you can't run it without a corresponding closed-bins.

Eventually, you should be able to browse the source via Cross Reference: nfs41-gate.

And a big thanks to David Marker for providing the help necessary to getting this to go live!


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Tuesday Sep 30, 2008

Why you should do basic triage before sending out a plea for help

So I think I have found a bug in our implementation of Mercurial. I'm all excited and I'm about to send the following email off to the ON gatekeepers:

Take a zfs clone of a workspace which has ssh://aus1500-home//pool/ws/th199096/spe-gate as a parent.

[th199096@jhereg ~]> zfs clone pool/builds/th199096/spe-gate@postjimw pool/builds/th199096/spe-fix
[th199096@jhereg ~]> ws /builds/th199096/spe-fix

Workspace			: /builds/th199096/spe-fix
Workspace Parent		: ssh://aus1500-home//pool/ws/th199096/spe-gate
Proto area ($ROOT)		: /builds/th199096/spe-fix/proto/root_i386
Root of source ($SRC)		: /builds/th199096/spe-fix/usr/src
Root of test source ($TSRC)  : /builds/th199096/spe-fix/usr/ontest
Current directory ($PWD)	: /builds/th199096/spe-fix

Reparent to ssh://aus1500-home//pool/ws/nfs41-clone

[th199096@jhereg spe-fix]> hg reparent ssh://aus1500-home//pool/ws/nfs41-clone
[th199096@jhereg spe-fix]> ws usr/closed/

Workspace			: /builds/th199096/spe-fix/usr/closed
Workspace Parent		: ssh://aus1500-home//pool/ws/th199096/spe-gate/usr/closed
Proto area ($ROOT)		: /builds/th199096/spe-fix/usr/closed/proto/root_i386
Root of source ($SRC)		: /builds/th199096/spe-fix/usr/closed/usr/src
Root of test source ($TSRC)  : /builds/th199096/spe-fix/usr/closed/usr/ontest
Current directory ($PWD)	: /builds/th199096/spe-fix/usr/closed

[th199096@jhereg closed]>  hg reparent ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed
[th199096@jhereg closed]> exit
exit
[th199096@jhereg spe-fix]> hg list
added:
      	usr/src/cmd/fs.d/nfs/sped/Makefile
	usr/src/cmd/fs.d/nfs/sped/sped.c
	usr/src/cmd/fs.d/nfs/sped/sped_dt.d
	usr/src/cmd/fs.d/nfs/sped/sped_server.c
	usr/src/cmd/fs.d/nfs/sped/spedaemon.c
	usr/src/cmd/fs.d/nfs/sped/spedaemon.h
	usr/src/cmd/fs.d/nfs/svc/spe.xml
	usr/src/head/rpcsvc/spe_prot.x
	usr/src/uts/common/fs/nfs/spe.c
	usr/src/uts/common/nfs/spe.h
	usr/src/uts/common/nfs/spe_attr.h
	usr/src/uts/common/nfs/spe_impl.h
modified:
        usr/src/cmd/fs.d/nfs/Makefile
	usr/src/cmd/fs.d/nfs/svc/Makefile
	usr/src/head/Makefile
	usr/src/head/rpcsvc/daemon_utils.h
	usr/src/lib/libshare/nfs/libshare_nfs.h
	usr/src/pkgdefs/SUNWhea/prototype_com
	usr/src/pkgdefs/SUNWhea/prototype_com
	usr/src/pkgdefs/SUNWnfscr/prototype_com
	usr/src/pkgdefs/SUNWnfscu/prototype_com
	usr/src/uts/common/Makefile
	usr/src/uts/common/Makefile.files
	usr/src/uts/common/dserv/dserv_mds.c
	usr/src/uts/common/fs/Makefile
	usr/src/uts/common/fs/nfs/ds_srv.c
	usr/src/uts/common/fs/nfs/nfs41_srv.c
	usr/src/uts/common/fs/nfs/nfs41_state.c
	usr/src/uts/common/fs/nfs/nfs_sys.c
	usr/src/uts/common/nfs/Makefile
	usr/src/uts/common/nfs/mds_state.h
	usr/src/uts/common/nfs/nfs4.h
	usr/src/uts/common/nfs/nfssys.h
	usr/src/uts/intel/nfs/Makefile
	usr/src/uts/sparc/nfs/Makefile
Time spent in user mode	  (CPU seconds) : 3.61s
Time spent in kernel mode (CPU seconds) : 5.44s
Total time				: 0:27.46s
CPU utilisation (percentage)		: 32.9%

End up making changes to usr/src/head/rpcsvc/ds_prot.x, usr/src/uts/common/fs/nfs/ds_srv.c, usr/src/uts/common/fs/nfs/nfs41_state.c, and usr/src/uts/common/nfs/nfs_serv_inst.h

[th199096@jhereg spe-fix]> vi usr/src/head/rpcsvc/ds_prot.x
[th199096@jhereg spe-fix]> vi usr/src/uts/common/dserv/dserv_mds.c
[th199096@jhereg spe-fix]> vi usr/src/uts/common/fs/nfs/ds_srv.c
[th199096@jhereg spe-fix]> vi usr/src/uts/common/fs/nfs/nfs41_srv.c
[th199096@jhereg spe-fix]> vi usr/src/uts/common/fs/nfs/nfs41_state.c
[th199096@jhereg spe-fix]> vi usr/src/uts/common/fs/nfs/nfs41_state.c
[th199096@jhereg spe-fix]> pushd usr/src/uts/common/fs/nfs
/builds/th199096/spe-fix/usr/src/uts/common/fs/nfs /builds/th199096/spe-fix
[th199096@jhereg nfs]> grep ds_guid_info_idx \*
ds_srv.c:	    mds_server->ds_guid_info_idx,
nfs41_state.c:rfs4_index_t \*ds_guid_info_idx;
nfs41_state.c:	instp->ds_guid_info_idx = rfs4_index_create(instp->ds_guid_info_tab,
[th199096@jhereg nfs]> vi ds_srv.c
[th199096@jhereg nfs]> popd
/builds/th199096/spe-fix
[th199096@jhereg spe-fix]> vi usr/src/uts/common/fs/nfs/nfs41_state.c
[th199096@jhereg spe-fix]> vi usr/src/uts/common/fs/nfs/nfs41_state.c\\
?
[th199096@jhereg spe-fix]> vi usr/src/uts/common/nfs/nfs_serv_inst.h

Reparent back to ssh://aus1500-home//pool/ws/th199096/spe-gate

[th199096@jhereg spe-fix]> hg reparent ssh://aus1500-home//pool/ws/th199096/spe-gate
[th199096@jhereg spe-fix]> ws usr/closed/

Workspace			: /builds/th199096/spe-fix/usr/closed
Workspace Parent		: ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed
Proto area ($ROOT)		: /builds/th199096/spe-fix/usr/closed/proto/root_i386
Root of source ($SRC)		: /builds/th199096/spe-fix/usr/closed/usr/src
Root of test source ($TSRC)  : /builds/th199096/spe-fix/usr/closed/usr/ontest
Current directory ($PWD)	: /builds/th199096/spe-fix/usr/closed

[th199096@jhereg closed]> hg reparent ssh://aus1500-home//pool/ws/th199096/spe-gate/usr/closed
[th199096@jhereg closed]> exit
exit

And no changes ???

[th199096@jhereg spe-fix]> hg outgoing
comparing with ssh://aus1500-home//pool/ws/th199096/spe-gate
searching for changes
no changes found
[th199096@jhereg spe-fix]> hg push
pushing to ssh://aus1500-home//pool/ws/th199096/spe-gate
searching for changes
no changes found
[th199096@jhereg spe-fix]>

Up until I've just been cutting and pasting what had happened, now I start to debug to show that I'm not just emailing without trying anything:

Hmm, I happen to have a copy of that gate on this machine:

[th199096@jhereg spe-fix]> diff usr/src/head/rpcsvc/ds_prot.x ../spe-gate/usr/src/head/rpcsvc/ds_prot.x
255d254
<	utf8string	ds_path;
[th199096@jhereg spe-fix]> diff usr/src/uts/common/dserv/dserv_mds.c ../spe-gate/usr/src/uts/common/dserv/dserv_mds.c
[th199096@jhereg spe-fix]> diff usr/src/uts/common/fs/nfs/ds_srv.c ../spe-gate/usr/src/uts/common/fs/nfs/ds_srv.c
139,141d138
<			UTF8STRING_FREE(res_ok->guid_map.guid_map_val[i].
<                           ds_path);
<
711,713d707
<			(void) utf8_copy(&pip->ds_path,
<                           &guid_map[count].ds_path);
<
[th199096@jhereg spe-fix]> diff usr/src/uts/common/fs/nfs/nfs41_srv.c ../spe-gate/usr/src/uts/common/fs/nfs/nfs41_srv.c
[th199096@jhereg spe-fix]> diff usr/src/uts/common/fs/nfs/nfs41_state.c ../spe-gate/usr/src/uts/common/fs/nfs/nfs41_state.c
1919c1919
<  \* this will populate the following MDS tables.
---
>  \* this will populste the following MDS tables.
2016c2016
<	ds_guid_info_t		\*pip;
---
>	mds_pool_info_t		\*pip;
2022c2022
<	rw_enter(&ds_guid_info_lock, RW_READER);
---
>	rw_enter(&mds_pool_info_lock, RW_READER);
2024c2024
<	pip = (ds_guid_info_t \*)rfs4_dbsearch(ds_guid_info_path_idx,
---
>	pip = (mds_pool_info_t \*)rfs4_dbsearch(mds_pool_info_path_idx,
2031c2031
<	rw_exit(&ds_guid_info_lock);
---
>	rw_exit(&mds_pool_info_lock);
2119c2119
< ds_guid_info_path_compare(rfs4_entry_t entry, void \*key)
---
> mds_pinfo_path_compare(rfs4_entry_t entry, void \*key)
2121c2121
<	ds_guid_info_t \*pip = (ds_guid_info_t \*)entry;
---
>	mds_pool_info_t \*pip = (mds_pool_info_t \*)entry;
2127c2127
< ds_guid_info_path_mkkey(rfs4_entry_t entry)
---
> mds_pinfo_path_mkkey(rfs4_entry_t entry)
2129c2129
<	ds_guid_info_t \*pip = (ds_guid_info_t \*)entry;
---
>	mds_pool_info_t \*pip = (mds_pool_info_t \*)entry;
2433,2438d2432
<	instp->ds_guid_info_path_idx =
<	    rfs4_index_create(instp->ds_guid_info_tab,
<	    "DS_guid_path-idx", ds_guid_info_hash, ds_guid_info_path_compare,
<	    ds_guid_info_path_mkkey,
<	    TRUE);
<
[th199096@jhereg spe-fix]> diff usr/src/uts/common/nfs/nfs_serv_inst.h ../spe-gate/usr/src/uts/common/nfs/nfs_serv_inst.h
193d192
<	rfs4_index_t \*ds_guid_info_path_idx;
[th199096@jhereg spe-fix]>

So the files are different

[th199096@jhereg spe-fix]> diff usr/src/uts/common/nfs/nfs_serv_inst.h /net/aus1500-home/pool/ws/th199096/spe-gate/usr/src/uts/common/nfs/nfs_serv_inst.h
193d192<	rfs4_index_t \*ds_guid_info_path_idx;

Yes, really they are

Why doesn't Mercurial think so?

It knows that the files have changed

[th199096@jhereg spe-fix]> hg list
modified:
        usr/src/head/rpcsvc/ds_prot.x
	usr/src/uts/common/fs/nfs/ds_srv.c
	usr/src/uts/common/fs/nfs/nfs41_state.c
	usr/src/uts/common/nfs/nfs_serv_inst.h

[th199096@jhereg spe-fix]> hg outgoing
comparing with ssh://aus1500-home//pool/ws/th199096/spe-gate
searching for changes
no changes found

And here the email stops as I RTFM and realize I forgot a step:

[th199096@jhereg spe-fix]> hg commit
[th199096@jhereg spe-fix]> hg outgoing
comparing with ssh://aus1500-home//pool/ws/th199096/spe-gate
searching for changes
changeset:   7779:ced6eccb4366
tag:		tip
user:		Thomas Haynes 
date:		Tue Sep 30 14:57:46 2008 -0500
summary:	Fix up for new NFS instances

Without a hg commit, even with changes, there is nothing to integrate back to the parent.

So, instead of an email, I blog about it...

[th199096@jhereg spe-fix]> hg push
pushing to ssh://aus1500-home//pool/ws/th199096/spe-gate
searching for changes
Are you sure you wish to push? [y/N]: y
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 4 changes to 4 files

Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

branch gatekeeping 101

So I maintain nfs41-gate which is a development branch of onnv-gate. With the introduction of Mercurial as our version control system, my life has changed a bit, but the basic tasks for gatekeeping stay the same:

Build binaries
When developers change the source base, build new BFU bits for QA and/or other developers. I.e., reference bits without anyone else's changes present.
Branch merge with the ON gate
When something we want is introduced into ON or we don't want to drift too far, we sync up with onnv-gate.

I've noticed that I find merging to be easier with Mercurial, so it tends to happen more often.

Building Binaries

I typically already have an existing workspace and I'll do an incremental build in it. Also, I do this for both sparc and i386. I don't have to worry about conflicts or merging since nothing changes in the child. A typical session would be:

[th199096@aus-build-x86 ~]> ws /builds/th199096/nfs41-gk

Workspace                    : /builds/th199096/nfs41-gk
Workspace Parent             : ssh://aus1500-home//pool/ws/nfs41-clone
Proto area ($ROOT)           : /builds/th199096/nfs41-gk/proto/root_i386
Root of source ($SRC)        : /builds/th199096/nfs41-gk/usr/src
Root of test source ($TSRC)  : /builds/th199096/nfs41-gk/usr/ontest
Current directory ($PWD)     : /builds/th199096/nfs41-gk

[th199096@aus-build-x86 nfs41-gk]>  hg pull -u
pulling from ssh://aus1500-home//pool/ws/nfs41-clone
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 93 changes to 93 files
93 files updated, 0 files merged, 0 files removed, 0 files unresolved
[th199096@aus-build-x86 nfs41-gk]> ws usr/closed/

Workspace                    : /builds/th199096/nfs41-gk/usr/closed
Workspace Parent             : ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed
Proto area ($ROOT)           : /builds/th199096/nfs41-gk/usr/closed/proto/root_i386
Root of source ($SRC)        : /builds/th199096/nfs41-gk/usr/closed/usr/src
Root of test source ($TSRC)  : /builds/th199096/nfs41-gk/usr/closed/usr/ontest
Current directory ($PWD)     : /builds/th199096/nfs41-gk/usr/closed

[th199096@aus-build-x86 closed]> hg pull -u
pulling from ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 3 changes to 3 files
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
[th199096@aus-build-x86 closed]> exit
exit
[th199096@aus-build-x86 nfs41-gk]> `which nightly` -in nightly.env 

At which point I go do something else, like blog about what I am doing.

I said typical, except that this change set (which gets rid of auth records in the MDS as a byproduct) happens to have touched something in closed. We rarely seem to make changes there.

Okay, the build is done (remember to check the logs) and in this case it did not fail. So push the BFU bits out and then send out email telling people about the new reference bits.

[th199096@aus-build-x86 nfs41-gk]> ~/gk/hg-push.sh nfs41-gk i386 2008-09-30
ARCH=i386
BASE=/builds/th199096
AUS=/net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386
NIGHTLY=/builds/th199096/nfs41-gk/archives/i386/nightly
NIGHTLY_ND=/builds/th199096/nfs41-gk/archives/i386/nightly-nd
DATER=/builds/th199096/nfs41-gk/archives/i386/2008-09-30
DATER_ND=/builds/th199096/nfs41-gk/archives/i386/2008-09-30-nd
+ mv /builds/th199096/nfs41-gk/archives/i386/nightly /builds/th199096/nfs41-gk/archives/i386/2008-09-30 
+ mv /builds/th199096/nfs41-gk/archives/i386/nightly-nd /builds/th199096/nfs41-gk/archives/i386/2008-09-30-nd 
+ cp -r /builds/th199096/nfs41-gk/archives/i386/2008-09-30 /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386/2008-09-30 
+ cp -r /builds/th199096/nfs41-gk/archives/i386/2008-09-30-nd /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386/2008-09-30-nd 
+ rm /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386/latest /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386/latest-nd 
+ cd /net/aus1500-home.central/pool/ws/nfs41-gate-hg-archives/i386 
+ ln -s 2008-09-30 latest 
+ ln -s 2008-09-30-nd latest-nd 

Branch merging

This case is a bit more complicated and can be summarized by:

  1. Take a clone of nfs41-clone - call it nfs41-sync
  2. Reparent it to onnv-clone (onnv-gate is write only)
  3. Pull across the changes and merge them.
    This is actually the only difficulty in the entire process
  4. On sparc and x86 build boxes, get clones of nfs41-sync and do full builds.
    Incrementals are not sufficient in this case.
  5. Populate a pNFS community (client, DS, and MDS) with these changes and make sure the Connectathon tests all pass.
    Depending on the scope of the changes and/or the difficulty of the merge, we may skip this item. -- pure judgment call. Also, note that these clones will become the basis for future incremental builds as described in the previous section.
  6. If there have been further integrations to nfs41-gate, reparent to nfs41-clone (which is automatically kept up to date with nfs41-gate), and do the pull/merge cycle until everything is up to date. You may have to rebuild and retest. Much easier with a small group of developers to ask them not to integrate.
  7. ZFS snapshot nfs41-gate to make rolling back the changes easier.
  8. Reparent nfs41-sync to nfs41-gate and integrate the changes.

By not changing the nfs41-gate until the final moment, I can throw everything away if needed. And believe me, as painful as that is, I've done it. Also, note that when I talk about a workspace above, I am also taling about working in parallel with the closed version of it.

But now onto a detailed example:

Get a backup snapshot of the gate:

[th199096@aus1500-home ~]> zfs snapshot pool/ws/nfs41-clone@sync99

Now grab your copy for merging

[th199096@aus1500-home ~]> cd /pool/ws/th199096/
[th199096@aus1500-home th199096]> ~/bin/hg-clone ssh://aus1500-home//pool/ws/nfs41-clone nfs41-syn
c
397b36b5473d
=== clone open tree: ssh://aus1500-home//pool/ws/nfs41-clone ===
requesting all changes
adding changesets
adding manifests
adding file changes
added 7515 changesets with 101024 changes to 51313 files
updating working directory
42507 files updated, 0 files merged, 0 files removed, 0 files unresolved
2a39f20bc20e
=== clone closed tree: ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed ===
requesting all changes
adding changesets
adding manifests
adding file changes
added 968 changesets with 8269 changes to 4389 files
updating working directory
2677 files updated, 0 files merged, 0 files removed, 0 files unresolved

~/bin/hg-clone is a simple script to get both the open and closed versions of the gate.< /p>

And reparent it to onnv-clone

[th199096@aus1500-home th199096]> ws nfs41-sync/

Workspace			: /pool/ws/th199096/nfs41-sync
Workspace Parent		: ssh://aus1500-home//pool/ws/nfs41-clone
Proto area ($ROOT)		: /pool/ws/th199096/nfs41-sync/proto/root_i386
Root of source ($SRC)		: /pool/ws/th199096/nfs41-sync/usr/src
Root of test source ($TSRC)  : /pool/ws/th199096/nfs41-sync/usr/ontest
Current directory ($PWD)	: /pool/ws/th199096/nfs41-sync

[th199096@aus1500-home nfs41-sync]> hg reparent ssh://onnv.eng//export/onnv-clone

Pull and merge

[th199096@aus1500-home nfs41-sync]> hg pull -u
pulling from ssh://onnv.eng//export/onnv-clone
searching for changes
adding changesets
adding manifests
adding file changes
added 210 changesets with 2122 changes to 1808 files (+1 heads)
not updating, since new heads added
(run 'hg heads' to see heads, 'hg merge' to merge)
[th199096@aus1500-home nfs41-sync]> hg merge
merging usr/src/cmd/Makefile
merging usr/src/cmd/zfs/zfs_iter.c
merging usr/src/cmd/zfs/zfs_main.c
merging usr/src/lib/Makefile
merging usr/src/lib/libzfs/common/libzfs.h
merging usr/src/lib/libzfs/common/libzfs_dataset.c
merging usr/src/lib/libzfs/common/mapfile-vers
merging usr/src/pkgdefs/Makefile
merging usr/src/pkgdefs/SUNWcsu/prototype_com
merging usr/src/pkgdefs/SUNWhea/prototype_com
merging usr/src/pkgdefs/etc/exception_list_sparc
merging usr/src/uts/common/Makefile.files
merging usr/src/uts/common/Makefile.rules
merging usr/src/uts/common/fs/zfs/dsl_dataset.c
merging usr/src/uts/common/fs/zfs/zfs_ioctl.c
merging usr/src/uts/common/sys/Makefile
merging usr/src/uts/common/sys/fs/zfs.h
merging usr/src/uts/intel/Makefile.intel.shared
merging usr/src/uts/intel/os/minor_perm
merging usr/src/uts/intel/os/name_to_major
merging usr/src/uts/sparc/Makefile.sparc.shared
merging usr/src/uts/sparc/os/minor_perm
merging usr/src/uts/sparc/os/name_to_major
1785 files updated, 23 files merged, 62 files removed, 0 files unresolved
(branch merge, don't forget to commit)

So, I used filemerge to do any manual editing in the above merge. It is invoked in my .hgrc:

# Merge tool
[merge-patterns]
\*\* = filemerge

[merge-tools]
filemerge.executable = /ws/onnv-tools/teamware/bin/filemerge
filemerge.args = -a $base $local $other $output
filemerge.checkchanged = true
filemerge.gui = true

Then I would commit and repeat the cycle for the closed branch:

[th199096@aus1500-home nfs41-sync]> hg commit
[th199096@aus1500-home nfs41-sync]> ws usr/closed

Workspace			: /pool/ws/th199096/nfs41-sync/usr/closed
Workspace Parent		: ssh://aus1500-home//pool/ws/nfs41-clone/usr/closed
Proto area ($ROOT)		: /pool/ws/th199096/nfs41-sync/usr/closed/proto/root_i386
Root of source ($SRC)		: /pool/ws/th199096/nfs41-sync/usr/closed/usr/src
Root of test source ($TSRC)  : /pool/ws/th199096/nfs41-sync/usr/closed/usr/ontest
Current directory ($PWD)	: /pool/ws/th199096/nfs41-sync/usr/closed

[th199096@aus1500-home closed]> hg reparent ssh://onnv.eng//export/onnv-clone/usr/closed
[th199096@aus1500-home closed]> hg pull -u
pulling from ssh://onnv.eng//export/onnv-clone/usr/closed
searching for changes
adding changesets
adding manifests
adding file changes
added 15 changesets with 145 changes to 137 files (+1 heads)
not updating, since new heads added
(run 'hg heads' to see heads, 'hg merge' to merge)
[th199096@aus1500-home closed]> hg merge
135 files updated, 0 files merged, 10 files removed, 0 files unresolved
(branch merge, don't forget to commit)
[th199096@aus1500-home closed]> hg commit

The next step is a clone/build on one of the build machines. As this looks a lot like the cloning in this section and the build from the prior, I'm going to leave it out.

After the build and verification is done, we prepare the nfs41-gate for the integration. Because of the branch merge comments not being in the approved RTI format, we need to turn off sanity checking for this operation. Note that it is okay to keep on for developers pushing to the gate:

[th199096@aus1500-home ~]> su - nfs4hg
Password:
Sun Microsystems Inc.	SunOS 5.11	snv_92	January 2008
[nfs4hg@aus1500-home ~]> cd /pool/ws/nfs41-gate/usr/closed/.hg
[nfs4hg@aus1500-home .hg]> cp hgrc hgrc.good
[nfs4hg@aus1500-home .hg]> vi hgrc
[nfs4hg@aus1500-home .hg]> diff hgrc hgrc.good
73c73
< #pretxnchangegroup.1 = python:hook.sanity.sanity
---
> pretxnchangegroup.1 = python:hook.sanity.sanity

Reparent to the gate and push

[th199096@aus1500-home closed]> hg reparent ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate/usr/clos
ed
[th199096@aus1500-home closed]> hg push
pushing to ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate/usr/closed
searching for changes
Are you sure you wish to push? [y/N]: y
pushing to ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate/usr/closed
...
remote: Preparing gk email...
remote: ...gk email sent

Fix the .hgrc back to turn on sanity checking and repeat for the open bits.


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily

Thursday Sep 25, 2008

Hit a Doh! whilst gatekeeping

I was trying to integrate a branch merge of the nfs41-gate and ON. And I'm keeping tight notes because I'm trying to get Piyush to be the gatekeeper. Anyway, here is what went wrong:

Reparent to the gate:

% hg reparent ssh://aus1500-home//pool/ws/nfs41-gate

And push:

% hg push
pushing to ssh://aus1500-home//pool/ws/nfs41-gate
searching for changes
Are you sure you wish to push? [y/N]: y
remote: abort: could not lock repository /pool/ws/nfs41-gate: Permission denied
abort: unexpected response: empty string

This worked last time I tried it. And Rich Brown was able to make a change this morning. Now, I could just blame him, but that is too easy. I use google and find [Volo-dev] I am unable to push my changes to the gate. I don't get past the comment:

You are trying to push back the changes as user "anon".

Before I realize what is going on. I forgot to reparent the gate correctly:

% hg reparent ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate
% hg push
pushing to ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate
searching for changes
Are you sure you wish to push? [y/N]: y
pushing to ssh://nfs4hg@aus1500-home//pool/ws/nfs41-gate
searching for changes
Are you sure you wish to push? [y/N]: y
remote: adding changesets
remote: adding manifests
...
remote: Preparing gk email...
remote: ...gk email sent

All is well and Rich is off the hook!


Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily
About

tdh

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today