Wednesday Jun 10, 2009

Comparing dtrace output using meld

Comparing dtrace and other debug logs using meld

meld is a powerful OpenSource graphical "diff" viewer. It is available from the OpenSolaris IPS repositories so can bee installed from the packagemanager in OpenSolaris or simply by typing

   $ pfexec pkg install SUNWmeld

It is very clever at identifying the real changes within files and highlighting where the difference start and end.


This example is using the output of two dtrace runs, tracing receive network activity in the a development version of the nge driver, once when it works, and once when it doesn't, trying to identify a bug in a development version.

First off the dtrace script is

   $ cat rx.d 
   #!/usr/sbin/dtrace -Fs
   fbt::nge_receive:entry
   {
       self->trace=1;
   }
   
   fbt::nge_receive:return
   /self->trace==1/
   {
       self->trace=0;
   }
   
   fbt:::entry
   /self->trace==1/
   {
       printf("%x",arg0);
   }
   
   fbt:::return
   /self->trace == 1/
   {
       printf("%x",arg0);
   }

This very simply traces all function calls from the nge_receive() function.


So I ran it twice, once when the network was working, and once when it wasn't and simply ran meld over the two files.

   $ meld rx.out rx.works.out

This throws up a gui as seen here





It's worth loading full size. What you see is on the right a large area of probes that have fired that do not exist within the one on the left. That implies a lot of code run in the working case that is missing from the failing case.

This is a picture of source code of nge_receive()



You can see it essentially does two things

   o Calls nge_recv_ring()
   o If that succeeds calls mac_rx()

Looking at the meld screenshot you can see the big green area starts at mac_rx. So in the failing case nge_receive() doesn't call mac_rx() (that'd explain why it fails to receive a packet).

Why doesn't it? Well it implies that nge_recv_ring() returns NULL. nge_recv_ring() is supposed to return an mblk, and it hasn't. Why is that? well looking in to the blue and red highlighted area in the meld window, we see another area in the working case that is missing in the failing case. Hey presto, this bit is the call to allocb(). allocb() is used to allocate an mblk.

So we know in the failing case the nge_recv_ring() function fails to allocate an mblk. Now just need to work out why.

I found this a powerful way of viewing complex data and quickly homing in on differences.

Monday Jun 01, 2009

It's Finally Here

I've decided I really need to get back to writing a blog occasionally, and what better day to choose than June 1 2009. Why? Well today we release OpenSolaris 2009.06, the latest OpenSource release of our operating system Solaris.

I know this all sounds a bit marketing, but actually there are some really good reasons for running OpenSolaris on your own machine.

First off, it is the most secure OS I know of. No need to Virus protection.

Second, it just works (mostly). I've just got a new Macbook Pro, I always find it easier to do development work on Solaris than any other platform so I like to run OpenSolaris. It installed pretty much seamlessly (just having to change the EFI disk label using the macOS fdisk utility as described here). The only thing that doesn't work out of the box is the Wifi - which is a pain. It's a broadcom chipset so I've got hold of a PCI3/4 Atheros card which works well

Third, all the development tools I need (and indeed anyone developing for or on Solaris) are available within the standard repositories. I found this page which is how I set up my laptop as a build machine.

From a day to day computing perspective it does everything I need. Mail, Web, chat all included, OpenOffice in the repositories for free (and simple) download. A new Media player in Elisa (in the repo), though unfortunately you have to buy the codecs for many common video formats.

So the next questions is, is it any different from 2008.11? Well it's hard for me to say as I've been upgrading every few weeks to the latest development builds (by using the opensolaris.com/dev repository). But I did install it fresh in side a VirtualBox VM and was impressed with the speed of the install. The auto installer is now more complete and can install SPARC machines (necessary for a good proportion of our customers). There are networking improvements, but generally the speed and usability is what you'll notice.

Oh and Fast reboot. Makes it much quicker to shutdown or reboot a machine.

Today I'm attending Comunity One (or C1 as we call it) and much more will be discussed about OpenSolaris and all our other OpenSource development efforts. I'll try to remember to write a blog about it (though don't hold your breath on recent form :-)




Monday Sep 22, 2008

In praise of the blogosphere - Upgrading OpenSolaris past build 91

I can't believe it's 4 months since I last blogged. Oh well it's been a hectic time. And this is just a quick not for my own memory jogging purposes really. I have been having a pain of a time upgrading from OpenSolaris build 91. I've been away alot (fantastic holiday thanks very much) and had been working on my laptop under VirtualBox. Any way. I would do my usual

$ pfexec pkg refresh --full
$ pfexec pkg install SUNWipkg
$ pfexec pkg image-update

However at the end of downloading all the packages It would give me an error like

pkg: attempt to mount opensola���2 failed.
pkg: image-update cannot be done on live image

I hadn't really given it much thought for a while but when I did google the message it took me straight to this page

http://louisbotterill.blogspot.com/2008/07/open-solaris-to-b93.html

Which pointed out the missing information(can't workout how I missed it - may be it was while I was away)

http://defect.opensolaris.org/bz/show_bug.cgi?id=2387

So any way upgrading it on a copy of the BE really helped and I am now on the latest bits. Thanks to Louise Botterill for the tips






Tuesday May 20, 2008

Interesting view on making money in business

I had the chance to be a key note speaker at PROMISE (and ICSE workshop) last week. The group uses datamining and A/I techniques to look for patterns and make predictions on a variety of things, like where in code defects might occur, or how much effort a project might entail. 

I was there because we had  a project a few years a go running to predict which bugs might cause customer escalations, using similar techniques. I was responsible for implementing the fixing of these bugs proactively. My talk was geared around how to put a business case together and run such a project, and ultimately why in this case it was wound up before major benefits were realised.

There were loads of other great talks and papers, but the other keynote speaker, Murray Cantor from IBM had some interesting points, one of which I wanted to pull out here.

He said that there are three things you can monetize. Innovation, Customer Relationships, and Cost structure. For example, you can Make money by having the first product to market, or a good close relationship with the customer or buy doing it cheaper than anyone else. He  drew this in a triangle like this

 Monetize Triange


So this got me thinking as to where Sun fits in to the picture. First off I'd say it's a different place from IBM who put huge resources in to having a close relationship with the customer (Murray indicated he felt IBM was somewhere on the Innovation/Customer line). However It isn't purely at the Innovation point either. We provide Innovative technologies to help lower costs both for our customers (hey Free Software anyone - check out http://opensolaris.com), but also by automating things like system management thus removing cost and complexity (Take a look at our xVM strategy to merge virtualization and system management at http://openxvm.org, thus removing some of the headaches to running a virtual data center). Oh and did I mention our coolthreads hardware? So I think we're probably somewhere between the Cost and Innovation points.

I'm not saying this is a full theory of business, but I found it a useful thought experiment to see the different value propositions of various companies business models.


Thursday Feb 14, 2008

Debugging sparc really (and I do mean really) early boot problems

For some work I've been doing I've had to work out how to debug the sparc boot process, before you can get to kmdb. And yes you can do it, it's just not that easy. So I thought I'd put it on my blog, in case I lose the notes I made in a mail to myself, and it might be of interest to some of you.

First off get as much of the diagnostics available from the OBP as possible 

 {1} ok setenv fcode-debug? true
fcode-debug? =          true
{1} ok
{1} ok setenv diag-switch? true
diag-switch? =          true
{1} ok reset-all

The reset-all is important as it saves the options the the nvram.

Now we try and boot it up - before anything is loaded. Note this requires a debug kernel, but if you're playing in this space and you're on sparc then you probably know that already

{ 1} ok boot disk0 -F kernel/unix -H 

You will see the boot fail like this

Rebooting with command: boot disk0 -F kernel/unix -H                 
Boot device: /pci@1c,600000/scsi@2/disk@0,0  File and args: -F kernel/unix -H
ufs-file-system
Halted with -H flag.
Warning: Fcode sequence resulted in a net stack depth change of 1

The file just loaded does not appear to be executable.

This is expected and how we get to start playing with breakpoints really early on. Note the unix module is not yet loaded so we now have to load the unix module. To do this we load the boot forth code and copy what it does

{1} ok see do-boot
: do-boot  
   parse-bootargs halt? l->n if   
      " Halted with -H flag. " type cr exit
   then  get-bootdev load-pkg mount-root zflag? nested? invert and
   l->n if   
      fs-name$ open-zfs-fs
   then  load-file setup-props exec-file
;

So by copying what do-boot does we can intercept the boot process

{1} ok get-bootdev load-pkg mount-root
{1} ok load-file setup-props
Loading: /platform/SUNW,Sun-Fire-V240/kernel/unix
Loading: /platform/sun4u/kernel/unix

{1} ok 

Now we can start some more magic. A DEBUG kernel will check the stop-me property in kobj_start(). This is something we have to populated in the boor properties which is why we've done all this messing around to get to this point

{1} ok cd /chosen
{1} ok 00 0 " stop-me" property
{1} ok .properties
stop-me                 
fs-package               ufs-file-system
whoami                   /platform/sun4u/kernel/unix
impl-arch-name           SUNW,Sun-Fire-V240
elfheader-length         001c55c0
elfheader-address        51000000
bootfs                   fed85a80
fstype                   ufs
bootargs                 -F kernel/unix -H
bootpath                 /pci@1c,600000/scsi@2/disk@0,0:a
mmu                      fff74080
memory                   fff74290
stdout                   fed97b90
stdin                    fed97ea8
stdout-#lines            ffffffff
name                     chosen

We can now start the boot process using exec-file. It will stop immediately because of the stop-me property (ctrace gives me the stacktrace)

{1} ok exec-file
Type  'go' to resume
{1} ok ctrace
PC: 0000.0000.f004.81e4
Last leaf: jmpl  0000.0000.f005.d274   from 0000.0000.0100.8aec client_handler+70 
     0 w  %o0-%o7: (f0000000 16 f0000000 6d 73 6 fedcb441 1008aec )

call 0000.0000.0106.bea8 p1275_sparc_cif_handler        from 0000.0000.0106.7de8 prom_enter_mon+24 
     1 w  %o0-%o7: (f005d274 fedcbda8 1839400 106af00 185fc00 f005d274 fedcb4f1 1067de8 )

call 0000.0000.0106.7dc4 prom_enter_mon        from 0000.0000.0101.9ed4 kobj_start+30 
     2 w  %o0-%o7: (0 10bdaf0 f002d224 1 1817700 1821dd8 fedcb5c1 1019ed4 )

call 0000.0000.0101.9ea4 kobj_start        from 0000.0000.0100.7ac8 _start+10 
     3 w  %o0-%o7: (f005d274 0 0 0 10bd800 181fc00 fedcb701 1007ac8 )

From this point we have access to the unix symbols and can start setting break points. For example

{1} ok load_primary +bp
{1} ok go
0000.0000.010a.c7b0 load_primary         save        %o6, ffffffffffffff30, %o6
{1} ok ctrace
PC: 0000.0000.010a.c7b0 load_primary    
Last leaf: call 0000.0000.010a.c7b0 load_primary        from 0000.0000.010a.b46c kobj_init+d8 
     0 w  %o0-%o7: (1879400 0 fedcbe78 184f000 1879340 181ac00 fedcb111 10ab46c )

call 0000.0000.010a.b394 kobj_init        from 0000.0000.0101.9fd0 kobj_start+12c
     1 w  %o0-%o7: (f005d274 185c800 184f000 fedcbe78 184f3f8 184e400 fedcb5c1 1019fd0 )

call 0000.0000.0101.9ea4 kobj_start        from 0000.0000.0100.7ac8 _start+10 
     2 w  %o0-%o7: (f005d274 7 0 51000040 51000000 51000040 fedcb701 1007ac8 )

I'm interested in getting some more module loading debug info out so lets set moddebug to 0xf

{1} ok moddebug l?
0

(displays current value of a long)

{1} ok F moddebug l!
{1} ok moddebug l?
f
{1} ok

(set the long to be F then display it again)

Now lets see what additional info I get

 {1} ok go
/kernel/fs/sparcv9/specfs symbol _info multiply defined
/kernel/fs/sparcv9/specfs symbol _init multiply defined
Returned from _info, retval = 1
init_stubs: couldn't find symbol in module fs/specfs
(Can't load specfs) Program terminated

OK That doesn't tell me much more but you get the idea. You can access the symbols - set break points, set variables. In addition you can  dump out memory with dump, single step with step and loads of other things that you might want to do, but this at least will act as a memory jogger for me

Let me know if you found this useful.

Chris















Thursday Nov 01, 2007

Installing Indiana/Opensolaris

For a few days recently I have been looking at the future of packaging, pkg(5) or IPS. IPS looks really powerful and quite simple. It will allow us to generate fixes and deliver them much more simply. What I've been thinking about is how and when will we generate fixes using this mechanism.

Any way as a result I've signed up for pkg-discuss-AT-opensolaris-DOT-org and indiana-discuss-AT-opensolaris.org. Both of these are very active and full of interesting discussions (and arguments) and ideas. Anyway, it's not surprising there has been so much activity recently. Today indiana-discuss announced the launch of the developer preview of the opensolaris binary distribution. So I tried it out on a couple of machines. My laptop first, an Acer Ferrari 4005. Everything just worked. The LiveCD booted up, really quickly actually, well done the team for getting the performance up so well. Even wireless worked, though that's probably because I've already swapped the Broadcom wireless miniPCI card for an Atheros one. Unfortunately I have no spare slices available on the laptop so I move on to my next machine.

This is my home PC, usually running WindowsXP for the kids, it has never successfully Solaris for reasons that will become apparent. I have just upgraded the hard drive so theres 60Gb partition free for me to do some damage.

Booting the livecd failed, or rather Xorg failed to display anything. My machine is an old Athlon XP2600 with an AGP radeon x1600pro graphics card. Great for games, but unfortunately the Solaris/OpenSolaris Radeon driver doesn't support it. Fortunately Stephan Hahn blogged about how to get Xorg to use the vesa driver from the livecd. With that in place I got the gnome gui up and gave the install a go.

The installer uses dwarf-caiman, a cut down slim line installer which is nice and easy to navigate. The install itself was really quick - there's only a CD's worth installed. The rest should be added later over the web from the IPS repository. Unfortunately that is where my old machine creaked too much. The onboard ethernet is an nforce2 gigabit ethernet. It should work with the nge driver but I think it's just too old. I tried adding an alias for it using 

# add_drv '"pci10de,66"' nge

 
But even though I could plumb it there was no traffic going through it :-( I guess I'll have to find another ethernet card.

The install claimed it failed, but it did come up fine after a reboot, though I had to add a user again at single user because the useradd hadn't worked. Warning here. root is just a role that users can take on - so you can't log in as root as you might expect from a "normal" solaris system.

I'm pretty impressed. Nice installer, lightweight liveCD to get you started. zfs root and pkg(5) to add new stuff (or it will when I get a new ethernet adapter. I wonder if I can get one of my old USB wireless sticks to work :-). Plus it seems to be more responsive under OpenSolaris than windows XP.

 Do give it a go, it is one vision of the future of opensolaris

 

Chris 

 



 

Monday Sep 24, 2007

On the road again

So I'm on the road again. The Sun Tech Days this times I'm in Rome and Milan is later in the week. I've just talked about "What is OpenSolaris" and "OpenSolaris Virtualization"

It's great to connect with real people who do or want to use OpenSolaris and interest in  out xVM and Zones based technologies.

Please take a look at the link and the presentations should be uploaded in the next few days.

Any way - great to be in Rome, just wish I was closer to the center. I made the trek in to see the Colosseum. Some say it's not as impressive as they were expecting. I have to say I had no expectations and was mightily impressed.  I will post a link to some photos when I've uploaded and checked them

 


 

 

 

 

Thursday Jul 19, 2007

Starting out with Solaris on Xen

As you may have seen from the announcement and John's blog we have a new set of Solaris on Xen bits available for download. A lot has changed in the (almost) year since the last drop. Certainly things are a lot easier set up than they were back then.

First big difference I notice is that you can install these bits straight from the DVD which means no mucking around with bfu.

Once it is installed also you have the joys of much newer Solaris builds including improvements to networking and removable media (but that isn't the point of this post).

Of course the thing you really want to do is run multiple operating systems so (while there are documents here I always think it's nice to see peoples use cases. Find out how they got things working.

I'm going to use zfs for storage so I made sure I had a large amount of space available for a zpool

# zpool create guests c2d0s7

First gotcha. After install the default boot entry in the grub menu.lst is for solaris on metal (ie not booting under Xen). You can change that before rebooting or select Solaris dom0 from the grub menu.

Check you are running under Xen by looking at uname -i

dominion# uname -i
i86xpv

(dominion is the name of my host)

If that says i86pc then you're not booted under Xen, i86xpv is the new platform modified to run on Xen.

I found that I accidentally booted on metal first time, and when I then booted under Xen the services weren't enabled. I had to manually enable them. (If you boot straight in to Dom 0 they start.

dominion# svcs -a | grep xctl
online         10:51:04 svc:/system/xctl/store:default
online         10:51:11 svc:/system/xctl/xend:default
online         10:51:11 svc:/system/xctl/console:default
online         10:51:16 svc:/system/xctl/domains:default

If it says anything other than online, enable them with

# svcadm enable "service name"

I use a zpool to create my disk devices for my domains. This has huge advantages, such as the ability to quickly snapshot a domain (say after install) so you can always return to that state. Also you can clone a snapshot so if you want to have many similar domains (say multiple solaris development environments) you can clone an install and then only the changes between the domains are stored (zfs being copy on write).

To set this up you need to create a zvol on your zpool

# zfs create -V 10G guests/solaris-pv

This creates a zvol of up to 10G in size. Unused space is still free for other users of the pool to allocate.

You can access the device for this zvol using

/dev/zvol/dsk/guests/solaris-pv

So that's simple - how do we install a Solaris domain? First off I create an install python config file. (Soon there will be a tool to manage the install for you but that's not really ready yet).

This python file describes some simple things about the domain like where the disk and cdrom is.

dominion# cat /guests/configs/solaris-pv-install.py 
name = "solaris-pv-install"
memory = "1024"
disk = [ 'file:/guests/isos/66-0613-nd.iso,6:cdrom,r', 'phy:/dev/zvol/dsk/guests/solaris-pv,0,w' ]
vif = [ '' ]
on_shutdown = 'destroy'
on_reboot = 'destroy'
on_crash = 'destroy'

Name is obvious, and I've copied the iso image to be a file to speed up install.

You can kick off the install just by starting the domain

dominion#  xm create -c /guests/configs/solaris-pv-install.py

This says start the domain and give me a serial console access to it. You then do a normal Solaris install. Once complete you should create a second python file to boot off the zvol. but first I'm going to snapshot it so I can quickly duplicate it (though I really should sys-unconfig it first to make me input the hostname and ip info again.)

dominion# zfs snapshot guests/solaris-pv@install
dominion# cat solaris-pv.py 
name = "solaris-pv"
memory = "1024"
root = "/dev/dsk/c0d0s0"
disk = [ 'phy:/dev/zvol/dsk/guests/solaris-pv,0,w' ]
vif = [ '' ]
on_shutdown = 'destroy'
on_reboot = 'destroy'
on_crash = 'destroy'

and create it with

# xm create -c solaris-pv.py

This then comes up as per a normal solaris boot, if you've given it an ip address during the install or set it to use dhcp you should be able to log in to it using ssh. The networking is effectively bridged, that is to say, you need a real IP address for each domain on the same network as the Dom0.

So the next question I always get is "Can I run windows as a domU". And the answer is "maybe". What we have done up till now is use a paravirualised domU. That is one that has been modified to run on Xen. Anything that would trigger a privileged operation (interrupt, privileged instruction etc) is modified to be a call to the hypervisor. This is nice and fast, but some operating systems haven't had this treatment.

However with the advent of the intel core2duo and Rev F Opteron/Athlon64 (socket AM2) processors, some hardware support for virtualisation has been built in to the chip. This detects these privileged operations and redirects control back to the hypervisor to do "the right thing"

With Xen these are referred to as HVM domains.

Russ is going to be blogging more about these so I won't go in to too much detail, but if you want to know if your system is HVM capable, I wrote this simple program to tell you

dominion# cat hvm-capable.c 
#include < sys/types.h>
#include < sys/stat.h>
#include < fcntl.h>
#include < unistd.h>
#include < string.h>
#include < errno.h>
#include < stdio.h>

static const char devname[] = "/dev/cpu/self/cpuid";

/\*ARGSUSED\*/
int
main(int argc, char \*argv[])
{
        struct {
                uint32_t r_eax, r_ebx, r_ecx, r_edx;
        } _r, \*rp = &_r;
        int d;
        char \*s;
        int isamd = 0;
        int isintel = 0;

        if ((d = open(devname, O_RDONLY)) == -1) {
                perror(devname);
                return (1);
        }

        if (pread(d, rp, sizeof (\*rp), 0) != sizeof (\*rp)) {
                perror(devname);
                goto fail;
        }

        s = (char \*)&rp->r_ebx;
        if (strncmp(s, "Auth" "cAMD" "enti", 12) == 0) {
                if (pread(d, rp, sizeof (\*rp), 0x80000001) == sizeof (\*rp)) {
                        (void) printf ("processor is AMD ");
                        /\*
                         \* Read secure virtual machine bit 
                         \* (bit 2 of ECX feature ID)
                         \*/
                        (void) close(d);
                        if ((rp->r_ecx >> 2) & 1) {
                                (void) printf("and processor supports SVM\\n");
                                return (0);
                        }
                        (void) printf("and does not support SVM\\n");
                } else {
                        (void) printf ("error reading features register");
                        (void) close(d);
                        return (1);
                }
        } else if (strncmp(s, "Genu" "ntel" "ineI", 12) == 0) {
                if (pread(d, rp, sizeof (\*rp), 0x00000001) == sizeof (\*rp)) {
                        (void) printf ("processor is Intel ");
                        /\*
                         \* Read VMXE feature bit
                         \* (bit 5 of ECX feature ID)
                         \*/
                        (void) close(d);
                        if ((rp->r_ecx >> 5) & 1) {
                                (void) printf("and processor supports VMX\\n");
                                return (0);
                        }
                        (void) printf("and does not support VMX\\n");
                } else {
                        (void) printf ("error reading features register");
                        (void) close(d);
                        return (1);
                }
        }
fail:
        (void) close(d);
        return (1);
}

SVM is AMD's implementation of HVM while VMX is Intel's.

And just a teaser of what you can expect. (right click - view image to see it full size)

Here you see a solaris paravirtualized vm being installed, a windows vista hvm domain. In the top left corner you can see the virtual machine manager. A new management gui that will help manage domains.

Sorry this is going to be pretty hard to see unless you view the image in it's original size (1600x1200, yes virtualisation helps you use up those wasted resources including screen real estate)

Monday Apr 30, 2007

Surfing kernow

I had a fantastic weekend down in Cornwall. Saturday's highlight was the boys up to their middles in the sa before they realised it was cold. Followed by my own vague attempts at surfing. I decided to take my camera to the beach to see if I could become a famous surf scene photographer. I don't think that'll happen but I'm pretty pleased with these.

What is he doing ?


It's a paddle !



I hope I look like this


but I know I don't

I even wish I managed a wipeout as good as this (and yes I did take hundreds of phtos for the exact wipeout)



Incidentaly these are just random snaps at the beach. If you were at watergate bay over last weekend (especially if you're in the piccies) please leave a comment.

Chris

Wednesday Apr 11, 2007

Two years of prevarication and 5 days hard work and finally ....

The wall is built.

A couple of years a go I leveled the area outside our back door and laid a patio. This meant I had to dig in to the slope that raises up away from our house, and that meant I was going to have to build a wall and some steps up to the lawn. For some reason I've found excuses not to do it ever since, until two weeks a go I was going in to Travis Perkins looking for fence panels (there is a fencing shortage in the UK at the moment) and came out with 400 bricks (well not literally, they did deliver them.

So I've spent most of Easter and a long weekend building it, but it is finally done. I think it looks pretty good. What do you think?

Friday Jan 26, 2007

Snow

A little late I know but we had some snow in the South of England this week. The usual 1 or 2 inches causing the rail and road infrastructure to break down, schools to be closed and food rationing to be enforced.

Fortunately our boys school wasn't closed but we did have a great snowball fight before and on the way to school.


Plus this was the first experience for our spaniel in the snow


I might post some more as I go through them.

Monday Jan 22, 2007

Using the OpenSolaris Mercurial repository


I finally decided to have a look at how you get OpenSolaris from the Mercurial repository I've been put off by the fact at the office we're behind a fire wall, so you have to pull it through a Socks proxy.

I turns out it's absolutely trivial to do (the instructions are all on the OpenSolaris WebSite.

If you're behind a firewall that requires you to have a proxy for ssh th first thing you need to do is set up ssh to use a proxy. I can't help you with that other than to say add the following line to your ssh config for opensolaris.org

$ cat ~/.ssh/config
Host \*.opensolaris.org
ProxyCommand /usr/lib/ssh/ssh-socks5-proxy-connect -h [Proxy IP address] %h %p
Compression yes

(thanks to Erik and Stephen for pointing out the compression option)
Putting in the IP address of your socks proxy.

Now you can clone the repository

$ hg clone ssh://anon-AT-hg.opensolaris-DOT-org/hg/onnv/onnv-gate
adding changesets
adding manifests
adding file changes
added 3487 changesets with 67524 changes to 43099 files
39742 files updated, 0 files merged, 0 files removed, 0 files unresolved


It took only 37 minutes to my home machine which is much faster than a full bringover in teamware

Now to start playing with some of the build tools

Monday Jan 08, 2007

Is it 2007 already?

Thanks to Patrick for tagging me and making me realise I hadn't touched my blog in ages.

I've been so busy, travelling to present at the Tech day in Prague, meeting our colleagues over in Grenoble. Working on FMA, and now finally being pulled in to porting Solaris on Xen to Solaris 10.

Busy times along with getting on with having a dog, buying a new camera. Christmas and all....

Any way Patrick tagged me to come up with 5 things you didn't (or didn't need to) know about me.

1. Despite my accent, I was born and grew up in the North East of England. In a town called Darlington

I sometimes have a hankering to move back.

2. I went to school with England Rugby player Tony Underwood (who has recentley also appeared on programme about Easyjet)

3. I went to UCL and studied Applied Physics at which the only course I really enjoyed were Planetary Geology (Colouring in photos of Mars) and Operating System Design (hence what I do now)

4. I'm a firm believer in rational process as defined by Kepner Tragoe. I find it helpful in defining a problem or situation I find myself in and giving me a structure to progressing it. I enjoy facilitating people to use this and used it recentley to help establish priorities for the future of FMA.

5. Apparentley if I was a Super Hero I'd be Spiderman (like many people around here...)
You are Spider-Man
Spider-Man
70%
Superman
65%
Robin
60%
Green Lantern
60%
Iron Man
55%
Supergirl
53%
Wonder Woman
43%
The Flash
40%
Catwoman
35%
Hulk
35%
Batman
35%
You are intelligent, witty,
a bit geeky and have great
power and responsibility.
Click here to take the Superhero Personality Test

So you may have know some of those, but I hope it was interesting. I here by tag Gavin, Peter, and Jon

Sunday Sep 24, 2006

EuroOSCON 2006

EuroOSCON 06

This week (18th - 21st September) I've had the opportunity to attend and be involved in EuroOSCON06. This was primarily to increase my understanding of opensource, but also to promote Open Solaris. This was my first OpenSource conference so first I'll make a few general observations before moving on to details about the sessions.

EuroOSCON06 was this year in Brussels, a city I'd not visited before. It's a surprisingly small city for the self proclaimed capital of europe. There are some very beautiful parts as well as some rather seedy parts, and being small the seedy and wonderful nestle uncomfortably together.

There were some other Sun employees (Martin Man Peter Dennis, Patrick Finch, Darren Kenney and Gary Pennington). I'd met a few of them before but we're from very different backgrounds so we had different reasons for wanting to attend EuroOSCON and promote OpenSolaris.

So I took the EuroStar from London (I was booking the trip just as the security scare happened last month so thought this would be easiest). Met with Peter Dennis on the train and worked through some demos we could show to people. We had a BOF and a Booth on Wednesday so thought we'd try and show some cool stuff.

The Demos

We decided we'd show how easy it was to set up and build OpenSolaris. I had a media kit with me on the train and by the time I was in Brussels had installed a build machine environment on my laptop and was happily building code, cool!. We also wanted to show some zfs features, and some zones features. There is a new facility in OpenSolaris to allow you to create a Zone on one system (preferably on a zfs file system, and then take a copy of it to create a new zone. If your using zfs it will sanpshot the filesystems rather than copying data meaning you get use the zfs snapshot facility meaning it is rather quick. This is done with zonecfg clone -s

You can then dettach that zone from your current system (using zoneadm dettach),and as the zpool was on an external disk we moved the USB disk to another laptop and imported the pool (zpool import ) and attach the zone to the new laptop (zoneadm -z attach -F -n ). It then just works as it did on the old system. I was amazed and can see how useful this is going to be.

Observations

Some things struck me. First - everyone uses Mac their. This surprised me as although it is based on an opensource OS, it is far from and OpenSource product. Second - There are a lot of people on the conference gravy train, they obviously go to a lot of these. That's fine, but it did distort the audience a bit. Finally Licensing both of Projects and data is still a big worry for this community.

It was well worth going this year to promote OpenSolaris. People were falling queing up at the booth to talk to us so on to some more detail

The Booth and the BOF

Wednesday had us setting up the booth for OpenSolaris and manning it. We had intended to take turns, but we were so busy for the whole of the morning that we all were talking pretty much all morning to various interested people. We had some "OpenSolaris Starter Kits" Containing the install media, the source and the compilers, along with a coupl of livedvd images. Overall they went like hot cakes everyone seemed excited by OpenSolaris. I even persuaded a guy from Google to take a look. The afternoon was a little quieter so we could go off to a few sessions, but Still a steady stream of interested people.

I gave more demos than I can remember - people were excited about how easy it was to use ZFS and how simple building opensolaris was. Dtrace is still a big thing and now it's in macOS the community can see am example of why they should be involved with OpenSolaris

The BOF was at the same time as one from Ubuntu's Mark Shuttleworth, so we weren't expecting many and we had a few, though not as many as I would have hoped. We talked through SMF, Zones, dtrace, ZFS among other things and the people seemed interested. The funny thing was that Mark Shuttleworth failed to turn up and the Ubuntu BOF got rather heated.

The Sessions

KeyNote Speakers: Tim Orielly, Tor Norretranders

Tim Orielly spoke about data, and licensing (as you'll see licensing seems to be a hotly discussed topic at the conference. It's not something that has ever interested me but it is clearly important. Tim was commenting that if you put it on Myspace but can't take it anywhere else is it really yours? Same goes for applications that run on the web server rather than in your own system, what is to stop the owner of the webserver doing what ever they want. I feeling he was blowing his own trumpet saying he'd pointed this out in 1998 or something but only now people are taking him seriously. He was trying to say (I think) that in the Web2.0 world where everyone contributes on line the things you put in your blog or on some one else collab site you may not own the rights to.

Tor gave a very funny speach about the motivations for participating in opensource. He stated it was about

Glow-> The nice feeling we get when interacting with someone else
Show -> Showing off and helping others gets you noticed (or laid as he put it)
Flow -> You are constantly changing.
2.0 -> We're getting back to a bartering type economy.

Certainly I enjoyed it and the first two points are clearly right, the rest felt a little forced, but then it was well made.

Industrial strength Email and Calendar: Flaorian von Kurnatowski

Without realising it I'd wandered in to the Products and Services track. Basically Opensource friendly companies promoting their products. That said he didn't push his company Scalix too much. What he observered was that you needed to have a true replacement for Outlook before people would be able to move away from Microsoft. It seems Outlook is very closely tied to all other Microsoft apps and if you remove them you loose a lot of functionality (and he said Outlook is 50% of the license fee too).

Until OpenOffice can provide that or has an equivalent it will not considered by many people. Also 90% of Admins have never done a migration of mail systems so they're scared of it, there need to be good migration tools.

Final point was that Calendar services do not have any standards which is why Calendar infrastructure is even harder to do than email.

Channeling OpenSourced in Europe: Ranga Tangachari

Back in the OpenSource world I was interested by this session. I'd assumed that this would be about getting the most out of OpenSource in Europe but instead it was a talk about how his company made money in Opensource by encouraging the Channel (resellers)

His assertion was that Communities provide innovations and companies provide Products (more the just projects, fully tests and supported things). In the middle are the Channel which adds value by things like locaization and training. Being a pool of deployment experts.

You need to encourage the Channel by giving them what they care about which is

1) Margin
2) Professional Services Opportunities
3) Maintenance (recurring revenue)

Think beyound Downloads they only mean one click, find examples of happy customers.

Big Data and the Open Warehouse: Roger Magoulas

This was a dissapointing presentation about what Orielly do about data storage and data mining. It was unfortunately simply a run down of tricks tips, products and techinques used by the Orielly guy in their data centre. There were some interesting things mentioned though which I will go and look at.

SecondLife and Opensource: Jim Purbrick

I'm intrigued by SecondLife, it's a game where the whole purpose is to make "Stuff" and "Hang Out" and generally share or sell what you do. I have looked at it and it is cool, but I haven't got my head round Why? yet.

SecondLife is not (yet) opensource but the Guy from LindenLab was explaining that the big difference with second life to other MMORPG games is that the players create the world. LindeLabs couldn't have provided enough content to keep people interested, but because it is created by the game community they reacon they get ~6500 man years of content development per year! (not even EA could manage that for one game I think) All of this is a course up for sale or copy depending on the desires of the community member

There are interesting aspects to the way LindenLab have architected their set up, like each now plot of land requires a new server so they're adding new servers at a huge rate.

Over all the crowd were excited by SecondLife. I didn't see them as engaged in any other session

Afternoon Keynotes: Steve Coast, Adrian Holovaty

Steve Coast: Open Data: An interesting view of "Good enough" data. He uses a gps held by volunteers as they go about there daily business to look for a plan of cities, making it available via OpenStreetMap. Most map data in the world is either govenement owned (like the Ordanance Survey) or not of great quality. Creating good enough Open Data which can be shared will be enough for some and will consequently bring down the price of Closed Data).

Adrian Holovaty: Journalism through Programming:

Facinating and slightly scarey view on providing access to the raw data used by journalists via web applications. He works for the Washington Post and is involved in a few projects one of which is Faces of the Fallen. Another one he quoted was to look at your MPs voting reacord TheyWorkForYou. Now I can see the point of this, and will certainly be checking on what my MP is doing, but I am slightly worried that this requires balance. We're asking the public to draw conclusions from only a small amount of the data (as the records published on line are incomplete), where as journalism is all about weighing up all the information and providing a balanced summary for consumption.

Open Useability: Jan Muehlig

A talk to encourage open source projects to include Useability eningeering in the project to make the user experience of Open Source Products as good or better than closed source, something we have often complained about. His greoup OpenUseability is working to promote this and it seems to boil down to publishing best practices, which noone does yet.

Making It Work: Louis Suarez-Potts

A talk about how to build succesful OpenSource projects. He comes from OpenOffice Which is both a succesful project and a really useful product so he should know what he's doing.

He talked about the two different approaches. The organic where a few friends start up with a common goal, and the Sponsored, where a corperate entity is driving towards a set of goals. Eitehr way you need to do the following

Pick the right license
Have a neutral environment (ie safe to contribute)
Have transparent governance and processes
Make decisions in public
Have clear decision paths
Use good communications tools (everyone liks IM these days
Have immediate gratification (easy and fast contribution)
Market your project
Have the right Product

I found this quite encouraging as I felt OpenSolaris has it about right

OpenSource and Freedom: Why Open Standards are crucial to protecting your linux investment: Jim Zemlin

This talk was aimed at promoting the LSB(Linux Standards Base To make sure applications will run on the largest number of Distros. LSB dictates the minimum number of components available within the Distro so your application can rely on them. This is to encourage growth over Microsoft. He quoted what happend in the Unix world when the standards fragmented and he is absolutely right

KeyNote: Florian Muller Roml Lefkowitz

Florian Muller Spoke about lobying in the European Parliament to limit the changes to Patent law which some companies are trying to tighten up to protect their IP, while OpenSource are trying to go the other way. I was left slightly disconcerted that someone with such a one sided view was having an effect on our laws.

Roml Lefkopwitz Spoke about the need to internationalize and localize the source code and languages used in opensource projects. Nice pie in the sky thinking, but misses the point that the source should not be the documentation, we need documentation before we can worry about such things.

Xgl and Compiz - New X11 features and the OpenGL Accelerated Desktop Matthias Hopf

Facinating talk about the future of desktop from Suze At last a talk with lots of technical details and a neat demo at the end demonstrating the desktop mapped on to a 3d cube running two movies and Quake 3 at the same time on different faces of the cube. All of this should soon also be possible in Solaris and I think it's vital we do it.

The End

If you got this far then well done :) It was a lot to read.

Technorati Tags: Solaris OpenSolaris EuroOSCON EuroOSCON06

Tuesday Sep 12, 2006

Building Opensolaris

This weekend I did something I've been meaning to do for a while. I've been putting it off due to lack of time to think about how to approach it without breaking anything.

Any way I finally tried downloading and installing Solaris Express the community edition (build 46) and downloaded all the build tools. I was amazed how easy it was. Within an hour offinishing the downloads, it was building opensolaris.

The main shock for me was the lack of an SCM (Source Code Management) system. Being fully entrenched in the world of ON (Os and Networking) for the last 10 years Teamware (the SCM we use) is just \*what we do\*. So I had to rethink how I'll manage the build. But then using opensolaris.sh from usr/src/tools/env did a good job.

So I know have my own build built from opensolaris running on my system, without anything from within Sun. Cool. Give it a go, it was easy and gives you the chance to play with how things work.

Next step for me - build the Xen bits from outside Sun :)


==== Nightly distributed build started: Mon Sep 11 11:27:18 BST 2006 ====
==== Nightly distributed build completed: Tue Sep 12 03:22:02 BST 2006 ====

==== Total build time ====

real 15:54:44

==== Nightly argument issues ====

Warning: the N option (do not run protocmp) is set; it probably shouldn't be

==== Build environment ====

/usr/bin/uname
SunOS osol-bld 5.11 snv_46 i86pc i386 i86pc

/opt/onbld/bin/nightly myopensolaris.sh
nightly.sh version 1.104 2006/08/29

/opt/SUNWspro/bin/dmake
dmake: Sun Distributed Make 7.7 2005/10/13
number of concurrent jobs = 4

32-bit compiler
/opt/onbld/bin/i386/cw -_cc
cw version 1.20
primary: /opt/SUNWspro/bin/cc
cc: Sun C 5.8 Patch 121016-02 2006/03/31
shadow: /usr/sfw/bin/gcc
gcc (GCC) 3.4.3 (csl-sol210-3_4-20050802)

64-bit compiler
/opt/onbld/bin/i386/cw -_cc
cw version 1.20
primary: /opt/SUNWspro/bin/cc
cc: Sun C 5.8 Patch 121016-02 2006/03/31
shadow: /usr/sfw/bin/gcc
gcc (GCC) 3.4.3 (csl-sol210-3_4-20050802)

/usr/java/bin/javac
java full version "1.5.0_08-b03"

/usr/ccs/bin/as
as: Sun Compiler Common 10 snv_46 08/03/2006

/usr/ccs/bin/ld
ld: Software Generation Utilities - Solaris Link Editors: 5.11-1.545

Build project: group.staff
Build taskid: 62

==== Build version ====

ws.opensolaris

==== Make clobber ERRORS ====

==== Make tools clobber ERRORS ====

==== Tools build errors ====

==== SCCS Noise (DEBUG) ====

==== Build errors (DEBUG) ====

==== Build warnings (DEBUG) ====

==== Elapsed build time (DEBUG) ====

real 11:42:57.3
user 4:41:00.3
sys 2:40:32.5




Technorati Tags:
Solaris OpenSolaris Xen

About

Chris W Beal

Search

Archives
« April 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today