Wednesday Feb 28, 2007

in.telnetd exploit doing the rounds

I am hearing talk about an exploit of the in.telnetd issue doing the rounds. This affects Solaris 10 and Solaris Express.

References at sans.org and asert.arbornetworks.com

Now would be a particularly good to disable your telnet daemon:

# svcadm disable telnet

If you must run telnetd, then you need to get the patches referred to in Sun Alert 102802. The patches are freely available on sunsolve.

120068-03 SunOS 5.10_sparc: in.telnetd Patch
120069-03 SunOS 5.10_x86: in.telnetd Patch

In spite of what the README says, these patches do not require a reboot.

Some Details

While things are still sketchy, it looks like it propogates by connecting as both "adm" and "lp" and copies sparc and x86 binaries into /var/adm/sa/.adm, along with crontab entries for both of these users.

A quick check to see if you have been infected is to check the mode of /var/adm/wtmpx. If it is 0646 and you have the aforementoined directory, it is likely yoyu are infected.

First off, disable the telnet daemon as described above. Clear out the cron entries that were added for adm and lp. There will also be a program running listening on port 32982. It will likely be called the same name as the only non-dot-file in /var/adm/sa/.adm. Make sure you kill the right one, as they choose from a number of solaris daemons for the naming. Run pfiles on it to look for port 32982.

Update

For more information see the security blog.

Technorati Tags: , , ,

Thursday Feb 15, 2007

more on the in.telnetd patches

As many folks have stated. Sun Alert 102802 and the patches are available on Sunsolve.

120068-02 SunOS 5.10_sparc: in.telnetd Patch
120069-02 SunOS 5.10_x86: in.telnetd Patch

I've had it pointed out to me that the patches are marked "Reboot after installation required". This is actually not the case and a bug has been logged to get the tags removed from the patch.

For what it is worth, I tested the fix by applying the patch while the systems were multiuser and the fix was immediate. I did not even have to restart the services. in.telnetd is fork/exec'd by ineted. It's generally short lived. Adding the patch replaces the binary that is exec'd. You do not need to perform a reboot to get this patch installed.

Technorati Tags: , , ,

Tuesday Feb 13, 2007

The in.telnetd vulnerability/exploit (3rd update)

Before I get into the meat of this posting, let me acknowledge that, yes, this was an almighty cock up and should not have happened. It did happen. Let's move on.

Also, while I might not agree with the publication of zero day exploits. Again, It happened. There's really not much I can do about that. There's really no point in being upset about it.

The upside to the posted exploit was the fact that because the code was available, the poster included an analysis of what was going wrong, pointing at the code that was broken. This almost certainly saved us some time in troubleshooting the issue. For this part of the post, you have my thanks.

I would certainly be interested if the person who posted the exploit could tell us how he found the problem; for no other reason, than I'm simply interested.

Anyway, this blog is supposed to be about getting it fixed.

All the times below are Australia/NSW.

One of our National SSEs (Rodney) was on-site with a large customer yesterday. This customer had asked him about a telnet exploit and described the problem to him. Rodney gave me a call and asked me about it at about 1pm. I hadn't and on hearing the description (initially only described to him as a root exploit) Ttwo of us (thanks for your help Chris) dove into the code to start looking at how telnet -l-froot could behave as it did. At this point I did not know about the zero day exploit posting. Once we worked out what was going on, I called Rodney back and explained the full implications of the bug to him so he could explain it to the customer.

We told them that they could block the root vulnerability by uncommenting the CONSOLE= line in /etc/default/login. Note that this has been the default since Solaris 10 update 2 almost forever. However, I still see lots of customer configurations where it is commented out. The only other way to protect against the other implications (login as any user without a password) would be to disable the telnet service until we could fix the issue. eg

# svcadm disable svc:/network/telnet:default

We then started looking through the code to determine the best way to fix this. I logged an internal escalation and was in the process of logging a high priority bug when I saw the following in the SCCS history of usr/src/cmd/cmd-inet/usr.sbin/in.telnetd.c

D 1.67  07/02/11 19:46:41 danmcd        90 89   00009/00010/04896
6523815 LARGE vulnerability in telnetd

I immediately had a look at the bug and banged of an email to Dan stating that I could probably get IDR patches built for on10 pretty quickly. After a brief discussion of the bug and the fix, he pointed me at the manager of the group responsible for the backport and I started the backport (actually a very simple fix).

I got the IDRs built and basic testing done by about 5pm and started writing the Sun Alert.

The documentation for how to write a Sun Alert and specifically the actions that need to be taken to get interim fixes available were spot on and I sent off the initial draft at about 6:45 along with sending a request for getting the IDR patches turned into ISR patches (Interim Security Relief) and getting them published.

Just before 9pm, I started getting into discussions with UK based folk in Derrick Scholl's group about getting the Sun Alert out and what needed to happen for me to get a gate open to get the fix back into the patch gate.

Thanks to Angela, Paul, Brent and Bill for working hard to get to the point that I could log the RTI at 10:10, and start doing the minor nit type stuff that needed to be done before Bill could pass the RTI onto the on10-patch gatekeepers and I could go home (at about 11:15).

As an aside, I missed my train connection by four minutes due to a late running North Shore train and spent an hour sat on a blacked out Hornsby station, getting home at about 2:30am)

While I was traveling (and sleeping) the heads up went out to the gatekeepers and all folk who needed to know about this so that when I came online at 8 this morning, I had very little to do before doing the actual putback into the patch gate (which happened at about 8:30).

The gatekeepers immediately closed the gate and started work on a patch.

The reason that I've detailed what I've been through with this is to point out one thing.

The speed at which I was able to do this and get to the point that an ISR patch will be shortly available publicly, is nothing short of phenomenal. For Sun to respond to and address a vulnerability like this in around 24 hours would have been completely unheard of even two to three years ago.

But it's not just the processes here. What really made for speed here was an incredibly focussed and helpful who had an interested in rapidly getting this addressed. Without the help of folks like Dan, Rodney, Chris, Angela, Paul, Brent, Bill and Seth, and not forgetting the gatekeeping team for pulling out the stops to start building a formal patch, none of this would be possible. If I've missed anyone, please forgive me, I didn't get a lot of sleep last night :)

I love working for a company that has people like this.


update 1

The ISR patches are available for free download from http://sunsolve.sun.com/tpatches. The details of the patches are:

   IDR125457-01 SunOS i386_x86: in.telnetd can call login with an
                option given as a username

   IDR125456-01 in.telnetd can call login with an option given as a
                username

update 2

Sun Alert 102802 is publicly available talking about this issue. Section 4 should shortly be modified to add the following paragraph:

Interim Security Relief (ISRs) are available from http://sunsolve.sun.com/tpatches for the following releases:

SPARC Platform

Solaris 10 IDR125456-01

x86 Platform

Solaris 10 IDR125457-01

Note: This document refers to one or more Interim Security Relief (ISRs) which are designed to address the concerns identified herein. Sun has limited experience with these (ISRs) due to their interim nature. As such, you should only install the (ISRs) on systems meeting the configurations described above. Sun may release full patches at a later date, however, Sun is under no obligation whatsoever to create, release, or distribute any such patch.

Update 3

I've just been informed that the formal patches are release ready and should be released to sunsolve in the next few hours. Keep an eye out for:

120068-02 SunOS 5.10_sparc: in.telnetd Patch
120069-02 SunOS 5.10_x86: in.telnetd Patch

As these are security patches, they will be publicly available.

Technorati Tags: , , ,

Monday Feb 12, 2007

b57 non-debug ON encumbered binaries

Sorry it's taken so long for these. On top of having one of the most hectic two weeks I think I've ever had in this job (with customer escalations), I tripped over a few things with having to upgrade the SPARC build machine and then having to rebuild my SUNWonbld stuff for both architectures as some things needed updating in it and then discovering a couple of little things I had updated on the build machines and not in my source tree (sigh). Anyway, they are up now.

They can be downloaded from The Download Centre. I've also updated the MD5 Checksums for the extra files.

Note that as of build 54, you no longer need the fiddle to usr/src/Makefile that was previously required to do non-debug builds, so have a look at the new instructions.

Dennis Clarke has also written a great step by step guide.

Technorati Tags: ,

Wednesday Jan 24, 2007

b56 non-debug ON encumbered binaries

They can be downloaded from The Download Centre. I've also updated the MD5 Checksums for the extra files.

Note that as of build 54, you no longer need the fiddle to usr/src/Makefile that was previously required to do non-debug builds, so have a look at the new instructions.

Dennis Clarke has also written a great step by step guide.

Technorati Tags: ,

Sunday Jan 07, 2007

b55 non-debug ON encumbered binaries

They can be downloaded from The Download Centre. I've also updated the MD5 Checksums for the extra files.

Note that as of build 54, you no longer need the fiddle to usr/src/Makefile that was previously required to do non-debug builds, so have a look at the new instructions.

Dennis Clarke has also written a great step by step guide.

Technorati Tags: ,

Tuesday Dec 12, 2006

Tamarack, Gnome 2.17 and external devices

This is a combination that I've been hanging on. vold never quite handled my cds, dvds and usb media quite right.

I'm happy to say that for the most part it all just works like it should.

Well, I have a multi-partitioned usb device which exhibits an interesting problem.

First, let's have a look at how the device (it's a 60gb usb 2 disk) is partitioned.

             Total disk size is 57231 cylinders
             Cylinder size is 2048 (512 byte) blocks

                                               Cylinders
      Partition   Status    Type          Start   End   Length    %
      =========   ======    ============  =====   ===   ======   ===
          1                 Ext Win95         1  28615    28615     50
          2                 Solaris2      28616  57230    28615     50 

Partition 1 has a fat-32 filesystem.
Partition 2 has an SMI label and slice 0 is a zfs pool with the imaginitve name 'usb'.

# fstyp /dev/dsk/c4t0d0p1
pcfs
# fstyp /dev/dsk/c4t0d0s0
zfs

If I insert this disk into my notebook running b53, I get the notice about not being able to mount the zfs pool (which I expect), but it does not notice the fat-32 partition, let alone mount it:

# rmmount -l
/dev/dsk/c4t0d0s0    rmdisk,rmdisk0,usb

I had a chat with one one of the developers about this. He found that if we place the zfs pool directly into the second partition, then everything works as you would expect. The different controller numbers and zfs pool name are purely due to using another disk and machine in anotehr continent.

# fstyp /dev/dsk/c2t0d0p1
pcfs
# fstyp /dev/dsk/c2t0d0p2
zfs
# rmmount -l
/dev/dsk/c2t0d0p0:1  rmdisk,rmdisk0,NONAME,/media/NONAME
/dev/dsk/c2t0d0s2    rmdisk,rmdisk0,lacie_p2

It turns out that if hal discovers an SMI label on any partition, it does not probe any of the other logical disks, leading to the logging of

6502219 If SMI label exists, hal does not probe logical disks

In the meantime as I really don't want to muck around with the existing zpool, I'll continue to mount my pcfs manually, but it will be nice when this one is fixed.

It's probably also worth mentioning that "zpool import usb" just works.

Technorati Tags: ,

Monday Dec 11, 2006

b54 non-debug ON encumbered binaries available

They can be downloaded from The Download Centre. I've also updated the MD5 Checksums for the extra files.

Note that this is the first of the named builds to incorporate the change to usr/src/Makefile to remove the fiddle previously required to do non-debug builds, so have a look at the new instructions.

Dennis Clarke has also written a great step by step guide.

Technorati Tags: ,

Thursday Nov 30, 2006

20061127 non-debug ON encumbered binaries available

As previously stated, the usr/src/Makefile fix went into ON just before build 54 closed. Coincidentally, this is the date of the source drop for this week.

As such, you no longer have to either rename the closed-bins directory or patch usr/src/Makefile.

Before I go any further, here's a link to the new closed binaries for this build.

Again, it's time to redo the instructions for building non-debug ON.

  1. Extract the encumbered binaries that you require.

  2. Modify the environment file. In my case this is generally called opensolaris.sh and I keep my copy in the directory that contains usr and proto, often referred to as ${CODEMGR_WS}.

    In order to only do a non-debug build go to the line that defines NIGHTLY_FLAGS and remove the D and F options. If you want nightly to build both, only remove the D option.

    You also want to ensure that in the environment file that ON_CLOSED_BINS points to the directory that contains the root_\* files.

Everything else should be as per how you would normally do a debug build and as long as you take these things into account Dennis Clarke's step by step guide is a good outline of how to build a non-debug ON consolidation.

Technorati Tags: ,

Saturday Nov 25, 2006

b53 non-debug encumbered binaries (and new usr/src/Makefile diff) available

The encumbered binaries for Build 53 of OpenSolaris are now available at The Download Centre along with the updated MD5 Checksums.

In order to put the usr/src/Makefile changes back into ON I've made a couple of minor changes to the diffs that I previously posted. The Makefile is now a little more clear about what it's doing with it's logging and error messages for doing the open build.

OK, the reason for this is that I managed to get caught out "testing" the Makefile change and forgot to copy the new Makefile into place. The logging that was in there didn't allow me to see that I was doing a non-debug build with the debug encumbered binaries (because it didn't have the new code in there). It now is clear about where it is copying the encumbered binaries from and about what it expected to find if it can't find them.

As such, it's probably time to re-do the instructions for how to build a non-debug ON tree, not forgetting to mention Dennis Clarke's great step by step guide.

First off, as the current usr/src/Makefile does not know about the directory name that the non-debug binaries extract in to, we have two options.

  1. Extract the encumbered binaries and then rename it such that the '-nd' suffix is now longer in the directory name.

    e.g.

    • closed/root_sparc (instead of closed/root_sparc-nd)
    • closed/root_i386 (instead of closed/root_i386-nd)
  2. Modify usr/src/Makefile as per the following diff output. This tells make to grab the non-debug encumbered binaries from the directory that the tarball extracts into. The upside of this is that it then anables you to build bfu archives for both debug and non-debug from nightly.

    ------- usr/src/Makefile -------
    
    Index: usr/src/Makefile
    --- /ws/onnv-gate/usr/src/Makefile      Mon Oct 16 15:10:55 2006
    +++ /export5/ah89892/onnv-6495870/usr/src/Makefile      Tue Nov 21 14:44:20 2006@@ -22,7 +22,7 @@
     # Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
     # Use is subject to license terms.
     #
    -# ident        "@(#)Makefile   1.224   06/10/16 SMI"
    +# ident        "@(#)Makefile   1.225   06/11/21 SMI"
     #
     # Makefile for system source
     #
    @@ -117,14 +117,17 @@
            @cd lib; pwd; $(MAKE) install
    
     closedbins: FRC $(ROOTDIRS)
    -       @if [ "$$CLOSED_IS_PRESENT" = no ]; then \\
    -               if [ ! -d "$$ON_CLOSED_BINS/root_$(MACH)" ]; then \\
    +       @CLOSED_ROOT="$$ON_CLOSED_BINS/root_$(MACH)$${RELEASE_BUILD+-nd}"; \\
    +       if [ "$$CLOSED_IS_PRESENT" = no ]; then \\
    +               if [ ! -d "$$CLOSED_ROOT" ]; then \\
                            $(ECHO) "Error: if closed sources are not present," \\
                                "ON_CLOSED_BINS must point to closed binaries."; \\
    +                       $(ECHO) "root_$(MACH)$${RELEASE_BUILD+-nd} is not" \\
    +                           "present in $$ON_CLOSED_BINS."; \\
                            exit 1; \\
                    fi; \\
    -               $(ECHO) "Copying closed binaries from $$ON_CLOSED_BINS"; \\
    -               (cd $$ON_CLOSED_BINS/root_$(MACH); tar cf - .) | \\
    +               $(ECHO) "Copying closed binaries from $$CLOSED_ROOT"; \\
    +               (cd $$CLOSED_ROOT; tar cf - .) | \\
                        (cd $(ROOT); tar xBpf -); \\
            fi
        

The next part is to modify the environment file. In my case this is generally called opensolaris.sh and I keep my copy in the directory that contains usr and proto, often referred to as $CODEMGR_WS.

In order to only do a non-debug build go to the line that defines NIGHTLY_FLAGS and remove the D and F options. If you want nightly to build both, only remove the D option.

Make sure that you have all of the encumbered-binaries files extracted into the right place and kick off the nightly as you normally would.

Technorati Tags: ,

Wednesday Nov 22, 2006

Some DTrace scripts I found useful last week

Last week I spent some time looking at applications that a customer was using to perform a data migration. It occurs to me that folks might be interested in a couple of the "one liner" type scripts that I found useful after turning them into 'stat' type tools. So, here they are.

funcs.d

#!/usr/sbin/dtrace -s

#pragma D option quiet

/\*
 \* Count all user space function calls
 \*
 \* $1 - time to run (eg 10s)
 \* $2 - pid to monitor
 \*/

pid$2:::entry {
	@[probefunc] = count();}
tick-$1 {
	printa(@);
	clear(@);
	printf("--------\\n");}

syscalls.d

#!/usr/sbin/dtrace -s

#pragma D option quiet

/\*
 \* Count the syscalls a process is making as a stat tool
 \*
 \* $1 time to wait (eg 10s)
 \* $2 target pid
 \*/

syscall:::entry /pid == $2/ {
	@[probefunc] = count();}
tick-$1 {
	printa(@);
	printf("--------\\n");
	clear(@);}

systimes.d

#!/usr/sbin/dtrace -s

#pragma D option quiet

/\*
 \* Count the syscalls a process is making as a stat tool
 \*
 \* $1 time to wait (eg 10s)
 \* $2 target pid
 \*/

syscall:::entry /pid == $2/ {
	self->start = vtimestamp;}
syscall:::return /self->start/ {
	@[probefunc] = quantize((vtimestamp - self->start)/1000);}
tick-$1 {
	printa(@);
	printf("--------\\n");
	clear(@);}

ustk.d

#!/usr/sbin/dtrace -s

#pragma D option quiet
/\*
 \* Aggregate user stacks calling a function in user space
 \*
 \* $1 - time to run (eg 10s)
 \* $2 - pid to monitor
 \* $3 - function to look for
 \*/

pid$2::$3:entry {
	@[ustack(30)] = count();}
tick-$1 {
	printa(@);
	clear(@);
	printf("--------\\n");}

Technorati Tags: ,

Sunday Nov 12, 2006

Solaris ufs bug in Month of Kernel Bugs

Just noticed that Solaris has an entry in Month of Kernel bugs.

While I agree that we have an issue that needs looking at, I also believe that the contributor is making much more of it than it really deserves.

First off, to paraphrase the issue:

If I give you a specially massaged filesystem and can convince someone with the appropriate privilege to mount it, it will crash the system.

I'd hardly call this a "denial of service", let alone exploitable.

First off, in order to perform a mount operation of a ufs filesystem, you need sys_mount privilege. In Solaris, we currently are runing under the concept of "least privilege". That is, a process is given the least amount of privilege that it needs to run. So, in order to exploit this you need to convince someone with the appropriate level of privilege to mount your filesystem. This would also invlove a bit of social engineering which went unmentioned.

That being said, they system should not panic off this filesystem and I will log a bug to this effect. It is a shame that the contributor did not make the crashdump files available as it would certainly speed up any analysis.

One other thing that I should add is that anyone who tries to mount an unknown ufs filesystem without at least running "fsck -n" over it probably deserves what they get.

OK, I have copied it to a relatively current nevada system and mounted it as /dev/lofi/1. On running "fsck -n" we see:

\*\* /dev/rlofi/1 (NO WRITE)
BAD SUPERBLOCK AT BLOCK 16: BAD VALUES IN SUPER BLOCK

LOOK FOR ALTERNATE SUPERBLOCKS WITH MKFS?  no


LOOK FOR ALTERNATE SUPERBLOCKS WITH NEWFS?  no

SEARCH FOR ALTERNATE SUPERBLOCKS FAILED.

USE GENERIC SUPERBLOCK FROM MKFS?  no


USE GENERIC SUPERBLOCK FROM NEWFS?  no

SEARCH FOR ALTERNATE SUPERBLOCKS FAILED. YOU MUST USE THE -o b OPTION
TO FSCK TO SPECIFY THE LOCATION OF A VALID ALTERNATE SUPERBLOCK TO
SUPPLY NEEDED INFORMATION; SEE fsck(1M).

In the normal course of events, would you mount this filesystem. I certainly would not. This however is not the normal course of events and I'm playing on a lab system.

v40z-c# uname -a
SunOS v40z-c 5.11 snv_46 i86pc i386 i86pc

Let's try a read only mount first.

v40z-c# mount -r /dev/lofi/1 /mnt
v40z-c# ls /mnt
lost+found
v40z-c# umount /mnt

OK, the read only mount is fine. Now the read/write, ... Bingo

v40z-c# mount /dev/lofi/1 /mnt

panic[cpu3]/thread=ffffffff9a6974e0: BAD TRAP: type=e (#pf Page fault) rp=fffffe8000c7f2c0 addr=fffffe80fe39d6c4

mount: #pf Page fault
Bad kernel fault at addr=0xfffffe80fe39d6c4
pid=2170, pc=0xfffffffffbb70950, sp=0xfffffe8000c7f3b0, eflags=0x10286
cr0: 8005003b cr4: 6f8
cr2: fffffe80fe39d6c4 cr3: 1ff76b000 cr8: c
	...
fffffe8000c7f1b0 unix:die+b1 ()
fffffe8000c7f2b0 unix:trap+1528 ()
fffffe8000c7f2c0 unix:_cmntrap+140 ()
fffffe8000c7f440 ufs:alloccgblk+42f ()
fffffe8000c7f4e0 ufs:alloccg+473 ()
fffffe8000c7f560 ufs:hashalloc+50 ()
fffffe8000c7f600 ufs:alloc+14f ()
fffffe8000c7f6c0 ufs:lufs_alloc+f3 ()
fffffe8000c7f770 ufs:lufs_enable+261 ()
fffffe8000c7f7e0 ufs:ufs_fiologenable+63 ()
fffffe8000c7fd60 ufs:ufs_ioctl+3e0 ()
fffffe8000c7fdc0 genunix:fop_ioctl+3b ()
fffffe8000c7fec0 genunix:ioctl+180 ()
fffffe8000c7ff10 unix:sys_syscall32+101 ()

OK, so we should now have a crashdump to look at.

While the machine is rebooting, it occurs to me that if we put this ufs onto an external USB device, we might actually have an exploitable issue here, once the new hal/rmvolmgr framework is in place (nv_51) if we try to automatically mount ufs devices.

core file:      /var/crash/v40z-c/vmcore.0
release:        5.11 (64-bit)
version:        snv_46
machine:        i86pc
node name:      v40z-c
domain:         aus.cte.sun.com
system type:    i86pc
hostid:         69e47dae
dump_conflags:  0x10000 (DUMP_KERNEL) on /dev/dsk/c1t1d0s1(517M)
time of crash:  Sun Nov 12 13:08:18 EST 2006
age of system:  34 days 1 hours 42 minutes 34.95 seconds
panic CPU:      3 (4 CPUs, 7.56G memory)
panic string:   BAD TRAP: type=e (#pf Page fault) rp=fffffe8000c7f2c0 addr=fffffe80fe39d6c4

sanity checks: settings...vmem...sysent...clock...misc...done

-- panic trap data  type: 0xe (Page fault)
  addr: 0xfffffe80fe39d6c4  rp: 0xfffffe8000c7f2c0
  savfp 0xfffffe8000c7f440  savpc 0xfffffffffbb70950
  %rbp  0xfffffe8000c7f440  %rsp  0xfffffe8000c7f3b0
  %rip  0xfffffffffbb70950  (ufs:alloccgblk+0x42f)

  0%rdi 0xffffffff8d60b000  1%rsi 0xffffffff8930c308  2%rdx               0xb5
  3%rcx               0xb5  4%r8  0xfffffe80fe39d6c0  5%r9              0x12f0

  %rax                 0x8  %rbx          0x361005a8
  %r10                   0  %r11  0xfffffffffbcd9ff0  %r12               0x5a8
  %r13  0xffffffff8930c000  %r14  0xffffffff8d60b000  %r15  0xffffffff99656c00
  %cs       0x28 (KCS_SEL)
  %ds       0x43 (UDS_SEL)
  %es       0x43 (UDS_SEL)
  %fs          0 (KFS_SEL)
  %gs      0x1c3 (LWPGS_SEL)
  %ss       0x30 (KDS_SEL)
  trapno     0xe (Page fault)
  err        0x2 (page not present,write,supervisor)
  %rfl   0x10286 (parity|negative|interrupt enable|resume)
  fsbase 0xffffffff80000000 gsbase 0xffffffff8901c800
ufs:alloccgblk+0x42f()
ufs:alloccg+0x473()
ufs:hashalloc+0x50()
ufs:alloc+0x14f()
ufs:lufs_alloc+0xf3()
ufs:lufs_enable+0x261()
ufs:ufs_fiologenable+0x63()
ufs:ufs_ioctl+0x3e0()
genunix:fop_ioctl+0x3b()
genunix:ioctl+0x180()
unix:_syscall32_save+0xbf()
-- switch to user thread's user stack --

The trap has occurred in alloccgblk+42f() in the ufs code.

ufs:alloccgblk+0x410            call   +0xe0a4  (ufs:clrblock+0x0)
ufs:alloccgblk+0x415            decl   0x1c(%r13)	; cgp->cg_cs.cs_nbfree--
ufs:alloccgblk+0x419            decl   0xc4(%r14)	; fs->fs_cstotal.cs_nbfree--
ufs:alloccgblk+0x420            movslq 0xc(%r13),%r8	; %r8 <- cgp->cg_cgx
ufs:alloccgblk+0x428            addq   0x2d8(%r14),%r8	; %r8 <- fs->fs_u.fs_csp[%r8]
ufs:alloccgblk+0x42f            decl   0x4(%r8) <-- panic here

We've just made a call to ufs:clrblock() and are decrementing something after a long list of pointer dereferencing. We only call clrblock() once in this routine, so that puts us at:

   428  #define fs_cs(fs, indx) fs_u.fs_csp[(indx)]

  1238          clrblock(fs, blksfree, (long)blkno);
  1239          /\*
  1240           \* the other cg/sb/si fields are TRANS'ed by the caller
  1241           \*/
  1242          cgp->cg_cs.cs_nbfree--;
  1243          fs->fs_cstotal.cs_nbfree--;
  1244          fs->fs_cs(fs, cgp->cg_cgx).cs_nbfree--;

Panicing on line 1244.

I should note at this point that the source I am quoting is from the same source tree as opensolaris.

After the macro expansion, it becomes

  1244          fs->fs_u.fs_csp[cgp->cg_cgx]--

So what is cgp->cg_cgx?

SolarisCAT(vmcore.0/11X)> sdump 0xffffffff8930c000 cg cg_cgx
   cg_cgx = 0x6f0000

This is probably a trifle on the largish side, which would explain how we have ended up in unmapped memory.

The address we end up with for the (struct csum \*) is 0xfffffe80fe39d6c0

If we go back to look at fs->fs_ncg, we see that there were only two cylinder groups allocated. We have an obvious inconsistancy.

Also, interestingly, this is not dieing in the mount(2) system call. It's dieing in a subsequent ioctl(2). This ioctl appears to be the one enabling ufs logging.

So how might we handle this?

Now, as the filesystem is already mounted and we are in a subsequent ioctl(), we can't fail the mount. Could we fail in alloccgblock() and have the error propogate back up to the process making the ioctl()?

Walking back up the stack, we see that alloccg() handles alloccgblk() returning 0. Now if alloccg() returns 0 to hashalloc(), in this instance, we'll first try to do a quadratic rehash, which will also fail, so we'll fall through to the brute force search. As this starts at cylinder group 2 and there are only 2 cylinder groups, this will call alloccg() once and fail, falling through to return 0 to alloc(). Note that no matter how many times we end up in alloccgblock() it has the same arguments, so it would fail the same way.

In alloc(), it notes that we did not get a block returned and assumes this is because some other thread grabbed the last block. It then treats the whole thing as if we ran out of space and returns ENOSPC to lufs_alloc(). lufs_alloc() catches this, frees up everything and returns the error (ENOSPC) to lufs_enable(), which in turn catches it cleans up and returns it to ufs_fiologenable() and the error is eventually passed back to user space. While not exactly the error we would have hoped, the end result would be that logging would not be turned on and the system would not panic due to this corrupted filesystem.

I'll log this as a bug against ufs for Solaris 10 and nevada

Update

I have logged CR 6492771 against this issue. The link for the CR should work some time in the next 24 hours, but the content as logged is pretty much a cut and paste from this blog entry.

Update 2

The bug I logged has been closed as a duplicate of

4732193 ufs will attempt to mount filesystem with blatantly-bad superblock

Technorati Tags: , , ,

Wednesday Nov 08, 2006

20061106 non-debug encumbered binaries available (sparc/x64)

They can be downloaded from The Download Centre. I've also updated the MD5 Checksums for the extra files.

Steve and I (at Rich Lowe's suggestion) have done a special "nightly" drop into The Download Centre.

Instructions can be found in an earlier blog.

Dennis Clarke has also written a great step by step guide.

You will also note that the downloads page for this build no longer contains the source code drop.

I had a chat with Steve Lau about this.

He has decided to no longer deliver it for the nightly (weekly) drops as the mercurial repository is available.

You may not have noticed, but Mercurial (hg) has been a part of Solaris since build 45 and lives in /usr/bin/hg.

Technorati Tags: ,

Sunday Nov 05, 2006

20061103 flag day non-debug encumbered binaries available (sparc/x64)

A special drop of encumbered binaries due to the flag day for IPSec Tunnel Reform (PSARC/2005/516).[Read More]

Friday Nov 03, 2006

You Tube vs UTube - Turn the problem on it's head for a solution?

I noticed YouTube faces UTube Lawsuit on El Reg when I came in this morning. While discussing it amongst colleagues (and I don't pretend to know the ins and outs o fthe legalities involved in the suit), it occurred to me that these folks were missing a giant opportunity here.

The way I read it is that the basis of their problems stems from:

In the suit, Universal Tube alleges that its traffic has soared from around 1,500 hits in a month to over two million because of vast numbers of visitors looking for "lewd and other disgusting video".

The answer, as I see it is really simple. Get hold of a Try and buy T2000 server to handle the load, and place some google ads on the page. The income from the ads would then fund the infrastructure upgrades.

Simple huh? :-)

Technorati Tags: , , T2000, ,

About

* - Solaris and Network Domain, Technical Support Centre


Alan is a kernel and performance engineer based in Australia who tends to have the nasty calls gravitate towards him

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Links
Blogroll

No bookmarks in folder

Sun Folk

No bookmarks in folder

Non-Sun Folk
Non-Sun Folks

No bookmarks in folder