Wednesday Aug 08, 2007

Finding undocumented command options

I had a colleague this morning asking about undocumented (ie not listed in usage or man pages) options in a command. The actual command doesn't really matter, but I was feeling a little lazy and couldn't bothered looking up the source code to the command (which actually wasn't in ON). Almost immediately I thought of DTrace.

Let's have a look at ls as an example. I'll give it a dummy directory as I really don't care about the output.

$ dtrace -q -n 'pid$target::getopt:entry {trace(copyinstr(arg2));}' -c 'ls /nosuchfile'                     
/nosuchfile: No such file or directory
aAbcCdeEfFghHilLmnopqrRsStux1@vV

As you can see, the second line of output is printing out the third argument (arg2) to getopt(3c), which will list every option that getopt(3c) will recognise for the command.

Of course I could have prettied it up, but it's a one liner, I know what the output means.

The point being, that DTrace is just another sysadmin tool to be used in day to day operations.

Technorati Tags: , ,


Yes I know I have been lax in my blogging, I'm going to start doing something about that, starting with this one :-)

Wednesday Feb 28, 2007

in.telnetd exploit doing the rounds

I am hearing talk about an exploit of the in.telnetd issue doing the rounds. This affects Solaris 10 and Solaris Express.

References at sans.org and asert.arbornetworks.com

Now would be a particularly good to disable your telnet daemon:

# svcadm disable telnet

If you must run telnetd, then you need to get the patches referred to in Sun Alert 102802. The patches are freely available on sunsolve.

120068-03 SunOS 5.10_sparc: in.telnetd Patch
120069-03 SunOS 5.10_x86: in.telnetd Patch

In spite of what the README says, these patches do not require a reboot.

Some Details

While things are still sketchy, it looks like it propogates by connecting as both "adm" and "lp" and copies sparc and x86 binaries into /var/adm/sa/.adm, along with crontab entries for both of these users.

A quick check to see if you have been infected is to check the mode of /var/adm/wtmpx. If it is 0646 and you have the aforementoined directory, it is likely yoyu are infected.

First off, disable the telnet daemon as described above. Clear out the cron entries that were added for adm and lp. There will also be a program running listening on port 32982. It will likely be called the same name as the only non-dot-file in /var/adm/sa/.adm. Make sure you kill the right one, as they choose from a number of solaris daemons for the naming. Run pfiles on it to look for port 32982.

Update

For more information see the security blog.

Technorati Tags: , , ,

Tuesday Feb 13, 2007

The in.telnetd vulnerability/exploit (3rd update)

Before I get into the meat of this posting, let me acknowledge that, yes, this was an almighty cock up and should not have happened. It did happen. Let's move on.

Also, while I might not agree with the publication of zero day exploits. Again, It happened. There's really not much I can do about that. There's really no point in being upset about it.

The upside to the posted exploit was the fact that because the code was available, the poster included an analysis of what was going wrong, pointing at the code that was broken. This almost certainly saved us some time in troubleshooting the issue. For this part of the post, you have my thanks.

I would certainly be interested if the person who posted the exploit could tell us how he found the problem; for no other reason, than I'm simply interested.

Anyway, this blog is supposed to be about getting it fixed.

All the times below are Australia/NSW.

One of our National SSEs (Rodney) was on-site with a large customer yesterday. This customer had asked him about a telnet exploit and described the problem to him. Rodney gave me a call and asked me about it at about 1pm. I hadn't and on hearing the description (initially only described to him as a root exploit) Ttwo of us (thanks for your help Chris) dove into the code to start looking at how telnet -l-froot could behave as it did. At this point I did not know about the zero day exploit posting. Once we worked out what was going on, I called Rodney back and explained the full implications of the bug to him so he could explain it to the customer.

We told them that they could block the root vulnerability by uncommenting the CONSOLE= line in /etc/default/login. Note that this has been the default since Solaris 10 update 2 almost forever. However, I still see lots of customer configurations where it is commented out. The only other way to protect against the other implications (login as any user without a password) would be to disable the telnet service until we could fix the issue. eg

# svcadm disable svc:/network/telnet:default

We then started looking through the code to determine the best way to fix this. I logged an internal escalation and was in the process of logging a high priority bug when I saw the following in the SCCS history of usr/src/cmd/cmd-inet/usr.sbin/in.telnetd.c

D 1.67  07/02/11 19:46:41 danmcd        90 89   00009/00010/04896
6523815 LARGE vulnerability in telnetd

I immediately had a look at the bug and banged of an email to Dan stating that I could probably get IDR patches built for on10 pretty quickly. After a brief discussion of the bug and the fix, he pointed me at the manager of the group responsible for the backport and I started the backport (actually a very simple fix).

I got the IDRs built and basic testing done by about 5pm and started writing the Sun Alert.

The documentation for how to write a Sun Alert and specifically the actions that need to be taken to get interim fixes available were spot on and I sent off the initial draft at about 6:45 along with sending a request for getting the IDR patches turned into ISR patches (Interim Security Relief) and getting them published.

Just before 9pm, I started getting into discussions with UK based folk in Derrick Scholl's group about getting the Sun Alert out and what needed to happen for me to get a gate open to get the fix back into the patch gate.

Thanks to Angela, Paul, Brent and Bill for working hard to get to the point that I could log the RTI at 10:10, and start doing the minor nit type stuff that needed to be done before Bill could pass the RTI onto the on10-patch gatekeepers and I could go home (at about 11:15).

As an aside, I missed my train connection by four minutes due to a late running North Shore train and spent an hour sat on a blacked out Hornsby station, getting home at about 2:30am)

While I was traveling (and sleeping) the heads up went out to the gatekeepers and all folk who needed to know about this so that when I came online at 8 this morning, I had very little to do before doing the actual putback into the patch gate (which happened at about 8:30).

The gatekeepers immediately closed the gate and started work on a patch.

The reason that I've detailed what I've been through with this is to point out one thing.

The speed at which I was able to do this and get to the point that an ISR patch will be shortly available publicly, is nothing short of phenomenal. For Sun to respond to and address a vulnerability like this in around 24 hours would have been completely unheard of even two to three years ago.

But it's not just the processes here. What really made for speed here was an incredibly focussed and helpful who had an interested in rapidly getting this addressed. Without the help of folks like Dan, Rodney, Chris, Angela, Paul, Brent, Bill and Seth, and not forgetting the gatekeeping team for pulling out the stops to start building a formal patch, none of this would be possible. If I've missed anyone, please forgive me, I didn't get a lot of sleep last night :)

I love working for a company that has people like this.


update 1

The ISR patches are available for free download from http://sunsolve.sun.com/tpatches. The details of the patches are:

   IDR125457-01 SunOS i386_x86: in.telnetd can call login with an
                option given as a username

   IDR125456-01 in.telnetd can call login with an option given as a
                username

update 2

Sun Alert 102802 is publicly available talking about this issue. Section 4 should shortly be modified to add the following paragraph:

Interim Security Relief (ISRs) are available from http://sunsolve.sun.com/tpatches for the following releases:

SPARC Platform

Solaris 10 IDR125456-01

x86 Platform

Solaris 10 IDR125457-01

Note: This document refers to one or more Interim Security Relief (ISRs) which are designed to address the concerns identified herein. Sun has limited experience with these (ISRs) due to their interim nature. As such, you should only install the (ISRs) on systems meeting the configurations described above. Sun may release full patches at a later date, however, Sun is under no obligation whatsoever to create, release, or distribute any such patch.

Update 3

I've just been informed that the formal patches are release ready and should be released to sunsolve in the next few hours. Keep an eye out for:

120068-02 SunOS 5.10_sparc: in.telnetd Patch
120069-02 SunOS 5.10_x86: in.telnetd Patch

As these are security patches, they will be publicly available.

Technorati Tags: , , ,

Monday Feb 12, 2007

b57 non-debug ON encumbered binaries

Sorry it's taken so long for these. On top of having one of the most hectic two weeks I think I've ever had in this job (with customer escalations), I tripped over a few things with having to upgrade the SPARC build machine and then having to rebuild my SUNWonbld stuff for both architectures as some things needed updating in it and then discovering a couple of little things I had updated on the build machines and not in my source tree (sigh). Anyway, they are up now.

They can be downloaded from The Download Centre. I've also updated the MD5 Checksums for the extra files.

Note that as of build 54, you no longer need the fiddle to usr/src/Makefile that was previously required to do non-debug builds, so have a look at the new instructions.

Dennis Clarke has also written a great step by step guide.

Technorati Tags: ,

Wednesday Jan 24, 2007

b56 non-debug ON encumbered binaries

They can be downloaded from The Download Centre. I've also updated the MD5 Checksums for the extra files.

Note that as of build 54, you no longer need the fiddle to usr/src/Makefile that was previously required to do non-debug builds, so have a look at the new instructions.

Dennis Clarke has also written a great step by step guide.

Technorati Tags: ,

Friday Jan 12, 2007

Modifying your predictions does not make them true

I've been hanging off writing this as I wanted to think about it a bit first.

I'm referring to Bob Cingley's predictions for 2006.

You might recall that I made some comments on him changing the prediction so he got it right last year. Well, he's playing the same games again this year.

The actual prediction was:

Sun's Woes Continue

Still no good news for Sun. Those Galaxy servers are very nice, but they aren't enough to support the company and Eric Schmidt is too smart (I hope) to bail out his old firm.

And in his evaluation of his predictions he writes:

4) More bad news for Sun. That's true.

Notice the subtle difference. More bad news in the evaluation and Still no good news in the prediction.

Reading the prediction, one has to think that if Sun had any good news last year, then he scored a clean miss.

I definitely think we had lots of good news last year.

Like many folks in other forums who have commented on this particular prediction (When was the last time you saw non-Sun folks standing up for Sun on slashdot!), I think the question has to be asked, What axe has Robert X Cringley got to grind against Sun Microsystems?

Bob, if you have any shred of credibility left, you really should come clean and score #4 from 2006 a clear miss.

One other interesting thing that I just noticed from my prior discussion on this. In January last year, the Sun prediction was #5, it's #4 now. Did a prediction get deleted?

Technorati Tags: , , ,

Sunday Jan 07, 2007

b55 non-debug ON encumbered binaries

They can be downloaded from The Download Centre. I've also updated the MD5 Checksums for the extra files.

Note that as of build 54, you no longer need the fiddle to usr/src/Makefile that was previously required to do non-debug builds, so have a look at the new instructions.

Dennis Clarke has also written a great step by step guide.

Technorati Tags: ,

Tuesday Dec 12, 2006

Tamarack, Gnome 2.17 and external devices

This is a combination that I've been hanging on. vold never quite handled my cds, dvds and usb media quite right.

I'm happy to say that for the most part it all just works like it should.

Well, I have a multi-partitioned usb device which exhibits an interesting problem.

First, let's have a look at how the device (it's a 60gb usb 2 disk) is partitioned.

             Total disk size is 57231 cylinders
             Cylinder size is 2048 (512 byte) blocks

                                               Cylinders
      Partition   Status    Type          Start   End   Length    %
      =========   ======    ============  =====   ===   ======   ===
          1                 Ext Win95         1  28615    28615     50
          2                 Solaris2      28616  57230    28615     50 

Partition 1 has a fat-32 filesystem.
Partition 2 has an SMI label and slice 0 is a zfs pool with the imaginitve name 'usb'.

# fstyp /dev/dsk/c4t0d0p1
pcfs
# fstyp /dev/dsk/c4t0d0s0
zfs

If I insert this disk into my notebook running b53, I get the notice about not being able to mount the zfs pool (which I expect), but it does not notice the fat-32 partition, let alone mount it:

# rmmount -l
/dev/dsk/c4t0d0s0    rmdisk,rmdisk0,usb

I had a chat with one one of the developers about this. He found that if we place the zfs pool directly into the second partition, then everything works as you would expect. The different controller numbers and zfs pool name are purely due to using another disk and machine in anotehr continent.

# fstyp /dev/dsk/c2t0d0p1
pcfs
# fstyp /dev/dsk/c2t0d0p2
zfs
# rmmount -l
/dev/dsk/c2t0d0p0:1  rmdisk,rmdisk0,NONAME,/media/NONAME
/dev/dsk/c2t0d0s2    rmdisk,rmdisk0,lacie_p2

It turns out that if hal discovers an SMI label on any partition, it does not probe any of the other logical disks, leading to the logging of

6502219 If SMI label exists, hal does not probe logical disks

In the meantime as I really don't want to muck around with the existing zpool, I'll continue to mount my pcfs manually, but it will be nice when this one is fixed.

It's probably also worth mentioning that "zpool import usb" just works.

Technorati Tags: ,

Gotcha bug for SXCR 53 and 54

A few folks have come across an issue after using live upgrade to go to build 54 of nevada, so to save some hassles for when it comes out as an SXCR, I thought I'd post this. It also seems that some folks hit it with 53.

It looks like after such a live upgrade the mountpoints for /tmp and /var/run are created with 0700 permissions. The /tmp permisssion will prevent a non-root login in gnome. The /var/run will cause more subtle issues with stuff that wants to access in there. One example of this would be from rhythmbox.

\*\* (rhythmbox:1439): WARNING \*\*: hal_initialize failed: Failed to connect to socket /var/run/dbus/system_bus_socket: Permission denied

The actual issue is the permissions of the mount points, and you can't fix these in any way that will be persistant across reboots (as they are tmpfs's) unless you unmount the filesystems. You could try booting with "-m milestone=none" and fixing the underlying moountpoints, or you could try the following. If you don't have anything using those mountpoints yet, you'll get away with it.

# umount /tmp
# umount /var/run
# chmod 1777 /tmp
# chmod 755 /var/run
# mount /tmp
# mount /var/run

This has been logged as

6497684 Unable to login due to live upgrade setting incorrect permissions on /tmp and /var/run

Which, unfortunately is not available externally (being logged against live-upgrade.

Of course the preventative thing to do would be to mount the alternate boot environment and fix the permissions before booting it.

Technorati Tags: ,

Monday Dec 11, 2006

b54 non-debug ON encumbered binaries available

They can be downloaded from The Download Centre. I've also updated the MD5 Checksums for the extra files.

Note that this is the first of the named builds to incorporate the change to usr/src/Makefile to remove the fiddle previously required to do non-debug builds, so have a look at the new instructions.

Dennis Clarke has also written a great step by step guide.

Technorati Tags: ,

Thursday Nov 30, 2006

20061127 non-debug ON encumbered binaries available

As previously stated, the usr/src/Makefile fix went into ON just before build 54 closed. Coincidentally, this is the date of the source drop for this week.

As such, you no longer have to either rename the closed-bins directory or patch usr/src/Makefile.

Before I go any further, here's a link to the new closed binaries for this build.

Again, it's time to redo the instructions for building non-debug ON.

  1. Extract the encumbered binaries that you require.

  2. Modify the environment file. In my case this is generally called opensolaris.sh and I keep my copy in the directory that contains usr and proto, often referred to as ${CODEMGR_WS}.

    In order to only do a non-debug build go to the line that defines NIGHTLY_FLAGS and remove the D and F options. If you want nightly to build both, only remove the D option.

    You also want to ensure that in the environment file that ON_CLOSED_BINS points to the directory that contains the root_\* files.

Everything else should be as per how you would normally do a debug build and as long as you take these things into account Dennis Clarke's step by step guide is a good outline of how to build a non-debug ON consolidation.

Technorati Tags: ,

Wednesday Nov 29, 2006

usr/src/Makefile now recognises non-debug encumbered binaries

OK folks, I managed to get the usr/src/Makefile fix into build 54 just before it closed (about an hour). The actual fix is the same diffs as I previously posted.

changeset:   3225:d04f67c1972c
user:        ah89892
date:        Mon Nov 27 21:56:37 2006 -0800
summary:     6495870 usr/src/Makefile needs to recognise the non-debug encumbered binary deliverable

This means that as of the source associated with the mercurial drop of d04f67c1972c and build 54, you no longer need to rename the closed binaries directory or fiddle with the Makefile. If you are doing a non-debug build, the make will look in the right place for the closed binaries (i.e. where they extracted).

I hope to have the encumbered binaries up for 20061127 sometime in the next few hours.

Technorati Tags: ,

Saturday Nov 25, 2006

b53 non-debug encumbered binaries (and new usr/src/Makefile diff) available

The encumbered binaries for Build 53 of OpenSolaris are now available at The Download Centre along with the updated MD5 Checksums.

In order to put the usr/src/Makefile changes back into ON I've made a couple of minor changes to the diffs that I previously posted. The Makefile is now a little more clear about what it's doing with it's logging and error messages for doing the open build.

OK, the reason for this is that I managed to get caught out "testing" the Makefile change and forgot to copy the new Makefile into place. The logging that was in there didn't allow me to see that I was doing a non-debug build with the debug encumbered binaries (because it didn't have the new code in there). It now is clear about where it is copying the encumbered binaries from and about what it expected to find if it can't find them.

As such, it's probably time to re-do the instructions for how to build a non-debug ON tree, not forgetting to mention Dennis Clarke's great step by step guide.

First off, as the current usr/src/Makefile does not know about the directory name that the non-debug binaries extract in to, we have two options.

  1. Extract the encumbered binaries and then rename it such that the '-nd' suffix is now longer in the directory name.

    e.g.

    • closed/root_sparc (instead of closed/root_sparc-nd)
    • closed/root_i386 (instead of closed/root_i386-nd)
  2. Modify usr/src/Makefile as per the following diff output. This tells make to grab the non-debug encumbered binaries from the directory that the tarball extracts into. The upside of this is that it then anables you to build bfu archives for both debug and non-debug from nightly.

    ------- usr/src/Makefile -------
    
    Index: usr/src/Makefile
    --- /ws/onnv-gate/usr/src/Makefile      Mon Oct 16 15:10:55 2006
    +++ /export5/ah89892/onnv-6495870/usr/src/Makefile      Tue Nov 21 14:44:20 2006@@ -22,7 +22,7 @@
     # Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
     # Use is subject to license terms.
     #
    -# ident        "@(#)Makefile   1.224   06/10/16 SMI"
    +# ident        "@(#)Makefile   1.225   06/11/21 SMI"
     #
     # Makefile for system source
     #
    @@ -117,14 +117,17 @@
            @cd lib; pwd; $(MAKE) install
    
     closedbins: FRC $(ROOTDIRS)
    -       @if [ "$$CLOSED_IS_PRESENT" = no ]; then \\
    -               if [ ! -d "$$ON_CLOSED_BINS/root_$(MACH)" ]; then \\
    +       @CLOSED_ROOT="$$ON_CLOSED_BINS/root_$(MACH)$${RELEASE_BUILD+-nd}"; \\
    +       if [ "$$CLOSED_IS_PRESENT" = no ]; then \\
    +               if [ ! -d "$$CLOSED_ROOT" ]; then \\
                            $(ECHO) "Error: if closed sources are not present," \\
                                "ON_CLOSED_BINS must point to closed binaries."; \\
    +                       $(ECHO) "root_$(MACH)$${RELEASE_BUILD+-nd} is not" \\
    +                           "present in $$ON_CLOSED_BINS."; \\
                            exit 1; \\
                    fi; \\
    -               $(ECHO) "Copying closed binaries from $$ON_CLOSED_BINS"; \\
    -               (cd $$ON_CLOSED_BINS/root_$(MACH); tar cf - .) | \\
    +               $(ECHO) "Copying closed binaries from $$CLOSED_ROOT"; \\
    +               (cd $$CLOSED_ROOT; tar cf - .) | \\
                        (cd $(ROOT); tar xBpf -); \\
            fi
        

The next part is to modify the environment file. In my case this is generally called opensolaris.sh and I keep my copy in the directory that contains usr and proto, often referred to as $CODEMGR_WS.

In order to only do a non-debug build go to the line that defines NIGHTLY_FLAGS and remove the D and F options. If you want nightly to build both, only remove the D option.

Make sure that you have all of the encumbered-binaries files extracted into the right place and kick off the nightly as you normally would.

Technorati Tags: ,

Wednesday Nov 08, 2006

20061106 non-debug encumbered binaries available (sparc/x64)

They can be downloaded from The Download Centre. I've also updated the MD5 Checksums for the extra files.

Steve and I (at Rich Lowe's suggestion) have done a special "nightly" drop into The Download Centre.

Instructions can be found in an earlier blog.

Dennis Clarke has also written a great step by step guide.

You will also note that the downloads page for this build no longer contains the source code drop.

I had a chat with Steve Lau about this.

He has decided to no longer deliver it for the nightly (weekly) drops as the mercurial repository is available.

You may not have noticed, but Mercurial (hg) has been a part of Solaris since build 45 and lives in /usr/bin/hg.

Technorati Tags: ,

Sunday Nov 05, 2006

20061103 flag day non-debug encumbered binaries available (sparc/x64)

A special drop of encumbered binaries due to the flag day for IPSec Tunnel Reform (PSARC/2005/516).[Read More]
About

* - Solaris and Network Domain, Technical Support Centre


Alan is a kernel and performance engineer based in Australia who tends to have the nasty calls gravitate towards him

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today
Links
Blogroll

No bookmarks in folder

Sun Folk

No bookmarks in folder

Non-Sun Folk
Non-Sun Folks

No bookmarks in folder