Don't let find drop into /proc

Today I had the pleasure of looking at a crash dump of a system where ps, prstat et al were hanging. I never got the chance to get to the bottom of it due to meetings. However there was one command that was running on that dump that jumped out as doing lots of harm to the performance of the system for no benefit. The command was “find / .....”. The find had descended into “/proc” where it would have been walking all the directories locking and unlocking processes when what it was looking for was never going to be in /proc. Clive pointed out similar issues from running ps, prstat on performance by walking /proc and supplied the dtrace to show it.

Given that it is so rarely the case that you would want a find starting form “/” to drop into /proc I wonder if it would not be better if either /proc was invisible in the same way as the “.zfs” directories can be invisible in ZFS. I'd be interested in comments on that.

In the mean time if you really want to run find from “/” do this:

find / \\( -fstype proc -prune \\) -o ....


It will not descend into procfs but in all other respects will be the same.

Comments:

Chris:

What a coincidence! We ran into this issue on one of our machines that was running Solaris 10 3/05HW1
:- ps, prstat, top, pgrep, basically anything that touched /proc was hung. Tools such as mpstat, vmstat, iostat, etc. showed nothing unusual. I frantically ran a couple of DTrace and other commands to glean as much info as possible but before I could find anything useful the machine was rebooted by a local sysadmin and he did not get a crash dump - bummer ;-(

Can one or more "find / ..." cause this much havoc? Could this potentially be a DOS attack? How do we recover from this?

Thank you posting this.

Posted by PJ on September 12, 2007 at 04:41 PM BST #

Well obviously you can beat the sysadmin until they learn to type "reboot -d" when these situations happen. However what you really should have done is take a live crash dump. Since all the processes that are hung are well hung the live dump contains all you should need.

"find / ..." in itself will not hang the box up however poking around in /proc is an expensive operation which effects the perfomance of the whole system and if NFS is involved there are ways you can end up hanging waiting for a down NFS server while you hold locks that prevent others scanning /proc. Specifically using commands that look at the address space can block until the NFS server returns. Hence choosing the forms of ps that don't grok in the address space (avoid the www options to get the full arguments) is a good thing.

Clearly avoid letting find go down into /proc unless you really need to reduces the risk.

Posted by guest on September 13, 2007 at 02:02 AM BST #

What's the method to take a live crash dump?

There is absolutely no way that /proc should be made invisible a la .zfs, thanks. People should just stop doing stupid things
(#include <allusions to the "rm -r /" travesty>) ;-)

Posted by Ceri Davies on September 13, 2007 at 03:53 AM BST #

Live crash dumps. Use savecore -L you need a dedicated dump device but that can just be a file and can be changed on the fly.

Posted by Chris Gerhard on September 13, 2007 at 04:11 AM BST #

Ace, thanks Chris.

Posted by Ceri Davies on September 13, 2007 at 05:39 AM BST #

Post a Comment:
Comments are closed for this entry.
About

This is the old blog of Chris Gerhard. It has mostly moved to http://chrisgerhard.wordpress.com

Search

Archives
« July 2014
MonTueWedThuFriSatSun
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
   
       
Today