Monday Apr 26, 2010

Grabbing information from the X server

Jay came into my office last week complaining his mouse was grabbed by a client and wouldn't let him click in any windows, but he couldn't tell which one. He remembered I said I had a script to find this out, but hadn't told him where it was. So I logged into his workstation, su'ed to root and ran a couple commands:

root@overthere:~# ps -ef | grep Xorg
jacotton   881   880   0   Apr 08 vt/2      274:24 /usr/bin/Xorg :0 -nolisten tcp -br -auth /var/run/gdm/auth-for-gdm-O_aWTb/datab
root@overthere:~# /usr/demo/Xserver/mdb/list_Xserver_devicegrab_client -p 881
Device "Virtual core pointer" id 2: 
  -- active grab 3a00000 by client 29

Device "Virtual core keyboard" id 3: 
  -- active grab 3a00000 by client 29

Device "Virtual core XTEST pointer" id 4: 
  -- no active grab on device

Device "Virtual core XTEST keyboard" id 5: 
  -- no active grab on device

Device "mouse" id 6: 
  -- no active grab on device

Device "input" id 7: 
  -- no active grab on device

Device "hotkey" id 8: 
  -- no active grab on device

Device "keyboard" id 9: 
  -- no active grab on device

root@overthere:~# /usr/demo/Xserver/mdb/list_Xserver_clients -p 881
currentMaxClients = 41
    0           0 ??? - NULL ClientPtr->osPrivate
    1          22  33 880 
    2          20  35 912 
    3           9  36 915 
   29     6926083  59 15839 
15822 /bin/bash /usr/bin/firefox
  15835 /bin/bash /usr/lib/firefox/ /usr/lib/firefox/firefox-bin
    15839 /usr/lib/firefox/firefox-bin

So we knew it was firefox holding the grab, and killing that process released it.

The scripts are actually simple wrappers around a loadable module for the Solaris mdb modular debugger. While I prefer source level debuggers like dbx & gdb for interactive use, the mdb module system works well for this style of use. The mdb modules are compiled C code, so can use the headers from the X server to get the type information, instead of requiring debugging information such as stabs or DWARF be built into the X server binary.

These were originally written for Solaris 9, in the dark days before DTrace, but are still useful, because they get information you don't know you need until after it's happened - for instance, with the Xserver DTrace probes you can get the client information from the client-connect and client-auth probes when the client connects, or see which clients send grab requests from the request probes when they make the calls - but when your server is already grabbed it's too late to trace unless you know how to reproduce the issue.

These have been kept in our source tree for a while, but weren't built as part of the regular builds so were often out of date when the server internals changed how things like the devPrivate fields were stored, and thus needed porting to the new X server versions when you needed them. And even when they were up-to-date, you needed to have a full X server source tree to build them, since they used headers not included in the server SDK package.

To solve this, in Nevada build 135 (sorry, that was just after the stable branch for the 2010.03 release forked off, so it won't be in there) I integrated them into the normal build process and delivered them in the X server packages. The mdb module is delivered in /usr/lib/mdb/proc/ with links to it named,, and so that it should be automatically loaded when mdb attaches to a process or core of any of those programs. The scripts were delivered in /usr/share/demo/Xserver/mdb, along with a README file explaining how to use either the direct mdb module or the wrapper scripts.

Porting these to other debuggers architectures for scripting should be possible, though you'd need to deal with either having the X server debugging information available or converting the C language headers & traversal algorithms to Studio dbx's ksh or gdb's Python or whatever scripting solution your debugger uses. The grab and client information should be identical across the X servers from the xorg-server sources on all platforms (though as noted above, varying by xorg-server version as the internals change), except for the client process id information. Most platforms get the client pid at connection time, so it can be logged in the client audit log if you run with -audit, or made available to the client-auth dtrace probe, but then discard that information. On Solaris & OpenSolaris however, we keep it in a client devPrivate for the SolarisIA extension to use to adjust the priority boost given to the process with focus, so the mdb module can look it up there. Of course, there's no guarantee that the client hasn't forked a child or somehow else passed control of its X connection in a way the server wasn't informed about, but it's accurate far more often than not. Since the file descriptor information is available on all platforms, you might be able to determine the client via that somehow, such as through lsof - for instance, I've implemented an enhancement to pfiles on Solaris to show peer processes for local IPC that hopefully I'll find time to get reviewed and integrated soon.

There's of course a lot more information in the X server that might be useful to find - these were each written to solve specific problems on machines we couldn't login to directly to debug ourselves. What other problems could be solved if we had similar modules to expose other information? We can certainly enhance these as we find more - let us know if you find some.


Engineer working on Oracle Solaris and with the X.Org open source community.


The views expressed on this blog are my own and do not necessarily reflect the views of Oracle, the X.Org Foundation, or anyone else.

See Also
Follow me on twitter


« July 2016