Grabbing information from the X server
By alanc on Apr 26, 2010
Jay came into my office last week complaining his mouse was grabbed by a client and wouldn't let him click in any windows, but he couldn't tell which one. He remembered I said I had a script to find this out, but hadn't told him where it was. So I logged into his workstation, su'ed to root and ran a couple commands:
root@overthere:~# ps -ef | grep Xorg jacotton 881 880 0 Apr 08 vt/2 274:24 /usr/bin/Xorg :0 -nolisten tcp -br -auth /var/run/gdm/auth-for-gdm-O_aWTb/datab root@overthere:~# /usr/demo/Xserver/mdb/list_Xserver_devicegrab_client -p 881 Device "Virtual core pointer" id 2: -- active grab 3a00000 by client 29 Device "Virtual core keyboard" id 3: -- active grab 3a00000 by client 29 Device "Virtual core XTEST pointer" id 4: -- no active grab on device Device "Virtual core XTEST keyboard" id 5: -- no active grab on device Device "mouse" id 6: -- no active grab on device Device "input" id 7: -- no active grab on device Device "hotkey" id 8: -- no active grab on device Device "keyboard" id 9: -- no active grab on device root@overthere:~# /usr/demo/Xserver/mdb/list_Xserver_clients -p 881 currentMaxClients = 41 CLIENT SEQUENCE # FD PIDS 0 0 ??? - NULL ClientPtr->osPrivate 1 22 33 880 2 20 35 912 3 9 36 915 [...] 29 6926083 59 15839 [...] 15822 /bin/bash /usr/bin/firefox 15835 /bin/bash /usr/lib/firefox/run-mozilla.sh /usr/lib/firefox/firefox-bin 15839 /usr/lib/firefox/firefox-bin [...]
So we knew it was firefox holding the grab, and killing that process released it.
The scripts are actually simple wrappers around a loadable module for the Solaris mdb modular debugger. While I prefer source level debuggers like dbx & gdb for interactive use, the mdb module system works well for this style of use. The mdb modules are compiled C code, so can use the headers from the X server to get the type information, instead of requiring debugging information such as stabs or DWARF be built into the X server binary.
These were originally written for Solaris 9, in the dark days before DTrace, but are still useful, because they get information you don't know you need until after it's happened - for instance, with the Xserver DTrace probes you can get the client information from the client-connect and client-auth probes when the client connects, or see which clients send grab requests from the request probes when they make the calls - but when your server is already grabbed it's too late to trace unless you know how to reproduce the issue.
These have been kept in our source tree for a while, but weren't built as part of the regular builds so were often out of date when the server internals changed how things like the devPrivate fields were stored, and thus needed porting to the new X server versions when you needed them. And even when they were up-to-date, you needed to have a full X server source tree to build them, since they used headers not included in the server SDK package.
To solve this, in Nevada build 135 (sorry, that was just after the stable branch for the 2010.03 release forked off, so it won't be in there) I integrated them into the normal build process and delivered them in the X server packages. The mdb module is delivered in /usr/lib/mdb/proc/ with links to it named Xorg.so, Xvfb.so, Xephyr.so and Xvnc.so so that it should be automatically loaded when mdb attaches to a process or core of any of those programs. The scripts were delivered in /usr/share/demo/Xserver/mdb, along with a README file explaining how to use either the direct mdb module or the wrapper scripts.
Porting these to other debuggers architectures for scripting should be possible, though you'd need to deal with either having the X server debugging information available or converting the C language headers & traversal algorithms to Studio dbx's ksh or gdb's Python or whatever scripting solution your debugger uses. The grab and client information should be identical across the X servers from the xorg-server sources on all platforms (though as noted above, varying by xorg-server version as the internals change), except for the client process id information. Most platforms get the client pid at connection time, so it can be logged in the client audit log if you run with -audit, or made available to the client-auth dtrace probe, but then discard that information. On Solaris & OpenSolaris however, we keep it in a client devPrivate for the SolarisIA extension to use to adjust the priority boost given to the process with focus, so the mdb module can look it up there. Of course, there's no guarantee that the client hasn't forked a child or somehow else passed control of its X connection in a way the server wasn't informed about, but it's accurate far more often than not. Since the file descriptor information is available on all platforms, you might be able to determine the client via that somehow, such as through lsof - for instance, I've implemented an enhancement to pfiles on Solaris to show peer processes for local IPC that hopefully I'll find time to get reviewed and integrated soon.
There's of course a lot more information in the X server that might be useful to find - these were each written to solve specific problems on machines we couldn't login to directly to debug ourselves. What other problems could be solved if we had similar modules to expose other information? We can certainly enhance these as we find more - let us know if you find some.