Wednesday Jun 22, 2011

Hidden exceptions

Occasionally you may find yourself in a Java application environment where exceptions in your code are being caught by the application framework and either silently swallowed or converted into a generic exception. Either way, the potentially useful details of your original exception are inaccessible. Wouldn't it be nice if there was a VM option that showed the stack trace for every exception thrown, whether or not it's caught? In fact, HotSpot includes such an option: -XX:+TraceExceptions. However, this option is only available in a debug build of HotSpot (search globals.hpp for TraceExceptions). And based on a quick skim of the HotSpot source code, this option only prints the exception class and message. A more useful capability would be to have the complete stack trace printed as well as the code location catching the exception. This is what the various TraceException* options in in Maxine do (and more). That said, there is a way to achieve a limited version of the same thing with a stock standard JVM. It involves the use of the -Xbootclasspath/p non-standard option. The trick is to modify the source of java.lang.Exception by inserting the following:

    private static final boolean logging = System.getProperty("TraceExceptions") != null;
    private void log() {
        if (logging && sun.misc.VM.isBooted()) {
            printStackTrace();
        }
    }

Then every constructor simply needs to be modified to call log() just before returning:

    public Exception(String message) {
        super(message);
        log();
    }

    public Exception(String message, Throwable cause) {
        super(message, cause);
        log();
    }

    // etc...

You now need to compile the modified Exception.java source and prepend the resulting class to the boot class path as well as add -DTraceExceptions to your java command line. Here's a console session showing these steps:

% mkdir boot
% javac -d boot Exception.java
% java -DTraceExceptions -Xbootclasspath/p:boot -cp com.oracle.max.vm/bin test.output.HelloWorld
java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.(ZipFile.java:127)
        at java.util.jar.JarFile.(JarFile.java:135)
        at java.util.jar.JarFile.(JarFile.java:72)
        at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:646)
        at sun.misc.URLClassPath$JarLoader.access$600(URLClassPath.java:540)
        at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:607)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:599)
        at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:583)
        at sun.misc.URLClassPath$3.run(URLClassPath.java:333)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.misc.URLClassPath.getLoader(URLClassPath.java:322)
        at sun.misc.URLClassPath.getLoader(URLClassPath.java:299)
        at sun.misc.URLClassPath.getResource(URLClassPath.java:168)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:194)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at sun.misc.Launcher$ExtClassLoader.findClass(Launcher.java:229)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:295)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
java.security.PrivilegedActionException
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:599)
        at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:583)
        at sun.misc.URLClassPath$3.run(URLClassPath.java:333)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.misc.URLClassPath.getLoader(URLClassPath.java:322)
        ...

It's worth pointing out that this is not as useful as direct VM support for tracing exceptions. It has (at least) the following limitations:

  • The trace is shown for every exception, whether it is thrown or not.
  • It only applies to subclasses of java.lang.Exception as there appears to be bootstrap issues when the modification is applied to Throwable.java.
  • It does not show you where the exception was caught.
  • It involves overriding a class in rt.jar, something should never be done in a non-development environment.

Thursday Apr 07, 2011

All the source

The src.zip file often distributed with a JDK only contains source files for some of the classes in rt.jar and the other jars files making up the JRE. This can be quite annoying when debugging. If you have access to the source code for the specific JDK version you are using, then creating a more complete src.zip is fairly straight forward. I find myself doing this quite regularly and so wrote a little script to simplify the task which I've posted here for anyone else who might find it useful.

Thursday Aug 26, 2010

Stacks with split personalities

After an update to one of our Linux test machines, stack overflow detection stopped working in Maxine on Linux. We employ the same mechanism as HotSpot of placing a yellow guard page at the end of the stack. That is, the lowest addressed page of the application accessible part of the stack is mprotected with PROT_NONE. This is in addition (and adjacent) to the red guard page placed by the thread library itself. On entry to every compiled Java method, a stacking banging instruction sequence loads at a fixed negative offset from the stack pointer. For example, on x64 the sequence is:

mov r11, [rsp - 0x3000]
If this address falls within the yellow guard page, an OS trap occurs. Control passes to the signal handler registered by the VM which will translate this as a stack overflow and throw a StackOverflowError. However, as a result of the aforementioned update, we were no longer seeing an OS trap when the yellow guard page was accessed on Linux. Something was changing the protection bits for the guard page. With a little help from gdb and strace, I discovered that when the VM loaded the libjava.so library from the JDK, dlopen triggered a call to a rather self-documenting function named __make_stacks_executable. What was this?! This function (source here) iterates over the list of current threads and calls mprotect on the entire stack of each thread with the flags PROT_READ | PROT_WRITE | PROT_EXEC. Of course, this removed the yellow guard page and hence the reason for the failure of stack overflow detection.

So, why did the linker decide to take this action? It turns out that it's all to do with the GNU_STACK ELF header, nicely described here and here. The Maxine launcher (maxvm) and shared library (libjvm.so) are compiled with the default stack-protection value for the GNU_STACK header which is RW (read/write). This means the thread library (pthreads) ensures all thread stacks are initially created with PROT_READ | PROT_WRITE protection flags. Now when the linker loads a shared library that does not have a GNU_STACK header (e.g. a legacy library compiled before gcc added this header by default) or has a GNU_STACK header whose value is RWX, then it calls (via a call-back registered by pthreads I think) the aforementioned __make_stacks_executable function. After this point, all currently active threads no longer have a yellow guard page on their stack. As I discovered, most of the shared libraries in the Linux JDK (build 1.6.0_20-b02) do not have a GNU_STACK header and hence indicate to the linker that they require executable stacks.

So, what's the solution for Maxine to handle this? A bit of experimentation revealed that if I added the default GNU_STACK header (with the execstack command) to all the shared libraries in the JDK, the Maxine VM stack overflow detection worked again. However, I'm not sure that this is safe in general as I don't know if HotSpot places executable code on the stack (much like gcc does to implement trampolines for local functions). The solution we've adopted for Maxine is to use the -z execstack option when linking the VM launcher. This basically makes the launcher equivalent to the JDK shared libraries with respect to controlling pthread's use of executable stacks (except that it's done with a non-default value of the GNU_STACK header, not its absence). I'd be curious to know what version of gcc is used to build the JDK for Linux as I'm not aware of a flag to omit this header in the current version of gcc. One (unfortunate) implication of this linker behaviour is that HotSpot (and Maxine) binaries & libraries must always specify executable stacks if they are to continue supporting loading of 3rd party libraries (part of JNI). Either that or invent another way to implement stack overflow detection that does not involve protecting pages on the stack.

Monday Aug 23, 2010

Yet another offline Java bytecode verifier

When directly manipulating or generating Java bytecode, it's not too hard to end up with bytecode that will not verify. The JVM will catch this code at runtime. However, the verifier in a JVM exists primarily to ensure secure execution of Java code, not to assist with bytecode-level programming. As a result, the verification error messages printed by a JVM are typically not so useful when trying to figure out why exactly your manipulated/generated bytecode does not verify. This is a well known problem and there already exists a number of standalone verifiers that provide very informative error messages. The ASM Java bytecode manipulation and analysis framework includes a builtin class verifier as does the BCEL library.

In the Maxine VM project, we use bytecode generation/manipulation as an alternative to writing assembler code. For example, we implement JNI stubs via generated bytecode (using a couple of bytecode extensions specific to Maxine). As such, we run into the usual issues of ensuring that the bytecode we generated was verifiable. Initially, we considered using one of the aforementioned libraries to address this. However, given that our goal is to develop a specification compliant JVM implementation we had to implement a verifier anyway. So, with the help of a sharp intern (thanks Dave!) we wrote one. Actually, we wrote two - one that performs type inferencing for class files with a version number less than 50 and one that does type checking for more recent class files (those compliant with the class file changes specified in JSR202). Now we have a verifier that not only passes the relevant JCK tests but is also a development aid whenever we do bytecode-level programming in the VM.

Useful functionality in the Maxine code base that can be made to work standalone is exposed by the max script. So, for anyone wanting yet another offline verifier, this script includes a verify sub-command. Here's how to get the usage message for this command:

~/maxine$ max help verify
max verify [options] patterns...

verifies a set methods using the Maxine bytecode verifier

    Run the Maxine verifier over a set of specified methods available
    on the class path. To extend the class path, use one of the global
    "-cp/p:" or "-cp/a:" options.

    See Patterns below for a description of the format expected for "patterns..."

    Use "max verify -help" to see what other options this command accepts.
    
    --- Patterns ---
    
    A pattern is a class name pattern followed by an optional method name...

Here's the output of using it to verify a negative-test (i.e. expected to cause a failure) from the JCK:

~/maxine$ max -cp/a:/Volumes/JCK-runtime-6/classes verify javasoft.sqe.tests.vm.classfmt.ins.instr_006.instr_00601m1t.instr_00601m1tn:m
Initializing verifier system...
Initialized verifier system
Finding specified methods...
Found 1 methods
Exception in thread "main" VerifyError: Missing stackmap frame for bytecode position 26 (branch target)
    while verifying javasoft.sqe.tests.vm.classfmt.ins.instr_006.instr_00601m1t.instr_00601m1tn.m() at bytecode position 1
        at com.sun.max.vm.classfile.ErrorContext.verifyError(ErrorContext.java:179)
        at com.sun.max.vm.verifier.MethodVerifier.verifyError(MethodVerifier.java:124)
        at com.sun.max.vm.verifier.TypeCheckingMethodVerifier.frameAt(TypeCheckingMethodVerifier.java:209)
        at com.sun.max.vm.verifier.TypeCheckingMethodVerifier.performBranch(TypeCheckingMethodVerifier.java:317)
        at com.sun.max.vm.verifier.TypeCheckingMethodVerifier$Interpreter.lookupswitch(TypeCheckingMethodVerifier.java:1579)
        at com.sun.max.vm.bytecode.BytecodeScanner.scanInstruction(BytecodeScanner.java:991)
        at com.sun.max.vm.bytecode.BytecodeScanner.scan(BytecodeScanner.java:1197)
        at com.sun.max.vm.verifier.TypeCheckingMethodVerifier.verifyBytecodes(TypeCheckingMethodVerifier.java:145)
        at com.sun.max.vm.verifier.TypeCheckingMethodVerifier.verify(TypeCheckingMethodVerifier.java:112)
        at com.sun.max.vm.verifier.TypeCheckingVerifier.verify(TypeCheckingVerifier.java:71)
        at test.com.sun.max.vm.verifier.CommandLineVerifier.main(CommandLineVerifier.java:106)

To see the abstract interpreter in action, use the -verbose option:

~/maxine$ max -cp/a:/Volumes/JCK-runtime-6/classes verify -verbose=3 javasoft.sqe.tests.vm.classfmt.ins.instr_006.instr_00601m1t.instr_00601m1tn:m
Initializing verifier system... 
Initialized verifier system
Finding specified methods...
Found 2 methods

Verifying javasoft.sqe.tests.vm.classfmt.ins.instr_006.instr_00601m1t.instr_00601m1tn.m()
Input bytecode:
Stack=1, Locals=1
0: iconst_1 | 4
1: lookupswitch default:20 1:26 | 171 0 0 0 0 0 19 0 0 0 1 0 0 0 1 0 0 0 25
20: return | 177
StackMapTable: number of entries = 1
  20: frame_type = 255 /\* full_frame \*/
    offset_delta = 20
    number_of_locals = 1
    locals = [ javasoft.sqe.tests.vm.classfmt.ins.instr_006.instr_00601m1t.instr_00601m1tn ]
    number_of_stack_items = 0
    stack = [  ]

StackMapTable frames:
0: local[0] = javasoft.sqe.tests.vm.classfmt.ins.instr_006.instr_00601m1t.instr_00601m1tn
20: local[0] = javasoft.sqe.tests.vm.classfmt.ins.instr_006.instr_00601m1t.instr_00601m1tn

Interpreting bytecode:
    local[0] = javasoft.sqe.tests.vm.classfmt.ins.instr_006.instr_00601m1t.instr_00601m1tn
0: iconst_1

    stack[0] = int
    local[0] = javasoft.sqe.tests.vm.classfmt.ins.instr_006.instr_00601m1t.instr_00601m1tn
1: lookupswitch

Exception in thread "main" VerifyError: Missing stackmap frame for bytecode position 26 (branch target)
    while verifying javasoft.sqe.tests.vm.classfmt.ins.instr_006.instr_00601m1t.instr_00601m1tn.m() at bytecode position 1
        at com.sun.max.vm.classfile.ErrorContext.verifyError(ErrorContext.java:179)
        at com.sun.max.vm.verifier.MethodVerifier.verifyError(MethodVerifier.java:122)
        at com.sun.max.vm.verifier.TypeCheckingMethodVerifier.frameAt(TypeCheckingMethodVerifier.java:209)
        at com.sun.max.vm.verifier.TypeCheckingMethodVerifier.performBranch(TypeCheckingMethodVerifier.java:317)
        at com.sun.max.vm.verifier.TypeCheckingMethodVerifier$Interpreter.lookupswitch(TypeCheckingMethodVerifier.java:1579)
        at com.sun.max.vm.bytecode.BytecodeScanner.scanInstruction(BytecodeScanner.java:991)
        at com.sun.max.vm.bytecode.BytecodeScanner.scan(BytecodeScanner.java:1197)
        at com.sun.max.vm.verifier.TypeCheckingMethodVerifier.verifyBytecodes(TypeCheckingMethodVerifier.java:145)
        at com.sun.max.vm.verifier.TypeCheckingMethodVerifier.verify(TypeCheckingMethodVerifier.java:112)
        at com.sun.max.vm.verifier.TypeCheckingVerifier.verify(TypeCheckingVerifier.java:71)
        at test.com.sun.max.vm.verifier.CommandLineVerifier.main(CommandLineVerifier.java:106)

The verify sub-command is available as of version 4278 in the main Maxine repository.

Tuesday Jul 20, 2010

Apple Keyboard with OpenSolaris

I use Macs most of the time so when at my OpenSolaris workstation, I'd like to use one of my spare Apple keyboards for a consistent experience. This almost works without any modification. However, the left Command/Apple key is not interpreted as "Alt" by OpenSolaris. Thankfully I found this thread which describes a workaround using xmodmap(1) and xev(1). The latter is a little X graphical app that shows you all X events sent to a window. Using this, I found that the keycode for the Command/Apple key is 115. Using xmodmap, I can now map this key to "Alt" with the following commands:

% xmodmap -e 'clear Mod4'
% xmodmap -e 'keycode 115 = Alt_L' 

To make this permanent, I simply put these two commands into a script (e.g. $HOME/bin/apple_keyboard.sh) and add it as a new startup program (System -> Preferences -> Sessions -> Add).

Wednesday Sep 16, 2009

Using task_for_pid on Mac OS X

Prior to the Mac OS X 10.5 (Tiger), it was completely legal for one process to modify another for the purpose of controlling its execution (single stepping, resuming, stopping etc) and inspecting/modifying its memory. In Tiger, this policy was modified so that only a process owned by root or with a primary effective group of procmod or procview has this privilege. In Leopard (Mac OS X 10.5), this policy was changed such that a debugger process now depends on the security framework to authorize use of the task_for_pid system service which gives a process the capability to control another process. The details are in the man page for the taskgated daemon. The default launch configuration for this daemon (in the file /System/Library/LaunchDaemons/com.apple.taskgated.plist) runs the daemon in the aforementioned Tiger mode.

The reason I mention all this is that the Maxine VM has a companion tool (called the Inspector) that is used for debugging a running instance of the VM. That is, the Inspector process needs the ability to control the VM process. Up to (and including) Leopard, the Inspector was granted this capability by means of a (somewhat insecure) workaround. Given that the Inspector is itself a Java program, one could simply modify the java executable used to run it. For example:

% sudo chgrp procmod /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java 
% sudo chmod g+s /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java 

It should be obvious of course, that this opens up potential security vulnerabilities that can be exploited by other Java programs run by the same executable. This was considered tolerable for those developers working on Inspector on a Mac. However, with the release of Snow Leopard (Mac OS X 10.6), this workaround was rendered ineffective. If one tries to run the inspector on Snow Leopard with the altered java executable, the result on the console is:

2009-09-16 11:14:23.307 java[1654:903] The application with bundle ID (null) is running setugid(), which is not allowed.

Not being a very knowledgeable Mac developer (nor wanting to invest the time to become one just yet!), I'm not exactly sure what this means. However, the outcome is that modifying the ownership and permission bits of the java executable is no longer possible on Snow Leopard. So, what is an Inspector user on Snow Leopard to do?! Unfortunately, the current solution is to force the Inspector to be launched as root via sudo.

However, the ideal solution is to use the Authorization Services on a Mac to dynamically obtain the privileges necessary for the Inspector to use task_for_pid. Unfortunately, use of this framework turns out not to be as straight forward as I thought it would (should!) be. Based on the sample code provided by Apple, I would have thought the following code is sufficient to acquire the privilege for calling task_for_pid:

#include "auth.h"
#include <Security/Authorization.h>
int acquireTaskportRight() {
    AuthorizationRef authorization;
    OSStatus status = AuthorizationCreate (NULL, kAuthorizationEmptyEnvironment, kAuthorizationFlagDefaults, &authorization);
    if (status != 0) {
        fprintf(stderr, "Error creating authorization reference\\n");
        return -1;
    }
    AuthorizationItem right = { "system.privilege.taskport", 0, 0 , 0 };
    AuthorizationItem items[] = { right };
    AuthorizationRights rights = { sizeof(items) / sizeof(items[0]), items };
    AuthorizationFlags flags = kAuthorizationFlagInteractionAllowed | kAuthorizationFlagExtendRights | kAuthorizationFlagPreAuthorize;

    status = AuthorizationCopyRights (authorization, &rights, kAuthorizationEmptyEnvironment, flags, NULL);
    if (status != 0) {
        fprintf(stderr, "Error authorizing current process with right to call task_for_pid\\n");
        return -1;
    }
    return 0;
}

When executed, this code results in the expected authentication dialog:

Authentication dialog for acquiring right to call task_for_pid

When I enter an administrator name and password, the dialog closes and the authorization appears to succeed. This suspicion is supported by the entry written to /var/log/secure.log:

Sep 16 11:11:28 isquawk com.apple.SecurityServer[21]: Succeeded authorizing right 'system.privilege.taskport' by client \\
    '/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java' for authorization created by \\
    '/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java'

However, a following call to task_for_pid returns an error code of 5 (i.e. a generic kernel failure). At this point, I'm at a loss as to what extra steps are required to convince the OS that I indeed have the permission to debug one of my own programs!

Sunday Jan 18, 2009

Opening multiple Eclipse instances on Mac OS X

While I'm generally only working within one Eclipse workspace, I occasionally have the need to open more than one Eclipse instance. One good example for this need is when debugging a regression. In this case, it can be quite useful to run side-by-side debug sessions; one on my current code, the other on an earlier version that I know does not contain the bug. To do so, I simply want to create a copy of the earlier version (using 'hg clone -r <good version number>' since I'm using Mercurial) and bring it up in a new instance of Eclipse. This is straight forward on most systems but on Mac OS X, launching an application by double clicking its icon simply brings a running instance of the app to the foreground. So, to achieve the desired effect on Mac, one needs to resort to the command line:

open -n /Applications/eclipse/Eclipse.app

This forces a new instance of the app to start. In theory, this should be sufficient. However, as I found out in my case, there is (at least) one other complication that my present itself. I have JDK 6 installed on my Mac and due to the way I've configured my PATH variable, it's the default version of Java launched from a terminal with an unqualified use of the java command. It appears as though this is how the Eclipse launcher starts its embedded Java process. The problem then is that Eclipse is a 32-bit app and JDK 6 on the Mac is 64-bit only and not surprisingly the result is a crash. The trick is therefore to modify one's path to ensure the JDK 5 java command is the default on the command line:

env PATH=/System/Library/Frameworks/JavaVM.framework/Versions/1.5/Commands:$PATH open -n /Applications/eclipse/Eclipse.app

I thought a better solution maybe to explicitly specify the JDK 5 java executable in /Applications/eclipse/Eclipse.app/Contents/MacOS/eclipse.ini with the following line:

-vm /System/Library/Frameworks/JavaVM.framework/Versions/1.5/Commands/java

but that doesn't work for some reason.


Wednesday Jun 04, 2008

VM documentation

Since joining Sun Labs in 2001, I've worked mostly on the design and implementation of language-level virtual machines. No prizes for guessing which language these VMs implement! Now that the second and third VM I've worked on have been released as open source, it's a good time to start a blog discussing the interesting parts of these VMs. That way, I can remember exactly what I was thinking when it comes time to debug/modify/discard the code. While most of these notes should be reflected in the javadoc associated with the code, I'm aware of some people's preference for reading webpages over reading javadoc. Maybe I should consider going straight to maintaining entries on Wikipedia - astute observers can then debug my VM design and implementation decisions with no more than a web browser ;-)

About

Doug Simon

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today