Friday Jan 29, 2016

Accessing Python objects from Nashorn JavaScript

Dynamic linker API for the Java platform (JEP 276) in JDK 9 Early Access defines dynamic linker API for the Java platform. Using this API, different language runtimes implemented for the Java platform can interoperate with each other.

Nashorn JavaScript engine has been a part of JDK platform since JDK 8. Jython is an implementation of the Python programming language designed to run on the Java platform. In this blog entry, I'll demonstrate seamless access of Python objects from a Nashorn script!

Steps:

My directory structure looks as follows:

With the above files in place, we just to have run the sample with this command:

jjs jython_linker.js

The output looks as follows:

With Dynalink, different language runtimes can interoperate with each other! Have fun with mixed language programming on the Java platform!

Monday Nov 30, 2015

Writing a pluggable dynalink linker and using it with Nashorn

JEP-276 (JEP 276: Dynamic Linking of Language-Defined Object Models) adds a facility for linking high-level operations on objects such as "read a property", "write a property", "invoke a callable object", etc., expressed as names in INVOKEDYNAMIC call sites. Nashorn JavaScript engine in OpenJDK uses dynalink for linking script and other objects.

JEP 276 allows user written ("pluggable") linkers to be loaded via service loader mechanism. Because of Dynalink, Nashorn also allows user-written dynalink linkers to be used along from nashorn scripts! You can write your own linker(s) and drop it as a "jar" in jjs -classpath. Your scripts can make use of those pluggable linkers to handle "special linkages".

If this sounds little abstract, you may want to look at the sample dynalink linker in nashorn OpenJDK repository - http://hg.openjdk.java.net/jdk9/dev/nashorn/rev/e9c4c02337cc. This sample adds a simple DOM Element linker DOMLinkerExporter.java. This linker allows scripts to access DOM child elements by child element tag name - rather than having to use DOM Java API to get child Node list and iterate to get the matching Element(s). dom_linker_gutenberg.js script uses "DOM linker". The script accesses child DOM elements by samply "_" followed by child element tag name.

You can imagine many other such useful dynalink linkers for your own Java classes or even JDK platform classes.How about a dynalink linker that supports easier access for SQL ResultSet objects - perhaps column names as properties? How about a dynalink linker that handles your domain object model complexity and exposes a simpler script access model -- without having to write any Java or script wrappers around it! All complexity is hidden in the linker code! No Java or script wrapper implies that easy script access does not result in reduced performance - because linking is just (mostly!) one time activity per callsite.

Friday Nov 27, 2015

Using Dynalink API with invokedynamic - a Java assembler example

Dynalink API JEP-276

Dynalink API (http://openjdk.java.net/jeps/276) provides a facility for linking high-level operations on objects such as "read a property", "write a property", "invoke a callable object", etc., expressed as names in INVOKEDYNAMIC call sites. Dynalink also provides a default linker for the usual semantics of these operations on plain Java objects (Java POJO "beans" linker), as well as a facility for installing your own language-specific linkers.

Dynalink has been used by Nashorn JavaScript engine internally starting from JDK 8. But, Dynalink is exposed as a public API starting from JDK 9. You can use dynalink to simplify invokedynamic generation when compiling your language for the Java platform.

To learn Dynalink API from Java code, you can check our dynalink API tests in nashorn repo. test/src/jdk/dynalink directory in Nashorn repo http://hg.openjdk.java.net/jdk9/dev/nashorn

There is a simple OpenJDK Java assembler based sample in nashorn repostory - $nashorn/samples/Main.asm - http://hg.openjdk.java.net/jdk9/dev/nashorn/file/376a63a077ee/samples/Main.asm. This example shows how "length" property can be linked using invokedynamic and dynalink. The Java beans linker that comes with Dynalink can link "length" property for Java arrays (array length) and Java Lists (List.size()). The example shows how "length" works for arrays and lists uniformly.

Tuesday Nov 10, 2015

Dynamic linker API for the Java platform (JEP 276)

JEP 276 defines dynamic linker API for Java. This JEP provides "facility for linking high-level operations on objects such as "read a property", "write a property", "invoke a callable object", etc., expressed as names in INVOKEDYNAMIC call sites. Provide a default linker for the usual semantics of these operations on plain Java objects, as well as a facility for installing language-specific linkers.

Nashorn JavaScript engine already uses "dynalink" library for linking properties, indexed access, calls on it's (script) objects as well as "foreign"/"host" Java objects (POJOs). With JEP-276, dynalink is exposed as a public (JDK specific) API in the java9 module named "jdk.dynalink".

Right now, the source code for JEP-276 lives in "jdk9 sandbox" OpenJDK repository (http://hg.openjdk.java.net/jdk9/sandbox) in the branched named "JEP-276-branch". This will eventually go into jdk9 repository. If you want to play with "dynalink" API, you can check out this forest and build "JEP-276-branch"

To play with any Java API, these days I use "jshell" (jshell) tool. The following is a jshell repl sample to demonstrate dynalink API from Java code.


import java.lang.invoke.*
// dynalink API lives in these packages
import jdk.dynalink.*
import jdk.dynalink.support.*

// dynamic 'operation' for a callsite. 'length' property
Operation op = new NamedOperation(StandardOperation.GET_PROPERTY, "length")

// method type of operation to be linked - length is 'int' value
MethodType mt = MethodType.methodType(int.class, Object.class)

// callsite descriptor
CallSiteDescriptor desc = new CallSiteDescriptor(MethodHandles.publicLookup(),
     op, mt)

// callsite
SimpleRelinkableCallSite cs = new SimpleRelinkableCallSite(desc)

// create a linker factory
DynamicLinkerFactory fac = new DynamicLinkerFactory()
// create dynalink linker
DynamicLinker linker = fac.createLinker()

// link the callsite
linker.link(cs)

// invoke it!
printf("array size %d\n", (int)cs.getTarget().invoke(new String[10]))

import java.util.ArrayList

// make a list and populate two elements
ArrayList<String> al = new ArrayList<>()
al.add("hello")
al.add("world")

// get 'length' of array list - which is nothing but size
printf("list size %d\n", (int)cs.getTarget().invoke(al))

The above repl prints "array size 10" and "list size 2" respectively. Note that "length" property is relinked automatically to be array length and then ArrayList size (when a different "this" object is passed). This linking (and relinking) of Java objects is handled by the "java beans linker" that comes with the dynalink implementation.

Wednesday Oct 28, 2015

Underscore is a keyword in Java 9, use this script to check your code!

Underscore ("_") is a keyword in Java 9. If you use "_" as an identifier, javac of JDK 8+ issues a warning. javac of JDK 9, issues an error! To check and migrate your Java code to avoid using "_" as an identifier, you can use the following Nashorn script with jjs tool. This script just parses each .java file and reports "_" variables with filename, line and column numbers. This script uses Javac Tool API and Compiler Tree API.

File: find_underscore.js
// Usage: jjs find_underscores.js -- <directory>

if (arguments.length == 0) {
    print("Usage: jjs find_underscores.js -- <directory>");
    exit(1);
}

// Java types used
var File = Java.type("java.io.File");
var Files = Java.type("java.nio.file.Files");
var StringArray = Java.type("java.lang.String[]");
var ToolProvider = Java.type("javax.tools.ToolProvider");
var Tree = Java.type("com.sun.source.tree.Tree");
var Trees = Java.type("com.sun.source.util.Trees");
var TreeScanner = Java.type("com.sun.source.util.TreeScanner");

function findUnderscores() {
    // get the system compiler tool
    var compiler = ToolProvider.systemJavaCompiler;
    // get standard file manager
    var fileMgr = compiler.getStandardFileManager(null, null, null);
    // Using Java.to convert script array (arguments) to a Java String[]
    var compUnits = fileMgr.getJavaFileObjects(Java.to(arguments, StringArray));
    // create a new compilation task
    var task = compiler.getTask(null, fileMgr, null, null, null, compUnits);
    var sourcePositions = Trees.instance(task).sourcePositions;
    // subclass SimpleTreeVisitor - to find underscore variable names
    var UnderscoreFinder = Java.extend(TreeScanner);

    var visitor = new UnderscoreFinder() {
        // override to capture information on current compilation unit
        visitCompilationUnit: function(compUnit, p) {
            this.compUnit = compUnit;
            this.lineMap = compUnit.lineMap;
            this.fileName = compUnit.sourceFile.name;

            return Java.super(visitor).visitCompilationUnit(compUnit, p);
        },

        // override to check variable name
        visitVariable: function(node, p) {
            if (node.name.toString() == "_") {
                var pos = sourcePositions.getStartPosition(this.compUnit, node);
                var line = this.lineMap.getLineNumber(pos);
                var col = this.lineMap.getColumnNumber(pos);
                print(node + " @ " + this.fileName + ":" + line + ":" + col);
            }

            return Java.super(visitor).visitVariable(node, p);
        }
    }

    for each (var cu in task.parse()) {
        cu.accept(visitor, null);
    }
}

// for each ".java" file in directory (recursively).
function main(dir) {
    var totalCount = 0;
    Files.walk(dir.toPath()).
      forEach(function(p) {
          var name = p.toFile().absolutePath;
          if (name.endsWith(".java")) {
              findUnderscores(p);
          }
      });
}

main(new File(arguments[0]));

Wednesday Sep 30, 2015

Playing with Java (java9) REPL - an example that uses nashorn engine in REPL

I love "Exploratory Programming" (https://en.wikipedia.org/wiki/Exploratory_programming) tools. If you've not already played with "Java REPL" project, you may want to do so! Check out Kulla project.

Clone kulla forest @ http://hg.openjdk.java.net/kulla and build using these commands:
  1. hg clone http://hg.openjdk.java.net/kulla/dev kulla-dev
  2. cd kulla-dev
  3. sh get_sources.sh
  4. sh configure --with-boot-jdk=/path/to/jdk1.8.0
  5. make clean images
  6. make install

Once build finishes, you can use the Java REPL tool "jshell" ( $kulla-dev/build//images/jdk/bin/jshell). I built on Windows. My jshell path is D:\src\kulla-dev\build\windows-x86_64-normal-server-release\images\jdk\bin\jshell.exe.

Now, two lines to print the squares of integers in the range [0, 100):


D:\src\kulla-dev\build\windows-x86_64-normal-server-release\images\jdk\bin>jshell
|  Welcome to JShell -- Version 1.9.0-internal
|  Type /help for help

-> import java.util.stream.*

-> IntStream.range(0, 100).map(x->x*x).forEach(System.out::println)

Slightly bigger repl sample that pulls weather data in JSON format, uses Nashorn script engine to parse JSON and print the statistics on it using Streams API.

File: weather.repl

import java.net.*
import java.io.*
import java.util.stream.*
import javax.script.*
import java.util.*

// URL to fetch JSON for weather data for Chennai, India
URL u = new URL(
  "http://api.openweathermap.org/data/2.5/forecast/daily?q=Chennai&mode=json&units=metric&cnt=7")

// read text from URL
String getText(URL u) {
    StringBuilder buf = new StringBuilder();
    try (BufferedReader reader =  new BufferedReader(
        new InputStreamReader(u.openStream()))) {
        reader.lines().forEach(l->buf.append(l));
    } catch (IOException exp) {
        exp.printStackTrace();
    }
    return buf.toString();
}

// create nashorn engine
ScriptEngine e = new ScriptEngineManager().getEngineByName("js")

// expose weather data JSON as global variable to nashorn engine
e.put("str", getText(u))

// massage JSON using nashorn and get the max. temp values
double[] values = (double[]) e.eval(
    "Java.to(JSON.parse(str).list.map(function(val) val.temp.max), Java.type('double[]'))")

// stat on max. temp values
printf(DoubleStream.of(values).summaryStatistics().toString())


You can evaluate the above source as follows:

jshell  weather.repl
|  Welcome to JShell -- Version 1.9.0-internal
|  Type /help for help
-> DoubleSummaryStatistics{count=7, sum=227.160000, min=30.210000, average=32.451429, max=36.5800
00}

Yep, Chennai is HOT! 32 C average! In Chennai, we have only three seasons - hot, hotter and the hottest :)

Wednesday Oct 21, 2009

Look Ma, javac tells me I am overriding static method wrongly!!


// File: SuperClass.java

public class SuperClass {
    public static int func() {
        return 0;
    }
}


// File: SubClass.java

public class SubClass extends SuperClass {
    public static boolean func() {
        return false;
    }
}

$ javac -fullversion
javac full version "1.6.0_15-b03-226"

javac SuperClass.java  SubClass.java 
SubClass.java:2: func() in SubClass cannot override func() in SuperClass; attempting to use incompatible return type
found   : boolean
required: int
    public static boolean func() { 
                          \^
1 error

The subclass uses a different return type for the same named method with same argument types. So, it is overloading SuperClass.func() and the overloading SubClass.func() differs only in return type. But, I am not sure of the error message....

Tuesday Jul 14, 2009

BTrace "unsafe" mode

We kept saying BTrace is a safe, dynamic tracing tool for Java. To ensure safety, BTrace does not permit many constructs of Java language and also allows calls only to BTraceUtils class.But, not everyone likes safety -- sometimes developers want "unsafe" facility ("I know what I am doing"-style). We have got comments of this nature about BTrace - for example here and here. We have added "unsafe" mode to BTrace - you can edit "btrace" script to change -Dcom.sun.btrace.unsafe=false to -Dcom.sun.btrace.unsafe=true. In "unsafe" mode BTrace permits all Java constructs and also allows calls to arbitrary code. Like every other unsafe facility in programming, you are on own your own with this facility! You may get weird errors while instrumenting target classes and/or running transformed code. But then you know what you are doing, right?

Monday Jul 06, 2009

BTrace project moved to kenai.com

We have recently moved BTrace project to kenai.com. If you are interested in BTrace, please continue to use it and help develop it from kenai.com.

Wednesday Dec 17, 2008

Debugging option for javac and javafxc

I work on JavaFX compiler these days. The command line (debugging) option that I often use is -doe ("dump on error"). This option prints stack trace of the compiler when error message is printed. NOTE: This is an internal option and can be removed any time without notice! But, it is useful for debugging. This option works for javac as well as javafxc. When I misspelled "class" as "clas" and run compiler with -doe option, I got the stack trace below:


$ javac -doe t.java
t.java:1: class, interface, or enum expected
clas t {}
\^
java.lang.RuntimeException
	at com.sun.tools.javac.util.Log.writeDiagnostic(Log.java:565)
	at com.sun.tools.javac.util.Log.report(Log.java:523)
	at com.sun.tools.javac.util.Log.error(Log.java:404)
	at com.sun.tools.javac.parser.Parser.reportSyntaxError(Parser.java:282)
	at com.sun.tools.javac.parser.Parser.syntaxError(Parser.java:267)
	at com.sun.tools.javac.parser.Parser.classOrInterfaceOrEnumDeclaration(Parser.java:2206)
	at com.sun.tools.javac.parser.Parser.typeDeclaration(Parser.java:2180)
	at com.sun.tools.javac.parser.Parser.compilationUnit(Parser.java:2126)
	at com.sun.tools.javac.main.JavaCompiler.parse(JavaCompiler.java:509)
	at com.sun.tools.javac.main.JavaCompiler.parse(JavaCompiler.java:550)
	at com.sun.tools.javac.main.JavaCompiler.parseFiles(JavaCompiler.java:801)
	at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:727)
	at com.sun.tools.javac.main.Main.compile(Main.java:353)
	at com.sun.tools.javac.main.Main.compile(Main.java:279)
	at com.sun.tools.javac.main.Main.compile(Main.java:270)
	at com.sun.tools.javac.Main.compile(Main.java:69)
	at com.sun.tools.javac.Main.main(Main.java:54)
1 error

Friday Aug 08, 2008

Scriptifying BTrace?

One of the issues reported with BTrace is that the trace authors have to write "verbose" code [some people say Java is "verbose"!]. In BTrace, we have to repeat the same set of imports, annotations in every btrace file and all methods have to be "public static void" and so on. Instead of inventing a new scripting language, I've added a simple C preprocessor like step in BTrace compilation. This preprocessor is based on the one in the GlueGen project. Thanks to Ken Russell for this code and for reviewing my changes specific to BTrace project. The preprocessor solution does not rule out a scripting solution in future :-) If you have nice ideas or would like contribute in this area, you are always welcome! But, I think preprocessor solution is simple and will be useful to some.


Simple Example:


btracedefs.h
ThreadBean.java

To run this sample, the following command can be used:

   btrace -I . <pid-of-the-traced-process> ThreadBean.java

Without the -I option in command line, BTrace skips the preprocessor step.

Friday Jul 25, 2008

Playing with JSqueak

Squeak is a open source implementation of Smalltalk. What is JSqueak? JSqueak is a Squeak interpreter written in Java. You can download JSqueak source code and play with it. I did the following:
  • Expanded the downloaded JSqueak.zip under a directory, say c:\\JSqueak.
  • cd c:\\JSqueak
  • javac -d . \*.java
  • copy mini.image.gz JSqueak
  • java JSqueak.Main
Even if you are not going to learn Smalltalk (why?), you can have the fun of reading Smalltalk VM implemented in Java. If you are lazy and don't want to compile, you can run directly by JNLP link from http://research.sun.com/projects/JSqueak/. Inside the Squeak environment, I wrote the legendary "Hello World" :-) This is how it looks...

Friday Jul 18, 2008

BTrace and JMX

You can dyanamically attach BTrace to a Java process to inject trace code into it. BTrace client classescollect the trace output via a socket -- these client classes are used by BTrace command line client as well as VisualVM plugin for BTrace. How about attaching a JMX client to collect BTrace's trace data? Yes, it is possible to access a BTrace class's static fields as attributes of a MBean with this RFE.

There are two MBean samples in the BTrace repository. I attached both BTrace samples to a "Java2D demo" process. And then I attached VisualVM to view the Mbean registered by these BTrace samples:

  1. ThreadCounterBean.java - this sample instruments java.lang.Thread.start() method to update a counter field. This counter field is accessible by JMX clients.



  2. HistogramBean.java - this sample collects histogram of java.awt.Component objects created by an application and exposes the histogram (map) as MBean attribute.


Thursday Jun 26, 2008

BTrace aggregations - contribution from community

If you have used DTrace, chances are that you have used aggregations. For performance issues, aggregated data is often more useful than individual data points. With BTrace, aggregating data is bit painful (you have to manage using Maps explicitly). It would be nice to have DTrace-style aggregation functions such as sum, max, min and so on. Glencross, Christian M (cited in my previous entry) has contributed code changes, doc and a sample for easy-to-use aggregation facility for BTrace. Please refer to the sample code (JdbcQueries.java) that demonstrates aggregations.

Now something unrelated to aggregations, but related to BTrace : I came to know about another use-case of BTrace. See also http://blog.igorminar.com/2008/06/btrace-dtrace-for-java.html

Monday Jun 16, 2008

BTrace in the real world

In the last few weeks, I came to know about two cases of real world use of BTrace.
  1. Glencross, Christian M (his blog?) wrote about attempting to write a script to track SQL statements executed by a Java application (private email). Thanks to him for permitting me to blog about his BTrace script. I've made few formatting changes to fit his code in this blog and added few explanatory comments (staring with "VERBOSE:").
    
    
    import static com.sun.btrace.BTraceUtils.\*;
    
    import java.sql.Statement;
    import java.util.Map;
    import java.util.concurrent.atomic.AtomicLong;
    
    import com.sun.btrace.\*;
    import com.sun.btrace.annotations.\*;
    
    /\*\*
     \* BTrace script to print timings for all executed JDBC statements on an event.
     \* <p>
     \* 
     \* @author Chris Glencross
     \*/
    @BTrace
    public class JdbcQueries {
    
        private static Map preparedStatementDescriptions = newWeakMap();
    
        private static Map statementDurations = newHashMap();
    
        // VERBOSE: @TLS makes the field "thread local" -- sort of like using java.lang.ThreadLocal
        @TLS
        private static String preparingStatement;
    
        @TLS
        private static long timeStampNanos;
    
        @TLS
        private static String executingStatement;
    
        /\*\*
         \* If "--stack" is passed on command line, print the Java stack trace of the JDBC statement.
         \*
         \* VERBOSE: Command line arguments to BTrace are accessed as $(N) where N is the command line arg position.
         \* 
         \* Otherwise we print the SQL.
         \*/
        private static boolean useStackTrace = $(2) != null && strcmp("--stack", $(2)) == 0;
    
        // The first couple of probes capture whenever prepared statement and callable statements are
        // instantiated, in order to let us track what SQL they contain.
    
        /\*\*
         \* Capture SQL used to create prepared statements.
         \*
         \* VERBOSE: +foo in clazz means foo and it's subtypes. Note the use of regular expression
         \* for method names. With that BTrace matches all methods starting with "prepare". The
         \* type "AnyType" matches any Java type.
         \* 
         \* @param args - the list of method parameters. args[1] is the SQL.
         \*/
        @OnMethod(clazz = "+java.sql.Connection", method = "/prepare.\*/")
        public static void onPrepare(AnyType[] args) {
            preparingStatement = useStackTrace ? jstackStr() : str(args[1]);
        }
    
        /\*\*
         \* Cache SQL associated with a prepared statement.
         \*
         \* VERBOSE: By default, @OnMethod matches method entry points. Modifying with @Location 
         \* annotation to match the method return points.
         \* 
         \* @param arg - the return value from the prepareXxx() method.
         \*/
        @OnMethod(clazz = "+java.sql.Connection", method = "/prepare.\*/", location = @Location(Kind.RETURN))
        public static void onPrepareReturn(AnyType arg) {
            if (preparingStatement != null) {
                print("P"); // Debug Prepared
                Statement preparedStatement = (Statement) arg;
                put(preparedStatementDescriptions, preparedStatement, preparingStatement);
                preparingStatement = null;
            }
        }
    
        // The next couple of probes intercept the execution of a statement. If it execute with no-args,
        // then it must be a prepared statement or callable statement. Get the SQL from the probes up above.
        // Otherwise the SQL is in the first argument.
    
        @OnMethod(clazz = "+java.sql.Statement", method = "/execute.\*/")
        public static void onExecute(AnyType[] args) {
            timeStampNanos = timeNanos();
            if (args.length == 1) {
                // No SQL argument; lookup the SQL from the prepared statement
                Statement currentStatement = (Statement) args[0]; // this
                executingStatement = get(preparedStatementDescriptions, currentStatement);
            } else {
                // Direct SQL in the first argument
                executingStatement = useStackTrace ? jstackStr() : str(args[1]);
            }
        }
    
        @OnMethod(clazz = "+java.sql.Statement", method = "/execute.\*/", location = @Location(Kind.RETURN))
        public static void onExecuteReturn() {
    
            if (executingStatement == null) {
                return;
            }
    
            print("X"); // Debug Executed
    
            long durationMicros = (timeNanos() - timeStampNanos) / 1000;
            AtomicLong ai = get(statementDurations, executingStatement);
            if (ai == null) {
                ai = newAtomicLong(durationMicros);
                put(statementDurations, executingStatement, ai);
            } else {
                addAndGet(ai, durationMicros);
            }
    
            executingStatement = null;
        }
    
        // VERBOSE: @OnEvent probe fires whenever BTrace client sends "event" command.
        // The command line BTrace client sends BTrace events when user pressed Ctrl-C 
        // (more precisely, on receiving SIGINT signal)
        @OnEvent
        public static void onEvent() {
            println("---------------------------------------------");
            printNumberMap("JDBC statement executions / microseconds:", statementDurations);
            println("---------------------------------------------");
        }
    
    }
    
    

    And he has expressed few wish lists for BTrace based on his experience with DTrace. We plan to investigate those items in near future.



  2. Binod P.G exchanged private e-mails about BTrace usage to track down a memory leak. Subsequently, he has blogged about the same.
About

sundararajan

Search

Archives
« May 2016
SunMonTueWedThuFriSat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
28
29
30
31
    
       
Today
Bookmarks
Links

No bookmarks in folder

Blogroll

No bookmarks in folder

News

No bookmarks in folder