Tuesday Jan 13, 2009

NetBeans Screenshot of the Week #41: Type Assertions

Python and Ruby are dynamically typed languages. We have a lot of heuristics in our editors to figure out the types - tracking assignments, analyzing your database migrations, etc - to help code completion, go to declaration and other IDE features work correctly. And Martin Krauskopf, who owns the Ruby editor now, has been improving the type inference for Ruby a lot lately.

However, there are cases where we just can't help. What if you're writing a brand new method you haven't called from anywhere, and you're trying to access methods on one of your parameters? We don't know the type of the parameter. In that case, NetBeans will provide "fallback code completion" - it will list ALL methods across ALL known classes in your project, libraries and current platform.

If you're looking for the documentation or parameters for a specific method - that's useful - after typing 2-3 characters you've typically narrowed the result set down to a handful of methods and you can pick the one you're looking for.

But what if you're not sure what the method is called? That exact scenario happened to me a couple of days ago. I was writing some Python code to do some string manipulation, and not being very familiar with Python I wanted to know what the equivalent of Java's indexOf method was in Python. I knew that the parameter in my method was a String, but I didn't have a way of telling the IDE that. I applied my usual workaround of adding a line above the current call:

x = ""

and then I could invoke x. to see the methods available for a String. However, that's definitely not a good way to do it. What if I forget to remove my fake assignment? (Yep, I did that too. I couldn't figure out why my code wasn't working until I debugged and realized I was clearing out the parameter with a bogus assignment... Doh!)

The "obvious" solution for this is supporting type assertions. We've had those for a long time in the Ruby editor
(picture), and obviously for JavaScript using the @type and @param {type} syntax. And the PHP editor just added support for type assertions as well (more here and here).

I've just added this for the Python editor as well. But I've gone one step further which I think makes the feature much more obvious, and easy to use. In code completion, if the editor detects that you are attempting to complete on a simple variable (as opposed to for example a complicated expression), and the type of that variable cannot be resolved, then we add a new special code completion item at the top of the list. Here's what it says (and a more detailed explanation in the documentation popup):

As soon as you press enter, the IDE will insert a special comment referencing the variable you were completing on, and automatically invoke code completion again to complete on the type statement:

The contents of this list has two parts: First, it lists the pseudo-types for many of the core builtin types, like ints, strings, lists and tuples. After that it will list all the known types in the system, such as the user and library classes. If you select one of these and try code completing the original expression again, you can now see that the type resolver knows what to do:

A couple of notes about this. First, note that the type assertions are just comments in your source. You can delete them - they are merely an aid to the IDE. Not only code completion uses these - go to declaration for example will also be more accurate if it is explicitly told the type of a variable it cannot figure out. You can also use multiple type assertions within a method in case the variable type changes during the method. This isn't the case with Ruby yet; the type assertions apply to parameters only, at the method level. But Martin has promised to look into this for Ruby as well so hopefully there will be full parity between the two editors shortly!

P.S. Just like in the PHP editor, mark occurrences and instant rename are type-assertion-aware:

Monday Jan 05, 2009

NetBeans Screenshot of the Week #40: Python Test Runner UI

In the 7.0 builds, we have a dedicated unit test runner for Python now. Running a file as a test, or running the test project action, will open a test runner with the output docked on its right hand side, instead of just the output window (click for full size screenshot):

Here's the test runner itself:

What you want to see in the output is a single green line which says "All n Tests Passed", where n is hopefully a large number. But when one or more of the tests fail, you see something like the above. Running the Next Error action (Cmd-.) will jump to the next failed test, just like it normally jumps to the next error in the current output window, or the next task in the tasklist.

One thing to notice (and this is new in 7.0) is that we now include the failure message right there in the test summary for each test, so you don't have to drill into an individual test to see the failure message. You can also hover over a test and any output from that test is shown in a tooltip. You can also right click on tests and select "Run Again" to run just one specific test over again.

This is the same window as the one we have for Ruby. In fact, Erno, who wrote the Ruby test runner, has modularized the code now such that it's the same implementation for both languages - we just have language specific plugins to actually hook into the various testing frameworks. Currently, for Python we support the two builtin testing frameworks: unittest and doctest. The above screenshot showed a regular unittest run. Here's a doctest:

One important thing to note is that you don't have to instrument your code with any test running code. As long as you place doctests in your docstrings, or as long as you have any classes extending unittest.TestCase, the test runner will find them. All you have to do is run Test Project:

and when you have done that, both unittests and doctests are found and included in the same test run - here's both failing doctests and unit tests:

Once again, kudos to Erno for writing the excellent test runner support! And as usual, let us know of any problems.

P.S. Jean-Yves Mengant has also done some excellent work recently. In addition to some cool new quickfixes in the Python editor for NetBeans, he has just integrated a multithreaded debugger for Python!

Tuesday Dec 16, 2008

NetBeans Screenshot of the Week #39: Ruby Code Coverage, Python Code Coverage

Here's a new feature for NetBeans 7.0: Code Coverage support!

First a screenshot:

There are a couple of key points about the code coverage support:

  • It is fully integrated into the IDE, and especially the editor. You don't get read-only views of your files with coverage information; the files themselves include coverage highlights. You can view and edit the code directly, use your normal IDE navigation commands to jump around, and so on.
  • Enabling and disabling code coverage is trivial. There is no need to modify your build files, run special rake targets etc. Just select the menu checkbox to enable coverage in the new Code Coverage project menu, and then continue as before - run unit tests, run files, run the tests in your project - and the results are updated live as soon as the executed process finishes.

I didn't think a screenshot really described this feature well, so I recorded a couple of quick sessions where I use the feature:

Here's a couple more images if you have problems loading the videos:

<script type="text/javascript">
digg_url = 'http://blogs.sun.com/tor/entry/netbeans_screenshot_of_the_week6';
<script src="http://digg.com/tools/diggthis.js" type="text/javascript"></script>

The Python version is already in the 7.0 builds. The Ruby support isn't integrated yet; it probably needs some tweaking to make it work on Windows.

Friday Nov 14, 2008

NetBeans Screenshot of the Week #36: Python Code Completion

Here are some screenshots of the code completion working for Python.
First, let's try it from an import statement:

Note that the documentation for the module is shown. Next, let's
try completing on imported symbols - again we've got the documentation:

I'm writing a unit test, extending TestCase. Now let's try overriding
some methods - completion after typing def:

Obviously, invoking methods on self. will show the inherited
methods - and their documentation:

Here's completion in my function (not on self) showing
the local variables, imported symbols and builtins. Notice that deprecated
methods are marked with strikethrough (and deprecated modules are in imports
as well, though I didn't show that above).

Notice also that the documentation here looks better - it has links,
colors, etc. This is because NetBeans supports the reStructuredText
Docstring Format (see PEP 287).
System libraries are being documented in this way now, so more of the
documentation will start looking like this.

When NetBeans finds embedded code, it will also syntax highlight the
code fragments in the documentation:

You can use this in your own documentation as well obviously.
Here's a file where I have restructured text. As I'm writing it,
I can -immediately- preview the text myself by just pressing
Ctrl-Shift-Space (Show Documentation - for those who don't know, you
can do this on regular code as well if you want to just get the documentation
for the code at the caret, rather than completion alternatives up to
the caret prefix.)

Here's a final example - completion after raise (or
exception) will show completion for all Error

I'll be quiet or about a week now -- I'm going on vacation! Not great
timing given the release date, but this has been planned since February...

Thursday Nov 13, 2008

NetBeans Screenshot of the Week #35: Support for Python's __all__ Variable

Let's start with a pretty simple module - an import, a function, and a couple of classes:

(Notice by the way the semantic highlighting here - the unused parameters and variables are marked with a slight grey underline,
method names in bold, parameters in orange. More on that another day.)

Now let's go to a different file and import this module. Code completion is context dependent and shows us
which symbols are being exported by the given module. We see not only our functions and classes, the imported symbol sys
is also being defined by module, meaning that if you do from compl4 import \* you would have sys defined as well:

Python has a way to deal with this: You can specify a variable named __all__, which lets you define exactly what the public API for
this module should be (more details).

Let's go in and define the names we want exported by our module. I will add __all__ = [ "Bar " ] somewhere in the file, to specify that only Bar should be exported:

Look at the file outline / navigator on the left hand side. As soon as I typed in the __all__ line,
the icons changed to show me that only the Bar class and its method are now "public"; there are little
lock icons on everything else! Let's see what code completion offers when we try to import this module again:

Yep, only the public API is exported now. Let's append to the __all__ list to export the func function:

You don't have to rely on icons to tell you if something is wrong though. Let's say I've forgotten all about this, and I decided
to rename my Bar class:

NetBeans checks that all the symbols you declare in your __all list actually exist, and points out any discrepancies. Here, the public Bar class no longer exists since we renamed it.

Here's another little bug I came across related to this. I was checking the __all__ handling on various library files,
in this case posixpath.py. Notice how in the navigator, all the functions are public, except one...

Sure enough, the code looks like it's registering this conditionally added method... Do you spot the problem? There's a dot missing!
As soon as we add it, we get this:

Of course, this last bug was already detected by the
unresolved symbol detector...

Tuesday Nov 11, 2008

NetBeans Screenshot of the Week #33: Finding Unresolved Symbols

In my last entry I showed that the Fix Imports action will alert you to an unresolved symbol. This is really helpful since it can help you track down runtime problems.

Obviously, you shouldn't have to go run Fix Imports on every file, periodically, to look for unresolved symbols!

NetBeans has a special semantic checker for this. It is off by default, for two reasons:

  1. I added it just a couple of days ago - during the stabilization period for EA, so I didn't dare enable it in case it's seriously broken.
  2. There are cases where it doesn't yet work. I'll describe that later.

To enable it, open up the Editor options dialog, and under Hints, select Python. You should now see the current batch of Python hints. Enable the "Unresolved Symbol" hint:

Now let's open up the datetime.py file again:

As you can see, the file is marked erroneous (by the red square above the scrollbar), and the scrollbar indicates where in the file the error is. Clicking on it to expose the error location you can see an underline under the unresolved symbol, and a tooltip explaining what the problem is. The error is also shown in the tasklist, so you don't have to open each file to look for problems - just let the tasklist scan your project for problems.

Here's another example. I opened up the Django 1.0 sources, and NetBeans pointed out this problem to me, in django.utils.simplejson.tool:

This shows that the import is unused, and that the attempts to call methods on the import do not work. The call would need to be changed to

obj = django.utils.simplejson.load(infile)

or the import changed to be

import django.utils.simplejson as simplejson

Here's one final one... Perhaps helpful to people having done a lot of Java development recently:

Finding unresolved symbols means that NetBeans must be able to find all the libraries you are importing. This means you can't just point NetBeans at a plain Python file - your project has to contain all the libraries you are using, and, you have to configure your source path correctly such that NetBeans computes the correct package names for your python modules. (I accidentally pointed to the "django" folder instead of its parent folder at first, which meant django. wasn't part of the package prefix and that meant that all kinds of symbols couldn't be resolved.)

Second, this won't work for dynamically generated symbols (eval etc). I haven't run into this a lot yet - but I'm sure there are areas where the unresolved detection can't find symbols that really do exist.

P.S. Is there a canonical Python term for what I've called "unresolved" here that we should use instead?

Monday Nov 10, 2008

NetBeans Screenshot of the Week #32: Python Import Management

We're about to release an early access release of Python support for NetBeans. We have ambitious goals for the Python support: NetBeans should be the best IDE for Python development, bar none!

Over the next couple of weeks I'll post screenshots of a bunch of editor related features for Python. Today I'll start with a feature near and dear to my heart: Import Management.

I used to be a die hard XEmacs user. When I started coding in Java, I continued using emacs for a couple of years. And the feature which finally made me abandon emacs for a Java IDE? You guessed it -- import management. If I wanted to use for example the List class, I could just press a keystroke to choose which List class I had in mind, and the import statement was added for it at the top of the file while I could continue working in the middle of the file somewhere.

To be sure, this is not rocket science, and Emacs support for Java has improved a lot over the years. But at the time, this was a huge productivity boost and coupled with other code-intelligence and navigation features was enough to lure my away from my comfort zone in the IDE to what was a less powerful plain text editor, but a more powerful Java coding tool.

Anyway, Python shares some similarities with Java. There are lots of libraries, and you need to import modules before you can use them. And this is where NetBeans' Python support can help.

Let's say I've tried calling the cmp method to compare files. I ran the code and realized I need to import cmp before I can use it. To get the IDE to import the symbol for me, I can just pull up the context menu and select Fast Import (or more realistically, just use the keybinding Alt-Shift-I):

This brings up an import dialog like this:

This tells us there are two definitions of a function named cmp - one in filecmp and one in functions. I'm interested in the first one. The label below the list tells me that I can hold the Alt key or the Shift key when I select the list entry to change the import statement. And below that, there is a preview of the import statement that will be generate. As soon as I press the Alt key, the window changes to this:

Here, the bottom line is showing that we will be doing a wildcard import of all symbols instead of just the target symbol. And finally, let's press Shift:

This will not just import the cmp symbol, it will instead import the whole module and change my current call to use the imported module symbol. When I press Enter my editor looks like this:

This is the import facility I ditched Emacs for ten years ago. But it's not how I manage imports for Java anymore. Instead, there is the Fix Imports action (it's also there in the menu -- Ctrl-Shift-I for short). Unlike Fast Import, which looks at the identifier under the editor caret, Fix Imports will look at all your imports in the file, and, well, fix them! This means sorting them, removing duplicates, even removing unused imports. And obviously, resolving imports automatically using the machinery I showed above. Here's how this works.

Here I've opened the SocketServer.py file, and then I've commented out all the imports:

The file is now broken - it has various references to functions that are not defined/imported anymore. Let's see what happens if I press Ctrl-Shift-I:

Tadaaa! It has figured out the missing imports and has inserted them. This is confirmed by a message in the lower left hand corner:

This just happened automatically because NetBeans could figure out, unambiguously, what the missing imports must be. Let's make it less clear-cut. I'm going to insert a call to get(). Now let's try again:

This time we get a dialog. You can see that it has figured out the sys and os imports - but it is not certain to do about the call to get(). There are two modules defining this function - unicodedata and webbrowser:

The corresponding function signatures are listed to help me decide which one I'm interested in. I can choose one of them, and when I press Ok, the imports are applied.

In some cases it may not find the symbol at all. Let's put in a bogus call:

Here the import manager can't figure out what to do - there is no known matching function. Only when NetBeans is certain about the imports does it proceed to clean up the imports without any user intervention.

This facility can also help find bugs! When I was testing this feature, I happened to invoke it in the datetime.py file which ships with Jython. Here's what it said:

That's right - the code is referencing nonexistent class ValuError. This clearly should be ValueError!
NetBeans has additional facilities for tracking down these types of bugs which I'll cover in another post.

So far I've focused on using the import facility to import new symbols. If you're writing code using the editor you may not need it, since when you use code completion to insert calls, it also automatically inserts import statements as needed.
However, another useful aspect of Fix Imports is cleaning up your imports: removing duplicates, sorting alphabetically (configurable), and even deleting unused imports.

If you open up a file with unused imports, NetBeans will show you that immediately:

There are quickfixes here, so if you click on the lightbulbs or press Alt-Enter, you can see the import actions:

The integration with quickfixes means that if you look in the tasklist, it will show files containing unused imports. But when you're editing, you can just press Ctrl-Shift-I to clean up the imports - it will do everything we've covered so far: Add missing imports, remove duplicates, remove unused, and sort the remaining imports.

Import management is also tied into source formatting. If you format your source, you can optionally have it clean up formats. Here are the formatting options for Python:

You can play with the checkboxes and the preview will show you the effect. Here's what happens if we enable sorting system libraries to the top. Note also that removing unused imports is not done by default during formatting (whereas Fix Imports will perform it automatically). You can change that in this formatting preference panel:

As I mentioned, we'll have an early access version out very soon. If you're feeling adventurous, you can always download builds from the continuous build server - http://deadlock.netbeans.org/hudson/job/python/. Please file bugs and ideas here.

Saturday Jul 26, 2008

Screenshot of the Week #31: Python, Groovy

In my language support blog entry last week I talked briefly about the PHP, Python and Scala work going into 6.5. That's not an exclusive list. We're also adding Groovy support, and it's making great progress. Martin Adamek is posting status updates on his blog; here's Groovy instant rename in action:

The Python work is making good progress too. Here's code completion now for methods and classes, with integrated documentation support:

Here's another screenshot: Python quickfixes:

Anyway, I'm going on "vacation" now for a few weeks. I put that in quotes because I'm moving, and honestly, packing, cleaning and moving is hard work. Coding, now that's fun. I can't wait for my real vacation to start when I get back to work.

Tuesday Jul 15, 2008

Multilingual NetBeans: PHP, Python, Scala

I've mostly been writing about Ruby and JavaScript on this blog, but NetBeans is getting strong language support in other languages as well!

First, take a look at the PHP support. Petr Pisl, Tomas Slota, Radek Matous, Tomas Mysik and others have been posting screenshots and
feature descriptions regularly on their team NetBeans PHP blog. Check it out - code completion,
quick fixes, semantic highlighting - we're going to have a fantastic PHP IDE! Here's a screenshot - look at the unused method detection for example:

The Python support is starting to shape up too - here's a screenshot of code folding, semantic highlighting and instant rename refactoring.

Coming soon - code completion and other editing goodies.

And I just now noticed a post from Caoyuan Deng
that his Scala support for NetBeans is now built on top of the native scala compiler. His editing support is already feature packed (take a look at his blog for some of the past entries) and now that he has 100% language parity things are looking very very good!

Last but not least: InfoWorld has reviewed 9 Ruby IDEs,
and NetBeans came out on top!


Tor Norbye


« June 2016