Ruby Screenshot of the Week #8: Live Code Templates and TextMate Snippets

I've received several requests for support for TextMate-style snippets. TextMate is a very popular editor among Ruby developers, and a lot of its power comes from its code snippet functionality. Snippets are basically identical to what NetBeans calls "Live Code Templates". The code templates are live in the sense that when you insert the template, you're in a mode where you can jump from one logical section of the template to another, and edits in one section can automatically be updated in other sections.

I've just implemented an import mechanism for TextMate bundles. Point at your own bundles (or even the ones distributed with TextMate, although I don't know what the legal issues are with that), and choose import. This will convert all\* the snippets into NetBeans Live Code Templates (\*=for some values of all).

There are two features in TextMate snippets that are not supported:

  • Command execution. A snippet can contain a `shell command` (in backquotes), and at template evaluation time, the command will be executed and the result inserted. In the snippet bundles I looked at, this was used only for a single thing: inserting parentheses on a conditional basis. So I just made the import handle parentheses directly instead.
  • Regular expression replacement. A template can perform regular expression replacement on different instances of the parameters. I can't even tell exactly how this is supposed to work, but for now, I skip snippets that do this.

TextMate is essentially a text editor; it doesn't (as far as I can tell, having never used it but having read their snippets and macros in detail) have any real semantic information about the code. Therefore, doing complicated text processing with regular expressions is the Way of TextMate to achieve logical code operations. Live Code Templates on the other hand can take advantage of all the information the IDE has about the code. I've added a bunch of logical code template variables you can use in templates. Rather than try to reproduce the regular expression operations of TextMate's snippets engine, I would like to create logical Ruby operations for the code templates and use those instead.

Here are some examples of what is possible with this mechanism. None of the imported snippets use these variables, but the imported snippets are useful in their own right - especially for those of you with muscle memory.

${unusedlocal defaults="i,j,k,x,y,foo,bar"}
This will evaluate to an unused local variable in the current scope. It will try to use the provided suggested names, in order. It checks whether each one is already in scope, and if so, tries the next candidate. (If none succeed, it starts appending numbers to find a unique name. If there are more Rubyish ways of generating unique variables, please let me know.)

First, we create the template, naming it "dob":

Then, in some method I want a do-block. I just type "dob" and hit Tab, and voila:

It inserts a do-end block. When it comes to the block variable, it has looked at the context and discovered that the first two preferred alternatives, i and j, are both taken - so it chooses k. This shows the advantage of using semantically aware code templates - you don't accidentally alias an existing variable. This wouldn't be a problem in Java, where it would complain that your new variable already exists in scope, but in Ruby, this is valid (and sometimes, but not usually, what you want.)

As another example, here's a silly template which spits out a hash that defines a bunch of local context:

Let's say I have the following silly file, and type "ctx":

If I now hit tab, I get this:

I will now be able to walk through (with the tab key) each of the variable sections and edit them. For an example like this, it's likely that we don't want these things to be edited by the user - so just add editable=false on each such variable (e.g. ${class editable=false}) and they will be processed but not edited by the user.

One of the things that makes this really powerful, is that it uses a lot of the context information the IDE has about your program. In this example, computing the super class of the current class isn't that hard - and a fancy Ruby shell script executed might be able to pull it off. But if I was simply redefining an existing class, let's say Integer, and there is no local reference to Integer's parent class, the live code template will spit out Numeric. Yes, it looks in the index.

Call for Help

As I've shown, the basic machinery is there now, but what we really need is a set of code templates to be bundled with the IDE. Some will be simple substitutions or linked variables, but we should be able to come up with some much stronger ones. Please let me know what kinds of logical parameters you can take advantage of. For example, perhaps a parameter which evaluates to the require statements necessary for the classes referenced in the file? Or a parameter which looks for database table names in the database? Perhaps some simple transformations which produce a constant name or a method name (upcasing and downcasing) from some other symbol? There are a lot of possibilities here, but since I'm not a real Ruby programmer I'm not the best person to dream this stuff up. If you can look at snippets you have written, import them into NetBeans, optionally edit them to add more semantic analysis if appropriate, and contribute them, I'd be very thankful!

Also, I have never used TextMate, so I would appreciate feedback on whether the templates work the way they do in TextMate, or if I misunderstood something from just reading the snippets themselves. Bug reports (preferably via the issue tracker) are much appreciated.

So, where can you get this stuff? I just integrated everything, but the Live Code Template management is not in stock NetBeans; it's in an optional download called "CodeTemplateTools" (in contrib/CodeTemplateTools), written by Sandip Chitale. More info about this module is here. I have built an NBM you can install directly - just download the module bundle file, then install it via the IDE via the Update Center (in the first panel choose manually downloaded NBM). Note however that you must have the other IDE fixes as well; they should be live on deadlock.nbextras.org soon but probably won't hit the Update Center for a couple of days. (I'll try to make the slim IDE bundle the tools). As always, see the wiki for installation instructions. Once you have installed the Code template tools, look for "Show Templates..." in the Edit menu. The import UI is still a bit rough - no progress bar etc - I'll worry about that later. There's one more rough spot: the "|" character must be escaped (by typing it twice) because historically, NetBeans used "|" to indicate the cursor position. There is the logical variable ${cursor} for that now, but there's still some issues we need to resolve in this area.

Comments:

Top man! Cheers!

Posted by Si on April 05, 2007 at 05:26 PM PDT #

Thanks Tor
Looking forward to trying this out! Can you confirm where you would like us to put comments noting there is your blog, netbeans ruby wiki, netbeans mailing list, netbeans bug tracker (I've not use this before).
Which of these areas would you prefer to receive:
a) feeback on your questions from the blog
b) bugs we find in the latest release drops
c) ideas for improvements re ruby/rails support
Cheers Greg

Posted by GregH on April 05, 2007 at 10:11 PM PDT #

Very Cool!! Looking forward to playing with it...

Posted by Brian Ehmann on April 05, 2007 at 11:15 PM PDT #

Hi Greg, this issue has come up a couple of times so I've created a wiki page for it: http://wiki.netbeans.org/wiki/view/RubyFeedback.

On top of what it says, for the TextMates functionality in particular I'm really curious how the editing part works compared to TextMate - having never tried TextMate I don't know, but I want to make sure that the muscle memory (and productivity) is the same.

Posted by Tor Norbye on April 06, 2007 at 12:28 AM PDT #

Hey Tor, I am a Emacs user and did try NBM and its ruby/rails support. What turned me off was:
  • No easy way to switch between contextual files. I have never used TextMate, but in emacs rinari mode, i can switch between an action and its corresponding view by just pressing C-c,C-v. Similary there are keyboard shortcuts to directly jump to public,config,log,controllers,migrations directories. With NBM, i was stuck with a mouse, which is not so fast. Also, with zenspider's toggle.el I can switch between a ruby file and its testcase file pretty quickly, by pressing C-c,u.
  • I found documentation a bit lacking, or rather plenty i should say, but hidden. emacs-rails mode, comes with a plain HTML file that lists all damn shortcuts. As a new netbeans, user i had to go and look into preferences for getting the keyboard shortcuts.
  • Last time i checked, there was no Javascript syntax highlighting in NBM. CSS mode wasn't good either. It certainly needs some work.
  • Do you guys have got Emacs kinda macro support? i mean, editing macro while replaying it?

Posted by Hemant Kumar on April 06, 2007 at 06:16 PM PDT #

Ok looks like C-S-v can do the trick of switching between views and actions, sorry for this bit. but as you understand, it would need some more tuning, because Emacs/Textmate support a lot more shortcuts for contextual switching.
for example, in emacs:
render :partial => "foobar", :layout => true
If i place cursor at =>, after :partial and press C-Enter, then it would take me to the partial.

Posted by Hemant Kumar on April 06, 2007 at 06:34 PM PDT #

I'm having trouble trying to import the textmate rails & ruby bundles I checked out (using svn).
I go Edit/Show Templates/Import, but then Netbeans is looking for a "Abbreviations" file, for which I can't find a file in the checked out Textmate bundle files that match?
Can anyone see where I'm going wrong?
Cheers Greg

Posted by GregH on April 06, 2007 at 09:03 PM PDT #

Amazing feature list, Tor!

Could you please give some hints on the basics, though: when is it advisable to run JRuby and when ordinary ruby binary is better?

At my first attempt, I configured Netbeans to use c:\\InstantRails\\ruby\\bin\\ruby, and then it started to reindex not just my app, but also all installed gems and Rails and it probably took 2 or 3 hours on my Centrino 1.7 laptop. It finished at last, but when I restarted Netbeans, the indexing seemed to begin all over the same gem directories again.

With JRuby, reindexing behaves better, but the difficulty is having to install second copy of all gems and plugins that already sit in InstantRails in order to work with the existing application. Another disadvantage would be the script execution performance. What is the current performance loss with JRuby at running, say, unit and functional tests?

One unrelated question - with some \*.rb files, the "Run" menu item is grayed out. Why is that? The same files run perfectly fine from console. For example, I cannot run script\\about within Netbeans, and neither my custom-made maintenance scripts. On the contrary, the test cases have that "Run" menu item, and the process running progress bar appears for a while, but the output remains empty and the logs are not written to, so probably something does not go right.

Finally, one feature that I miss from Eclipse is being able to see what files in the file tree are covered by version control and what files are not - it helps a lot when committing a subset of modified files.

  • Features/Ruby 0.58.0
  • Netbeans 200703291800 on Java 1.6.0_01

Regards
--
Yar

Posted by Yar Dmitriev on April 07, 2007 at 01:06 AM PDT #

Hemant, thank you for your feedback, and especially your comments regarding keyboard navigation. I will be adding a shortcut to jump to tests soon - the toggle.el reference was helpful. The JavaScript support is being beefed up for 6.0, but that isn't bundled in the builds yet; I will enable that very soon. And yes, there is macro support (see Macro commands in the edit menu) but I haven't used them. FYI - if you're an emacs guy you'll probably want the incremental search toolbar; thats a separate module not included (but I think there are plans to integrate the functionality into the core.

Greg, have you tried selecting the -directory- named ".tmbundle" (e.g. Ruby.tmbundle) and importing that? I know, I should make a separate filter for it and use icons in the file chooser on these directories to make it more obvious.

Yar, perfect timing! You have version 0.58; in version 0.59 indxing should be -immediate- for native ruby! (I actually integrated this into version 0.58 and it worked on my mac, but I just noticed there was a problem on Linux that I made a quick fix for.). There was another problem related to symlinks on Linux which made it index all of Rails twice; thats also gone now.

In general, I'd say JRuby is for when you want to write Ruby applications that access Java classes, or for very easy out of the box configuration (for example, the system supplied Ruby on Mac's doesnt work well and it's nontrivial to build your own - most systems don't have cc installed etc). In the future, JRuby may also be very fast; they're making a lot of strides in this area.

I'm not sure what the deal is with grayed out Run files. If you could mail one to me (along with a description of where in the project the file sits) I can take a look (tor.norbye@sun.com).

I'll pass the version control comment on to the version control group. But in NetBeans you get a lot of help in this area at checkin time (try it - it will also include new files not included by version control as checkin suggestions - within reason (E.g. it skips files named \*~, .backup, .class, etc.). I'm using the NetBeans CVS support daily and have never needed the icon badging that I believe we had in the old days for NetBeans; the new CVS support changed the way this works. On the other hand, if you really like a different workflow we should try to support it.

Posted by Tor Norbye on April 07, 2007 at 02:54 AM PDT #

One more thing regarding indexing speed: Maybe I spoke to soon. On my year-old mac, indexing takes 6 minutes. On my three years old Ultra 20 running Linux, it takes 8 minutes. I've never seen it take 3 hours, and somebody earlier reported 8 (eight!) hours. So there might be something going on in some Ruby distributions that I'm not handling properly - or perhaps there are some really large gems I don't have stalled (large either in terms of number of files or lines of code). If after this fix you still find indexing to be slow, can you tell me what it seems to be doing? (The status bar should at least list the root of what it's indexing - that should tell me which gem it's in etc.).

Posted by Tor Norbye on April 07, 2007 at 03:15 AM PDT #

On how to write additional intelligent code template parameters see this blog entry:

http://blogs.sun.com/scblog/entry/define_your_own_ code_template

Posted by Sandip on April 07, 2007 at 04:03 AM PDT #

Great, 0.60 improved the indexing speed in my setup (with native ruby) - it was finished within minutes. However, after restarting the application some of the indexing starts again. For most gem dirs it happens quickly and says "scanning". But my project dirs, such as "helpers", "vendors" are indexed again and again. Maybe I should drop the config line:

netbeans_extraclusters="c:/Program Files/netbeans/my_clusters/ruby1"

and just put ruby1 into C:\\Documents and Settings\\yar\\.netbeans\\ (is that correct?).

BTW, for clarity, I delete C:\\Documents and Settings\\yar\\.netbeans\\ and C:\\Documents and Settings\\yar\\.netbeans-derby\\ every time I replace ruby1 with new downloaded copy.

As for not-runnable files, here is a screenshot.

Posted by Yar Dmitriev on April 07, 2007 at 05:27 PM PDT #

I've checked in a "Go To Test" action (bound to ctrl-shift-e or make ctrl-alt-e). It's in the Goto context menu in the editor. It will first try to find the matching file based on file path and name patterns (same as toggle.el). If that fails, it will use parse information and the code index; if the current file ends with "Test" it will strip it off and look for a class of that name, otherwise is will append Test and look for that. Are there other name patterns common in Ruby? I also remember a Test::Unit convention of using files named tc_\* so I might add that. Let me know whether it works. The keybinding is bad at the moment; it's using a global shortcut rather than a mimetype specific one. I've gotta work something out with the JUnit test guys since they also have a Goto Test action. (This is in version 0.61).

Regarding code indexing: It's true that it will index the -user- directories over again, at every startup (well, it should be looking at file timestamps and comparing to the index time stamps). It doesn't do this for the libraries and gems. The assumption is that user-files may have been edited by other tools or editors in between editor sessions so it does a check. I was assuming this check would be pretty quick, and that IDE startup is not that frequent so avoiding an out-of-date index was more important. Perhaps I should introduce a flag to suppress this behavior? Note that this was much worse until two days ago, because it turned out that on some Linux systems, the Rails command created symlinks in the user directory (in rails' vendor/ directory) back into the Rails libraries, so the indexer was happily rescanning all of Rails every startup. Ouch. That should be fixed now.

Regarding some files being disabled: From the screenshot it looks like the issue is that these are files which do not correspond to a "source root" (the directories in the logical/projects view show the source roots). I should still be able to execute these things, so I'll look into that.

Posted by Tor Norbye on April 08, 2007 at 01:19 PM PDT #

It's quite cool now ! But I really want to know how to set the rails's encoding in Netbeans? For instance ,if I want to use UTF-8 as my default encoding,how to do that??

Posted by Aaron on April 08, 2007 at 02:10 PM PDT #

Regarding UTF-8 encoding, does this help? http://ditoinfo.wordpress.com/2007/02/26/netbeans-and-utf8-encoding-2/ ?

Posted by Tor Norbye on April 09, 2007 at 12:49 AM PDT #

Could you please place a link for up to date builds on the side for handy reference?

Posted by Ralph on April 09, 2007 at 06:57 AM PDT #

I subscribed to nbusers mailing list, but wasn't sure, if i should ask ruby mode specific questions.
  • How do i update netbeans itself, when i have done a binary install in Linux. I am currently using ML-8.
  • Just updating the cluster would update ruby/rails only features, right?
  • I am getting a really nasty exception, when using Control-Space, both with Jruby and native Ruby. Where shall i file the bug report.
  • There is no support for manual edit while replaying a macro in NBM. In emacs, i can pause a macro while replaying/recording and do some recursive/manual edit and resume the macro. Its very powerful feature, but most IDEs I have came across ignore it.
Thanks for the work on toggle.el.

Posted by Hemant Kumar on April 09, 2007 at 07:41 AM PDT #

Ok I got the link to issue tracker, sorry for that.

Posted by Hemant Kumar on April 09, 2007 at 07:43 AM PDT #

Just now I tried the Netbeans M8. In this version, the Java-Project's encoding could be setted in its project properties, while Rails project could not. And I have tried the method in "http://ditoinfo.wordpress.com/2007/02/26/netbeans-and-utf8-encoding-2/", but it doesn't work well in my system. I used simplified chinese Windows-XP, JDK6.

Posted by Aaron on April 09, 2007 at 11:48 AM PDT #

In case anyone is interested I have uploaded the Code Template Tools module on my update center. Read all about it here:

http://blogs.sun.com/scblog/entry/netbeans_module_update_center

The Code Template Tools module also has online help that can be accessed using Help:Content.

Posted by Sandip on April 09, 2007 at 04:18 PM PDT #

Hudson build seems to be failing, regardless of success status. Certainly no artifacts are being produced: BUILD FAILED /hudson/workdir/jobs/ruby/workspace/scripting/ruby/build.xml:26: The following error occurred while executing this line: /hudson/workdir/jobs/ruby/workspace/nbbuild/templates/projectized.xml:53: Cannot compile against a module: /hudson/workdir/jobs/ruby/workspace/nbbuild/netbeans/ruby1/modules/org-netbeans-api-gsf.jar because of dependency: org.netbeans.api.gsf/1 > 0.20.0

Posted by Si on April 09, 2007 at 08:25 PM PDT #

Hi Si, yes - there was a bug in the build script which meant I didn't notice the build was broken! It was missing a file from the checkin. I have checked those files in AND fixed the build script so this won't happen again. As of build #857 everything should be okay again.

It's probably a good idea to grab the updated bits; I've fixed a bunch of issues. First, I believe I have fixed the "Run File" command being disabled that Yar reported. Second, there were some corrupted data in the native ruby code indices, which caused some ugly problems with code completion (this is probably Hemant's issue), third, I've made the Go To Test action work a bit better (in addition to fixing a problem with ZenTest, I noticed that Ruby tests often use "Test" as a prefix rather than a suffix), and a few other things.

Aaron, I have an open task to add a encoding property to the Ruby project types the way the Java project types recently did.

There's some information on installation and updating here: http://wiki.netbeans.org/wiki/view/RubyInstallation.

Posted by Tor Norbye on April 09, 2007 at 11:38 PM PDT #

Thanks Tor, I was missing my cluster updates :)

WRT "ugly problems" you mention in your post, the bug that worst affects me by far happens during rhtml editing. Would this fall under scripting/ruby, or is it handled elsewhere currently? (scripting/www?).

It seems to be document structure related. For example swapping repeated p tags to li tags, for a better semantic match, I get Assertion errors popping up every keystroke. Portions of rhtml start dropping from the editor view, needing the file to be reloaded in the editor.

Posted by Si on April 10, 2007 at 02:28 AM PDT #

Thanks Si. This does fall in the scripting/ruby category. I hacked together the RHTML support pretty quickly a while back and haven't touched it since, but the good news is that another engineer (Marek Fukala) is implementing it properly it in a branch right now and it's going to get a lot better pretty soon. I'll post something here (as a separate blog entry) when it gets integrated.

Posted by Tor Norbye on April 10, 2007 at 03:11 AM PDT #

Post a Comment:
Comments are closed for this entry.
About

Tor Norbye

Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today