The power of the AST view in debugging your grammar rules is discussed in detail here.
However, these two views are potentially confusing for end users (i.e., for people who are not module developers or, even, those who are module developers but do not need to write a Schliemann editor). Hence, in the RC releases, certainly in RC2, these views are not easy to find. We don't want the unwary user to stumble across them and get confused or worried or something similar. So, now, if you want to see these views... you must have an NBS file. You'd probably begin creating your Schliemann editor via the Language Support wizard in the New File wizard. So, now you have an NBS file (i.e., "NetBeans Scripting", which defines your tokens, grammar, and feature mappings). Now you can right-click this file and choose "Tokens View" or "AST View". And then these views open and now... they show you the AST and tokens of your NBS file.
But you don't want that! You want to see the AST and tokens of your own file. No worries. Just leave the window open, now that you've opened it. Leave it open! Then... select your own file. Thanks to selection management in NetBeans IDE, the views show the AST and tokens of the currently selected file. So now you can go ahead and select your own file and the windows will be populated with the correct AST and tokens. Drag the window somewhere else if that will make working with it more comfortable for you. You can even undock it (Alt-Shift-D) and then it will be freed from the NetBeans IDE frame.
Yes, maybe slightly cumbersome (but a lot less than this blog entry makes it seem). However, since these views are intended for a microscopically tiny fraction of developers, when compared to the entire NetBeans user base, it is provided in such a way that only those specifically needing them can can get to them. An alternative might have been to only show the views in the Window menu if an NBS file is created, but that would be contrary to all design principles. And the problems that this solution would cause would make the solution totally meaningless. Hence, under the circumstances, it all makes complete sense. However, it is good to know what happened and... now you know too. I will add this blog entry to our fantastic NetBeans Developer FAQ for future reference.