Okay, so which one to choose? (In fact, someone asked me this very question yesterday.) Everything you ever wanted to know about 'Lexer' (what it is and what's so special about the NetBeans approach to it) is discussed here, in an interview with Mila Metelka, the author of the NetBeans Lexer API:
Some of the content in the above interview is now out of date, but the principle statements there are still accurate. However, there's also 'Schliemann', the declarative approach to language programming. The core concept to understand in relation to Schliemann is—you don't need to be a brain surgeon to use this approach, whereas Lexer is still pretty complex, although much simpler than the old Syntax class. In fact, you don't even need to know Java. Read that again and let it sink in. That's why it's there, principally for scripting language authors, who might not even know Java themselves. These people need a rough and ready way to introduce their scripting language to the IDE, thus providing tooling support for their scripting language, which in turn will serve to popularize it. So, what happens when the world's simplest syntax coloring approach (i.e., Schliemann) meets the world's simplest language notation (i.e., JFugue)? Well, here's everything I needed to do to recreate my coloring in the Schliemann world:
# definition of tokens
TOKEN:instrument:( "I" "[" ["A"-"Z" "_"]\* "]"\* )
TOKEN:noteStart:( ["A"-"G"] )
TOKEN:noteEnd:( ["0"-"9"] "q" | "i" | "w" | "h" | "s" )
TOKEN:tempo:( "T" ["0"-"9"]\* )
By the way, the above is not a code snippet. It is the entire file. When I open the file, I see exactly what you see above, nothing else, not one additional character, is in the actual file. And the result? Well, look here:
Now, what are the benefits of this approach? Well, when I check in my changes, guess what, all of the Java classes, and one settings file, shown below, can be deleted:
That's great news—when debugging, I will not need to dig through 10 different classes to find my problem. Instead I will have one simple declarative file, which I can scan with the naked eye, without needing to translate from the Java world to my own non-Java brain. And why did I choose Schliemann instead of Lexer? Because Schliemann is faster to implement (as evidenced above, since no syntax coloring implementation could possibly be simpler than that). I simply need a MIME resolver and the above file, which I register in the XML layer, and I am done. However, although I can add a variety of other language features, they'll always be coarse grained. My understanding of the distinction between Lexer and Schliemann is that you can achieve a lot more with Lexer. For example, all the semantic analysis done in the new Java editor, as well as in the Ruby editor, isn't possible when you're using Schliemann. That's when you need Lexer. However, I imagine that Schliemann will get more and more powerful, in fact, it is already possible to refer to Java snippets from the above Schliemann file, so that you can create a lot more finetuned code for analyzing a syntax. However, for notations such as JFugue which have simplicity at its heart, Schliemann is a gift from heaven. No, not from heaven. From Sun.
In other news. I haven't committed my Schliemann implementation to CVS yet, so the application still runs on 5.5. Waiting to add some more Schliemann features, and doing some testing, prior to committing them, probably to a 6.0 branch of the JFugue Music NotePad.