Btrace is essentially a (new and) portable DTrace - a safe (read, not write), low-overhead, probe-based dynamic tracing tool. Btrace offers Java annotations to define what could be considered as troubleshooting interceptors ("probe points").
While BTrace applies to any Java application development, in the case of GlassFish, BTrace can be used to observe specific parts of the system or better yet, correlate events between it various parts. Athijegannathan has a recent post about a BTrace script tracking SQL statements (courtesy of a BTrace user) while Binod shares his experience of SailFin troubleshooting with BTrace.
Because BTrace is guaranteed to be safe (read-only, time-bound), it cannot throw/catch exceptions, create new objects/arrays, etc... These restrictions and more importantly all you need to get started is available in the BTrace Users's Guide. Check also the recent JavaOne slides.
Finally, a VisualVM plugin for BTrace is in the works and will make a nice addition to the GlassFish plugin for VisualVM.