Tuesday Jun 23, 2009

New Plugin for VisualVM - Buffer Monitor

Buffer Monitor Plugin monitors the resources associated with pools of buffers. Plugin can monitor direct buffers that are allocated outside of the garbage-collected heap (by ByteBuffer.allocateDirect), and memory-mapped buffers, created by mapping a region of a file into memory (using FileChannel.map). The plugin uses newly added java.nio.BufferPoolMXBean to monitor the buffers usage. BufferPoolMXBean is available only in JDK 7 (build 36 and newer), so if you want to use this plugin, your monitored application must run on the latest JDK 7 build available here. More details about buffer pools monitoring are available in Alan Bateman's blog. The plugin is available at VisualVM 1.1.1 plugin center.

Thursday Jun 04, 2009

JavaOne 2009 Slides: Monitoring and Troubleshooting Java Platform Applications with JDK Software

On June 3rd there was a "Monitoring and Troubleshooting Java™ Platform Applications with JDK™ Software" (BOF-4724) session held at JavaOne 2009. The presentation mentioned monitoring and troubleshooting tools available in JDK, several demos presented how typical problems of Java applications can be detected and debugged using the VisualVM tool. Many attendees requested the slides to be made available online, many others who didn't make it to JavaOne this year may be interested in the contents. Click the image below to get the slides:

You may also download the sources for presentation demos, the NetBeans project archive is available here.

Monday Jun 01, 2009

Exporting Data From Profiler

Until NetBeans 6.7M3 the only way data could be exported from profiler was through profiler snapshot (.nps). The catch is, that profiler snapshot can't be opened by external applications like spreadsheet or browser. For this reason we decided to add ability to export data to other formats:

  • CSV - For analysis in spreadsheet
  • XML - For automated analysis
  • HTML - For quick and easy visualization in browser

The feature is available for both live results and snapshot for CPU and Memory profiling. In addition the raw data for VM Telemetry graphs can be exported too. It is accessible through the “Export to...” button in toolbar of each view.


Note that for all snapshots the profiler snapshot file (.nps) is always the default choice so there is no need to change workflow in case this feature is not interesting for you.

Thursday Feb 26, 2009

OQL support in the HeapWalker

With the arrival of the NetBeans 6.7M2 release the HeapWalker is enhanced with the OQL (stands for Object Query Language) support for analysing the heap-dump contents.

The OQL specifications are particularly complex and there is no complete implementation of the specifications yet. What more, there are various implementations enhancing the original OQL with their own additions. For the sakes of compatibility the jHat OQL implementation was chosen as the query engine for the HeapWalker. This decision also makes it possible to enhance the analytical capabilities of the OQL by custom JavaScript functions.

The actual OQL implementation comprises of the query engine and the visual console embedded within the HeapWalker. The visual console offers basic OQL language syntax highlighting and code completion. It also provides a contextual help linking to the rather exhausting jHat OQL documentation explaining the basics of the OQL as well as providing many working examples. The query results are bound back to the rest of the HeapWalker for the ease of navigation.

Demo of the OQL integration can be seen here.

Monday Feb 23, 2009

Visual GC Plugin 2.0 for VisualVM

The recently released VisualVM 1.1.1 introduced an updated Visual GC tool integration into VisualVM. From today the Visual GC Plugin 2.0 is also available for all other VisualVM and Java VisualVM releases via the Plugin Center.

Visual GC is an experimental Visual Garbage Collection Monitoring Tool - a graphical tool for monitoring the HotSpot Garbage Collector, Compiler, and class loader. The tool was originally developed as demonstration software for JavaOne 2001 where it was used in a presentation to help describe how various tunable parameters affect the operations of the HotSpot generational garbage collector. In 2004 it was presented in technical session Using jvmstat and visualgc to Solve Memory Management Problems.

Detailed information on using the Visual GC tool can be found on the Visual GC page. The latest standalone Visual GC tool can be downloaded here.

Using the Visual GC plugin for VisualVM has several advantages over the standalone tool:

  • No command-line launcher. The standalone tool has to be launched from a command-line with at least one argument specifying the PID of the Java process to be monitored. When using the VisualVM plugin, the Visual GC GUI is displayed in a separate subtab for each application.
  • Remote monitoring without any special setup. When the jstatd daemon is already running on a remote server to enable the VisualVM to discover running remote Java applications, no special setup is required for Visual GC to monitor these applications.
  • More user-friendly UI. The plugin changes the default Visual GC appearance to fit into VisualVM UI and provide better user experience. The most significant change is using white background instead of the default black. Changing the background color also required to change the graphs colors. The Application Information section has been removed - the information is already available in Overview subtab of each application. The plugin also sets default font for the labels, fixes several layout problems and allows to configure the refresh rate during runtime. Note that all these customizations can be switched off by starting VisualVM with a special argument -J-Dcom.sun.tools.visualvm.modules.visualgc.originalUI=true.

  

To install the Visual GC plugin for VisualVM, use Tools | Plugins in VisualVM menu bar and select the 'Available Plugins' tab. Alternatively, if previous version of the plugin is already installed, the updated plugin will be displayed in the Updates tab. You may need to click the 'Reload Catalog' button to refresh the available plugins if the older plugin version (1.1.1) is cached.

When installed, the Visual GC subtab is added for each Java application. The graphs are available for all applications running Sun JDK 1.4.2+ with jvmstat available - this means that both local and remote applications can be monitored without any limitations (jstatd daemon must be running on remote hosts). The Histogram view may or may not be displayed by default depending on the application configuration - when the monitored VM uses Parallel Scavenge collection (using Server VM or setting -XX:+UseParallelGC or -XX:+AggressiveHeap), Histogram view is hidden by default.

The refresh rate of Visual GC graphs is customizable during runtime, it can be changed using the Refresh Rate dropdown. The value is specified in milliseconds and the default value is Auto which means using the global refresh rate defined in Tools | Options | Polling - Monitored Data.

Tuesday Jan 13, 2009

New Command-Line Options in VisualVM 1.1

Recently released VisualVM 1.1 tool introduces three new command-line options: --openfile, --openpid and --openjmx. These options allow to use VisualVM as an application or file viewer. This can be handy especially for viewing heap dumps and thread dumps or for integration with third-party tools.

A nice feature of VisualVM command-line support is the ability to reuse already running VisualVM instance. If you request to open a file for the first time, VisualVM is launched and the file is opened. If you request to open another file, it will be opened in the tool which is already running. This is possible thanks to the NetBeans Command Line Parsing API used by this feature.

The --openfile option

syntax: visualvm --openfile [filename] where filename is full path to a supported file

allows to open a supported file in VisualVM. Supported are the same file types as in the Load dialog in VisualVM GUI - by default thread dumps (.tdump), heap dumps (.hprof) and profiler snapshots (.nps) are supported. Since the heap dumps are recognized by the file header and not just according to their extension, also .bin and other heap dumps can be opened by VisualVM. If a third-party tool adds support for another file type, it will be automatically available also from the command-line.

The --openpid option

syntax: visualvm --openpid [pid] where pid is process id of running Java application

allows to open local Java process in VisualVM. This is an alternative to the Open action in VisualVM GUI. Note that there's a predefined timeout 5sec for which the VisualVM waits for the PID to become available - this allows to start VisualVM along with the monitored application. The --openpid option may be used for example in batch commands for monitoring several predefined applications (application servers etc.) or for integration with third-party tools (like the integration module for the Eclipse IDE).

The --openjmx option

syntax: visualvm --openjmx [hostname:port] where hostname:port defines the JMX connection

allows to connect to local or remote Java process using the JMX connection. This is an alternative to the Add JMX Connection action in VisualVM GUI. JMX connection defined from the command-line is not restored after restarting VisualVM. The --openjmx option may be used for example in batch commands for monitoring several predefined applications (application servers etc.) or just like a simple remote application monitor/viewer.

Thursday Jul 31, 2008

Profiling With VisualVM, Part 2

This article describes advanced profiling features of the VisualVM tool which give the user more control over the profiling session and collected data. You may also want to read the Profiling With VisualVM, Part 1 article describing the basic profiling features of VisualVM.

By default, the Profiler UI in VisualVM doesn't display any settings, the initial configuration ensures that even an unexperienced user will get some useful results with zero configuration by just clicking the CPU or Memory button. However, an advanced user still has the possibility to fine-tune the settings and fully utilize the power of the NetBeans profiler engine.

To display the settings area, Settings checkbox has to be selected in the topright corner of Profiler tab. The settings area consists of two tabs, CPU settings and Memory settings, containing controls for each profiling mode.

Note that the settings can only be changed when the profiling session is not in progress, during profiling the settings area is disabled.

There's also a Restore Defaults button available at the bottom of the area which restores the initial configuration. This is useful when the values were customized incorrectly and no profiling data are being collected.

Profiling Performance

For performance profiling there are two important settings available: profiling roots and instrumentation filter. In general, profiling roots define when the profiler should start collecting the data whereas instrumentation filter defines packages/classes from which the data should/shouldn't be collected.

Start profiling from classes textbox defines the profiling roots. If all the methods were instrumented, the profiling overhead would be enormous and profiling would eventually become unusable. That's why the NetBeans profiler introduces a notion of Root Methods - a way to profile just the code of interest while the rest of the application runs at full speed.

A typical usecase is for example profiling performance of an action invoked by a button click - if the appropriate actionPerformed() method is set as a root method, the profiler starts instrumenting and profiling the application once the actionPerformed() method is invoked and only profiles the methods called from the actionPerformed(). This way the profiling overhead is very small and only relevant profiling results are collected which greatly improves their readability.

Whereas in NetBeans profiler selecting a root method is very natural because of its tight integration with the editor, in VisualVM this might be a bit tricky and not exactly user friendly. That's why VisualVM defines profiling roots by classes - all methods defined in these classes and their innerclasses become root methods. In the similar way, package names can be used as profiling roots. That's how the default profiling roots are defined: <main_class_package>.** which means that all methods from <main_class_package> and subpackages are profiled.

The default settings work in most cases, but for large applications like application servers it might be useful to tweak the profiling roots to point to a concrete code of interest. For example, when profiling a web application it's much better to set the servlet's package or classname as a profiling root instead of profiling the whole server.

The format for entering the profiling roots in VisualVM is following:

  • org.mypackage.** defines all methods in org.mypackage and all subpackages as root methods
  • org.mypackage.* defines all methods in org.mypackage as root methods
  • org.mypackage.MyClass defines all methods in org.mypackage.MyClass and its innerclasses as root methods
  • empty textbox means that all methods executed in the application will be profiled

Multiple profiling roots (separated by a space and/or comma and/or newline) can be defined for a single profiling session.

Note that it doesn't make sense to define org.mypackage.MyClass.* or org.mypackage.MyClass.** as profiling roots since the tool always selects all methods of the class and its innerclasses as profiling roots.

Profile only classes / Do not profile classes textbox defines the instrumentation filter. This is exactly the same option as Quick Filter in the NetBeans profiler. It either defines the only packages/classes to be profiled or packages/classes not to be profiled. For example, if only an application's code should be profiled without any outgoing calls to other frameworks or JDK code, this option should be set to Profile only classes, <application_package>.*. If all the code except JDK classes from java and javax packages should be profiled, this option should be set to Do not profile classes, java.*, javax.*. The default instrumentation filter is Do not profile classes, java.*, javax.*, sun.*, sunw.*, com.sun.* (+ com.apple.*, apple.awt.* on Mac OS X) which filters out all the JDK code.

The format for entering instrumentation filter is following:

  • org.mypackage.* selects all classes from org.mypackage and subpackages to be profiled or to be excluded
  • org.mypackage. selects all classes from org.mypackage to be profiled or to be excluded
  • empty textbox selects all classes of the application to be profiled

Multiple filter values (separated by a space and/or comma and/or newline) can be defined for a single profiling session.

Profiling Memory

For memory profiling there are three important settings available: profiling scope, density of tracked objects and recording allocations stack traces.

The profiling scope is defined by one of two choices: for the Profile object allocations option only object allocations are monitored. This is useful when evaluating how many objects are allocated by the application. For checking and identifying memory leaks the second choice is more useful - Profile object allocations and GC. In this mode the allocated objects are monitored during the whole lifecycle, from their allocation till garbage collection. This allows to easily identify live objects currently allocated on the heap at certain point. Also, in this mode the Surviving Generations metrics is available (see this article for more information). By default, Profile object allocations and GC option is selected.

The Track every Nth object allocations option allows to define density of tracked objects. In most applications it isn't necessary to monitor each single instance, the overall allocations behavior is the same when tracking just each Nth object. The advantage of this approach is lower profiling overhead. To get exact number of allocated or live objects, this option has to be set to 1. By default, each 10th object is tracked.

The third control is the Record allocations stack traces checkbox. When selected, a stack trace is taken for each monitored object when allocated. This allows to identify the concrete method (and its call tree) which has allocated the object. By default the collecting of allocation stack traces is disabled to lower the profiling overhead.

Note that by default some columns of memory results are hidden. You can display more metrics using the top-right corner button in Profiling results or memory snapshot tables.

Comparison With The NetBeans Profiler

This section compares profiling settings available in the NetBeans profiler and VisualVM. For the settings which cannot be customized in VisualVM the predefined value is listed.

CPU profiling settings:

Settings NetBeans profiler VisualVM
Profiling roots packages/classes/methods/project methods packages/classes
Instrumentation filter predefined sets/user defined/project code user defined
Profiling points enabled/disabled N/A
Profiling technique instrumentation only/instrumentation & sampled time instrumentation only
Exclude Thread.sleep() & Object.wait() time on/off always on
Profile underlying framework startup on/off always off
Profiling technique instrumentation only/instrumentation & sampled time instrumentation only
Profile new Threads/Runnables on/off on/off
Profiled threads limit 1 to unlimited always 32
Thread CPU timer (Solaris only) on/off always off
Instrumentation scheme total/eager/lazy always lazy
Instrument Method.invoke() on/off always on
Instrument getters/setter on/off always off
Instrument empty methods on/off always off

Memory profiling settings:

Settings NetBeans profiler VisualVM
Profiling scope object allocations / object allocations & GC object allocations / object allocations & GC
Density of tracked objects track every to every Nth object track every to every Nth object
Allocations stack traces on/off on/off
Profiling points enabled/disabled N/A
Limit stack traces depth 1 frame to unlimited depth always unlimited depth
Run GC when getting results on/off always on

Note that you can easily determine the used profiling settings in both NetBeans profiler and VisualVM by selecting the Info tab of the CPU or memory snapshot.

Further Reading

For more information see the VisualVM Documentation, especially the Profiling Applications section.

You may also find the NetBeans Profiler 5.5 documentation useful, especially the Instrumenting a Root Method, CPU Profiling settings or Memory Profiling Settings sections.

Monday Jul 28, 2008

Profiling With VisualVM, Part 1

This article describes the out-of-the-box profiling features of the VisualVM tool available without any additional configuration, just using the default settings.

Beside various monitoring features, the tool contains a built-in profiler based on the NetBeans profiler. While the profiler UI in VisualVM looks simple (especially when compared to the NetBeans profiler), the profiling capabilities are almost as powerful as in NetBeans. The main difference between these two tools is their workflow: whereas NetBeans profiler is project/sources-centered and is a great tool for iterative testing/improving performance during application development, VisualVM can be used as a standalone profiler which works with any running application without having any context of the source code.

Note: For profiling applications running on JDK 6.0 Update 6 and earlier, the applications need to be started with the Xshare:off flag which disables the class sharing, otherwise the profiling may cause a crash of profiled JVM. For applications running on JDK 6.0 Update 7 and higher the profiling should work without any modifications.

The profiling UI is very simple, by default it shows just three buttons for analyzing performance (CPU button), memory usage (Memory button) and stopping the profiling session (Stop button). The other UI elements are Status area where the actual profiling status is shown and Profiling results section which displays live profiling data collected during the profiling session. For advanced profiling there's also a Settings section which is hidden by default. It allows to fine-tune the profiling settings, this is described in detail in Profiling With VisualVM, Part 2 article.

Note: Whereas the Profiler tab is available for each application which can be profiled, only one application can be profiled at a time, concurrent profiling sessions are not supported.

Profiling Performance

When the CPU button in Profiler tab is pressed, the profiler attaches to the application and starts profiling its performance. At the beginning the profiler needs to instrument some methods of the application, the actual number of instrumented methods is displayed in the Status area. Since the profiler uses dynamic instrumentation, the number of instrumented methods may change during the profiling session. After the profiler is attached and starts collecting the data, the view looks like on the screenshot:

You can see live profiling data for the application, for each profiled method number of invocations and total execution time is displayed. Note that not all methods are profiled, by default the profiler doesn't profile methods from java.*, javax.*, sun.*, sunw.*, com.sun.* (+ com.apple.*, apple.awt.* on Mac OS X) packages. Time spent in methods from these packages is added to execution time of profiled methods calling them, for example constructor execution time of a custom class (CustomClass.<init>) includes execution time of super constructor etc.

The toolbar of Profiling results section contains the following actions (from the left):

  • Update Results Automatically: results are periodically refreshed approx. each 1.2sec
  • Update Results Now: immediately fetches actual data, typically used when automatic refresh is disabled
  • Run Garbage Collection: invokes GC in profiled application
  • Reset Collected Results Buffer: resets the data collected so far, useful for measuring delta results
  • Take Snapshot of Collected Results: saves all the collected data into a persistent snapshot for more detailed analysis
  • Save current view to image: saves the results table into a .png image

The most important action is Take Snapshot of Collected Results (available also as Profiler Snapshot in context menu of profiled application in Applications tree). It takes a snapshot of collected profiling results (compatible with NetBeans profiler snapshot) and opens it in a separate tab. This snapshot provides several different views: Call Tree displaying methods call tree starting by threads, Hot Spots displaying a list of all profiled methods sorted by their execution time and Combined view showing both call tree and list of profiled methods. The last view Info displays basic snapshot information and detailed profiling configuration of the snapshot. There's one handy action available in context menu of each method, it's called Show Back Traces and displays all the places from where the method is being called.

  

Multiple profiler snapshots can be taken for a single profiling session for example to compare performance in various situations / loads / configurations of the application. All the profiler snapshots can be then saved into a single Application Snapshot (use Application Snapshot action in context menu of profiled application in Applications tree) to archive them in one file or to send them to application developers/quality engineers.

Profiling Memory

When Memory button in Profiler tab is pressed, the profiler attaches to the application and starts profiling its memory consumption. At the beginning the profiler needs to instrument all classes of the application, the actual number of instrumented classes is displayed in the Status area. Because all the application classes need to be instrumented, it takes some time until the profiler starts to collect the data. To lower the profiling overhead, each 10th object allocation is tracked by default (also displayed in Status area). Keep this in mind when evaluating the profiling results. After the profiler is attached and starts collecting the data, the view looks like on the screenshot:

You can see live profiling data for the application, for each class number of live instances and their size is displayed. The rightmost column shows Surviving Generations metrics which is very useful for detecting memory leaks. Detailed explanation of Surviving Generations is available in this article. You can also monitor allocations history of a particular class during time (History subtab at the bottom), to do this invoke the Log Class History action in context menu of the class.

The toolbar of memory profiling results is the same as for CPU profiling results. You can also save the collected results into a snapshot using the Take Snapshot of Collected Results action in Profiling results section toolbar or Profiler Snapshot context menu action. This is especially useful for detecting memory leaks, using the Compare Memory Snapshots action available in File menu you can visually display the difference between two comparable memory results and for example evaluate if/which objects remained on the heap after executing some action.

Note: When stopping the profiling session (both CPU an Memory) the profiler restores original application bytecode. This can take some time and the application and/or the whole system may not respond for a while.

Profiler Calibration

When VisualVM is started for the first time, the profiler performs an initial calibration. During this process the overall system and actual JDK performance is being measured and these data are later used to subtract the profiling overhead from collected performance data. The profiler requires a "stable" performance of the system, this means that any technique for dynamic CPU frequency switching like SpeedStep or PowerNow! should be disabled during both the calibration and profiling.

If the system performance changes or a different JDK than the calibrated one is used for profiling, the collected performance results may become biased and the profiler needs to be re-calibrated. Unfortunately there's no UI action available for recalibration in VisualVM 1.0, the only way to invoke the calibration again is to manually delete calibration data file in <user_directory>/.nbprofiler/machinedata.jdk1X and restart the VisualVM.

Further Reading

For more information see the VisualVM Documentation, especially the Profiling Applications section.

You may also find the NetBeans Profiler 5.5 documentation useful, especially the Profiling Results - Application Performance, Profiling Results - Memory Usage, CPU Snapshot or Memory Snapshot sections.

Tuesday May 06, 2008

VisualVM 1.0 RC Released

VisualVM 1.0 Release Candidate is available, you can download it from https://visualvm.dev.java.net/. The most significant news are:

  • Improved performance
  • Improved memory management
  • Improved stability
  • Start Page with links to VisualVM documentation and JDK monitoring and troubleshooting guides

Wednesday Feb 07, 2007

What Do The Surviving Generations Metrics Mean?

In VM Telemetry graphs of the Profiler you can see Surviving Generations metrics which can detect potential memory leaks. Here's a three-lines definition of the what the metrics mean:

  • a Generation is a set of instances created within the same GC interval (between two garbage collections)
  • a Surviving Generation is a Generation that survives at least one garbage collection. The number of survived garbage collections - the generation's age - is its unique identifier
  • Surviving Generations (metrics) value is the number of differerent Surviving Generations that are currently alive on the heap (number of Generations with different generation ages)

Typically there are several long-lived objects (like an application's main JFrame etc.) in an application representing one or a few Surviving Generations. There are also many short-lived objects created very frequently (such as Dimension etc.) and released soon, typically within only a few garbage collections. They also represent just several Surviving Generations.

After some period of time the number of Surviving Generations in an application should become stable because all long-lived objects have already been created and newly-created short-lived objects are periodically being released from the heap. However, if there is a memory leak in an application which prevents newly-created objects from being released from the heap (for example objects stored in a Collection and never removed), the number of Surviving Generations grows and that's exactly what the Surviving Generations metrics is able to detect regardless of how much memory is wasted by such a leak.

Note: This is an excerpt from an article published at netbeans.org. You can read the full story with a simple four-steps tutorial here.

Monday Feb 05, 2007

Redesign of Select Profiling Task Dialog

We are thinking of redesigning the Select Profiling Task dialog for Profiler 6.0. The main reasons are to add support for controlling new Profiler features and allow future extensibility, improve usability and fix accessibility problems.

Here are some screenshots of how the new dialog should look like, we will welcome any feedback:

CPU - Analyze Performance predefined task - Basic settings

CPU - Analyze Performance predefined task - Advanced settings CPU - Analyze Performance predefined task - Load generator support CPU - Analyze Performance predefined task - Attach Profiler

Friday Sep 22, 2006

Let Us Know What Do You Need!

Do you need a feature for your profiling which the NetBeans Profiler lacks? Just send us a feedback to feedback@profiler.netbeans.org or file a RFE in our IssueZilla.

Do you have any problem with your profiling? Ask us and the community for help and assistance at users@profiler.netbeans.org. Or try to find a solution in Profiler documentation.

To help you get started with the Profiler there are tutorials and flash demos available. You will find them at the NetBeans Profiler homepage. More articles can be found in NetBeans articles repository.

About

A blog by NetBeans Profiler & VisualVM developers.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today