Tool for Diagnosing Failed GlassFish Redeployments/Undeployments on Windows due to Locked JARs

If you see redeployment or undeployment errors on Windows, with GlassFish saying it cannot rename or delete files, you can now help diagnose the problem yourself on your system. This will save you from having to send us your app and it will save us from installing and trying your app, all of which will make for a faster resolution. Plus you don't have to share your app, which is a problem for many people.

Background

When GlassFish deploys an app, it creates an internal directory to hold the files contained in the app. Do not depend on this convention, but currently by default GlassFish places an enterprise app's files in $install-dir/domains/$domain-name/applications/j2ee-apps/$app-name. As it redeploys an app, GlassFish starts by renaming the existing internal app directory to ${app-name}_old. This allows GlassFish to restore the original app in case of errors during the redeployment. And, naturally, during undeployment GlassFish removes the app's directory and its contents.

One aspect of Windows (you decide if it's a feature or a bug!) is that an open file cannot be deleted or overwritten nor can its containing directory be renamed.

Put these facts together and you can see there can be trouble on Windows if, for some reason, a file in the application's directory is open when you try to redeploy or undeploy the application. Perhaps in another post I'll talk about why files might be open. Suffice it to say that this does happen from time to time. The result is failed redeployments or undeployments. Typically restarting the server clears up the problem, but that's a real nuisance especially for developers in the edit/build/deploy/test/repeat cycle.

It's important to find out which file or files are open and why. It could be due to behavior in GlassFish, in the application code, in Java itself, or in a third-party library. The process for solving the underlying problem is different in each case. (I should point out that we continue to work on ways of letting GlassFish tolerate such problems without causing deployment or undeployment failures, regardless of the underlying problem. But it's still good to discover the root cause.)

A tool to help

If it's a JAR that's open and causing the trouble, I can now offer a tool that helps identify what code opens the JAR.

Disclaimer: This tool is unsupported!

It's also a bit rough around the edges but it gives information that's extremely valuable in tracking down where JAR files are opened. (A link to the JAR for the tool is below; I plan to post the source soon.)

How the tool works

The basic idea is this: Whenever a JAR file is opened, a new instance of the java.util.jar.JarFile class is created. JarFile extends java.util.zip.ZipFile which has some very useful information about the file. The tool uses the Java Debugger Interface (JDI) (JDI API spec, intro page for the overall Java Platform Debugger Architecture or JPDA) to connect to the GlassFish JVM. It sets a breakpoint in the ZipFile constructor (which is invoked whenever a zip file is opened) and another in its close method. When the tool detects that the GlassFish JVM reaches the constructor breakpoint it collects some information about the current ZipFile object: the current stack trace, the file path, and the object's hash code. The tool saves this information locally and then lets the GlassFish JVM continue. When the GlassFish JVM reaches the breakpoint for the close method, the tool finds the object's hash code and discards the previously-saved information about that ZipFile instance.

So at any given moment the tool has a record of all zip files (JAR files) that are opened but not yet closed by the GlassFish JVM. This could be a lot of JARs, so the tool allows you to select which files to monitor using a filter.

The tool also responds to two commands (“show” and “exit”). The “show” command will display the recorded information about every currently-open zip file in the GlassFish JVM that matches the file path filter. This output includes the path of the file and the stack trace for each instantiation of ZipFile corresponding to that same file.

The net effect of all this is that if you issue the “show” command immediately after a redeployment or undeployment has failed, you should see stack traces that lead you to the code that opened – but has not closed – the locked JAR file.

Using the tool

Download the tool

At least for the moment, get the tool from here. It's a single, small JAR file. Save it somewhere where you can find it again.

Set up GlassFish for debugging

Because the utility uses the Java debugger API to connect to GlassFish, you'll need to configure your GlassFish instance to accept debugger attaches. If you have not done this before the following instructions may seem long but it's not that bad a process, really. You can either use the admin GUI :

  • Choose Application Server from the left-hand panel.

  • Choose the JVM Settings tab.

  • Make sure the Debug check box is checked.

  • Look for the debug options and notice what port number is in the "address" setting or set it to something you want.

or edit the domain.xml file manually (and carefully!) to accomplish the same things:

  • The <java-config> element has a debug-enabled attribute; set it to "true"

  • The same element has a debug-options attribute where the port number is set. Note its value or set it to something you want.

Start GlassFish

  • Start GlassFish and wait for it to settle down. (The admin start-domain command will return before the app server is completely started.)

Start the utility

  • Start the utility in a command window using

java -classpath <path-to-the-utility-jar>;%JAVA_HOME%/lib/tools.jar util.ZipFileMonitor -host yourhost -port 9009 -filter xyz


    Replace 9009 if you changed the port number that was already set. Replace “yourhost” with the host where you started GlassFish.


    Replace “xyz” with a filter expression that will match the JARs in your application. A good choice is the name of the application itself, since all JARs in an app will appear within a subdirectory tree named after the module name. The default module name for a deployed app is the name of the archive file – foobar for the file foobar.ear, for example.

  • From another command window or from the admin GUI deploy the application in question.

  • Redeploy or undeploy the application in question, repeating if needed until you see the error.

And when it fails...?

In the command window for the utility enter the “show” command and capture the output. If the output indicates that your application code opened the JAR then decide where the application should close the file and change the logic. If it looks like something else opened the JAR, see if someone has already reported a similar problem in the GlassFish forum. If not, post your output with a brief description of the problem and the community will take a look.

A little more about the utility

The full set of command-line options are:

Option

Usage

Default Value

-host

The host name where GlassFish is running.

localhost

-port

The port number where GlassFish is listening for debugger attaches.

9009

-timeout

Time in milliseconds that the utility will wait for GlassFish to accept its debugger connection before aborting.

infinite

-filter

Comma-separated list of strings. If the full path of a JAR file being opened contains any of the strings in the filter then the utility will monitor that JAR.

Enterprise (for no particularly good reason)



During deployment and redeployment you'll see quite a few messages displayed on the utility's System.out. This is not the real information we're after but the messages are reassuring that the files you want to monitor (and only those) are in fact being monitored.

Once you have started the utility it accepts two commands: show and exit. The show command will list out every JAR file that is still open and will display the stack trace from which it was opened. Note that a single jar file may be opened from more than one place, so you may see multiple traces for a given JAR. Note also that if you deploy the app and then use the show command you will see a large number of stack traces. This is normal, because those JARs are legitimately still open while the app is deployed. What will be of interest is the set of stack traces after a redeployment or undeployment fails. You can issue the show command as many times as you want.

The exit command detaches from the debugger in the GlassFish JVM and exits.

As I said, there are some rough edges. For example, if you stop GlassFish the utility will report this but does not automatically exit. When you then give it the exit command it throws an exception because it tries to disconnect from the GlassFish JVM even though the connection is no longer in place.

Feedback

For comments on the utility itself, please feel free to add a comment to this posting or send me an e-mail message.

If you encounter a locking problem and you use the tool to help diagnose it, please first search the GlassFish forum to see if someone has already seen a similar problem. If there is none, please create a new topic there. This way the community can cooperate in understanding and solving the problem and you will help other users who may run into the same problem later.

GlassFish JPDA



Comments:

Hi Tim, what an absolutely great tool! I enjoyed using it a lot, job very well done. Additional information (which will help increase the usefullness for non-Glassfish users): it also works well for other application-servers and servlet containers (e.g. Tomcat), not only for Glassfish! regards, Martin

Posted by Martin Marinschek on September 20, 2006 at 12:24 AM CDT #

Every time I try to use the tool I get this exception: java.lang.NullPointerException at com.sun.tools.jdi.MirrorImpl.validateMirror(MirrorImpl.java:49) at com.sun.tools.jdi.EventRequestManagerImpl.createBreakpointRequest(Eve ntRequestManagerImpl.java:789) at util.ZipFileMonitor.addJarFileConstructorBreakpoint(ZipFileMonitor.ja va:358) at util.ZipFileMonitor.setBreakpoints(ZipFileMonitor.java:338) at util.ZipFileMonitor.run(ZipFileMonitor.java:104) at util.ZipFileMonitor.main(ZipFileMonitor.java:81) I am using it with Java 6. Any ideas? Thanks

Posted by vladchuk on February 01, 2007 at 03:03 AM CST #

As soon as I can I will see if I get the same problem using Java SE 6. Would it be an option for you to use Java SE 5 - even if temporarily - to see if that solves the immediate problem with the tool and allows you to identify why the file is being locked?

Posted by Tim Quinn on February 01, 2007 at 03:32 AM CST #

Tim, I just tried with Java 5 and got exactly the same exception. Is is some configuration issue?

Posted by vladchuk on February 01, 2007 at 04:44 AM CST #

Hi Tim, your original entry in August mentioned you might be posting the source for your tool soon. Has that already happened? If not, do you still plan to release the source? Thanks for the great tool.

Posted by Andrew McCulloch on May 22, 2007 at 04:09 AM CDT #

I get connection error when I try to connect remotely. Isn't a login required? Is this tool intended to be used remotely? C:\\JDev>java -classpath c:\\HOLD\\ZipFileMonitor.jar;%JAVA_HOME%/lib/tools.jar util.ZipFileMonitor -host www.hhartley.com -port 9009 ZipFileMonitor(18-Mar-2007) java.net.ConnectException: Connection refused: connect at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(Unknown Source) at java.net.PlainSocketImpl.connectToAddress(Unknown Source) at java.net.PlainSocketImpl.connect(Unknown Source) at java.net.SocksSocketImpl.connect(Unknown Source) at java.net.Socket.connect(Unknown Source) at com.sun.tools.jdi.SocketTransportService.attach(SocketTransportService.java:204) at com.sun.tools.jdi.GenericAttachingConnector.attach(GenericAttachingConnector.java:98) at com.sun.tools.jdi.SocketAttachingConnector.attach(SocketAttachingConnector.java:72) at util.ZipFileMonitor.connectToRemoteJVM(ZipFileMonitor.java:220) at util.ZipFileMonitor.run(ZipFileMonitor.java:92) at util.ZipFileMonitor.main(ZipFileMonitor.java:75)

Posted by ITVGuy2000 on July 12, 2007 at 06:57 AM CDT #

Wouldn't it be better to have the default value of the filter to be none, where there is \*no\* filter? Or perhaps some support for wildcards? I have tons of tiny apps of one or a few jsp's. I believe it is a cumulative effect of deploying and compiling that causes me problems.

Posted by ITVGuy2000 on July 12, 2007 at 07:11 AM CDT #

About Andrew's (now very old) comment... Sorry about the late response. I think there were some comment-notification problems and I have only just now noticed your comment. I have not yet posted the source. I would like to do so but that's kind of a low priority for us at the moment. When the time comes - soon, I hope - I'll post another comment. Thanks for your interest.

Posted by Tim Quinn on July 12, 2007 at 07:20 AM CDT #

RE: ITVGuy2000's comments... 1. Are you sure you enabled debugging in the server that you wanted to test? You can try using any JPDA-compliant debugger (such as NetBean's, for example) to connect to your server remotely. If that works, then this tool should work since it layers on top of the JPDA API. 2. As for the default filter value... There undoubtedly lots of improvements possible in the tool, your suggestion among them. My original plan was in fact for the filter to be regular expression, but I had a \*very\* pressing need to get the tool working so I relegated that to the list of possible enhancements. And yes, I know, if the source were posted then people could make their own mods! - Tim

Posted by Tim Quinn on July 12, 2007 at 07:29 AM CDT #

I am able to connect with Netbeans.

Posted by ITVGuy2000 on July 12, 2007 at 08:06 AM CDT #

Sorry the tool is not working remotely. I'll need to debug it at some point.

In the meantime does it work if you run it on the same system as the server you want to work with?

Posted by Tim Quinn on July 12, 2007 at 09:17 AM CDT #

Windows users with this problem can grab Unlocker from 'http://ccollomb.free.fr/unlocker/'.

Set up an ant task to run it before doing a war deploy:

--- Properties ---
# Use the Unlocker program on Windows machines to force unlocking of locked files before an install.
# Download Unlocker from 'http://ccollomb.free.fr/unlocker/' (or download.com).
# Uncomment do.use.unlocker to enable.
do.use.unlocker=Set this line to Anything to run Unlocker.
unlocker.path="c:/program files/unlocker/unlocker.exe"
#Set unlocker.flags to blank to show the dialog every time.
unlocker.flags=-S
# Unlocker requres the correct slash direction for windows.
unlocker.dir.to.unlock="${env.JAVAEE_HOME}\\\\domains\\\\domain1\\\\applications\\\\j2ee-modules\\\\yourDomainName\\\\WEB-INF\\\\lib"

---- Ant ------
<target name="_unlock_windows_files" if="do.use.unlocker" >
<echo message="Executing: ${unlocker.path} ${unlocker.dir.to.unlock} ${unlocker.flags}"/>
<exec executable="${unlocker.path}">
<arg value="${unlocker.dir.to.unlock}"/>
<arg value="${unlocker.flags}"/>
</exec>
</target>

Posted by Geoff Granum on September 06, 2007 at 07:15 AM CDT #

I have no experience with this tool and do not know precisely how it works, although I can guess. It seems to me that if the Java runtime thinks the file is still open but this unlocker tool has reached inside the process and forced a file closed at the Windows I/O level, then users could see very odd errors as Java tried to use files that it thought were open but had been closed out from under it.

I'd be interested to hear from GlassFish users who see locked JARs and use the unlocker tool.

Note that GlassFish V2 contains significantly revamped deployment/undeployment/redeployment logic that is a huge help in avoiding these sorts of problems.

- Tim

Posted by Tim Quinn on September 06, 2007 at 07:41 AM CDT #

The original problem is in the WebAppClassloader that the glassfish uses. This classloader keeps the opened jars in an array. It closes the jars only in case of undeploy, and not in case of redeploy or autodeploy. I created a workaround that works with eclipse. I created a simple JSP that retrieves its own classloader and climbs up until it finds the WebAppClassLoader ancestor. Then it calls the public closeJARs(true) method to close the jars.

I also created an eclipse plugin that extends the IStartup extension point. It subscribes as a ServerLifeCycleListener. It always subscribes to any server as a publishlistener. Then in the publishstarted method I simply call the JSP above to make the classloader to release the jars.

It works fine, and I am satisfied :)

Posted by Lipi on February 09, 2009 at 05:24 PM CST #

Having the same Connection refused issue ITVGuy posted about. I am trying to run on Windows 7 Ultimate 64 bit with JDK 1.6.0_22 32 bit and GF 3.1. I know debugging is enabled and working being I can attach and debug using IDEA with the same host/port settings. I am trying to run the tool from the same host that GF is running on as well.

Posted by NBW on March 15, 2011 at 08:22 AM CDT #

NBW,

I'm sorry the tool is not working for you.

I have not tried it myself on a Windows 7 system. I am not sure that is the problem but it certainly could be a factor.

Is there a chance that the GlassFish server is running one Java release and when you run the tool you are using another? I would expect that to work but we cannot rule anything out at this point.

Posted by Timothy Quinn on March 15, 2011 at 08:42 AM CDT #

The link to the file ZipFileMonitor.jar is broken.

Posted by guest on March 17, 2012 at 03:30 PM CDT #

The link is fixed now. Thanks for pointing that out.

Posted by Tim on March 17, 2012 at 03:41 PM CDT #

This tool, if run correctly and successfully, appears to have a gap of some kind. It may fail to account for a close() call.

Please refer to https://issues.apache.org/jira/browse/WICKET-4458

I am suspecting that there is still a GlassFish issue, so this appears to be quite critical to pinpoint such a GlassFish issue.

Posted by guest on March 29, 2012 at 08:59 PM CDT #

I disagree that the case has been made that this is GlassFish issue or that the tool has a gap.

Please see my comment on the WICKET issue referenced earlier.

Posted by Tim on March 30, 2012 at 04:52 AM CDT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

News and musings on the technology I work on at Oracle.

The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today