Addressing locked JAR problems

Many GlassFish users have had problems undeploying or redeploying certain applications on Windows because one or more JAR files in the app remain open and, on Windows, locked. Windows does not allow open files to be deleted or renamed, and the deployment logic in GlassFish would attempt both. GlassFish would try to rename the working directory it creates for the application so it can restore it in case of an error during a redeployment. And during undeployment GlassFish tries to delete the working directory for the app. The result is that undeployments or redeployments would fail with a variety of errors. There are plenty of places to find discussions along these lines. Issue 539 is just one, and the GlassFish user forum is laced with topics on this problem. The workaround to this problem was to restart the server to release the locked files – a very painful and time-consuming recourse indeed!

The good news is that Hong has checked in a set of coordinated changes that she and I have made that should improve this situation markedly for Windows users doing archive deployments. (These changes do not affect directory deployments.)

When you deploy an application archive (an EAR, a WAR, an EJB JAR, or an app client JAR) GlassFish creates a private directory for that app with the same name as the application name you assign (xxx in this example). Among other things, any library JAR files such as foo.jar in the archive are stored in the corresponding relative place in this directory.

[Prior to the recent changes, if the directory already existed during a deployment or redeployment, GlassFish used to attempt to rename it from xxx to xxx_old. This would fail if a file anywhere in the directory was locked. GlassFish no longer attempts that rename operation.

Also, during undeployment GlassFish would try to delete the directory and all its contents. This would fail and the undeployment would abort if any file could not be removed, and future attempts to deploy the app again would fail because GlassFish could not clean out the pre-existing directory. Errors deleting files from the app's directory no longer cause the operations to fail.]

During redeployment GlassFish tries to clean out the existing directory so it can populate it from the new version of the archive. In the absence of locked JARs already in the directory, all the old files are deleted and the directory is completely replaced with the contents from the application archive.

Suppose now that foo.jar from an earlier deployment is locked. It stays in place after GlassFish has tried to clean out the directory for the redeployment. So does the containing directory structure, by the way. The inability to clean out the directory no longer causes a failure. So during the redeployment GlassFish overwrites the contents of the old foo.jar file (blue outline around foo.jar) with the new (yellow) contents of the foo.jar file from the archive.

There is a possible problem.

Here, the developer has removed foo.jar from the application – it is no longer in the archive being deployed. But the foo.jar file in the app directory was locked and stayed in-place when GlassFish tried to clean out the directory. Because foo.jar no longer appears in the new archive, GlassFish does not have new content to write into the existing foo.jar. So foo.jar remains in the application directory with its old contents (blue) and may interfere with the application.

This can happen either during redeployment of xxx or if you undeploy xxx and then deploy a different application with the same name.

To make you aware of this potential problem, GlassFish keeps track of which left-over files it overwrites. If it finds any left-over file that it does not overwrite, GlassFish reports this in the server.log and in the status message sent back to the client, such as the asadmin utility. This way, at least you know about the possible risks. We have some good ideas for handling that case as well but haven't had a chance to implement anything along those lines yet.

For the first three cases above GlassFish does not log any warnings or errors to the client or in server.log. If you want to, you can set UTIL logging to FINE to see details about left-over files, attempts to delete them, etc. in server.log. If you set DEPLOYMENT logging to FINE then GlassFish will write messages to server.log about all left-over files, whether they are overwritten or not.

Here is an example of the message asadmin might display in the last case:

Command deploy executed successfully with following warning messages: WARNING: The following file(s) were left(locked) from the previous deployment; they were not updated during this deployment and could interfere with the application:

GlassFish is telling you that the deployment itself succeeded but that the left-over JARs are still there and could cause your application to behave differently from what you expect.

For those of you who have wrestled with this problem, these recent changes go a long way toward resolving it.


Hi Tim, Any chance it solves the issue #354 as well ? Rgs, JB

Posted by bjb on October 11, 2006 at 10:30 PM CDT #

Yes, these changes should also clean up Issue 354. I am going to update each issue and forum topic related to JAR locking with a link to this blog entry this morning. You just beat me to it! - Tim

Posted by Tim Quinn on October 11, 2006 at 11:59 PM CDT #

Tim, I'm one of the people that spent some time complaining on the forums about this...thank you, thank you, THANK you for pursuing this issue and potentially eliminating the bug. Having to start/stop Glassfish repetitively during development is a huge waste of time (and doesn't always fix the problem.) This will have a huge impact on my company using Glassfish/SJAS in production going forward. I thought I had read that including xercesImpl.jar (2.8.0) in my app would fix this? I was not able to get it to work w/ that method, unfortunately.

Posted by V. Jenks on October 12, 2006 at 02:54 AM CDT #

Well done! I've yet to get my hands on a copy of the new version but this problem has stood in the way of us moving 50+ Windows developers to use SJSAS (From JBoss). The problem was with Struts - but JBoss found a nice way to handle the problem, and now SJSAS does too. Eagerly looking forward to downloading the update when its available!

Posted by Corey Johnston on October 13, 2006 at 07:04 PM CDT #

I've just tried out the most recent nightly build for this new feature. It works like a gem - and all of our redeployment issues on Windows are now gone. Any chance this will make its way into GlassFish UR1 (AS 9.1 PE ?)

Posted by Corey Johnston on October 17, 2006 at 12:43 PM CDT #

Hi, Corey. Glad to hear that the changes are working so well for you. About the release labeling... Sun Java Systems Application Server 9.0 corresponds to GlassFish V1. SJS AS 9.1 corresponds to GF V2. I don't think we'll be backporting these changes to GF V1 UR1 (a.k.a. SJS AS 9.0UR1). UR1 is basically in final form at this point and I don't think there are plans to open it up again. - Tim

Posted by Tim Quinn on October 17, 2006 at 01:18 PM CDT #

We have had this problem off and on for years... Thanks for the fix!

Posted by junkgui on November 21, 2006 at 03:34 AM CST #

It gets really frustrating to stop and start the server and manually delete the jars everytime we deploy the app.
This fix on SJSAS 9.1 is great and I guess it makes life lot more easier.
We still use SJSAS 8.2 and can't be upgrading to SJSAS 9.1 any time soon.
Is there any patch available that fixes this issue for SJSAS 8.2 ??


Posted by kanynomour on November 05, 2007 at 06:53 AM CST #

can we please have this for Sun App Server 8.2?

Posted by Anil on January 13, 2008 at 04:30 PM CST #

Hi, Anil.

As I mentioned to kanynomour, the best course of action would be for you to work through Sun Services to report the problem and ask for a solution for 8.2. I do not know if kanynomour did so but if more people ask for a fix formally through the support organization the more likely a fix will become available.

- Tim

Posted by Tim Quinn on January 14, 2008 at 12:53 AM CST #

If you create your struts2 project using this module in netbeans and deploy/run the project on app server (incl. problematic 8.2) do not get locked jar issues.

I dont know how they did it (maybe newer version of struts.jar)....btw shouldnt apache struts guys be fixing this issue of locked jars?


Posted by Bhaarat on April 09, 2008 at 05:06 AM CDT #

URL for the module

Posted by Bhaarat on April 09, 2008 at 05:08 AM CDT #

Post a Comment:
  • HTML Syntax: NOT allowed

News and musings on the technology I work on at Oracle.

The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.


« April 2014