Addressing locked JAR problems
By tjquinn on Oct 11, 2006
Many GlassFish users have had problems undeploying or redeploying certain applications on Windows because one or more JAR files in the app remain open and, on Windows, locked. Windows does not allow open files to be deleted or renamed, and the deployment logic in GlassFish would attempt both. GlassFish would try to rename the working directory it creates for the application so it can restore it in case of an error during a redeployment. And during undeployment GlassFish tries to delete the working directory for the app. The result is that undeployments or redeployments would fail with a variety of errors. There are plenty of places to find discussions along these lines. Issue 539 is just one, and the GlassFish user forum is laced with topics on this problem. The workaround to this problem was to restart the server to release the locked files – a very painful and time-consuming recourse indeed!
The good news is that Hong has checked in a set of coordinated changes that she and I have made that should improve this situation markedly for Windows users doing archive deployments. (These changes do not affect directory deployments.)
When you deploy an application archive (an EAR, a WAR, an EJB JAR, or an app client JAR) GlassFish creates a private directory for that app with the same name as the application name you assign (xxx in this example). Among other things, any library JAR files such as foo.jar in the archive are stored in the corresponding relative place in this directory.
[Prior to the recent changes, if the directory already existed during a deployment or redeployment, GlassFish used to attempt to rename it from xxx to xxx_old. This would fail if a file anywhere in the directory was locked. GlassFish no longer attempts that rename operation.
Also, during undeployment GlassFish would try to delete the directory and all its contents. This would fail and the undeployment would abort if any file could not be removed, and future attempts to deploy the app again would fail because GlassFish could not clean out the pre-existing directory. Errors deleting files from the app's directory no longer cause the operations to fail.]
During redeployment GlassFish tries to clean out the existing directory so it can populate it from the new version of the archive. In the absence of locked JARs already in the directory, all the old files are deleted and the directory is completely replaced with the contents from the application archive.
Suppose now that foo.jar from an earlier deployment is locked. It stays in place after GlassFish has tried to clean out the directory for the redeployment. So does the containing directory structure, by the way. The inability to clean out the directory no longer causes a failure. So during the redeployment GlassFish overwrites the contents of the old foo.jar file (blue outline around foo.jar) with the new (yellow) contents of the foo.jar file from the archive.
There is a possible problem.
Here, the developer has removed foo.jar from the application – it is no longer in the archive being deployed. But the foo.jar file in the app directory was locked and stayed in-place when GlassFish tried to clean out the directory. Because foo.jar no longer appears in the new archive, GlassFish does not have new content to write into the existing foo.jar. So foo.jar remains in the application directory with its old contents (blue) and may interfere with the application.
This can happen either during redeployment of xxx or if you undeploy xxx and then deploy a different application with the same name.
To make you aware of this potential problem, GlassFish keeps track of which left-over files it overwrites. If it finds any left-over file that it does not overwrite, GlassFish reports this in the server.log and in the status message sent back to the client, such as the asadmin utility. This way, at least you know about the possible risks. We have some good ideas for handling that case as well but haven't had a chance to implement anything along those lines yet.
For the first three cases above GlassFish does not log any warnings or errors to the client or in server.log. If you want to, you can set UTIL logging to FINE to see details about left-over files, attempts to delete them, etc. in server.log. If you set DEPLOYMENT logging to FINE then GlassFish will write messages to server.log about all left-over files, whether they are overwritten or not.
Here is an example of the message asadmin might display in the last case:
executed successfully with following warning messages: WARNING: The
following file(s) were left(locked) from the previous deployment;
they were not updated during this deployment and could interfere with
GlassFish is telling you that the deployment itself succeeded but that the left-over JARs are still there and could cause your application to behave differently from what you expect.
For those of you who have wrestled with this problem, these recent changes go a long way toward resolving it.