2005-03-24 21:23:20 UTC
- Tomcat Quartz Repair
- Quartz Tomcat 8
- Quartz Tomcat Shutdown Problem
- Tomcat Quartz Countertops
- Java Quartz Tomcat
The scheduler threads hang and I have to kill tomcat manually. I use tomcat-6.0.16 spring 2.5.1 and java 1.6. When I manually call the bean destroy method by looking up the bean everything work fine.
![Quartz Quartz](https://git.oschina.net/uploads/images/2017/0607/173420_38eb81e2_606783.png)
![Tomcat quartzite Tomcat quartzite](/uploads/1/3/7/7/137799254/488338881.jpg)
Environment
Webobjects 5.2.3
Eclipse 2.1.3
Tomcat 5.0
Quartz Scheduler
I have an application that will run every 15 minutes. I am using the
quartz scheduler to perform this task. In my Application.java, I have
added the following code to start the job:
public void didFinishLaunching() {
super.didFinishLaunching();
try {
this.startJobScheduler();
}
catch (SchedulerException e){
log4JLogger.error(e);
}
catch (ParseException e){
log4JLogger.error(e);
}
}
/*
* Attempting to use the Quartz scheduler
*/
public void startJobScheduler() throws SchedulerException,
ParseException {
SchedulerFactory schedFact = new
org.quartz.impl.StdSchedulerFactory();
Scheduler sched = schedFact.getScheduler();
sched.start();
JobDetail jobDetail = new
JobDetail('UberBlogsJob',Scheduler.DEFAULT_GROUP,UberBlogsJob.class);
Trigger trigger = new
CronTrigger('HalfHourlyTrigger',Scheduler.DEFAULT_GROUP,'1 0/1 * * *
?');
trigger = new
SimpleTrigger('HalfHourlyTrigger',Scheduler.DEFAULT_GROUP);
sched.scheduleJob(jobDetail,trigger);
}
The job starts up and runs successfully. My problem occurs when I try to
shutdown Tomcat. No error occurs but it does not stop the scheduler
process. I have attempted to insert code into the terminate() method of
Application.java but this is never called. I have to manually stop the
process before I can do a new build or restart Tomcat. Does anybody know
of a way to stop the process when tomcat is shutdown?
Thanks,
Tom Tessitore
Skip to end of metadataGo to start of metadataWebobjects 5.2.3
Eclipse 2.1.3
Tomcat 5.0
Quartz Scheduler
I have an application that will run every 15 minutes. I am using the
quartz scheduler to perform this task. In my Application.java, I have
added the following code to start the job:
public void didFinishLaunching() {
super.didFinishLaunching();
try {
this.startJobScheduler();
}
catch (SchedulerException e){
log4JLogger.error(e);
}
catch (ParseException e){
log4JLogger.error(e);
}
}
/*
* Attempting to use the Quartz scheduler
*/
public void startJobScheduler() throws SchedulerException,
ParseException {
SchedulerFactory schedFact = new
org.quartz.impl.StdSchedulerFactory();
Scheduler sched = schedFact.getScheduler();
sched.start();
JobDetail jobDetail = new
JobDetail('UberBlogsJob',Scheduler.DEFAULT_GROUP,UberBlogsJob.class);
Trigger trigger = new
CronTrigger('HalfHourlyTrigger',Scheduler.DEFAULT_GROUP,'1 0/1 * * *
?');
trigger = new
SimpleTrigger('HalfHourlyTrigger',Scheduler.DEFAULT_GROUP);
sched.scheduleJob(jobDetail,trigger);
}
The job starts up and runs successfully. My problem occurs when I try to
shutdown Tomcat. No error occurs but it does not stop the scheduler
process. I have attempted to insert code into the terminate() method of
Application.java but this is never called. I have to manually stop the
process before I can do a new build or restart Tomcat. Does anybody know
of a way to stop the process when tomcat is shutdown?
Thanks,
Tom Tessitore
For some time Tomcat has had some means of protection against memory leaks when stopping or redeploying applications. This page tries to list them, and shows the situations where leaks can be detected and fixed.
Starting with tomcat 6.0.25, the manager webapp has a new 'Find Leaks' button. When triggered, it displays a list of webapps (their context path) that have been stopped (this includes undeployed and redeployed ones) but whose classloader failed to be GCed.
If a leaking webapp is redeployed several times, it will appear as many times as it actually leaked.
Caution: This diagnosis calls
System.gc()
which may not be desirable in production environments.When a webapp execution is stopped (this encompassed redeploy and undeploy), tomcat tries to detect and fix leaks.
Starting with tomcat 6.0.24, messages are logged to indicate the kind of leak that was detected.
Leak cause | Detected by tomcat | Fixed by tomcat | Possible enhancements |
>=6.0.24 | >= 7.0.6 | ||
>=6.0.24 | >= 7.0.6 | ||
no | >= 7.0.6 | ||
>=6.0.24 | >= 7.0.6 | ||
>=6.0.24 | In 6.0.24-6.0.26 TimerThread are stopped but it may lead to problems. Optional from 6.0.27 with the clearReferencesStopTimerThreads flag. Other threads may be stopped with the clearReferencesStopThreads flag, but it's unsafe. | Fix the application to stop the thread when the application is stopped | |
>=6.0.24 | In 6.0.24-6.0.26 TimerThread are stopped but it may lead to problems. Optional from 6.0.27 with the clearReferencesStopTimerThreads flag. Other threads may be stopped with the clearReferencesStopThreads flag, but it's unsafe. | Fix the offending code (set the correct CCL when spawning the thread) | |
no | >=6.0.24 pre-spawns some known offenders | ||
no | > 6.0.? . Disabled by default with tomcat 7 | ||
> 6.0.? | |||
> 6.0.? | > 6.0.? | ||
> 6.0.? |
ThreadLocal leaks
Classloader leaks because of uncleaned
ThreadLocal
variables are quite common. Depending on the use cases, they can be detected or not.Custom ThreadLocal class
Suppose we have the following 3 classes in our webapp :
If the
LeakingServlet
is invoked at least once and the Thread that served it is not stopped, then we created a classloader leak !The leak is caused because we have a custom class for the
ThreadLocal
instance, and also a custom class for the value bound to the Thread. Actually the important thing is that both classes were loaded by the webapp classloader.Hopefully tomcat 6.0.24 can detect the leak when the application is stopped: each Thread in the JVM is examined, and the internal structures of the Thread and
ThreadLocal
classes are introspected to see if either the ThreadLocal
instance or the value bound to it were loaded by the WebAppClassLoader
of the application being stopped.In this particular case, the leak is detected and a message is logged. Tomcat 6.0.24 to 6.0.26 modify internal structures of the JDK (
ThreadLocalMap
) to remove the reference to the ThreadLocal
instance, but this is unsafe (see #48895) so that it became optional and disabled by default from 6.0.27. The national song book. Starting with Tomcat 7.0.6, the threads of the pool are renewed so that the leak is safely fixed. Note: this particular leak was actually already cured by previous versions of tomcat 6, because static references of classes loaded by the webappclassloader are nullified (see later).
Webapp class instance as ThreadLocal value
Suppose that we have the following class in the common classpath (for instance in a jar in tomcat/lib) :
![Quartz Quartz](https://git.oschina.net/uploads/images/2017/0607/173420_38eb81e2_606783.png)
And those 2 classes in the webapp :
Tomcat Quartz Repair
If the servlet is invoked at least once, the webapp classloader would not be GCed when the app is stopped: since the classloader of
ThreadScopedHolder
is the common classloader, it remains forever which is as expected. But its ThreadLocal
instance has a value bound to it (for the non-terminated thread(s) that served the sevlet), which is an instance of a class loaded by the webapp classloader..Here again, tomcat >=6.0.24 will detect the leak :
Webapp class instance indirectly held through a ThreadLocal value
Suppose we have the same
ThreadScopedHolder
class (in the common classloader) and MyCounter
class in the webapp, but with the following servlet :We have more or less the same kind of leak as the previous one, but this time tomcat does not detect the leak when stopping the application. The problem is that when it inspects the entries of
ThreadLocalMap
, it checks whether either the key or the value is an instance of a class loaded by the webapp classloader. Here the key is an instance of ThreadLocal
, and the value is an instance of java.util.ArrayList
.The 'Find leaks' button in tomcat manager will report the leak when asked :
But it does not give any clue about what caused the leak, we would need to make a heapdump and analyse it with some tool like Eclipse MAT.
Tomcat 7.0.6 and later fix this leak by renewing threads in the pool.
ThreadLocal pseudo-leak
Suppose we have the same
MyCounter
class as above (in the webapp) and the following servlet :Notice that the
ThreadLocal
instance is referenced through an instance variable, not a static one.Sun's implementation of
ThreadLocal
(and WeakHashMap
) too is such that ThreadLocalMap
entries whose key is GCed are not immediately removed. (The key is a weak reference to the ThreadLocal
instance, see java.lang.ThreadLocal.ThreadLocalMap.Entry<T> in JDK 5/6
. And there's no daemon thread waiting on a ReferenceQueue
). Instead, it's only during subsequent uses of ThreadLocal
features that each Thread removes the abandoned ThreadLocalMap.Entry
entries (see ThreadLocalMap.expungeStaleEntries()
.![Tomcat quartzite Tomcat quartzite](/uploads/1/3/7/7/137799254/488338881.jpg)
If many threads were used to serve our leaking webapp, but after we stop it only a couple of threads are enough to serve other webapps, one could have some threads that are no longer used, waiting for some work. Since those threads are blocked, they have no interaction with their
ThreadLocalMap
(i.e. there's no ThreadLocal
value bound to them or removed), so that there's no opportunity to expungeStaleEntries()
.Tomcat 6.0.24-6.0.26 'speeds up' the removal of stale entries (and thus fixes the pseudo-leak), by calling
expungeStaleEntries()
for each thread that has some stale entries. Since it's not thread-safe, it has been made optional and disabled by default from 6.0.27.Tomcat 7.0.6 and later fix the problem by renewing threads in the pool.
Threads ContextClassLoader
Threads spawned by webapps
If a webapp creates a thread, by default its context classloader is set to the one of the parent thread (the thread that created the new thread). In a webapp, this parent thread is one of tomcat worker threads, whose context classloader is set to the webapp classloader when it executes webapp code.
Furthermore, the spawned thread may be executing (or blocked in) some code that involves classes loaded by the webapp, thus preventing the webapp classloader from being collected.
So, if the spawned thread is not properly terminated when the application is stopped, the webapp classloader will leak because of the strong reference held by the spawned thread.
Quartz Tomcat 8
Example :
Here, when the app is stopped, the webapp classloader is still referenced by the spawned thread both through its context classloader and its current call stack (the anonymous Thread subclass is loaded by the webapp classloader).
When stopping an application, tomcat checks the context classloader of every Thread, and if it is the same as the app being stopped, it logs the following message :
Now, if we uncomment the line
leakingThread.setContextClassLoader(null);
in the above example, tomcat (6.0.24) no longer detect the leak when the application is stopped because the spawned thread context classloader is no longer the webapp's. (the 'Find leaks' feature in the manager will report it though)Threads spawned by classes loaded by the common classloader
Suppose, we have the Commons Pool library in the classpath of the server (e.g. the jar is in tomcat/lib), and the following servlet :
The call to
GenericObjectPool.setTimeBetweenEvictionRunsMillis)
actually starts or reuses a java.lang.Timer
shared between all GenericObjectPool
instances. As long as a pool is running and at least one pool is using the timer eviction feature, the Timer lives.If there's no other webapp using commons-pool, there's no leak : when we stop the webapp, the servlet is stopped and the pool is closed. If it was the only pool in use, the Timer thread is also stopped and there's no leak.
Now, imagine that there are 2 webapps using commons-pool with the timer eviction feature (imagine the above servlet is deployed in 2 webapps A and B). Suppose webapp A is deployed, then B. Since the commons-pool jar is shared between both webapps, only one Timer thread is spawned, with 2
TimerTask
, one for each webapp, to handle the eviction of each pool instance.Then, if we stop webapp A, boom it's leaking! the Timer thread has its context classloader set to the
WebAppClassLoader
of webapp A. This is somehow a bug of commons-pool, but tomcat 6.0.24 tries to help :So the leak is fixed, but unfortunately there's a side effect : it broke webapp B eviction timer. That's why stopping
TimerThread
has been made optional from 6.0.27.Threads spawned by JRE classes
Just like third-party libraries may spawn threads that provoke leaks, some JRE classes also spawn threads that inherit from the current class loader and thus provoke leaks.
Instead of trying to stop such threads, tomcat prefers to force the creation of such threads when the container is started, before webapps are started. The
JreMemoryLeakPreventionListener
does it for a few known offenders in the JRE.static class variables
When an app is stopped, Tomcat (even before 6.0.24) nullifies the value of all static class variables of classes loaded by the
WebAppClassLoader
. In some cases, it may fix a classloader leak (for example because of a custom ThreadLocal
class, see above), but even if we still have a leak, it may decrease the amount of memory lost:Imagine a class with the following variable :
Normally, the 1MB buffer should be freed when the app is stopped, but only if the classloader itself can be garbage-collected. Since there are still possibilities to have a leak of the classloader, clearing the BUFFER variable allows to recover 1MB of memory.
LogFactory
to be completed
JavaBean Introspector cache
Tomcat calls
java.beans.Introspector.flushCaches();
when an app is stoppped.JDBC driver registration
If a webapp contains a JDBC driver (e.g. in WEB-INF/lib), the driver will be registered with the
DriverManager
when it is first used. When the application is stopped, the driver should be deregistered with DriverManager
to avoid a classloader leak. Since applications usually forget this, tomcat helps by deregistering the driver.Quartz Tomcat Shutdown Problem
RMI target
to be completed
References
- Eclipse Memory Analysis Tool
Tomcat Quartz Countertops
Related issues
Java Quartz Tomcat
- 49159 - Improve ThreadLocal memory leak clean-up
- Sun bug 4957990 - In some cases the Server JVM fails to collect classloaders. According to this page it should have been fixed with java 6u16 but actually it was not. It seems to be fixed with 6u21 (documented here and verified by the author of this wiki page).
- Sun bug 6916498 - An exception can keep a classloader in memory if the stack trace that was recorded when it was created contains a reference to one of its classes. Fixes were done in Tomcat for its own classes that had this issue (see BZ 50460), but some library or JRE code may still create a leak that is undetected by tools because of this JVM bug. See also BZ 53936 for a workaround that you can implement if you are unable to fix a buggy library.