[JBEWS-92] Tomcat with mod_cluster refuses to shut down Created: 30/Aug/12  Updated: 17/Jan/13  Resolved: 22/Oct/12

Status: Closed
Project: JBoss Enterprise Web Server
Component/s: None
Affects Version/s: EWS 2.0.0 ER8, EWS 2.0.0 ER10, EWS 2.0.0 ER11
Fix Version/s: TBD EWS

Type: Bug Priority: Blocker
Reporter: Michal Karm Babacek Assignee: Permaine Cheung
Resolution: Done Votes: 0
Labels: tomcat
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Tomcat6 confirmed


Attachments: Text File after_shutdown.txt     Text File before_shutdown.txt     File full-thread-dump     Java Archive File mod_cluster-container-tomcat6.jar     File patch.JBPAPP-9788    
Issue Links:
Related
Bugzilla Update:
Perform

 Description   

Follow BZ 849924



 Comments   
Comment by Jean-Frederic Clere [ 30/Aug/12 ]

I can't reproduce it.

Comment by Jean-Frederic Clere [ 19/Sep/12 ]

Please retest I can't reproduce the issue.

Comment by Michal Karm Babacek [ 19/Sep/12 ]

I have a live box for you with hanging Tomcat.

It might be just some silly error related to the way it was started/configured:

/root/workspace/jboss-ews-2.0/tomcat6/bin/startup.sh 870367228

Note: The weird number is actually only a label with which I can track and possibly kill this particular instance (grepping ps aux...).
Also note it is being started with roots privileges.

Now I have:

root      2094  4.4  8.0 3431968 400412 pts/0  Sl   14:58   0:27 /root/jdk1.7.0_last//bin/java -d64 -Djava.util.logging.config.file=/root/workspace/jboss-ews-2.0/tomcat6/conf/logging.properties -Djava.library.path=/root/workspace/jboss-ews-2.0/tomcat6/lib -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/root/workspace/jboss-ews-2.0/tomcat6/endorsed -classpath /root/workspace/jboss-ews-2.0/tomcat6/bin/bootstrap.jar -Dcatalina.base=/root/workspace/jboss-ews-2.0/tomcat6 -Dcatalina.home=/root/workspace/jboss-ews-2.0/tomcat6 -Djava.io.tmpdir=/root/workspace/jboss-ews-2.0/tomcat6/temp org.apache.catalina.startup.Bootstrap 870367228 start

Attempts like /root/workspace/jboss-ews-2.0/tomcat6/bin/shutdown.sh are fruitless:

Using CATALINA_BASE:   /root/workspace/jboss-ews-2.0/tomcat6
Using CATALINA_HOME:   /root/workspace/jboss-ews-2.0/tomcat6
Using CATALINA_TMPDIR: /root/workspace/jboss-ews-2.0/tomcat6/temp
Using JRE_HOME:        /root/jdk1.7.0_last/
Using CLASSPATH:       /root/workspace/jboss-ews-2.0/tomcat6/bin/bootstrap.jar
Sep 19, 2012 3:09:37 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop: 
java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
	at java.net.Socket.connect(Socket.java:579)
	at java.net.Socket.connect(Socket.java:528)
	at java.net.Socket.<init>(Socket.java:425)
	at java.net.Socket.<init>(Socket.java:208)
	at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:422)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:338)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:416)

I will give you access to that box via email.

Comment by Michal Karm Babacek [ 19/Sep/12 ]

Lowering to Major as the reproducibility is still questionable.

Comment by Jean-Frederic Clere [ 20/Sep/12 ]

I have reproduced it with a tomcat6 installed in $HOME/tc6.0.x/output/build with the following script:
+++
$HOME/tc6.0.x/output/build
while true
do
bin/startup.sh
while true
do
curl -v --cookie JSESSIONID=data.jvm1 http://localhost:8080/examples/jsp/num/numguess.jsp 2>&1 >/dev/null
if [ $? -eq 0 ]; then
bin/shutdown.sh
break
fi
done
sleep 2
ps -ef | grep tc6.0.x | grep -v grep
if [ $? -eq 0 ]; then
break
fi
done
+++

Comment by Jean-Frederic Clere [ 20/Sep/12 ]

the whole problem it due to the fact tomcat6 doesn't send:
lifecycleEvent: StandardServer[8005] before_destroy
lifecycleEvent: StandardServer[8005] after_destroy
I think a fix is to use the:
lifecycleEvent: StandardServer[8005] after_stop to destroy the listener doing the advertise.

Comment by Michal Karm Babacek [ 20/Sep/12 ]

Hmm, I have just tried the attached jar out with the following result:

  • having tomcat open in terminal and hitting ^C shuts it down all right
  • using ./shutdown.sh still leads to the old

    Using CATALINA_BASE:   /root/workspace/jboss-ews-2.0/tomcat-6-19
    Using CATALINA_HOME:   /root/workspace/jboss-ews-2.0/tomcat-6-19
    Using CATALINA_TMPDIR: /root/workspace/jboss-ews-2.0/tomcat-6-19/temp
    Using JRE_HOME:        /root/jdk1.7.0_last/
    Using CLASSPATH:       /root/workspace/jboss-ews-2.0/tomcat-6-19/bin/bootstrap.jar
    Sep 20, 2012 10:21:24 AM org.apache.catalina.startup.Catalina stopServer
    SEVERE: Catalina.stop: 
    java.net.ConnectException: Connection refused
    	at java.net.PlainSocketImpl.socketConnect(Native Method)
    	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
    	at java.net.Socket.connect(Socket.java:579)
    	at java.net.Socket.connect(Socket.java:528)
    	at java.net.Socket.<init>(Socket.java:425)
    	at java.net.Socket.<init>(Socket.java:208)
    	at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:422)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:601)
    	at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:338)
    	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:416)
     

I will come back to it later...

Comment by Jean-Frederic Clere [ 20/Sep/12 ]

patch for the problem.

Comment by Libor Fuka [ 25/Sep/12 ]

Its still bug in ER11

Comment by Jean-Frederic Clere [ 25/Sep/12 ]

Sure but could someone test the stuff with the patch please.

Comment by Libor Fuka [ 25/Sep/12 ]

I was testing with your patch on MS Windows 2008R2 by all mod_cluster tests and it works!
We need to verify also on RHEL and Solaris, but i don't expect problems.

The patch is for Tomcat6 and how about Tomcat7 ? Can you check Tomcat7 sources if there is a same problem ?

Comment by Jean-Frederic Clere [ 25/Sep/12 ]

Tomcat6 and Tomcat7 sources are completely different. It works for Tomcat7 I have checked it.
Look to the comment I made the 20.

Comment by Libor Fuka [ 25/Sep/12 ]

OK, so, we will see our testing in RHEL and Solaris

Comment by Radim Hatlapatka [ 27/Sep/12 ]

If I apply your patch, I am unable to even start the tomcat (I have tested it on RHEL 5 and 6, and also it happens on Solaris machines).

I am getting this exception in catalina.out (without patch it is started correctly):

SEVERE: Begin event threw error
java.util.ServiceConfigurationError: No org.jboss.modcluster.container.catalina.LifecycleListenerFactory service provider found.
        at org.jboss.modcluster.container.catalina.standalone.ModClusterListener$1.run(ModClusterListener.java:105)
        at org.jboss.modcluster.container.catalina.standalone.ModClusterListener$1.run(ModClusterListener.java:99)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.jboss.modcluster.container.catalina.standalone.ModClusterListener.loadFactory(ModClusterListener.java:108)
        at org.jboss.modcluster.container.catalina.standalone.ModClusterListener.<init>(ModClusterListener.java:95)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at java.lang.Class.newInstance0(Class.java:355)
        at java.lang.Class.newInstance(Class.java:308)
        at org.apache.tomcat.util.digester.ObjectCreateRule.begin(ObjectCreateRule.java:206)
        at org.apache.tomcat.util.digester.Rule.begin(Rule.java:153)
        at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1356)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:501)
        at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:179)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1343)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2756)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
        at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1642)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:524)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:582)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.util.ServiceConfigurationError: No org.jboss.modcluster.container.catalina.LifecycleListenerFactory service provider found.
        at org.jboss.modcluster.container.catalina.standalone.ModClusterListener$1.run(ModClusterListener.java:105)
        at org.jboss.modcluster.container.catalina.standalone.ModClusterListener$1.run(ModClusterListener.java:99)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.jboss.modcluster.container.catalina.standalone.ModClusterListener.loadFactory(ModClusterListener.java:108)
        at org.jboss.modcluster.container.catalina.standalone.ModClusterListener.<init>(ModClusterListener.java:95)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

Comment by Jean-Frederic Clere [ 27/Sep/12 ]

you must be doing something wrong as previous comment said it is working on windoze (and for me on Fedora17).

Comment by Jean-Frederic Clere [ 01/Oct/12 ]

Please add the patch to 1.2.2.Final

Comment by Radim Hatlapatka [ 01/Oct/12 ]

Jean-Frederic Clere even after applying the patch there are still some running tomcats remaining

Comment by Jean-Frederic Clere [ 02/Oct/12 ]

I need a thread dump or stack trace of one of those or a "easy" way to reproduce.

Comment by Ladislav Thon [ 03/Oct/12 ]

Attaching stack traces of Tomcat with the patch applied 1. before running shutdown.sh 2. after running shutdown.sh as obtained using jstack $PID. I'm sending you login informations for that particular machine via email, in case you are interested.

Comment by Jean-Frederic Clere [ 03/Oct/12 ]

The patch is not in /opt/workspace/jboss-ews-2.0/tomcat6/lib/mod_cluster-container-tomcat6.jar (the classes are from september 12). So it is not fixed in ER11

Comment by Ladislav Thon [ 03/Oct/12 ]

That's weird, the /opt/workspace/jboss-ews-2.0/tomcat6/lib/mod_cluster-container-tomcat6.jar file is the same file as the mod_cluster-container-tomcat6.jar attachment in this issue (at least according to md5sum and sha1sum)... Wrong, I was looking at a bad file. You're right, it isn't fixed in ER11.

Comment by Jean-Frederic Clere [ 03/Oct/12 ]

it should be in CR2.

Comment by Permaine Cheung [ 05/Oct/12 ]

Mladen, I've built mod_cluster-1.2.2-4.Final_redhat_2.ep6.el5
mod_cluster-1.2.2-4.Final_redhat_2.ep6.el6

Can you please update that for the other platforms? Thanks!

(I suspect you don't need the commit id in git, but in case you do, it is the jb-eap-6-rhel-6 branch of mod_cluster, commit id abe36072f0595ab6bf5e05822135bcac12e770f6)

Comment by Radim Hatlapatka [ 08/Oct/12 ]

The problem with shutting down the tomcat remains in certain cases. If I deploy too many contexts (based on mod_cluster listener property Maxcontext) it causes release of the shutdown port without actually stopping whole tomcat.

Because this is not so much a problem of shutting down as problem of releasing the shutdown port before really shutting down the tomcat server, creating new JIRA for it https://issues.jboss.org/browse/JBPAPP-10147

Comment by Jean-Frederic Clere [ 08/Oct/12 ]

looking to JBPAPP-10147 it is MODCLUSTER-325 which is a minor bug.

The clean up time is a parameter if you want a fast clean up you may have to adjust the corresponding parameter, additionally is you look to the catalina.out you will see that the clean up time out log messages in it.

Comment by Jean-Frederic Clere [ 11/Oct/12 ]

Note that the patch was missing in CR1 so please make sure it is in CR2.

Comment by Weinan Li [ 12/Oct/12 ]

Fixed in CR2.

Comment by Radim Hatlapatka [ 17/Oct/12 ]

The problem remains on Solaris machines, it looks like there wasn't applied the patch. Even after applying the patch from this JIRA the problem remains, but in CR1 it worked with this patch was applied.

Comment by Jean-Frederic Clere [ 17/Oct/12 ]

that is a weird exception:
java.io.InterruptedIOException: operation interrupted
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
at java.net.DatagramSocket.receive(DatagramSocket.java:725)
at org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl$AdvertiseListenerWorker.run(AdvertiseListenerImpl.java:354)
at java.lang.Thread.run(Thread.java:662)

I don't see why you don't get it with CR1 + patch.

Comment by Misha Ali [ 17/Oct/12 ]

Documented as a Known Issue for 2.0. Is there a known workaround to add?

Comment by Jean-Frederic Clere [ 18/Oct/12 ]

I have a fix that MUST go in CR3.

Comment by Radim Hatlapatka [ 18/Oct/12 ]

Misha Ali This issue shouldn't be in release notes, this should be fixed in CR3

Comment by Jean-Frederic Clere [ 18/Oct/12 ]

Please the tag 1.2.3.Final.

Comment by Permaine Cheung [ 18/Oct/12 ]

mod_cluster-1.2.3-1.Final_redhat_1.ep6.el6
mod_cluster-native-1.2.3-2.Final.ep6.el6

mod_cluster-1.2.3-1.Final_redhat_1.ep6.el5
mod_cluster-native-1.2.3-2.Final.ep6.el5

are built.

Comment by Weinan Li [ 22/Oct/12 ]

Resolved in CR3.

Comment by RH Bugzilla Integration [ 17/Jan/13 ]

This issue has been migrated to Bugzilla bug 900828. Please note that this JIRA issue has been closed as part of the migration and therefore you will need to check the Bugzilla issue to find the current status.

Generated at Tue Nov 20 13:48:39 EST 2018 using Jira 7.12.1#712002-sha1:609a50578ba6bc73dbf8b05dddd7c04a04b6807c.